From knepley at gmail.com Tue Oct 1 07:01:14 2024 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 1 Oct 2024 08:01:14 -0400 Subject: [petsc-users] ISView() in PETSc 3.22 In-Reply-To: References: Message-ID: On Mon, Sep 30, 2024 at 10:15?PM Adrian Croucher wrote: > hi, I am testing my (Fortran) code on PETSc 3.22 and have got it to > build. However I am getting some unusual new behaviour when I write an > IS to an HDF5 file using ISView(). > > The attached minimal example shows the issue. It creates a simple > 10-element IS and writes it to HDF5. With previous versions of PETSc > this would give me a 10x1 dataset containing the values 0 - 9, as expected. > > When I run it with PETSc 3.22 (in serial), I again get the expected > values written on stdout, so it looks like the IS itself is correct. But > in the HDF5 file I get a 1x3 dataset containing the values (10,1,0). > > Has something changed here? > Yes. We now compress IS data sets by default. You can turn it off using -is_view_compress 0. I am not sure what the best way to manage this is, but it makes a huge difference in file size for checkpointing. Thanks, Matt > - Adrian > > -- > Dr Adrian Croucher > Senior Research Fellow > Department of Engineering Science > Waipapa Taumata Rau / University of Auckland, New Zealand > email: a.croucher at auckland.ac.nz > tel: +64 (0)9 923 4611 > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!ejndUdBS4n9snwyvw_Y8jvSAdi6ga-LyUR7oasMzcbtNb3GQ_PbPcBenOALwXgN3HaoVuMUgTtv9bCTCkhv7$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From mmolinos at us.es Tue Oct 1 09:56:26 2024 From: mmolinos at us.es (MIGUEL MOLINOS PEREZ) Date: Tue, 1 Oct 2024 14:56:26 +0000 Subject: [petsc-users] Ghost particles for DMSWARM (or similar) In-Reply-To: <7698089E-0909-429F-9E89-9D1AD636ACBF@us.es> References: <8FBAC7A5-B6AE-4B21-8FEB-52BE1C04A265@us.es> <1B9B1277-9566-444C-9DA8-7ED17684FE01@us.es> <24337E11-33D2-4FFC-89E2-12520AD487FF@us.es> <681E96BD-62A4-4566-A13E-E034B2F19D54@us.es> <7698089E-0909-429F-9E89-9D1AD636ACBF@us.es> Message-ID: <562B2CA0-7462-4AF2-AAF4-E44DDD00B222@us.es> Thank you Matt, it works! The implementation is straightforward: - 1? Define the paddle regions using DMGetLocalBoundingBox with the background DMDA mesh as an auxiliary mesh for the domain-partitioning. - 2? Create an integer to count the local number of particles to be used as ghost particle for other processors (N_ghost). One particle can be counted more than one time. At the same time, fill two arrays: - one with the index of the "main particle? (local particle), - and the other with the target rank of the "main particle?. - 3? Create the new particles using DMSwarmAddNPoints - 4? Fill the new particles with the information of the ?main particle? but set the internal variable DMSwarmField_rank with the target rank. - 5? Call DMSwarmMigrate(*,PETSC_TRUE). Therefore, we send the ghost particles to the corresponding processors and we delete them from the original processor. - 6? Do stuff? - 7? Delete ghost particles. This is very easy, we just have to call DMSwarmRemovePoint N_ghost times. I think this can be easily implemented as closed routine for the DMSwarm class. The remaining question is: how to do the communication between the ?original" particle and the ghost particles? For instance, if we update some particle variable (locally) inside of a SNES context, this same variable should be updated in the ghost particles at the other processors. Thanks, Miguel PS: Hope this helps someone in the future :-) On Sep 27, 2024, at 10:50?AM, MIGUEL MOLINOS PEREZ wrote: Thank you Matt, let me give it try. Miguel On Sep 27, 2024, at 3:44?AM, Matthew Knepley wrote: On Thu, Sep 26, 2024 at 7:18?PM MIGUEL MOLINOS PEREZ > wrote: I see, you mean: Create the ghost particles at the local cell with the same properties as particle 1 (duplicate the original particle) but different value DMSwarmField_rank. Then, call DMSwarmMigrate(*,PETSC_FALSE) so we do the migration and delete the local copies of the particle 1. Right? Yep. I think it will work, from what I know about BASIC. Thanks, Matt Thanks, Miguel On Sep 26, 2024, at 11:09?PM, Matthew Knepley > wrote: On Thu, Sep 26, 2024 at 11:20?AM MIGUEL MOLINOS PEREZ > wrote: Thank you Matt. Okey, let me have a careful look to the DMSwarmMigrate_Push_Basic implementation to see if there is some workaround. The idea of adding new particles is interesting. However, in that case, we need to initialize the new (ghost) particles using the fields of the ?real? particle, right? This can be done using something like: VecGhostUpdateBegin(Vec globalout,InsertMode ADD_VALUES, ScatterMode SCATTER_REVERSE); VecGhostUpdateEnd(Vec globalout,InsertMode ADD_VALUES, ScatterMode SCATTER_REVERSE); for the particle fields (?). I think we can just copy from the local particle. For example, suppose I decide that particle 1 should go to rank 5, 12, and 27. Then I first set p1.rank = 5, then I add two new particles with the same values as particle 1, but with rank = 12 and 27. Then when I call migrate, it will move these three particles to the correct processes, and delete the original particles and the copies from the local set. Thanks, Matt Thanks, Miguel On Sep 26, 2024, at 3:53?PM, Matthew Knepley > wrote: On Thu, Sep 26, 2024 at 6:31?AM MIGUEL MOLINOS PEREZ > wrote: Hi Matt et al, I?ve been working on the scheme that you proposed to create ghost particles (atoms in my case), and it works! With a couple of caveats: -1? In general the overlap particles will be migrate from their own rank to more than one neighbor rank, this is specially relevant for those located close to the corners. Therefore, you'll need to call DMSwarmMigrate several times (27 times for 3D cells), during the migration process. That is terrible. Let's just fix DMSwarmMigrate to have a mode that sends the particle to all overlapping neighbors at once. It can't be that hard. -2? You need to set DMSWARM_MIGRATE_BASIC. Otherwise the proposed algorithm will not work at all! Oh, I should have thought of that. Sorry. I can help code up that extension. Can you take a quick look at the BASIC code? Right now, we just use the rank attached to the particle to send it. We could have an arrays of ranks, but that seems crazy, and would blow up particle storage. How about just adding new particles with the other ranks right before migration? Thanks, Matt Hope this helps to other folks! I have a follow-up question about periodic bcc on this context, should I open a new thread of keep posting here? Thanks, Miguel On Aug 7, 2024, at 4:22?AM, MIGUEL MOLINOS PEREZ > wrote: Thanks Matt, I think I'll start by making a small program as a proof of concept. Then, if it works I'll implement it in my code and I'll be happy to share it too :-) Miguel On Aug 4, 2024, at 3:30?AM, Matthew Knepley > wrote: On Fri, Aug 2, 2024 at 7:15?PM MIGUEL MOLINOS PEREZ > wrote: Thanks again Matt, that makes a lot more sense !! Just to check that we are on the same page. You are saying: 1. create a field define a field called "owner rank" for each particle. 2. Identify the phantom particles and modify the internal variable defined by the DMSwarmField_rank variable. 3. Call DMSwarmMigrate(*,PETSC_FALSE), do the calculations using the new local vector including the ghost particles. 4. Then, once the calculations are done, rename the DMSwarmField_rank variable using the "owner rank" variable and call DMSwarmMigrate(*,PETSC_FALSE) once again. I don't think we need this last step. We can just remove those ghost particles for the next step I think. Thanks, Matt Thank you, Miguel On Aug 2, 2024, at 5:33?PM, Matthew Knepley > wrote: On Fri, Aug 2, 2024 at 11:15?AM MIGUEL MOLINOS PEREZ > wrote: Thank you Matt for your time, What you describe seems to me the ideal approach. 1) Add a particle field 'ghost' that identifies ghost vs owned particles. I think it needs options OWNED, OVERLAP, and GHOST This means, locally, I need to allocate Nlocal + ghost particles (duplicated) for my model? I would do it another way. I would allocate the particles with no overlap and set them up. Then I would identify the halo particles, mark them as OVERLAP, call DMSwarmMigrate(), and mark the migrated particles as GHOST, then unmark the OVERLAP particles. Shoot! That marking will not work since we cannot tell the difference between particles we received and particles we sent. Okay, instead of the `ghost` field we need an `owner rank` field. So then we 1) Setup the non-overlapping particles 2) Identify the halo particles 3) Change the `rank`, but not the `owner rank` 4) Call DMSwarmMigrate() Now we can identify ghost particles by the `owner rank` If that so, how to do the communication between the ghost particles living in the rank i and their ?real? counterpart in the rank j. Algo, as an alternative, what about: 1) Use an IS tag which contains, for each rank, a list of the global index of the neighbors particles outside of the rank. 2) Use VecCreateGhost to create a new vector which contains extra local space for the ghost components of the vector. 3) Use VecScatterCreate, VecScatterBegin, and VecScatterEnd to do the transference of data between a vector obtained with DMSwarmCreateGlobalVectorFromField 4) Do necessary computations using the vectors created with VecCreateGhost. This is essentially what Migrate() does. I was trying to reuse the code. Thanks, Matt Thanks, Miguel On Aug 2, 2024, at 8:58?AM, Matthew Knepley > wrote: On Thu, Aug 1, 2024 at 4:40?PM MIGUEL MOLINOS PEREZ > wrote: This Message Is From an External Sender This message came from outside your organization. Dear all, I am implementing a Molecular Dynamics (MD) code using the DMSWARM interface. In the MD simulations we evaluate on each particle (atoms) some kind of scalar functional using data from the neighbouring atoms. My problem lies in the parallel implementation of the model, because sometimes, some of these neighbours lie on a different processor. This is usually solved by using ghost particles. A similar approach (with nodes instead) is already implemented for other PETSc mesh structures like DMPlexConstructGhostCells. Unfortunately, I don't see this kind of constructs for DMSWARM. Am I missing something? I this could be done by applying a buffer region by exploiting the background DMDA mesh that I already use to do domain decomposition. Then using the buffer region of each cell to locate the ghost particles and finally using VecCreateGhost. Is this feasible? Or is there an easier approach using other PETSc functions. This is feasible, but it would be good to develop a set of best practices, since we have been mainly focused on the case of non-redundant particles. Here is how I think I would do what you want. 1) Add a particle field 'ghost' that identifies ghost vs owned particles. I think it needs options OWNED, OVERLAP, and GHOST 2) At some interval identify particles that should be sent to other processes as ghosts. I would call these "overlap particles". The determination seems application specific, so I would leave this determination to the user right now. We do two things to these particles a) Mark chosen particles as OVERLAP b) Change rank to process we are sending to 3) Call DMSwarmMigrate with PETSC_FALSE for the particle deletion flag 4) Mark OVERLAP particles as GHOST when they arrive There is one problem in the above algorithm. It does not allow sending particles to multiple ranks. We would have to do this in phases right now, or make a small adjustment to the interface allowing replication of particles when a set of ranks is specified. THanks, Matt Thank you, Miguel -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!ablD9pb2fiyrlqlMgBzXeFAeumTEYAKq4LK7cZ088w-bJQczl7dQ4JLZw-CF4che3Dn6Jrms-cjdvYK6FRJIIA$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!ablD9pb2fiyrlqlMgBzXeFAeumTEYAKq4LK7cZ088w-bJQczl7dQ4JLZw-CF4che3Dn6Jrms-cjdvYK6FRJIIA$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!ablD9pb2fiyrlqlMgBzXeFAeumTEYAKq4LK7cZ088w-bJQczl7dQ4JLZw-CF4che3Dn6Jrms-cjdvYK6FRJIIA$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!ablD9pb2fiyrlqlMgBzXeFAeumTEYAKq4LK7cZ088w-bJQczl7dQ4JLZw-CF4che3Dn6Jrms-cjdvYK6FRJIIA$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!ablD9pb2fiyrlqlMgBzXeFAeumTEYAKq4LK7cZ088w-bJQczl7dQ4JLZw-CF4che3Dn6Jrms-cjdvYK6FRJIIA$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!ablD9pb2fiyrlqlMgBzXeFAeumTEYAKq4LK7cZ088w-bJQczl7dQ4JLZw-CF4che3Dn6Jrms-cjdvYK6FRJIIA$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave.mayhem23 at gmail.com Tue Oct 1 10:22:59 2024 From: dave.mayhem23 at gmail.com (Dave May) Date: Tue, 1 Oct 2024 08:22:59 -0700 Subject: [petsc-users] Ghost particles for DMSWARM (or similar) In-Reply-To: <562B2CA0-7462-4AF2-AAF4-E44DDD00B222@us.es> References: <8FBAC7A5-B6AE-4B21-8FEB-52BE1C04A265@us.es> <1B9B1277-9566-444C-9DA8-7ED17684FE01@us.es> <24337E11-33D2-4FFC-89E2-12520AD487FF@us.es> <681E96BD-62A4-4566-A13E-E034B2F19D54@us.es> <7698089E-0909-429F-9E89-9D1AD636ACBF@us.es> <562B2CA0-7462-4AF2-AAF4-E44DDD00B222@us.es> Message-ID: Hi Miguel, On Tue 1. Oct 2024 at 07:56, MIGUEL MOLINOS PEREZ wrote: > Thank you Matt, it works! > > The implementation is straightforward: > - 1? Define the paddle regions using DMGetLocalBoundingBox with the > background DMDA mesh as an auxiliary mesh for the domain-partitioning. > - 2? Create an integer to count the local number of particles to be used > as ghost particle for other processors (N_ghost). One particle can be > counted more than one time. At the same time, fill two arrays: > - one with the index of the "main particle? (local particle), > - and the other with the target rank of the "main particle?. > - 3? Create the new particles using DMSwarmAddNPoints > - 4? Fill the new particles with the information of the ?main particle? > but set the internal variable DMSwarmField_rank with the target rank. > - 5? Call DMSwarmMigrate(*,PETSC_TRUE). Therefore, we send the ghost > particles to the corresponding processors and we delete them from the > original processor. > - 6? Do stuff? > - 7? Delete ghost particles. This is very easy, we just have to call > DMSwarmRemovePoint N_ghost times. > > I think this can be easily implemented as closed routine for the DMSwarm > class. > > The remaining question is: how to do the communication between the > ?original" particle and the ghost particles? For instance, if we update > some particle variable (locally) inside of a SNES context, this same > variable should be updated in the ghost particles at the other processors. > I think what you are asking about is an operation similar to VecGhostUpdate{Begin,End}(). In the case of a DMSwarm I?m not sure how to define the InsertMode = ADD_VALUES? Some swarm fields do not make sense to be added. INSERT_VALUES is fine. One solution might be to have something like this DMSwarmCollectViewUpdateGhostOwners(DM dm, InsertMode mode, PetscInt nfields, const char *fieldNames[]); where one can specify the insert mode and the fields on which the insert mode will apply. Would something like that work? Cheers, Dave > PS: Hope this helps someone in the future :-) > > > On Sep 27, 2024, at 10:50?AM, MIGUEL MOLINOS PEREZ wrote: > > Thank you Matt, let me give it try. > > Miguel > > On Sep 27, 2024, at 3:44?AM, Matthew Knepley wrote: > > On Thu, Sep 26, 2024 at 7:18?PM MIGUEL MOLINOS PEREZ > wrote: > >> I see, you mean: >> >> Create the ghost particles at the local cell with the same properties as >> particle 1 (duplicate the original particle) but different value >> DMSwarmField_rank. Then, call DMSwarmMigrate(*,PETSC_FALSE) so we do the >> migration and delete the local copies of the particle 1. Right? >> > > Yep. I think it will work, from what I know about BASIC. > > Thanks, > > Matt > > >> Thanks, >> Miguel >> >> On Sep 26, 2024, at 11:09?PM, Matthew Knepley wrote: >> >> On Thu, Sep 26, 2024 at 11:20?AM MIGUEL MOLINOS PEREZ >> wrote: >> >>> Thank you Matt. >>> >>> Okey, let me have a careful look to the DMSwarmMigrate_Push_Basic implementation >>> to see if there is some workaround. >>> >>> The idea of adding new particles is interesting. However, in that case, >>> we need to initialize the new (ghost) particles using the fields of the >>> ?real? particle, right? This can be done using something like: >>> >>> VecGhostUpdateBegin (Vec globalout,InsertMode ADD_VALUES , ScatterMode SCATTER_REVERSE );VecGhostUpdateEnd (Vec globalout,InsertMode ADD_VALUES , ScatterMode SCATTER_REVERSE ); >>> >>> for the particle fields (?). >>> >> >> I think we can just copy from the local particle. For example, suppose I >> decide that particle 1 should go to rank 5, 12, and 27. Then >> I first set p1.rank = 5, then I add two new particles with the same >> values as particle 1, but with rank = 12 and 27. Then when I call migrate, >> it will move these three particles to the correct processes, and delete the >> original particles and the copies from the local set. >> >> Thanks, >> >> Matt >> >> >>> Thanks, >>> Miguel >>> >>> >>> On Sep 26, 2024, at 3:53?PM, Matthew Knepley wrote: >>> >>> On Thu, Sep 26, 2024 at 6:31?AM MIGUEL MOLINOS PEREZ >>> wrote: >>> >>>> Hi Matt et al, >>>> >>>> I?ve been working on the scheme that you proposed to create ghost >>>> particles (atoms in my case), and it works! With a couple of caveats: >>>> -1? In general the overlap particles will be migrate from their own >>>> rank to more than one neighbor rank, this is specially relevant for those >>>> located close to the corners. Therefore, you'll need to call DMSwarmMigrate >>>> several times (27 times for 3D cells), during the migration process. >>>> >>> >>> That is terrible. Let's just fix DMSwarmMigrate to have a mode that >>> sends the particle to all overlapping neighbors at once. It can't be that >>> hard. >>> >>> >>>> -2? You need to set DMSWARM_MIGRATE_BASIC. Otherwise the proposed >>>> algorithm will not work at all! >>>> >>> >>> Oh, I should have thought of that. Sorry. >>> >>> I can help code up that extension. Can you take a quick look at the >>> BASIC code? Right now, we just use the rank attached to the particle >>> to send it. We could have an arrays of ranks, but that seems crazy, and >>> would blow up particle storage. How about just adding new particles >>> with the other ranks right before migration? >>> >>> Thanks, >>> >>> Matt >>> >>> >>>> Hope this helps to other folks! >>>> >>>> I have a follow-up question about periodic bcc on this context, should >>>> I open a new thread of keep posting here? >>>> >>>> Thanks, >>>> Miguel >>>> >>>> On Aug 7, 2024, at 4:22?AM, MIGUEL MOLINOS PEREZ >>>> wrote: >>>> >>>> Thanks Matt, I think I'll start by making a small program as a proof of >>>> concept. Then, if it works I'll implement it in my code and I'll be happy >>>> to share it too :-) >>>> >>>> Miguel >>>> >>>> On Aug 4, 2024, at 3:30?AM, Matthew Knepley wrote: >>>> >>>> On Fri, Aug 2, 2024 at 7:15?PM MIGUEL MOLINOS PEREZ >>>> wrote: >>>> >>>>> Thanks again Matt, that makes a lot more sense !! >>>>> >>>>> Just to check that we are on the same page. You are saying: >>>>> >>>>> 1. create a field define a field called "owner rank" for each >>>>> particle. >>>>> >>>>> 2. Identify the phantom particles and modify the internal variable >>>>> defined by the DMSwarmField_rank variable. >>>>> >>>>> 3. Call DMSwarmMigrate(*,PETSC_FALSE), do the calculations using the >>>>> new local vector including the ghost particles. >>>>> >>>>> 4. Then, once the calculations are done, rename the DMSwarmField_rank >>>>> variable using the "owner rank" variable and call >>>>> DMSwarmMigrate(*,PETSC_FALSE) once again. >>>>> >>>> >>>> I don't think we need this last step. We can just remove those ghost >>>> particles for the next step I think. >>>> >>>> Thanks, >>>> >>>> Matt >>>> >>>> >>>>> Thank you, >>>>> Miguel >>>>> >>>>> >>>>> On Aug 2, 2024, at 5:33?PM, Matthew Knepley wrote: >>>>> >>>>> On Fri, Aug 2, 2024 at 11:15?AM MIGUEL MOLINOS PEREZ >>>>> wrote: >>>>> >>>>>> Thank you Matt for your time, >>>>>> >>>>>> What you describe seems to me the ideal approach. >>>>>> >>>>>> 1) Add a particle field 'ghost' that identifies ghost vs owned >>>>>> particles. I think it needs options OWNED, OVERLAP, and GHOST >>>>>> >>>>>> This means, locally, I need to allocate Nlocal + ghost particles >>>>>> (duplicated) for my model? >>>>>> >>>>> >>>>> I would do it another way. I would allocate the particles with no >>>>> overlap and set them up. Then I would identify the halo particles, mark >>>>> them as OVERLAP, call DMSwarmMigrate(), and mark the migrated particles as >>>>> GHOST, then unmark the OVERLAP particles. Shoot! That marking will not work >>>>> since we cannot tell the difference between particles we received and >>>>> particles we sent. Okay, instead of the `ghost` field we need an `owner >>>>> rank` field. So then we >>>>> >>>>> 1) Setup the non-overlapping particles >>>>> >>>>> 2) Identify the halo particles >>>>> >>>>> 3) Change the `rank`, but not the `owner rank` >>>>> >>>>> 4) Call DMSwarmMigrate() >>>>> >>>>> Now we can identify ghost particles by the `owner rank` >>>>> >>>>> >>>>>> If that so, how to do the communication between the ghost particles >>>>>> living in the rank i and their ?real? counterpart in the rank j. >>>>>> >>>>>> Algo, as an alternative, what about: >>>>>> 1) Use an IS tag which contains, for each rank, a list of the global >>>>>> index of the neighbors particles outside of the rank. >>>>>> 2) Use VecCreateGhost to create a new vector which contains extra >>>>>> local space for the ghost components of the vector. >>>>>> 3) Use VecScatterCreate, VecScatterBegin, and VecScatterEnd to do >>>>>> the transference of data between a vector obtained with >>>>>> DMSwarmCreateGlobalVectorFromField >>>>>> 4) Do necessary computations using the vectors created with >>>>>> VecCreateGhost. >>>>>> >>>>> >>>>> This is essentially what Migrate() does. I was trying to reuse the >>>>> code. >>>>> >>>>> Thanks, >>>>> >>>>> Matt >>>>> >>>>> >>>>>> Thanks, >>>>>> Miguel >>>>>> >>>>>> On Aug 2, 2024, at 8:58?AM, Matthew Knepley >>>>>> wrote: >>>>>> >>>>>> On Thu, Aug 1, 2024 at 4:40?PM MIGUEL MOLINOS PEREZ >>>>>> wrote: >>>>>> >>>>>>> This Message Is From an External Sender >>>>>>> This message came from outside your organization. >>>>>>> >>>>>>> >>>>>>> Dear all, >>>>>>> >>>>>>> I am implementing a Molecular Dynamics (MD) code using the DMSWARM interface. In the MD simulations we evaluate on each particle (atoms) some kind of scalar functional using data from the neighbouring atoms. My problem lies in the parallel implementation of the model, because sometimes, some of these neighbours lie on a different processor. >>>>>>> >>>>>>> This is usually solved by using ghost particles. A similar approach (with nodes instead) is already implemented for other PETSc mesh structures like DMPlexConstructGhostCells. Unfortunately, I don't see this kind of constructs for DMSWARM. Am I missing something? >>>>>>> >>>>>>> I this could be done by applying a buffer region by exploiting the background DMDA mesh that I already use to do domain decomposition. Then using the buffer region of each cell to locate the ghost particles and finally using VecCreateGhost. Is this feasible? Or is there an easier approach using other PETSc functions. >>>>>>> >>>>>>> >>>>>> This is feasible, but it would be good to develop a set of best >>>>>> practices, since we have been mainly focused on the case of non-redundant >>>>>> particles. Here is how I think I would do what you want. >>>>>> >>>>>> 1) Add a particle field 'ghost' that identifies ghost vs owned >>>>>> particles. I think it needs options OWNED, OVERLAP, and GHOST >>>>>> >>>>>> 2) At some interval identify particles that should be sent to other >>>>>> processes as ghosts. I would call these "overlap particles". The >>>>>> determination >>>>>> seems application specific, so I would leave this determination >>>>>> to the user right now. We do two things to these particles >>>>>> >>>>>> a) Mark chosen particles as OVERLAP >>>>>> >>>>>> b) Change rank to process we are sending to >>>>>> >>>>>> 3) Call DMSwarmMigrate with PETSC_FALSE for the particle deletion flag >>>>>> >>>>>> 4) Mark OVERLAP particles as GHOST when they arrive >>>>>> >>>>>> There is one problem in the above algorithm. It does not allow >>>>>> sending particles to multiple ranks. We would have to do this >>>>>> in phases right now, or make a small adjustment to the interface >>>>>> allowing replication of particles when a set of ranks is specified. >>>>>> >>>>>> THanks, >>>>>> >>>>>> Matt >>>>>> >>>>>> >>>>>>> Thank you, >>>>>>> Miguel >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> -- >>>>>> What most experimenters take for granted before they begin their >>>>>> experiments is infinitely more interesting than any results to which their >>>>>> experiments lead. >>>>>> -- Norbert Wiener >>>>>> >>>>>> https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!bZXXf6uG-_sIhXmwCjiE6lnmtykfj5x1NmdfSKglwmtTzvFaM04Mt9F1ZU8HPt4Uc5pSoVVeWA7KbLQNFFlAECyeEhTc$ >>>>>> >>>>>> >>>>>> >>>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their >>>>> experiments is infinitely more interesting than any results to which their >>>>> experiments lead. >>>>> -- Norbert Wiener >>>>> >>>>> https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!bZXXf6uG-_sIhXmwCjiE6lnmtykfj5x1NmdfSKglwmtTzvFaM04Mt9F1ZU8HPt4Uc5pSoVVeWA7KbLQNFFlAECyeEhTc$ >>>>> >>>>> >>>>> >>>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>>> https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!bZXXf6uG-_sIhXmwCjiE6lnmtykfj5x1NmdfSKglwmtTzvFaM04Mt9F1ZU8HPt4Uc5pSoVVeWA7KbLQNFFlAECyeEhTc$ >>>> >>>> >>>> >>>> >>>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!bZXXf6uG-_sIhXmwCjiE6lnmtykfj5x1NmdfSKglwmtTzvFaM04Mt9F1ZU8HPt4Uc5pSoVVeWA7KbLQNFFlAECyeEhTc$ >>> >>> >>> >>> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!bZXXf6uG-_sIhXmwCjiE6lnmtykfj5x1NmdfSKglwmtTzvFaM04Mt9F1ZU8HPt4Uc5pSoVVeWA7KbLQNFFlAECyeEhTc$ >> >> >> >> > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!bZXXf6uG-_sIhXmwCjiE6lnmtykfj5x1NmdfSKglwmtTzvFaM04Mt9F1ZU8HPt4Uc5pSoVVeWA7KbLQNFFlAECyeEhTc$ > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mmolinos at us.es Tue Oct 1 10:56:26 2024 From: mmolinos at us.es (MIGUEL MOLINOS PEREZ) Date: Tue, 1 Oct 2024 15:56:26 +0000 Subject: [petsc-users] Ghost particles for DMSWARM (or similar) In-Reply-To: References: <8FBAC7A5-B6AE-4B21-8FEB-52BE1C04A265@us.es> <1B9B1277-9566-444C-9DA8-7ED17684FE01@us.es> <24337E11-33D2-4FFC-89E2-12520AD487FF@us.es> <681E96BD-62A4-4566-A13E-E034B2F19D54@us.es> <7698089E-0909-429F-9E89-9D1AD636ACBF@us.es> <562B2CA0-7462-4AF2-AAF4-E44DDD00B222@us.es> Message-ID: <5C293345-E026-436B-B4D0-E5DC109A0701@us.es> Hi Dave, Would something like that work? Yes, this should work! Any idea on where to look so I can try to implement it myself? Best, Miguel On Oct 1, 2024, at 5:22?PM, Dave May wrote: Hi Miguel, On Tue 1. Oct 2024 at 07:56, MIGUEL MOLINOS PEREZ > wrote: Thank you Matt, it works! The implementation is straightforward: - 1? Define the paddle regions using DMGetLocalBoundingBox with the background DMDA mesh as an auxiliary mesh for the domain-partitioning. - 2? Create an integer to count the local number of particles to be used as ghost particle for other processors (N_ghost). One particle can be counted more than one time. At the same time, fill two arrays: - one with the index of the "main particle? (local particle), - and the other with the target rank of the "main particle?. - 3? Create the new particles using DMSwarmAddNPoints - 4? Fill the new particles with the information of the ?main particle? but set the internal variable DMSwarmField_rank with the target rank. - 5? Call DMSwarmMigrate(*,PETSC_TRUE). Therefore, we send the ghost particles to the corresponding processors and we delete them from the original processor. - 6? Do stuff? - 7? Delete ghost particles. This is very easy, we just have to call DMSwarmRemovePoint N_ghost times. I think this can be easily implemented as closed routine for the DMSwarm class. The remaining question is: how to do the communication between the ?original" particle and the ghost particles? For instance, if we update some particle variable (locally) inside of a SNES context, this same variable should be updated in the ghost particles at the other processors. I think what you are asking about is an operation similar to VecGhostUpdate{Begin,End}(). In the case of a DMSwarm I?m not sure how to define the InsertMode = ADD_VALUES? Some swarm fields do not make sense to be added. INSERT_VALUES is fine. One solution might be to have something like this DMSwarmCollectViewUpdateGhostOwners(DM dm, InsertMode mode, PetscInt nfields, const char *fieldNames[]); where one can specify the insert mode and the fields on which the insert mode will apply. Would something like that work? Cheers, Dave PS: Hope this helps someone in the future :-) On Sep 27, 2024, at 10:50?AM, MIGUEL MOLINOS PEREZ > wrote: Thank you Matt, let me give it try. Miguel On Sep 27, 2024, at 3:44?AM, Matthew Knepley > wrote: On Thu, Sep 26, 2024 at 7:18?PM MIGUEL MOLINOS PEREZ > wrote: I see, you mean: Create the ghost particles at the local cell with the same properties as particle 1 (duplicate the original particle) but different value DMSwarmField_rank. Then, call DMSwarmMigrate(*,PETSC_FALSE) so we do the migration and delete the local copies of the particle 1. Right? Yep. I think it will work, from what I know about BASIC. Thanks, Matt Thanks, Miguel On Sep 26, 2024, at 11:09?PM, Matthew Knepley > wrote: On Thu, Sep 26, 2024 at 11:20?AM MIGUEL MOLINOS PEREZ > wrote: Thank you Matt. Okey, let me have a careful look to the DMSwarmMigrate_Push_Basic implementation to see if there is some workaround. The idea of adding new particles is interesting. However, in that case, we need to initialize the new (ghost) particles using the fields of the ?real? particle, right? This can be done using something like: VecGhostUpdateBegin(Vec globalout,InsertMode ADD_VALUES, ScatterMode SCATTER_REVERSE); VecGhostUpdateEnd(Vec globalout,InsertMode ADD_VALUES, ScatterMode SCATTER_REVERSE); for the particle fields (?). I think we can just copy from the local particle. For example, suppose I decide that particle 1 should go to rank 5, 12, and 27. Then I first set p1.rank = 5, then I add two new particles with the same values as particle 1, but with rank = 12 and 27. Then when I call migrate, it will move these three particles to the correct processes, and delete the original particles and the copies from the local set. Thanks, Matt Thanks, Miguel On Sep 26, 2024, at 3:53?PM, Matthew Knepley > wrote: On Thu, Sep 26, 2024 at 6:31?AM MIGUEL MOLINOS PEREZ > wrote: Hi Matt et al, I?ve been working on the scheme that you proposed to create ghost particles (atoms in my case), and it works! With a couple of caveats: -1? In general the overlap particles will be migrate from their own rank to more than one neighbor rank, this is specially relevant for those located close to the corners. Therefore, you'll need to call DMSwarmMigrate several times (27 times for 3D cells), during the migration process. That is terrible. Let's just fix DMSwarmMigrate to have a mode that sends the particle to all overlapping neighbors at once. It can't be that hard. -2? You need to set DMSWARM_MIGRATE_BASIC. Otherwise the proposed algorithm will not work at all! Oh, I should have thought of that. Sorry. I can help code up that extension. Can you take a quick look at the BASIC code? Right now, we just use the rank attached to the particle to send it. We could have an arrays of ranks, but that seems crazy, and would blow up particle storage. How about just adding new particles with the other ranks right before migration? Thanks, Matt Hope this helps to other folks! I have a follow-up question about periodic bcc on this context, should I open a new thread of keep posting here? Thanks, Miguel On Aug 7, 2024, at 4:22?AM, MIGUEL MOLINOS PEREZ > wrote: Thanks Matt, I think I'll start by making a small program as a proof of concept. Then, if it works I'll implement it in my code and I'll be happy to share it too :-) Miguel On Aug 4, 2024, at 3:30?AM, Matthew Knepley > wrote: On Fri, Aug 2, 2024 at 7:15?PM MIGUEL MOLINOS PEREZ > wrote: Thanks again Matt, that makes a lot more sense !! Just to check that we are on the same page. You are saying: 1. create a field define a field called "owner rank" for each particle. 2. Identify the phantom particles and modify the internal variable defined by the DMSwarmField_rank variable. 3. Call DMSwarmMigrate(*,PETSC_FALSE), do the calculations using the new local vector including the ghost particles. 4. Then, once the calculations are done, rename the DMSwarmField_rank variable using the "owner rank" variable and call DMSwarmMigrate(*,PETSC_FALSE) once again. I don't think we need this last step. We can just remove those ghost particles for the next step I think. Thanks, Matt Thank you, Miguel On Aug 2, 2024, at 5:33?PM, Matthew Knepley > wrote: On Fri, Aug 2, 2024 at 11:15?AM MIGUEL MOLINOS PEREZ > wrote: Thank you Matt for your time, What you describe seems to me the ideal approach. 1) Add a particle field 'ghost' that identifies ghost vs owned particles. I think it needs options OWNED, OVERLAP, and GHOST This means, locally, I need to allocate Nlocal + ghost particles (duplicated) for my model? I would do it another way. I would allocate the particles with no overlap and set them up. Then I would identify the halo particles, mark them as OVERLAP, call DMSwarmMigrate(), and mark the migrated particles as GHOST, then unmark the OVERLAP particles. Shoot! That marking will not work since we cannot tell the difference between particles we received and particles we sent. Okay, instead of the `ghost` field we need an `owner rank` field. So then we 1) Setup the non-overlapping particles 2) Identify the halo particles 3) Change the `rank`, but not the `owner rank` 4) Call DMSwarmMigrate() Now we can identify ghost particles by the `owner rank` If that so, how to do the communication between the ghost particles living in the rank i and their ?real? counterpart in the rank j. Algo, as an alternative, what about: 1) Use an IS tag which contains, for each rank, a list of the global index of the neighbors particles outside of the rank. 2) Use VecCreateGhost to create a new vector which contains extra local space for the ghost components of the vector. 3) Use VecScatterCreate, VecScatterBegin, and VecScatterEnd to do the transference of data between a vector obtained with DMSwarmCreateGlobalVectorFromField 4) Do necessary computations using the vectors created with VecCreateGhost. This is essentially what Migrate() does. I was trying to reuse the code. Thanks, Matt Thanks, Miguel On Aug 2, 2024, at 8:58?AM, Matthew Knepley > wrote: On Thu, Aug 1, 2024 at 4:40?PM MIGUEL MOLINOS PEREZ > wrote: This Message Is From an External Sender This message came from outside your organization. Dear all, I am implementing a Molecular Dynamics (MD) code using the DMSWARM interface. In the MD simulations we evaluate on each particle (atoms) some kind of scalar functional using data from the neighbouring atoms. My problem lies in the parallel implementation of the model, because sometimes, some of these neighbours lie on a different processor. This is usually solved by using ghost particles. A similar approach (with nodes instead) is already implemented for other PETSc mesh structures like DMPlexConstructGhostCells. Unfortunately, I don't see this kind of constructs for DMSWARM. Am I missing something? I this could be done by applying a buffer region by exploiting the background DMDA mesh that I already use to do domain decomposition. Then using the buffer region of each cell to locate the ghost particles and finally using VecCreateGhost. Is this feasible? Or is there an easier approach using other PETSc functions. This is feasible, but it would be good to develop a set of best practices, since we have been mainly focused on the case of non-redundant particles. Here is how I think I would do what you want. 1) Add a particle field 'ghost' that identifies ghost vs owned particles. I think it needs options OWNED, OVERLAP, and GHOST 2) At some interval identify particles that should be sent to other processes as ghosts. I would call these "overlap particles". The determination seems application specific, so I would leave this determination to the user right now. We do two things to these particles a) Mark chosen particles as OVERLAP b) Change rank to process we are sending to 3) Call DMSwarmMigrate with PETSC_FALSE for the particle deletion flag 4) Mark OVERLAP particles as GHOST when they arrive There is one problem in the above algorithm. It does not allow sending particles to multiple ranks. We would have to do this in phases right now, or make a small adjustment to the interface allowing replication of particles when a set of ranks is specified. THanks, Matt Thank you, Miguel -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!ZaGTGzrAiCWH2bjuzt8ctb2mKZNr3c6POEI393i7Guk8etZWMAfdgzaAXP6Pewvzlgm-rmfEGgIAL50c3Hge8g$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!ZaGTGzrAiCWH2bjuzt8ctb2mKZNr3c6POEI393i7Guk8etZWMAfdgzaAXP6Pewvzlgm-rmfEGgIAL50c3Hge8g$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!ZaGTGzrAiCWH2bjuzt8ctb2mKZNr3c6POEI393i7Guk8etZWMAfdgzaAXP6Pewvzlgm-rmfEGgIAL50c3Hge8g$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!ZaGTGzrAiCWH2bjuzt8ctb2mKZNr3c6POEI393i7Guk8etZWMAfdgzaAXP6Pewvzlgm-rmfEGgIAL50c3Hge8g$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!ZaGTGzrAiCWH2bjuzt8ctb2mKZNr3c6POEI393i7Guk8etZWMAfdgzaAXP6Pewvzlgm-rmfEGgIAL50c3Hge8g$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!ZaGTGzrAiCWH2bjuzt8ctb2mKZNr3c6POEI393i7Guk8etZWMAfdgzaAXP6Pewvzlgm-rmfEGgIAL50c3Hge8g$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From mmolinos at us.es Tue Oct 1 11:05:00 2024 From: mmolinos at us.es (MIGUEL MOLINOS PEREZ) Date: Tue, 1 Oct 2024 16:05:00 +0000 Subject: [petsc-users] Ghost particles for DMSWARM (or similar) In-Reply-To: <5C293345-E026-436B-B4D0-E5DC109A0701@us.es> References: <8FBAC7A5-B6AE-4B21-8FEB-52BE1C04A265@us.es> <1B9B1277-9566-444C-9DA8-7ED17684FE01@us.es> <24337E11-33D2-4FFC-89E2-12520AD487FF@us.es> <681E96BD-62A4-4566-A13E-E034B2F19D54@us.es> <7698089E-0909-429F-9E89-9D1AD636ACBF@us.es> <562B2CA0-7462-4AF2-AAF4-E44DDD00B222@us.es> <5C293345-E026-436B-B4D0-E5DC109A0701@us.es> Message-ID: <37036553-254F-4532-8D55-0020CACB5F08@us.es> Sorry, I forgot to add: You think VecScatterCreate is the way to go? Thanks, Miguel On Oct 1, 2024, at 5:56?PM, MIGUEL MOLINOS PEREZ wrote: Hi Dave, Would something like that work? Yes, this should work! Any idea on where to look so I can try to implement it myself? Best, Miguel On Oct 1, 2024, at 5:22?PM, Dave May wrote: Hi Miguel, On Tue 1. Oct 2024 at 07:56, MIGUEL MOLINOS PEREZ > wrote: Thank you Matt, it works! The implementation is straightforward: - 1? Define the paddle regions using DMGetLocalBoundingBox with the background DMDA mesh as an auxiliary mesh for the domain-partitioning. - 2? Create an integer to count the local number of particles to be used as ghost particle for other processors (N_ghost). One particle can be counted more than one time. At the same time, fill two arrays: - one with the index of the "main particle? (local particle), - and the other with the target rank of the "main particle?. - 3? Create the new particles using DMSwarmAddNPoints - 4? Fill the new particles with the information of the ?main particle? but set the internal variable DMSwarmField_rank with the target rank. - 5? Call DMSwarmMigrate(*,PETSC_TRUE). Therefore, we send the ghost particles to the corresponding processors and we delete them from the original processor. - 6? Do stuff? - 7? Delete ghost particles. This is very easy, we just have to call DMSwarmRemovePoint N_ghost times. I think this can be easily implemented as closed routine for the DMSwarm class. The remaining question is: how to do the communication between the ?original" particle and the ghost particles? For instance, if we update some particle variable (locally) inside of a SNES context, this same variable should be updated in the ghost particles at the other processors. I think what you are asking about is an operation similar to VecGhostUpdate{Begin,End}(). In the case of a DMSwarm I?m not sure how to define the InsertMode = ADD_VALUES? Some swarm fields do not make sense to be added. INSERT_VALUES is fine. One solution might be to have something like this DMSwarmCollectViewUpdateGhostOwners(DM dm, InsertMode mode, PetscInt nfields, const char *fieldNames[]); where one can specify the insert mode and the fields on which the insert mode will apply. Would something like that work? Cheers, Dave PS: Hope this helps someone in the future :-) On Sep 27, 2024, at 10:50?AM, MIGUEL MOLINOS PEREZ > wrote: Thank you Matt, let me give it try. Miguel On Sep 27, 2024, at 3:44?AM, Matthew Knepley > wrote: On Thu, Sep 26, 2024 at 7:18?PM MIGUEL MOLINOS PEREZ > wrote: I see, you mean: Create the ghost particles at the local cell with the same properties as particle 1 (duplicate the original particle) but different value DMSwarmField_rank. Then, call DMSwarmMigrate(*,PETSC_FALSE) so we do the migration and delete the local copies of the particle 1. Right? Yep. I think it will work, from what I know about BASIC. Thanks, Matt Thanks, Miguel On Sep 26, 2024, at 11:09?PM, Matthew Knepley > wrote: On Thu, Sep 26, 2024 at 11:20?AM MIGUEL MOLINOS PEREZ > wrote: Thank you Matt. Okey, let me have a careful look to the DMSwarmMigrate_Push_Basic implementation to see if there is some workaround. The idea of adding new particles is interesting. However, in that case, we need to initialize the new (ghost) particles using the fields of the ?real? particle, right? This can be done using something like: VecGhostUpdateBegin(Vec globalout,InsertMode ADD_VALUES, ScatterMode SCATTER_REVERSE); VecGhostUpdateEnd(Vec globalout,InsertMode ADD_VALUES, ScatterMode SCATTER_REVERSE); for the particle fields (?). I think we can just copy from the local particle. For example, suppose I decide that particle 1 should go to rank 5, 12, and 27. Then I first set p1.rank = 5, then I add two new particles with the same values as particle 1, but with rank = 12 and 27. Then when I call migrate, it will move these three particles to the correct processes, and delete the original particles and the copies from the local set. Thanks, Matt Thanks, Miguel On Sep 26, 2024, at 3:53?PM, Matthew Knepley > wrote: On Thu, Sep 26, 2024 at 6:31?AM MIGUEL MOLINOS PEREZ > wrote: Hi Matt et al, I?ve been working on the scheme that you proposed to create ghost particles (atoms in my case), and it works! With a couple of caveats: -1? In general the overlap particles will be migrate from their own rank to more than one neighbor rank, this is specially relevant for those located close to the corners. Therefore, you'll need to call DMSwarmMigrate several times (27 times for 3D cells), during the migration process. That is terrible. Let's just fix DMSwarmMigrate to have a mode that sends the particle to all overlapping neighbors at once. It can't be that hard. -2? You need to set DMSWARM_MIGRATE_BASIC. Otherwise the proposed algorithm will not work at all! Oh, I should have thought of that. Sorry. I can help code up that extension. Can you take a quick look at the BASIC code? Right now, we just use the rank attached to the particle to send it. We could have an arrays of ranks, but that seems crazy, and would blow up particle storage. How about just adding new particles with the other ranks right before migration? Thanks, Matt Hope this helps to other folks! I have a follow-up question about periodic bcc on this context, should I open a new thread of keep posting here? Thanks, Miguel On Aug 7, 2024, at 4:22?AM, MIGUEL MOLINOS PEREZ > wrote: Thanks Matt, I think I'll start by making a small program as a proof of concept. Then, if it works I'll implement it in my code and I'll be happy to share it too :-) Miguel On Aug 4, 2024, at 3:30?AM, Matthew Knepley > wrote: On Fri, Aug 2, 2024 at 7:15?PM MIGUEL MOLINOS PEREZ > wrote: Thanks again Matt, that makes a lot more sense !! Just to check that we are on the same page. You are saying: 1. create a field define a field called "owner rank" for each particle. 2. Identify the phantom particles and modify the internal variable defined by the DMSwarmField_rank variable. 3. Call DMSwarmMigrate(*,PETSC_FALSE), do the calculations using the new local vector including the ghost particles. 4. Then, once the calculations are done, rename the DMSwarmField_rank variable using the "owner rank" variable and call DMSwarmMigrate(*,PETSC_FALSE) once again. I don't think we need this last step. We can just remove those ghost particles for the next step I think. Thanks, Matt Thank you, Miguel On Aug 2, 2024, at 5:33?PM, Matthew Knepley > wrote: On Fri, Aug 2, 2024 at 11:15?AM MIGUEL MOLINOS PEREZ > wrote: Thank you Matt for your time, What you describe seems to me the ideal approach. 1) Add a particle field 'ghost' that identifies ghost vs owned particles. I think it needs options OWNED, OVERLAP, and GHOST This means, locally, I need to allocate Nlocal + ghost particles (duplicated) for my model? I would do it another way. I would allocate the particles with no overlap and set them up. Then I would identify the halo particles, mark them as OVERLAP, call DMSwarmMigrate(), and mark the migrated particles as GHOST, then unmark the OVERLAP particles. Shoot! That marking will not work since we cannot tell the difference between particles we received and particles we sent. Okay, instead of the `ghost` field we need an `owner rank` field. So then we 1) Setup the non-overlapping particles 2) Identify the halo particles 3) Change the `rank`, but not the `owner rank` 4) Call DMSwarmMigrate() Now we can identify ghost particles by the `owner rank` If that so, how to do the communication between the ghost particles living in the rank i and their ?real? counterpart in the rank j. Algo, as an alternative, what about: 1) Use an IS tag which contains, for each rank, a list of the global index of the neighbors particles outside of the rank. 2) Use VecCreateGhost to create a new vector which contains extra local space for the ghost components of the vector. 3) Use VecScatterCreate, VecScatterBegin, and VecScatterEnd to do the transference of data between a vector obtained with DMSwarmCreateGlobalVectorFromField 4) Do necessary computations using the vectors created with VecCreateGhost. This is essentially what Migrate() does. I was trying to reuse the code. Thanks, Matt Thanks, Miguel On Aug 2, 2024, at 8:58?AM, Matthew Knepley > wrote: On Thu, Aug 1, 2024 at 4:40?PM MIGUEL MOLINOS PEREZ > wrote: This Message Is From an External Sender This message came from outside your organization. Dear all, I am implementing a Molecular Dynamics (MD) code using the DMSWARM interface. In the MD simulations we evaluate on each particle (atoms) some kind of scalar functional using data from the neighbouring atoms. My problem lies in the parallel implementation of the model, because sometimes, some of these neighbours lie on a different processor. This is usually solved by using ghost particles. A similar approach (with nodes instead) is already implemented for other PETSc mesh structures like DMPlexConstructGhostCells. Unfortunately, I don't see this kind of constructs for DMSWARM. Am I missing something? I this could be done by applying a buffer region by exploiting the background DMDA mesh that I already use to do domain decomposition. Then using the buffer region of each cell to locate the ghost particles and finally using VecCreateGhost. Is this feasible? Or is there an easier approach using other PETSc functions. This is feasible, but it would be good to develop a set of best practices, since we have been mainly focused on the case of non-redundant particles. Here is how I think I would do what you want. 1) Add a particle field 'ghost' that identifies ghost vs owned particles. I think it needs options OWNED, OVERLAP, and GHOST 2) At some interval identify particles that should be sent to other processes as ghosts. I would call these "overlap particles". The determination seems application specific, so I would leave this determination to the user right now. We do two things to these particles a) Mark chosen particles as OVERLAP b) Change rank to process we are sending to 3) Call DMSwarmMigrate with PETSC_FALSE for the particle deletion flag 4) Mark OVERLAP particles as GHOST when they arrive There is one problem in the above algorithm. It does not allow sending particles to multiple ranks. We would have to do this in phases right now, or make a small adjustment to the interface allowing replication of particles when a set of ranks is specified. THanks, Matt Thank you, Miguel -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!byP8aADeuSz41pfrGZxxlkJxopkQ6GifOfA2gTRtDBb9PKIZ3shG8z42l13mIE_5xEQbMrr49UBAuTAkH1ECBQ$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!byP8aADeuSz41pfrGZxxlkJxopkQ6GifOfA2gTRtDBb9PKIZ3shG8z42l13mIE_5xEQbMrr49UBAuTAkH1ECBQ$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!byP8aADeuSz41pfrGZxxlkJxopkQ6GifOfA2gTRtDBb9PKIZ3shG8z42l13mIE_5xEQbMrr49UBAuTAkH1ECBQ$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!byP8aADeuSz41pfrGZxxlkJxopkQ6GifOfA2gTRtDBb9PKIZ3shG8z42l13mIE_5xEQbMrr49UBAuTAkH1ECBQ$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!byP8aADeuSz41pfrGZxxlkJxopkQ6GifOfA2gTRtDBb9PKIZ3shG8z42l13mIE_5xEQbMrr49UBAuTAkH1ECBQ$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!byP8aADeuSz41pfrGZxxlkJxopkQ6GifOfA2gTRtDBb9PKIZ3shG8z42l13mIE_5xEQbMrr49UBAuTAkH1ECBQ$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave.mayhem23 at gmail.com Tue Oct 1 12:20:47 2024 From: dave.mayhem23 at gmail.com (Dave May) Date: Tue, 1 Oct 2024 10:20:47 -0700 Subject: [petsc-users] Ghost particles for DMSWARM (or similar) In-Reply-To: <5C293345-E026-436B-B4D0-E5DC109A0701@us.es> References: <8FBAC7A5-B6AE-4B21-8FEB-52BE1C04A265@us.es> <1B9B1277-9566-444C-9DA8-7ED17684FE01@us.es> <24337E11-33D2-4FFC-89E2-12520AD487FF@us.es> <681E96BD-62A4-4566-A13E-E034B2F19D54@us.es> <7698089E-0909-429F-9E89-9D1AD636ACBF@us.es> <562B2CA0-7462-4AF2-AAF4-E44DDD00B222@us.es> <5C293345-E026-436B-B4D0-E5DC109A0701@us.es> Message-ID: On Tue, 1 Oct 2024 at 08:56, MIGUEL MOLINOS PEREZ wrote: > Hi Dave, > > Would something like that work? > > Yes, this should work! Any idea on where to look so I can try to implement > it myself? > I am adding support for this right now. > > Best, > Miguel > > On Oct 1, 2024, at 5:22?PM, Dave May wrote: > > Hi Miguel, > > On Tue 1. Oct 2024 at 07:56, MIGUEL MOLINOS PEREZ wrote: > >> Thank you Matt, it works! >> >> The implementation is straightforward: >> - 1? Define the paddle regions using DMGetLocalBoundingBox with the >> background DMDA mesh as an auxiliary mesh for the domain-partitioning. >> - 2? Create an integer to count the local number of particles to be used >> as ghost particle for other processors (N_ghost). One particle can be >> counted more than one time. At the same time, fill two arrays: >> - one with the index of the "main particle? (local particle), >> - and the other with the target rank of the "main particle?. >> - 3? Create the new particles using DMSwarmAddNPoints >> - 4? Fill the new particles with the information of the ?main particle? >> but set the internal variable DMSwarmField_rank with the target rank. >> - 5? Call DMSwarmMigrate(*,PETSC_TRUE). Therefore, we send the ghost >> particles to the corresponding processors and we delete them from the >> original processor. >> - 6? Do stuff? >> - 7? Delete ghost particles. This is very easy, we just have to call >> DMSwarmRemovePoint N_ghost times. >> >> I think this can be easily implemented as closed routine for the DMSwarm >> class. >> >> The remaining question is: how to do the communication between the >> ?original" particle and the ghost particles? For instance, if we update >> some particle variable (locally) inside of a SNES context, this same >> variable should be updated in the ghost particles at the other processors. >> > > I think what you are asking about is an operation similar to > VecGhostUpdate{Begin,End}(). In the case of a DMSwarm I?m not sure how to > define the InsertMode = ADD_VALUES? Some swarm fields do not make sense to > be added. INSERT_VALUES is fine. > > One solution might be to have something like this > > DMSwarmCollectViewUpdateGhostOwners(DM dm, InsertMode mode, PetscInt > nfields, const char *fieldNames[]); > > where one can specify the insert mode and the fields on which the insert > mode will apply. > > Would something like that work? > > Cheers, > Dave > > > > > >> PS: Hope this helps someone in the future :-) >> >> >> On Sep 27, 2024, at 10:50?AM, MIGUEL MOLINOS PEREZ >> wrote: >> >> Thank you Matt, let me give it try. >> >> Miguel >> >> On Sep 27, 2024, at 3:44?AM, Matthew Knepley wrote: >> >> On Thu, Sep 26, 2024 at 7:18?PM MIGUEL MOLINOS PEREZ >> wrote: >> >>> I see, you mean: >>> >>> Create the ghost particles at the local cell with the same properties as >>> particle 1 (duplicate the original particle) but different value >>> DMSwarmField_rank. Then, call DMSwarmMigrate(*,PETSC_FALSE) so we do >>> the migration and delete the local copies of the particle 1. Right? >>> >> >> Yep. I think it will work, from what I know about BASIC. >> >> Thanks, >> >> Matt >> >> >>> Thanks, >>> Miguel >>> >>> On Sep 26, 2024, at 11:09?PM, Matthew Knepley wrote: >>> >>> On Thu, Sep 26, 2024 at 11:20?AM MIGUEL MOLINOS PEREZ >>> wrote: >>> >>>> Thank you Matt. >>>> >>>> Okey, let me have a careful look to the DMSwarmMigrate_Push_Basic implementation >>>> to see if there is some workaround. >>>> >>>> The idea of adding new particles is interesting. However, in that case, >>>> we need to initialize the new (ghost) particles using the fields of >>>> the ?real? particle, right? This can be done using something like: >>>> >>>> VecGhostUpdateBegin (Vec globalout,InsertMode ADD_VALUES , ScatterMode SCATTER_REVERSE );VecGhostUpdateEnd (Vec globalout,InsertMode ADD_VALUES , ScatterMode SCATTER_REVERSE ); >>>> >>>> for the particle fields (?). >>>> >>> >>> I think we can just copy from the local particle. For example, suppose I >>> decide that particle 1 should go to rank 5, 12, and 27. Then >>> I first set p1.rank = 5, then I add two new particles with the same >>> values as particle 1, but with rank = 12 and 27. Then when I call migrate, >>> it will move these three particles to the correct processes, and delete the >>> original particles and the copies from the local set. >>> >>> Thanks, >>> >>> Matt >>> >>> >>>> Thanks, >>>> Miguel >>>> >>>> >>>> On Sep 26, 2024, at 3:53?PM, Matthew Knepley wrote: >>>> >>>> On Thu, Sep 26, 2024 at 6:31?AM MIGUEL MOLINOS PEREZ >>>> wrote: >>>> >>>>> Hi Matt et al, >>>>> >>>>> I?ve been working on the scheme that you proposed to create ghost >>>>> particles (atoms in my case), and it works! With a couple of caveats: >>>>> -1? In general the overlap particles will be migrate from their own >>>>> rank to more than one neighbor rank, this is specially relevant for those >>>>> located close to the corners. Therefore, you'll need to call DMSwarmMigrate >>>>> several times (27 times for 3D cells), during the migration process. >>>>> >>>> >>>> That is terrible. Let's just fix DMSwarmMigrate to have a mode that >>>> sends the particle to all overlapping neighbors at once. It can't be that >>>> hard. >>>> >>>> >>>>> -2? You need to set DMSWARM_MIGRATE_BASIC. Otherwise the proposed >>>>> algorithm will not work at all! >>>>> >>>> >>>> Oh, I should have thought of that. Sorry. >>>> >>>> I can help code up that extension. Can you take a quick look at the >>>> BASIC code? Right now, we just use the rank attached to the particle >>>> to send it. We could have an arrays of ranks, but that seems crazy, and >>>> would blow up particle storage. How about just adding new particles >>>> with the other ranks right before migration? >>>> >>>> Thanks, >>>> >>>> Matt >>>> >>>> >>>>> Hope this helps to other folks! >>>>> >>>>> I have a follow-up question about periodic bcc on this context, should >>>>> I open a new thread of keep posting here? >>>>> >>>>> Thanks, >>>>> Miguel >>>>> >>>>> On Aug 7, 2024, at 4:22?AM, MIGUEL MOLINOS PEREZ >>>>> wrote: >>>>> >>>>> Thanks Matt, I think I'll start by making a small program as a proof >>>>> of concept. Then, if it works I'll implement it in my code and I'll be >>>>> happy to share it too :-) >>>>> >>>>> Miguel >>>>> >>>>> On Aug 4, 2024, at 3:30?AM, Matthew Knepley wrote: >>>>> >>>>> On Fri, Aug 2, 2024 at 7:15?PM MIGUEL MOLINOS PEREZ >>>>> wrote: >>>>> >>>>>> Thanks again Matt, that makes a lot more sense !! >>>>>> >>>>>> Just to check that we are on the same page. You are saying: >>>>>> >>>>>> 1. create a field define a field called "owner rank" for each >>>>>> particle. >>>>>> >>>>>> 2. Identify the phantom particles and modify the internal variable >>>>>> defined by the DMSwarmField_rank variable. >>>>>> >>>>>> 3. Call DMSwarmMigrate(*,PETSC_FALSE), do the calculations using the >>>>>> new local vector including the ghost particles. >>>>>> >>>>>> 4. Then, once the calculations are done, rename the DMSwarmField_rank >>>>>> variable using the "owner rank" variable and call >>>>>> DMSwarmMigrate(*,PETSC_FALSE) once again. >>>>>> >>>>> >>>>> I don't think we need this last step. We can just remove those ghost >>>>> particles for the next step I think. >>>>> >>>>> Thanks, >>>>> >>>>> Matt >>>>> >>>>> >>>>>> Thank you, >>>>>> Miguel >>>>>> >>>>>> >>>>>> On Aug 2, 2024, at 5:33?PM, Matthew Knepley >>>>>> wrote: >>>>>> >>>>>> On Fri, Aug 2, 2024 at 11:15?AM MIGUEL MOLINOS PEREZ >>>>>> wrote: >>>>>> >>>>>>> Thank you Matt for your time, >>>>>>> >>>>>>> What you describe seems to me the ideal approach. >>>>>>> >>>>>>> 1) Add a particle field 'ghost' that identifies ghost vs owned >>>>>>> particles. I think it needs options OWNED, OVERLAP, and GHOST >>>>>>> >>>>>>> This means, locally, I need to allocate Nlocal + ghost particles >>>>>>> (duplicated) for my model? >>>>>>> >>>>>> >>>>>> I would do it another way. I would allocate the particles with no >>>>>> overlap and set them up. Then I would identify the halo particles, mark >>>>>> them as OVERLAP, call DMSwarmMigrate(), and mark the migrated particles as >>>>>> GHOST, then unmark the OVERLAP particles. Shoot! That marking will not work >>>>>> since we cannot tell the difference between particles we received and >>>>>> particles we sent. Okay, instead of the `ghost` field we need an `owner >>>>>> rank` field. So then we >>>>>> >>>>>> 1) Setup the non-overlapping particles >>>>>> >>>>>> 2) Identify the halo particles >>>>>> >>>>>> 3) Change the `rank`, but not the `owner rank` >>>>>> >>>>>> 4) Call DMSwarmMigrate() >>>>>> >>>>>> Now we can identify ghost particles by the `owner rank` >>>>>> >>>>>> >>>>>>> If that so, how to do the communication between the ghost particles >>>>>>> living in the rank i and their ?real? counterpart in the rank j. >>>>>>> >>>>>>> Algo, as an alternative, what about: >>>>>>> 1) Use an IS tag which contains, for each rank, a list of the >>>>>>> global index of the neighbors particles outside of the rank. >>>>>>> 2) Use VecCreateGhost to create a new vector which contains extra >>>>>>> local space for the ghost components of the vector. >>>>>>> 3) Use VecScatterCreate, VecScatterBegin, and VecScatterEnd to do >>>>>>> the transference of data between a vector obtained with >>>>>>> DMSwarmCreateGlobalVectorFromField >>>>>>> 4) Do necessary computations using the vectors created with >>>>>>> VecCreateGhost. >>>>>>> >>>>>> >>>>>> This is essentially what Migrate() does. I was trying to reuse the >>>>>> code. >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Matt >>>>>> >>>>>> >>>>>>> Thanks, >>>>>>> Miguel >>>>>>> >>>>>>> On Aug 2, 2024, at 8:58?AM, Matthew Knepley >>>>>>> wrote: >>>>>>> >>>>>>> On Thu, Aug 1, 2024 at 4:40?PM MIGUEL MOLINOS PEREZ >>>>>>> wrote: >>>>>>> >>>>>>>> This Message Is From an External Sender >>>>>>>> This message came from outside your organization. >>>>>>>> >>>>>>>> >>>>>>>> Dear all, >>>>>>>> >>>>>>>> I am implementing a Molecular Dynamics (MD) code using the DMSWARM interface. In the MD simulations we evaluate on each particle (atoms) some kind of scalar functional using data from the neighbouring atoms. My problem lies in the parallel implementation of the model, because sometimes, some of these neighbours lie on a different processor. >>>>>>>> >>>>>>>> This is usually solved by using ghost particles. A similar approach (with nodes instead) is already implemented for other PETSc mesh structures like DMPlexConstructGhostCells. Unfortunately, I don't see this kind of constructs for DMSWARM. Am I missing something? >>>>>>>> >>>>>>>> I this could be done by applying a buffer region by exploiting the background DMDA mesh that I already use to do domain decomposition. Then using the buffer region of each cell to locate the ghost particles and finally using VecCreateGhost. Is this feasible? Or is there an easier approach using other PETSc functions. >>>>>>>> >>>>>>>> >>>>>>> This is feasible, but it would be good to develop a set of best >>>>>>> practices, since we have been mainly focused on the case of non-redundant >>>>>>> particles. Here is how I think I would do what you want. >>>>>>> >>>>>>> 1) Add a particle field 'ghost' that identifies ghost vs owned >>>>>>> particles. I think it needs options OWNED, OVERLAP, and GHOST >>>>>>> >>>>>>> 2) At some interval identify particles that should be sent to other >>>>>>> processes as ghosts. I would call these "overlap particles". The >>>>>>> determination >>>>>>> seems application specific, so I would leave this determination >>>>>>> to the user right now. We do two things to these particles >>>>>>> >>>>>>> a) Mark chosen particles as OVERLAP >>>>>>> >>>>>>> b) Change rank to process we are sending to >>>>>>> >>>>>>> 3) Call DMSwarmMigrate with PETSC_FALSE for the particle deletion >>>>>>> flag >>>>>>> >>>>>>> 4) Mark OVERLAP particles as GHOST when they arrive >>>>>>> >>>>>>> There is one problem in the above algorithm. It does not allow >>>>>>> sending particles to multiple ranks. We would have to do this >>>>>>> in phases right now, or make a small adjustment to the interface >>>>>>> allowing replication of particles when a set of ranks is specified. >>>>>>> >>>>>>> THanks, >>>>>>> >>>>>>> Matt >>>>>>> >>>>>>> >>>>>>>> Thank you, >>>>>>>> Miguel >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> -- >>>>>>> What most experimenters take for granted before they begin their >>>>>>> experiments is infinitely more interesting than any results to which their >>>>>>> experiments lead. >>>>>>> -- Norbert Wiener >>>>>>> >>>>>>> https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!Zqnh2zBsDQzOkX4E17kP-dfMuQV6iRncPCnY9VmJabssauVRMGbFE988fEWBH0MwqDqAF6K6rpQWCtGT9U31XbYP8ig3$ >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> -- >>>>>> What most experimenters take for granted before they begin their >>>>>> experiments is infinitely more interesting than any results to which their >>>>>> experiments lead. >>>>>> -- Norbert Wiener >>>>>> >>>>>> https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!Zqnh2zBsDQzOkX4E17kP-dfMuQV6iRncPCnY9VmJabssauVRMGbFE988fEWBH0MwqDqAF6K6rpQWCtGT9U31XbYP8ig3$ >>>>>> >>>>>> >>>>>> >>>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their >>>>> experiments is infinitely more interesting than any results to which their >>>>> experiments lead. >>>>> -- Norbert Wiener >>>>> >>>>> https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!Zqnh2zBsDQzOkX4E17kP-dfMuQV6iRncPCnY9VmJabssauVRMGbFE988fEWBH0MwqDqAF6K6rpQWCtGT9U31XbYP8ig3$ >>>>> >>>>> >>>>> >>>>> >>>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>>> https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!Zqnh2zBsDQzOkX4E17kP-dfMuQV6iRncPCnY9VmJabssauVRMGbFE988fEWBH0MwqDqAF6K6rpQWCtGT9U31XbYP8ig3$ >>>> >>>> >>>> >>>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!Zqnh2zBsDQzOkX4E17kP-dfMuQV6iRncPCnY9VmJabssauVRMGbFE988fEWBH0MwqDqAF6K6rpQWCtGT9U31XbYP8ig3$ >>> >>> >>> >>> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!Zqnh2zBsDQzOkX4E17kP-dfMuQV6iRncPCnY9VmJabssauVRMGbFE988fEWBH0MwqDqAF6K6rpQWCtGT9U31XbYP8ig3$ >> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave.mayhem23 at gmail.com Tue Oct 1 12:22:02 2024 From: dave.mayhem23 at gmail.com (Dave May) Date: Tue, 1 Oct 2024 10:22:02 -0700 Subject: [petsc-users] Ghost particles for DMSWARM (or similar) In-Reply-To: <37036553-254F-4532-8D55-0020CACB5F08@us.es> References: <8FBAC7A5-B6AE-4B21-8FEB-52BE1C04A265@us.es> <1B9B1277-9566-444C-9DA8-7ED17684FE01@us.es> <24337E11-33D2-4FFC-89E2-12520AD487FF@us.es> <681E96BD-62A4-4566-A13E-E034B2F19D54@us.es> <7698089E-0909-429F-9E89-9D1AD636ACBF@us.es> <562B2CA0-7462-4AF2-AAF4-E44DDD00B222@us.es> <5C293345-E026-436B-B4D0-E5DC109A0701@us.es> <37036553-254F-4532-8D55-0020CACB5F08@us.es> Message-ID: Hi Miguel, On Tue, 1 Oct 2024 at 09:05, MIGUEL MOLINOS PEREZ wrote: > Sorry, I forgot to add: > > You think VecScatterCreate is the way to go? > No. The semantics you seek are very similar to what VecGhost provides. I was only using VecGhostUpdateBegin/End by way of example. Cheers, Dave > Thanks, > Miguel > > On Oct 1, 2024, at 5:56?PM, MIGUEL MOLINOS PEREZ wrote: > > Hi Dave, > > Would something like that work? > > Yes, this should work! Any idea on where to look so I can try to implement > it myself? > > Best, > Miguel > > On Oct 1, 2024, at 5:22?PM, Dave May wrote: > > Hi Miguel, > > On Tue 1. Oct 2024 at 07:56, MIGUEL MOLINOS PEREZ wrote: > >> Thank you Matt, it works! >> >> The implementation is straightforward: >> - 1? Define the paddle regions using DMGetLocalBoundingBox with the >> background DMDA mesh as an auxiliary mesh for the domain-partitioning. >> - 2? Create an integer to count the local number of particles to be used >> as ghost particle for other processors (N_ghost). One particle can be >> counted more than one time. At the same time, fill two arrays: >> - one with the index of the "main particle? (local particle), >> - and the other with the target rank of the "main particle?. >> - 3? Create the new particles using DMSwarmAddNPoints >> - 4? Fill the new particles with the information of the ?main particle? >> but set the internal variable DMSwarmField_rank with the target rank. >> - 5? Call DMSwarmMigrate(*,PETSC_TRUE). Therefore, we send the ghost >> particles to the corresponding processors and we delete them from the >> original processor. >> - 6? Do stuff? >> - 7? Delete ghost particles. This is very easy, we just have to call >> DMSwarmRemovePoint N_ghost times. >> >> I think this can be easily implemented as closed routine for the DMSwarm >> class. >> >> The remaining question is: how to do the communication between the >> ?original" particle and the ghost particles? For instance, if we update >> some particle variable (locally) inside of a SNES context, this same >> variable should be updated in the ghost particles at the other processors. >> > > I think what you are asking about is an operation similar to > VecGhostUpdate{Begin,End}(). In the case of a DMSwarm I?m not sure how to > define the InsertMode = ADD_VALUES? Some swarm fields do not make sense to > be added. INSERT_VALUES is fine. > > One solution might be to have something like this > > DMSwarmCollectViewUpdateGhostOwners(DM dm, InsertMode mode, PetscInt > nfields, const char *fieldNames[]); > > where one can specify the insert mode and the fields on which the insert > mode will apply. > > Would something like that work? > > Cheers, > Dave > > > > > >> PS: Hope this helps someone in the future :-) >> >> >> On Sep 27, 2024, at 10:50?AM, MIGUEL MOLINOS PEREZ >> wrote: >> >> Thank you Matt, let me give it try. >> >> Miguel >> >> On Sep 27, 2024, at 3:44?AM, Matthew Knepley wrote: >> >> On Thu, Sep 26, 2024 at 7:18?PM MIGUEL MOLINOS PEREZ >> wrote: >> >>> I see, you mean: >>> >>> Create the ghost particles at the local cell with the same properties as >>> particle 1 (duplicate the original particle) but different value >>> DMSwarmField_rank. Then, call DMSwarmMigrate(*,PETSC_FALSE) so we do >>> the migration and delete the local copies of the particle 1. Right? >>> >> >> Yep. I think it will work, from what I know about BASIC. >> >> Thanks, >> >> Matt >> >> >>> Thanks, >>> Miguel >>> >>> On Sep 26, 2024, at 11:09?PM, Matthew Knepley wrote: >>> >>> On Thu, Sep 26, 2024 at 11:20?AM MIGUEL MOLINOS PEREZ >>> wrote: >>> >>>> Thank you Matt. >>>> >>>> Okey, let me have a careful look to the DMSwarmMigrate_Push_Basic implementation >>>> to see if there is some workaround. >>>> >>>> The idea of adding new particles is interesting. However, in that case, >>>> we need to initialize the new (ghost) particles using the fields of >>>> the ?real? particle, right? This can be done using something like: >>>> >>>> VecGhostUpdateBegin (Vec globalout,InsertMode ADD_VALUES , ScatterMode SCATTER_REVERSE );VecGhostUpdateEnd (Vec globalout,InsertMode ADD_VALUES , ScatterMode SCATTER_REVERSE ); >>>> >>>> for the particle fields (?). >>>> >>> >>> I think we can just copy from the local particle. For example, suppose I >>> decide that particle 1 should go to rank 5, 12, and 27. Then >>> I first set p1.rank = 5, then I add two new particles with the same >>> values as particle 1, but with rank = 12 and 27. Then when I call migrate, >>> it will move these three particles to the correct processes, and delete the >>> original particles and the copies from the local set. >>> >>> Thanks, >>> >>> Matt >>> >>> >>>> Thanks, >>>> Miguel >>>> >>>> >>>> On Sep 26, 2024, at 3:53?PM, Matthew Knepley wrote: >>>> >>>> On Thu, Sep 26, 2024 at 6:31?AM MIGUEL MOLINOS PEREZ >>>> wrote: >>>> >>>>> Hi Matt et al, >>>>> >>>>> I?ve been working on the scheme that you proposed to create ghost >>>>> particles (atoms in my case), and it works! With a couple of caveats: >>>>> -1? In general the overlap particles will be migrate from their own >>>>> rank to more than one neighbor rank, this is specially relevant for those >>>>> located close to the corners. Therefore, you'll need to call DMSwarmMigrate >>>>> several times (27 times for 3D cells), during the migration process. >>>>> >>>> >>>> That is terrible. Let's just fix DMSwarmMigrate to have a mode that >>>> sends the particle to all overlapping neighbors at once. It can't be that >>>> hard. >>>> >>>> >>>>> -2? You need to set DMSWARM_MIGRATE_BASIC. Otherwise the proposed >>>>> algorithm will not work at all! >>>>> >>>> >>>> Oh, I should have thought of that. Sorry. >>>> >>>> I can help code up that extension. Can you take a quick look at the >>>> BASIC code? Right now, we just use the rank attached to the particle >>>> to send it. We could have an arrays of ranks, but that seems crazy, and >>>> would blow up particle storage. How about just adding new particles >>>> with the other ranks right before migration? >>>> >>>> Thanks, >>>> >>>> Matt >>>> >>>> >>>>> Hope this helps to other folks! >>>>> >>>>> I have a follow-up question about periodic bcc on this context, should >>>>> I open a new thread of keep posting here? >>>>> >>>>> Thanks, >>>>> Miguel >>>>> >>>>> On Aug 7, 2024, at 4:22?AM, MIGUEL MOLINOS PEREZ >>>>> wrote: >>>>> >>>>> Thanks Matt, I think I'll start by making a small program as a proof >>>>> of concept. Then, if it works I'll implement it in my code and I'll be >>>>> happy to share it too :-) >>>>> >>>>> Miguel >>>>> >>>>> On Aug 4, 2024, at 3:30?AM, Matthew Knepley wrote: >>>>> >>>>> On Fri, Aug 2, 2024 at 7:15?PM MIGUEL MOLINOS PEREZ >>>>> wrote: >>>>> >>>>>> Thanks again Matt, that makes a lot more sense !! >>>>>> >>>>>> Just to check that we are on the same page. You are saying: >>>>>> >>>>>> 1. create a field define a field called "owner rank" for each >>>>>> particle. >>>>>> >>>>>> 2. Identify the phantom particles and modify the internal variable >>>>>> defined by the DMSwarmField_rank variable. >>>>>> >>>>>> 3. Call DMSwarmMigrate(*,PETSC_FALSE), do the calculations using the >>>>>> new local vector including the ghost particles. >>>>>> >>>>>> 4. Then, once the calculations are done, rename the DMSwarmField_rank >>>>>> variable using the "owner rank" variable and call >>>>>> DMSwarmMigrate(*,PETSC_FALSE) once again. >>>>>> >>>>> >>>>> I don't think we need this last step. We can just remove those ghost >>>>> particles for the next step I think. >>>>> >>>>> Thanks, >>>>> >>>>> Matt >>>>> >>>>> >>>>>> Thank you, >>>>>> Miguel >>>>>> >>>>>> >>>>>> On Aug 2, 2024, at 5:33?PM, Matthew Knepley >>>>>> wrote: >>>>>> >>>>>> On Fri, Aug 2, 2024 at 11:15?AM MIGUEL MOLINOS PEREZ >>>>>> wrote: >>>>>> >>>>>>> Thank you Matt for your time, >>>>>>> >>>>>>> What you describe seems to me the ideal approach. >>>>>>> >>>>>>> 1) Add a particle field 'ghost' that identifies ghost vs owned >>>>>>> particles. I think it needs options OWNED, OVERLAP, and GHOST >>>>>>> >>>>>>> This means, locally, I need to allocate Nlocal + ghost particles >>>>>>> (duplicated) for my model? >>>>>>> >>>>>> >>>>>> I would do it another way. I would allocate the particles with no >>>>>> overlap and set them up. Then I would identify the halo particles, mark >>>>>> them as OVERLAP, call DMSwarmMigrate(), and mark the migrated particles as >>>>>> GHOST, then unmark the OVERLAP particles. Shoot! That marking will not work >>>>>> since we cannot tell the difference between particles we received and >>>>>> particles we sent. Okay, instead of the `ghost` field we need an `owner >>>>>> rank` field. So then we >>>>>> >>>>>> 1) Setup the non-overlapping particles >>>>>> >>>>>> 2) Identify the halo particles >>>>>> >>>>>> 3) Change the `rank`, but not the `owner rank` >>>>>> >>>>>> 4) Call DMSwarmMigrate() >>>>>> >>>>>> Now we can identify ghost particles by the `owner rank` >>>>>> >>>>>> >>>>>>> If that so, how to do the communication between the ghost particles >>>>>>> living in the rank i and their ?real? counterpart in the rank j. >>>>>>> >>>>>>> Algo, as an alternative, what about: >>>>>>> 1) Use an IS tag which contains, for each rank, a list of the >>>>>>> global index of the neighbors particles outside of the rank. >>>>>>> 2) Use VecCreateGhost to create a new vector which contains extra >>>>>>> local space for the ghost components of the vector. >>>>>>> 3) Use VecScatterCreate, VecScatterBegin, and VecScatterEnd to do >>>>>>> the transference of data between a vector obtained with >>>>>>> DMSwarmCreateGlobalVectorFromField >>>>>>> 4) Do necessary computations using the vectors created with >>>>>>> VecCreateGhost. >>>>>>> >>>>>> >>>>>> This is essentially what Migrate() does. I was trying to reuse the >>>>>> code. >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Matt >>>>>> >>>>>> >>>>>>> Thanks, >>>>>>> Miguel >>>>>>> >>>>>>> On Aug 2, 2024, at 8:58?AM, Matthew Knepley >>>>>>> wrote: >>>>>>> >>>>>>> On Thu, Aug 1, 2024 at 4:40?PM MIGUEL MOLINOS PEREZ >>>>>>> wrote: >>>>>>> >>>>>>>> This Message Is From an External Sender >>>>>>>> This message came from outside your organization. >>>>>>>> >>>>>>>> >>>>>>>> Dear all, >>>>>>>> >>>>>>>> I am implementing a Molecular Dynamics (MD) code using the DMSWARM interface. In the MD simulations we evaluate on each particle (atoms) some kind of scalar functional using data from the neighbouring atoms. My problem lies in the parallel implementation of the model, because sometimes, some of these neighbours lie on a different processor. >>>>>>>> >>>>>>>> This is usually solved by using ghost particles. A similar approach (with nodes instead) is already implemented for other PETSc mesh structures like DMPlexConstructGhostCells. Unfortunately, I don't see this kind of constructs for DMSWARM. Am I missing something? >>>>>>>> >>>>>>>> I this could be done by applying a buffer region by exploiting the background DMDA mesh that I already use to do domain decomposition. Then using the buffer region of each cell to locate the ghost particles and finally using VecCreateGhost. Is this feasible? Or is there an easier approach using other PETSc functions. >>>>>>>> >>>>>>>> >>>>>>> This is feasible, but it would be good to develop a set of best >>>>>>> practices, since we have been mainly focused on the case of non-redundant >>>>>>> particles. Here is how I think I would do what you want. >>>>>>> >>>>>>> 1) Add a particle field 'ghost' that identifies ghost vs owned >>>>>>> particles. I think it needs options OWNED, OVERLAP, and GHOST >>>>>>> >>>>>>> 2) At some interval identify particles that should be sent to other >>>>>>> processes as ghosts. I would call these "overlap particles". The >>>>>>> determination >>>>>>> seems application specific, so I would leave this determination >>>>>>> to the user right now. We do two things to these particles >>>>>>> >>>>>>> a) Mark chosen particles as OVERLAP >>>>>>> >>>>>>> b) Change rank to process we are sending to >>>>>>> >>>>>>> 3) Call DMSwarmMigrate with PETSC_FALSE for the particle deletion >>>>>>> flag >>>>>>> >>>>>>> 4) Mark OVERLAP particles as GHOST when they arrive >>>>>>> >>>>>>> There is one problem in the above algorithm. It does not allow >>>>>>> sending particles to multiple ranks. We would have to do this >>>>>>> in phases right now, or make a small adjustment to the interface >>>>>>> allowing replication of particles when a set of ranks is specified. >>>>>>> >>>>>>> THanks, >>>>>>> >>>>>>> Matt >>>>>>> >>>>>>> >>>>>>>> Thank you, >>>>>>>> Miguel >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> -- >>>>>>> What most experimenters take for granted before they begin their >>>>>>> experiments is infinitely more interesting than any results to which their >>>>>>> experiments lead. >>>>>>> -- Norbert Wiener >>>>>>> >>>>>>> https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!aIhw4GigszsZG9HmvzDp8Lxt-OKLh4BxuMAUcfn8901TVodKwWeu7X3DFBjBZ5X0IGbICPb6T0PIj0GDVOAk63NPQ60c$ >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> -- >>>>>> What most experimenters take for granted before they begin their >>>>>> experiments is infinitely more interesting than any results to which their >>>>>> experiments lead. >>>>>> -- Norbert Wiener >>>>>> >>>>>> https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!aIhw4GigszsZG9HmvzDp8Lxt-OKLh4BxuMAUcfn8901TVodKwWeu7X3DFBjBZ5X0IGbICPb6T0PIj0GDVOAk63NPQ60c$ >>>>>> >>>>>> >>>>>> >>>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their >>>>> experiments is infinitely more interesting than any results to which their >>>>> experiments lead. >>>>> -- Norbert Wiener >>>>> >>>>> https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!aIhw4GigszsZG9HmvzDp8Lxt-OKLh4BxuMAUcfn8901TVodKwWeu7X3DFBjBZ5X0IGbICPb6T0PIj0GDVOAk63NPQ60c$ >>>>> >>>>> >>>>> >>>>> >>>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>>> https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!aIhw4GigszsZG9HmvzDp8Lxt-OKLh4BxuMAUcfn8901TVodKwWeu7X3DFBjBZ5X0IGbICPb6T0PIj0GDVOAk63NPQ60c$ >>>> >>>> >>>> >>>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!aIhw4GigszsZG9HmvzDp8Lxt-OKLh4BxuMAUcfn8901TVodKwWeu7X3DFBjBZ5X0IGbICPb6T0PIj0GDVOAk63NPQ60c$ >>> >>> >>> >>> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!aIhw4GigszsZG9HmvzDp8Lxt-OKLh4BxuMAUcfn8901TVodKwWeu7X3DFBjBZ5X0IGbICPb6T0PIj0GDVOAk63NPQ60c$ >> >> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mmolinos at us.es Tue Oct 1 12:42:11 2024 From: mmolinos at us.es (MIGUEL MOLINOS PEREZ) Date: Tue, 1 Oct 2024 17:42:11 +0000 Subject: [petsc-users] Ghost particles for DMSWARM (or similar) In-Reply-To: References: <8FBAC7A5-B6AE-4B21-8FEB-52BE1C04A265@us.es> <1B9B1277-9566-444C-9DA8-7ED17684FE01@us.es> <24337E11-33D2-4FFC-89E2-12520AD487FF@us.es> <681E96BD-62A4-4566-A13E-E034B2F19D54@us.es> <7698089E-0909-429F-9E89-9D1AD636ACBF@us.es> <562B2CA0-7462-4AF2-AAF4-E44DDD00B222@us.es> <5C293345-E026-436B-B4D0-E5DC109A0701@us.es> Message-ID: <55056A2B-85E9-4896-9B4B-869A14F4B2C8@us.es> Wow, thank you Dave that?s awesome, let me know if there's anything I can help you with! Miguel On Oct 1, 2024, at 7:20?PM, Dave May wrote: On Tue, 1 Oct 2024 at 08:56, MIGUEL MOLINOS PEREZ > wrote: Hi Dave, Would something like that work? Yes, this should work! Any idea on where to look so I can try to implement it myself? I am adding support for this right now. Best, Miguel On Oct 1, 2024, at 5:22?PM, Dave May > wrote: Hi Miguel, On Tue 1. Oct 2024 at 07:56, MIGUEL MOLINOS PEREZ > wrote: Thank you Matt, it works! The implementation is straightforward: - 1? Define the paddle regions using DMGetLocalBoundingBox with the background DMDA mesh as an auxiliary mesh for the domain-partitioning. - 2? Create an integer to count the local number of particles to be used as ghost particle for other processors (N_ghost). One particle can be counted more than one time. At the same time, fill two arrays: - one with the index of the "main particle? (local particle), - and the other with the target rank of the "main particle?. - 3? Create the new particles using DMSwarmAddNPoints - 4? Fill the new particles with the information of the ?main particle? but set the internal variable DMSwarmField_rank with the target rank. - 5? Call DMSwarmMigrate(*,PETSC_TRUE). Therefore, we send the ghost particles to the corresponding processors and we delete them from the original processor. - 6? Do stuff? - 7? Delete ghost particles. This is very easy, we just have to call DMSwarmRemovePoint N_ghost times. I think this can be easily implemented as closed routine for the DMSwarm class. The remaining question is: how to do the communication between the ?original" particle and the ghost particles? For instance, if we update some particle variable (locally) inside of a SNES context, this same variable should be updated in the ghost particles at the other processors. I think what you are asking about is an operation similar to VecGhostUpdate{Begin,End}(). In the case of a DMSwarm I?m not sure how to define the InsertMode = ADD_VALUES? Some swarm fields do not make sense to be added. INSERT_VALUES is fine. One solution might be to have something like this DMSwarmCollectViewUpdateGhostOwners(DM dm, InsertMode mode, PetscInt nfields, const char *fieldNames[]); where one can specify the insert mode and the fields on which the insert mode will apply. Would something like that work? Cheers, Dave PS: Hope this helps someone in the future :-) On Sep 27, 2024, at 10:50?AM, MIGUEL MOLINOS PEREZ > wrote: Thank you Matt, let me give it try. Miguel On Sep 27, 2024, at 3:44?AM, Matthew Knepley > wrote: On Thu, Sep 26, 2024 at 7:18?PM MIGUEL MOLINOS PEREZ > wrote: I see, you mean: Create the ghost particles at the local cell with the same properties as particle 1 (duplicate the original particle) but different value DMSwarmField_rank. Then, call DMSwarmMigrate(*,PETSC_FALSE) so we do the migration and delete the local copies of the particle 1. Right? Yep. I think it will work, from what I know about BASIC. Thanks, Matt Thanks, Miguel On Sep 26, 2024, at 11:09?PM, Matthew Knepley > wrote: On Thu, Sep 26, 2024 at 11:20?AM MIGUEL MOLINOS PEREZ > wrote: Thank you Matt. Okey, let me have a careful look to the DMSwarmMigrate_Push_Basic implementation to see if there is some workaround. The idea of adding new particles is interesting. However, in that case, we need to initialize the new (ghost) particles using the fields of the ?real? particle, right? This can be done using something like: VecGhostUpdateBegin(Vec globalout,InsertMode ADD_VALUES, ScatterMode SCATTER_REVERSE); VecGhostUpdateEnd(Vec globalout,InsertMode ADD_VALUES, ScatterMode SCATTER_REVERSE); for the particle fields (?). I think we can just copy from the local particle. For example, suppose I decide that particle 1 should go to rank 5, 12, and 27. Then I first set p1.rank = 5, then I add two new particles with the same values as particle 1, but with rank = 12 and 27. Then when I call migrate, it will move these three particles to the correct processes, and delete the original particles and the copies from the local set. Thanks, Matt Thanks, Miguel On Sep 26, 2024, at 3:53?PM, Matthew Knepley > wrote: On Thu, Sep 26, 2024 at 6:31?AM MIGUEL MOLINOS PEREZ > wrote: Hi Matt et al, I?ve been working on the scheme that you proposed to create ghost particles (atoms in my case), and it works! With a couple of caveats: -1? In general the overlap particles will be migrate from their own rank to more than one neighbor rank, this is specially relevant for those located close to the corners. Therefore, you'll need to call DMSwarmMigrate several times (27 times for 3D cells), during the migration process. That is terrible. Let's just fix DMSwarmMigrate to have a mode that sends the particle to all overlapping neighbors at once. It can't be that hard. -2? You need to set DMSWARM_MIGRATE_BASIC. Otherwise the proposed algorithm will not work at all! Oh, I should have thought of that. Sorry. I can help code up that extension. Can you take a quick look at the BASIC code? Right now, we just use the rank attached to the particle to send it. We could have an arrays of ranks, but that seems crazy, and would blow up particle storage. How about just adding new particles with the other ranks right before migration? Thanks, Matt Hope this helps to other folks! I have a follow-up question about periodic bcc on this context, should I open a new thread of keep posting here? Thanks, Miguel On Aug 7, 2024, at 4:22?AM, MIGUEL MOLINOS PEREZ > wrote: Thanks Matt, I think I'll start by making a small program as a proof of concept. Then, if it works I'll implement it in my code and I'll be happy to share it too :-) Miguel On Aug 4, 2024, at 3:30?AM, Matthew Knepley > wrote: On Fri, Aug 2, 2024 at 7:15?PM MIGUEL MOLINOS PEREZ > wrote: Thanks again Matt, that makes a lot more sense !! Just to check that we are on the same page. You are saying: 1. create a field define a field called "owner rank" for each particle. 2. Identify the phantom particles and modify the internal variable defined by the DMSwarmField_rank variable. 3. Call DMSwarmMigrate(*,PETSC_FALSE), do the calculations using the new local vector including the ghost particles. 4. Then, once the calculations are done, rename the DMSwarmField_rank variable using the "owner rank" variable and call DMSwarmMigrate(*,PETSC_FALSE) once again. I don't think we need this last step. We can just remove those ghost particles for the next step I think. Thanks, Matt Thank you, Miguel On Aug 2, 2024, at 5:33?PM, Matthew Knepley > wrote: On Fri, Aug 2, 2024 at 11:15?AM MIGUEL MOLINOS PEREZ > wrote: Thank you Matt for your time, What you describe seems to me the ideal approach. 1) Add a particle field 'ghost' that identifies ghost vs owned particles. I think it needs options OWNED, OVERLAP, and GHOST This means, locally, I need to allocate Nlocal + ghost particles (duplicated) for my model? I would do it another way. I would allocate the particles with no overlap and set them up. Then I would identify the halo particles, mark them as OVERLAP, call DMSwarmMigrate(), and mark the migrated particles as GHOST, then unmark the OVERLAP particles. Shoot! That marking will not work since we cannot tell the difference between particles we received and particles we sent. Okay, instead of the `ghost` field we need an `owner rank` field. So then we 1) Setup the non-overlapping particles 2) Identify the halo particles 3) Change the `rank`, but not the `owner rank` 4) Call DMSwarmMigrate() Now we can identify ghost particles by the `owner rank` If that so, how to do the communication between the ghost particles living in the rank i and their ?real? counterpart in the rank j. Algo, as an alternative, what about: 1) Use an IS tag which contains, for each rank, a list of the global index of the neighbors particles outside of the rank. 2) Use VecCreateGhost to create a new vector which contains extra local space for the ghost components of the vector. 3) Use VecScatterCreate, VecScatterBegin, and VecScatterEnd to do the transference of data between a vector obtained with DMSwarmCreateGlobalVectorFromField 4) Do necessary computations using the vectors created with VecCreateGhost. This is essentially what Migrate() does. I was trying to reuse the code. Thanks, Matt Thanks, Miguel On Aug 2, 2024, at 8:58?AM, Matthew Knepley > wrote: On Thu, Aug 1, 2024 at 4:40?PM MIGUEL MOLINOS PEREZ > wrote: This Message Is From an External Sender This message came from outside your organization. Dear all, I am implementing a Molecular Dynamics (MD) code using the DMSWARM interface. In the MD simulations we evaluate on each particle (atoms) some kind of scalar functional using data from the neighbouring atoms. My problem lies in the parallel implementation of the model, because sometimes, some of these neighbours lie on a different processor. This is usually solved by using ghost particles. A similar approach (with nodes instead) is already implemented for other PETSc mesh structures like DMPlexConstructGhostCells. Unfortunately, I don't see this kind of constructs for DMSWARM. Am I missing something? I this could be done by applying a buffer region by exploiting the background DMDA mesh that I already use to do domain decomposition. Then using the buffer region of each cell to locate the ghost particles and finally using VecCreateGhost. Is this feasible? Or is there an easier approach using other PETSc functions. This is feasible, but it would be good to develop a set of best practices, since we have been mainly focused on the case of non-redundant particles. Here is how I think I would do what you want. 1) Add a particle field 'ghost' that identifies ghost vs owned particles. I think it needs options OWNED, OVERLAP, and GHOST 2) At some interval identify particles that should be sent to other processes as ghosts. I would call these "overlap particles". The determination seems application specific, so I would leave this determination to the user right now. We do two things to these particles a) Mark chosen particles as OVERLAP b) Change rank to process we are sending to 3) Call DMSwarmMigrate with PETSC_FALSE for the particle deletion flag 4) Mark OVERLAP particles as GHOST when they arrive There is one problem in the above algorithm. It does not allow sending particles to multiple ranks. We would have to do this in phases right now, or make a small adjustment to the interface allowing replication of particles when a set of ranks is specified. THanks, Matt Thank you, Miguel -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!dX4svD_xMjcyayuVbLHxmpSYzp15tYt-FL88U5OCYP_ohterg8KIz-4VMtUzOtiwZKMFPnf6DAs5PmFlUPs3SA$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!dX4svD_xMjcyayuVbLHxmpSYzp15tYt-FL88U5OCYP_ohterg8KIz-4VMtUzOtiwZKMFPnf6DAs5PmFlUPs3SA$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!dX4svD_xMjcyayuVbLHxmpSYzp15tYt-FL88U5OCYP_ohterg8KIz-4VMtUzOtiwZKMFPnf6DAs5PmFlUPs3SA$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!dX4svD_xMjcyayuVbLHxmpSYzp15tYt-FL88U5OCYP_ohterg8KIz-4VMtUzOtiwZKMFPnf6DAs5PmFlUPs3SA$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!dX4svD_xMjcyayuVbLHxmpSYzp15tYt-FL88U5OCYP_ohterg8KIz-4VMtUzOtiwZKMFPnf6DAs5PmFlUPs3SA$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!dX4svD_xMjcyayuVbLHxmpSYzp15tYt-FL88U5OCYP_ohterg8KIz-4VMtUzOtiwZKMFPnf6DAs5PmFlUPs3SA$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From gkluhana at cs.ubc.ca Tue Oct 1 14:37:17 2024 From: gkluhana at cs.ubc.ca (Gautam Luhana) Date: Tue, 1 Oct 2024 12:37:17 -0700 Subject: [petsc-users] Multigrid Preconditioned CG on cell-centered Poisson with DMStag Message-ID: <1ec910d0-1b23-435c-b23a-ae1843c3bb3e@cs.ubc.ca> Hello, I am solving the stokes equations on the MAC grid using DMStag with an approximate Schur complement based preconditioner. The preconditioner needs a subdomain solve of the Poisson equation on both the velocity as well as the pressure space. I am using multigrid preconditioned CG for both the subdomain solves. While the subdomain solve on the velocity space converges in 4-5 iterations and is independent of mesh-size, the pressure solve is not mesh-size independent. Relevant information: Pressure space operator has Neumann boundary conditions. Coarse-grid operators are being generated using the Galerkin approach (A = P*Af*Pt) for both of the subdomains. For the pressure space, the jacobi and sor smoothers were tried with several omega and damping parameter choices. Symmetric smoothing (same number of pre- and post-smoothing iterations) Has anyone tried solving the poisson operator on the cell-centered pressure space with DMStag? Or know what might be going wrong with the mg-cg convergence? Thanks! -Gautam From bsmith at petsc.dev Tue Oct 1 14:47:14 2024 From: bsmith at petsc.dev (Barry Smith) Date: Tue, 1 Oct 2024 15:47:14 -0400 Subject: [petsc-users] Multigrid Preconditioned CG on cell-centered Poisson with DMStag In-Reply-To: <1ec910d0-1b23-435c-b23a-ae1843c3bb3e@cs.ubc.ca> References: <1ec910d0-1b23-435c-b23a-ae1843c3bb3e@cs.ubc.ca> Message-ID: <633763A2-98AA-44FF-AEED-0C74F35219F3@petsc.dev> Are you handling the null space? Like with MatSetNullSpace()? Can you try src/ksp/ksp/tutorials/ex32.c it solves the same problem but with a simpler DMDA. I don't see any reason why using DMStag with the cell-centered values should be different. Barry > On Oct 1, 2024, at 3:37?PM, Gautam Luhana wrote: > > Hello, > > I am solving the stokes equations on the MAC grid using DMStag with an approximate Schur complement based preconditioner. The preconditioner needs a subdomain solve of the Poisson equation on both the velocity as well as the pressure space. I am using multigrid preconditioned CG for both the subdomain solves. While the subdomain solve on the velocity space converges in 4-5 iterations and is independent of mesh-size, the pressure solve is not mesh-size independent. Relevant information: > > Pressure space operator has Neumann boundary conditions. > > Coarse-grid operators are being generated using the Galerkin approach (A = P*Af*Pt) for both of the subdomains. > > For the pressure space, the jacobi and sor smoothers were tried with several omega and damping parameter choices. > > Symmetric smoothing (same number of pre- and post-smoothing iterations) > > Has anyone tried solving the poisson operator on the cell-centered pressure space with DMStag? Or know what might be going wrong with the mg-cg convergence? > > Thanks! > > -Gautam > > From a.croucher at auckland.ac.nz Tue Oct 1 17:39:23 2024 From: a.croucher at auckland.ac.nz (Adrian Croucher) Date: Wed, 2 Oct 2024 11:39:23 +1300 Subject: [petsc-users] ISView() in PETSc 3.22 In-Reply-To: References: Message-ID: hi Matt On 2/10/24 1:01 am, Matthew Knepley wrote: > On Mon, Sep 30, 2024 at 10:15?PM Adrian Croucher > wrote: > > hi, I am testing my (Fortran) code on PETSc 3.22 and have got it to > build. However I am getting some unusual new behaviour when I > write an > IS to an HDF5 file using ISView(). > > The attached minimal example shows the issue. It creates a simple > 10-element IS and writes it to HDF5. With previous versions of PETSc > this would give me a 10x1 dataset containing the values 0 - 9, as > expected. > > When I run it with PETSc 3.22 (in serial), I again get the expected > values written on stdout, so it looks like the IS itself is > correct. But > in the HDF5 file I get a 1x3 dataset containing the values (10,1,0). > > Has something changed here? > > > Yes. We now compress IS data sets by default. You can turn it off > using -is_view_compress 0. I am not sure > what the best way to manage this is, but it makes a huge difference in > file size for checkpointing. Thanks. I guess that's fine if it's being read back in by PETSc, but I have users with non-PETSc post-processing codes needing to read my output, and they aren't going to know what to do with these compressed datasets. I only write one IS at the start of each simulation, so it's a negligible contribution to the file size. So I think I will need to disable the compression in my code. Is there a function call I can use to do that, to make sure it's always done without users needing to pass -is_view_compress 0? - Adrian -- Dr Adrian Croucher Senior Research Fellow Department of Engineering Science Waipapa Taumata Rau / University of Auckland, New Zealand email:a.croucher at auckland.ac.nz tel: +64 (0)9 923 4611 -------------- next part -------------- An HTML attachment was scrubbed... URL: From a.croucher at auckland.ac.nz Tue Oct 1 19:29:03 2024 From: a.croucher at auckland.ac.nz (Adrian Croucher) Date: Wed, 2 Oct 2024 13:29:03 +1300 Subject: [petsc-users] ISView() in PETSc 3.22 In-Reply-To: References: Message-ID: On 2/10/24 11:39 am, Adrian Croucher wrote: > So I think I will need to disable the compression in my code. Is there > a function call I can use to do that, to make sure it's always done > without users needing to pass -is_view_compress 0? > Looks like there is no specific function to do this, but I can use PetscOptionsSetValue(). Maybe worth having something about this new compression stuff in the PETSc 3.22 change log (https://urldefense.us/v3/__https://petsc.org/release/changes/322/__;!!G_uCfscf7eWS!fRCgVJ_aunT3NNJpeVSGKkoRNgN5wnYrYQWnrDzYna1sNQVHGx_EpPC0jgnjKHO5q0rRqrwbbctxKW_n4eNy28Ia6w99USE7$ ), in case it trips anyone else up? - Adrian -- Dr Adrian Croucher Senior Research Fellow Department of Engineering Science Waipapa Taumata Rau / University of Auckland, New Zealand email: a.croucher at auckland.ac.nz tel: +64 (0)9 923 4611 From knepley at gmail.com Tue Oct 1 21:01:40 2024 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 1 Oct 2024 22:01:40 -0400 Subject: [petsc-users] ISView() in PETSc 3.22 In-Reply-To: References: Message-ID: On Tue, Oct 1, 2024 at 8:29?PM Adrian Croucher wrote: > On 2/10/24 11:39 am, Adrian Croucher wrote: > > > So I think I will need to disable the compression in my code. Is there > > a function call I can use to do that, to make sure it's always done > > without users needing to pass -is_view_compress 0? > > > Looks like there is no specific function to do this, but I can use > PetscOptionsSetValue(). > > Maybe worth having something about this new compression stuff in the > PETSc 3.22 change log (https://urldefense.us/v3/__https://petsc.org/release/changes/322/__;!!G_uCfscf7eWS!YT4Ki8iXApGvjhAbuUGKbRNIKj2t-ysAWb_4Us3bGuW6-CR7e5bklc4UHe09hsB0RZeuLqSuTgHGswrT9Rlk$ ), in case > it trips anyone else up? > Yep, I will put it in. Thanks, Matt > - Adrian > > -- > Dr Adrian Croucher > Senior Research Fellow > Department of Engineering Science > Waipapa Taumata Rau / University of Auckland, New Zealand > email: a.croucher at auckland.ac.nz > tel: +64 (0)9 923 4611 > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!YT4Ki8iXApGvjhAbuUGKbRNIKj2t-ysAWb_4Us3bGuW6-CR7e5bklc4UHe09hsB0RZeuLqSuTgHGs0Cnxtn3$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From langtian.liu at icloud.com Wed Oct 2 03:56:50 2024 From: langtian.liu at icloud.com (=?utf-8?B?5YiY5rWq5aSp?=) Date: Wed, 2 Oct 2024 08:56:50 +0000 (UTC) Subject: [petsc-users] issue of MatCreateDense in the CUDA codes Message-ID: Hi all, I am using the PETSc and SLEPc to solve the Faddeev equation of baryons. I encounter a problem of function MatCreateDense when changing from CPU to CPU-GPU computations. At first, I write the codes in purely CPU computation in the following way and it works. ``` Eigen :: MatrixXcd H_KER ; Eigen :: MatrixXcd G0 ; printf( " \n Compute the propagator matrix. \n " ) ; prop_matrix_nucleon_sc_av( Mn , pp_nodes , cos1_nodes) ; printf( " \n Compute the propagator matrix done. \n " ) ; printf( " \n Compute the kernel matrix. \n " ) ; bse_kernel_nucleon_sc_av( Mn , pp_nodes , pp_weights , cos1_nodes , cos1_weights) ; printf( " \n Compute the kernel matrix done. \n " ) ; printf( " \n Compute the full kernel matrix by multiplying kernel and propagator matrix. \n " ) ; MatrixXcd kernel_temp = H_KER * G0 ; printf( " \n Compute the full kernel matrix done. \n " ) ; // Solve the eigen system with SLEPc printf( " \n Solve the eigen system in the rest frame. \n " ) ; // Get the size of the Eigen matrix int nRows = ( int ) kernel_temp. rows () ; int nCols = ( int ) kernel_temp. cols () ; // Create PETSc matrix and share the data of kernel_temp Mat kernel ; PetscCall(MatCreateDense(PETSC_COMM_WORLD , PETSC_DECIDE , PETSC_DECIDE , nRows , nCols , kernel_temp.data() , &kernel)) ; PetscCall(MatAssemblyBegin(kernel , MAT_FINAL_ASSEMBLY)) ; PetscCall(MatAssemblyEnd(kernel , MAT_FINAL_ASSEMBLY)) ; ``` Now I change to compute the propagator and kernel matrices in GPU and then compute the largest eigenvalues in CPU using SLEPc in the ways below. ``` cuDoubleComplex * h_propmat ; cuDoubleComplex * h_kernelmat ; int dim = EIGHT * NP * NZ ; printf( " \n Compute the propagator matrix. \n " ) ; prop_matrix_nucleon_sc_av_cuda( Mn , pp_nodes .data() , cos1_nodes .data()) ; printf( " \n Compute the propagator matrix done. \n " ) ; printf( " \n Compute the kernel matrix. \n " ) ; kernel_matrix_nucleon_sc_av_cuda( Mn , pp_nodes .data() , pp_weights .data() , cos1_nodes .data() , cos1_weights .data()) ; printf( " \n Compute the kernel matrix done. \n " ) ; printf( " \n Compute the full kernel matrix by multiplying kernel and propagator matrix. \n " ) ; // Map the raw arrays to Eigen matrices (column-major order) auto * h_kernel_temp = new cuDoubleComplex [ dim * dim ] ; matmul_cublas_cuDoubleComplex( h_kernelmat , h_propmat , h_kernel_temp , dim , dim , dim ) ; printf( " \n Compute the full kernel matrix done. \n " ) ; // Solve the eigen system with SLEPc printf( " \n Solve the eigen system in the rest frame. \n " ) ; int nRows = dim ; int nCols = dim ; // Create PETSc matrix and share the data of kernel_temp Mat kernel ; auto * h_kernel = ( std :: complex < double >*)( h_kernel_temp ) ; PetscCall (MatCreateDense( PETSC_COMM_WORLD , PETSC_DECIDE , PETSC_DECIDE , nRows , nCols , h_kernel_temp , & kernel )) ; PetscCall (MatAssemblyBegin( kernel , MAT_FINAL_ASSEMBLY )) ; PetscCall (MatAssemblyEnd( kernel , MAT_FINAL_ASSEMBLY )) ; But in this case, the compiler told me that the MatCreateDense function uses the data pointer as type of "thrust::complex" instead of "std::complex". I am sure I only configured and install PETSc in purely CPU without GPU and this codes are written in the host function. Why the function changes its behavior? Did you also meet this problem when writing the cuda codes and how to solve this problem. I tried to copy the data to a new thrust::complex matrix but this is very time consuming since my matrix is very big. Is there a way to create the Mat from the original data without changing the data type to thrust::complex in the cuda applications? Any response will be appreciated. Thank you! Best wishes, Langtian Liu ------ Institute for Theorectical Physics, Justus-Liebig-University Giessen Heinrich-Buff-Ring 16, 35392 Giessen Germany -------------- next part -------------- An HTML attachment was scrubbed... URL: From jroman at dsic.upv.es Wed Oct 2 04:18:21 2024 From: jroman at dsic.upv.es (Jose E. Roman) Date: Wed, 2 Oct 2024 09:18:21 +0000 Subject: [petsc-users] issue of MatCreateDense in the CUDA codes In-Reply-To: References: Message-ID: <1D918E07-839C-4369-9B82-695A08AEB0A3@dsic.upv.es> For the CUDA case you should use MatCreateDenseCUDA() instead of MatCreateDense(). With this you pass a pointer with the data on the GPU memory. But I guess "new cuDoubleComplex[dim*dim]" is allocating on the CPU, you should use cudaMalloc() instead. Jose > El 2 oct 2024, a las 10:56, ??? via petsc-users escribi?: > > Hi all, > > I am using the PETSc and SLEPc to solve the Faddeev equation of baryons. I encounter a problem of function MatCreateDense when changing from CPU to CPU-GPU computations. > At first, I write the codes in purely CPU computation in the following way and it works. > ``` > Eigen::MatrixXcd H_KER; > Eigen::MatrixXcd G0; > printf("\nCompute the propagator matrix.\n"); > prop_matrix_nucleon_sc_av(Mn, pp_nodes, cos1_nodes); > printf("\nCompute the propagator matrix done.\n"); > printf("\nCompute the kernel matrix.\n"); > bse_kernel_nucleon_sc_av(Mn, pp_nodes, pp_weights, cos1_nodes, cos1_weights); > printf("\nCompute the kernel matrix done.\n"); > printf("\nCompute the full kernel matrix by multiplying kernel and propagator matrix.\n"); > MatrixXcd kernel_temp = H_KER * G0; > printf("\nCompute the full kernel matrix done.\n"); > > // Solve the eigen system with SLEPc > printf("\nSolve the eigen system in the rest frame.\n"); > // Get the size of the Eigen matrix > int nRows = (int) kernel_temp.rows(); > int nCols = (int) kernel_temp.cols(); > // Create PETSc matrix and share the data of kernel_temp > Mat kernel; > PetscCall(MatCreateDense(PETSC_COMM_WORLD, PETSC_DECIDE, PETSC_DECIDE, nRows, nCols, kernel_temp.data(), &kernel)); > PetscCall(MatAssemblyBegin(kernel, MAT_FINAL_ASSEMBLY)); > PetscCall(MatAssemblyEnd(kernel, MAT_FINAL_ASSEMBLY)); > ``` > Now I change to compute the propagator and kernel matrices in GPU and then compute the largest eigenvalues in CPU using SLEPc in the ways below. > ``` > cuDoubleComplex *h_propmat; > cuDoubleComplex *h_kernelmat; > int dim = EIGHT * NP * NZ; > printf("\nCompute the propagator matrix.\n"); > prop_matrix_nucleon_sc_av_cuda(Mn, pp_nodes.data(), cos1_nodes.data()); > printf("\nCompute the propagator matrix done.\n"); > printf("\nCompute the kernel matrix.\n"); > kernel_matrix_nucleon_sc_av_cuda(Mn, pp_nodes.data(), pp_weights.data(), cos1_nodes.data(), cos1_weights.data()); > printf("\nCompute the kernel matrix done.\n"); > printf("\nCompute the full kernel matrix by multiplying kernel and propagator matrix.\n"); > // Map the raw arrays to Eigen matrices (column-major order) > auto *h_kernel_temp = new cuDoubleComplex [dim*dim]; > matmul_cublas_cuDoubleComplex(h_kernelmat,h_propmat,h_kernel_temp,dim,dim,dim); > printf("\nCompute the full kernel matrix done.\n"); > > // Solve the eigen system with SLEPc > printf("\nSolve the eigen system in the rest frame.\n"); > int nRows = dim; > int nCols = dim; > // Create PETSc matrix and share the data of kernel_temp > Mat kernel; > auto* h_kernel = (std::complex*)(h_kernel_temp); > PetscCall(MatCreateDense(PETSC_COMM_WORLD, PETSC_DECIDE, PETSC_DECIDE, nRows, nCols, h_kernel_temp, &kernel)); > PetscCall(MatAssemblyBegin(kernel, MAT_FINAL_ASSEMBLY)); > PetscCall(MatAssemblyEnd(kernel, MAT_FINAL_ASSEMBLY)); > But in this case, the compiler told me that the MatCreateDense function uses the data pointer as type of "thrust::complex" instead of "std::complex". > I am sure I only configured and install PETSc in purely CPU without GPU and this codes are written in the host function. > Why the function changes its behavior? Did you also meet this problem when writing the cuda codes and how to solve this problem. > I tried to copy the data to a new thrust::complex matrix but this is very time consuming since my matrix is very big. Is there a way to create the Mat from the original data without changing the data type to thrust::complex in the cuda applications? Any response will be appreciated. Thank you! > > Best wishes, > Langtian Liu > > ------ > Institute for Theorectical Physics, Justus-Liebig-University Giessen > Heinrich-Buff-Ring 16, 35392 Giessen Germany > From langtian.liu at icloud.com Wed Oct 2 04:23:57 2024 From: langtian.liu at icloud.com (=?utf-8?B?5YiY5rWq5aSp?=) Date: Wed, 2 Oct 2024 09:23:57 +0000 (UTC) Subject: [petsc-users] issue of MatCreateDense in the CUDA codes In-Reply-To: <1D918E07-839C-4369-9B82-695A08AEB0A3@dsic.upv.es> References: <1D918E07-839C-4369-9B82-695A08AEB0A3@dsic.upv.es> Message-ID: Hi Jose, Since my matrix is two large, I cannot create the Mat on GPU. So I still want to create and compute the eigenvalues of this matrix on CPU using SLEPc. Best, -------------------- Langtian Liu Institute for Theorectical Physics, Justus-Liebig-University Giessen Heinrich-Buff-Ring 16, 35392 Giessen Germany email: langtian.liu at icloud.com Tel: (+49)641 99 33342 On Oct 2, 2024, at 11:18 AM, Jose E. Roman wrote: For the CUDA case you should use MatCreateDenseCUDA() instead of MatCreateDense(). With this you pass a pointer with the data on the GPU memory. But I guess "new cuDoubleComplex[dim*dim]" is allocating on the CPU, you should use cudaMalloc() instead. Jose El 2 oct 2024, a las 10:56, ??? via petsc-users escribi?: Hi all, I am using the PETSc and SLEPc to solve the Faddeev equation of baryons. I encounter a problem of function MatCreateDense when changing from CPU to CPU-GPU computations. At first, I write the codes in purely CPU computation in the following way and it works. ``` Eigen::MatrixXcd H_KER; Eigen::MatrixXcd G0; printf("\nCompute the propagator matrix.\n"); prop_matrix_nucleon_sc_av(Mn, pp_nodes, cos1_nodes); printf("\nCompute the propagator matrix done.\n"); printf("\nCompute the kernel matrix.\n"); bse_kernel_nucleon_sc_av(Mn, pp_nodes, pp_weights, cos1_nodes, cos1_weights); printf("\nCompute the kernel matrix done.\n"); printf("\nCompute the full kernel matrix by multiplying kernel and propagator matrix.\n"); MatrixXcd kernel_temp = H_KER * G0; printf("\nCompute the full kernel matrix done.\n"); // Solve the eigen system with SLEPc printf("\nSolve the eigen system in the rest frame.\n"); // Get the size of the Eigen matrix int nRows = (int) kernel_temp.rows(); int nCols = (int) kernel_temp.cols(); // Create PETSc matrix and share the data of kernel_temp Mat kernel; PetscCall(MatCreateDense(PETSC_COMM_WORLD, PETSC_DECIDE, PETSC_DECIDE, nRows, nCols, kernel_temp.data(), &kernel)); PetscCall(MatAssemblyBegin(kernel, MAT_FINAL_ASSEMBLY)); PetscCall(MatAssemblyEnd(kernel, MAT_FINAL_ASSEMBLY)); ``` Now I change to compute the propagator and kernel matrices in GPU and then compute the largest eigenvalues in CPU using SLEPc in the ways below. ``` cuDoubleComplex *h_propmat; cuDoubleComplex *h_kernelmat; int dim = EIGHT * NP * NZ; printf("\nCompute the propagator matrix.\n"); prop_matrix_nucleon_sc_av_cuda(Mn, pp_nodes.data(), cos1_nodes.data()); printf("\nCompute the propagator matrix done.\n"); printf("\nCompute the kernel matrix.\n"); kernel_matrix_nucleon_sc_av_cuda(Mn, pp_nodes.data(), pp_weights.data(), cos1_nodes.data(), cos1_weights.data()); printf("\nCompute the kernel matrix done.\n"); printf("\nCompute the full kernel matrix by multiplying kernel and propagator matrix.\n"); // Map the raw arrays to Eigen matrices (column-major order) auto *h_kernel_temp = new cuDoubleComplex [dim*dim]; matmul_cublas_cuDoubleComplex(h_kernelmat,h_propmat,h_kernel_temp,dim,dim,dim); printf("\nCompute the full kernel matrix done.\n"); // Solve the eigen system with SLEPc printf("\nSolve the eigen system in the rest frame.\n"); int nRows = dim; int nCols = dim; // Create PETSc matrix and share the data of kernel_temp Mat kernel; auto* h_kernel = (std::complex*)(h_kernel_temp); PetscCall(MatCreateDense(PETSC_COMM_WORLD, PETSC_DECIDE, PETSC_DECIDE, nRows, nCols, h_kernel_temp, &kernel)); PetscCall(MatAssemblyBegin(kernel, MAT_FINAL_ASSEMBLY)); PetscCall(MatAssemblyEnd(kernel, MAT_FINAL_ASSEMBLY)); But in this case, the compiler told me that the MatCreateDense function uses the data pointer as type of "thrust::complex" instead of "std::complex". I am sure I only configured and install PETSc in purely CPU without GPU and this codes are written in the host function. Why the function changes its behavior? Did you also meet this problem when writing the cuda codes and how to solve this problem. I tried to copy the data to a new thrust::complex matrix but this is very time consuming since my matrix is very big. Is there a way to create the Mat from the original data without changing the data type to thrust::complex in the cuda applications? Any response will be appreciated. Thank you! Best wishes, Langtian Liu ------ Institute for Theorectical Physics, Justus-Liebig-University Giessen Heinrich-Buff-Ring 16, 35392 Giessen Germany -------------- next part -------------- An HTML attachment was scrubbed... URL: From jroman at dsic.upv.es Wed Oct 2 04:31:19 2024 From: jroman at dsic.upv.es (Jose E. Roman) Date: Wed, 2 Oct 2024 09:31:19 +0000 Subject: [petsc-users] issue of MatCreateDense in the CUDA codes In-Reply-To: References: <1D918E07-839C-4369-9B82-695A08AEB0A3@dsic.upv.es> Message-ID: <71D0EEC2-9AF5-4D0E-9744-13DEA46A8F05@dsic.upv.es> Does it work if you declare everything as PetscScalar instead of cuDoubleComplex? > El 2 oct 2024, a las 11:23, ??? escribi?: > > Hi Jose, > > Since my matrix is two large, I cannot create the Mat on GPU. So I still want to create and compute the eigenvalues of this matrix on CPU using SLEPc. > > Best, > -------------------- Langtian Liu Institute for Theorectical Physics, Justus-Liebig-University Giessen Heinrich-Buff-Ring 16, 35392 Giessen Germany email: langtian.liu at icloud.com Tel: (+49)641 99 33342 > >> On Oct 2, 2024, at 11:18 AM, Jose E. Roman wrote: >> >> >> For the CUDA case you should use MatCreateDenseCUDA() instead of MatCreateDense(). With this you pass a pointer with the data on the GPU memory. But I guess "new cuDoubleComplex[dim*dim]" is allocating on the CPU, you should use cudaMalloc() instead. >> >> Jose >> >> >>> El 2 oct 2024, a las 10:56, ??? via petsc-users escribi?: >>> >>> Hi all, >>> >>> I am using the PETSc and SLEPc to solve the Faddeev equation of baryons. I encounter a problem of function MatCreateDense when changing from CPU to CPU-GPU computations. >>> At first, I write the codes in purely CPU computation in the following way and it works. >>> ``` >>> Eigen::MatrixXcd H_KER; >>> Eigen::MatrixXcd G0; >>> printf("\nCompute the propagator matrix.\n"); >>> prop_matrix_nucleon_sc_av(Mn, pp_nodes, cos1_nodes); >>> printf("\nCompute the propagator matrix done.\n"); >>> printf("\nCompute the kernel matrix.\n"); >>> bse_kernel_nucleon_sc_av(Mn, pp_nodes, pp_weights, cos1_nodes, cos1_weights); >>> printf("\nCompute the kernel matrix done.\n"); >>> printf("\nCompute the full kernel matrix by multiplying kernel and propagator matrix.\n"); >>> MatrixXcd kernel_temp = H_KER * G0; >>> printf("\nCompute the full kernel matrix done.\n"); >>> >>> // Solve the eigen system with SLEPc >>> printf("\nSolve the eigen system in the rest frame.\n"); >>> // Get the size of the Eigen matrix >>> int nRows = (int) kernel_temp.rows(); >>> int nCols = (int) kernel_temp.cols(); >>> // Create PETSc matrix and share the data of kernel_temp >>> Mat kernel; >>> PetscCall(MatCreateDense(PETSC_COMM_WORLD, PETSC_DECIDE, PETSC_DECIDE, nRows, nCols, kernel_temp.data(), &kernel)); >>> PetscCall(MatAssemblyBegin(kernel, MAT_FINAL_ASSEMBLY)); >>> PetscCall(MatAssemblyEnd(kernel, MAT_FINAL_ASSEMBLY)); >>> ``` >>> Now I change to compute the propagator and kernel matrices in GPU and then compute the largest eigenvalues in CPU using SLEPc in the ways below. >>> ``` >>> cuDoubleComplex *h_propmat; >>> cuDoubleComplex *h_kernelmat; >>> int dim = EIGHT * NP * NZ; >>> printf("\nCompute the propagator matrix.\n"); >>> prop_matrix_nucleon_sc_av_cuda(Mn, pp_nodes.data(), cos1_nodes.data()); >>> printf("\nCompute the propagator matrix done.\n"); >>> printf("\nCompute the kernel matrix.\n"); >>> kernel_matrix_nucleon_sc_av_cuda(Mn, pp_nodes.data(), pp_weights.data(), cos1_nodes.data(), cos1_weights.data()); >>> printf("\nCompute the kernel matrix done.\n"); >>> printf("\nCompute the full kernel matrix by multiplying kernel and propagator matrix.\n"); >>> // Map the raw arrays to Eigen matrices (column-major order) >>> auto *h_kernel_temp = new cuDoubleComplex [dim*dim]; >>> matmul_cublas_cuDoubleComplex(h_kernelmat,h_propmat,h_kernel_temp,dim,dim,dim); >>> printf("\nCompute the full kernel matrix done.\n"); >>> >>> // Solve the eigen system with SLEPc >>> printf("\nSolve the eigen system in the rest frame.\n"); >>> int nRows = dim; >>> int nCols = dim; >>> // Create PETSc matrix and share the data of kernel_temp >>> Mat kernel; >>> auto* h_kernel = (std::complex*)(h_kernel_temp); >>> PetscCall(MatCreateDense(PETSC_COMM_WORLD, PETSC_DECIDE, PETSC_DECIDE, nRows, nCols, h_kernel_temp, &kernel)); >>> PetscCall(MatAssemblyBegin(kernel, MAT_FINAL_ASSEMBLY)); >>> PetscCall(MatAssemblyEnd(kernel, MAT_FINAL_ASSEMBLY)); >>> But in this case, the compiler told me that the MatCreateDense function uses the data pointer as type of "thrust::complex" instead of "std::complex". >>> I am sure I only configured and install PETSc in purely CPU without GPU and this codes are written in the host function. >>> Why the function changes its behavior? Did you also meet this problem when writing the cuda codes and how to solve this problem. >>> I tried to copy the data to a new thrust::complex matrix but this is very time consuming since my matrix is very big. Is there a way to create the Mat from the original data without changing the data type to thrust::complex in the cuda applications? Any response will be appreciated. Thank you! >>> >>> Best wishes, >>> Langtian Liu >>> >>> ------ >>> Institute for Theorectical Physics, Justus-Liebig-University Giessen >>> Heinrich-Buff-Ring 16, 35392 Giessen Germany > From langtian.liu at icloud.com Wed Oct 2 05:10:57 2024 From: langtian.liu at icloud.com (=?utf-8?B?5YiY5rWq5aSp?=) Date: Wed, 2 Oct 2024 10:10:57 +0000 (UTC) Subject: [petsc-users] issue of MatCreateDense in the CUDA codes In-Reply-To: <71D0EEC2-9AF5-4D0E-9744-13DEA46A8F05@dsic.upv.es> References: <1D918E07-839C-4369-9B82-695A08AEB0A3@dsic.upv.es> <71D0EEC2-9AF5-4D0E-9744-13DEA46A8F05@dsic.upv.es> Message-ID: <80322b28-133f-4d7f-a1b3-24d3b37b455a@me.com> I cannot declare everything as PetscScalar, my strategy is computing the elements of matrix on GPU blocks by blocks and copying them back to the CPU. Finally computing the eigenvalues using SLEPc on CPU. -------------------- Langtian Liu Institute for Theorectical Physics, Justus-Liebig-University Giessen Heinrich-Buff-Ring 16, 35392 Giessen Germany email: langtian.liu at icloud.com Tel: (+49)641 99 33342 On Oct 2, 2024, at 11:31 AM, Jose E. Roman wrote: Does it work if you declare everything as PetscScalar instead of cuDoubleComplex? El 2 oct 2024, a las 11:23, ??? escribi?: Hi Jose, Since my matrix is two large, I cannot create the Mat on GPU. So I still want to create and compute the eigenvalues of this matrix on CPU using SLEPc. Best, -------------------- Langtian Liu Institute for Theorectical Physics, Justus-Liebig-University Giessen Heinrich-Buff-Ring 16, 35392 Giessen Germany email: langtian.liu at icloud.com Tel: (+49)641 99 33342 On Oct 2, 2024, at 11:18 AM, Jose E. Roman wrote: For the CUDA case you should use MatCreateDenseCUDA() instead of MatCreateDense(). With this you pass a pointer with the data on the GPU memory. But I guess "new cuDoubleComplex[dim*dim]" is allocating on the CPU, you should use cudaMalloc() instead. Jose El 2 oct 2024, a las 10:56, ??? via petsc-users escribi?: Hi all, I am using the PETSc and SLEPc to solve the Faddeev equation of baryons. I encounter a problem of function MatCreateDense when changing from CPU to CPU-GPU computations. At first, I write the codes in purely CPU computation in the following way and it works. ``` Eigen::MatrixXcd H_KER; Eigen::MatrixXcd G0; printf("\nCompute the propagator matrix.\n"); prop_matrix_nucleon_sc_av(Mn, pp_nodes, cos1_nodes); printf("\nCompute the propagator matrix done.\n"); printf("\nCompute the kernel matrix.\n"); bse_kernel_nucleon_sc_av(Mn, pp_nodes, pp_weights, cos1_nodes, cos1_weights); printf("\nCompute the kernel matrix done.\n"); printf("\nCompute the full kernel matrix by multiplying kernel and propagator matrix.\n"); MatrixXcd kernel_temp = H_KER * G0; printf("\nCompute the full kernel matrix done.\n"); // Solve the eigen system with SLEPc printf("\nSolve the eigen system in the rest frame.\n"); // Get the size of the Eigen matrix int nRows = (int) kernel_temp.rows(); int nCols = (int) kernel_temp.cols(); // Create PETSc matrix and share the data of kernel_temp Mat kernel; PetscCall(MatCreateDense(PETSC_COMM_WORLD, PETSC_DECIDE, PETSC_DECIDE, nRows, nCols, kernel_temp.data(), &kernel)); PetscCall(MatAssemblyBegin(kernel, MAT_FINAL_ASSEMBLY)); PetscCall(MatAssemblyEnd(kernel, MAT_FINAL_ASSEMBLY)); ``` Now I change to compute the propagator and kernel matrices in GPU and then compute the largest eigenvalues in CPU using SLEPc in the ways below. ``` cuDoubleComplex *h_propmat; cuDoubleComplex *h_kernelmat; int dim = EIGHT * NP * NZ; printf("\nCompute the propagator matrix.\n"); prop_matrix_nucleon_sc_av_cuda(Mn, pp_nodes.data(), cos1_nodes.data()); printf("\nCompute the propagator matrix done.\n"); printf("\nCompute the kernel matrix.\n"); kernel_matrix_nucleon_sc_av_cuda(Mn, pp_nodes.data(), pp_weights.data(), cos1_nodes.data(), cos1_weights.data()); printf("\nCompute the kernel matrix done.\n"); printf("\nCompute the full kernel matrix by multiplying kernel and propagator matrix.\n"); // Map the raw arrays to Eigen matrices (column-major order) auto *h_kernel_temp = new cuDoubleComplex [dim*dim]; matmul_cublas_cuDoubleComplex(h_kernelmat,h_propmat,h_kernel_temp,dim,dim,dim); printf("\nCompute the full kernel matrix done.\n"); // Solve the eigen system with SLEPc printf("\nSolve the eigen system in the rest frame.\n"); int nRows = dim; int nCols = dim; // Create PETSc matrix and share the data of kernel_temp Mat kernel; auto* h_kernel = (std::complex*)(h_kernel_temp); PetscCall(MatCreateDense(PETSC_COMM_WORLD, PETSC_DECIDE, PETSC_DECIDE, nRows, nCols, h_kernel_temp, &kernel)); PetscCall(MatAssemblyBegin(kernel, MAT_FINAL_ASSEMBLY)); PetscCall(MatAssemblyEnd(kernel, MAT_FINAL_ASSEMBLY)); But in this case, the compiler told me that the MatCreateDense function uses the data pointer as type of "thrust::complex" instead of "std::complex". I am sure I only configured and install PETSc in purely CPU without GPU and this codes are written in the host function. Why the function changes its behavior? Did you also meet this problem when writing the cuda codes and how to solve this problem. I tried to copy the data to a new thrust::complex matrix but this is very time consuming since my matrix is very big. Is there a way to create the Mat from the original data without changing the data type to thrust::complex in the cuda applications? Any response will be appreciated. Thank you! Best wishes, Langtian Liu ------ Institute for Theorectical Physics, Justus-Liebig-University Giessen Heinrich-Buff-Ring 16, 35392 Giessen Germany -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Oct 2 06:49:39 2024 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 2 Oct 2024 07:49:39 -0400 Subject: [petsc-users] issue of MatCreateDense in the CUDA codes In-Reply-To: <80322b28-133f-4d7f-a1b3-24d3b37b455a@me.com> References: <1D918E07-839C-4369-9B82-695A08AEB0A3@dsic.upv.es> <71D0EEC2-9AF5-4D0E-9744-13DEA46A8F05@dsic.upv.es> <80322b28-133f-4d7f-a1b3-24d3b37b455a@me.com> Message-ID: On Wed, Oct 2, 2024 at 6:11?AM ??? via petsc-users wrote: > I cannot declare everything as PetscScalar, my strategy is computing the > elements of matrix on GPU blocks by blocks and copying them back to the > CPU. Finally computing the eigenvalues using SLEPc on CPU. > Then you have to either a) have a temporary array where you copy the GPU results to the CPU type, or b) if you know the types are the same size, you can cast the pointer. Thanks, Matt > -------------------- Langtian Liu Institute for Theorectical Physics, > Justus-Liebig-University Giessen Heinrich-Buff-Ring 16, 35392 Giessen > Germany email: langtian.liu at icloud.com Tel: (+49)641 99 33342 > > On Oct 2, 2024, at 11:31 AM, Jose E. Roman wrote: > > > Does it work if you declare everything as PetscScalar instead of > cuDoubleComplex? > > El 2 oct 2024, a las 11:23, ??? escribi?: > > Hi Jose, > > Since my matrix is two large, I cannot create the Mat on GPU. So I still > want to create and compute the eigenvalues of this matrix on CPU using > SLEPc. > > Best, > -------------------- Langtian Liu Institute for Theorectical Physics, > Justus-Liebig-University Giessen Heinrich-Buff-Ring 16, 35392 Giessen > Germany email: langtian.liu at icloud.com Tel: (+49)641 99 33342 > > On Oct 2, 2024, at 11:18 AM, Jose E. Roman wrote: > > > For the CUDA case you should use MatCreateDenseCUDA() instead of > MatCreateDense(). With this you pass a pointer with the data on the GPU > memory. But I guess "new cuDoubleComplex[dim*dim]" is allocating on the > CPU, you should use cudaMalloc() instead. > > Jose > > > El 2 oct 2024, a las 10:56, ??? via petsc-users > escribi?: > > Hi all, > > I am using the PETSc and SLEPc to solve the Faddeev equation of baryons. I > encounter a problem of function MatCreateDense when changing from CPU to > CPU-GPU computations. > At first, I write the codes in purely CPU computation in the following way > and it works. > ``` > Eigen::MatrixXcd H_KER; > Eigen::MatrixXcd G0; > printf("\nCompute the propagator matrix.\n"); > prop_matrix_nucleon_sc_av(Mn, pp_nodes, cos1_nodes); > printf("\nCompute the propagator matrix done.\n"); > printf("\nCompute the kernel matrix.\n"); > bse_kernel_nucleon_sc_av(Mn, pp_nodes, pp_weights, cos1_nodes, > cos1_weights); > printf("\nCompute the kernel matrix done.\n"); > printf("\nCompute the full kernel matrix by multiplying kernel and > propagator matrix.\n"); > MatrixXcd kernel_temp = H_KER * G0; > printf("\nCompute the full kernel matrix done.\n"); > > // Solve the eigen system with SLEPc > printf("\nSolve the eigen system in the rest frame.\n"); > // Get the size of the Eigen matrix > int nRows = (int) kernel_temp.rows(); > int nCols = (int) kernel_temp.cols(); > // Create PETSc matrix and share the data of kernel_temp > Mat kernel; > PetscCall(MatCreateDense(PETSC_COMM_WORLD, PETSC_DECIDE, PETSC_DECIDE, > nRows, nCols, kernel_temp.data(), &kernel)); > PetscCall(MatAssemblyBegin(kernel, MAT_FINAL_ASSEMBLY)); > PetscCall(MatAssemblyEnd(kernel, MAT_FINAL_ASSEMBLY)); > ``` > Now I change to compute the propagator and kernel matrices in GPU and then > compute the largest eigenvalues in CPU using SLEPc in the ways below. > ``` > cuDoubleComplex *h_propmat; > cuDoubleComplex *h_kernelmat; > int dim = EIGHT * NP * NZ; > printf("\nCompute the propagator matrix.\n"); > prop_matrix_nucleon_sc_av_cuda(Mn, pp_nodes.data(), cos1_nodes.data()); > printf("\nCompute the propagator matrix done.\n"); > printf("\nCompute the kernel matrix.\n"); > kernel_matrix_nucleon_sc_av_cuda(Mn, pp_nodes.data(), pp_weights.data(), > cos1_nodes.data(), cos1_weights.data()); > printf("\nCompute the kernel matrix done.\n"); > printf("\nCompute the full kernel matrix by multiplying kernel and > propagator matrix.\n"); > // Map the raw arrays to Eigen matrices (column-major order) > auto *h_kernel_temp = new cuDoubleComplex [dim*dim]; > > matmul_cublas_cuDoubleComplex(h_kernelmat,h_propmat,h_kernel_temp,dim,dim,dim); > printf("\nCompute the full kernel matrix done.\n"); > > // Solve the eigen system with SLEPc > printf("\nSolve the eigen system in the rest frame.\n"); > int nRows = dim; > int nCols = dim; > // Create PETSc matrix and share the data of kernel_temp > Mat kernel; > auto* h_kernel = (std::complex*)(h_kernel_temp); > PetscCall(MatCreateDense(PETSC_COMM_WORLD, PETSC_DECIDE, PETSC_DECIDE, > nRows, nCols, h_kernel_temp, &kernel)); > PetscCall(MatAssemblyBegin(kernel, MAT_FINAL_ASSEMBLY)); > PetscCall(MatAssemblyEnd(kernel, MAT_FINAL_ASSEMBLY)); > But in this case, the compiler told me that the MatCreateDense function > uses the data pointer as type of "thrust::complex" instead of > "std::complex". > I am sure I only configured and install PETSc in purely CPU without GPU > and this codes are written in the host function. > Why the function changes its behavior? Did you also meet this problem when > writing the cuda codes and how to solve this problem. > I tried to copy the data to a new thrust::complex matrix but this > is very time consuming since my matrix is very big. Is there a way to > create the Mat from the original data without changing the data type to > thrust::complex in the cuda applications? Any response will be > appreciated. Thank you! > > Best wishes, > Langtian Liu > > ------ > Institute for Theorectical Physics, Justus-Liebig-University Giessen > Heinrich-Buff-Ring 16, 35392 Giessen Germany > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fYVABMc9HqFTe-LOQjYCjmdl2bvWNOESlGNrBaRjwG8Kk2w8kDck2d-5ka6MqEO-gH_ppuZo46QFEgIgkLH6$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Wed Oct 2 09:42:11 2024 From: bsmith at petsc.dev (Barry Smith) Date: Wed, 2 Oct 2024 10:42:11 -0400 Subject: [petsc-users] Multigrid Preconditioned CG on cell-centered Poisson with DMStag In-Reply-To: <5893cce0-b1ee-4998-9eba-0c6332a6485d@cs.ubc.ca> References: <1ec910d0-1b23-435c-b23a-ae1843c3bb3e@cs.ubc.ca> <633763A2-98AA-44FF-AEED-0C74F35219F3@petsc.dev> <5893cce0-b1ee-4998-9eba-0c6332a6485d@cs.ubc.ca> Message-ID: <25F7552D-86F8-4E39-B88D-6A29C27A6A19@petsc.dev> Using PetscCall(MatNullSpaceCreate(PetscObjectComm((PetscObject)dmPressure), PETSC_TRUE, 0, nullptr, &matNullSpace)); should be fine (the other approach is equivalent). Any change if you use -ksp_type gmres? Please send all the output from a sample run with -ksp_monitor_true_residual -ksp_view > On Oct 1, 2024, at 6:42?PM, Gautam Luhana wrote: > > I did think that I accounted for that but now I see that I am getting the same number of iterations in the pressure space with and without the call to MatSetNullspace so maybe I am not doing that correctly and that's where the problem is. > > I tried it in the following way first: > > PetscCall(MatNullSpaceCreate(PetscObjectComm((PetscObject)dmPressure), PETSC_TRUE, 0, nullptr, &matNullSpace)); > > PetscCall(MatSetNullSpace(*A, matNullSpace)); > > and then also as: > > PetscCall(DMGetGlobalVector(dmPressure, &constantPressure)); > PetscCall(VecSet(constantPressure, 1.0)); > PetscCall(VecNorm(constantPressure, NORM_2, &nrm)); > PetscCall(VecScale(constantPressure, 1.0 / nrm)); > PetscCall(DMCreateGlobalVector(dmPressure, &basis)); > PetscCall(VecCopy(constantPressure, basis)); > PetscCall(MatNullSpaceCreate(PetscObjectComm((PetscObject)dmPressure), PETSC_FALSE, 1, &basis, &matNullSpace)); > PetscCall(MatSetNullSpace(*A, matNullSpace)); > > Is this the right way? are there other options that need to be set? > > -Gautam > > On 2024-10-01 12:47 p.m., Barry Smith wrote: >> [CAUTION: Non-UBC Email] >> >> Are you handling the null space? Like with MatSetNullSpace()? >> >> Can you try src/ksp/ksp/tutorials/ex32.c it solves the same problem but with a simpler DMDA. I don't see any reason why using DMStag with the cell-centered values should be different. >> >> Barry >> >> >>> On Oct 1, 2024, at 3:37?PM, Gautam Luhana wrote: >>> >>> Hello, >>> >>> I am solving the stokes equations on the MAC grid using DMStag with an approximate Schur complement based preconditioner. The preconditioner needs a subdomain solve of the Poisson equation on both the velocity as well as the pressure space. I am using multigrid preconditioned CG for both the subdomain solves. While the subdomain solve on the velocity space converges in 4-5 iterations and is independent of mesh-size, the pressure solve is not mesh-size independent. Relevant information: >>> >>> Pressure space operator has Neumann boundary conditions. >>> >>> Coarse-grid operators are being generated using the Galerkin approach (A = P*Af*Pt) for both of the subdomains. >>> >>> For the pressure space, the jacobi and sor smoothers were tried with several omega and damping parameter choices. >>> >>> Symmetric smoothing (same number of pre- and post-smoothing iterations) >>> >>> Has anyone tried solving the poisson operator on the cell-centered pressure space with DMStag? Or know what might be going wrong with the mg-cg convergence? >>> >>> Thanks! >>> >>> -Gautam >>> >>> From junchao.zhang at gmail.com Wed Oct 2 10:05:16 2024 From: junchao.zhang at gmail.com (Junchao Zhang) Date: Wed, 2 Oct 2024 10:05:16 -0500 Subject: [petsc-users] issue of MatCreateDense in the CUDA codes In-Reply-To: References: Message-ID: On Wed, Oct 2, 2024 at 3:57?AM ??? via petsc-users wrote: > Hi all, > > I am using the PETSc and SLEPc to solve the Faddeev equation of baryons. I > encounter a problem of function MatCreateDense when changing from CPU to > CPU-GPU computations. > At first, I write the codes in purely CPU computation in the following way > and it works. > ``` > > Eigen::MatrixXcd H_KER > *;*Eigen::MatrixXcd G0*;* > > printf("\nCompute the propagator matrix.\n") > *;*prop_matrix_nucleon_sc_av(Mn*, *pp_nodes*, *cos1_nodes) > *;*printf("\nCompute the propagator matrix done.\n") > *;*printf("\nCompute the kernel matrix.\n") > *;*bse_kernel_nucleon_sc_av(Mn*, *pp_nodes*, *pp_weights*, *cos1_nodes*, *cos1_weights) > *;*printf("\nCompute the kernel matrix done.\n") > *;*printf("\nCompute the full kernel matrix by multiplying kernel and propagator matrix.\n") > *;*MatrixXcd kernel_temp = H_KER * G0 > *;*printf("\nCompute the full kernel matrix done.\n") > *;* > // Solve the eigen system with SLEPc > printf("\nSolve the eigen system in the rest frame.\n") > *;*// Get the size of the Eigen matrix > int nRows = (int) kernel_temp.rows() > *;*int nCols = (int) kernel_temp.cols()*;* > > // Create PETSc matrix and share the data of kernel_temp > Mat kernel > *;*PetscCall(MatCreateDense(PETSC_COMM_WORLD*, *PETSC_DECIDE*, *PETSC_DECIDE*, *nRows*, *nCols*, *kernel_temp.data()*, *&kernel)) > *;*PetscCall(MatAssemblyBegin(kernel*, *MAT_FINAL_ASSEMBLY)) > *;*PetscCall(MatAssemblyEnd(kernel*, *MAT_FINAL_ASSEMBLY))*;* > > ``` > Now I change to compute the propagator and kernel matrices in GPU and then > compute the largest eigenvalues in CPU using SLEPc in the ways below. > ``` > > cuDoubleComplex *h_propmat > *;*cuDoubleComplex *h_kernelmat*;* > > int dim = EIGHT * NP * NZ > *;*printf("\nCompute the propagator matrix.\n") > *;*prop_matrix_nucleon_sc_av_cuda(Mn*, *pp_nodes.data()*, *cos1_nodes.data()) > *;*printf("\nCompute the propagator matrix done.\n") > *;*printf("\nCompute the kernel matrix.\n") > *;*kernel_matrix_nucleon_sc_av_cuda(Mn*, *pp_nodes.data()*, *pp_weights.data()*, *cos1_nodes.data()*, *cos1_weights.data()) > *;*printf("\nCompute the kernel matrix done.\n") > *;*printf("\nCompute the full kernel matrix by multiplying kernel and propagator matrix.\n") > *;*// Map the raw arrays to Eigen matrices (column-major order) > auto *h_kernel_temp = new cuDoubleComplex [dim*dim] > *;*matmul_cublas_cuDoubleComplex(h_kernelmat*,*h_propmat*,*h_kernel_temp*,*dim*,*dim*,*dim) > *;*printf("\nCompute the full kernel matrix done.\n") > *;* > // Solve the eigen system with SLEPc > printf("\nSolve the eigen system in the rest frame.\n") > *;*int nRows = dim > *;*int nCols = dim*;* > > // Create PETSc matrix and share the data of kernel_temp > Mat kernel > *;*auto* h_kernel = (std::complex*)(h_kernel_temp) > *;*PetscCall(MatCreateDense(PETSC_COMM_WORLD*, *PETSC_DECIDE*, *PETSC_DECIDE*, *nRows*, *nCols*, *h_kernel_temp*, *&kernel)) > *;*PetscCall(MatAssemblyBegin(kernel*, **MAT_FINAL_ASSEMBLY*)) > *;*PetscCall(MatAssemblyEnd(kernel*, **MAT_FINAL_ASSEMBLY*))*;* > > But in this case, the compiler told me that the MatCreateDense function > uses the data pointer as type of "thrust::complex" instead of > "std::complex". > I am sure I only configured and install PETSc in purely CPU without GPU > and this codes are written in the host function. > Please double check that your PETSc was purely CPU configured. You can find it at the end of your configure.log to see if petsc is configured with CUDA. Since *thrust::complex *is a result of a petsc/cuda configuration, I have this doubt. > Why the function changes its behavior? Did you also meet this problem when > writing the cuda codes and how to solve this problem. > I tried to copy the data to a new thrust::complex matrix but this > is very time consuming since my matrix is very big. Is there a way to > create the Mat from the original data without changing the data type to > thrust::complex in the cuda applications? Any response will be > appreciated. Thank you! > > Best wishes, > Langtian Liu > > ------ > Institute for Theorectical Physics, Justus-Liebig-University Giessen > Heinrich-Buff-Ring 16, 35392 Giessen Germany > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Wed Oct 2 11:40:58 2024 From: bsmith at petsc.dev (Barry Smith) Date: Wed, 2 Oct 2024 12:40:58 -0400 Subject: [petsc-users] Multigrid Preconditioned CG on cell-centered Poisson with DMStag In-Reply-To: <6fa7e617-00d3-48f3-9d05-4c8d973d4df0@cs.ubc.ca> References: <1ec910d0-1b23-435c-b23a-ae1843c3bb3e@cs.ubc.ca> <633763A2-98AA-44FF-AEED-0C74F35219F3@petsc.dev> <5893cce0-b1ee-4998-9eba-0c6332a6485d@cs.ubc.ca> <25F7552D-86F8-4E39-B88D-6A29C27A6A19@petsc.dev> <6fa7e617-00d3-48f3-9d05-4c8d973d4df0@cs.ubc.ca> Message-ID: Use -mg_coarse_pc_type svd Verify that when you apply the restriction to a constant vector (all ones) it returns a constant vector, maybe scaled Verify that when you apply the interpolation to a constant vector it returns a constant vector > On Oct 2, 2024, at 12:29?PM, Gautam Luhana wrote: > > GMRES is similar i.e. converges but iteration counts go +1-2 with every doubling of the mesh. > > I am attaching the output from a cg and gmres run for a single stokes iteration with a velocity and pressure subdomain solves, subsequent iterations are similar. The pressure subdomain ksp is titled pre_mgcg. > > Adding more information in case it is relevant to isolating the issue: > > -> The coarse grid operators are formed by setting PCMGSetGalerkin to PC_MG_GALERKIN_BOTH. > > -> The interpolation / restriction operators are formed using the DMCreateInterpolation and DMCreateRestriction functions on coarsened DMs (DMStag implementation) and sent to PCMGSetInterpolation/PCMGSetRestriction. > > Thanks for the help! > > -Gautam > > > On 2024-10-02 7:42 a.m., Barry Smith wrote: >> [CAUTION: Non-UBC Email] >> >> Using >> >> PetscCall(MatNullSpaceCreate(PetscObjectComm((PetscObject)dmPressure), PETSC_TRUE, 0, nullptr, &matNullSpace)); >> >> should be fine (the other approach is equivalent). >> >> Any change if you use -ksp_type gmres? >> >> Please send all the output from a sample run with -ksp_monitor_true_residual -ksp_view >> >> >> >> >> >> >> >>> On Oct 1, 2024, at 6:42?PM, Gautam Luhana wrote: >>> >>> I did think that I accounted for that but now I see that I am getting the same number of iterations in the pressure space with and without the call to MatSetNullspace so maybe I am not doing that correctly and that's where the problem is. >>> >>> I tried it in the following way first: >>> >>> PetscCall(MatNullSpaceCreate(PetscObjectComm((PetscObject)dmPressure), PETSC_TRUE, 0, nullptr, &matNullSpace)); >>> >>> PetscCall(MatSetNullSpace(*A, matNullSpace)); >>> >>> and then also as: >>> >>> PetscCall(DMGetGlobalVector(dmPressure, &constantPressure)); >>> PetscCall(VecSet(constantPressure, 1.0)); >>> PetscCall(VecNorm(constantPressure, NORM_2, &nrm)); >>> PetscCall(VecScale(constantPressure, 1.0 / nrm)); >>> PetscCall(DMCreateGlobalVector(dmPressure, &basis)); >>> PetscCall(VecCopy(constantPressure, basis)); >>> PetscCall(MatNullSpaceCreate(PetscObjectComm((PetscObject)dmPressure), PETSC_FALSE, 1, &basis, &matNullSpace)); >>> PetscCall(MatSetNullSpace(*A, matNullSpace)); >>> >>> Is this the right way? are there other options that need to be set? >>> >>> -Gautam >>> >>> On 2024-10-01 12:47 p.m., Barry Smith wrote: >>>> [CAUTION: Non-UBC Email] >>>> >>>> Are you handling the null space? Like with MatSetNullSpace()? >>>> >>>> Can you try src/ksp/ksp/tutorials/ex32.c it solves the same problem but with a simpler DMDA. I don't see any reason why using DMStag with the cell-centered values should be different. >>>> >>>> Barry >>>> >>>> >>>>> On Oct 1, 2024, at 3:37?PM, Gautam Luhana wrote: >>>>> >>>>> Hello, >>>>> >>>>> I am solving the stokes equations on the MAC grid using DMStag with an approximate Schur complement based preconditioner. The preconditioner needs a subdomain solve of the Poisson equation on both the velocity as well as the pressure space. I am using multigrid preconditioned CG for both the subdomain solves. While the subdomain solve on the velocity space converges in 4-5 iterations and is independent of mesh-size, the pressure solve is not mesh-size independent. Relevant information: >>>>> >>>>> Pressure space operator has Neumann boundary conditions. >>>>> >>>>> Coarse-grid operators are being generated using the Galerkin approach (A = P*Af*Pt) for both of the subdomains. >>>>> >>>>> For the pressure space, the jacobi and sor smoothers were tried with several omega and damping parameter choices. >>>>> >>>>> Symmetric smoothing (same number of pre- and post-smoothing iterations) >>>>> >>>>> Has anyone tried solving the poisson operator on the cell-centered pressure space with DMStag? Or know what might be going wrong with the mg-cg convergence? >>>>> >>>>> Thanks! >>>>> >>>>> -Gautam >>>>> -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: output_gmres.txt URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: output_cg.txt URL: -------------- next part -------------- From mfadams at lbl.gov Wed Oct 2 12:41:13 2024 From: mfadams at lbl.gov (Mark Adams) Date: Wed, 2 Oct 2024 13:41:13 -0400 Subject: [petsc-users] CALL FOR PAPERS - PASC25 Message-ID: CALL FOR PAPERS Platform for Advanced Scientific Computing PASC25 FHNW Campus Brugg-Windisch Switzerland 16-18 Jun 2025 https://urldefense.us/v3/__https://pasc25.pasc-conference.org__;!!G_uCfscf7eWS!dfOCIpA5acVRpEBeEEnttRTu64tX35YckKo1Vy6rCY5Ak_SS72AmNGjPTuNOmgBA9kFggmHgNxdcVxTlKj3j9Yg$ Deadline: 06 Dec 2024 (no extensions) The Platform for Advanced Scientific Computing (PASC) invites research paper submissions for PASC25, co-sponsored by the Association for Computing Machinery (ACM) and the Swiss National Supercomputing Centre (CSCS). PASC25 will be held from June 16 to 18, 2025, at FHNW (University of Applied Sciences and Arts Northwestern Switzerland) Campus Brugg-Windisch, Switzerland. The guidelines for submissions are published at https://urldefense.us/v3/__https://pasc25.pasc-conference.org/submission/guidelines-for-papers/__;!!G_uCfscf7eWS!dfOCIpA5acVRpEBeEEnttRTu64tX35YckKo1Vy6rCY5Ak_SS72AmNGjPTuNOmgBA9kFggmHgNxdcVxTlxGfDHBs$ . The PASC Conference series is an international and interdisciplinary platform for the exchange of knowledge in scientific computing and computational science with a strong focus on methods, tools, algorithms, workflows, application challenges, and novel techniques in the context of scientific usage of high performance computing. The technical program of PASC25 is organized around the following scientific domains: Chemistry and Materials; Climate, Weather, and Earth Sciences; Applied Social Sciences and Humanities; Engineering; Life Sciences; Physics; Computational Methods and Applied Mathematics. PASC25 solicits high-quality contributions of original research related to scientific computing in all of these domains. Proposals that emphasize the theme of PASC25 - ?Supercomputing for Sustainable Development? - are particularly welcome. Papers accepted for PASC25 will be presented as talks, and published in the Proceedings of the PASC Conference, accessible via the ACM Digital Library. A selection of the highest quality papers may be given the opportunity of a plenary presentation. The PASC25 Papers Program Committee is responsible for the paper evaluation process. The committee is chaired by Sally Ellingson (University of Kentucky, US), and (Xiaoye) Sherry Li (Lawrence Berkeley National Laboratory, US), and comprises of domain co-chairs and committee members who are specialists in their scientific fields. Papers will be evaluated on their significance, technical soundness, originality, and quality of communication by reviewers listed as part of the PASC25 papers committee. Contributions must be submitted through the PASC Conference online submission portal (https://urldefense.us/v3/__https://submissions.pasc-conference.org__;!!G_uCfscf7eWS!dfOCIpA5acVRpEBeEEnttRTu64tX35YckKo1Vy6rCY5Ak_SS72AmNGjPTuNOmgBA9kFggmHgNxdcVxTlmt_YX-Y$ ). SUBMISSION DEADLINES The deadline for submissions for Papers is December 06, 2024 at 11:59 pm anywhere on earth (?AoE? or ?UTC-12?). * 06 Dec 2024: Deadline for paper submissions (no extensions!) * 04 Feb 2025: Review notifications * 16 March 2025: Deadline for paper revisions * 15 Apr 2025: Acceptance notifications -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Wed Oct 2 14:36:00 2024 From: bsmith at petsc.dev (Barry Smith) Date: Wed, 2 Oct 2024 15:36:00 -0400 Subject: [petsc-users] PETSc 3.22 release Message-ID: We are pleased to announce the release of PETSc version 3.22.0 at https://urldefense.us/v3/__https://petsc.org/release/download/__;!!G_uCfscf7eWS!bXkkk84oS5mc65RT9pvzbJJOcqwIII2FeYksMUT7WSki552oD55C8jQ3xwT_sVS4e2dJ2idgUUWFf50aqPJfiPM$ A list of the major changes and updates can be found at https://urldefense.us/v3/__https://petsc.org/release/changes/322/__;!!G_uCfscf7eWS!bXkkk84oS5mc65RT9pvzbJJOcqwIII2FeYksMUT7WSki552oD55C8jQ3xwT_sVS4e2dJ2idgUUWFf50aBobzQd8$ The final update to petsc-3.21 i.e petsc-3.21.6 is also available We recommend upgrading to PETSc 3.22.0 soon. As always, please report problems to petsc-maint at mcs.anl.gov and ask questions at petsc-users at mcs.anl.gov This release includes contributions from Alexander Lindsay Audic XU Barry Smith Blaise Bourdin Brad Aagaard Chris Kees Chris Richardson Daiane Iglesia Dolci Darsh Nathawani David Kamensky David Salac Eric Chamberland Etienne Malaboeuf Hansol Suh Ilya Fursov Iulian Grindeanu Jack Betteridge James Wright Jed Brown Jeremy L Thompson Joe Wallwork Jose Roman Junchao Zhang Koki Sagiyama Lisandro Dalcin liuyangzhuan Lukas Mark Adams Martin Diehl Matthew Knepley mint Yzr Mr. Hong Zhang Namjae Choi Pablo Brubeck Pierre Jolivet Ren? Chenard Richard Tran Mills Sarah Ahmed Satish Balay Sebastian Grimberg Stefano Zampini Stephan Kramer Toby Isaac YANG Zongze Zach Atkins and bug reports/proposed improvements received from Adrian Croucher Alexandre Halbach Cheng Tian Pei Christophe Prudhomme Christopher Douglas Derek.Teaney Eric Chamberland Frank Bramkamp Jose E. Roman Junchao Zhang Lisandro Dalcin Mehmet Sahin "Nelson, Lucas Horace" Nick OBrien via petsc-users Olivier Jamond Pierre Marchand Richard Katz Rui Martins Sebastian Blauth Sreeram R Venkat Yongzhong Li As always, thanks for your support, Barry -------------- next part -------------- An HTML attachment was scrubbed... URL: From s_g at berkeley.edu Wed Oct 2 15:08:46 2024 From: s_g at berkeley.edu (Sanjay Govindjee) Date: Wed, 2 Oct 2024 13:08:46 -0700 Subject: [petsc-users] 3.22 fortran changes Message-ID: In the release notes, it mentions the introduction of PETSC_NULL_INTEGER_ARRAY. Am I correct in interpreting this to mean that I can/should change my usage of PETSC_NULL_INTEGER(1) in function calls to PETSc routines to PETSC_NULL_INTEGER_ARRAY? -sanjay -------------- next part -------------- An HTML attachment was scrubbed... URL: From langtian.liu at icloud.com Wed Oct 2 15:11:16 2024 From: langtian.liu at icloud.com (=?utf-8?B?5YiY5rWq5aSp?=) Date: Wed, 2 Oct 2024 20:11:16 +0000 (UTC) Subject: [petsc-users] issue of MatCreateDense in the CUDA codes In-Reply-To: References: Message-ID: <4a0a535b-04cd-4989-af60-25179e7ad429@me.com> Hi Junchao, I check it, I haven't use cuda when install pure cpu version of petsc. The configure.log has been attached. Thank you for your reply. Best wishes, -------------------- Langtian Liu Institute for Theorectical Physics, Justus-Liebig-University Giessen Heinrich-Buff-Ring 16, 35392 Giessen Germany email: langtian.liu at icloud.com Tel: (+49)641 99 33342 On Oct 2, 2024, at 5:05 PM, Junchao Zhang wrote: On Wed, Oct 2, 2024 at 3:57 AM ??? via petsc-users < petsc-users at mcs.anl.gov > wrote: Hi all, I am using the PETSc and SLEPc to solve the Faddeev equation of baryons. I encounter a problem of function MatCreateDense when changing from CPU to CPU-GPU computations. At first, I write the codes in purely CPU computation in the following way and it works. ``` Eigen :: MatrixXcd H_KER ; Eigen :: MatrixXcd G0 ; printf( " \n Compute the propagator matrix. \n " ) ; prop_matrix_nucleon_sc_av( Mn , pp_nodes , cos1_nodes) ; printf( " \n Compute the propagator matrix done. \n " ) ; printf( " \n Compute the kernel matrix. \n " ) ; bse_kernel_nucleon_sc_av( Mn , pp_nodes , pp_weights , cos1_nodes , cos1_weights) ; printf( " \n Compute the kernel matrix done. \n " ) ; printf( " \n Compute the full kernel matrix by multiplying kernel and propagator matrix. \n " ) ; MatrixXcd kernel_temp = H_KER * G0 ; printf( " \n Compute the full kernel matrix done. \n " ) ; // Solve the eigen system with SLEPc printf( " \n Solve the eigen system in the rest frame. \n " ) ; // Get the size of the Eigen matrix int nRows = ( int ) kernel_temp. rows () ; int nCols = ( int ) kernel_temp. cols () ; // Create PETSc matrix and share the data of kernel_temp Mat kernel ; PetscCall(MatCreateDense(PETSC_COMM_WORLD , PETSC_DECIDE , PETSC_DECIDE , nRows , nCols , kernel_temp.data() , &kernel)) ; PetscCall(MatAssemblyBegin(kernel , MAT_FINAL_ASSEMBLY)) ; PetscCall(MatAssemblyEnd(kernel , MAT_FINAL_ASSEMBLY)) ; ``` Now I change to compute the propagator and kernel matrices in GPU and then compute the largest eigenvalues in CPU using SLEPc in the ways below. ``` cuDoubleComplex * h_propmat ; cuDoubleComplex * h_kernelmat ; int dim = EIGHT * NP * NZ ; printf( " \n Compute the propagator matrix. \n " ) ; prop_matrix_nucleon_sc_av_cuda( Mn , pp_nodes .data() , cos1_nodes .data()) ; printf( " \n Compute the propagator matrix done. \n " ) ; printf( " \n Compute the kernel matrix. \n " ) ; kernel_matrix_nucleon_sc_av_cuda( Mn , pp_nodes .data() , pp_weights .data() , cos1_nodes .data() , cos1_weights .data()) ; printf( " \n Compute the kernel matrix done. \n " ) ; printf( " \n Compute the full kernel matrix by multiplying kernel and propagator matrix. \n " ) ; // Map the raw arrays to Eigen matrices (column-major order) auto * h_kernel_temp = new cuDoubleComplex [ dim * dim ] ; matmul_cublas_cuDoubleComplex( h_kernelmat , h_propmat , h_kernel_temp , dim , dim , dim ) ; printf( " \n Compute the full kernel matrix done. \n " ) ; // Solve the eigen system with SLEPc printf( " \n Solve the eigen system in the rest frame. \n " ) ; int nRows = dim ; int nCols = dim ; // Create PETSc matrix and share the data of kernel_temp Mat kernel ; auto * h_kernel = ( std :: complex < double >*)( h_kernel_temp ) ; PetscCall (MatCreateDense( PETSC_COMM_WORLD , PETSC_DECIDE , PETSC_DECIDE , nRows , nCols , h_kernel_temp , & kernel )) ; PetscCall (MatAssemblyBegin( kernel , MAT_FINAL_ASSEMBLY )) ; PetscCall (MatAssemblyEnd( kernel , MAT_FINAL_ASSEMBLY )) ; But in this case, the compiler told me that the MatCreateDense function uses the data pointer as type of "thrust::complex" instead of "std::complex". I am sure I only configured and install PETSc in purely CPU without GPU and this codes are written in the host function. Please double check that your PETSc was purely CPU configured. You can find it at the end of your configure.log to see if petsc is configured with CUDA. Since thrust::complex is a result of a petsc/cuda configuration, I have this doubt. Why the function changes its behavior? Did you also meet this problem when writing the cuda codes and how to solve this problem. I tried to copy the data to a new thrust::complex matrix but this is very time consuming since my matrix is very big. Is there a way to create the Mat from the original data without changing the data type to thrust::complex in the cuda applications? Any response will be appreciated. Thank you! Best wishes, Langtian Liu ------ Institute for Theorectical Physics, Justus-Liebig-University Giessen Heinrich-Buff-Ring 16, 35392 Giessen Germany -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: configure.log Type: text/x-log Size: 1330087 bytes Desc: not available URL: From bsmith at petsc.dev Wed Oct 2 15:14:19 2024 From: bsmith at petsc.dev (Barry Smith) Date: Wed, 2 Oct 2024 16:14:19 -0400 Subject: [petsc-users] 3.22 fortran changes In-Reply-To: References: Message-ID: <002D684B-2EC0-4805-9396-F1DEFE69BB71@petsc.dev> Arguments that expect an array of integers and you want to pass nothing you should use PETSC_NULL_INTEGER_ARRAY > On Oct 2, 2024, at 4:08?PM, Sanjay Govindjee via petsc-users wrote: > > In the release notes, it mentions the introduction of PETSC_NULL_INTEGER_ARRAY. Am I correct in interpreting this to mean that I can/should change my usage of > PETSC_NULL_INTEGER(1) in function calls to PETSc routines to PETSC_NULL_INTEGER_ARRAY? > > -sanjay -------------- next part -------------- An HTML attachment was scrubbed... URL: From junchao.zhang at gmail.com Wed Oct 2 17:12:04 2024 From: junchao.zhang at gmail.com (Junchao Zhang) Date: Wed, 2 Oct 2024 17:12:04 -0500 Subject: [petsc-users] issue of MatCreateDense in the CUDA codes In-Reply-To: <4a0a535b-04cd-4989-af60-25179e7ad429@me.com> References: <4a0a535b-04cd-4989-af60-25179e7ad429@me.com> Message-ID: Hi, Langtian, Thanks for the configure.log and I now see what's wrong. Since you compiled your code with nvcc, we mistakenly thought petsc was configured with cuda. It is fixed in https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/merge_requests/7909__;!!G_uCfscf7eWS!c2Xrs8q_dNGecGM-HpdhjKUScE6v8qSOSZa0ApqRhZJb2JUEhBq23ydXnFivVh45LPetYtrx-9xUkF_Z7VKbetWpWzW-$ , which will be in petsc/release and main. Thanks. --Junchao Zhang On Wed, Oct 2, 2024 at 3:11?PM ??? wrote: > Hi Junchao, > > I check it, I haven't use cuda when install pure cpu version of petsc. > The configure.log has been attached. Thank you for your reply. > > Best wishes, > -------------------- Langtian Liu Institute for Theorectical Physics, > Justus-Liebig-University Giessen Heinrich-Buff-Ring 16, 35392 Giessen > Germany email: langtian.liu at icloud.com Tel: (+49)641 99 33342 > > On Oct 2, 2024, at 5:05 PM, Junchao Zhang wrote: > > > > > On Wed, Oct 2, 2024 at 3:57?AM ??? via petsc-users < > petsc-users at mcs.anl.gov> wrote: > >> Hi all, >> >> I am using the PETSc and SLEPc to solve the Faddeev equation of baryons. >> I encounter a problem of function MatCreateDense when changing from CPU to >> CPU-GPU computations. >> At first, I write the codes in purely CPU computation in the following >> way and it works. >> ``` >> >> Eigen::MatrixXcd H_KER >> *;*Eigen::MatrixXcd G0*;* >> >> printf("\nCompute the propagator matrix.\n") >> *;*prop_matrix_nucleon_sc_av(Mn*, *pp_nodes*, *cos1_nodes) >> *;*printf("\nCompute the propagator matrix done.\n") >> *;*printf("\nCompute the kernel matrix.\n") >> *;*bse_kernel_nucleon_sc_av(Mn*, *pp_nodes*, *pp_weights*, *cos1_nodes*, *cos1_weights) >> *;*printf("\nCompute the kernel matrix done.\n") >> *;*printf("\nCompute the full kernel matrix by multiplying kernel and propagator matrix.\n") >> *;*MatrixXcd kernel_temp = H_KER * G0 >> *;*printf("\nCompute the full kernel matrix done.\n") >> >> *;*// Solve the eigen system with SLEPc >> printf("\nSolve the eigen system in the rest frame.\n") >> *;*// Get the size of the Eigen matrix >> int nRows = (int) kernel_temp.rows() >> *;*int nCols = (int) kernel_temp.cols()*;* >> >> // Create PETSc matrix and share the data of kernel_temp >> Mat kernel >> *;*PetscCall(MatCreateDense(PETSC_COMM_WORLD*, *PETSC_DECIDE*, *PETSC_DECIDE*, *nRows*, *nCols*, *kernel_temp.data()*, *&kernel)) >> *;*PetscCall(MatAssemblyBegin(kernel*, *MAT_FINAL_ASSEMBLY)) >> *;*PetscCall(MatAssemblyEnd(kernel*, *MAT_FINAL_ASSEMBLY))*;* >> >> ``` >> Now I change to compute the propagator and kernel matrices in GPU and >> then compute the largest eigenvalues in CPU using SLEPc in the ways below. >> ``` >> >> cuDoubleComplex *h_propmat >> *;*cuDoubleComplex *h_kernelmat*;* >> >> int dim = EIGHT * NP * NZ >> *;*printf("\nCompute the propagator matrix.\n") >> *;*prop_matrix_nucleon_sc_av_cuda(Mn*, *pp_nodes.data()*, *cos1_nodes.data()) >> *;*printf("\nCompute the propagator matrix done.\n") >> *;*printf("\nCompute the kernel matrix.\n") >> *;*kernel_matrix_nucleon_sc_av_cuda(Mn*, *pp_nodes.data()*, *pp_weights.data()*, *cos1_nodes.data()*, *cos1_weights.data()) >> *;*printf("\nCompute the kernel matrix done.\n") >> *;*printf("\nCompute the full kernel matrix by multiplying kernel and propagator matrix.\n") >> *;*// Map the raw arrays to Eigen matrices (column-major order) >> auto *h_kernel_temp = new cuDoubleComplex [dim*dim] >> *;*matmul_cublas_cuDoubleComplex(h_kernelmat*,*h_propmat*,*h_kernel_temp*,*dim*,*dim*,*dim) >> *;*printf("\nCompute the full kernel matrix done.\n") >> >> *;*// Solve the eigen system with SLEPc >> printf("\nSolve the eigen system in the rest frame.\n") >> *;*int nRows = dim >> *;*int nCols = dim*;* >> >> // Create PETSc matrix and share the data of kernel_temp >> Mat kernel >> *;*auto* h_kernel = (std::complex*)(h_kernel_temp) >> *;*PetscCall(MatCreateDense(PETSC_COMM_WORLD*, *PETSC_DECIDE*, *PETSC_DECIDE*, *nRows*, *nCols*, *h_kernel_temp*, *&kernel)) >> *;*PetscCall(MatAssemblyBegin(kernel*, **MAT_FINAL_ASSEMBLY*)) >> *;*PetscCall(MatAssemblyEnd(kernel*, **MAT_FINAL_ASSEMBLY*))*;* >> >> But in this case, the compiler told me that the MatCreateDense function >> uses the data pointer as type of "thrust::complex" instead of >> "std::complex". >> I am sure I only configured and install PETSc in purely CPU without GPU >> and this codes are written in the host function. >> > Please double check that your PETSc was purely CPU configured. You can > find it at the end of your configure.log to see if petsc is configured with > CUDA. > Since *thrust::complex *is a result of a petsc/cuda > configuration, I have this doubt. > > > >> >> Why the function changes its behavior? Did you also meet this problem >> when writing the cuda codes and how to solve this problem. >> I tried to copy the data to a new thrust::complex matrix but this >> is very time consuming since my matrix is very big. Is there a way to >> create the Mat from the original data without changing the data type to >> thrust::complex in the cuda applications? Any response will be >> appreciated. Thank you! >> >> Best wishes, >> Langtian Liu >> >> ------ >> Institute for Theorectical Physics, Justus-Liebig-University Giessen >> Heinrich-Buff-Ring 16, 35392 Giessen Germany >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From langtian.liu at icloud.com Thu Oct 3 02:55:23 2024 From: langtian.liu at icloud.com (=?utf-8?B?5YiY5rWq5aSp?=) Date: Thu, 3 Oct 2024 07:55:23 +0000 (UTC) Subject: [petsc-users] issue of MatCreateDense in the CUDA codes In-Reply-To: References: <4a0a535b-04cd-4989-af60-25179e7ad429@me.com> Message-ID: Hello Junchao, Okay. Thank you for helping find this bug. I pull the newest version of petsc today. It seems this error has not been fixed in the present release version. Maybe I should wait for some days. Best wishes, Langtian On Oct 3, 2024, at 12:12 AM, Junchao Zhang wrote: Hi, Langtian, Thanks for the configure.log and I now see what's wrong. Since you compiled your code with nvcc, we mistakenly thought petsc was configured with cuda. It is fixed in https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/merge_requests/7909__;!!G_uCfscf7eWS!dIQhZmHAR37vEaIvWtn-b9ggurJgaZ9R4aznsluuwVI2-1OfMczg_v0sTOQyO7Xkgb22zZuJiua8J3OFDzsQcUK3LiNP$ , which will be in petsc/release and main. Thanks. --Junchao Zhang On Wed, Oct 2, 2024 at 3:11 PM ??? < langtian.liu at icloud.com > wrote: Hi Junchao, I check it, I haven't use cuda when install pure cpu version of petsc. The configure.log has been attached. Thank you for your reply. Best wishes, -------------------- Langtian Liu Institute for Theorectical Physics, Justus-Liebig-University Giessen Heinrich-Buff-Ring 16, 35392 Giessen Germany email: langtian.liu at icloud.com Tel: (+49)641 99 33342 On Oct 2, 2024, at 5:05 PM, Junchao Zhang < junchao.zhang at gmail.com > wrote: On Wed, Oct 2, 2024 at 3:57 AM ??? via petsc-users < petsc-users at mcs.anl.gov > wrote: Hi all, I am using the PETSc and SLEPc to solve the Faddeev equation of baryons. I encounter a problem of function MatCreateDense when changing from CPU to CPU-GPU computations. At first, I write the codes in purely CPU computation in the following way and it works. ``` Eigen :: MatrixXcd H_KER ; Eigen :: MatrixXcd G0 ; printf( " \n Compute the propagator matrix. \n " ) ; prop_matrix_nucleon_sc_av( Mn , pp_nodes , cos1_nodes) ; printf( " \n Compute the propagator matrix done. \n " ) ; printf( " \n Compute the kernel matrix. \n " ) ; bse_kernel_nucleon_sc_av( Mn , pp_nodes , pp_weights , cos1_nodes , cos1_weights) ; printf( " \n Compute the kernel matrix done. \n " ) ; printf( " \n Compute the full kernel matrix by multiplying kernel and propagator matrix. \n " ) ; MatrixXcd kernel_temp = H_KER * G0 ; printf( " \n Compute the full kernel matrix done. \n " ) ; // Solve the eigen system with SLEPc printf( " \n Solve the eigen system in the rest frame. \n " ) ; // Get the size of the Eigen matrix int nRows = ( int ) kernel_temp. rows () ; int nCols = ( int ) kernel_temp. cols () ; // Create PETSc matrix and share the data of kernel_temp Mat kernel ; PetscCall(MatCreateDense(PETSC_COMM_WORLD , PETSC_DECIDE , PETSC_DECIDE , nRows , nCols , kernel_temp.data() , &kernel)) ; PetscCall(MatAssemblyBegin(kernel , MAT_FINAL_ASSEMBLY)) ; PetscCall(MatAssemblyEnd(kernel , MAT_FINAL_ASSEMBLY)) ; ``` Now I change to compute the propagator and kernel matrices in GPU and then compute the largest eigenvalues in CPU using SLEPc in the ways below. ``` cuDoubleComplex * h_propmat ; cuDoubleComplex * h_kernelmat ; int dim = EIGHT * NP * NZ ; printf( " \n Compute the propagator matrix. \n " ) ; prop_matrix_nucleon_sc_av_cuda( Mn , pp_nodes .data() , cos1_nodes .data()) ; printf( " \n Compute the propagator matrix done. \n " ) ; printf( " \n Compute the kernel matrix. \n " ) ; kernel_matrix_nucleon_sc_av_cuda( Mn , pp_nodes .data() , pp_weights .data() , cos1_nodes .data() , cos1_weights .data()) ; printf( " \n Compute the kernel matrix done. \n " ) ; printf( " \n Compute the full kernel matrix by multiplying kernel and propagator matrix. \n " ) ; // Map the raw arrays to Eigen matrices (column-major order) auto * h_kernel_temp = new cuDoubleComplex [ dim * dim ] ; matmul_cublas_cuDoubleComplex( h_kernelmat , h_propmat , h_kernel_temp , dim , dim , dim ) ; printf( " \n Compute the full kernel matrix done. \n " ) ; // Solve the eigen system with SLEPc printf( " \n Solve the eigen system in the rest frame. \n " ) ; int nRows = dim ; int nCols = dim ; // Create PETSc matrix and share the data of kernel_temp Mat kernel ; auto * h_kernel = ( std :: complex < double >*)( h_kernel_temp ) ; PetscCall (MatCreateDense( PETSC_COMM_WORLD , PETSC_DECIDE , PETSC_DECIDE , nRows , nCols , h_kernel_temp , & kernel )) ; PetscCall (MatAssemblyBegin( kernel , MAT_FINAL_ASSEMBLY )) ; PetscCall (MatAssemblyEnd( kernel , MAT_FINAL_ASSEMBLY )) ; But in this case, the compiler told me that the MatCreateDense function uses the data pointer as type of "thrust::complex" instead of "std::complex". I am sure I only configured and install PETSc in purely CPU without GPU and this codes are written in the host function. Please double check that your PETSc was purely CPU configured. You can find it at the end of your configure.log to see if petsc is configured with CUDA. Since thrust::complex is a result of a petsc/cuda configuration, I have this doubt. Why the function changes its behavior? Did you also meet this problem when writing the cuda codes and how to solve this problem. I tried to copy the data to a new thrust::complex matrix but this is very time consuming since my matrix is very big. Is there a way to create the Mat from the original data without changing the data type to thrust::complex in the cuda applications? Any response will be appreciated. Thank you! Best wishes, Langtian Liu ------ Institute for Theorectical Physics, Justus-Liebig-University Giessen Heinrich-Buff-Ring 16, 35392 Giessen Germany -------------- next part -------------- An HTML attachment was scrubbed... URL: From jroman at dsic.upv.es Thu Oct 3 02:58:18 2024 From: jroman at dsic.upv.es (Jose E. Roman) Date: Thu, 3 Oct 2024 07:58:18 +0000 Subject: [petsc-users] issue of MatCreateDense in the CUDA codes In-Reply-To: References: <4a0a535b-04cd-4989-af60-25179e7ad429@me.com> Message-ID: <83227ED1-668F-470C-8F4D-CE0B308505BF@dsic.upv.es> You have to wait until the merge request has the label "Merged" instead of "Open". > El 3 oct 2024, a las 9:55, ??? via petsc-users escribi?: > > Hello Junchao, > > Okay. Thank you for helping find this bug. I pull the newest version of petsc today. It seems this error has not been fixed in the present release version. Maybe I should wait for some days. > > Best wishes, > Langtian > > >> On Oct 3, 2024, at 12:12 AM, Junchao Zhang wrote: >> >> >> Hi, Langtian, >> Thanks for the configure.log and I now see what's wrong. Since you compiled your code with nvcc, we mistakenly thought petsc was configured with cuda. >> It is fixed in https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/merge_requests/7909__;!!G_uCfscf7eWS!aeorPzG-t4hjjYuvNrXAN30S9m17SudfU_aVI1QhsSjB6SJ1ezv8tqvb5tC5u3kC-MAPhaI_HsiKr5Pt_XnJjIrC$ , which will be in petsc/release and main. >> >> Thanks. >> --Junchao Zhang >> >> >> On Wed, Oct 2, 2024 at 3:11?PM ??? wrote: >> Hi Junchao, >> >> I check it, I haven't use cuda when install pure cpu version of petsc. >> The configure.log has been attached. Thank you for your reply. >> >> Best wishes, >> -------------------- Langtian Liu Institute for Theorectical Physics, Justus-Liebig-University Giessen Heinrich-Buff-Ring 16, 35392 Giessen Germany email: langtian.liu at icloud.com Tel: (+49)641 99 33342 >> >>> On Oct 2, 2024, at 5:05 PM, Junchao Zhang wrote: >>> >>> >>> >>> >>> On Wed, Oct 2, 2024 at 3:57?AM ??? via petsc-users wrote: >>> Hi all, >>> >>> I am using the PETSc and SLEPc to solve the Faddeev equation of baryons. I encounter a problem of function MatCreateDense when changing from CPU to CPU-GPU computations. >>> At first, I write the codes in purely CPU computation in the following way and it works. >>> ``` >>> Eigen::MatrixXcd H_KER; >>> Eigen::MatrixXcd G0; >>> printf("\nCompute the propagator matrix.\n"); >>> prop_matrix_nucleon_sc_av(Mn, pp_nodes, cos1_nodes); >>> printf("\nCompute the propagator matrix done.\n"); >>> printf("\nCompute the kernel matrix.\n"); >>> bse_kernel_nucleon_sc_av(Mn, pp_nodes, pp_weights, cos1_nodes, cos1_weights); >>> printf("\nCompute the kernel matrix done.\n"); >>> printf("\nCompute the full kernel matrix by multiplying kernel and propagator matrix.\n"); >>> MatrixXcd kernel_temp = H_KER * G0; >>> printf("\nCompute the full kernel matrix done.\n"); >>> >>> // Solve the eigen system with SLEPc >>> printf("\nSolve the eigen system in the rest frame.\n"); >>> // Get the size of the Eigen matrix >>> int nRows = (int) kernel_temp.rows(); >>> int nCols = (int) kernel_temp.cols(); >>> // Create PETSc matrix and share the data of kernel_temp >>> Mat kernel; >>> PetscCall(MatCreateDense(PETSC_COMM_WORLD, PETSC_DECIDE, PETSC_DECIDE, nRows, nCols, kernel_temp.data(), &kernel)); >>> PetscCall(MatAssemblyBegin(kernel, MAT_FINAL_ASSEMBLY)); >>> PetscCall(MatAssemblyEnd(kernel, MAT_FINAL_ASSEMBLY)); >>> ``` >>> Now I change to compute the propagator and kernel matrices in GPU and then compute the largest eigenvalues in CPU using SLEPc in the ways below. >>> ``` >>> cuDoubleComplex *h_propmat; >>> cuDoubleComplex *h_kernelmat; >>> int dim = EIGHT * NP * NZ; >>> printf("\nCompute the propagator matrix.\n"); >>> prop_matrix_nucleon_sc_av_cuda(Mn, pp_nodes.data(), cos1_nodes.data()); >>> printf("\nCompute the propagator matrix done.\n"); >>> printf("\nCompute the kernel matrix.\n"); >>> kernel_matrix_nucleon_sc_av_cuda(Mn, pp_nodes.data(), pp_weights.data(), cos1_nodes.data(), cos1_weights.data()); >>> printf("\nCompute the kernel matrix done.\n"); >>> printf("\nCompute the full kernel matrix by multiplying kernel and propagator matrix.\n"); >>> // Map the raw arrays to Eigen matrices (column-major order) >>> auto *h_kernel_temp = new cuDoubleComplex [dim*dim]; >>> matmul_cublas_cuDoubleComplex(h_kernelmat,h_propmat,h_kernel_temp,dim,dim,dim); >>> printf("\nCompute the full kernel matrix done.\n"); >>> >>> // Solve the eigen system with SLEPc >>> printf("\nSolve the eigen system in the rest frame.\n"); >>> int nRows = dim; >>> int nCols = dim; >>> // Create PETSc matrix and share the data of kernel_temp >>> Mat kernel; >>> auto* h_kernel = (std::complex*)(h_kernel_temp); >>> PetscCall(MatCreateDense(PETSC_COMM_WORLD, PETSC_DECIDE, PETSC_DECIDE, nRows, nCols, h_kernel_temp, &kernel)); >>> PetscCall(MatAssemblyBegin(kernel, MAT_FINAL_ASSEMBLY)); >>> PetscCall(MatAssemblyEnd(kernel, MAT_FINAL_ASSEMBLY)); >>> But in this case, the compiler told me that the MatCreateDense function uses the data pointer as type of "thrust::complex" instead of "std::complex". >>> I am sure I only configured and install PETSc in purely CPU without GPU and this codes are written in the host function. >>> Please double check that your PETSc was purely CPU configured. You can find it at the end of your configure.log to see if petsc is configured with CUDA. >>> Since thrust::complex is a result of a petsc/cuda configuration, I have this doubt. >>> >>> >>> >>> Why the function changes its behavior? Did you also meet this problem when writing the cuda codes and how to solve this problem. >>> I tried to copy the data to a new thrust::complex matrix but this is very time consuming since my matrix is very big. Is there a way to create the Mat from the original data without changing the data type to thrust::complex in the cuda applications? Any response will be appreciated. Thank you! >>> >>> Best wishes, >>> Langtian Liu >>> >>> ------ >>> Institute for Theorectical Physics, Justus-Liebig-University Giessen >>> Heinrich-Buff-Ring 16, 35392 Giessen Germany >>> >> > From langtian.liu at icloud.com Thu Oct 3 03:03:32 2024 From: langtian.liu at icloud.com (=?utf-8?B?5YiY5rWq5aSp?=) Date: Thu, 3 Oct 2024 08:03:32 +0000 (UTC) Subject: [petsc-users] issue of MatCreateDense in the CUDA codes In-Reply-To: <83227ED1-668F-470C-8F4D-CE0B308505BF@dsic.upv.es> References: <4a0a535b-04cd-4989-af60-25179e7ad429@me.com> <83227ED1-668F-470C-8F4D-CE0B308505BF@dsic.upv.es> Message-ID: Okay. I see :D -------------------- Langtian Liu Institute for Theorectical Physics, Justus-Liebig-University Giessen Heinrich-Buff-Ring 16, 35392 Giessen Germany email: langtian.liu at icloud.com Tel: (+49)641 99 33342 On Oct 3, 2024, at 9:58 AM, Jose E. Roman wrote: You have to wait until the merge request has the label "Merged" instead of "Open". El 3 oct 2024, a las 9:55, ??? via petsc-users escribi?: Hello Junchao, Okay. Thank you for helping find this bug. I pull the newest version of petsc today. It seems this error has not been fixed in the present release version. Maybe I should wait for some days. Best wishes, Langtian On Oct 3, 2024, at 12:12 AM, Junchao Zhang wrote: Hi, Langtian, Thanks for the configure.log and I now see what's wrong. Since you compiled your code with nvcc, we mistakenly thought petsc was configured with cuda. It is fixed in https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/merge_requests/7909__;!!G_uCfscf7eWS!aqNRfMgxRIZOjZPHUylOegaKwo0fVaU4tiE7HxYpmfzEDt1AeL7boycnCp17c5H5jJ_a35QHydt5TCOvzH_U65F6mBW3$ , which will be in petsc/release and main. Thanks. --Junchao Zhang On Wed, Oct 2, 2024 at 3:11 PM ??? wrote: Hi Junchao, I check it, I haven't use cuda when install pure cpu version of petsc. The configure.log has been attached. Thank you for your reply. Best wishes, -------------------- Langtian Liu Institute for Theorectical Physics, Justus-Liebig-University Giessen Heinrich-Buff-Ring 16, 35392 Giessen Germany email: langtian.liu at icloud.com Tel: (+49)641 99 33342 On Oct 2, 2024, at 5:05 PM, Junchao Zhang wrote: On Wed, Oct 2, 2024 at 3:57 AM ??? via petsc-users wrote: Hi all, I am using the PETSc and SLEPc to solve the Faddeev equation of baryons. I encounter a problem of function MatCreateDense when changing from CPU to CPU-GPU computations. At first, I write the codes in purely CPU computation in the following way and it works. ``` Eigen::MatrixXcd H_KER; Eigen::MatrixXcd G0; printf("\nCompute the propagator matrix.\n"); prop_matrix_nucleon_sc_av(Mn, pp_nodes, cos1_nodes); printf("\nCompute the propagator matrix done.\n"); printf("\nCompute the kernel matrix.\n"); bse_kernel_nucleon_sc_av(Mn, pp_nodes, pp_weights, cos1_nodes, cos1_weights); printf("\nCompute the kernel matrix done.\n"); printf("\nCompute the full kernel matrix by multiplying kernel and propagator matrix.\n"); MatrixXcd kernel_temp = H_KER * G0; printf("\nCompute the full kernel matrix done.\n"); // Solve the eigen system with SLEPc printf("\nSolve the eigen system in the rest frame.\n"); // Get the size of the Eigen matrix int nRows = (int) kernel_temp.rows(); int nCols = (int) kernel_temp.cols(); // Create PETSc matrix and share the data of kernel_temp Mat kernel; PetscCall(MatCreateDense(PETSC_COMM_WORLD, PETSC_DECIDE, PETSC_DECIDE, nRows, nCols, kernel_temp.data(), &kernel)); PetscCall(MatAssemblyBegin(kernel, MAT_FINAL_ASSEMBLY)); PetscCall(MatAssemblyEnd(kernel, MAT_FINAL_ASSEMBLY)); ``` Now I change to compute the propagator and kernel matrices in GPU and then compute the largest eigenvalues in CPU using SLEPc in the ways below. ``` cuDoubleComplex *h_propmat; cuDoubleComplex *h_kernelmat; int dim = EIGHT * NP * NZ; printf("\nCompute the propagator matrix.\n"); prop_matrix_nucleon_sc_av_cuda(Mn, pp_nodes.data(), cos1_nodes.data()); printf("\nCompute the propagator matrix done.\n"); printf("\nCompute the kernel matrix.\n"); kernel_matrix_nucleon_sc_av_cuda(Mn, pp_nodes.data(), pp_weights.data(), cos1_nodes.data(), cos1_weights.data()); printf("\nCompute the kernel matrix done.\n"); printf("\nCompute the full kernel matrix by multiplying kernel and propagator matrix.\n"); // Map the raw arrays to Eigen matrices (column-major order) auto *h_kernel_temp = new cuDoubleComplex [dim*dim]; matmul_cublas_cuDoubleComplex(h_kernelmat,h_propmat,h_kernel_temp,dim,dim,dim); printf("\nCompute the full kernel matrix done.\n"); // Solve the eigen system with SLEPc printf("\nSolve the eigen system in the rest frame.\n"); int nRows = dim; int nCols = dim; // Create PETSc matrix and share the data of kernel_temp Mat kernel; auto* h_kernel = (std::complex*)(h_kernel_temp); PetscCall(MatCreateDense(PETSC_COMM_WORLD, PETSC_DECIDE, PETSC_DECIDE, nRows, nCols, h_kernel_temp, &kernel)); PetscCall(MatAssemblyBegin(kernel, MAT_FINAL_ASSEMBLY)); PetscCall(MatAssemblyEnd(kernel, MAT_FINAL_ASSEMBLY)); But in this case, the compiler told me that the MatCreateDense function uses the data pointer as type of "thrust::complex" instead of "std::complex". I am sure I only configured and install PETSc in purely CPU without GPU and this codes are written in the host function. Please double check that your PETSc was purely CPU configured. You can find it at the end of your configure.log to see if petsc is configured with CUDA. Since thrust::complex is a result of a petsc/cuda configuration, I have this doubt. Why the function changes its behavior? Did you also meet this problem when writing the cuda codes and how to solve this problem. I tried to copy the data to a new thrust::complex matrix but this is very time consuming since my matrix is very big. Is there a way to create the Mat from the original data without changing the data type to thrust::complex in the cuda applications? Any response will be appreciated. Thank you! Best wishes, Langtian Liu ------ Institute for Theorectical Physics, Justus-Liebig-University Giessen Heinrich-Buff-Ring 16, 35392 Giessen Germany -------------- next part -------------- An HTML attachment was scrubbed... URL: From edoardo.alinovi at gmail.com Thu Oct 3 05:13:52 2024 From: edoardo.alinovi at gmail.com (Edoardo alinovi) Date: Thu, 3 Oct 2024 12:13:52 +0200 Subject: [petsc-users] getinfo for Mat of type nest Message-ID: Hi Barry, I am trying to port this preconditioner I found presented at the last PETSc conference into may code: https://urldefense.us/v3/__https://github.com/ndjinga/SOLVERLAB/blob/master/CoreFlows/examples/C/SaddlePointPreconditioner/SaddlePointLinearSolver.c__;!!G_uCfscf7eWS!a-JnrBsta75n_WDtxUNd7Xb8_m7tg1pWSxwWNTmUBSfpBwISq4IxJkbM6QgADd-x23Q_2s-d6d6bOWCKGRMxMzHyc9DDm7A$ So far so good, I am just translating C to Fortran, however when I try to solve my linear system with: call KSPSetOperators(ksp, Ahat, Pmat, ierr) call KSPSolve(ksp, bhat, xhat, ierr) with Ahat and Pmat being indeed two nest MatNest, created from an array of Mat as: call MatCreateNest(PETSC_COMM_WORLD, 2, PETSC_NULL_IS, 2, PETSC_NULL_IS, Mat_array, Ahat, ierr) call MatCreateNest(PETSC_COMM_WORLD, 2, PETSC_NULL_IS, 2, PETSC_NULL_IS, Mat_array, Pmat, ierr) However I am getting this error on getinfo: from C to Fortran [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: No support for this operation for this object type [0]PETSC ERROR: No method getinfo for Mat of type nest [0]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!a-JnrBsta75n_WDtxUNd7Xb8_m7tg1pWSxwWNTmUBSfpBwISq4IxJkbM6QgADd-x23Q_2s-d6d6bOWCKGRMxMzHyRUMnyvE$ for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.20.4, unknown How can I get rid of this issue? Many thanks! -------------- next part -------------- An HTML attachment was scrubbed... URL: From junchao.zhang at gmail.com Thu Oct 3 09:38:30 2024 From: junchao.zhang at gmail.com (Junchao Zhang) Date: Thu, 3 Oct 2024 09:38:30 -0500 Subject: [petsc-users] issue of MatCreateDense in the CUDA codes In-Reply-To: References: <4a0a535b-04cd-4989-af60-25179e7ad429@me.com> <83227ED1-668F-470C-8F4D-CE0B308505BF@dsic.upv.es> Message-ID: The MR is merged to petsc/release. BTW, in MatCreateDense, the data pointer has to be a host pointer. It is better to always use PetscScalar* (instead of std::complex*) to do the cast. --Junchao Zhang On Thu, Oct 3, 2024 at 3:03?AM ??? wrote: > Okay. I see :D > -------------------- Langtian Liu Institute for Theorectical Physics, > Justus-Liebig-University Giessen Heinrich-Buff-Ring 16, 35392 Giessen > Germany email: langtian.liu at icloud.com Tel: (+49)641 99 33342 > > On Oct 3, 2024, at 9:58 AM, Jose E. Roman wrote: > > > You have to wait until the merge request has the label "Merged" instead of > "Open". > > > El 3 oct 2024, a las 9:55, ??? via petsc-users > escribi?: > > Hello Junchao, > > Okay. Thank you for helping find this bug. I pull the newest version of > petsc today. It seems this error has not been fixed in the present release > version. Maybe I should wait for some days. > > Best wishes, > Langtian > > > On Oct 3, 2024, at 12:12 AM, Junchao Zhang > wrote: > > > Hi, Langtian, > Thanks for the configure.log and I now see what's wrong. Since you > compiled your code with nvcc, we mistakenly thought petsc was configured > with cuda. > It is fixed in https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/merge_requests/7909__;!!G_uCfscf7eWS!ckm_ynevPO4eoYiGVdofOAnizLKzlNsHf92YV6b-QZr4V0eo-HEk7MWC5flyF7VZFiqAwC3eoy-BPWGyUs0bDjx4iNhW$ , > which will be in petsc/release and main. > > Thanks. > --Junchao Zhang > > > On Wed, Oct 2, 2024 at 3:11?PM ??? wrote: > Hi Junchao, > > I check it, I haven't use cuda when install pure cpu version of petsc. > The configure.log has been attached. Thank you for your reply. > > Best wishes, > -------------------- Langtian Liu Institute for Theorectical Physics, > Justus-Liebig-University Giessen Heinrich-Buff-Ring 16, 35392 Giessen > Germany email: langtian.liu at icloud.com Tel: (+49)641 99 33342 > > On Oct 2, 2024, at 5:05 PM, Junchao Zhang wrote: > > > > > On Wed, Oct 2, 2024 at 3:57?AM ??? via petsc-users < > petsc-users at mcs.anl.gov> wrote: > Hi all, > > I am using the PETSc and SLEPc to solve the Faddeev equation of baryons. I > encounter a problem of function MatCreateDense when changing from CPU to > CPU-GPU computations. > At first, I write the codes in purely CPU computation in the following way > and it works. > ``` > Eigen::MatrixXcd H_KER; > Eigen::MatrixXcd G0; > printf("\nCompute the propagator matrix.\n"); > prop_matrix_nucleon_sc_av(Mn, pp_nodes, cos1_nodes); > printf("\nCompute the propagator matrix done.\n"); > printf("\nCompute the kernel matrix.\n"); > bse_kernel_nucleon_sc_av(Mn, pp_nodes, pp_weights, cos1_nodes, > cos1_weights); > printf("\nCompute the kernel matrix done.\n"); > printf("\nCompute the full kernel matrix by multiplying kernel and > propagator matrix.\n"); > MatrixXcd kernel_temp = H_KER * G0; > printf("\nCompute the full kernel matrix done.\n"); > > // Solve the eigen system with SLEPc > printf("\nSolve the eigen system in the rest frame.\n"); > // Get the size of the Eigen matrix > int nRows = (int) kernel_temp.rows(); > int nCols = (int) kernel_temp.cols(); > // Create PETSc matrix and share the data of kernel_temp > Mat kernel; > PetscCall(MatCreateDense(PETSC_COMM_WORLD, PETSC_DECIDE, PETSC_DECIDE, > nRows, nCols, kernel_temp.data(), &kernel)); > PetscCall(MatAssemblyBegin(kernel, MAT_FINAL_ASSEMBLY)); > PetscCall(MatAssemblyEnd(kernel, MAT_FINAL_ASSEMBLY)); > ``` > Now I change to compute the propagator and kernel matrices in GPU and then > compute the largest eigenvalues in CPU using SLEPc in the ways below. > ``` > cuDoubleComplex *h_propmat; > cuDoubleComplex *h_kernelmat; > int dim = EIGHT * NP * NZ; > printf("\nCompute the propagator matrix.\n"); > prop_matrix_nucleon_sc_av_cuda(Mn, pp_nodes.data(), cos1_nodes.data()); > printf("\nCompute the propagator matrix done.\n"); > printf("\nCompute the kernel matrix.\n"); > kernel_matrix_nucleon_sc_av_cuda(Mn, pp_nodes.data(), pp_weights.data(), > cos1_nodes.data(), cos1_weights.data()); > printf("\nCompute the kernel matrix done.\n"); > printf("\nCompute the full kernel matrix by multiplying kernel and > propagator matrix.\n"); > // Map the raw arrays to Eigen matrices (column-major order) > auto *h_kernel_temp = new cuDoubleComplex [dim*dim]; > > matmul_cublas_cuDoubleComplex(h_kernelmat,h_propmat,h_kernel_temp,dim,dim,dim); > printf("\nCompute the full kernel matrix done.\n"); > > // Solve the eigen system with SLEPc > printf("\nSolve the eigen system in the rest frame.\n"); > int nRows = dim; > int nCols = dim; > // Create PETSc matrix and share the data of kernel_temp > Mat kernel; > auto* h_kernel = (std::complex*)(h_kernel_temp); > PetscCall(MatCreateDense(PETSC_COMM_WORLD, PETSC_DECIDE, PETSC_DECIDE, > nRows, nCols, h_kernel_temp, &kernel)); > PetscCall(MatAssemblyBegin(kernel, MAT_FINAL_ASSEMBLY)); > PetscCall(MatAssemblyEnd(kernel, MAT_FINAL_ASSEMBLY)); > But in this case, the compiler told me that the MatCreateDense function > uses the data pointer as type of "thrust::complex" instead of > "std::complex". > I am sure I only configured and install PETSc in purely CPU without GPU > and this codes are written in the host function. > Please double check that your PETSc was purely CPU configured. You can > find it at the end of your configure.log to see if petsc is configured with > CUDA. > Since thrust::complex is a result of a petsc/cuda configuration, I > have this doubt. > > > > Why the function changes its behavior? Did you also meet this problem when > writing the cuda codes and how to solve this problem. > I tried to copy the data to a new thrust::complex matrix but this > is very time consuming since my matrix is very big. Is there a way to > create the Mat from the original data without changing the data type to > thrust::complex in the cuda applications? Any response will be > appreciated. Thank you! > > Best wishes, > Langtian Liu > > ------ > Institute for Theorectical Physics, Justus-Liebig-University Giessen > Heinrich-Buff-Ring 16, 35392 Giessen Germany > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From langtian.liu at icloud.com Thu Oct 3 10:23:55 2024 From: langtian.liu at icloud.com (=?utf-8?B?5YiY5rWq5aSp?=) Date: Thu, 3 Oct 2024 15:23:55 +0000 (UTC) Subject: [petsc-users] issue of MatCreateDense in the CUDA codes In-Reply-To: References: <4a0a535b-04cd-4989-af60-25179e7ad429@me.com> <83227ED1-668F-470C-8F4D-CE0B308505BF@dsic.upv.es> Message-ID: Hi Junchao, I pulled and reinstalled. Although the PetscScalar still refers to thrust::complex, based on your suggestion, cast the pointer to PetscScalar * can work. It has the same storage order as std::complex. Mat kernel ; auto * h_kernel = reinterpret_cast < PetscScalar *>( h_kernel_temp ) ; PetscCall (MatCreateDense( PETSC_COMM_WORLD , PETSC_DECIDE , PETSC_DECIDE , nRows , nCols , h_kernel , & kernel )) ; PetscCall (MatAssemblyBegin( kernel , MAT_FINAL_ASSEMBLY )) ; PetscCall (MatAssemblyEnd( kernel , MAT_FINAL_ASSEMBLY )) ; Thank you. Best wishes, -------------------- Langtian Liu On Oct 3, 2024, at 4:38 PM, Junchao Zhang wrote: The MR is merged to petsc/release. BTW, in MatCreateDense, the data pointer has to be a host pointer. It is better to always use PetscScalar* (instead of std::complex*) to do the cast. --Junchao Zhang On Thu, Oct 3, 2024 at 3:03 AM ??? < langtian.liu at icloud.com > wrote: Okay. I see :D -------------------- Langtian Liu Institute for Theorectical Physics, Justus-Liebig-University Giessen Heinrich-Buff-Ring 16, 35392 Giessen Germany email: langtian.liu at icloud.com Tel: (+49)641 99 33342 On Oct 3, 2024, at 9:58 AM, Jose E. Roman < jroman at dsic.upv.es > wrote: You have to wait until the merge request has the label "Merged" instead of "Open". El 3 oct 2024, a las 9:55, ??? via petsc-users < petsc-users at mcs.anl.gov > escribi?: Hello Junchao, Okay. Thank you for helping find this bug. I pull the newest version of petsc today. It seems this error has not been fixed in the present release version. Maybe I should wait for some days. Best wishes, Langtian On Oct 3, 2024, at 12:12 AM, Junchao Zhang < junchao.zhang at gmail.com > wrote: Hi, Langtian, Thanks for the configure.log and I now see what's wrong. Since you compiled your code with nvcc, we mistakenly thought petsc was configured with cuda. It is fixed in https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/merge_requests/7909__;!!G_uCfscf7eWS!bCwFqQQgKIKkOMR4nXJ16cEjufaLSNw5qOU_Cjwd-NEbx1k6ulpqcOvdMotHBhwSz573y-hBlUo-BcdxxyDtfYwpg9is$ , which will be in petsc/release and main. Thanks. --Junchao Zhang On Wed, Oct 2, 2024 at 3:11 PM ??? < langtian.liu at icloud.com > wrote: Hi Junchao, I check it, I haven't use cuda when install pure cpu version of petsc. The configure.log has been attached. Thank you for your reply. Best wishes, -------------------- Langtian Liu Institute for Theorectical Physics, Justus-Liebig-University Giessen Heinrich-Buff-Ring 16, 35392 Giessen Germany email: langtian.liu at icloud.com Tel: (+49)641 99 33342 On Oct 2, 2024, at 5:05 PM, Junchao Zhang < junchao.zhang at gmail.com > wrote: On Wed, Oct 2, 2024 at 3:57 AM ??? via petsc-users < petsc-users at mcs.anl.gov > wrote: Hi all, I am using the PETSc and SLEPc to solve the Faddeev equation of baryons. I encounter a problem of function MatCreateDense when changing from CPU to CPU-GPU computations. At first, I write the codes in purely CPU computation in the following way and it works. ``` Eigen::MatrixXcd H_KER; Eigen::MatrixXcd G0; printf("\nCompute the propagator matrix.\n"); prop_matrix_nucleon_sc_av(Mn, pp_nodes, cos1_nodes); printf("\nCompute the propagator matrix done.\n"); printf("\nCompute the kernel matrix.\n"); bse_kernel_nucleon_sc_av(Mn, pp_nodes, pp_weights, cos1_nodes, cos1_weights); printf("\nCompute the kernel matrix done.\n"); printf("\nCompute the full kernel matrix by multiplying kernel and propagator matrix.\n"); MatrixXcd kernel_temp = H_KER * G0; printf("\nCompute the full kernel matrix done.\n"); // Solve the eigen system with SLEPc printf("\nSolve the eigen system in the rest frame.\n"); // Get the size of the Eigen matrix int nRows = (int) kernel_temp.rows(); int nCols = (int) kernel_temp.cols(); // Create PETSc matrix and share the data of kernel_temp Mat kernel; PetscCall(MatCreateDense(PETSC_COMM_WORLD, PETSC_DECIDE, PETSC_DECIDE, nRows, nCols, kernel_temp.data(), &kernel)); PetscCall(MatAssemblyBegin(kernel, MAT_FINAL_ASSEMBLY)); PetscCall(MatAssemblyEnd(kernel, MAT_FINAL_ASSEMBLY)); ``` Now I change to compute the propagator and kernel matrices in GPU and then compute the largest eigenvalues in CPU using SLEPc in the ways below. ``` cuDoubleComplex *h_propmat; cuDoubleComplex *h_kernelmat; int dim = EIGHT * NP * NZ; printf("\nCompute the propagator matrix.\n"); prop_matrix_nucleon_sc_av_cuda(Mn, pp_nodes.data(), cos1_nodes.data()); printf("\nCompute the propagator matrix done.\n"); printf("\nCompute the kernel matrix.\n"); kernel_matrix_nucleon_sc_av_cuda(Mn, pp_nodes.data(), pp_weights.data(), cos1_nodes.data(), cos1_weights.data()); printf("\nCompute the kernel matrix done.\n"); printf("\nCompute the full kernel matrix by multiplying kernel and propagator matrix.\n"); // Map the raw arrays to Eigen matrices (column-major order) auto *h_kernel_temp = new cuDoubleComplex [dim*dim]; matmul_cublas_cuDoubleComplex(h_kernelmat,h_propmat,h_kernel_temp,dim,dim,dim); printf("\nCompute the full kernel matrix done.\n"); // Solve the eigen system with SLEPc printf("\nSolve the eigen system in the rest frame.\n"); int nRows = dim; int nCols = dim; // Create PETSc matrix and share the data of kernel_temp Mat kernel; auto* h_kernel = (std::complex*)(h_kernel_temp); PetscCall(MatCreateDense(PETSC_COMM_WORLD, PETSC_DECIDE, PETSC_DECIDE, nRows, nCols, h_kernel_temp, &kernel)); PetscCall(MatAssemblyBegin(kernel, MAT_FINAL_ASSEMBLY)); PetscCall(MatAssemblyEnd(kernel, MAT_FINAL_ASSEMBLY)); But in this case, the compiler told me that the MatCreateDense function uses the data pointer as type of "thrust::complex" instead of "std::complex". I am sure I only configured and install PETSc in purely CPU without GPU and this codes are written in the host function. Please double check that your PETSc was purely CPU configured. You can find it at the end of your configure.log to see if petsc is configured with CUDA. Since thrust::complex is a result of a petsc/cuda configuration, I have this doubt. Why the function changes its behavior? Did you also meet this problem when writing the cuda codes and how to solve this problem. I tried to copy the data to a new thrust::complex matrix but this is very time consuming since my matrix is very big. Is there a way to create the Mat from the original data without changing the data type to thrust::complex in the cuda applications? Any response will be appreciated. Thank you! Best wishes, Langtian Liu ------ Institute for Theorectical Physics, Justus-Liebig-University Giessen Heinrich-Buff-Ring 16, 35392 Giessen Germany -------------- next part -------------- An HTML attachment was scrubbed... URL: From langtian.liu at icloud.com Thu Oct 3 10:49:02 2024 From: langtian.liu at icloud.com (=?utf-8?B?5YiY5rWq5aSp?=) Date: Thu, 3 Oct 2024 15:49:02 +0000 (UTC) Subject: [petsc-users] issue of MatCreateDense in the CUDA codes In-Reply-To: References: <4a0a535b-04cd-4989-af60-25179e7ad429@me.com> <83227ED1-668F-470C-8F4D-CE0B308505BF@dsic.upv.es> Message-ID: "How did you compile your code, using nvcc, mpicc or mpicxx? I expect PetscScalar to be std::complex with C++ or _Complex with C. So I don't understand why "PetscScalar still refers to thrust::complex"." I use mpicxx and nvcc. I also attached my cmakelists.txt Best, -------------------- Langtian Liu On Oct 3, 2024, at 5:30 PM, ??? via petsc-users wrote: Hi Junchao, I pulled and reinstalled. Although the PetscScalar still refers to thrust::complex, based on your suggestion, cast the pointer to PetscScalar * can work. It has the same storage order as std::complex. Mat kernel ; auto * h_kernel = reinterpret_cast < PetscScalar *>( h_kernel_temp ) ; PetscCall (MatCreateDense( PETSC_COMM_WORLD , PETSC_DECIDE , PETSC_DECIDE , nRows , nCols , h_kernel , & kernel )) ; PetscCall (MatAssemblyBegin( kernel , MAT_FINAL_ASSEMBLY )) ; PetscCall (MatAssemblyEnd( kernel , MAT_FINAL_ASSEMBLY )) ; Thank you. Best wishes, -------------------- Langtian Liu On Oct 3, 2024, at 4:38 PM, Junchao Zhang wrote: The MR is merged to petsc/release. BTW, in MatCreateDense, the data pointer has to be a host pointer. It is better to always use PetscScalar* (instead of std::complex*) to do the cast. --Junchao Zhang On Thu, Oct 3, 2024 at 3:03 AM ??? < langtian.liu at icloud.com > wrote: Okay. I see :D -------------------- Langtian Liu Institute for Theorectical Physics, Justus-Liebig-University Giessen Heinrich-Buff-Ring 16, 35392 Giessen Germany email: langtian.liu at icloud.com Tel: (+49)641 99 33342 On Oct 3, 2024, at 9:58 AM, Jose E. Roman < jroman at dsic.upv.es > wrote: You have to wait until the merge request has the label "Merged" instead of "Open". El 3 oct 2024, a las 9:55, ??? via petsc-users < petsc-users at mcs.anl.gov > escribi?: Hello Junchao, Okay. Thank you for helping find this bug. I pull the newest version of petsc today. It seems this error has not been fixed in the present release version. Maybe I should wait for some days. Best wishes, Langtian On Oct 3, 2024, at 12:12 AM, Junchao Zhang < junchao.zhang at gmail.com > wrote: Hi, Langtian, Thanks for the configure.log and I now see what's wrong. Since you compiled your code with nvcc, we mistakenly thought petsc was configured with cuda. It is fixed in https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/merge_requests/7909__;!!G_uCfscf7eWS!dSgKWIaV5BY-uv8ISNa3eXsKMNM05aYYIUvUH_itJjdOJuCaKezoVd0x6fSAoxpkx0olJmjdn_tyXDUSRqYXAKcTW7eN$ , which will be in petsc/release and main. Thanks. --Junchao Zhang On Wed, Oct 2, 2024 at 3:11 PM ??? < langtian.liu at icloud.com > wrote: Hi Junchao, I check it, I haven't use cuda when install pure cpu version of petsc. The configure.log has been attached. Thank you for your reply. Best wishes, -------------------- Langtian Liu Institute for Theorectical Physics, Justus-Liebig-University Giessen Heinrich-Buff-Ring 16, 35392 Giessen Germany email: langtian.liu at icloud.com Tel: (+49)641 99 33342 On Oct 2, 2024, at 5:05 PM, Junchao Zhang < junchao.zhang at gmail.com > wrote: On Wed, Oct 2, 2024 at 3:57 AM ??? via petsc-users < petsc-users at mcs.anl.gov > wrote: Hi all, I am using the PETSc and SLEPc to solve the Faddeev equation of baryons. I encounter a problem of function MatCreateDense when changing from CPU to CPU-GPU computations. At first, I write the codes in purely CPU computation in the following way and it works. ``` Eigen::MatrixXcd H_KER; Eigen::MatrixXcd G0; printf("\nCompute the propagator matrix.\n"); prop_matrix_nucleon_sc_av(Mn, pp_nodes, cos1_nodes); printf("\nCompute the propagator matrix done.\n"); printf("\nCompute the kernel matrix.\n"); bse_kernel_nucleon_sc_av(Mn, pp_nodes, pp_weights, cos1_nodes, cos1_weights); printf("\nCompute the kernel matrix done.\n"); printf("\nCompute the full kernel matrix by multiplying kernel and propagator matrix.\n"); MatrixXcd kernel_temp = H_KER * G0; printf("\nCompute the full kernel matrix done.\n"); // Solve the eigen system with SLEPc printf("\nSolve the eigen system in the rest frame.\n"); // Get the size of the Eigen matrix int nRows = (int) kernel_temp.rows(); int nCols = (int) kernel_temp.cols(); // Create PETSc matrix and share the data of kernel_temp Mat kernel; PetscCall(MatCreateDense(PETSC_COMM_WORLD, PETSC_DECIDE, PETSC_DECIDE, nRows, nCols, kernel_temp.data(), &kernel)); PetscCall(MatAssemblyBegin(kernel, MAT_FINAL_ASSEMBLY)); PetscCall(MatAssemblyEnd(kernel, MAT_FINAL_ASSEMBLY)); ``` Now I change to compute the propagator and kernel matrices in GPU and then compute the largest eigenvalues in CPU using SLEPc in the ways below. ``` cuDoubleComplex *h_propmat; cuDoubleComplex *h_kernelmat; int dim = EIGHT * NP * NZ; printf("\nCompute the propagator matrix.\n"); prop_matrix_nucleon_sc_av_cuda(Mn, pp_nodes.data(), cos1_nodes.data()); printf("\nCompute the propagator matrix done.\n"); printf("\nCompute the kernel matrix.\n"); kernel_matrix_nucleon_sc_av_cuda(Mn, pp_nodes.data(), pp_weights.data(), cos1_nodes.data(), cos1_weights.data()); printf("\nCompute the kernel matrix done.\n"); printf("\nCompute the full kernel matrix by multiplying kernel and propagator matrix.\n"); // Map the raw arrays to Eigen matrices (column-major order) auto *h_kernel_temp = new cuDoubleComplex [dim*dim]; matmul_cublas_cuDoubleComplex(h_kernelmat,h_propmat,h_kernel_temp,dim,dim,dim); printf("\nCompute the full kernel matrix done.\n"); // Solve the eigen system with SLEPc printf("\nSolve the eigen system in the rest frame.\n"); int nRows = dim; int nCols = dim; // Create PETSc matrix and share the data of kernel_temp Mat kernel; auto* h_kernel = (std::complex*)(h_kernel_temp); PetscCall(MatCreateDense(PETSC_COMM_WORLD, PETSC_DECIDE, PETSC_DECIDE, nRows, nCols, h_kernel_temp, &kernel)); PetscCall(MatAssemblyBegin(kernel, MAT_FINAL_ASSEMBLY)); PetscCall(MatAssemblyEnd(kernel, MAT_FINAL_ASSEMBLY)); But in this case, the compiler told me that the MatCreateDense function uses the data pointer as type of "thrust::complex" instead of "std::complex". I am sure I only configured and install PETSc in purely CPU without GPU and this codes are written in the host function. Please double check that your PETSc was purely CPU configured. You can find it at the end of your configure.log to see if petsc is configured with CUDA. Since thrust::complex is a result of a petsc/cuda configuration, I have this doubt. Why the function changes its behavior? Did you also meet this problem when writing the cuda codes and how to solve this problem. I tried to copy the data to a new thrust::complex matrix but this is very time consuming since my matrix is very big. Is there a way to create the Mat from the original data without changing the data type to thrust::complex in the cuda applications? Any response will be appreciated. Thank you! Best wishes, Langtian Liu ------ Institute for Theorectical Physics, Justus-Liebig-University Giessen Heinrich-Buff-Ring 16, 35392 Giessen Germany -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: CMakeLists.txt URL: From junchao.zhang at gmail.com Thu Oct 3 12:01:16 2024 From: junchao.zhang at gmail.com (Junchao Zhang) Date: Thu, 3 Oct 2024 12:01:16 -0500 Subject: [petsc-users] getinfo for Mat of type nest In-Reply-To: References: Message-ID: Could you also provide input files to run your code? --Junchao Zhang On Thu, Oct 3, 2024 at 5:14?AM Edoardo alinovi wrote: > Hi Barry, > > I am trying to port this preconditioner I found presented at the last > PETSc conference into may code: > > https://urldefense.us/v3/__https://github.com/ndjinga/SOLVERLAB/blob/master/CoreFlows/examples/C/SaddlePointPreconditioner/SaddlePointLinearSolver.c__;!!G_uCfscf7eWS!ctF0rf5w_mUMxCH-BnPk3_1r0krQf0FzTtFxPvh461NlCNc-9c1KtZclWZZcGRUWeAsZyQjcoMeRpbtYhtZ4fNBmUlYJ$ > > > So far so good, I am just translating C to Fortran, however when I try to > solve my linear system with: > > call KSPSetOperators(ksp, Ahat, Pmat, ierr) > call KSPSolve(ksp, bhat, xhat, ierr) > > with Ahat and Pmat being indeed two nest MatNest, created from an array of > Mat as: > > call MatCreateNest(PETSC_COMM_WORLD, 2, PETSC_NULL_IS, 2, PETSC_NULL_IS, > Mat_array, Ahat, ierr) > call MatCreateNest(PETSC_COMM_WORLD, 2, PETSC_NULL_IS, 2, PETSC_NULL_IS, > Mat_array, Pmat, ierr) > > > However I am getting this error on getinfo: > > from C to Fortran > > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: No support for this operation for this object type > [0]PETSC ERROR: No method getinfo for Mat of type nest > [0]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!ctF0rf5w_mUMxCH-BnPk3_1r0krQf0FzTtFxPvh461NlCNc-9c1KtZclWZZcGRUWeAsZyQjcoMeRpbtYhtZ4fG4Obqto$ > > for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.20.4, unknown > > How can I get rid of this issue? > > Many thanks! > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Thu Oct 3 12:05:02 2024 From: jed at jedbrown.org (Jed Brown) Date: Thu, 03 Oct 2024 11:05:02 -0600 Subject: [petsc-users] getinfo for Mat of type nest In-Reply-To: References: Message-ID: <87cykhw20x.fsf@jedbrown.org> Could you please show the rest of the stack trace? The code you cited there isn't checking return values (using PetscCall or otherwise) so it might be truncated. You can run in a debugger to see a full stack trace without modifying the code (though we recommend checking return values). MatGetInfo_Nest doesn't exist because it's unclear what the fields should mean (and people often use Nest with matrix-free types). However, it probably doesn't need to be called. Note that this preconditioner looks a lot like PCFIELDSPLIT, which is more feasible for us to help with. Edoardo alinovi writes: > Hi Barry, > > I am trying to port this preconditioner I found presented at the last PETSc > conference into may code: > https://urldefense.us/v3/__https://github.com/ndjinga/SOLVERLAB/blob/master/CoreFlows/examples/C/SaddlePointPreconditioner/SaddlePointLinearSolver.c__;!!G_uCfscf7eWS!a-JnrBsta75n_WDtxUNd7Xb8_m7tg1pWSxwWNTmUBSfpBwISq4IxJkbM6QgADd-x23Q_2s-d6d6bOWCKGRMxMzHyc9DDm7A$ > > So far so good, I am just translating C to Fortran, however when I try to > solve my linear system with: > > call KSPSetOperators(ksp, Ahat, Pmat, ierr) > call KSPSolve(ksp, bhat, xhat, ierr) > > with Ahat and Pmat being indeed two nest MatNest, created from an array of > Mat as: > > call MatCreateNest(PETSC_COMM_WORLD, 2, PETSC_NULL_IS, 2, PETSC_NULL_IS, > Mat_array, Ahat, ierr) > call MatCreateNest(PETSC_COMM_WORLD, 2, PETSC_NULL_IS, 2, PETSC_NULL_IS, > Mat_array, Pmat, ierr) > > > However I am getting this error on getinfo: > > from C to Fortran > > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: No support for this operation for this object type > [0]PETSC ERROR: No method getinfo for Mat of type nest > [0]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!a-JnrBsta75n_WDtxUNd7Xb8_m7tg1pWSxwWNTmUBSfpBwISq4IxJkbM6QgADd-x23Q_2s-d6d6bOWCKGRMxMzHyRUMnyvE$ for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.20.4, unknown > > How can I get rid of this issue? > > Many thanks! From edoardo.alinovi at gmail.com Fri Oct 4 01:50:09 2024 From: edoardo.alinovi at gmail.com (Edoardo alinovi) Date: Fri, 4 Oct 2024 08:50:09 +0200 Subject: [petsc-users] getinfo for Mat of type nest In-Reply-To: References: Message-ID: Jed, Junchao, Thanks for getting me back! Junchao, I cannot provide you with an input file as I am plugging just part of that code into mine. I am building blocks of the full matrix on my own, not reading from the file unfortunately. I can share a branch of my code with a 3x3 cavity toy case, but that would require compiling it and it might be a bit inconvenient for you. Jed, I am using fieldsplit indeed, it's just the matrix that is assembled in a 2x2 block form. Here is the full backtrace: *[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------* *[0]PETSC ERROR: No support for this operation for this object type* *[0]PETSC ERROR: No method getinfo for Mat of type nest* *[0]PETSC ERROR: WARNING! There are unused option(s) set! Could be the program crashed before usage or a spelling mistake, etc!* *[0]PETSC ERROR: Option left: name:-UPeqn_fieldsplit_p_pc_type value: hypre source: command line* *[0]PETSC ERROR: Option left: name:-UPeqn_fieldsplit_u_pc_type value: bjacobi source: command line* *[0]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!a6ejiT9ML4BpeAAcqVuIMX-J7sA75OiyYcHeUd6-INXK8sR-gcQPA3ndqsgpzphK4tIBnE13dbzycfmUkwST7aXF6oxXipY$ for trouble shooting.* *[0]PETSC ERROR: Petsc Release Version 3.20.4, unknown * *[0]PETSC ERROR: flubio_coupled on a linux_x86_64 named localhost.localdomain by edo Thu Oct 3 14:17:20 2024* *[0]PETSC ERROR: Configure options PETSC_ARCH=linux_x86_64 FOPTFLAGS=-O3 COPTFLAGS=-O3 CXXOPTFLAGS=-O3 -with-debugging=no -download-fblaslapack=1 -download-superlu_dist -download-mumps -download-hypre -download-metis -download-parmetis -download-scalapack -download-ml -download-slepc -download-hpddm -download-cmake -with-mpi-dir=/home/edo/software_repo/openmpi-5.0.1/build/* *[0]PETSC ERROR: #1 MatGetInfo() at /home/edo/software_repo/petsc/src/mat/interface/matrix.c:3005* *[0]PETSC ERROR: #2 PCSetUp_GAMG() at /home/edo/software_repo/petsc/src/ksp/pc/impls/gamg/gamg.c:619* *[0]PETSC ERROR: #3 PCSetUp() at /home/edo/software_repo/petsc/src/ksp/pc/interface/precon.c:1080* *[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------* *[0]PETSC ERROR: It appears a new error in the code was triggered after a previous error, possibly because:* *[0]PETSC ERROR: - The first error was not properly handled via (for example) the use of* *[0]PETSC ERROR: PetscCall(TheFunctionThatErrors()); or* *[0]PETSC ERROR: - The second error was triggered while handling the first error.* *[0]PETSC ERROR: Above is the traceback for the previous unhandled error, below the traceback for the next error* *[0]PETSC ERROR: ALL ERRORS in the PETSc libraries are fatal, you should add the appropriate error checking to the code* *[0]PETSC ERROR: No support for this operation for this object type* *[0]PETSC ERROR: No method getinfo for Mat of type nest* *[0]PETSC ERROR: WARNING! There are unused option(s) set! Could be the program crashed before usage or a spelling mistake, etc!* *[0]PETSC ERROR: Option left: name:-UPeqn_fieldsplit_p_pc_type value: hypre source: command line* *[0]PETSC ERROR: Option left: name:-UPeqn_fieldsplit_u_pc_type value: bjacobi source: command line* *[0]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!a6ejiT9ML4BpeAAcqVuIMX-J7sA75OiyYcHeUd6-INXK8sR-gcQPA3ndqsgpzphK4tIBnE13dbzycfmUkwST7aXF6oxXipY$ for trouble shooting.* *[0]PETSC ERROR: Petsc Release Version 3.20.4, unknown * *[0]PETSC ERROR: flubio_coupled on a linux_x86_64 named localhost.localdomain by edo Thu Oct 3 14:17:20 2024* *[0]PETSC ERROR: Configure options PETSC_ARCH=linux_x86_64 FOPTFLAGS=-O3 COPTFLAGS=-O3 CXXOPTFLAGS=-O3 -with-debugging=no -download-fblaslapack=1 -download-superlu_dist -download-mumps -download-hypre -download-metis -download-parmetis -download-scalapack -download-ml -download-slepc -download-hpddm -download-cmake -with-mpi-dir=/home/edo/software_repo/openmpi-5.0.1/build/* *[0]PETSC ERROR: #1 MatGetInfo() at /home/edo/software_repo/petsc/src/mat/interface/matrix.c:3005* *[0]PETSC ERROR: #2 PCSetUp_GAMG() at /home/edo/software_repo/petsc/src/ksp/pc/impls/gamg/gamg.c:619* *[0]PETSC ERROR: #3 PCSetUp() at /home/edo/software_repo/petsc/src/ksp/pc/interface/precon.c:1080* *[0]PETSC ERROR: #4 KSPSetUp() at /home/edo/software_repo/petsc/src/ksp/ksp/interface/itfunc.c:415* *[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------* *[0]PETSC ERROR: It appears a new error in the code was triggered after a previous error, possibly because:* *[0]PETSC ERROR: - The first error was not properly handled via (for example) the use of* *[0]PETSC ERROR: PetscCall(TheFunctionThatErrors()); or* *[0]PETSC ERROR: - The second error was triggered while handling the first error.* *[0]PETSC ERROR: Above is the traceback for the previous unhandled error, below the traceback for the next error* *[0]PETSC ERROR: ALL ERRORS in the PETSc libraries are fatal, you should add the appropriate error checking to the code* *[0]PETSC ERROR: No support for this operation for this object type* *[0]PETSC ERROR: No method getinfo for Mat of type nest* *[0]PETSC ERROR: WARNING! There are unused option(s) set! Could be the program crashed before usage or a spelling mistake, etc!* *[0]PETSC ERROR: Option left: name:-UPeqn_fieldsplit_p_pc_type value: hypre source: command line* *[0]PETSC ERROR: Option left: name:-UPeqn_fieldsplit_u_pc_type value: bjacobi source: command line* *[0]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!a6ejiT9ML4BpeAAcqVuIMX-J7sA75OiyYcHeUd6-INXK8sR-gcQPA3ndqsgpzphK4tIBnE13dbzycfmUkwST7aXF6oxXipY$ for trouble shooting.* *[0]PETSC ERROR: Petsc Release Version 3.20.4, unknown * *[0]PETSC ERROR: flubio_coupled on a linux_x86_64 named localhost.localdomain by edo Thu Oct 3 14:17:20 2024* *[0]PETSC ERROR: Configure options PETSC_ARCH=linux_x86_64 FOPTFLAGS=-O3 COPTFLAGS=-O3 CXXOPTFLAGS=-O3 -with-debugging=no -download-fblaslapack=1 -download-superlu_dist -download-mumps -download-hypre -download-metis -download-parmetis -download-scalapack -download-ml -download-slepc -download-hpddm -download-cmake -with-mpi-dir=/home/edo/software_repo/openmpi-5.0.1/build/* *[0]PETSC ERROR: #1 MatGetInfo() at /home/edo/software_repo/petsc/src/mat/interface/matrix.c:3005* *[0]PETSC ERROR: #2 PCSetUp_GAMG() at /home/edo/software_repo/petsc/src/ksp/pc/impls/gamg/gamg.c:619* *[0]PETSC ERROR: #3 PCSetUp() at /home/edo/software_repo/petsc/src/ksp/pc/interface/precon.c:1080* *[0]PETSC ERROR: #4 KSPSetUp() at /home/edo/software_repo/petsc/src/ksp/ksp/interface/itfunc.c:415* *[0]PETSC ERROR: #5 KSPSolve_Private() at /home/edo/software_repo/petsc/src/ksp/ksp/interface/itfunc.c:832* *[0]PETSC ERROR: #6 KSPSolve() at /home/edo/software_repo/petsc/src/ksp/ksp/interface/itfunc.c:1079* This the code I am doing right now: * M = this%setMomentumBlockMatrix() C = this%setPressureBlockMatrix() G = this%setPressureGradBlockMatrix() D = this%setDivBlockMatrix() bhat = this%setPressureMomentumRhs() !------------------! ! Create Ahat ! !-----------------! ! Bottom left block of Ahat Mat_array(4) = M ! Extraction of the diagonal of M call MatCreateVecs(M, PETSC_NULL_VEC, v, ierr) !v has the parallel distribution of M call MatGetDiagonal(M, v, ierr) call MatCreateDiagonal(v, diag_2M, ierr) call MatScale(diag_2M, 2.0, ierr) ! store 2*diagonal part of M call MatConvert(diag_2M, MATAIJ, MAT_INPLACE_MATRIX, diag_2M, ierr) call VecReciprocal(v, ierr) ! Creation of D_M_inv_G = D_M_inv*G call MatDuplicate(G,MAT_COPY_VALUES, D_M_inv_G, ierr) ! D_M_inv_G contains G call MatCreateVecs(D_M_inv_G, PETSC_NULL_VEC, v_redistributed, ierr) ! v_redistributed has the parallel distribution of D_M_inv_G call VecGetOwnershipRange(v, col_min, col_max, ierr) call ISCreateStride(PETSC_COMM_WORLD, col_max-col_min, col_min, 1, is_from, ierr) call VecGetOwnershipRange(v_redistributed, col_min, col_max, ierr) call ISCreateStride(PETSC_COMM_WORLD, col_max-col_min, col_min, 1, is_to, ierr) call VecScatterCreate(v,is_from,v_redistributed,is_to, scat, ierr) call VecScatterBegin(scat, v, v_redistributed,INSERT_VALUES,SCATTER_FORWARD, ierr) call VecScatterEnd( scat, v, v_redistributed,INSERT_VALUES,SCATTER_FORWARD, ierr) call MatDiagonalScale( D_M_inv_G, v_redistributed, PETSC_NULL_VEC, ierr) ! D_M_inv_G contains D_M_inv*G call ISDestroy(is_to, ierr) call ISDestroy(is_from, ierr) ! Creation of Chat (PETSC_DEFAULT_REAL -> PETSC_DETERMINE) call MatMatMult(D, D_M_inv_G, MAT_INITIAL_MATRIX, PETSC_DEFAULT_REAL, Chat, ierr) ! Chat contains D*D_M_inv*G call MatAXPY( Chat, 1.0, C, SUBSET_NONZERO_PATTERN, ierr) ! Chat contains C + D*D_M_inv*G Mat_array(1) = Chat ! Top left block of Ahat ! Creation of Ghat call MatMatMult(M, D_M_inv_G, MAT_INITIAL_MATRIX, PETSC_DEFAULT_REAL, Ghat, ierr) ! Ghat contains M*D_M_inv*G call MatAYPX( Ghat, -1.0, G, UNKNOWN_NONZERO_PATTERN, ierr) ! Ghat contains G - M*D_M_inv*G Mat_array(3) = Ghat ! Bottom left block of Ahat ! Creation of -D call MatScale(D,-1.0, ierr) Mat_array(2) = D ! Top right block of Ahat ! Creation of Ahat = transformed+ reordered A_input call MatCreateNest(PETSC_COMM_WORLD, 2, PETSC_NULL_IS, 2, PETSC_NULL_IS, Mat_array, Ahat, ierr) ! Creation of Pmat Mat_array(4) = diag_2M Mat_array(2) = PETSC_NULL_MAT ! Cancel top right block call MatCreateNest(PETSC_COMM_WORLD, 2, PETSC_NULL_IS, 2, PETSC_NULL_IS, Mat_array, Pmat, ierr); ! Finalisation of the preconditioner call ISCreateStride(PETSC_COMM_WORLD, numberOfElements, 0, 1, is_P_hat, ierr); call ISCreateStride(PETSC_COMM_WORLD, (3-bdim)*numberOfElements, numberOfElements, 1, is_U_hat, ierr); !----------------! ! KSP PART ! !----------------! call KSPSetOperators(this%ksp, Ahat, Pmat, ierr) call KSPGetPC(this%ksp, pc, ierr) call PCSetFromOptions(pc, ierr) call PCSetUp(pc, ierr) call KSPSetFromOptions(this%ksp, ierr) call KSPSetUp(this%ksp, ierr) call VecDuplicate(bhat, xhat, ierr) ! I think it makes sense to move this part where fieldsplit preconditioner is defined call PCFieldSplitSetIS(pc, "phat", is_P_hat, ierr) call PCFieldSplitSetIS(pc, "uhat", is_U_hat, ierr) call KSPSolve(this%ksp, bhat, xhat, ierr)* KSP id FGMRES + fieldsplit with default settings. Many thanks! Il giorno gio 3 ott 2024 alle ore 19:01 Junchao Zhang < junchao.zhang at gmail.com> ha scritto: > Could you also provide input files to run your code? > > --Junchao Zhang > > > On Thu, Oct 3, 2024 at 5:14?AM Edoardo alinovi > wrote: > >> Hi Barry, >> >> I am trying to port this preconditioner I found presented at the last >> PETSc conference into may code: >> >> https://urldefense.us/v3/__https://github.com/ndjinga/SOLVERLAB/blob/master/CoreFlows/examples/C/SaddlePointPreconditioner/SaddlePointLinearSolver.c__;!!G_uCfscf7eWS!a6ejiT9ML4BpeAAcqVuIMX-J7sA75OiyYcHeUd6-INXK8sR-gcQPA3ndqsgpzphK4tIBnE13dbzycfmUkwST7aXF2QaZ-jc$ >> >> >> So far so good, I am just translating C to Fortran, however when I try to >> solve my linear system with: >> >> call KSPSetOperators(ksp, Ahat, Pmat, ierr) >> call KSPSolve(ksp, bhat, xhat, ierr) >> >> with Ahat and Pmat being indeed two nest MatNest, created from an array >> of Mat as: >> >> call MatCreateNest(PETSC_COMM_WORLD, 2, PETSC_NULL_IS, 2, PETSC_NULL_IS, >> Mat_array, Ahat, ierr) >> call MatCreateNest(PETSC_COMM_WORLD, 2, PETSC_NULL_IS, 2, PETSC_NULL_IS, >> Mat_array, Pmat, ierr) >> >> >> However I am getting this error on getinfo: >> >> from C to Fortran >> >> [0]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> [0]PETSC ERROR: No support for this operation for this object type >> [0]PETSC ERROR: No method getinfo for Mat of type nest >> [0]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!a6ejiT9ML4BpeAAcqVuIMX-J7sA75OiyYcHeUd6-INXK8sR-gcQPA3ndqsgpzphK4tIBnE13dbzycfmUkwST7aXF6oxXipY$ >> >> for trouble shooting. >> [0]PETSC ERROR: Petsc Release Version 3.20.4, unknown >> >> How can I get rid of this issue? >> >> Many thanks! >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From pierre at joliv.et Fri Oct 4 02:02:00 2024 From: pierre at joliv.et (Pierre Jolivet) Date: Fri, 4 Oct 2024 09:02:00 +0200 Subject: [petsc-users] getinfo for Mat of type nest In-Reply-To: References: Message-ID: <12A77563-E267-45B3-B087-2790112DFFF4@joliv.et> > On 4 Oct 2024, at 8:50?AM, Edoardo alinovi wrote: > > Jed, Junchao, > > Thanks for getting me back! > > Junchao, I cannot provide you with an input file as I am plugging just part of that code into mine. I am building blocks of the full matrix on my own, not reading from the file unfortunately. I can share a branch of my code with a 3x3 cavity toy case, but that would require compiling it and it might be a bit inconvenient for you. > > Jed, I am using fieldsplit indeed, it's just the matrix that is assembled in a 2x2 block form. Here is the full backtrace: > > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [0]PETSC ERROR: No support for this operation for this object type > [0]PETSC ERROR: No method getinfo for Mat of type nest > [0]PETSC ERROR: WARNING! There are unused option(s) set! Could be the program crashed before usage or a spelling mistake, etc! > [0]PETSC ERROR: Option left: name:-UPeqn_fieldsplit_p_pc_type value: hypre source: command line > [0]PETSC ERROR: Option left: name:-UPeqn_fieldsplit_u_pc_type value: bjacobi source: command line > [0]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!firbtqTDrSSyq61w7-MZvVgFlq4mbFVINLG5rFdWel7q49yeL78quSjJJ_-BZVfe7WAvCMwoKHfM8YSs9bCJ1w$ for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.20.4, unknown > [0]PETSC ERROR: flubio_coupled on a linux_x86_64 named localhost.localdomain by edo Thu Oct 3 14:17:20 2024 > [0]PETSC ERROR: Configure options PETSC_ARCH=linux_x86_64 FOPTFLAGS=-O3 COPTFLAGS=-O3 CXXOPTFLAGS=-O3 -with-debugging=no -download-fblaslapack=1 -download-superlu_dist -download-mumps -download-hypre -download-metis -download-parmetis -download-scalapack -download-ml -download-slepc -download-hpddm -download-cmake -with-mpi-dir=/home/edo/software_repo/openmpi-5.0.1/build/ > [0]PETSC ERROR: #1 MatGetInfo() at /home/edo/software_repo/petsc/src/mat/interface/matrix.c:3005 > [0]PETSC ERROR: #2 PCSetUp_GAMG() at /home/edo/software_repo/petsc/src/ksp/pc/impls/gamg/gamg.c:619 > [0]PETSC ERROR: #3 PCSetUp() at /home/edo/software_repo/petsc/src/ksp/pc/interface/precon.c:1080 > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [0]PETSC ERROR: It appears a new error in the code was triggered after a previous error, possibly because: > [0]PETSC ERROR: - The first error was not properly handled via (for example) the use of > [0]PETSC ERROR: PetscCall(TheFunctionThatErrors()); or > [0]PETSC ERROR: - The second error was triggered while handling the first error. > [0]PETSC ERROR: Above is the traceback for the previous unhandled error, below the traceback for the next error > [0]PETSC ERROR: ALL ERRORS in the PETSc libraries are fatal, you should add the appropriate error checking to the code > [0]PETSC ERROR: No support for this operation for this object type > [0]PETSC ERROR: No method getinfo for Mat of type nest > [0]PETSC ERROR: WARNING! There are unused option(s) set! Could be the program crashed before usage or a spelling mistake, etc! > [0]PETSC ERROR: Option left: name:-UPeqn_fieldsplit_p_pc_type value: hypre source: command line > [0]PETSC ERROR: Option left: name:-UPeqn_fieldsplit_u_pc_type value: bjacobi source: command line > [0]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!firbtqTDrSSyq61w7-MZvVgFlq4mbFVINLG5rFdWel7q49yeL78quSjJJ_-BZVfe7WAvCMwoKHfM8YSs9bCJ1w$ for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.20.4, unknown > [0]PETSC ERROR: flubio_coupled on a linux_x86_64 named localhost.localdomain by edo Thu Oct 3 14:17:20 2024 > [0]PETSC ERROR: Configure options PETSC_ARCH=linux_x86_64 FOPTFLAGS=-O3 COPTFLAGS=-O3 CXXOPTFLAGS=-O3 -with-debugging=no -download-fblaslapack=1 -download-superlu_dist -download-mumps -download-hypre -download-metis -download-parmetis -download-scalapack -download-ml -download-slepc -download-hpddm -download-cmake -with-mpi-dir=/home/edo/software_repo/openmpi-5.0.1/build/ > [0]PETSC ERROR: #1 MatGetInfo() at /home/edo/software_repo/petsc/src/mat/interface/matrix.c:3005 > [0]PETSC ERROR: #2 PCSetUp_GAMG() at /home/edo/software_repo/petsc/src/ksp/pc/impls/gamg/gamg.c:619 > [0]PETSC ERROR: #3 PCSetUp() at /home/edo/software_repo/petsc/src/ksp/pc/interface/precon.c:1080 > [0]PETSC ERROR: #4 KSPSetUp() at /home/edo/software_repo/petsc/src/ksp/ksp/interface/itfunc.c:415 > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [0]PETSC ERROR: It appears a new error in the code was triggered after a previous error, possibly because: > [0]PETSC ERROR: - The first error was not properly handled via (for example) the use of > [0]PETSC ERROR: PetscCall(TheFunctionThatErrors()); or > [0]PETSC ERROR: - The second error was triggered while handling the first error. > [0]PETSC ERROR: Above is the traceback for the previous unhandled error, below the traceback for the next error > [0]PETSC ERROR: ALL ERRORS in the PETSc libraries are fatal, you should add the appropriate error checking to the code > [0]PETSC ERROR: No support for this operation for this object type > [0]PETSC ERROR: No method getinfo for Mat of type nest > [0]PETSC ERROR: WARNING! There are unused option(s) set! Could be the program crashed before usage or a spelling mistake, etc! > [0]PETSC ERROR: Option left: name:-UPeqn_fieldsplit_p_pc_type value: hypre source: command line > [0]PETSC ERROR: Option left: name:-UPeqn_fieldsplit_u_pc_type value: bjacobi source: command line > [0]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!firbtqTDrSSyq61w7-MZvVgFlq4mbFVINLG5rFdWel7q49yeL78quSjJJ_-BZVfe7WAvCMwoKHfM8YSs9bCJ1w$ for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.20.4, unknown > [0]PETSC ERROR: flubio_coupled on a linux_x86_64 named localhost.localdomain by edo Thu Oct 3 14:17:20 2024 > [0]PETSC ERROR: Configure options PETSC_ARCH=linux_x86_64 FOPTFLAGS=-O3 COPTFLAGS=-O3 CXXOPTFLAGS=-O3 -with-debugging=no -download-fblaslapack=1 -download-superlu_dist -download-mumps -download-hypre -download-metis -download-parmetis -download-scalapack -download-ml -download-slepc -download-hpddm -download-cmake -with-mpi-dir=/home/edo/software_repo/openmpi-5.0.1/build/ > [0]PETSC ERROR: #1 MatGetInfo() at /home/edo/software_repo/petsc/src/mat/interface/matrix.c:3005 > [0]PETSC ERROR: #2 PCSetUp_GAMG() at /home/edo/software_repo/petsc/src/ksp/pc/impls/gamg/gamg.c:619 > [0]PETSC ERROR: #3 PCSetUp() at /home/edo/software_repo/petsc/src/ksp/pc/interface/precon.c:1080 > [0]PETSC ERROR: #4 KSPSetUp() at /home/edo/software_repo/petsc/src/ksp/ksp/interface/itfunc.c:415 > [0]PETSC ERROR: #5 KSPSolve_Private() at /home/edo/software_repo/petsc/src/ksp/ksp/interface/itfunc.c:832 > [0]PETSC ERROR: #6 KSPSolve() at /home/edo/software_repo/petsc/src/ksp/ksp/interface/itfunc.c:1079 > > This the code I am doing right now: > > M = this%setMomentumBlockMatrix() > C = this%setPressureBlockMatrix() > G = this%setPressureGradBlockMatrix() > D = this%setDivBlockMatrix() > bhat = this%setPressureMomentumRhs() > > !------------------! > ! Create Ahat ! > !-----------------! > > ! Bottom left block of Ahat > Mat_array(4) = M > > ! Extraction of the diagonal of M > call MatCreateVecs(M, PETSC_NULL_VEC, v, ierr) !v has the parallel distribution of M > call MatGetDiagonal(M, v, ierr) > > call MatCreateDiagonal(v, diag_2M, ierr) > call MatScale(diag_2M, 2.0, ierr) ! store 2*diagonal part of M > call MatConvert(diag_2M, MATAIJ, MAT_INPLACE_MATRIX, diag_2M, ierr) > call VecReciprocal(v, ierr) > > ! Creation of D_M_inv_G = D_M_inv*G > call MatDuplicate(G,MAT_COPY_VALUES, D_M_inv_G, ierr) ! D_M_inv_G contains G > call MatCreateVecs(D_M_inv_G, PETSC_NULL_VEC, v_redistributed, ierr) ! v_redistributed has the parallel distribution of D_M_inv_G > > call VecGetOwnershipRange(v, col_min, col_max, ierr) > call ISCreateStride(PETSC_COMM_WORLD, col_max-col_min, col_min, 1, is_from, ierr) > call VecGetOwnershipRange(v_redistributed, col_min, col_max, ierr) > call ISCreateStride(PETSC_COMM_WORLD, col_max-col_min, col_min, 1, is_to, ierr) > > call VecScatterCreate(v,is_from,v_redistributed,is_to, scat, ierr) > call VecScatterBegin(scat, v, v_redistributed,INSERT_VALUES,SCATTER_FORWARD, ierr) > call VecScatterEnd( scat, v, v_redistributed,INSERT_VALUES,SCATTER_FORWARD, ierr) > call MatDiagonalScale( D_M_inv_G, v_redistributed, PETSC_NULL_VEC, ierr) ! D_M_inv_G contains D_M_inv*G > > call ISDestroy(is_to, ierr) > call ISDestroy(is_from, ierr) > > ! Creation of Chat (PETSC_DEFAULT_REAL -> PETSC_DETERMINE) > call MatMatMult(D, D_M_inv_G, MAT_INITIAL_MATRIX, PETSC_DEFAULT_REAL, Chat, ierr) ! Chat contains D*D_M_inv*G > call MatAXPY( Chat, 1.0, C, SUBSET_NONZERO_PATTERN, ierr) ! Chat contains C + D*D_M_inv*G > Mat_array(1) = Chat ! Top left block of Ahat > > ! Creation of Ghat > call MatMatMult(M, D_M_inv_G, MAT_INITIAL_MATRIX, PETSC_DEFAULT_REAL, Ghat, ierr) ! Ghat contains M*D_M_inv*G > call MatAYPX( Ghat, -1.0, G, UNKNOWN_NONZERO_PATTERN, ierr) ! Ghat contains G - M*D_M_inv*G > Mat_array(3) = Ghat ! Bottom left block of Ahat > > ! Creation of -D > call MatScale(D,-1.0, ierr) > Mat_array(2) = D ! Top right block of Ahat > > ! Creation of Ahat = transformed+ reordered A_input > call MatCreateNest(PETSC_COMM_WORLD, 2, PETSC_NULL_IS, 2, PETSC_NULL_IS, Mat_array, Ahat, ierr) > > ! Creation of Pmat > Mat_array(4) = diag_2M > Mat_array(2) = PETSC_NULL_MAT ! Cancel top right block > call MatCreateNest(PETSC_COMM_WORLD, 2, PETSC_NULL_IS, 2, PETSC_NULL_IS, Mat_array, Pmat, ierr); > > ! Finalisation of the preconditioner > call ISCreateStride(PETSC_COMM_WORLD, numberOfElements, 0, 1, is_P_hat, ierr); > call ISCreateStride(PETSC_COMM_WORLD, (3-bdim)*numberOfElements, numberOfElements, 1, is_U_hat, ierr); > > !----------------! > ! KSP PART ! > !----------------! > > call KSPSetOperators(this%ksp, Ahat, Pmat, ierr) > call KSPGetPC(this%ksp, pc, ierr) > > call PCSetFromOptions(pc, ierr) > call PCSetUp(pc, ierr) > call KSPSetFromOptions(this%ksp, ierr) > call KSPSetUp(this%ksp, ierr) > > call VecDuplicate(bhat, xhat, ierr) > > ! I think it makes sense to move this part where fieldsplit preconditioner is defined > call PCFieldSplitSetIS(pc, "phat", is_P_hat, ierr) > call PCFieldSplitSetIS(pc, "uhat", is_U_hat, ierr) > > call KSPSolve(this%ksp, bhat, xhat, ierr) > > KSP id FGMRES + fieldsplit with default settings. Apparently not, otherwise GAMG would not be called. In particular, it looks like the code is currently trying to call GAMG on the global outer MatNest, whereas you likely want to call it on just one of the inner block. Please send the complete code, complete options file, and complete command line arguments you are using, because with just the snippet you sent and the stack trace, the code is indeed not expected to work, e.g., I see no occurence of -pc_type fieldsplit or PCSetType(pc, PCFIELDSPLIT). Thanks, Pierre > Many thanks! > > Il giorno gio 3 ott 2024 alle ore 19:01 Junchao Zhang > ha scritto: >> Could you also provide input files to run your code? >> >> --Junchao Zhang >> >> >> On Thu, Oct 3, 2024 at 5:14?AM Edoardo alinovi > wrote: >>> Hi Barry, >>> >>> I am trying to port this preconditioner I found presented at the last PETSc conference into may code: >>> https://urldefense.us/v3/__https://github.com/ndjinga/SOLVERLAB/blob/master/CoreFlows/examples/C/SaddlePointPreconditioner/SaddlePointLinearSolver.c__;!!G_uCfscf7eWS!firbtqTDrSSyq61w7-MZvVgFlq4mbFVINLG5rFdWel7q49yeL78quSjJJ_-BZVfe7WAvCMwoKHfM8YS_aI5DUg$ >>> >>> So far so good, I am just translating C to Fortran, however when I try to solve my linear system with: >>> >>> call KSPSetOperators(ksp, Ahat, Pmat, ierr) >>> call KSPSolve(ksp, bhat, xhat, ierr) >>> >>> with Ahat and Pmat being indeed two nest MatNest, created from an array of Mat as: >>> >>> call MatCreateNest(PETSC_COMM_WORLD, 2, PETSC_NULL_IS, 2, PETSC_NULL_IS, Mat_array, Ahat, ierr) >>> call MatCreateNest(PETSC_COMM_WORLD, 2, PETSC_NULL_IS, 2, PETSC_NULL_IS, Mat_array, Pmat, ierr) >>> >>> However I am getting this error on getinfo: >>> >>> from C to Fortran >>> >>> [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >>> [0]PETSC ERROR: No support for this operation for this object type >>> [0]PETSC ERROR: No method getinfo for Mat of type nest >>> [0]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!firbtqTDrSSyq61w7-MZvVgFlq4mbFVINLG5rFdWel7q49yeL78quSjJJ_-BZVfe7WAvCMwoKHfM8YSs9bCJ1w$ for trouble shooting. >>> [0]PETSC ERROR: Petsc Release Version 3.20.4, unknown >>> >>> How can I get rid of this issue? >>> >>> Many thanks! >>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From edoardo.alinovi at gmail.com Fri Oct 4 03:52:08 2024 From: edoardo.alinovi at gmail.com (Edoardo alinovi) Date: Fri, 4 Oct 2024 10:52:08 +0200 Subject: [petsc-users] getinfo for Mat of type nest In-Reply-To: <12A77563-E267-45B3-B087-2790112DFFF4@joliv.et> References: <12A77563-E267-45B3-B087-2790112DFFF4@joliv.et> Message-ID: Hello Pierre, you are right , it looks like my fieldspit settings are missing. They are not really in the code I posted, but they are set in a class during setup phase, not at solve level. Good catch! For the time being I'll try to set up KSP in the same part and get you back when done! -------------- next part -------------- An HTML attachment was scrubbed... URL: From yangzongze at gmail.com Mon Oct 7 09:45:53 2024 From: yangzongze at gmail.com (Zongze Yang) Date: Mon, 7 Oct 2024 22:45:53 +0800 Subject: [petsc-users] ParMmg crashes when run in parallel: Assertion failed: (isfinite(dd)), function PMMG_hashNorver_normals, file analys_pmmg.c, line 1072 Message-ID: <6FFBDE5A-E87D-48A6-AAC9-6F4994345D9E@gmail.com> Hi PETSc team, I noticed that the ParMmg interface for overlap (https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/merge_requests/7593__;!!G_uCfscf7eWS!bh5eQHxA8QXn10TtTUOI5lmjFbfxFBtZz9CQoiYgyZSR7oDMYp8NK2HHP2UaWRXmU82ddDBpMx3ctAfX8GFQAuL_$ ) has been fixed. I tried using the updated version in my code, but it did not perform as expected when refining the attached mesh (or fine mesh of the same geometry) in parallel. Here are the details output: The first two sequential runs work correctly. However, when running the code in parallel without refinement, some warnings are generated. After refining the mesh in parallel, the code crashes. ``` ? mpiexec -n 1 python test_adapt.py -dm_adaptor parmmg ? mpiexec -n 1 python test_adapt.py -dm_adaptor parmmg -refine 1 ? mpiexec -n 2 python test_adapt.py -dm_adaptor parmmg ## Warning: MMG5_interpreg_ani: 608: unexpected case (null normal), impossible interpolation. ## Warning: MMG5_movbdyregpt_ani: unable to compute optimal position for at least 1 point. ## Warning: MMG5_interpreg_ani: 608: unexpected case (null normal), impossible interpolation. ## Warning: MMG5_movbdyregpt_ani: unable to compute optimal position for at least 1 point. ? mpiexec -n 2 python test_adapt.py -dm_adaptor parmmg -refine 1 Assertion failed: (isfinite(dd)), function PMMG_hashNorver_normals, file analys_pmmg.c, line 1072. Assertion failed: (isfinite(dd)), function PMMG_hashNorver_normals, file analys_pmmg.c, line 1072. =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = PID 22704 RUNNING AT yzzs-mac.local = EXIT CODE: 6 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES =================================================================================== YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Abort trap: 6 (signal 6) This typically refers to a problem with your application. Please see the FAQ page for debugging suggestions ? mpiexec -n 2 python test_adapt.py -dm_adaptor parmmg -overlap ## Warning: MMG5_interpreg_ani: 608: unexpected case (null normal), impossible interpolation. ## Warning: MMG5_movbdyregpt_ani: unable to compute optimal position for at least 1 point. ## Warning: MMG5_interpreg_ani: 608: unexpected case (null normal), impossible interpolation. ## Warning: MMG5_movbdyregpt_ani: unable to compute optimal position for at least 1 point. ? mpiexec -n 2 python test_adapt.py -dm_adaptor parmmg -overlap -refine Assertion failed: (isfinite(dd)), function PMMG_hashNorver_normals, file analys_pmmg.c, line 1072. Assertion failed: (isfinite(dd)), function PMMG_hashNorver_normals, file analys_pmmg.c, line 1072. =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = PID 25424 RUNNING AT yzzs-mac.local = EXIT CODE: 6 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES =================================================================================== YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Abort trap: 6 (signal 6) This typically refers to a problem with your application. Please see the FAQ page for debugging suggestions ``` The mesh geometry is a cylinder with an interface. The geometry file of gmsh, the mesh file, and the code are attached. Could someone take a look and help identify the issue? Thank you! Best regards, Zongze -------------- next part -------------- A non-text attachment was scrubbed... Name: cylinder_test.msh Type: application/octet-stream Size: 70782 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: test_adapt.py Type: text/x-python-script Size: 1388 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: geometry.png Type: image/png Size: 81097 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: cylinder.geo Type: application/octet-stream Size: 714 bytes Desc: not available URL: -------------- next part -------------- From pierre at joliv.et Mon Oct 7 10:34:05 2024 From: pierre at joliv.et (Pierre Jolivet) Date: Mon, 7 Oct 2024 17:34:05 +0200 Subject: [petsc-users] ParMmg crashes when run in parallel: Assertion failed: (isfinite(dd)), function PMMG_hashNorver_normals, file analys_pmmg.c, line 1072 In-Reply-To: <6FFBDE5A-E87D-48A6-AAC9-6F4994345D9E@gmail.com> References: <6FFBDE5A-E87D-48A6-AAC9-6F4994345D9E@gmail.com> Message-ID: Your mesh is tiny, you should not use ParMmg but plain Mmg instead which is much more robust. I don?t know how easy it is to do with DMPlex though (gather a DM and a metric on a single process), especially from the command line. Thanks, Pierre > On 7 Oct 2024, at 4:46?PM, Zongze Yang wrote: > > ?Hi PETSc team, > > I noticed that the ParMmg interface for overlap (https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/merge_requests/7593__;!!G_uCfscf7eWS!bh5eQHxA8QXn10TtTUOI5lmjFbfxFBtZz9CQoiYgyZSR7oDMYp8NK2HHP2UaWRXmU82ddDBpMx3ctAfX8GFQAuL_$ ) has been fixed. I tried using the updated version in my code, but it did not perform as expected when refining the attached mesh (or fine mesh of the same geometry) in parallel. > > Here are the details output: The first two sequential runs work correctly. However, when running the code in parallel without refinement, some warnings are generated. After refining the mesh in parallel, the code crashes. > > ``` > ? mpiexec -n 1 python test_adapt.py -dm_adaptor parmmg > > ? mpiexec -n 1 python test_adapt.py -dm_adaptor parmmg -refine 1 > > ? mpiexec -n 2 python test_adapt.py -dm_adaptor parmmg > ## Warning: MMG5_interpreg_ani: 608: unexpected case (null normal), impossible interpolation. > > ## Warning: MMG5_movbdyregpt_ani: unable to compute optimal position for at least 1 point. > ## Warning: MMG5_interpreg_ani: 608: unexpected case (null normal), impossible interpolation. > > ## Warning: MMG5_movbdyregpt_ani: unable to compute optimal position for at least 1 point. > > ? mpiexec -n 2 python test_adapt.py -dm_adaptor parmmg -refine 1 > Assertion failed: (isfinite(dd)), function PMMG_hashNorver_normals, file analys_pmmg.c, line 1072. > Assertion failed: (isfinite(dd)), function PMMG_hashNorver_normals, file analys_pmmg.c, line 1072. > > =================================================================================== > = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES > = PID 22704 RUNNING AT yzzs-mac.local > = EXIT CODE: 6 > = CLEANING UP REMAINING PROCESSES > = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES > =================================================================================== > YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Abort trap: 6 (signal 6) > This typically refers to a problem with your application. > Please see the FAQ page for debugging suggestions > > > ? mpiexec -n 2 python test_adapt.py -dm_adaptor parmmg -overlap > ## Warning: MMG5_interpreg_ani: 608: unexpected case (null normal), impossible interpolation. > > ## Warning: MMG5_movbdyregpt_ani: unable to compute optimal position for at least 1 point. > ## Warning: MMG5_interpreg_ani: 608: unexpected case (null normal), impossible interpolation. > > ## Warning: MMG5_movbdyregpt_ani: unable to compute optimal position for at least 1 point. > > > ? mpiexec -n 2 python test_adapt.py -dm_adaptor parmmg -overlap -refine > Assertion failed: (isfinite(dd)), function PMMG_hashNorver_normals, file analys_pmmg.c, line 1072. > Assertion failed: (isfinite(dd)), function PMMG_hashNorver_normals, file analys_pmmg.c, line 1072. > > =================================================================================== > = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES > = PID 25424 RUNNING AT yzzs-mac.local > = EXIT CODE: 6 > = CLEANING UP REMAINING PROCESSES > = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES > =================================================================================== > YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Abort trap: 6 (signal 6) > This typically refers to a problem with your application. > Please see the FAQ page for debugging suggestions > ``` > > The mesh geometry is a cylinder with an interface. The geometry file of gmsh, the mesh file, and the code are attached. Could someone take a look and help identify the issue? > > Thank you! > > Best regards, > Zongze > > > > > > > > From yangzongze at gmail.com Mon Oct 7 11:11:19 2024 From: yangzongze at gmail.com (Zongze Yang) Date: Tue, 8 Oct 2024 00:11:19 +0800 Subject: [petsc-users] ParMmg crashes when run in parallel: Assertion failed: (isfinite(dd)), function PMMG_hashNorver_normals, file analys_pmmg.c, line 1072 In-Reply-To: References: <6FFBDE5A-E87D-48A6-AAC9-6F4994345D9E@gmail.com> Message-ID: <7466C614-D2E4-408F-AF24-B5E83E883617@gmail.com> Hi Pierre, Thank you for the advice. I will look into implementing it as suggested. This is just a toy example; the mesh may be larger in the actual calculations. Best wishes, Zongze > On 7 Oct 2024, at 23:34, Pierre Jolivet wrote: > > Your mesh is tiny, you should not use ParMmg but plain Mmg instead which is much more robust. > I don?t know how easy it is to do with DMPlex though (gather a DM and a metric on a single process), especially from the command line. > > Thanks, > Pierre > >> On 7 Oct 2024, at 4:46?PM, Zongze Yang > wrote: >> >> ?Hi PETSc team, >> >> I noticed that the ParMmg interface for overlap (https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/merge_requests/7593__;!!G_uCfscf7eWS!bh5eQHxA8QXn10TtTUOI5lmjFbfxFBtZz9CQoiYgyZSR7oDMYp8NK2HHP2UaWRXmU82ddDBpMx3ctAfX8GFQAuL_$ ) has been fixed. I tried using the updated version in my code, but it did not perform as expected when refining the attached mesh (or fine mesh of the same geometry) in parallel. >> >> Here are the details output: The first two sequential runs work correctly. However, when running the code in parallel without refinement, some warnings are generated. After refining the mesh in parallel, the code crashes. >> >> ``` >> ? mpiexec -n 1 python test_adapt.py -dm_adaptor parmmg >> >> ? mpiexec -n 1 python test_adapt.py -dm_adaptor parmmg -refine 1 >> >> ? mpiexec -n 2 python test_adapt.py -dm_adaptor parmmg >> ## Warning: MMG5_interpreg_ani: 608: unexpected case (null normal), impossible interpolation. >> >> ## Warning: MMG5_movbdyregpt_ani: unable to compute optimal position for at least 1 point. >> ## Warning: MMG5_interpreg_ani: 608: unexpected case (null normal), impossible interpolation. >> >> ## Warning: MMG5_movbdyregpt_ani: unable to compute optimal position for at least 1 point. >> >> ? mpiexec -n 2 python test_adapt.py -dm_adaptor parmmg -refine 1 >> Assertion failed: (isfinite(dd)), function PMMG_hashNorver_normals, file analys_pmmg.c, line 1072. >> Assertion failed: (isfinite(dd)), function PMMG_hashNorver_normals, file analys_pmmg.c, line 1072. >> >> =================================================================================== >> = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES >> = PID 22704 RUNNING AT yzzs-mac.local >> = EXIT CODE: 6 >> = CLEANING UP REMAINING PROCESSES >> = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES >> =================================================================================== >> YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Abort trap: 6 (signal 6) >> This typically refers to a problem with your application. >> Please see the FAQ page for debugging suggestions >> >> >> ? mpiexec -n 2 python test_adapt.py -dm_adaptor parmmg -overlap >> ## Warning: MMG5_interpreg_ani: 608: unexpected case (null normal), impossible interpolation. >> >> ## Warning: MMG5_movbdyregpt_ani: unable to compute optimal position for at least 1 point. >> ## Warning: MMG5_interpreg_ani: 608: unexpected case (null normal), impossible interpolation. >> >> ## Warning: MMG5_movbdyregpt_ani: unable to compute optimal position for at least 1 point. >> >> >> ? mpiexec -n 2 python test_adapt.py -dm_adaptor parmmg -overlap -refine >> Assertion failed: (isfinite(dd)), function PMMG_hashNorver_normals, file analys_pmmg.c, line 1072. >> Assertion failed: (isfinite(dd)), function PMMG_hashNorver_normals, file analys_pmmg.c, line 1072. >> >> =================================================================================== >> = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES >> = PID 25424 RUNNING AT yzzs-mac.local >> = EXIT CODE: 6 >> = CLEANING UP REMAINING PROCESSES >> = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES >> =================================================================================== >> YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Abort trap: 6 (signal 6) >> This typically refers to a problem with your application. >> Please see the FAQ page for debugging suggestions >> ``` >> >> The mesh geometry is a cylinder with an interface. The geometry file of gmsh, the mesh file, and the code are attached. Could someone take a look and help identify the issue? >> >> Thank you! >> >> Best regards, >> Zongze >> >> >> >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From pierre at joliv.et Mon Oct 7 11:35:22 2024 From: pierre at joliv.et (Pierre Jolivet) Date: Mon, 7 Oct 2024 18:35:22 +0200 Subject: [petsc-users] ParMmg crashes when run in parallel: Assertion failed: (isfinite(dd)), function PMMG_hashNorver_normals, file analys_pmmg.c, line 1072 In-Reply-To: <7466C614-D2E4-408F-AF24-B5E83E883617@gmail.com> References: <7466C614-D2E4-408F-AF24-B5E83E883617@gmail.com> Message-ID: <47FBFD10-E625-412D-89F5-46EC132F6D66@joliv.et> An HTML attachment was scrubbed... URL: From yangzongze at gmail.com Mon Oct 7 11:47:51 2024 From: yangzongze at gmail.com (Zongze Yang) Date: Tue, 8 Oct 2024 00:47:51 +0800 Subject: [petsc-users] ParMmg crashes when run in parallel: Assertion failed: (isfinite(dd)), function PMMG_hashNorver_normals, file analys_pmmg.c, line 1072 In-Reply-To: <47FBFD10-E625-412D-89F5-46EC132F6D66@joliv.et> References: <7466C614-D2E4-408F-AF24-B5E83E883617@gmail.com> <47FBFD10-E625-412D-89F5-46EC132F6D66@joliv.et> Message-ID: Thanks! I?ll try using Mmg for smaller meshes. Once I reach a mesh scale of a couple of million tetrahedra and if the error persists, I will get back to you. I appreciate your guidance! Best wishes, Zongze > On 8 Oct 2024, at 00:35, Pierre Jolivet wrote: > > The ParMmg developers have the following rule of thumb: if the adaptation process takes less than 10 minutes, stick to Mmg. > I have experimented on numerous occasions that this is indeed a good rule. > Once you have reached such a scale (that?s about a couple of millions of tetrahedra, at least how we do it in FreeFEM without DMPlex) and the error persists, please let me know. > > Thanks, > Pierre > >> On 7 Oct 2024, at 6:12?PM, Zongze Yang wrote: >> >> ? >> Hi Pierre, >> >> Thank you for the advice. I will look into implementing it as suggested. >> >> This is just a toy example; the mesh may be larger in the actual calculations. >> >> Best wishes, >> Zongze >> >>> On 7 Oct 2024, at 23:34, Pierre Jolivet wrote: >>> >>> Your mesh is tiny, you should not use ParMmg but plain Mmg instead which is much more robust. >>> I don?t know how easy it is to do with DMPlex though (gather a DM and a metric on a single process), especially from the command line. >>> >>> Thanks, >>> Pierre >>> >>>> On 7 Oct 2024, at 4:46?PM, Zongze Yang > wrote: >>>> >>>> ?Hi PETSc team, >>>> >>>> I noticed that the ParMmg interface for overlap (https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/merge_requests/7593__;!!G_uCfscf7eWS!bh5eQHxA8QXn10TtTUOI5lmjFbfxFBtZz9CQoiYgyZSR7oDMYp8NK2HHP2UaWRXmU82ddDBpMx3ctAfX8GFQAuL_$ ) has been fixed. I tried using the updated version in my code, but it did not perform as expected when refining the attached mesh (or fine mesh of the same geometry) in parallel. >>>> >>>> Here are the details output: The first two sequential runs work correctly. However, when running the code in parallel without refinement, some warnings are generated. After refining the mesh in parallel, the code crashes. >>>> >>>> ``` >>>> ? mpiexec -n 1 python test_adapt.py -dm_adaptor parmmg >>>> >>>> ? mpiexec -n 1 python test_adapt.py -dm_adaptor parmmg -refine 1 >>>> >>>> ? mpiexec -n 2 python test_adapt.py -dm_adaptor parmmg >>>> ## Warning: MMG5_interpreg_ani: 608: unexpected case (null normal), impossible interpolation. >>>> >>>> ## Warning: MMG5_movbdyregpt_ani: unable to compute optimal position for at least 1 point. >>>> ## Warning: MMG5_interpreg_ani: 608: unexpected case (null normal), impossible interpolation. >>>> >>>> ## Warning: MMG5_movbdyregpt_ani: unable to compute optimal position for at least 1 point. >>>> >>>> ? mpiexec -n 2 python test_adapt.py -dm_adaptor parmmg -refine 1 >>>> Assertion failed: (isfinite(dd)), function PMMG_hashNorver_normals, file analys_pmmg.c, line 1072. >>>> Assertion failed: (isfinite(dd)), function PMMG_hashNorver_normals, file analys_pmmg.c, line 1072. >>>> >>>> =================================================================================== >>>> = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES >>>> = PID 22704 RUNNING AT yzzs-mac.local >>>> = EXIT CODE: 6 >>>> = CLEANING UP REMAINING PROCESSES >>>> = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES >>>> =================================================================================== >>>> YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Abort trap: 6 (signal 6) >>>> This typically refers to a problem with your application. >>>> Please see the FAQ page for debugging suggestions >>>> >>>> >>>> ? mpiexec -n 2 python test_adapt.py -dm_adaptor parmmg -overlap >>>> ## Warning: MMG5_interpreg_ani: 608: unexpected case (null normal), impossible interpolation. >>>> >>>> ## Warning: MMG5_movbdyregpt_ani: unable to compute optimal position for at least 1 point. >>>> ## Warning: MMG5_interpreg_ani: 608: unexpected case (null normal), impossible interpolation. >>>> >>>> ## Warning: MMG5_movbdyregpt_ani: unable to compute optimal position for at least 1 point. >>>> >>>> >>>> ? mpiexec -n 2 python test_adapt.py -dm_adaptor parmmg -overlap -refine >>>> Assertion failed: (isfinite(dd)), function PMMG_hashNorver_normals, file analys_pmmg.c, line 1072. >>>> Assertion failed: (isfinite(dd)), function PMMG_hashNorver_normals, file analys_pmmg.c, line 1072. >>>> >>>> =================================================================================== >>>> = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES >>>> = PID 25424 RUNNING AT yzzs-mac.local >>>> = EXIT CODE: 6 >>>> = CLEANING UP REMAINING PROCESSES >>>> = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES >>>> =================================================================================== >>>> YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Abort trap: 6 (signal 6) >>>> This typically refers to a problem with your application. >>>> Please see the FAQ page for debugging suggestions >>>> ``` >>>> >>>> The mesh geometry is a cylinder with an interface. The geometry file of gmsh, the mesh file, and the code are attached. Could someone take a look and help identify the issue? >>>> >>>> Thank you! >>>> >>>> Best regards, >>>> Zongze >>>> >>>> >>>> >>>> >>>> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From boneill at rnet-tech.com Fri Oct 11 10:13:01 2024 From: boneill at rnet-tech.com (Ben O'Neill) Date: Fri, 11 Oct 2024 11:13:01 -0400 Subject: [petsc-users] Graphical User Interfaces for PETSc applications. Message-ID: PETSc users, I have been developing an open-source framework called the VnV Toolkit that helps developers build graphical user interfaces around scientific applications. Like most documentation generators, the toolkit uses comments/markup defined in your codebase to generate customized graphical interfaces for scientific applications. I just wanted to reach out and see if there are any PETSc users who might be interested in trying out the framework in their applications, and/or providing feedback. I have already created a few petsc based POC interfaces, and am ready to start trying things out in some real applications. If you are interested, and would like more info, please feel free to let me know. More information about the toolkit can be found at the github page and on our demo site Thank you, Ben O'Neill -------------- next part -------------- An HTML attachment was scrubbed... URL: From daniel.pino_munoz at mines-paristech.fr Fri Oct 11 16:03:02 2024 From: daniel.pino_munoz at mines-paristech.fr (Daniel Pino Munoz) Date: Fri, 11 Oct 2024 23:03:02 +0200 Subject: [petsc-users] Compilation error Message-ID: <5a59fa8b-262b-41d2-a2f5-cb08b05fd558@mines-paristech.fr> Dear all, I am currently trying to compile my code but the compilation fails with the following error: error "PETSc was configured with OpenMPI but now appears to be compiling using a non-OpenMPI mpi.h I have obviously checked and I am compiling my sources using the same compiler used to compile Petsc. To check this, I? looked into the configure.log file generated by petsc and verified that the compiler listed was indeed the one currently been used to compile my sources. Does anyone have any idea of where the problem might be coming from? Thank you, ? D. From junchao.zhang at gmail.com Fri Oct 11 17:04:47 2024 From: junchao.zhang at gmail.com (Junchao Zhang) Date: Fri, 11 Oct 2024 17:04:47 -0500 Subject: [petsc-users] Compilation error In-Reply-To: <5a59fa8b-262b-41d2-a2f5-cb08b05fd558@mines-paristech.fr> References: <5a59fa8b-262b-41d2-a2f5-cb08b05fd558@mines-paristech.fr> Message-ID: Check they are the same MPI wrappers(if you use them), for example with "which mpicc" or "make V=1" to compare the exact executable path --Junchao Zhang On Fri, Oct 11, 2024 at 4:10?PM Daniel Pino Munoz < daniel.pino_munoz at mines-paristech.fr> wrote: > Dear all, > > I am currently trying to compile my code but the compilation fails with > the following error: > > error "PETSc was configured with OpenMPI but now appears to be compiling > using a non-OpenMPI mpi.h > > I have obviously checked and I am compiling my sources using the same > compiler used to compile Petsc. To check this, I looked into the > configure.log file generated by petsc and verified that the compiler > listed was indeed the one currently been used to compile my sources. > > Does anyone have any idea of where the problem might be coming from? > > Thank you, > > D. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From daniel.pino_munoz at mines-paristech.fr Fri Oct 11 17:13:59 2024 From: daniel.pino_munoz at mines-paristech.fr (Daniel Pino Munoz) Date: Sat, 12 Oct 2024 00:13:59 +0200 Subject: [petsc-users] Compilation error In-Reply-To: References: <5a59fa8b-262b-41d2-a2f5-cb08b05fd558@mines-paristech.fr> Message-ID: <86411ad2-4198-4778-81db-73f1c895c2ba@mines-paristech.fr> I did that too, and I am using the same compiler. In fact I also checked the compilation of my code by setting VERBOSE=1 and looking at the compilation command. It is indeed the same compiler... It's pretty odd... Thanks anyway, ? D. On 12/10/2024 00:04, Junchao Zhang wrote: > Check they are the same MPI wrappers(if you use them), for example > with "which mpicc" or "make V=1" to compare the exact executable path > > --Junchao Zhang > > > On Fri, Oct 11, 2024 at 4:10?PM Daniel Pino Munoz > wrote: > > Dear all, > > I am currently trying to compile my code but the compilation fails > with > the following error: > > error "PETSc was configured with OpenMPI but now appears to be > compiling > using a non-OpenMPI mpi.h > > I have obviously checked and I am compiling my sources using the same > compiler used to compile Petsc. To check this, I? looked into the > configure.log file generated by petsc and verified that the compiler > listed was indeed the one currently been used to compile my sources. > > Does anyone have any idea of where the problem might be coming from? > > Thank you, > > ?? D. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From junchao.zhang at gmail.com Fri Oct 11 18:13:27 2024 From: junchao.zhang at gmail.com (Junchao Zhang) Date: Fri, 11 Oct 2024 18:13:27 -0500 Subject: [petsc-users] Compilation error In-Reply-To: <86411ad2-4198-4778-81db-73f1c895c2ba@mines-paristech.fr> References: <5a59fa8b-262b-41d2-a2f5-cb08b05fd558@mines-paristech.fr> <86411ad2-4198-4778-81db-73f1c895c2ba@mines-paristech.fr> Message-ID: Perhaps you need to attach your configure.log --Junchao Zhang On Fri, Oct 11, 2024 at 5:14?PM Daniel Pino Munoz < daniel.pino_munoz at mines-paristech.fr> wrote: > I did that too, and I am using the same compiler. > > In fact I also checked the compilation of my code by setting VERBOSE=1 and > looking at the compilation command. It is indeed the same compiler... It's > pretty odd... > > Thanks anyway, > > D. > On 12/10/2024 00:04, Junchao Zhang wrote: > > Check they are the same MPI wrappers(if you use them), for example with > "which mpicc" or "make V=1" to compare the exact executable path > > --Junchao Zhang > > > On Fri, Oct 11, 2024 at 4:10?PM Daniel Pino Munoz < > daniel.pino_munoz at mines-paristech.fr> wrote: > >> Dear all, >> >> I am currently trying to compile my code but the compilation fails with >> the following error: >> >> error "PETSc was configured with OpenMPI but now appears to be compiling >> using a non-OpenMPI mpi.h >> >> I have obviously checked and I am compiling my sources using the same >> compiler used to compile Petsc. To check this, I looked into the >> configure.log file generated by petsc and verified that the compiler >> listed was indeed the one currently been used to compile my sources. >> >> Does anyone have any idea of where the problem might be coming from? >> >> Thank you, >> >> D. >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay.anl at fastmail.org Fri Oct 11 18:17:36 2024 From: balay.anl at fastmail.org (Satish Balay) Date: Fri, 11 Oct 2024 18:17:36 -0500 (CDT) Subject: [petsc-users] Compilation error In-Reply-To: <86411ad2-4198-4778-81db-73f1c895c2ba@mines-paristech.fr> References: <5a59fa8b-262b-41d2-a2f5-cb08b05fd558@mines-paristech.fr> <86411ad2-4198-4778-81db-73f1c895c2ba@mines-paristech.fr> Message-ID: Here is one way to check. - check make.log for the build - it might have something like: ----------------------------------------- Using system modules: Using mpi.h: # 1 "/home/balay/soft/mpich-4.0.1/include/mpi.h" 1 ----------------------------------------- - now check your application build where you are getting the error - compile that source code with the additional option "-E" to see which mpi.h is getting used. balay at p1 /home/balay/petsc/src/ksp/ksp/tutorials (main =) $ make ex1.o CPPFLAGS=-E mpicc -o ex1.o -c -fPIC -Wall -Wwrite-strings -Wno-unknown-pragmas -Wno-lto-type-mismatch -Wno-stringop-overflow -fstack-protector -fvisibility=hidden -Wall -Wwrite-strings -Wno-unknown-pragmas -Wno-lto-type-mismatch -Wno-stringop-overflow -fstack-protector -fvisibility=hidden -g3 -O0 -I/home/balay/petsc/include -I/home/balay/petsc/arch-linux-c-debug/include -E ex1.c balay at p1 /home/balay/petsc/src/ksp/ksp/tutorials (main =) $ grep 'mpi\.h' ex1.o |head # 1 "/home/balay/soft/mpich-4.0.1/include/mpi.h" 1 # 40 "/home/balay/soft/mpich-4.0.1/include/mpi.h" # 66 "/home/balay/soft/mpich-4.0.1/include/mpi.h" # 84 "/home/balay/soft/mpich-4.0.1/include/mpi.h" # 144 "/home/balay/soft/mpich-4.0.1/include/mpi.h" # 156 "/home/balay/soft/mpich-4.0.1/include/mpi.h" # 198 "/home/balay/soft/mpich-4.0.1/include/mpi.h" # 244 "/home/balay/soft/mpich-4.0.1/include/mpi.h" # 262 "/home/balay/soft/mpich-4.0.1/include/mpi.h" # 333 "/home/balay/soft/mpich-4.0.1/include/mpi.h" balay at p1 /home/balay/petsc/src/ksp/ksp/tutorials (main =) $ Ok here they match. If the same mpi.h is used for your usecase in both PETSc build - and in your application code - but you are still getting the error - send us the build logs - configure.log, make.log [you can use petsc-maint at mcs.anl.gov] Satish On Sat, 12 Oct 2024, Daniel Pino Munoz wrote: > I did that too, and I am using the same compiler. > > In fact I also checked the compilation of my code by setting VERBOSE=1 and > looking at the compilation command. It is indeed the same compiler... It's > pretty odd... > > Thanks anyway, > > ? D. > > On 12/10/2024 00:04, Junchao Zhang wrote: > > Check they are the same MPI wrappers(if you use them), for example with > > "which mpicc" or "make V=1" to compare the exact executable path > > > > --Junchao Zhang > > > > > > On Fri, Oct 11, 2024 at 4:10?PM Daniel Pino Munoz > > wrote: > > > > Dear all, > > > > I am currently trying to compile my code but the compilation fails > > with > > the following error: > > > > error "PETSc was configured with OpenMPI but now appears to be > > compiling > > using a non-OpenMPI mpi.h > > > > I have obviously checked and I am compiling my sources using the same > > compiler used to compile Petsc. To check this, I? looked into the > > configure.log file generated by petsc and verified that the compiler > > listed was indeed the one currently been used to compile my sources. > > > > Does anyone have any idea of where the problem might be coming from? > > > > Thank you, > > > > ?? D. > > > From daniel.pino_munoz at mines-paristech.fr Sat Oct 12 09:31:43 2024 From: daniel.pino_munoz at mines-paristech.fr (Daniel Pino Munoz) Date: Sat, 12 Oct 2024 16:31:43 +0200 Subject: [petsc-users] Compilation error In-Reply-To: References: <5a59fa8b-262b-41d2-a2f5-cb08b05fd558@mines-paristech.fr> <86411ad2-4198-4778-81db-73f1c895c2ba@mines-paristech.fr> Message-ID: <90f703bc-4a2d-445a-ab6b-70beaf69c23d@mines-paristech.fr> Thank you Satish and Junchao, Actually following Satish's instructions I found that I am indeed using the same compiler but for some reason my command does not include the mpi include path. I am going to check now my cmake files because it was initially found so I should be doing something wrong somewhere else. Thank you and I think the problem is solved now. Best, ? Daniel On 12/10/2024 01:17, Satish Balay wrote: > Here is one way to check. > > - check make.log for the build - it might have something like: > > ----------------------------------------- > Using system modules: > Using mpi.h: # 1 "/home/balay/soft/mpich-4.0.1/include/mpi.h" 1 > ----------------------------------------- > > - now check your application build where you are getting the error - compile that source code with the additional option "-E" to see which mpi.h is getting used. > > balay at p1 /home/balay/petsc/src/ksp/ksp/tutorials (main =) > $ make ex1.o CPPFLAGS=-E > mpicc -o ex1.o -c -fPIC -Wall -Wwrite-strings -Wno-unknown-pragmas -Wno-lto-type-mismatch -Wno-stringop-overflow -fstack-protector -fvisibility=hidden -Wall -Wwrite-strings -Wno-unknown-pragmas -Wno-lto-type-mismatch -Wno-stringop-overflow -fstack-protector -fvisibility=hidden -g3 -O0 -I/home/balay/petsc/include -I/home/balay/petsc/arch-linux-c-debug/include -E ex1.c > balay at p1 /home/balay/petsc/src/ksp/ksp/tutorials (main =) > $ grep 'mpi\.h' ex1.o |head > # 1 "/home/balay/soft/mpich-4.0.1/include/mpi.h" 1 > # 40 "/home/balay/soft/mpich-4.0.1/include/mpi.h" > # 66 "/home/balay/soft/mpich-4.0.1/include/mpi.h" > # 84 "/home/balay/soft/mpich-4.0.1/include/mpi.h" > # 144 "/home/balay/soft/mpich-4.0.1/include/mpi.h" > # 156 "/home/balay/soft/mpich-4.0.1/include/mpi.h" > # 198 "/home/balay/soft/mpich-4.0.1/include/mpi.h" > # 244 "/home/balay/soft/mpich-4.0.1/include/mpi.h" > # 262 "/home/balay/soft/mpich-4.0.1/include/mpi.h" > # 333 "/home/balay/soft/mpich-4.0.1/include/mpi.h" > balay at p1 /home/balay/petsc/src/ksp/ksp/tutorials (main =) > $ > > Ok here they match. > > If the same mpi.h is used for your usecase in both PETSc build - and > in your application code - but you are still getting the error - send > us the build logs - configure.log, make.log [you can use petsc-maint at mcs.anl.gov] > > Satish > > On Sat, 12 Oct 2024, Daniel Pino Munoz wrote: > >> I did that too, and I am using the same compiler. >> >> In fact I also checked the compilation of my code by setting VERBOSE=1 and >> looking at the compilation command. It is indeed the same compiler... It's >> pretty odd... >> >> Thanks anyway, >> >> ? D. >> >> On 12/10/2024 00:04, Junchao Zhang wrote: >>> Check they are the same MPI wrappers(if you use them), for example with >>> "which mpicc" or "make V=1" to compare the exact executable path >>> >>> --Junchao Zhang >>> >>> >>> On Fri, Oct 11, 2024 at 4:10?PM Daniel Pino Munoz >>> wrote: >>> >>> Dear all, >>> >>> I am currently trying to compile my code but the compilation fails >>> with >>> the following error: >>> >>> error "PETSc was configured with OpenMPI but now appears to be >>> compiling >>> using a non-OpenMPI mpi.h >>> >>> I have obviously checked and I am compiling my sources using the same >>> compiler used to compile Petsc. To check this, I? looked into the >>> configure.log file generated by petsc and verified that the compiler >>> listed was indeed the one currently been used to compile my sources. >>> >>> Does anyone have any idea of where the problem might be coming from? >>> >>> Thank you, >>> >>> ?? D. >>> From bsmith at petsc.dev Sat Oct 12 12:46:20 2024 From: bsmith at petsc.dev (Barry Smith) Date: Sat, 12 Oct 2024 12:46:20 -0500 Subject: [petsc-users] Graphical User Interfaces for PETSc applications. In-Reply-To: References: Message-ID: <73DA2446-FB9A-40D5-A304-7589B2A9B71F@petsc.dev> Ben, This is interesting; it seems like it may be useful inside the PETSc library code. Please take a quick look at https://urldefense.us/v3/__https://bitbucket.org/saws/saws/src/master/__;!!G_uCfscf7eWS!eBNC69y99tie5v_KSbJ3PosiD5GwewD_jS-U2-1sd7l6XOaXwSy97N9job28xHc_2UBnSz-cuOUpWS-u6j58rBo$ it has some overlap with your work. PETSc talks to it through an API embedded in the PETSc source, perhaps some of that same infrastructure could be leveraged to work with VnV. Barry > On Oct 11, 2024, at 10:13?AM, Ben O'Neill wrote: > > PETSc users, > > I have been developing an open-source framework called the VnV Toolkit that helps developers build graphical user interfaces around scientific applications. Like most documentation generators, the toolkit uses comments/markup defined in your codebase to generate customized graphical interfaces for scientific applications. > > I just wanted to reach out and see if there are any PETSc users who might be interested in trying out the framework in their applications, and/or providing feedback. I have already created a few petsc based POC interfaces, and am ready to start trying things out in some real applications. If you are interested, and would like more info, please feel free to let me know. More information about the toolkit can be found at the github page and on our demo site > > > Thank you, > > Ben O'Neill > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Sat Oct 12 18:47:41 2024 From: bsmith at petsc.dev (Barry Smith) Date: Sat, 12 Oct 2024 19:47:41 -0400 Subject: [petsc-users] Code In-Reply-To: References: <8130bff6-e77d-480f-844c-986ab533d8ba@cs.ubc.ca> Message-ID: <3632C9F0-12FF-4A8E-A117-213E4A78E922@petsc.dev> Sorry for the long delay in getting back to you. I think there is not directly a problem with the code; rather, with these Q0 elements, Galerkin is not a good approach. I went back to src/ksp/ksp/tutorials/ex32.c and ran a little deeper and noticed that when running without Galerkin it gives very good, expected convergence. But with Galerkin it does not. I had to make some fixes in refinement for DMDA to run how I wanted they are in the branch in https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/merge_requests/7940__;!!G_uCfscf7eWS!fXCpNUu10XTTCmVzPF3jsEz72QFuFGPkzeF0He6viicBt934KYfgROb3ncB2BZnfEMWRwJnoelHdOTKlzPHkaiU$ So I think the correct answer is for you to not use Galerkin but instead discretize on each level the matrix as is done in ex32.c For this to work in your code you will need to change how you scale the matrix (and right hand side) you generate, see ComputeMatrix() in ex32.c Note how the matrix entries have the scaling HxdHy etc. Similarly the RHS has the scaling by Hx for Hy. Also DMStag uses .25 in the interpolation but DMDA uses 1; I am not sure why the 1 that DMDA uses works, but if you rescale your matrices and right hand side and don't get use Galerkin and still get poor convergence then try different scaling for the interpolation. Here is the performance for ex32. Note how fantastic it is, I think you should be virtually the same. /ex32 -pc_type mg -pc_mg_levels 5 -ksp_monitor_true_residual -ksp_rtol 1.e-10 -ksp_type cg -pc_mg_galerkin none -mg_levels_pc_type sor -mg_levels_ksp_type richardson -mg_levels_ksp_max_it 2 -mg_coarse_pc_type svd -da_refine 5 0 KSP preconditioned resid norm 8.707575268782e-01 true resid norm 2.842788112864e-03 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 5.067478574947e-03 true resid norm 8.807970167024e-05 ||r(i)||/||b|| 3.098356197273e-02 2 KSP preconditioned resid norm 5.566700402730e-05 true resid norm 2.014981529883e-06 ||r(i)||/||b|| 7.088046839529e-04 3 KSP preconditioned resid norm 2.698851702073e-07 true resid norm 3.013479554557e-08 ||r(i)||/||b|| 1.060043673646e-05 4 KSP preconditioned resid norm 2.689594854129e-09 true resid norm 2.536151839771e-10 ||r(i)||/||b|| 8.921353752308e-08 5 KSP preconditioned resid norm 2.232993540537e-11 true resid norm 2.360948191859e-12 ||r(i)||/||b|| 8.305044548259e-10 Barry > On Oct 8, 2024, at 2:09?PM, Gautam Luhana wrote: > > Hi Barry, > > Did you get time to look through the code and see where the issue might be? > > Regards, > Gautam > > On 2024-10-02 11:58 a.m., Gautam Luhana wrote: >> Hi Barry, >> >> I've stripped away the multiphyics functionality and put the remaining stokes part in the repo here: >> >> https://urldefense.us/v3/__https://github.com/gkluhana/stokesToy__;!!G_uCfscf7eWS!fXCpNUu10XTTCmVzPF3jsEz72QFuFGPkzeF0He6viicBt934KYfgROb3ncB2BZnfEMWRwJnoelHdOTKl4GWg6pc$ >> >> Let me know if you have trouble accessing the code. >> >> I apologize for old routines/multiphysics functionality that may still be there. >> >> Thanks & Regards, >> Gautam >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From boneill at rnet-tech.com Sun Oct 13 20:13:13 2024 From: boneill at rnet-tech.com (Ben O'Neill) Date: Sun, 13 Oct 2024 21:13:13 -0400 Subject: [petsc-users] Graphical User Interfaces for PETSc applications. In-Reply-To: <73DA2446-FB9A-40D5-A304-7589B2A9B71F@petsc.dev> References: <73DA2446-FB9A-40D5-A304-7589B2A9B71F@petsc.dev> Message-ID: Barry, Thanks for looking into VnV. I am more than happy to put together a proof-of-concept for VnV support in PETSc codebase. After a quick look at SAWs, I think we could, at the very least, hook into the SAWs calls for data collection (although supporting two-way communication would be awesome). I think the SAWs interface together with a couple of VnV based monitors, a log handler, and maybe by hooking into the perfstubs API, we could get a pretty good non-intrusive VnV integration. Thanks, Ben . On Sat, Oct 12, 2024 at 1:46?PM Barry Smith wrote: > > Ben, > > This is interesting; it seems like it may be useful inside the PETSc > library code. Please take a quick look at > https://urldefense.us/v3/__https://bitbucket.org/saws/saws/src/master/__;!!G_uCfscf7eWS!aoA7vvIY_qtiinq8T2qJqpJysRjXC2ZhwLPIqH-qe19ezxS8G-u-ZUZF8P59prIwOFiJzezXg9cKvkpMMeXLZDt8$ it has some overlap with your > work. PETSc talks to it through an API embedded in the PETSc source, > perhaps some of that same infrastructure could be leveraged to work with > VnV. > > Barry > > > On Oct 11, 2024, at 10:13?AM, Ben O'Neill wrote: > > PETSc users, > > I have been developing an open-source framework called the VnV Toolkit > that helps developers build graphical user interfaces around scientific > applications. Like most documentation generators, the toolkit uses > comments/markup defined in your codebase to generate customized graphical > interfaces for scientific applications. > > I just wanted to reach out and see if there are any PETSc users who might > be interested in trying out the framework in their applications, and/or > providing feedback. I have already created a few petsc based POC > interfaces, and am ready to start trying things out in some real > applications. If you are interested, and would like more info, please feel > free to let me know. More information about the toolkit can be found at the github > page > and > on our demo site > > > > Thank you, > > Ben O'Neill > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From nicolas.tardieu at edf.fr Mon Oct 14 02:43:20 2024 From: nicolas.tardieu at edf.fr (TARDIEU Nicolas) Date: Mon, 14 Oct 2024 07:43:20 +0000 Subject: [petsc-users] Compilation fails in tao Fortran modules Message-ID: Dear PETSc Team, The compilation of the PETSc release version fails on my Debian machine with the message : # --------------------------------------------------------------------------------------------------------------------------- FC arch-linux-c-debug/obj/src/ts/f90-mod/petsctsmod.o FC arch-linux-c-debug/obj/src/tao/f90-mod/petsctaomod.o /home/UserName/dev/petsc_aster/include/../src/tao/f90-mod/ftn-auto-interfaces/petsctao.h90:2:17: Tao a ! Tao 1 Error: Derived type ?ttao? at (1) is being used before it is defined /home/UserName/dev/petsc_aster/include/../src/tao/f90-mod/ftn-auto-interfaces/petsctao.h90:7:17: # --------------------------------------------------------------------------------------------------------------------------- Thanks in advance for your help. Regards, Nicolas -- Nicolas Tardieu Ing PhD Computational Mechanics EDF - R&D Dpt ERMES PARIS-SACLAY, FRANCE Ce message et toutes les pi?ces jointes (ci-apr?s le 'Message') sont ?tablis ? l'intention exclusive des destinataires et les informations qui y figurent sont strictement confidentielles. Toute utilisation de ce Message non conforme ? sa destination, toute diffusion ou toute publication totale ou partielle, est interdite sauf autorisation expresse. Si vous n'?tes pas le destinataire de ce Message, il vous est interdit de le copier, de le faire suivre, de le divulguer ou d'en utiliser tout ou partie. Si vous avez re?u ce Message par erreur, merci de le supprimer de votre syst?me, ainsi que toutes ses copies, et de n'en garder aucune trace sur quelque support que ce soit. Nous vous remercions ?galement d'en avertir imm?diatement l'exp?diteur par retour du message. Il est impossible de garantir que les communications par messagerie ?lectronique arrivent en temps utile, sont s?curis?es ou d?nu?es de toute erreur ou virus. ____________________________________________________ This message and any attachments (the 'Message') are intended solely for the addressees. The information contained in this Message is confidential. Any use of information contained in this Message not in accord with its purpose, any dissemination or disclosure, either whole or partial, is prohibited except formal approval. If you are not the addressee, you may not copy, forward, disclose or use any part of it. If you have received this message in error, please delete it and all copies from your system and notify the sender immediately by return message. E-mail communication cannot be guaranteed to be timely secure, error or virus-free. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: configure.log Type: text/x-log Size: 1085335 bytes Desc: configure.log URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: make.log Type: text/x-log Size: 150422 bytes Desc: make.log URL: From knepley at gmail.com Mon Oct 14 06:28:57 2024 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 14 Oct 2024 07:28:57 -0400 Subject: [petsc-users] Compilation fails in tao Fortran modules In-Reply-To: References: Message-ID: On Mon, Oct 14, 2024 at 3:44?AM TARDIEU Nicolas via petsc-users < petsc-users at mcs.anl.gov> wrote: > Dear PETSc Team, > > The compilation of the PETSc release version fails on my Debian machine > with the message : > > # > --------------------------------------------------------------------------------------------------------------------------- > FC arch-linux-c-debug/obj/src/ts/f90-mod/petsctsmod.o > FC arch-linux-c-debug/obj/src/tao/f90-mod/petsctaomod.o > > /home/UserName/dev/petsc_aster/include/../src/tao/f90-mod/ftn-auto-interfaces/petsctao.h90:2:17: > > Tao a ! Tao > 1 > Error: Derived type ?ttao? at (1) is being used before it is defined > > /home/UserName/dev/petsc_aster/include/../src/tao/f90-mod/ftn-auto-interfaces/petsctao.h90:7:17: > # > --------------------------------------------------------------------------------------------------------------------------- > > Thanks in advance for your help. > You have old Fotran stub under '/src'. I think you need to run make deleteallfortranstubs make allfortranstubs make Thanks, Matt > Regards, > Nicolas > -- > Nicolas Tardieu > Ing PhD Computational Mechanics > EDF - R&D Dpt ERMES > PARIS-SACLAY, FRANCE > > > > Ce message et toutes les pi?ces jointes (ci-apr?s le 'Message') sont > ?tablis ? l'intention exclusive des destinataires et les informations qui y > figurent sont strictement confidentielles. Toute utilisation de ce Message > non conforme ? sa destination, toute diffusion ou toute publication totale > ou partielle, est interdite sauf autorisation expresse. > > Si vous n'?tes pas le destinataire de ce Message, il vous est interdit de > le copier, de le faire suivre, de le divulguer ou d'en utiliser tout ou > partie. Si vous avez re?u ce Message par erreur, merci de le supprimer de > votre syst?me, ainsi que toutes ses copies, et de n'en garder aucune trace > sur quelque support que ce soit. Nous vous remercions ?galement d'en > avertir imm?diatement l'exp?diteur par retour du message. > > Il est impossible de garantir que les communications par messagerie > ?lectronique arrivent en temps utile, sont s?curis?es ou d?nu?es de toute > erreur ou virus. > ____________________________________________________ > > This message and any attachments (the 'Message') are intended solely for > the addressees. The information contained in this Message is confidential. > Any use of information contained in this Message not in accord with its > purpose, any dissemination or disclosure, either whole or partial, is > prohibited except formal approval. > > If you are not the addressee, you may not copy, forward, disclose or use > any part of it. If you have received this message in error, please delete > it and all copies from your system and notify the sender immediately by > return message. > > E-mail communication cannot be guaranteed to be timely secure, error or > virus-free. > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!bgIKi1-oKxz5PBKrVf24n6VF49j5HwlMXbU6SN8oRBjbzWTH8xlh6wTG0cLbN1nYumK2Th5fONJcg-u_rIq5$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From nicolas.tardieu at edf.fr Mon Oct 14 06:58:37 2024 From: nicolas.tardieu at edf.fr (TARDIEU Nicolas) Date: Mon, 14 Oct 2024 11:58:37 +0000 Subject: [petsc-users] Compilation fails in tao Fortran modules In-Reply-To: References: Message-ID: You are right Matt! Thank you so much for your help. Nicolas ________________________________ From: knepley at gmail.com Sent: Monday, October 14, 2024 1:28:57 PM To: TARDIEU Nicolas Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] Compilation fails in tao Fortran modules Exp?diteur externe : V?rifier l?exp?diteur avant de cliquer sur les liens ou pi?ces-jointes External Sender : Check the sender before clicking any links or attachments On Mon, Oct 14, 2024 at 3:44?AM TARDIEU Nicolas via petsc-users > wrote: Dear PETSc Team, The compilation of the PETSc release version fails on my Debian machine with the message : # --------------------------------------------------------------------------------------------------------------------------- FC arch-linux-c-debug/obj/src/ts/f90-mod/petsctsmod.o FC arch-linux-c-debug/obj/src/tao/f90-mod/petsctaomod.o /home/UserName/dev/petsc_aster/include/../src/tao/f90-mod/ftn-auto-interfaces/petsctao.h90:2:17: Tao a ! Tao 1 Error: Derived type ?ttao? at (1) is being used before it is defined /home/UserName/dev/petsc_aster/include/../src/tao/f90-mod/ftn-auto-interfaces/petsctao.h90:7:17: # --------------------------------------------------------------------------------------------------------------------------- Thanks in advance for your help. You have old Fotran stub under '/src'. I think you need to run make deleteallfortranstubs make allfortranstubs make Thanks, Matt Regards, Nicolas -- Nicolas Tardieu Ing PhD Computational Mechanics EDF - R&D Dpt ERMES PARIS-SACLAY, FRANCE Ce message et toutes les pi?ces jointes (ci-apr?s le 'Message') sont ?tablis ? l'intention exclusive des destinataires et les informations qui y figurent sont strictement confidentielles. Toute utilisation de ce Message non conforme ? sa destination, toute diffusion ou toute publication totale ou partielle, est interdite sauf autorisation expresse. Si vous n'?tes pas le destinataire de ce Message, il vous est interdit de le copier, de le faire suivre, de le divulguer ou d'en utiliser tout ou partie. Si vous avez re?u ce Message par erreur, merci de le supprimer de votre syst?me, ainsi que toutes ses copies, et de n'en garder aucune trace sur quelque support que ce soit. Nous vous remercions ?galement d'en avertir imm?diatement l'exp?diteur par retour du message. Il est impossible de garantir que les communications par messagerie ?lectronique arrivent en temps utile, sont s?curis?es ou d?nu?es de toute erreur ou virus. ____________________________________________________ This message and any attachments (the 'Message') are intended solely for the addressees. The information contained in this Message is confidential. Any use of information contained in this Message not in accord with its purpose, any dissemination or disclosure, either whole or partial, is prohibited except formal approval. If you are not the addressee, you may not copy, forward, disclose or use any part of it. If you have received this message in error, please delete it and all copies from your system and notify the sender immediately by return message. E-mail communication cannot be guaranteed to be timely secure, error or virus-free. -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!enplDkXM9iqE1nA4xnpNBloUTvT0sIPWSe7G73ydEgij3TxpZXWHx_bEZEFz6G5hlua89wT8Z3mY93o0HbaoNvYGvSOAdG8w$ Ce message et toutes les pi?ces jointes (ci-apr?s le 'Message') sont ?tablis ? l'intention exclusive des destinataires et les informations qui y figurent sont strictement confidentielles. Toute utilisation de ce Message non conforme ? sa destination, toute diffusion ou toute publication totale ou partielle, est interdite sauf autorisation expresse. Si vous n'?tes pas le destinataire de ce Message, il vous est interdit de le copier, de le faire suivre, de le divulguer ou d'en utiliser tout ou partie. Si vous avez re?u ce Message par erreur, merci de le supprimer de votre syst?me, ainsi que toutes ses copies, et de n'en garder aucune trace sur quelque support que ce soit. Nous vous remercions ?galement d'en avertir imm?diatement l'exp?diteur par retour du message. Il est impossible de garantir que les communications par messagerie ?lectronique arrivent en temps utile, sont s?curis?es ou d?nu?es de toute erreur ou virus. ____________________________________________________ This message and any attachments (the 'Message') are intended solely for the addressees. The information contained in this Message is confidential. Any use of information contained in this Message not in accord with its purpose, any dissemination or disclosure, either whole or partial, is prohibited except formal approval. If you are not the addressee, you may not copy, forward, disclose or use any part of it. If you have received this message in error, please delete it and all copies from your system and notify the sender immediately by return message. E-mail communication cannot be guaranteed to be timely secure, error or virus-free. -------------- next part -------------- An HTML attachment was scrubbed... URL: From matteo.semplice at uninsubria.it Mon Oct 21 11:21:19 2024 From: matteo.semplice at uninsubria.it (Matteo Semplice) Date: Mon, 21 Oct 2024 18:21:19 +0200 Subject: [petsc-users] local/global DMPlex Vec output Message-ID: Dear petsc-users, ??? I am having issues with output of parallel data attached to a DMPlex (or maybe more fundamental ones about DMPlex...). So I currently 1. create a DMPlex (DMPlexCreateGmshFromFile or DMPlexCreateBoxMesh) 2. partition it 3. and create a section for my data layout with DMPlexCreateSection(ctx.dmMesh, NULL, numComp, numDof, numBC, NULL, NULL, NULL, NULL, &sUavg) 4. DMSetLocalSection(ctx.dmMesh, sUavg) 5. create solLoc and solGlob vectors with DMCreateGlobalVector and DMCreateLocalVector 6. solve .... 7. VecView(ctx.solGlob, vtkViewer) on a .vtu file but when I load data in ParaView I get more cells than expected and it is as if the cells in the halo are put twice in output. (I could create a MWE if the above is not clear) I guess that the culprit is point (4), but if I replace it with DMSetGlobalSection then I cannot create the local vector at point (5). How should I handle this properly? In my code I need to create both local and global vectors, to perform at least GlobalToLocal and to save the global data. (On a side note, I tried also HDF5 but then it complains about the DM not having a DS...; really, any working solution that allows data to be explored with Paraview is fine) Thanks for any advice! Matteo Semplice -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Oct 21 12:15:52 2024 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 21 Oct 2024 13:15:52 -0400 Subject: [petsc-users] local/global DMPlex Vec output In-Reply-To: References: Message-ID: On Mon, Oct 21, 2024 at 12:22?PM Matteo Semplice via petsc-users < petsc-users at mcs.anl.gov> wrote: > Dear petsc-users, > > I am having issues with output of parallel data attached to a DMPlex > (or maybe more fundamental ones about DMPlex...). > > So I currently > > 1. create a DMPlex (DMPlexCreateGmshFromFile or DMPlexCreateBoxMesh) > 2. partition it > 3. and create a section for my data layout with > DMPlexCreateSection(ctx.dmMesh, NULL, numComp, numDof, numBC, NULL, NULL, > NULL, NULL, &sUavg) > 4. DMSetLocalSection(ctx.dmMesh, sUavg) > 5. create solLoc and solGlob vectors with DMCreateGlobalVector and > DMCreateLocalVector > 6. solve .... > 7. VecView(ctx.solGlob, vtkViewer) on a .vtu file > > but when I load data in ParaView I get more cells than expected and it is > as if the cells in the halo are put twice in output. (I could create a MWE > if the above is not clear) > > I think we need an MWE here, because from the explanation above, it should work. However, I can try to guess the problem. When you partition the mesh, I am guessing that you have cells in the overlap. These cells must be in the point SF in order for the global section to give them a unique owner. Perhaps something has gone wrong here. Thanks, Matt > I guess that the culprit is point (4), but if I replace it with > DMSetGlobalSection then I cannot create the local vector at point (5). > > How should I handle this properly? In my code I need to create both local > and global vectors, to perform at least GlobalToLocal and to save the > global data. > > (On a side note, I tried also HDF5 but then it complains about the DM not > having a DS...; really, any working solution that allows data to be > explored with Paraview is fine) > > Thanks for any advice! > > Matteo Semplice > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!dyUl2snizAM-nnjAz0JsZ-BQgR8C9AVaDc6IMX6AAMGg4BCq5vOHECpDJvryh20xQRl6SKI_KLzhXk6P7Co7$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From matteo.semplice at uninsubria.it Mon Oct 21 17:24:25 2024 From: matteo.semplice at uninsubria.it (Matteo Semplice) Date: Tue, 22 Oct 2024 00:24:25 +0200 Subject: [petsc-users] local/global DMPlex Vec output In-Reply-To: References: Message-ID: Dear Matt, ??? I guess you're right: thresholding by rank==0 and rank==1 in paraview reveals that it is indeed the overlap cells that are appear twice in the output. The attached file is not exactly minimal but hopefully short enough. If I run it in serial, all is ok, but with ??? mpirun -np 2 ./saveDemo it creates a 10x10 grid, but I get "output.vtu" with a total of 120 cells. However the pointSF of the DMPlex seems correct. Thanks ??? Matteo Il 21/10/24 19:15, Matthew Knepley ha scritto: > On Mon, Oct 21, 2024 at 12:22?PM Matteo Semplice via petsc-users > wrote: > > Dear petsc-users, > > ??? I am having issues with output of parallel data attached to a > DMPlex (or maybe more fundamental ones about DMPlex...). > > So I currently > > 1. create a DMPlex (DMPlexCreateGmshFromFile or DMPlexCreateBoxMesh) > 2. partition it > 3. and create a section for my data layout with > DMPlexCreateSection(ctx.dmMesh, NULL, numComp, numDof, numBC, > NULL, NULL, NULL, NULL, &sUavg) > 4. DMSetLocalSection(ctx.dmMesh, sUavg) > 5. create solLoc and solGlob vectors with DMCreateGlobalVector > and DMCreateLocalVector > 6. solve .... > 7. VecView(ctx.solGlob, vtkViewer) on a .vtu file > > but when I load data in ParaView I get more cells than expected > and it is as if the cells in the halo are put twice in output. (I > could create a MWE if the above is not clear) > > I think we need an MWE here, because from the explanation above, it > should work. > > However, I can try to guess the problem. When you partition the mesh, > I am guessing that you have cells in the overlap. These cells > must be in the point SF in order for the global section to give them a > unique owner. Perhaps something has gone wrong here. > > ? Thanks, > > ? ? ?Matt > > I guess that the culprit is point (4), but if I replace it with > DMSetGlobalSection then I cannot create the local vector at point (5). > > How should I handle this properly? In my code I need to create > both local and global vectors, to perform at least GlobalToLocal > and to save the global data. > > (On a side note, I tried also HDF5 but then it complains about the > DM not having a DS...; really, any working solution that allows > data to be explored with Paraview is fine) > > Thanks for any advice! > > Matteo Semplice > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!buvrehNNjQgd0-6_n_Z6VF4Vu2psZ4V9MxJaRNwIIb6jWpe0QCnQJthiMrRDFW2RNEmqkynIU62eShkLj670fPczTBr67mFEnvBJig$ > -- --- Professore Associato in Analisi Numerica Dipartimento di Scienza e Alta Tecnologia Universit? degli Studi dell'Insubria Via Valleggio, 11 - Como -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: saveDemo.cpp Type: text/x-c++src Size: 4718 bytes Desc: not available URL: From cpraveen at gmail.com Wed Oct 23 07:23:49 2024 From: cpraveen at gmail.com (Praveen C) Date: Wed, 23 Oct 2024 17:53:49 +0530 Subject: [petsc-users] Problem running geoclaw example with petsc 3.22 Message-ID: <1C63D19F-6EBC-43A3-8B40-65EA34991AFA@gmail.com> Dear all I am not able to run the boussinesq example from geoclaw using petsc at 3.22.0 https://urldefense.us/v3/__https://github.com/clawpack/geoclaw/tree/3303883f46572c58130d161986b8a87a57ca7816/examples/bouss__;!!G_uCfscf7eWS!Z8LSV15aCfI8foq0rASSCLydMRS0fpl-nlhu4Y9dStP4hL2ldhlvXBICnvs5reaLfjg4po-XOWnV294i0QNAJQ$ It runs with petsc at 3.21.6 The error I get is given below. After printing this, the code does not progress. I use the following petsc options # set min numbers of matrix rows per MPI rank (default is 10000) -mpi_linear_solve_minimum_count_per_rank 5000 # Krylov linear solver: -mpi_linear_solver_server -mpi_linear_solver_server_view -ksp_type gmres -ksp_max_it 200 -ksp_reuse_preconditioner -ksp_rtol 1.e-9 # preconditioner: -pc_type gamg I installed petsc and other dependencies for clawpack using miniforge. Thanks pc ==> Use Bouss. in water deeper than 1.0000000000000000 Using a PETSc solver Using Bouss equations from the start rnode allocated... node allocated... listOfGrids allocated... Storage allocated... bndList allocated... Gridding level 1 at t = 0.000000E+00: 4 grids with 10000 cells Setting initial dt to 2.9999999999999999E-002 max threads set to 6 Done reading data, starting computation ... Total zeta at initial time: 39269.907650665169 GEOCLAW: Frame 0 output files done at time t = 0.000000D+00 [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Petsc has generated inconsistent data [0]PETSC ERROR: Unable to locate PCMPI allocated shared address 0x55e6d750ae20 [0]PETSC ERROR: WARNING! There are unused option(s) set! Could be the program crashed before usage or a spelling mistake, etc! [0]PETSC ERROR: Option left: name:-ksp_max_it value: 200 source: file [0]PETSC ERROR: Option left: name:-ksp_reuse_preconditioner (no value) source: file [0]PETSC ERROR: Option left: name:-ksp_rtol value: 1.e-9 source: file [0]PETSC ERROR: Option left: name:-ksp_type value: gmres source: file [0]PETSC ERROR: Option left: name:-mpi_linear_solve_minimum_count_per_rank value: 5000 source: file [0]PETSC ERROR: Option left: name:-mpi_linear_solver_server_view (no value) source: file [0]PETSC ERROR: Option left: name:-pc_type value: gamg source: file [0]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!Z8LSV15aCfI8foq0rASSCLydMRS0fpl-nlhu4Y9dStP4hL2ldhlvXBICnvs5reaLfjg4po-XOWnV29626dJiJA$ for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.22.0, Sep 28, 2024 [0]PETSC ERROR: /home/praveen/bouss/radial_flat/xgeoclaw with 6 MPI process(es) and PETSC_ARCH on euler by praveen Thu Oct 17 21:49:54 2024 [0]PETSC ERROR: Configure options: AR=${PREFIX}/bin/x86_64-conda-linux-gnu-ar CC=mpicc CXX=mpicxx FC=mpifort CFLAGS="-march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /opt/miniforge/envs/claw/include " CPPFLAGS="-DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /opt/miniforge/envs/claw/include" CXXFLAGS="-fvisibility-inlines-hidden -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /opt/miniforge/envs/claw/include " FFLAGS="-march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /opt/miniforge/envs/claw/include -Wl,--no-as-needed" LDFLAGS="-pthread -fopenmp -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,--allow-shlib-undefined -Wl,-rpath,/opt/miniforge/envs/claw/lib -Wl,-rpath-link,/opt/miniforge/envs/claw/lib -L/opt/miniforge/envs/claw/lib -Wl,-rpath-link,/opt/miniforge/envs/claw/lib" LIBS="-Wl,-rpath,/opt/miniforge/envs/claw/lib -lmpi_mpifh -lgfortran" --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 --FOPTFLAGS=-O3 --with-clib-autodetect=0 --with-cxxlib-autodetect=0 --with-fortranlib-autodetect=0 --with-debugging=0 --with-blas-lib=libblas.so --with-lapack-lib=liblapack.so --with-yaml=1 --with-hdf5=1 --with-fftw=1 --with-hwloc=0 --with-hypre=1 --with-metis=1 --with-mpi=1 --with-mumps=1 --with-parmetis=1 --with-pthread=1 --with-ptscotch=1 --with-shared-libraries --with-ssl=0 --with-scalapack=1 --with-superlu=1 --with-superlu_dist=1 --with-superlu_dist-include=/opt/miniforge/envs/claw/include/superlu-dist --with-superlu_dist-lib=-lsuperlu_dist --with-suitesparse=1 --with-suitesparse-dir=/opt/miniforge/envs/claw --with-x=0 --with-scalar-type=real --with-cuda=0 --prefix=/opt/miniforge/envs/claw [0]PETSC ERROR: #1 PetscShmgetMapAddresses() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/sys/utils/server.c:114 [0]PETSC ERROR: #2 PCMPISetMat() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/pc/impls/mpi/pcmpi.c:269 [0]PETSC ERROR: #3 PCSetUp_MPI() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/pc/impls/mpi/pcmpi.c:853 [0]PETSC ERROR: #4 PCSetUp() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/pc/interface/precon.c:1071 [0]PETSC ERROR: #5 KSPSetUp() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/ksp/interface/itfunc.c:415 [0]PETSC ERROR: #6 KSPSolve_Private() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/ksp/interface/itfunc.c:826 [0]PETSC ERROR: #7 KSPSolve() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/ksp/interface/itfunc.c:1075 From cpraveen at gmail.com Wed Oct 23 07:23:49 2024 From: cpraveen at gmail.com (Praveen C) Date: Wed, 23 Oct 2024 17:53:49 +0530 Subject: [petsc-users] Problem running geoclaw example with petsc 3.22 Message-ID: <1C63D19F-6EBC-43A3-8B40-65EA34991AFA@gmail.com> Dear all I am not able to run the boussinesq example from geoclaw using petsc at 3.22.0 https://urldefense.us/v3/__https://github.com/clawpack/geoclaw/tree/3303883f46572c58130d161986b8a87a57ca7816/examples/bouss__;!!G_uCfscf7eWS!e3VQ4NHKmXGstRsQW5vtI7fmKfUT9zmJkMJcPbcvPyIjicyfJpNoMgx3wZ-qyGcKNSjIkNZkzilec8MnHN6PMw$ It runs with petsc at 3.21.6 The error I get is given below. After printing this, the code does not progress. I use the following petsc options # set min numbers of matrix rows per MPI rank (default is 10000) -mpi_linear_solve_minimum_count_per_rank 5000 # Krylov linear solver: -mpi_linear_solver_server -mpi_linear_solver_server_view -ksp_type gmres -ksp_max_it 200 -ksp_reuse_preconditioner -ksp_rtol 1.e-9 # preconditioner: -pc_type gamg I installed petsc and other dependencies for clawpack using miniforge. Thanks pc ==> Use Bouss. in water deeper than 1.0000000000000000 Using a PETSc solver Using Bouss equations from the start rnode allocated... node allocated... listOfGrids allocated... Storage allocated... bndList allocated... Gridding level 1 at t = 0.000000E+00: 4 grids with 10000 cells Setting initial dt to 2.9999999999999999E-002 max threads set to 6 Done reading data, starting computation ... Total zeta at initial time: 39269.907650665169 GEOCLAW: Frame 0 output files done at time t = 0.000000D+00 [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Petsc has generated inconsistent data [0]PETSC ERROR: Unable to locate PCMPI allocated shared address 0x55e6d750ae20 [0]PETSC ERROR: WARNING! There are unused option(s) set! Could be the program crashed before usage or a spelling mistake, etc! [0]PETSC ERROR: Option left: name:-ksp_max_it value: 200 source: file [0]PETSC ERROR: Option left: name:-ksp_reuse_preconditioner (no value) source: file [0]PETSC ERROR: Option left: name:-ksp_rtol value: 1.e-9 source: file [0]PETSC ERROR: Option left: name:-ksp_type value: gmres source: file [0]PETSC ERROR: Option left: name:-mpi_linear_solve_minimum_count_per_rank value: 5000 source: file [0]PETSC ERROR: Option left: name:-mpi_linear_solver_server_view (no value) source: file [0]PETSC ERROR: Option left: name:-pc_type value: gamg source: file [0]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!e3VQ4NHKmXGstRsQW5vtI7fmKfUT9zmJkMJcPbcvPyIjicyfJpNoMgx3wZ-qyGcKNSjIkNZkzilec8MvNjNo7A$ for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.22.0, Sep 28, 2024 [0]PETSC ERROR: /home/praveen/bouss/radial_flat/xgeoclaw with 6 MPI process(es) and PETSC_ARCH on euler by praveen Thu Oct 17 21:49:54 2024 [0]PETSC ERROR: Configure options: AR=${PREFIX}/bin/x86_64-conda-linux-gnu-ar CC=mpicc CXX=mpicxx FC=mpifort CFLAGS="-march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /opt/miniforge/envs/claw/include " CPPFLAGS="-DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /opt/miniforge/envs/claw/include" CXXFLAGS="-fvisibility-inlines-hidden -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /opt/miniforge/envs/claw/include " FFLAGS="-march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /opt/miniforge/envs/claw/include -Wl,--no-as-needed" LDFLAGS="-pthread -fopenmp -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,--allow-shlib-undefined -Wl,-rpath,/opt/miniforge/envs/claw/lib -Wl,-rpath-link,/opt/miniforge/envs/claw/lib -L/opt/miniforge/envs/claw/lib -Wl,-rpath-link,/opt/miniforge/envs/claw/lib" LIBS="-Wl,-rpath,/opt/miniforge/envs/claw/lib -lmpi_mpifh -lgfortran" --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 --FOPTFLAGS=-O3 --with-clib-autodetect=0 --with-cxxlib-autodetect=0 --with-fortranlib-autodetect=0 --with-debugging=0 --with-blas-lib=libblas.so --with-lapack-lib=liblapack.so --with-yaml=1 --with-hdf5=1 --with-fftw=1 --with-hwloc=0 --with-hypre=1 --with-metis=1 --with-mpi=1 --with-mumps=1 --with-parmetis=1 --with-pthread=1 --with-ptscotch=1 --with-shared-libraries --with-ssl=0 --with-scalapack=1 --with-superlu=1 --with-superlu_dist=1 --with-superlu_dist-include=/opt/miniforge/envs/claw/include/superlu-dist --with-superlu_dist-lib=-lsuperlu_dist --with-suitesparse=1 --with-suitesparse-dir=/opt/miniforge/envs/claw --with-x=0 --with-scalar-type=real --with-cuda=0 --prefix=/opt/miniforge/envs/claw [0]PETSC ERROR: #1 PetscShmgetMapAddresses() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/sys/utils/server.c:114 [0]PETSC ERROR: #2 PCMPISetMat() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/pc/impls/mpi/pcmpi.c:269 [0]PETSC ERROR: #3 PCSetUp_MPI() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/pc/impls/mpi/pcmpi.c:853 [0]PETSC ERROR: #4 PCSetUp() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/pc/interface/precon.c:1071 [0]PETSC ERROR: #5 KSPSetUp() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/ksp/interface/itfunc.c:415 [0]PETSC ERROR: #6 KSPSolve_Private() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/ksp/interface/itfunc.c:826 [0]PETSC ERROR: #7 KSPSolve() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/ksp/interface/itfunc.c:1075 From bsmith at petsc.dev Wed Oct 23 08:38:36 2024 From: bsmith at petsc.dev (Barry Smith) Date: Wed, 23 Oct 2024 09:38:36 -0400 Subject: [petsc-users] Problem running geoclaw example with petsc 3.22 In-Reply-To: <1C63D19F-6EBC-43A3-8B40-65EA34991AFA@gmail.com> References: <1C63D19F-6EBC-43A3-8B40-65EA34991AFA@gmail.com> Message-ID: Please take a look at the notes in https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/PetscShmgetAllocateArray/__;!!G_uCfscf7eWS!fTBuqCCGHRprXT35JlP-Fm7YsLqOM33bEFcdtlsJEH1q0worVtAiE6xo-LviQiuClRi9lhg4z48P1hVdAp7sD5E$ For some reason your program is not able to access/use the Unix shared memory; check if you are already using the shared memory (so it is not available for a new run) or the limits are too low to access enough memory. Barry > On Oct 23, 2024, at 8:23?AM, Praveen C wrote: > > Dear all > > I am not able to run the boussinesq example from geoclaw using petsc at 3.22.0 > > https://urldefense.us/v3/__https://github.com/clawpack/geoclaw/tree/3303883f46572c58130d161986b8a87a57ca7816/examples/bouss__;!!G_uCfscf7eWS!e3VQ4NHKmXGstRsQW5vtI7fmKfUT9zmJkMJcPbcvPyIjicyfJpNoMgx3wZ-qyGcKNSjIkNZkzilec8MnHN6PMw$ > > It runs with petsc at 3.21.6 > > The error I get is given below. After printing this, the code does not progress. > > I use the following petsc options > > # set min numbers of matrix rows per MPI rank (default is 10000) > -mpi_linear_solve_minimum_count_per_rank 5000 > > > # Krylov linear solver: > -mpi_linear_solver_server > -mpi_linear_solver_server_view > -ksp_type gmres > -ksp_max_it 200 > -ksp_reuse_preconditioner > -ksp_rtol 1.e-9 > > # preconditioner: > -pc_type gamg > > I installed petsc and other dependencies for clawpack using miniforge. > > Thanks > pc > > ==> Use Bouss. in water deeper than 1.0000000000000000 Using a PETSc solver > Using Bouss equations from the start > rnode allocated... > node allocated... > listOfGrids allocated... > Storage allocated... > bndList allocated... > Gridding level 1 at t = 0.000000E+00: 4 grids with 10000 cells > Setting initial dt to 2.9999999999999999E-002 > max threads set to 6 > Done reading data, starting computation ... Total zeta at initial time: 39269.907650665169 GEOCLAW: Frame 0 output files done at time t = 0.000000D+00 > > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [0]PETSC ERROR: Petsc has generated inconsistent data > [0]PETSC ERROR: Unable to locate PCMPI allocated shared address 0x55e6d750ae20 > [0]PETSC ERROR: WARNING! There are unused option(s) set! Could be the program crashed before usage or a spelling mistake, etc! > [0]PETSC ERROR: Option left: name:-ksp_max_it value: 200 source: file > [0]PETSC ERROR: Option left: name:-ksp_reuse_preconditioner (no value) source: file > [0]PETSC ERROR: Option left: name:-ksp_rtol value: 1.e-9 source: file > [0]PETSC ERROR: Option left: name:-ksp_type value: gmres source: file > [0]PETSC ERROR: Option left: name:-mpi_linear_solve_minimum_count_per_rank value: 5000 source: file > [0]PETSC ERROR: Option left: name:-mpi_linear_solver_server_view (no value) source: file > [0]PETSC ERROR: Option left: name:-pc_type value: gamg source: file > [0]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!e3VQ4NHKmXGstRsQW5vtI7fmKfUT9zmJkMJcPbcvPyIjicyfJpNoMgx3wZ-qyGcKNSjIkNZkzilec8MvNjNo7A$ for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.22.0, Sep 28, 2024 [0]PETSC ERROR: /home/praveen/bouss/radial_flat/xgeoclaw with 6 MPI process(es) and PETSC_ARCH on euler by praveen Thu Oct 17 21:49:54 2024 > [0]PETSC ERROR: Configure options: AR=${PREFIX}/bin/x86_64-conda-linux-gnu-ar CC=mpicc CXX=mpicxx FC=mpifort CFLAGS="-march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /opt/miniforge/envs/claw/include " CPPFLAGS="-DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /opt/miniforge/envs/claw/include" CXXFLAGS="-fvisibility-inlines-hidden -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /opt/miniforge/envs/claw/include " FFLAGS="-march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /opt/miniforge/envs/claw/include -Wl,--no-as-needed" LDFLAGS="-pthread -fopenmp -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,--allow-shlib-undefined -Wl,-rpath,/opt/miniforge/envs/claw/lib -Wl,-rpath-link,/opt/miniforge/envs/claw/lib -L/opt/miniforge/envs/claw/lib -Wl,-rpath-link,/opt/miniforge/envs/claw/lib" LIBS="-Wl,-rpath,/opt/miniforge/envs/claw/lib -lmpi_mpifh -lgfortran" --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 --FOPTFLAGS=-O3 --with-clib-autodetect=0 --with-cxxlib-autodetect=0 --with-fortranlib-autodetect=0 --with-debugging=0 --with-blas-lib=libblas.so --with-lapack-lib=liblapack.so --with-yaml=1 --with-hdf5=1 --with-fftw=1 --with-hwloc=0 --with-hypre=1 --with-metis=1 --with-mpi=1 --with-mumps=1 --with-parmetis=1 --with-pthread=1 --with-ptscotch=1 --with-shared-libraries --with-ssl=0 --with-scalapack=1 --with-superlu=1 --with-superlu_dist=1 --with-superlu_dist-include=/opt/miniforge/envs/claw/include/superlu-dist --with-superlu_dist-lib=-lsuperlu_dist --with-suitesparse=1 --with-suitesparse-dir=/opt/miniforge/envs/claw --with-x=0 --with-scalar-type=real --with-cuda=0 --prefix=/opt/miniforge/envs/claw > [0]PETSC ERROR: #1 PetscShmgetMapAddresses() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/sys/utils/server.c:114 > [0]PETSC ERROR: #2 PCMPISetMat() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/pc/impls/mpi/pcmpi.c:269 > [0]PETSC ERROR: #3 PCSetUp_MPI() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/pc/impls/mpi/pcmpi.c:853 > [0]PETSC ERROR: #4 PCSetUp() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/pc/interface/precon.c:1071 > [0]PETSC ERROR: #5 KSPSetUp() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/ksp/interface/itfunc.c:415 > [0]PETSC ERROR: #6 KSPSolve_Private() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/ksp/interface/itfunc.c:826 > [0]PETSC ERROR: #7 KSPSolve() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/ksp/interface/itfunc.c:1075 -------------- next part -------------- An HTML attachment was scrubbed... URL: From cpraveen at gmail.com Wed Oct 23 09:20:13 2024 From: cpraveen at gmail.com (Praveen C) Date: Wed, 23 Oct 2024 19:50:13 +0530 Subject: [petsc-users] Problem running geoclaw example with petsc 3.22 In-Reply-To: References: <1C63D19F-6EBC-43A3-8B40-65EA34991AFA@gmail.com> Message-ID: Hello Barry I see this $ ipcs -m ------ Shared Memory Segments -------- key shmid owner perms bytes nattch status 0x0000000a 11 praveen 666 240 6 and I am observing same error as below. Thanks praveen > On 23 Oct 2024, at 7:08?PM, Barry Smith wrote: > > > Please take a look at the notes in https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/PetscShmgetAllocateArray/__;!!G_uCfscf7eWS!aV1t0ykXK5XZXD7-BJX9A_zGNHNR8THmsF66BnZkVDiDJF0FKfSG354RNSvXarj4iea2kTc7lgzUzX55rI7bsw$ For some reason your program is not able to access/use the Unix shared memory; check if you are already using the shared memory (so it is not available for a new run) or the limits are too low to access enough memory. > > Barry > >> On Oct 23, 2024, at 8:23?AM, Praveen C wrote: >> >> Dear all >> >> I am not able to run the boussinesq example from geoclaw using petsc at 3.22.0 >> >> https://urldefense.us/v3/__https://github.com/clawpack/geoclaw/tree/3303883f46572c58130d161986b8a87a57ca7816/examples/bouss__;!!G_uCfscf7eWS!e3VQ4NHKmXGstRsQW5vtI7fmKfUT9zmJkMJcPbcvPyIjicyfJpNoMgx3wZ-qyGcKNSjIkNZkzilec8MnHN6PMw$ >> >> It runs with petsc at 3.21.6 >> >> The error I get is given below. After printing this, the code does not progress. >> >> I use the following petsc options >> >> # set min numbers of matrix rows per MPI rank (default is 10000) >> -mpi_linear_solve_minimum_count_per_rank 5000 >> >> >> # Krylov linear solver: >> -mpi_linear_solver_server >> -mpi_linear_solver_server_view >> -ksp_type gmres >> -ksp_max_it 200 >> -ksp_reuse_preconditioner >> -ksp_rtol 1.e-9 >> >> # preconditioner: >> -pc_type gamg >> >> I installed petsc and other dependencies for clawpack using miniforge. >> >> Thanks >> pc >> >> ==> Use Bouss. in water deeper than 1.0000000000000000 Using a PETSc solver >> Using Bouss equations from the start >> rnode allocated... >> node allocated... >> listOfGrids allocated... >> Storage allocated... >> bndList allocated... >> Gridding level 1 at t = 0.000000E+00: 4 grids with 10000 cells >> Setting initial dt to 2.9999999999999999E-002 >> max threads set to 6 >> Done reading data, starting computation ... Total zeta at initial time: 39269.907650665169 GEOCLAW: Frame 0 output files done at time t = 0.000000D+00 >> >> [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >> [0]PETSC ERROR: Petsc has generated inconsistent data >> [0]PETSC ERROR: Unable to locate PCMPI allocated shared address 0x55e6d750ae20 >> [0]PETSC ERROR: WARNING! There are unused option(s) set! Could be the program crashed before usage or a spelling mistake, etc! >> [0]PETSC ERROR: Option left: name:-ksp_max_it value: 200 source: file >> [0]PETSC ERROR: Option left: name:-ksp_reuse_preconditioner (no value) source: file >> [0]PETSC ERROR: Option left: name:-ksp_rtol value: 1.e-9 source: file >> [0]PETSC ERROR: Option left: name:-ksp_type value: gmres source: file >> [0]PETSC ERROR: Option left: name:-mpi_linear_solve_minimum_count_per_rank value: 5000 source: file >> [0]PETSC ERROR: Option left: name:-mpi_linear_solver_server_view (no value) source: file >> [0]PETSC ERROR: Option left: name:-pc_type value: gamg source: file >> [0]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!e3VQ4NHKmXGstRsQW5vtI7fmKfUT9zmJkMJcPbcvPyIjicyfJpNoMgx3wZ-qyGcKNSjIkNZkzilec8MvNjNo7A$ for trouble shooting. >> [0]PETSC ERROR: Petsc Release Version 3.22.0, Sep 28, 2024 [0]PETSC ERROR: /home/praveen/bouss/radial_flat/xgeoclaw with 6 MPI process(es) and PETSC_ARCH on euler by praveen Thu Oct 17 21:49:54 2024 >> [0]PETSC ERROR: Configure options: AR=${PREFIX}/bin/x86_64-conda-linux-gnu-ar CC=mpicc CXX=mpicxx FC=mpifort CFLAGS="-march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /opt/miniforge/envs/claw/include " CPPFLAGS="-DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /opt/miniforge/envs/claw/include" CXXFLAGS="-fvisibility-inlines-hidden -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /opt/miniforge/envs/claw/include " FFLAGS="-march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /opt/miniforge/envs/claw/include -Wl,--no-as-needed" LDFLAGS="-pthread -fopenmp -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,--allow-shlib-undefined -Wl,-rpath,/opt/miniforge/envs/claw/lib -Wl,-rpath-link,/opt/miniforge/envs/claw/lib -L/opt/miniforge/envs/claw/lib -Wl,-rpath-link,/opt/miniforge/envs/claw/lib" LIBS="-Wl,-rpath,/opt/miniforge/envs/claw/lib -lmpi_mpifh -lgfortran" --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 --FOPTFLAGS=-O3 --with-clib-autodetect=0 --with-cxxlib-autodetect=0 --with-fortranlib-autodetect=0 --with-debugging=0 --with-blas-lib=libblas.so --with-lapack-lib=liblapack.so --with-yaml=1 --with-hdf5=1 --with-fftw=1 --with-hwloc=0 --with-hypre=1 --with-metis=1 --with-mpi=1 --with-mumps=1 --with-parmetis=1 --with-pthread=1 --with-ptscotch=1 --with-shared-libraries --with-ssl=0 --with-scalapack=1 --with-superlu=1 --with-superlu_dist=1 --with-superlu_dist-include=/opt/miniforge/envs/claw/include/superlu-dist --with-superlu_dist-lib=-lsuperlu_dist --with-suitesparse=1 --with-suitesparse-dir=/opt/miniforge/envs/claw --with-x=0 --with-scalar-type=real --with-cuda=0 --prefix=/opt/miniforge/envs/claw >> [0]PETSC ERROR: #1 PetscShmgetMapAddresses() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/sys/utils/server.c:114 >> [0]PETSC ERROR: #2 PCMPISetMat() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/pc/impls/mpi/pcmpi.c:269 >> [0]PETSC ERROR: #3 PCSetUp_MPI() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/pc/impls/mpi/pcmpi.c:853 >> [0]PETSC ERROR: #4 PCSetUp() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/pc/interface/precon.c:1071 >> [0]PETSC ERROR: #5 KSPSetUp() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/ksp/interface/itfunc.c:415 >> [0]PETSC ERROR: #6 KSPSolve_Private() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/ksp/interface/itfunc.c:826 >> [0]PETSC ERROR: #7 KSPSolve() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/ksp/interface/itfunc.c:1075 > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Wed Oct 23 09:26:21 2024 From: bsmith at petsc.dev (Barry Smith) Date: Wed, 23 Oct 2024 10:26:21 -0400 Subject: [petsc-users] Problem running geoclaw example with petsc 3.22 In-Reply-To: References: <1C63D19F-6EBC-43A3-8B40-65EA34991AFA@gmail.com> Message-ID: Try ipcrm -m 11 ipcs -m Try running the program again If failed check ipcs -m again > On Oct 23, 2024, at 10:20?AM, Praveen C wrote: > > Hello Barry > > I see this > > $ ipcs -m > > ------ Shared Memory Segments -------- > key shmid owner perms bytes nattch status > 0x0000000a 11 praveen 666 240 6 > > and I am observing same error as below. > > Thanks > praveen > >> On 23 Oct 2024, at 7:08?PM, Barry Smith wrote: >> >> >> Please take a look at the notes in https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/PetscShmgetAllocateArray/__;!!G_uCfscf7eWS!aFrFZtkivHqm4h8CCjiTYRVosesIL2-ZKUsQYJczi7-l8ZEO2G1Hnn1WpIl76gLAKbAzDIJeAaKXAudi5UARJTU$ For some reason your program is not able to access/use the Unix shared memory; check if you are already using the shared memory (so it is not available for a new run) or the limits are too low to access enough memory. >> >> Barry >> >>> On Oct 23, 2024, at 8:23?AM, Praveen C wrote: >>> >>> Dear all >>> >>> I am not able to run the boussinesq example from geoclaw using petsc at 3.22.0 >>> >>> https://urldefense.us/v3/__https://github.com/clawpack/geoclaw/tree/3303883f46572c58130d161986b8a87a57ca7816/examples/bouss__;!!G_uCfscf7eWS!e3VQ4NHKmXGstRsQW5vtI7fmKfUT9zmJkMJcPbcvPyIjicyfJpNoMgx3wZ-qyGcKNSjIkNZkzilec8MnHN6PMw$ >>> >>> It runs with petsc at 3.21.6 >>> >>> The error I get is given below. After printing this, the code does not progress. >>> >>> I use the following petsc options >>> >>> # set min numbers of matrix rows per MPI rank (default is 10000) >>> -mpi_linear_solve_minimum_count_per_rank 5000 >>> >>> >>> # Krylov linear solver: >>> -mpi_linear_solver_server >>> -mpi_linear_solver_server_view >>> -ksp_type gmres >>> -ksp_max_it 200 >>> -ksp_reuse_preconditioner >>> -ksp_rtol 1.e-9 >>> >>> # preconditioner: >>> -pc_type gamg >>> >>> I installed petsc and other dependencies for clawpack using miniforge. >>> >>> Thanks >>> pc >>> >>> ==> Use Bouss. in water deeper than 1.0000000000000000 Using a PETSc solver >>> Using Bouss equations from the start >>> rnode allocated... >>> node allocated... >>> listOfGrids allocated... >>> Storage allocated... >>> bndList allocated... >>> Gridding level 1 at t = 0.000000E+00: 4 grids with 10000 cells >>> Setting initial dt to 2.9999999999999999E-002 >>> max threads set to 6 >>> Done reading data, starting computation ... Total zeta at initial time: 39269.907650665169 GEOCLAW: Frame 0 output files done at time t = 0.000000D+00 >>> >>> [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >>> [0]PETSC ERROR: Petsc has generated inconsistent data >>> [0]PETSC ERROR: Unable to locate PCMPI allocated shared address 0x55e6d750ae20 >>> [0]PETSC ERROR: WARNING! There are unused option(s) set! Could be the program crashed before usage or a spelling mistake, etc! >>> [0]PETSC ERROR: Option left: name:-ksp_max_it value: 200 source: file >>> [0]PETSC ERROR: Option left: name:-ksp_reuse_preconditioner (no value) source: file >>> [0]PETSC ERROR: Option left: name:-ksp_rtol value: 1.e-9 source: file >>> [0]PETSC ERROR: Option left: name:-ksp_type value: gmres source: file >>> [0]PETSC ERROR: Option left: name:-mpi_linear_solve_minimum_count_per_rank value: 5000 source: file >>> [0]PETSC ERROR: Option left: name:-mpi_linear_solver_server_view (no value) source: file >>> [0]PETSC ERROR: Option left: name:-pc_type value: gamg source: file >>> [0]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!e3VQ4NHKmXGstRsQW5vtI7fmKfUT9zmJkMJcPbcvPyIjicyfJpNoMgx3wZ-qyGcKNSjIkNZkzilec8MvNjNo7A$ for trouble shooting. >>> [0]PETSC ERROR: Petsc Release Version 3.22.0, Sep 28, 2024 [0]PETSC ERROR: /home/praveen/bouss/radial_flat/xgeoclaw with 6 MPI process(es) and PETSC_ARCH on euler by praveen Thu Oct 17 21:49:54 2024 >>> [0]PETSC ERROR: Configure options: AR=${PREFIX}/bin/x86_64-conda-linux-gnu-ar CC=mpicc CXX=mpicxx FC=mpifort CFLAGS="-march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /opt/miniforge/envs/claw/include " CPPFLAGS="-DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /opt/miniforge/envs/claw/include" CXXFLAGS="-fvisibility-inlines-hidden -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /opt/miniforge/envs/claw/include " FFLAGS="-march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /opt/miniforge/envs/claw/include -Wl,--no-as-needed" LDFLAGS="-pthread -fopenmp -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,--allow-shlib-undefined -Wl,-rpath,/opt/miniforge/envs/claw/lib -Wl,-rpath-link,/opt/miniforge/envs/claw/lib -L/opt/miniforge/envs/claw/lib -Wl,-rpath-link,/opt/miniforge/envs/claw/lib" LIBS="-Wl,-rpath,/opt/miniforge/envs/claw/lib -lmpi_mpifh -lgfortran" --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 --FOPTFLAGS=-O3 --with-clib-autodetect=0 --with-cxxlib-autodetect=0 --with-fortranlib-autodetect=0 --with-debugging=0 --with-blas-lib=libblas.so --with-lapack-lib=liblapack.so --with-yaml=1 --with-hdf5=1 --with-fftw=1 --with-hwloc=0 --with-hypre=1 --with-metis=1 --with-mpi=1 --with-mumps=1 --with-parmetis=1 --with-pthread=1 --with-ptscotch=1 --with-shared-libraries --with-ssl=0 --with-scalapack=1 --with-superlu=1 --with-superlu_dist=1 --with-superlu_dist-include=/opt/miniforge/envs/claw/include/superlu-dist --with-superlu_dist-lib=-lsuperlu_dist --with-suitesparse=1 --with-suitesparse-dir=/opt/miniforge/envs/claw --with-x=0 --with-scalar-type=real --with-cuda=0 --prefix=/opt/miniforge/envs/claw >>> [0]PETSC ERROR: #1 PetscShmgetMapAddresses() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/sys/utils/server.c:114 >>> [0]PETSC ERROR: #2 PCMPISetMat() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/pc/impls/mpi/pcmpi.c:269 >>> [0]PETSC ERROR: #3 PCSetUp_MPI() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/pc/impls/mpi/pcmpi.c:853 >>> [0]PETSC ERROR: #4 PCSetUp() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/pc/interface/precon.c:1071 >>> [0]PETSC ERROR: #5 KSPSetUp() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/ksp/interface/itfunc.c:415 >>> [0]PETSC ERROR: #6 KSPSolve_Private() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/ksp/interface/itfunc.c:826 >>> [0]PETSC ERROR: #7 KSPSolve() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/ksp/interface/itfunc.c:1075 >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cpraveen at gmail.com Wed Oct 23 09:31:00 2024 From: cpraveen at gmail.com (Praveen C) Date: Wed, 23 Oct 2024 20:01:00 +0530 Subject: [petsc-users] Problem running geoclaw example with petsc 3.22 In-Reply-To: References: <1C63D19F-6EBC-43A3-8B40-65EA34991AFA@gmail.com> Message-ID: <0136027D-C2CE-47D0-AF97-ACDE1DA0C382@gmail.com> I get same error and now it shows $ ipcs -m ------ Shared Memory Segments -------- key shmid owner perms bytes nattch status 0x0000000a 32788 praveen 666 240 6 Note that the code seems to be still running after printing those error message, but it is not printing any progress which it should do. Thanks praveen > On 23 Oct 2024, at 7:56?PM, Barry Smith wrote: > > > Try > > ipcrm -m 11 > > ipcs -m > > Try running the program again > > If failed check > > ipcs -m > > again > > > >> On Oct 23, 2024, at 10:20?AM, Praveen C wrote: >> >> Hello Barry >> >> I see this >> >> $ ipcs -m >> >> ------ Shared Memory Segments -------- >> key shmid owner perms bytes nattch status >> 0x0000000a 11 praveen 666 240 6 >> >> and I am observing same error as below. >> >> Thanks >> praveen >> >>> On 23 Oct 2024, at 7:08?PM, Barry Smith wrote: >>> >>> >>> Please take a look at the notes in https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/PetscShmgetAllocateArray/__;!!G_uCfscf7eWS!fpsbNP5z6XCWQ6tD3oyfDrMPZn-6CqXz9A-Lqv4A0-ss21oyjEdcRNGSEw8Kl1XFP9QBS7Iz3yheqkQrhkRkEg$ For some reason your program is not able to access/use the Unix shared memory; check if you are already using the shared memory (so it is not available for a new run) or the limits are too low to access enough memory. >>> >>> Barry >>> >>>> On Oct 23, 2024, at 8:23?AM, Praveen C wrote: >>>> >>>> Dear all >>>> >>>> I am not able to run the boussinesq example from geoclaw using petsc at 3.22.0 >>>> >>>> https://urldefense.us/v3/__https://github.com/clawpack/geoclaw/tree/3303883f46572c58130d161986b8a87a57ca7816/examples/bouss__;!!G_uCfscf7eWS!e3VQ4NHKmXGstRsQW5vtI7fmKfUT9zmJkMJcPbcvPyIjicyfJpNoMgx3wZ-qyGcKNSjIkNZkzilec8MnHN6PMw$ >>>> >>>> It runs with petsc at 3.21.6 >>>> >>>> The error I get is given below. After printing this, the code does not progress. >>>> >>>> I use the following petsc options >>>> >>>> # set min numbers of matrix rows per MPI rank (default is 10000) >>>> -mpi_linear_solve_minimum_count_per_rank 5000 >>>> >>>> >>>> # Krylov linear solver: >>>> -mpi_linear_solver_server >>>> -mpi_linear_solver_server_view >>>> -ksp_type gmres >>>> -ksp_max_it 200 >>>> -ksp_reuse_preconditioner >>>> -ksp_rtol 1.e-9 >>>> >>>> # preconditioner: >>>> -pc_type gamg >>>> >>>> I installed petsc and other dependencies for clawpack using miniforge. >>>> >>>> Thanks >>>> pc >>>> >>>> ==> Use Bouss. in water deeper than 1.0000000000000000 Using a PETSc solver >>>> Using Bouss equations from the start >>>> rnode allocated... >>>> node allocated... >>>> listOfGrids allocated... >>>> Storage allocated... >>>> bndList allocated... >>>> Gridding level 1 at t = 0.000000E+00: 4 grids with 10000 cells >>>> Setting initial dt to 2.9999999999999999E-002 >>>> max threads set to 6 >>>> Done reading data, starting computation ... Total zeta at initial time: 39269.907650665169 GEOCLAW: Frame 0 output files done at time t = 0.000000D+00 >>>> >>>> [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >>>> [0]PETSC ERROR: Petsc has generated inconsistent data >>>> [0]PETSC ERROR: Unable to locate PCMPI allocated shared address 0x55e6d750ae20 >>>> [0]PETSC ERROR: WARNING! There are unused option(s) set! Could be the program crashed before usage or a spelling mistake, etc! >>>> [0]PETSC ERROR: Option left: name:-ksp_max_it value: 200 source: file >>>> [0]PETSC ERROR: Option left: name:-ksp_reuse_preconditioner (no value) source: file >>>> [0]PETSC ERROR: Option left: name:-ksp_rtol value: 1.e-9 source: file >>>> [0]PETSC ERROR: Option left: name:-ksp_type value: gmres source: file >>>> [0]PETSC ERROR: Option left: name:-mpi_linear_solve_minimum_count_per_rank value: 5000 source: file >>>> [0]PETSC ERROR: Option left: name:-mpi_linear_solver_server_view (no value) source: file >>>> [0]PETSC ERROR: Option left: name:-pc_type value: gamg source: file >>>> [0]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!e3VQ4NHKmXGstRsQW5vtI7fmKfUT9zmJkMJcPbcvPyIjicyfJpNoMgx3wZ-qyGcKNSjIkNZkzilec8MvNjNo7A$ for trouble shooting. >>>> [0]PETSC ERROR: Petsc Release Version 3.22.0, Sep 28, 2024 [0]PETSC ERROR: /home/praveen/bouss/radial_flat/xgeoclaw with 6 MPI process(es) and PETSC_ARCH on euler by praveen Thu Oct 17 21:49:54 2024 >>>> [0]PETSC ERROR: Configure options: AR=${PREFIX}/bin/x86_64-conda-linux-gnu-ar CC=mpicc CXX=mpicxx FC=mpifort CFLAGS="-march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /opt/miniforge/envs/claw/include " CPPFLAGS="-DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /opt/miniforge/envs/claw/include" CXXFLAGS="-fvisibility-inlines-hidden -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /opt/miniforge/envs/claw/include " FFLAGS="-march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /opt/miniforge/envs/claw/include -Wl,--no-as-needed" LDFLAGS="-pthread -fopenmp -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,--allow-shlib-undefined -Wl,-rpath,/opt/miniforge/envs/claw/lib -Wl,-rpath-link,/opt/miniforge/envs/claw/lib -L/opt/miniforge/envs/claw/lib -Wl,-rpath-link,/opt/miniforge/envs/claw/lib" LIBS="-Wl,-rpath,/opt/miniforge/envs/claw/lib -lmpi_mpifh -lgfortran" --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 --FOPTFLAGS=-O3 --with-clib-autodetect=0 --with-cxxlib-autodetect=0 --with-fortranlib-autodetect=0 --with-debugging=0 --with-blas-lib=libblas.so --with-lapack-lib=liblapack.so --with-yaml=1 --with-hdf5=1 --with-fftw=1 --with-hwloc=0 --with-hypre=1 --with-metis=1 --with-mpi=1 --with-mumps=1 --with-parmetis=1 --with-pthread=1 --with-ptscotch=1 --with-shared-libraries --with-ssl=0 --with-scalapack=1 --with-superlu=1 --with-superlu_dist=1 --with-superlu_dist-include=/opt/miniforge/envs/claw/include/superlu-dist --with-superlu_dist-lib=-lsuperlu_dist --with-suitesparse=1 --with-suitesparse-dir=/opt/miniforge/envs/claw --with-x=0 --with-scalar-type=real --with-cuda=0 --prefix=/opt/miniforge/envs/claw >>>> [0]PETSC ERROR: #1 PetscShmgetMapAddresses() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/sys/utils/server.c:114 >>>> [0]PETSC ERROR: #2 PCMPISetMat() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/pc/impls/mpi/pcmpi.c:269 >>>> [0]PETSC ERROR: #3 PCSetUp_MPI() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/pc/impls/mpi/pcmpi.c:853 >>>> [0]PETSC ERROR: #4 PCSetUp() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/pc/interface/precon.c:1071 >>>> [0]PETSC ERROR: #5 KSPSetUp() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/ksp/interface/itfunc.c:415 >>>> [0]PETSC ERROR: #6 KSPSolve_Private() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/ksp/interface/itfunc.c:826 >>>> [0]PETSC ERROR: #7 KSPSolve() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/ksp/interface/itfunc.c:1075 >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Wed Oct 23 09:56:14 2024 From: bsmith at petsc.dev (Barry Smith) Date: Wed, 23 Oct 2024 10:56:14 -0400 Subject: [petsc-users] Problem running geoclaw example with petsc 3.22 In-Reply-To: <0136027D-C2CE-47D0-AF97-ACDE1DA0C382@gmail.com> References: <1C63D19F-6EBC-43A3-8B40-65EA34991AFA@gmail.com> <0136027D-C2CE-47D0-AF97-ACDE1DA0C382@gmail.com> Message-ID: <4726A341-54DB-4592-BBAE-843AF271F28E@petsc.dev> Hmm, so it is creating the first shared memory region with ID of 10 (A in hex) puts it in a linked list in PETSc but then when it tries to find it in the linked list it cannot find it. I don't know how to reproduce this or debug it remotely. Can you build on a completely different machine or with completely different compilers? Barry > On Oct 23, 2024, at 10:31?AM, Praveen C wrote: > > I get same error and now it shows > > $ ipcs -m > > ------ Shared Memory Segments -------- > key shmid owner perms bytes nattch status > 0x0000000a 32788 praveen 666 240 6 > > Note that the code seems to be still running after printing those error message, but it is not printing any progress which it should do. > > Thanks > praveen > >> On 23 Oct 2024, at 7:56?PM, Barry Smith wrote: >> >> >> Try >> >> ipcrm -m 11 >> >> ipcs -m >> >> Try running the program again >> >> If failed check >> >> ipcs -m >> >> again >> >> >> >>> On Oct 23, 2024, at 10:20?AM, Praveen C wrote: >>> >>> Hello Barry >>> >>> I see this >>> >>> $ ipcs -m >>> >>> ------ Shared Memory Segments -------- >>> key shmid owner perms bytes nattch status >>> 0x0000000a 11 praveen 666 240 6 >>> >>> and I am observing same error as below. >>> >>> Thanks >>> praveen >>> >>>> On 23 Oct 2024, at 7:08?PM, Barry Smith wrote: >>>> >>>> >>>> Please take a look at the notes in https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/PetscShmgetAllocateArray/__;!!G_uCfscf7eWS!bQF1Ha5L7JIXTnw9qnMNuyiV_XldBWdWhmzhKBu5P99ZZNiosqJx6XbzQab2pfjMrRYRfp2qnm25RDoEeefcAbM$ For some reason your program is not able to access/use the Unix shared memory; check if you are already using the shared memory (so it is not available for a new run) or the limits are too low to access enough memory. >>>> >>>> Barry >>>> >>>>> On Oct 23, 2024, at 8:23?AM, Praveen C wrote: >>>>> >>>>> Dear all >>>>> >>>>> I am not able to run the boussinesq example from geoclaw using petsc at 3.22.0 >>>>> >>>>> https://urldefense.us/v3/__https://github.com/clawpack/geoclaw/tree/3303883f46572c58130d161986b8a87a57ca7816/examples/bouss__;!!G_uCfscf7eWS!e3VQ4NHKmXGstRsQW5vtI7fmKfUT9zmJkMJcPbcvPyIjicyfJpNoMgx3wZ-qyGcKNSjIkNZkzilec8MnHN6PMw$ >>>>> >>>>> It runs with petsc at 3.21.6 >>>>> >>>>> The error I get is given below. After printing this, the code does not progress. >>>>> >>>>> I use the following petsc options >>>>> >>>>> # set min numbers of matrix rows per MPI rank (default is 10000) >>>>> -mpi_linear_solve_minimum_count_per_rank 5000 >>>>> >>>>> >>>>> # Krylov linear solver: >>>>> -mpi_linear_solver_server >>>>> -mpi_linear_solver_server_view >>>>> -ksp_type gmres >>>>> -ksp_max_it 200 >>>>> -ksp_reuse_preconditioner >>>>> -ksp_rtol 1.e-9 >>>>> >>>>> # preconditioner: >>>>> -pc_type gamg >>>>> >>>>> I installed petsc and other dependencies for clawpack using miniforge. >>>>> >>>>> Thanks >>>>> pc >>>>> >>>>> ==> Use Bouss. in water deeper than 1.0000000000000000 Using a PETSc solver >>>>> Using Bouss equations from the start >>>>> rnode allocated... >>>>> node allocated... >>>>> listOfGrids allocated... >>>>> Storage allocated... >>>>> bndList allocated... >>>>> Gridding level 1 at t = 0.000000E+00: 4 grids with 10000 cells >>>>> Setting initial dt to 2.9999999999999999E-002 >>>>> max threads set to 6 >>>>> Done reading data, starting computation ... Total zeta at initial time: 39269.907650665169 GEOCLAW: Frame 0 output files done at time t = 0.000000D+00 >>>>> >>>>> [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >>>>> [0]PETSC ERROR: Petsc has generated inconsistent data >>>>> [0]PETSC ERROR: Unable to locate PCMPI allocated shared address 0x55e6d750ae20 >>>>> [0]PETSC ERROR: WARNING! There are unused option(s) set! Could be the program crashed before usage or a spelling mistake, etc! >>>>> [0]PETSC ERROR: Option left: name:-ksp_max_it value: 200 source: file >>>>> [0]PETSC ERROR: Option left: name:-ksp_reuse_preconditioner (no value) source: file >>>>> [0]PETSC ERROR: Option left: name:-ksp_rtol value: 1.e-9 source: file >>>>> [0]PETSC ERROR: Option left: name:-ksp_type value: gmres source: file >>>>> [0]PETSC ERROR: Option left: name:-mpi_linear_solve_minimum_count_per_rank value: 5000 source: file >>>>> [0]PETSC ERROR: Option left: name:-mpi_linear_solver_server_view (no value) source: file >>>>> [0]PETSC ERROR: Option left: name:-pc_type value: gamg source: file >>>>> [0]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!e3VQ4NHKmXGstRsQW5vtI7fmKfUT9zmJkMJcPbcvPyIjicyfJpNoMgx3wZ-qyGcKNSjIkNZkzilec8MvNjNo7A$ for trouble shooting. >>>>> [0]PETSC ERROR: Petsc Release Version 3.22.0, Sep 28, 2024 [0]PETSC ERROR: /home/praveen/bouss/radial_flat/xgeoclaw with 6 MPI process(es) and PETSC_ARCH on euler by praveen Thu Oct 17 21:49:54 2024 >>>>> [0]PETSC ERROR: Configure options: AR=${PREFIX}/bin/x86_64-conda-linux-gnu-ar CC=mpicc CXX=mpicxx FC=mpifort CFLAGS="-march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /opt/miniforge/envs/claw/include " CPPFLAGS="-DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /opt/miniforge/envs/claw/include" CXXFLAGS="-fvisibility-inlines-hidden -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /opt/miniforge/envs/claw/include " FFLAGS="-march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /opt/miniforge/envs/claw/include -Wl,--no-as-needed" LDFLAGS="-pthread -fopenmp -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,--allow-shlib-undefined -Wl,-rpath,/opt/miniforge/envs/claw/lib -Wl,-rpath-link,/opt/miniforge/envs/claw/lib -L/opt/miniforge/envs/claw/lib -Wl,-rpath-link,/opt/miniforge/envs/claw/lib" LIBS="-Wl,-rpath,/opt/miniforge/envs/claw/lib -lmpi_mpifh -lgfortran" --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 --FOPTFLAGS=-O3 --with-clib-autodetect=0 --with-cxxlib-autodetect=0 --with-fortranlib-autodetect=0 --with-debugging=0 --with-blas-lib=libblas.so --with-lapack-lib=liblapack.so --with-yaml=1 --with-hdf5=1 --with-fftw=1 --with-hwloc=0 --with-hypre=1 --with-metis=1 --with-mpi=1 --with-mumps=1 --with-parmetis=1 --with-pthread=1 --with-ptscotch=1 --with-shared-libraries --with-ssl=0 --with-scalapack=1 --with-superlu=1 --with-superlu_dist=1 --with-superlu_dist-include=/opt/miniforge/envs/claw/include/superlu-dist --with-superlu_dist-lib=-lsuperlu_dist --with-suitesparse=1 --with-suitesparse-dir=/opt/miniforge/envs/claw --with-x=0 --with-scalar-type=real --with-cuda=0 --prefix=/opt/miniforge/envs/claw >>>>> [0]PETSC ERROR: #1 PetscShmgetMapAddresses() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/sys/utils/server.c:114 >>>>> [0]PETSC ERROR: #2 PCMPISetMat() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/pc/impls/mpi/pcmpi.c:269 >>>>> [0]PETSC ERROR: #3 PCSetUp_MPI() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/pc/impls/mpi/pcmpi.c:853 >>>>> [0]PETSC ERROR: #4 PCSetUp() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/pc/interface/precon.c:1071 >>>>> [0]PETSC ERROR: #5 KSPSetUp() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/ksp/interface/itfunc.c:415 >>>>> [0]PETSC ERROR: #6 KSPSolve_Private() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/ksp/interface/itfunc.c:826 >>>>> [0]PETSC ERROR: #7 KSPSolve() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/ksp/interface/itfunc.c:1075 >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Wed Oct 23 09:56:14 2024 From: bsmith at petsc.dev (Barry Smith) Date: Wed, 23 Oct 2024 10:56:14 -0400 Subject: [petsc-users] Problem running geoclaw example with petsc 3.22 In-Reply-To: <0136027D-C2CE-47D0-AF97-ACDE1DA0C382@gmail.com> References: <1C63D19F-6EBC-43A3-8B40-65EA34991AFA@gmail.com> <0136027D-C2CE-47D0-AF97-ACDE1DA0C382@gmail.com> Message-ID: <4726A341-54DB-4592-BBAE-843AF271F28E@petsc.dev> Hmm, so it is creating the first shared memory region with ID of 10 (A in hex) puts it in a linked list in PETSc but then when it tries to find it in the linked list it cannot find it. I don't know how to reproduce this or debug it remotely. Can you build on a completely different machine or with completely different compilers? Barry > On Oct 23, 2024, at 10:31?AM, Praveen C wrote: > > I get same error and now it shows > > $ ipcs -m > > ------ Shared Memory Segments -------- > key shmid owner perms bytes nattch status > 0x0000000a 32788 praveen 666 240 6 > > Note that the code seems to be still running after printing those error message, but it is not printing any progress which it should do. > > Thanks > praveen > >> On 23 Oct 2024, at 7:56?PM, Barry Smith wrote: >> >> >> Try >> >> ipcrm -m 11 >> >> ipcs -m >> >> Try running the program again >> >> If failed check >> >> ipcs -m >> >> again >> >> >> >>> On Oct 23, 2024, at 10:20?AM, Praveen C wrote: >>> >>> Hello Barry >>> >>> I see this >>> >>> $ ipcs -m >>> >>> ------ Shared Memory Segments -------- >>> key shmid owner perms bytes nattch status >>> 0x0000000a 11 praveen 666 240 6 >>> >>> and I am observing same error as below. >>> >>> Thanks >>> praveen >>> >>>> On 23 Oct 2024, at 7:08?PM, Barry Smith wrote: >>>> >>>> >>>> Please take a look at the notes in https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/PetscShmgetAllocateArray/__;!!G_uCfscf7eWS!d8hbkBgHfGN-5jZI0yh7_fRBXUe2m7vGeec0JeTo3GvP69O4CajQgQsIdFO7QSfK4ae5V5L0GOBMjnY4xLoT-2U$ For some reason your program is not able to access/use the Unix shared memory; check if you are already using the shared memory (so it is not available for a new run) or the limits are too low to access enough memory. >>>> >>>> Barry >>>> >>>>> On Oct 23, 2024, at 8:23?AM, Praveen C wrote: >>>>> >>>>> Dear all >>>>> >>>>> I am not able to run the boussinesq example from geoclaw using petsc at 3.22.0 >>>>> >>>>> https://urldefense.us/v3/__https://github.com/clawpack/geoclaw/tree/3303883f46572c58130d161986b8a87a57ca7816/examples/bouss__;!!G_uCfscf7eWS!e3VQ4NHKmXGstRsQW5vtI7fmKfUT9zmJkMJcPbcvPyIjicyfJpNoMgx3wZ-qyGcKNSjIkNZkzilec8MnHN6PMw$ >>>>> >>>>> It runs with petsc at 3.21.6 >>>>> >>>>> The error I get is given below. After printing this, the code does not progress. >>>>> >>>>> I use the following petsc options >>>>> >>>>> # set min numbers of matrix rows per MPI rank (default is 10000) >>>>> -mpi_linear_solve_minimum_count_per_rank 5000 >>>>> >>>>> >>>>> # Krylov linear solver: >>>>> -mpi_linear_solver_server >>>>> -mpi_linear_solver_server_view >>>>> -ksp_type gmres >>>>> -ksp_max_it 200 >>>>> -ksp_reuse_preconditioner >>>>> -ksp_rtol 1.e-9 >>>>> >>>>> # preconditioner: >>>>> -pc_type gamg >>>>> >>>>> I installed petsc and other dependencies for clawpack using miniforge. >>>>> >>>>> Thanks >>>>> pc >>>>> >>>>> ==> Use Bouss. in water deeper than 1.0000000000000000 Using a PETSc solver >>>>> Using Bouss equations from the start >>>>> rnode allocated... >>>>> node allocated... >>>>> listOfGrids allocated... >>>>> Storage allocated... >>>>> bndList allocated... >>>>> Gridding level 1 at t = 0.000000E+00: 4 grids with 10000 cells >>>>> Setting initial dt to 2.9999999999999999E-002 >>>>> max threads set to 6 >>>>> Done reading data, starting computation ... Total zeta at initial time: 39269.907650665169 GEOCLAW: Frame 0 output files done at time t = 0.000000D+00 >>>>> >>>>> [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >>>>> [0]PETSC ERROR: Petsc has generated inconsistent data >>>>> [0]PETSC ERROR: Unable to locate PCMPI allocated shared address 0x55e6d750ae20 >>>>> [0]PETSC ERROR: WARNING! There are unused option(s) set! Could be the program crashed before usage or a spelling mistake, etc! >>>>> [0]PETSC ERROR: Option left: name:-ksp_max_it value: 200 source: file >>>>> [0]PETSC ERROR: Option left: name:-ksp_reuse_preconditioner (no value) source: file >>>>> [0]PETSC ERROR: Option left: name:-ksp_rtol value: 1.e-9 source: file >>>>> [0]PETSC ERROR: Option left: name:-ksp_type value: gmres source: file >>>>> [0]PETSC ERROR: Option left: name:-mpi_linear_solve_minimum_count_per_rank value: 5000 source: file >>>>> [0]PETSC ERROR: Option left: name:-mpi_linear_solver_server_view (no value) source: file >>>>> [0]PETSC ERROR: Option left: name:-pc_type value: gamg source: file >>>>> [0]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!e3VQ4NHKmXGstRsQW5vtI7fmKfUT9zmJkMJcPbcvPyIjicyfJpNoMgx3wZ-qyGcKNSjIkNZkzilec8MvNjNo7A$ for trouble shooting. >>>>> [0]PETSC ERROR: Petsc Release Version 3.22.0, Sep 28, 2024 [0]PETSC ERROR: /home/praveen/bouss/radial_flat/xgeoclaw with 6 MPI process(es) and PETSC_ARCH on euler by praveen Thu Oct 17 21:49:54 2024 >>>>> [0]PETSC ERROR: Configure options: AR=${PREFIX}/bin/x86_64-conda-linux-gnu-ar CC=mpicc CXX=mpicxx FC=mpifort CFLAGS="-march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /opt/miniforge/envs/claw/include " CPPFLAGS="-DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /opt/miniforge/envs/claw/include" CXXFLAGS="-fvisibility-inlines-hidden -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /opt/miniforge/envs/claw/include " FFLAGS="-march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /opt/miniforge/envs/claw/include -Wl,--no-as-needed" LDFLAGS="-pthread -fopenmp -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,--allow-shlib-undefined -Wl,-rpath,/opt/miniforge/envs/claw/lib -Wl,-rpath-link,/opt/miniforge/envs/claw/lib -L/opt/miniforge/envs/claw/lib -Wl,-rpath-link,/opt/miniforge/envs/claw/lib" LIBS="-Wl,-rpath,/opt/miniforge/envs/claw/lib -lmpi_mpifh -lgfortran" --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 --FOPTFLAGS=-O3 --with-clib-autodetect=0 --with-cxxlib-autodetect=0 --with-fortranlib-autodetect=0 --with-debugging=0 --with-blas-lib=libblas.so --with-lapack-lib=liblapack.so --with-yaml=1 --with-hdf5=1 --with-fftw=1 --with-hwloc=0 --with-hypre=1 --with-metis=1 --with-mpi=1 --with-mumps=1 --with-parmetis=1 --with-pthread=1 --with-ptscotch=1 --with-shared-libraries --with-ssl=0 --with-scalapack=1 --with-superlu=1 --with-superlu_dist=1 --with-superlu_dist-include=/opt/miniforge/envs/claw/include/superlu-dist --with-superlu_dist-lib=-lsuperlu_dist --with-suitesparse=1 --with-suitesparse-dir=/opt/miniforge/envs/claw --with-x=0 --with-scalar-type=real --with-cuda=0 --prefix=/opt/miniforge/envs/claw >>>>> [0]PETSC ERROR: #1 PetscShmgetMapAddresses() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/sys/utils/server.c:114 >>>>> [0]PETSC ERROR: #2 PCMPISetMat() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/pc/impls/mpi/pcmpi.c:269 >>>>> [0]PETSC ERROR: #3 PCSetUp_MPI() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/pc/impls/mpi/pcmpi.c:853 >>>>> [0]PETSC ERROR: #4 PCSetUp() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/pc/interface/precon.c:1071 >>>>> [0]PETSC ERROR: #5 KSPSetUp() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/ksp/interface/itfunc.c:415 >>>>> [0]PETSC ERROR: #6 KSPSolve_Private() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/ksp/interface/itfunc.c:826 >>>>> [0]PETSC ERROR: #7 KSPSolve() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/ksp/interface/itfunc.c:1075 >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cpraveen at gmail.com Wed Oct 23 21:55:37 2024 From: cpraveen at gmail.com (Praveen C) Date: Thu, 24 Oct 2024 08:25:37 +0530 Subject: [petsc-users] Problem running geoclaw example with petsc 3.22 In-Reply-To: <4726A341-54DB-4592-BBAE-843AF271F28E@petsc.dev> References: <1C63D19F-6EBC-43A3-8B40-65EA34991AFA@gmail.com> <0136027D-C2CE-47D0-AF97-ACDE1DA0C382@gmail.com> <4726A341-54DB-4592-BBAE-843AF271F28E@petsc.dev> Message-ID: <27B47E2D-00D5-4326-95A9-7595DFF0E952@gmail.com> I get very similar error on my mac with $ gfortran -v Using built-in specs. COLLECT_GCC=gfortran COLLECT_LTO_WRAPPER=/opt/homebrew/Caskroom/miniforge/base/envs/claw/libexec/gcc/arm64-apple-darwin20.0.0/13.2.0/lto-wrapper Target: arm64-apple-darwin20.0.0 Configured with: ../configure --prefix=/opt/homebrew/Caskroom/miniforge/base/envs/claw --build=x86_64-apple-darwin13.4.0 --host=arm64-apple-darwin20.0.0 --target=arm64-apple-darwin20.0.0 --with-libiconv-prefix=/opt/homebrew/Caskroom/miniforge/base/envs/claw --enable-languages=fortran --disable-multilib --enable-checking=release --disable-bootstrap --disable-libssp --with-gmp=/opt/homebrew/Caskroom/miniforge/base/envs/claw --with-mpfr=/opt/homebrew/Caskroom/miniforge/base/envs/claw --with-mpc=/opt/homebrew/Caskroom/miniforge/base/envs/claw --with-isl=/opt/homebrew/Caskroom/miniforge/base/envs/claw --enable-darwin-at-rpath Thread model: posix Supported LTO compression algorithms: zlib gcc version 13.2.0 (GCC) Before starting $ ipcs -m IPC status from as of Thu Oct 24 08:02:11 IST 2024 T ID KEY MODE OWNER GROUP Shared Memory: and when I run the code Using a PETSc solver Using Bouss equations from the start rnode allocated... node allocated... listOfGrids allocated... Storage allocated... bndList allocated... Gridding level 1 at t = 0.000000E+00: 4 grids with 10000 cells Setting initial dt to 2.9999999999999999E-002 max threads set to 1 Done reading data, starting computation ... Total zeta at initial time: 39269.907650665169 GEOCLAW: Frame 0 output files done at time t = 0.000000D+00 [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Petsc has generated inconsistent data [0]PETSC ERROR: Unable to locate PCMPI allocated shared address 0x130698000 [0]PETSC ERROR: WARNING! There are unused option(s) set! Could be the program crashed before usage or a spelling mistake, etc! [0]PETSC ERROR: Option left: name:-ksp_type value: preonly source: file [0]PETSC ERROR: Option left: name:-mpi_ksp_max_it value: 200 source: file [0]PETSC ERROR: Option left: name:-mpi_ksp_reuse_preconditioner (no value) source: file [0]PETSC ERROR: Option left: name:-mpi_ksp_rtol value: 1.e-9 source: file [0]PETSC ERROR: Option left: name:-mpi_ksp_type value: gmres source: file [0]PETSC ERROR: Option left: name:-mpi_linear_solver_server_view (no value) source: file [0]PETSC ERROR: Option left: name:-mpi_pc_gamg_sym_graph value: true source: file [0]PETSC ERROR: Option left: name:-mpi_pc_gamg_symmetrize_graph value: true source: file [0]PETSC ERROR: Option left: name:-mpi_pc_type value: gamg source: file [0]PETSC ERROR: Option left: name:-pc_mpi_minimum_count_per_rank value: 5000 source: file [0]PETSC ERROR: Option left: name:-pc_type value: mpi source: file [0]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!ZGVQF1mXRVL7_W_U2YYltlhjpJnbZse3y51RCYHH0aJtnt9WRtDMxGfCLD_5ua4VLxp0_IMDDAycDKWb2Yr0Pw$ for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.22.0, Sep 28, 2024 [0]PETSC ERROR: /Users/praveen/work/bouss/radial_flat/xgeoclaw with 6 MPI process(es) and PETSC_ARCH on MacMiniHome.local by praveen Thu Oct 24 08:04:27 2024 [0]PETSC ERROR: Configure options: AR=arm64-apple-darwin20.0.0-ar CC=mpicc CXX=mpicxx FC=mpifort CFLAGS="-ftree-vectorize -fPIC -fstack-protector-strong -O2 -pipe -isystem /opt/homebrew/Caskroom/miniforge/base/envs/claw/include " CPPFLAGS="-D_FORTIFY_SOURCE=2 -isystem /opt/homebrew/Caskroom/miniforge/base/envs/claw/include -mmacosx-version-min=11.0 -mmacosx-version-min=11.0" CXXFLAGS="-ftree-vectorize -fPIC -fstack-protector-strong -O2 -pipe -stdlib=libc++ -fvisibility-inlines-hidden -fmessage-length=0 -isystem /opt/homebrew/Caskroom/miniforge/base/envs/claw/include " FFLAGS="-march=armv8.3-a -ftree-vectorize -fPIC -fno-stack-protector -O2 -pipe -isystem /opt/homebrew/Caskroom/miniforge/base/envs/claw/include " LDFLAGS="-Wl,-headerpad_max_install_names -Wl,-dead_strip_dylibs -Wl,-rpath,/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib -L/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib" LIBS="-Wl,-rpath,/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib -lmpi_mpifh -lgfortran" --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 --FOPTFLAGS=-O3 --with-clib-autodetect=0 --with-cxxlib-autodetect=0 --with-fortranlib-autodetect=0 --with-debugging=0 --with-blas-lib=libblas.dylib --with-lapack-lib=liblapack.dylib --with-yaml=1 --with-hdf5=1 --with-fftw=1 --with-hwloc=0 --with-hypre=1 --with-metis=1 --with-mpi=1 --with-mumps=1 --with-parmetis=1 --with-pthread=1 --with-ptscotch=1 --with-shared-libraries --with-ssl=0 --with-scalapack=1 --with-superlu=1 --with-superlu_dist=1 --with-superlu_dist-include=/opt/homebrew/Caskroom/miniforge/base/envs/claw/include/superlu-dist --with-superlu_dist-lib=-lsuperlu_dist --with-suitesparse=1 --with-suitesparse-dir=/opt/homebrew/Caskroom/miniforge/base/envs/claw --with-x=0 --with-scalar-type=real --with-cuda=0 --with-batch --prefix=/opt/homebrew/Caskroom/miniforge/base/envs/claw [0]PETSC ERROR: #1 PetscShmgetMapAddresses() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/sys/utils/server.c:114 [0]PETSC ERROR: #2 PCMPISetMat() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/ksp/pc/impls/mpi/pcmpi.c:269 [0]PETSC ERROR: #3 PCSetUp_MPI() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/ksp/pc/impls/mpi/pcmpi.c:853 [0]PETSC ERROR: #4 PCSetUp() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/ksp/pc/interface/precon.c:1071 [0]PETSC ERROR: #5 KSPSetUp() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/ksp/ksp/interface/itfunc.c:415 [0]PETSC ERROR: #6 KSPSolve_Private() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/ksp/ksp/interface/itfunc.c:826 [0]PETSC ERROR: #7 KSPSolve() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/ksp/ksp/interface/itfunc.c:1075 Code does not progress and I kill it ^CTraceback (most recent call last): File "/Users/praveen/Applications/clawpack/clawutil/src/python/clawutil/runclaw.py", line 341, in runclaw(*args) File "/Users/praveen/Applications/clawpack/clawutil/src/python/clawutil/runclaw.py", line 242, in runclaw proc = subprocess.check_call(cmd_split, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib/python3.12/subprocess.py", line 408, in check_call retcode = call(*popenargs, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib/python3.12/subprocess.py", line 391, in call return p.wait(timeout=timeout) ^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib/python3.12/subprocess.py", line 1264, in wait return self._wait(timeout=timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib/python3.12/subprocess.py", line 2053, in _wait (pid, sts) = self._try_wait(0) ^^^^^^^^^^^^^^^^^ File "/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib/python3.12/subprocess.py", line 2011, in _try_wait (pid, sts) = os.waitpid(self.pid, wait_flags) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ KeyboardInterrupt make[1]: *** [output] Interrupt: 2 make: *** [.output] Interrupt: 2 Now it says $ ipcs -m IPC status from as of Thu Oct 24 08:05:06 IST 2024 T ID KEY MODE OWNER GROUP Shared Memory: m 720896 0x0000000a --rw-rw-rw- praveen staff Thanks praveen > On 23 Oct 2024, at 8:26?PM, Barry Smith wrote: > > > Hmm, so it is creating the first shared memory region with ID of 10 (A in hex) puts it in a linked list in PETSc but then when it tries to find it in the linked list it cannot find it. > > I don't know how to reproduce this or debug it remotely. > > Can you build on a completely different machine or with completely different compilers? > > Barry > > > > >> On Oct 23, 2024, at 10:31?AM, Praveen C wrote: >> >> I get same error and now it shows >> >> $ ipcs -m >> >> ------ Shared Memory Segments -------- >> key shmid owner perms bytes nattch status >> 0x0000000a 32788 praveen 666 240 6 >> >> Note that the code seems to be still running after printing those error message, but it is not printing any progress which it should do. >> >> Thanks >> praveen >> >>> On 23 Oct 2024, at 7:56?PM, Barry Smith wrote: >>> >>> >>> Try >>> >>> ipcrm -m 11 >>> >>> ipcs -m >>> >>> Try running the program again >>> >>> If failed check >>> >>> ipcs -m >>> >>> again >>> >>> >>> >>>> On Oct 23, 2024, at 10:20?AM, Praveen C wrote: >>>> >>>> Hello Barry >>>> >>>> I see this >>>> >>>> $ ipcs -m >>>> >>>> ------ Shared Memory Segments -------- >>>> key shmid owner perms bytes nattch status >>>> 0x0000000a 11 praveen 666 240 6 >>>> >>>> and I am observing same error as below. >>>> >>>> Thanks >>>> praveen >>>> >>>>> On 23 Oct 2024, at 7:08?PM, Barry Smith wrote: >>>>> >>>>> >>>>> Please take a look at the notes in https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/PetscShmgetAllocateArray/__;!!G_uCfscf7eWS!ZGVQF1mXRVL7_W_U2YYltlhjpJnbZse3y51RCYHH0aJtnt9WRtDMxGfCLD_5ua4VLxp0_IMDDAycDKVkQg5Zlw$ For some reason your program is not able to access/use the Unix shared memory; check if you are already using the shared memory (so it is not available for a new run) or the limits are too low to access enough memory. >>>>> >>>>> Barry >>>>> >>>>>> On Oct 23, 2024, at 8:23?AM, Praveen C wrote: >>>>>> >>>>>> Dear all >>>>>> >>>>>> I am not able to run the boussinesq example from geoclaw using petsc at 3.22.0 >>>>>> >>>>>> https://urldefense.us/v3/__https://github.com/clawpack/geoclaw/tree/3303883f46572c58130d161986b8a87a57ca7816/examples/bouss__;!!G_uCfscf7eWS!e3VQ4NHKmXGstRsQW5vtI7fmKfUT9zmJkMJcPbcvPyIjicyfJpNoMgx3wZ-qyGcKNSjIkNZkzilec8MnHN6PMw$ >>>>>> >>>>>> It runs with petsc at 3.21.6 >>>>>> >>>>>> The error I get is given below. After printing this, the code does not progress. >>>>>> >>>>>> I use the following petsc options >>>>>> >>>>>> # set min numbers of matrix rows per MPI rank (default is 10000) >>>>>> -mpi_linear_solve_minimum_count_per_rank 5000 >>>>>> >>>>>> >>>>>> # Krylov linear solver: >>>>>> -mpi_linear_solver_server >>>>>> -mpi_linear_solver_server_view >>>>>> -ksp_type gmres >>>>>> -ksp_max_it 200 >>>>>> -ksp_reuse_preconditioner >>>>>> -ksp_rtol 1.e-9 >>>>>> >>>>>> # preconditioner: >>>>>> -pc_type gamg >>>>>> >>>>>> I installed petsc and other dependencies for clawpack using miniforge. >>>>>> >>>>>> Thanks >>>>>> pc >>>>>> >>>>>> ==> Use Bouss. in water deeper than 1.0000000000000000 Using a PETSc solver >>>>>> Using Bouss equations from the start >>>>>> rnode allocated... >>>>>> node allocated... >>>>>> listOfGrids allocated... >>>>>> Storage allocated... >>>>>> bndList allocated... >>>>>> Gridding level 1 at t = 0.000000E+00: 4 grids with 10000 cells >>>>>> Setting initial dt to 2.9999999999999999E-002 >>>>>> max threads set to 6 >>>>>> Done reading data, starting computation ... Total zeta at initial time: 39269.907650665169 GEOCLAW: Frame 0 output files done at time t = 0.000000D+00 >>>>>> >>>>>> [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >>>>>> [0]PETSC ERROR: Petsc has generated inconsistent data >>>>>> [0]PETSC ERROR: Unable to locate PCMPI allocated shared address 0x55e6d750ae20 >>>>>> [0]PETSC ERROR: WARNING! There are unused option(s) set! Could be the program crashed before usage or a spelling mistake, etc! >>>>>> [0]PETSC ERROR: Option left: name:-ksp_max_it value: 200 source: file >>>>>> [0]PETSC ERROR: Option left: name:-ksp_reuse_preconditioner (no value) source: file >>>>>> [0]PETSC ERROR: Option left: name:-ksp_rtol value: 1.e-9 source: file >>>>>> [0]PETSC ERROR: Option left: name:-ksp_type value: gmres source: file >>>>>> [0]PETSC ERROR: Option left: name:-mpi_linear_solve_minimum_count_per_rank value: 5000 source: file >>>>>> [0]PETSC ERROR: Option left: name:-mpi_linear_solver_server_view (no value) source: file >>>>>> [0]PETSC ERROR: Option left: name:-pc_type value: gamg source: file >>>>>> [0]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!e3VQ4NHKmXGstRsQW5vtI7fmKfUT9zmJkMJcPbcvPyIjicyfJpNoMgx3wZ-qyGcKNSjIkNZkzilec8MvNjNo7A$ for trouble shooting. >>>>>> [0]PETSC ERROR: Petsc Release Version 3.22.0, Sep 28, 2024 [0]PETSC ERROR: /home/praveen/bouss/radial_flat/xgeoclaw with 6 MPI process(es) and PETSC_ARCH on euler by praveen Thu Oct 17 21:49:54 2024 >>>>>> [0]PETSC ERROR: Configure options: AR=${PREFIX}/bin/x86_64-conda-linux-gnu-ar CC=mpicc CXX=mpicxx FC=mpifort CFLAGS="-march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /opt/miniforge/envs/claw/include " CPPFLAGS="-DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /opt/miniforge/envs/claw/include" CXXFLAGS="-fvisibility-inlines-hidden -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /opt/miniforge/envs/claw/include " FFLAGS="-march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /opt/miniforge/envs/claw/include -Wl,--no-as-needed" LDFLAGS="-pthread -fopenmp -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,--allow-shlib-undefined -Wl,-rpath,/opt/miniforge/envs/claw/lib -Wl,-rpath-link,/opt/miniforge/envs/claw/lib -L/opt/miniforge/envs/claw/lib -Wl,-rpath-link,/opt/miniforge/envs/claw/lib" LIBS="-Wl,-rpath,/opt/miniforge/envs/claw/lib -lmpi_mpifh -lgfortran" --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 --FOPTFLAGS=-O3 --with-clib-autodetect=0 --with-cxxlib-autodetect=0 --with-fortranlib-autodetect=0 --with-debugging=0 --with-blas-lib=libblas.so --with-lapack-lib=liblapack.so --with-yaml=1 --with-hdf5=1 --with-fftw=1 --with-hwloc=0 --with-hypre=1 --with-metis=1 --with-mpi=1 --with-mumps=1 --with-parmetis=1 --with-pthread=1 --with-ptscotch=1 --with-shared-libraries --with-ssl=0 --with-scalapack=1 --with-superlu=1 --with-superlu_dist=1 --with-superlu_dist-include=/opt/miniforge/envs/claw/include/superlu-dist --with-superlu_dist-lib=-lsuperlu_dist --with-suitesparse=1 --with-suitesparse-dir=/opt/miniforge/envs/claw --with-x=0 --with-scalar-type=real --with-cuda=0 --prefix=/opt/miniforge/envs/claw >>>>>> [0]PETSC ERROR: #1 PetscShmgetMapAddresses() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/sys/utils/server.c:114 >>>>>> [0]PETSC ERROR: #2 PCMPISetMat() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/pc/impls/mpi/pcmpi.c:269 >>>>>> [0]PETSC ERROR: #3 PCSetUp_MPI() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/pc/impls/mpi/pcmpi.c:853 >>>>>> [0]PETSC ERROR: #4 PCSetUp() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/pc/interface/precon.c:1071 >>>>>> [0]PETSC ERROR: #5 KSPSetUp() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/ksp/interface/itfunc.c:415 >>>>>> [0]PETSC ERROR: #6 KSPSolve_Private() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/ksp/interface/itfunc.c:826 >>>>>> [0]PETSC ERROR: #7 KSPSolve() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/ksp/interface/itfunc.c:1075 >>>>> >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Thu Oct 24 09:25:01 2024 From: bsmith at petsc.dev (Barry Smith) Date: Thu, 24 Oct 2024 10:25:01 -0400 Subject: [petsc-users] Problem running geoclaw example with petsc 3.22 In-Reply-To: <27B47E2D-00D5-4326-95A9-7595DFF0E952@gmail.com> References: <1C63D19F-6EBC-43A3-8B40-65EA34991AFA@gmail.com> <0136027D-C2CE-47D0-AF97-ACDE1DA0C382@gmail.com> <4726A341-54DB-4592-BBAE-843AF271F28E@petsc.dev> <27B47E2D-00D5-4326-95A9-7595DFF0E952@gmail.com> Message-ID: Ok, super strange, I've run many times on my Mac Can you please try to remove that allocated memory with ipcrm and then cd $PETSC_DIR/src/ksp/ksp/tutorials make ex89f mpiexec -n 3 ./ex89f -n 20 -mpi_linear_solver_server -mpi_linear_solver_server -mpi_linear_solver_server_ksp_view -ksp_monitor -ksp_converged_reason -ksp_view -mpi_linear_solver_server_minimum_count_per_rank 5 This does the same thing as the GeoClaw code but is much simpler. Barry > On Oct 23, 2024, at 10:55?PM, Praveen C wrote: > > I get very similar error on my mac with > > $ gfortran -v > Using built-in specs. > COLLECT_GCC=gfortran > COLLECT_LTO_WRAPPER=/opt/homebrew/Caskroom/miniforge/base/envs/claw/libexec/gcc/arm64-apple-darwin20.0.0/13.2.0/lto-wrapper > Target: arm64-apple-darwin20.0.0 > Configured with: ../configure --prefix=/opt/homebrew/Caskroom/miniforge/base/envs/claw --build=x86_64-apple-darwin13.4.0 --host=arm64-apple-darwin20.0.0 --target=arm64-apple-darwin20.0.0 --with-libiconv-prefix=/opt/homebrew/Caskroom/miniforge/base/envs/claw --enable-languages=fortran --disable-multilib --enable-checking=release --disable-bootstrap --disable-libssp --with-gmp=/opt/homebrew/Caskroom/miniforge/base/envs/claw --with-mpfr=/opt/homebrew/Caskroom/miniforge/base/envs/claw --with-mpc=/opt/homebrew/Caskroom/miniforge/base/envs/claw --with-isl=/opt/homebrew/Caskroom/miniforge/base/envs/claw --enable-darwin-at-rpath > Thread model: posix > Supported LTO compression algorithms: zlib > gcc version 13.2.0 (GCC) > > Before starting > > $ ipcs -m > IPC status from as of Thu Oct 24 08:02:11 IST 2024 > T ID KEY MODE OWNER GROUP > Shared Memory: > > and when I run the code > > Using a PETSc solver > Using Bouss equations from the start > rnode allocated... > node allocated... > listOfGrids allocated... > Storage allocated... > bndList allocated... > Gridding level 1 at t = 0.000000E+00: 4 grids with 10000 cells > Setting initial dt to 2.9999999999999999E-002 > max threads set to 1 > > Done reading data, starting computation ... > > Total zeta at initial time: 39269.907650665169 > GEOCLAW: Frame 0 output files done at time t = 0.000000D+00 > > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [0]PETSC ERROR: Petsc has generated inconsistent data > [0]PETSC ERROR: Unable to locate PCMPI allocated shared address 0x130698000 > [0]PETSC ERROR: WARNING! There are unused option(s) set! Could be the program crashed before usage or a spelling mistake, etc! > [0]PETSC ERROR: Option left: name:-ksp_type value: preonly source: file > [0]PETSC ERROR: Option left: name:-mpi_ksp_max_it value: 200 source: file > [0]PETSC ERROR: Option left: name:-mpi_ksp_reuse_preconditioner (no value) source: file > [0]PETSC ERROR: Option left: name:-mpi_ksp_rtol value: 1.e-9 source: file > [0]PETSC ERROR: Option left: name:-mpi_ksp_type value: gmres source: file > [0]PETSC ERROR: Option left: name:-mpi_linear_solver_server_view (no value) source: file > [0]PETSC ERROR: Option left: name:-mpi_pc_gamg_sym_graph value: true source: file > [0]PETSC ERROR: Option left: name:-mpi_pc_gamg_symmetrize_graph value: true source: file > [0]PETSC ERROR: Option left: name:-mpi_pc_type value: gamg source: file > [0]PETSC ERROR: Option left: name:-pc_mpi_minimum_count_per_rank value: 5000 source: file > [0]PETSC ERROR: Option left: name:-pc_type value: mpi source: file > [0]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!bpaAXGQ_lR0oWQBFY_WN-vIyC5TaIwS8E1FfnQbLpgX0LCqFj8xzFvY7F6MGWmGqzB_3JjhTC_ApnH63wNWGfxg$ for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.22.0, Sep 28, 2024 > [0]PETSC ERROR: /Users/praveen/work/bouss/radial_flat/xgeoclaw with 6 MPI process(es) and PETSC_ARCH on MacMiniHome.local by praveen Thu Oct 24 08:04:27 2024 > [0]PETSC ERROR: Configure options: AR=arm64-apple-darwin20.0.0-ar CC=mpicc CXX=mpicxx FC=mpifort CFLAGS="-ftree-vectorize -fPIC -fstack-protector-strong -O2 -pipe -isystem /opt/homebrew/Caskroom/miniforge/base/envs/claw/include " CPPFLAGS="-D_FORTIFY_SOURCE=2 -isystem /opt/homebrew/Caskroom/miniforge/base/envs/claw/include -mmacosx-version-min=11.0 -mmacosx-version-min=11.0" CXXFLAGS="-ftree-vectorize -fPIC -fstack-protector-strong -O2 -pipe -stdlib=libc++ -fvisibility-inlines-hidden -fmessage-length=0 -isystem /opt/homebrew/Caskroom/miniforge/base/envs/claw/include " FFLAGS="-march=armv8.3-a -ftree-vectorize -fPIC -fno-stack-protector -O2 -pipe -isystem /opt/homebrew/Caskroom/miniforge/base/envs/claw/include " LDFLAGS="-Wl,-headerpad_max_install_names -Wl,-dead_strip_dylibs -Wl,-rpath,/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib -L/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib" LIBS="-Wl,-rpath,/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib -lmpi_mpifh -lgfortran" --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 --FOPTFLAGS=-O3 --with-clib-autodetect=0 --with-cxxlib-autodetect=0 --with-fortranlib-autodetect=0 --with-debugging=0 --with-blas-lib=libblas.dylib --with-lapack-lib=liblapack.dylib --with-yaml=1 --with-hdf5=1 --with-fftw=1 --with-hwloc=0 --with-hypre=1 --with-metis=1 --with-mpi=1 --with-mumps=1 --with-parmetis=1 --with-pthread=1 --with-ptscotch=1 --with-shared-libraries --with-ssl=0 --with-scalapack=1 --with-superlu=1 --with-superlu_dist=1 --with-superlu_dist-include=/opt/homebrew/Caskroom/miniforge/base/envs/claw/include/superlu-dist --with-superlu_dist-lib=-lsuperlu_dist --with-suitesparse=1 --with-suitesparse-dir=/opt/homebrew/Caskroom/miniforge/base/envs/claw --with-x=0 --with-scalar-type=real --with-cuda=0 --with-batch --prefix=/opt/homebrew/Caskroom/miniforge/base/envs/claw > [0]PETSC ERROR: #1 PetscShmgetMapAddresses() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/sys/utils/server.c:114 > [0]PETSC ERROR: #2 PCMPISetMat() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/ksp/pc/impls/mpi/pcmpi.c:269 > [0]PETSC ERROR: #3 PCSetUp_MPI() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/ksp/pc/impls/mpi/pcmpi.c:853 > [0]PETSC ERROR: #4 PCSetUp() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/ksp/pc/interface/precon.c:1071 > [0]PETSC ERROR: #5 KSPSetUp() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/ksp/ksp/interface/itfunc.c:415 > [0]PETSC ERROR: #6 KSPSolve_Private() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/ksp/ksp/interface/itfunc.c:826 > [0]PETSC ERROR: #7 KSPSolve() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/ksp/ksp/interface/itfunc.c:1075 > > Code does not progress and I kill it > > ^CTraceback (most recent call last): > File "/Users/praveen/Applications/clawpack/clawutil/src/python/clawutil/runclaw.py", line 341, in > runclaw(*args) > File "/Users/praveen/Applications/clawpack/clawutil/src/python/clawutil/runclaw.py", line 242, in runclaw > proc = subprocess.check_call(cmd_split, > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > File "/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib/python3.12/subprocess.py", line 408, in check_call > retcode = call(*popenargs, **kwargs) > ^^^^^^^^^^^^^^^^^^^^^^^^^^ > File "/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib/python3.12/subprocess.py", line 391, in call > return p.wait(timeout=timeout) > ^^^^^^^^^^^^^^^^^^^^^^^ > File "/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib/python3.12/subprocess.py", line 1264, in wait > return self._wait(timeout=timeout) > ^^^^^^^^^^^^^^^^^^^^^^^^^^^ > File "/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib/python3.12/subprocess.py", line 2053, in _wait > (pid, sts) = self._try_wait(0) > ^^^^^^^^^^^^^^^^^ > File "/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib/python3.12/subprocess.py", line 2011, in _try_wait > (pid, sts) = os.waitpid(self.pid, wait_flags) > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > KeyboardInterrupt > make[1]: *** [output] Interrupt: 2 > make: *** [.output] Interrupt: 2 > > Now it says > > $ ipcs -m > IPC status from as of Thu Oct 24 08:05:06 IST 2024 > T ID KEY MODE OWNER GROUP > Shared Memory: > m 720896 0x0000000a --rw-rw-rw- praveen staff > > Thanks > praveen > >> On 23 Oct 2024, at 8:26?PM, Barry Smith wrote: >> >> >> Hmm, so it is creating the first shared memory region with ID of 10 (A in hex) puts it in a linked list in PETSc but then when it tries to find it in the linked list it cannot find it. >> >> I don't know how to reproduce this or debug it remotely. >> >> Can you build on a completely different machine or with completely different compilers? >> >> Barry >> >> >> >> >>> On Oct 23, 2024, at 10:31?AM, Praveen C wrote: >>> >>> I get same error and now it shows >>> >>> $ ipcs -m >>> >>> ------ Shared Memory Segments -------- >>> key shmid owner perms bytes nattch status >>> 0x0000000a 32788 praveen 666 240 6 >>> >>> Note that the code seems to be still running after printing those error message, but it is not printing any progress which it should do. >>> >>> Thanks >>> praveen >>> >>>> On 23 Oct 2024, at 7:56?PM, Barry Smith wrote: >>>> >>>> >>>> Try >>>> >>>> ipcrm -m 11 >>>> >>>> ipcs -m >>>> >>>> Try running the program again >>>> >>>> If failed check >>>> >>>> ipcs -m >>>> >>>> again >>>> >>>> >>>> >>>>> On Oct 23, 2024, at 10:20?AM, Praveen C wrote: >>>>> >>>>> Hello Barry >>>>> >>>>> I see this >>>>> >>>>> $ ipcs -m >>>>> >>>>> ------ Shared Memory Segments -------- >>>>> key shmid owner perms bytes nattch status >>>>> 0x0000000a 11 praveen 666 240 6 >>>>> >>>>> and I am observing same error as below. >>>>> >>>>> Thanks >>>>> praveen >>>>> >>>>>> On 23 Oct 2024, at 7:08?PM, Barry Smith wrote: >>>>>> >>>>>> >>>>>> Please take a look at the notes in https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/PetscShmgetAllocateArray/__;!!G_uCfscf7eWS!bpaAXGQ_lR0oWQBFY_WN-vIyC5TaIwS8E1FfnQbLpgX0LCqFj8xzFvY7F6MGWmGqzB_3JjhTC_ApnH637bfMOgs$ For some reason your program is not able to access/use the Unix shared memory; check if you are already using the shared memory (so it is not available for a new run) or the limits are too low to access enough memory. >>>>>> >>>>>> Barry >>>>>> >>>>>>> On Oct 23, 2024, at 8:23?AM, Praveen C wrote: >>>>>>> >>>>>>> Dear all >>>>>>> >>>>>>> I am not able to run the boussinesq example from geoclaw using petsc at 3.22.0 >>>>>>> >>>>>>> https://urldefense.us/v3/__https://github.com/clawpack/geoclaw/tree/3303883f46572c58130d161986b8a87a57ca7816/examples/bouss__;!!G_uCfscf7eWS!e3VQ4NHKmXGstRsQW5vtI7fmKfUT9zmJkMJcPbcvPyIjicyfJpNoMgx3wZ-qyGcKNSjIkNZkzilec8MnHN6PMw$ >>>>>>> >>>>>>> It runs with petsc at 3.21.6 >>>>>>> >>>>>>> The error I get is given below. After printing this, the code does not progress. >>>>>>> >>>>>>> I use the following petsc options >>>>>>> >>>>>>> # set min numbers of matrix rows per MPI rank (default is 10000) >>>>>>> -mpi_linear_solve_minimum_count_per_rank 5000 >>>>>>> >>>>>>> >>>>>>> # Krylov linear solver: >>>>>>> -mpi_linear_solver_server >>>>>>> -mpi_linear_solver_server_view >>>>>>> -ksp_type gmres >>>>>>> -ksp_max_it 200 >>>>>>> -ksp_reuse_preconditioner >>>>>>> -ksp_rtol 1.e-9 >>>>>>> >>>>>>> # preconditioner: >>>>>>> -pc_type gamg >>>>>>> >>>>>>> I installed petsc and other dependencies for clawpack using miniforge. >>>>>>> >>>>>>> Thanks >>>>>>> pc >>>>>>> >>>>>>> ==> Use Bouss. in water deeper than 1.0000000000000000 Using a PETSc solver >>>>>>> Using Bouss equations from the start >>>>>>> rnode allocated... >>>>>>> node allocated... >>>>>>> listOfGrids allocated... >>>>>>> Storage allocated... >>>>>>> bndList allocated... >>>>>>> Gridding level 1 at t = 0.000000E+00: 4 grids with 10000 cells >>>>>>> Setting initial dt to 2.9999999999999999E-002 >>>>>>> max threads set to 6 >>>>>>> Done reading data, starting computation ... Total zeta at initial time: 39269.907650665169 GEOCLAW: Frame 0 output files done at time t = 0.000000D+00 >>>>>>> >>>>>>> [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >>>>>>> [0]PETSC ERROR: Petsc has generated inconsistent data >>>>>>> [0]PETSC ERROR: Unable to locate PCMPI allocated shared address 0x55e6d750ae20 >>>>>>> [0]PETSC ERROR: WARNING! There are unused option(s) set! Could be the program crashed before usage or a spelling mistake, etc! >>>>>>> [0]PETSC ERROR: Option left: name:-ksp_max_it value: 200 source: file >>>>>>> [0]PETSC ERROR: Option left: name:-ksp_reuse_preconditioner (no value) source: file >>>>>>> [0]PETSC ERROR: Option left: name:-ksp_rtol value: 1.e-9 source: file >>>>>>> [0]PETSC ERROR: Option left: name:-ksp_type value: gmres source: file >>>>>>> [0]PETSC ERROR: Option left: name:-mpi_linear_solve_minimum_count_per_rank value: 5000 source: file >>>>>>> [0]PETSC ERROR: Option left: name:-mpi_linear_solver_server_view (no value) source: file >>>>>>> [0]PETSC ERROR: Option left: name:-pc_type value: gamg source: file >>>>>>> [0]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!e3VQ4NHKmXGstRsQW5vtI7fmKfUT9zmJkMJcPbcvPyIjicyfJpNoMgx3wZ-qyGcKNSjIkNZkzilec8MvNjNo7A$ for trouble shooting. >>>>>>> [0]PETSC ERROR: Petsc Release Version 3.22.0, Sep 28, 2024 [0]PETSC ERROR: /home/praveen/bouss/radial_flat/xgeoclaw with 6 MPI process(es) and PETSC_ARCH on euler by praveen Thu Oct 17 21:49:54 2024 >>>>>>> [0]PETSC ERROR: Configure options: AR=${PREFIX}/bin/x86_64-conda-linux-gnu-ar CC=mpicc CXX=mpicxx FC=mpifort CFLAGS="-march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /opt/miniforge/envs/claw/include " CPPFLAGS="-DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /opt/miniforge/envs/claw/include" CXXFLAGS="-fvisibility-inlines-hidden -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /opt/miniforge/envs/claw/include " FFLAGS="-march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /opt/miniforge/envs/claw/include -Wl,--no-as-needed" LDFLAGS="-pthread -fopenmp -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,--allow-shlib-undefined -Wl,-rpath,/opt/miniforge/envs/claw/lib -Wl,-rpath-link,/opt/miniforge/envs/claw/lib -L/opt/miniforge/envs/claw/lib -Wl,-rpath-link,/opt/miniforge/envs/claw/lib" LIBS="-Wl,-rpath,/opt/miniforge/envs/claw/lib -lmpi_mpifh -lgfortran" --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 --FOPTFLAGS=-O3 --with-clib-autodetect=0 --with-cxxlib-autodetect=0 --with-fortranlib-autodetect=0 --with-debugging=0 --with-blas-lib=libblas.so --with-lapack-lib=liblapack.so --with-yaml=1 --with-hdf5=1 --with-fftw=1 --with-hwloc=0 --with-hypre=1 --with-metis=1 --with-mpi=1 --with-mumps=1 --with-parmetis=1 --with-pthread=1 --with-ptscotch=1 --with-shared-libraries --with-ssl=0 --with-scalapack=1 --with-superlu=1 --with-superlu_dist=1 --with-superlu_dist-include=/opt/miniforge/envs/claw/include/superlu-dist --with-superlu_dist-lib=-lsuperlu_dist --with-suitesparse=1 --with-suitesparse-dir=/opt/miniforge/envs/claw --with-x=0 --with-scalar-type=real --with-cuda=0 --prefix=/opt/miniforge/envs/claw >>>>>>> [0]PETSC ERROR: #1 PetscShmgetMapAddresses() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/sys/utils/server.c:114 >>>>>>> [0]PETSC ERROR: #2 PCMPISetMat() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/pc/impls/mpi/pcmpi.c:269 >>>>>>> [0]PETSC ERROR: #3 PCSetUp_MPI() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/pc/impls/mpi/pcmpi.c:853 >>>>>>> [0]PETSC ERROR: #4 PCSetUp() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/pc/interface/precon.c:1071 >>>>>>> [0]PETSC ERROR: #5 KSPSetUp() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/ksp/interface/itfunc.c:415 >>>>>>> [0]PETSC ERROR: #6 KSPSolve_Private() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/ksp/interface/itfunc.c:826 >>>>>>> [0]PETSC ERROR: #7 KSPSolve() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/ksp/interface/itfunc.c:1075 >>>>>> >>>>> >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cpraveen at gmail.com Thu Oct 24 10:49:41 2024 From: cpraveen at gmail.com (Praveen C) Date: Thu, 24 Oct 2024 21:19:41 +0530 Subject: [petsc-users] Problem running geoclaw example with petsc 3.22 In-Reply-To: References: <1C63D19F-6EBC-43A3-8B40-65EA34991AFA@gmail.com> <0136027D-C2CE-47D0-AF97-ACDE1DA0C382@gmail.com> <4726A341-54DB-4592-BBAE-843AF271F28E@petsc.dev> <27B47E2D-00D5-4326-95A9-7595DFF0E952@gmail.com> Message-ID: I get this $ mpiexec -n 3 ./ex89f -n 20 -mpi_linear_solver_server -mpi_linear_solver_server -mpi_linear_solver_server_ksp_view -ksp_monitor -ksp_converged_reason -ksp_view -mpi_linear_solver_server_minimum_count_per_rank 5 [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see https://urldefense.us/v3/__https://petsc.org/release/faq/*valgrind__;Iw!!G_uCfscf7eWS!de6DU4GTEEdz3Krd2vMFGCVHtIp9eWFSEiOkgNKineAMPAQMV1cZm_xpwAfn8Sb4NuTam1AssCqSd-aHA73oOg$ and https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!de6DU4GTEEdz3Krd2vMFGCVHtIp9eWFSEiOkgNKineAMPAQMV1cZm_xpwAfn8Sb4NuTam1AssCqSd-ZGxfjmXQ$ [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [0]PETSC ERROR: to get more information on the crash. [0]PETSC ERROR: Run with -malloc_debug to check if memory corruption is causing the crash. -------------------------------------------------------------------------- MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_SELF Proc: [[47380,1],0] Errorcode: 59 NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. You may or may not see output from other processes, depending on exactly when Open MPI kills them. -------------------------------------------------------------------------- -------------------------------------------------------------------------- prterun has exited due to process rank 0 with PID 0 on node MacMiniHome calling "abort". This may have caused other processes in the application to be terminated by signals sent by prterun (as reported here). ????????????????????????????????????? and I have this after the code exits $ ipcs -m IPC status from as of Thu Oct 24 21:17:39 IST 2024 T ID KEY MODE OWNER GROUP Shared Memory: m 1572864 0x0000000b --rw-rw-rw- praveen staff m 524289 0x0000000c --rw-rw-rw- praveen staff m 655362 0x0000000d --rw-rw-rw- praveen staff m 262147 0x0000000e --rw-rw-rw- praveen staff m 262148 0x0000000f --rw-rw-rw- praveen staff m 393221 0x0000000a --rw-rw-rw- praveen staff This is with petsc installed with miniforge, which I also use with clawpack. With spack installed petsc, I can run the ex89f example. Thanks praveen > On 24 Oct 2024, at 7:55?PM, Barry Smith wrote: > > > Ok, super strange, I've run many times on my Mac > > Can you please try to remove that allocated memory with ipcrm and then > > cd $PETSC_DIR/src/ksp/ksp/tutorials > make ex89f > mpiexec -n 3 ./ex89f -n 20 -mpi_linear_solver_server -mpi_linear_solver_server -mpi_linear_solver_server_ksp_view -ksp_monitor -ksp_converged_reason -ksp_view -mpi_linear_solver_server_minimum_count_per_rank 5 > > This does the same thing as the GeoClaw code but is much simpler. > > Barry > > > > >> On Oct 23, 2024, at 10:55?PM, Praveen C wrote: >> >> I get very similar error on my mac with >> >> $ gfortran -v >> Using built-in specs. >> COLLECT_GCC=gfortran >> COLLECT_LTO_WRAPPER=/opt/homebrew/Caskroom/miniforge/base/envs/claw/libexec/gcc/arm64-apple-darwin20.0.0/13.2.0/lto-wrapper >> Target: arm64-apple-darwin20.0.0 >> Configured with: ../configure --prefix=/opt/homebrew/Caskroom/miniforge/base/envs/claw --build=x86_64-apple-darwin13.4.0 --host=arm64-apple-darwin20.0.0 --target=arm64-apple-darwin20.0.0 --with-libiconv-prefix=/opt/homebrew/Caskroom/miniforge/base/envs/claw --enable-languages=fortran --disable-multilib --enable-checking=release --disable-bootstrap --disable-libssp --with-gmp=/opt/homebrew/Caskroom/miniforge/base/envs/claw --with-mpfr=/opt/homebrew/Caskroom/miniforge/base/envs/claw --with-mpc=/opt/homebrew/Caskroom/miniforge/base/envs/claw --with-isl=/opt/homebrew/Caskroom/miniforge/base/envs/claw --enable-darwin-at-rpath >> Thread model: posix >> Supported LTO compression algorithms: zlib >> gcc version 13.2.0 (GCC) >> >> Before starting >> >> $ ipcs -m >> IPC status from as of Thu Oct 24 08:02:11 IST 2024 >> T ID KEY MODE OWNER GROUP >> Shared Memory: >> >> and when I run the code >> >> Using a PETSc solver >> Using Bouss equations from the start >> rnode allocated... >> node allocated... >> listOfGrids allocated... >> Storage allocated... >> bndList allocated... >> Gridding level 1 at t = 0.000000E+00: 4 grids with 10000 cells >> Setting initial dt to 2.9999999999999999E-002 >> max threads set to 1 >> >> Done reading data, starting computation ... >> >> Total zeta at initial time: 39269.907650665169 >> GEOCLAW: Frame 0 output files done at time t = 0.000000D+00 >> >> [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >> [0]PETSC ERROR: Petsc has generated inconsistent data >> [0]PETSC ERROR: Unable to locate PCMPI allocated shared address 0x130698000 >> [0]PETSC ERROR: WARNING! There are unused option(s) set! Could be the program crashed before usage or a spelling mistake, etc! >> [0]PETSC ERROR: Option left: name:-ksp_type value: preonly source: file >> [0]PETSC ERROR: Option left: name:-mpi_ksp_max_it value: 200 source: file >> [0]PETSC ERROR: Option left: name:-mpi_ksp_reuse_preconditioner (no value) source: file >> [0]PETSC ERROR: Option left: name:-mpi_ksp_rtol value: 1.e-9 source: file >> [0]PETSC ERROR: Option left: name:-mpi_ksp_type value: gmres source: file >> [0]PETSC ERROR: Option left: name:-mpi_linear_solver_server_view (no value) source: file >> [0]PETSC ERROR: Option left: name:-mpi_pc_gamg_sym_graph value: true source: file >> [0]PETSC ERROR: Option left: name:-mpi_pc_gamg_symmetrize_graph value: true source: file >> [0]PETSC ERROR: Option left: name:-mpi_pc_type value: gamg source: file >> [0]PETSC ERROR: Option left: name:-pc_mpi_minimum_count_per_rank value: 5000 source: file >> [0]PETSC ERROR: Option left: name:-pc_type value: mpi source: file >> [0]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!de6DU4GTEEdz3Krd2vMFGCVHtIp9eWFSEiOkgNKineAMPAQMV1cZm_xpwAfn8Sb4NuTam1AssCqSd-ZGxfjmXQ$ for trouble shooting. >> [0]PETSC ERROR: Petsc Release Version 3.22.0, Sep 28, 2024 >> [0]PETSC ERROR: /Users/praveen/work/bouss/radial_flat/xgeoclaw with 6 MPI process(es) and PETSC_ARCH on MacMiniHome.local by praveen Thu Oct 24 08:04:27 2024 >> [0]PETSC ERROR: Configure options: AR=arm64-apple-darwin20.0.0-ar CC=mpicc CXX=mpicxx FC=mpifort CFLAGS="-ftree-vectorize -fPIC -fstack-protector-strong -O2 -pipe -isystem /opt/homebrew/Caskroom/miniforge/base/envs/claw/include " CPPFLAGS="-D_FORTIFY_SOURCE=2 -isystem /opt/homebrew/Caskroom/miniforge/base/envs/claw/include -mmacosx-version-min=11.0 -mmacosx-version-min=11.0" CXXFLAGS="-ftree-vectorize -fPIC -fstack-protector-strong -O2 -pipe -stdlib=libc++ -fvisibility-inlines-hidden -fmessage-length=0 -isystem /opt/homebrew/Caskroom/miniforge/base/envs/claw/include " FFLAGS="-march=armv8.3-a -ftree-vectorize -fPIC -fno-stack-protector -O2 -pipe -isystem /opt/homebrew/Caskroom/miniforge/base/envs/claw/include " LDFLAGS="-Wl,-headerpad_max_install_names -Wl,-dead_strip_dylibs -Wl,-rpath,/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib -L/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib" LIBS="-Wl,-rpath,/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib -lmpi_mpifh -lgfortran" --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 --FOPTFLAGS=-O3 --with-clib-autodetect=0 --with-cxxlib-autodetect=0 --with-fortranlib-autodetect=0 --with-debugging=0 --with-blas-lib=libblas.dylib --with-lapack-lib=liblapack.dylib --with-yaml=1 --with-hdf5=1 --with-fftw=1 --with-hwloc=0 --with-hypre=1 --with-metis=1 --with-mpi=1 --with-mumps=1 --with-parmetis=1 --with-pthread=1 --with-ptscotch=1 --with-shared-libraries --with-ssl=0 --with-scalapack=1 --with-superlu=1 --with-superlu_dist=1 --with-superlu_dist-include=/opt/homebrew/Caskroom/miniforge/base/envs/claw/include/superlu-dist --with-superlu_dist-lib=-lsuperlu_dist --with-suitesparse=1 --with-suitesparse-dir=/opt/homebrew/Caskroom/miniforge/base/envs/claw --with-x=0 --with-scalar-type=real --with-cuda=0 --with-batch --prefix=/opt/homebrew/Caskroom/miniforge/base/envs/claw >> [0]PETSC ERROR: #1 PetscShmgetMapAddresses() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/sys/utils/server.c:114 >> [0]PETSC ERROR: #2 PCMPISetMat() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/ksp/pc/impls/mpi/pcmpi.c:269 >> [0]PETSC ERROR: #3 PCSetUp_MPI() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/ksp/pc/impls/mpi/pcmpi.c:853 >> [0]PETSC ERROR: #4 PCSetUp() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/ksp/pc/interface/precon.c:1071 >> [0]PETSC ERROR: #5 KSPSetUp() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/ksp/ksp/interface/itfunc.c:415 >> [0]PETSC ERROR: #6 KSPSolve_Private() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/ksp/ksp/interface/itfunc.c:826 >> [0]PETSC ERROR: #7 KSPSolve() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/ksp/ksp/interface/itfunc.c:1075 >> >> Code does not progress and I kill it >> >> ^CTraceback (most recent call last): >> File "/Users/praveen/Applications/clawpack/clawutil/src/python/clawutil/runclaw.py", line 341, in >> runclaw(*args) >> File "/Users/praveen/Applications/clawpack/clawutil/src/python/clawutil/runclaw.py", line 242, in runclaw >> proc = subprocess.check_call(cmd_split, >> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ >> File "/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib/python3.12/subprocess.py", line 408, in check_call >> retcode = call(*popenargs, **kwargs) >> ^^^^^^^^^^^^^^^^^^^^^^^^^^ >> File "/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib/python3.12/subprocess.py", line 391, in call >> return p.wait(timeout=timeout) >> ^^^^^^^^^^^^^^^^^^^^^^^ >> File "/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib/python3.12/subprocess.py", line 1264, in wait >> return self._wait(timeout=timeout) >> ^^^^^^^^^^^^^^^^^^^^^^^^^^^ >> File "/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib/python3.12/subprocess.py", line 2053, in _wait >> (pid, sts) = self._try_wait(0) >> ^^^^^^^^^^^^^^^^^ >> File "/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib/python3.12/subprocess.py", line 2011, in _try_wait >> (pid, sts) = os.waitpid(self.pid, wait_flags) >> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ >> KeyboardInterrupt >> make[1]: *** [output] Interrupt: 2 >> make: *** [.output] Interrupt: 2 >> >> Now it says >> >> $ ipcs -m >> IPC status from as of Thu Oct 24 08:05:06 IST 2024 >> T ID KEY MODE OWNER GROUP >> Shared Memory: >> m 720896 0x0000000a --rw-rw-rw- praveen staff >> >> Thanks >> praveen >> >>> On 23 Oct 2024, at 8:26?PM, Barry Smith wrote: >>> >>> >>> Hmm, so it is creating the first shared memory region with ID of 10 (A in hex) puts it in a linked list in PETSc but then when it tries to find it in the linked list it cannot find it. >>> >>> I don't know how to reproduce this or debug it remotely. >>> >>> Can you build on a completely different machine or with completely different compilers? >>> >>> Barry >>> >>> >>> >>> >>>> On Oct 23, 2024, at 10:31?AM, Praveen C wrote: >>>> >>>> I get same error and now it shows >>>> >>>> $ ipcs -m >>>> >>>> ------ Shared Memory Segments -------- >>>> key shmid owner perms bytes nattch status >>>> 0x0000000a 32788 praveen 666 240 6 >>>> >>>> Note that the code seems to be still running after printing those error message, but it is not printing any progress which it should do. >>>> >>>> Thanks >>>> praveen >>>> >>>>> On 23 Oct 2024, at 7:56?PM, Barry Smith wrote: >>>>> >>>>> >>>>> Try >>>>> >>>>> ipcrm -m 11 >>>>> >>>>> ipcs -m >>>>> >>>>> Try running the program again >>>>> >>>>> If failed check >>>>> >>>>> ipcs -m >>>>> >>>>> again >>>>> >>>>> >>>>> >>>>>> On Oct 23, 2024, at 10:20?AM, Praveen C wrote: >>>>>> >>>>>> Hello Barry >>>>>> >>>>>> I see this >>>>>> >>>>>> $ ipcs -m >>>>>> >>>>>> ------ Shared Memory Segments -------- >>>>>> key shmid owner perms bytes nattch status >>>>>> 0x0000000a 11 praveen 666 240 6 >>>>>> >>>>>> and I am observing same error as below. >>>>>> >>>>>> Thanks >>>>>> praveen >>>>>> >>>>>>> On 23 Oct 2024, at 7:08?PM, Barry Smith wrote: >>>>>>> >>>>>>> >>>>>>> Please take a look at the notes in https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/PetscShmgetAllocateArray/__;!!G_uCfscf7eWS!de6DU4GTEEdz3Krd2vMFGCVHtIp9eWFSEiOkgNKineAMPAQMV1cZm_xpwAfn8Sb4NuTam1AssCqSd-bL3C5qOw$ For some reason your program is not able to access/use the Unix shared memory; check if you are already using the shared memory (so it is not available for a new run) or the limits are too low to access enough memory. >>>>>>> >>>>>>> Barry >>>>>>> >>>>>>>> On Oct 23, 2024, at 8:23?AM, Praveen C wrote: >>>>>>>> >>>>>>>> Dear all >>>>>>>> >>>>>>>> I am not able to run the boussinesq example from geoclaw using petsc at 3.22.0 >>>>>>>> >>>>>>>> https://urldefense.us/v3/__https://github.com/clawpack/geoclaw/tree/3303883f46572c58130d161986b8a87a57ca7816/examples/bouss__;!!G_uCfscf7eWS!e3VQ4NHKmXGstRsQW5vtI7fmKfUT9zmJkMJcPbcvPyIjicyfJpNoMgx3wZ-qyGcKNSjIkNZkzilec8MnHN6PMw$ >>>>>>>> >>>>>>>> It runs with petsc at 3.21.6 >>>>>>>> >>>>>>>> The error I get is given below. After printing this, the code does not progress. >>>>>>>> >>>>>>>> I use the following petsc options >>>>>>>> >>>>>>>> # set min numbers of matrix rows per MPI rank (default is 10000) >>>>>>>> -mpi_linear_solve_minimum_count_per_rank 5000 >>>>>>>> >>>>>>>> >>>>>>>> # Krylov linear solver: >>>>>>>> -mpi_linear_solver_server >>>>>>>> -mpi_linear_solver_server_view >>>>>>>> -ksp_type gmres >>>>>>>> -ksp_max_it 200 >>>>>>>> -ksp_reuse_preconditioner >>>>>>>> -ksp_rtol 1.e-9 >>>>>>>> >>>>>>>> # preconditioner: >>>>>>>> -pc_type gamg >>>>>>>> >>>>>>>> I installed petsc and other dependencies for clawpack using miniforge. >>>>>>>> >>>>>>>> Thanks >>>>>>>> pc >>>>>>>> >>>>>>>> ==> Use Bouss. in water deeper than 1.0000000000000000 Using a PETSc solver >>>>>>>> Using Bouss equations from the start >>>>>>>> rnode allocated... >>>>>>>> node allocated... >>>>>>>> listOfGrids allocated... >>>>>>>> Storage allocated... >>>>>>>> bndList allocated... >>>>>>>> Gridding level 1 at t = 0.000000E+00: 4 grids with 10000 cells >>>>>>>> Setting initial dt to 2.9999999999999999E-002 >>>>>>>> max threads set to 6 >>>>>>>> Done reading data, starting computation ... Total zeta at initial time: 39269.907650665169 GEOCLAW: Frame 0 output files done at time t = 0.000000D+00 >>>>>>>> >>>>>>>> [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >>>>>>>> [0]PETSC ERROR: Petsc has generated inconsistent data >>>>>>>> [0]PETSC ERROR: Unable to locate PCMPI allocated shared address 0x55e6d750ae20 >>>>>>>> [0]PETSC ERROR: WARNING! There are unused option(s) set! Could be the program crashed before usage or a spelling mistake, etc! >>>>>>>> [0]PETSC ERROR: Option left: name:-ksp_max_it value: 200 source: file >>>>>>>> [0]PETSC ERROR: Option left: name:-ksp_reuse_preconditioner (no value) source: file >>>>>>>> [0]PETSC ERROR: Option left: name:-ksp_rtol value: 1.e-9 source: file >>>>>>>> [0]PETSC ERROR: Option left: name:-ksp_type value: gmres source: file >>>>>>>> [0]PETSC ERROR: Option left: name:-mpi_linear_solve_minimum_count_per_rank value: 5000 source: file >>>>>>>> [0]PETSC ERROR: Option left: name:-mpi_linear_solver_server_view (no value) source: file >>>>>>>> [0]PETSC ERROR: Option left: name:-pc_type value: gamg source: file >>>>>>>> [0]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!e3VQ4NHKmXGstRsQW5vtI7fmKfUT9zmJkMJcPbcvPyIjicyfJpNoMgx3wZ-qyGcKNSjIkNZkzilec8MvNjNo7A$ for trouble shooting. >>>>>>>> [0]PETSC ERROR: Petsc Release Version 3.22.0, Sep 28, 2024 [0]PETSC ERROR: /home/praveen/bouss/radial_flat/xgeoclaw with 6 MPI process(es) and PETSC_ARCH on euler by praveen Thu Oct 17 21:49:54 2024 >>>>>>>> [0]PETSC ERROR: Configure options: AR=${PREFIX}/bin/x86_64-conda-linux-gnu-ar CC=mpicc CXX=mpicxx FC=mpifort CFLAGS="-march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /opt/miniforge/envs/claw/include " CPPFLAGS="-DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /opt/miniforge/envs/claw/include" CXXFLAGS="-fvisibility-inlines-hidden -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /opt/miniforge/envs/claw/include " FFLAGS="-march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /opt/miniforge/envs/claw/include -Wl,--no-as-needed" LDFLAGS="-pthread -fopenmp -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,--allow-shlib-undefined -Wl,-rpath,/opt/miniforge/envs/claw/lib -Wl,-rpath-link,/opt/miniforge/envs/claw/lib -L/opt/miniforge/envs/claw/lib -Wl,-rpath-link,/opt/miniforge/envs/claw/lib" LIBS="-Wl,-rpath,/opt/miniforge/envs/claw/lib -lmpi_mpifh -lgfortran" --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 --FOPTFLAGS=-O3 --with-clib-autodetect=0 --with-cxxlib-autodetect=0 --with-fortranlib-autodetect=0 --with-debugging=0 --with-blas-lib=libblas.so --with-lapack-lib=liblapack.so --with-yaml=1 --with-hdf5=1 --with-fftw=1 --with-hwloc=0 --with-hypre=1 --with-metis=1 --with-mpi=1 --with-mumps=1 --with-parmetis=1 --with-pthread=1 --with-ptscotch=1 --with-shared-libraries --with-ssl=0 --with-scalapack=1 --with-superlu=1 --with-superlu_dist=1 --with-superlu_dist-include=/opt/miniforge/envs/claw/include/superlu-dist --with-superlu_dist-lib=-lsuperlu_dist --with-suitesparse=1 --with-suitesparse-dir=/opt/miniforge/envs/claw --with-x=0 --with-scalar-type=real --with-cuda=0 --prefix=/opt/miniforge/envs/claw >>>>>>>> [0]PETSC ERROR: #1 PetscShmgetMapAddresses() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/sys/utils/server.c:114 >>>>>>>> [0]PETSC ERROR: #2 PCMPISetMat() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/pc/impls/mpi/pcmpi.c:269 >>>>>>>> [0]PETSC ERROR: #3 PCSetUp_MPI() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/pc/impls/mpi/pcmpi.c:853 >>>>>>>> [0]PETSC ERROR: #4 PCSetUp() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/pc/interface/precon.c:1071 >>>>>>>> [0]PETSC ERROR: #5 KSPSetUp() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/ksp/interface/itfunc.c:415 >>>>>>>> [0]PETSC ERROR: #6 KSPSolve_Private() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/ksp/interface/itfunc.c:826 >>>>>>>> [0]PETSC ERROR: #7 KSPSolve() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/ksp/interface/itfunc.c:1075 >>>>>>> >>>>>> >>>>> >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Thu Oct 24 11:07:30 2024 From: bsmith at petsc.dev (Barry Smith) Date: Thu, 24 Oct 2024 12:07:30 -0400 Subject: [petsc-users] Problem running geoclaw example with petsc 3.22 In-Reply-To: References: <1C63D19F-6EBC-43A3-8B40-65EA34991AFA@gmail.com> <0136027D-C2CE-47D0-AF97-ACDE1DA0C382@gmail.com> <4726A341-54DB-4592-BBAE-843AF271F28E@petsc.dev> <27B47E2D-00D5-4326-95A9-7595DFF0E952@gmail.com> Message-ID: <490B5A4A-BBD8-490A-91FD-7CEC9E01A99B@petsc.dev> Good, some helpful information with the runs you made. In the crash below it made more progress it was able to allocate multiple regions of shared memory and access them. I don't know why it would crash later. Can you tell me all the steps with miniforge (which seems to be related to the failure) you use? I've never used miniforge. If I can get to an environment that reproduces the problem I can debug it and fix it. Barry > On Oct 24, 2024, at 11:49?AM, Praveen C wrote: > > I get this > > $ mpiexec -n 3 ./ex89f -n 20 -mpi_linear_solver_server -mpi_linear_solver_server -mpi_linear_solver_server_ksp_view -ksp_monitor -ksp_converged_reason -ksp_view -mpi_linear_solver_server_minimum_count_per_rank 5 > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range > [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [0]PETSC ERROR: or see https://urldefense.us/v3/__https://petsc.org/release/faq/*valgrind__;Iw!!G_uCfscf7eWS!a83YNiE6vfZTxyhWxnJxmh95Z63M4HapLLI6b0Q_hFwekNKgjv2j3t61lo38kvy2gLDVjWO5IScFYBVUNuMuh34$ and https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!a83YNiE6vfZTxyhWxnJxmh95Z63M4HapLLI6b0Q_hFwekNKgjv2j3t61lo38kvy2gLDVjWO5IScFYBVUJaIxcoc$ > [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run > [0]PETSC ERROR: to get more information on the crash. > [0]PETSC ERROR: Run with -malloc_debug to check if memory corruption is causing the crash. > -------------------------------------------------------------------------- > MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_SELF > Proc: [[47380,1],0] > Errorcode: 59 > > NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. > You may or may not see output from other processes, depending on > exactly when Open MPI kills them. > -------------------------------------------------------------------------- > -------------------------------------------------------------------------- > prterun has exited due to process rank 0 with PID 0 on node MacMiniHome calling > "abort". This may have caused other processes in the application to be > terminated by signals sent by prterun (as reported here). > ????????????????????????????????????? > > and I have this after the code exits > > $ ipcs -m > IPC status from as of Thu Oct 24 21:17:39 IST 2024 > T ID KEY MODE OWNER GROUP > Shared Memory: > m 1572864 0x0000000b --rw-rw-rw- praveen staff > m 524289 0x0000000c --rw-rw-rw- praveen staff > m 655362 0x0000000d --rw-rw-rw- praveen staff > m 262147 0x0000000e --rw-rw-rw- praveen staff > m 262148 0x0000000f --rw-rw-rw- praveen staff > m 393221 0x0000000a --rw-rw-rw- praveen staff > > This is with petsc installed with miniforge, which I also use with clawpack. With spack installed petsc, I can run the ex89f example. > > Thanks > praveen > >> On 24 Oct 2024, at 7:55?PM, Barry Smith wrote: >> >> >> Ok, super strange, I've run many times on my Mac >> >> Can you please try to remove that allocated memory with ipcrm and then >> >> cd $PETSC_DIR/src/ksp/ksp/tutorials >> make ex89f >> mpiexec -n 3 ./ex89f -n 20 -mpi_linear_solver_server -mpi_linear_solver_server -mpi_linear_solver_server_ksp_view -ksp_monitor -ksp_converged_reason -ksp_view -mpi_linear_solver_server_minimum_count_per_rank 5 >> >> This does the same thing as the GeoClaw code but is much simpler. >> >> Barry >> >> >> >> >>> On Oct 23, 2024, at 10:55?PM, Praveen C wrote: >>> >>> I get very similar error on my mac with >>> >>> $ gfortran -v >>> Using built-in specs. >>> COLLECT_GCC=gfortran >>> COLLECT_LTO_WRAPPER=/opt/homebrew/Caskroom/miniforge/base/envs/claw/libexec/gcc/arm64-apple-darwin20.0.0/13.2.0/lto-wrapper >>> Target: arm64-apple-darwin20.0.0 >>> Configured with: ../configure --prefix=/opt/homebrew/Caskroom/miniforge/base/envs/claw --build=x86_64-apple-darwin13.4.0 --host=arm64-apple-darwin20.0.0 --target=arm64-apple-darwin20.0.0 --with-libiconv-prefix=/opt/homebrew/Caskroom/miniforge/base/envs/claw --enable-languages=fortran --disable-multilib --enable-checking=release --disable-bootstrap --disable-libssp --with-gmp=/opt/homebrew/Caskroom/miniforge/base/envs/claw --with-mpfr=/opt/homebrew/Caskroom/miniforge/base/envs/claw --with-mpc=/opt/homebrew/Caskroom/miniforge/base/envs/claw --with-isl=/opt/homebrew/Caskroom/miniforge/base/envs/claw --enable-darwin-at-rpath >>> Thread model: posix >>> Supported LTO compression algorithms: zlib >>> gcc version 13.2.0 (GCC) >>> >>> Before starting >>> >>> $ ipcs -m >>> IPC status from as of Thu Oct 24 08:02:11 IST 2024 >>> T ID KEY MODE OWNER GROUP >>> Shared Memory: >>> >>> and when I run the code >>> >>> Using a PETSc solver >>> Using Bouss equations from the start >>> rnode allocated... >>> node allocated... >>> listOfGrids allocated... >>> Storage allocated... >>> bndList allocated... >>> Gridding level 1 at t = 0.000000E+00: 4 grids with 10000 cells >>> Setting initial dt to 2.9999999999999999E-002 >>> max threads set to 1 >>> >>> Done reading data, starting computation ... >>> >>> Total zeta at initial time: 39269.907650665169 >>> GEOCLAW: Frame 0 output files done at time t = 0.000000D+00 >>> >>> [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >>> [0]PETSC ERROR: Petsc has generated inconsistent data >>> [0]PETSC ERROR: Unable to locate PCMPI allocated shared address 0x130698000 >>> [0]PETSC ERROR: WARNING! There are unused option(s) set! Could be the program crashed before usage or a spelling mistake, etc! >>> [0]PETSC ERROR: Option left: name:-ksp_type value: preonly source: file >>> [0]PETSC ERROR: Option left: name:-mpi_ksp_max_it value: 200 source: file >>> [0]PETSC ERROR: Option left: name:-mpi_ksp_reuse_preconditioner (no value) source: file >>> [0]PETSC ERROR: Option left: name:-mpi_ksp_rtol value: 1.e-9 source: file >>> [0]PETSC ERROR: Option left: name:-mpi_ksp_type value: gmres source: file >>> [0]PETSC ERROR: Option left: name:-mpi_linear_solver_server_view (no value) source: file >>> [0]PETSC ERROR: Option left: name:-mpi_pc_gamg_sym_graph value: true source: file >>> [0]PETSC ERROR: Option left: name:-mpi_pc_gamg_symmetrize_graph value: true source: file >>> [0]PETSC ERROR: Option left: name:-mpi_pc_type value: gamg source: file >>> [0]PETSC ERROR: Option left: name:-pc_mpi_minimum_count_per_rank value: 5000 source: file >>> [0]PETSC ERROR: Option left: name:-pc_type value: mpi source: file >>> [0]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!a83YNiE6vfZTxyhWxnJxmh95Z63M4HapLLI6b0Q_hFwekNKgjv2j3t61lo38kvy2gLDVjWO5IScFYBVUJaIxcoc$ for trouble shooting. >>> [0]PETSC ERROR: Petsc Release Version 3.22.0, Sep 28, 2024 >>> [0]PETSC ERROR: /Users/praveen/work/bouss/radial_flat/xgeoclaw with 6 MPI process(es) and PETSC_ARCH on MacMiniHome.local by praveen Thu Oct 24 08:04:27 2024 >>> [0]PETSC ERROR: Configure options: AR=arm64-apple-darwin20.0.0-ar CC=mpicc CXX=mpicxx FC=mpifort CFLAGS="-ftree-vectorize -fPIC -fstack-protector-strong -O2 -pipe -isystem /opt/homebrew/Caskroom/miniforge/base/envs/claw/include " CPPFLAGS="-D_FORTIFY_SOURCE=2 -isystem /opt/homebrew/Caskroom/miniforge/base/envs/claw/include -mmacosx-version-min=11.0 -mmacosx-version-min=11.0" CXXFLAGS="-ftree-vectorize -fPIC -fstack-protector-strong -O2 -pipe -stdlib=libc++ -fvisibility-inlines-hidden -fmessage-length=0 -isystem /opt/homebrew/Caskroom/miniforge/base/envs/claw/include " FFLAGS="-march=armv8.3-a -ftree-vectorize -fPIC -fno-stack-protector -O2 -pipe -isystem /opt/homebrew/Caskroom/miniforge/base/envs/claw/include " LDFLAGS="-Wl,-headerpad_max_install_names -Wl,-dead_strip_dylibs -Wl,-rpath,/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib -L/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib" LIBS="-Wl,-rpath,/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib -lmpi_mpifh -lgfortran" --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 --FOPTFLAGS=-O3 --with-clib-autodetect=0 --with-cxxlib-autodetect=0 --with-fortranlib-autodetect=0 --with-debugging=0 --with-blas-lib=libblas.dylib --with-lapack-lib=liblapack.dylib --with-yaml=1 --with-hdf5=1 --with-fftw=1 --with-hwloc=0 --with-hypre=1 --with-metis=1 --with-mpi=1 --with-mumps=1 --with-parmetis=1 --with-pthread=1 --with-ptscotch=1 --with-shared-libraries --with-ssl=0 --with-scalapack=1 --with-superlu=1 --with-superlu_dist=1 --with-superlu_dist-include=/opt/homebrew/Caskroom/miniforge/base/envs/claw/include/superlu-dist --with-superlu_dist-lib=-lsuperlu_dist --with-suitesparse=1 --with-suitesparse-dir=/opt/homebrew/Caskroom/miniforge/base/envs/claw --with-x=0 --with-scalar-type=real --with-cuda=0 --with-batch --prefix=/opt/homebrew/Caskroom/miniforge/base/envs/claw >>> [0]PETSC ERROR: #1 PetscShmgetMapAddresses() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/sys/utils/server.c:114 >>> [0]PETSC ERROR: #2 PCMPISetMat() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/ksp/pc/impls/mpi/pcmpi.c:269 >>> [0]PETSC ERROR: #3 PCSetUp_MPI() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/ksp/pc/impls/mpi/pcmpi.c:853 >>> [0]PETSC ERROR: #4 PCSetUp() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/ksp/pc/interface/precon.c:1071 >>> [0]PETSC ERROR: #5 KSPSetUp() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/ksp/ksp/interface/itfunc.c:415 >>> [0]PETSC ERROR: #6 KSPSolve_Private() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/ksp/ksp/interface/itfunc.c:826 >>> [0]PETSC ERROR: #7 KSPSolve() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/ksp/ksp/interface/itfunc.c:1075 >>> >>> Code does not progress and I kill it >>> >>> ^CTraceback (most recent call last): >>> File "/Users/praveen/Applications/clawpack/clawutil/src/python/clawutil/runclaw.py", line 341, in >>> runclaw(*args) >>> File "/Users/praveen/Applications/clawpack/clawutil/src/python/clawutil/runclaw.py", line 242, in runclaw >>> proc = subprocess.check_call(cmd_split, >>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ >>> File "/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib/python3.12/subprocess.py", line 408, in check_call >>> retcode = call(*popenargs, **kwargs) >>> ^^^^^^^^^^^^^^^^^^^^^^^^^^ >>> File "/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib/python3.12/subprocess.py", line 391, in call >>> return p.wait(timeout=timeout) >>> ^^^^^^^^^^^^^^^^^^^^^^^ >>> File "/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib/python3.12/subprocess.py", line 1264, in wait >>> return self._wait(timeout=timeout) >>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^ >>> File "/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib/python3.12/subprocess.py", line 2053, in _wait >>> (pid, sts) = self._try_wait(0) >>> ^^^^^^^^^^^^^^^^^ >>> File "/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib/python3.12/subprocess.py", line 2011, in _try_wait >>> (pid, sts) = os.waitpid(self.pid, wait_flags) >>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ >>> KeyboardInterrupt >>> make[1]: *** [output] Interrupt: 2 >>> make: *** [.output] Interrupt: 2 >>> >>> Now it says >>> >>> $ ipcs -m >>> IPC status from as of Thu Oct 24 08:05:06 IST 2024 >>> T ID KEY MODE OWNER GROUP >>> Shared Memory: >>> m 720896 0x0000000a --rw-rw-rw- praveen staff >>> >>> Thanks >>> praveen >>> >>>> On 23 Oct 2024, at 8:26?PM, Barry Smith wrote: >>>> >>>> >>>> Hmm, so it is creating the first shared memory region with ID of 10 (A in hex) puts it in a linked list in PETSc but then when it tries to find it in the linked list it cannot find it. >>>> >>>> I don't know how to reproduce this or debug it remotely. >>>> >>>> Can you build on a completely different machine or with completely different compilers? >>>> >>>> Barry >>>> >>>> >>>> >>>> >>>>> On Oct 23, 2024, at 10:31?AM, Praveen C wrote: >>>>> >>>>> I get same error and now it shows >>>>> >>>>> $ ipcs -m >>>>> >>>>> ------ Shared Memory Segments -------- >>>>> key shmid owner perms bytes nattch status >>>>> 0x0000000a 32788 praveen 666 240 6 >>>>> >>>>> Note that the code seems to be still running after printing those error message, but it is not printing any progress which it should do. >>>>> >>>>> Thanks >>>>> praveen >>>>> >>>>>> On 23 Oct 2024, at 7:56?PM, Barry Smith wrote: >>>>>> >>>>>> >>>>>> Try >>>>>> >>>>>> ipcrm -m 11 >>>>>> >>>>>> ipcs -m >>>>>> >>>>>> Try running the program again >>>>>> >>>>>> If failed check >>>>>> >>>>>> ipcs -m >>>>>> >>>>>> again >>>>>> >>>>>> >>>>>> >>>>>>> On Oct 23, 2024, at 10:20?AM, Praveen C wrote: >>>>>>> >>>>>>> Hello Barry >>>>>>> >>>>>>> I see this >>>>>>> >>>>>>> $ ipcs -m >>>>>>> >>>>>>> ------ Shared Memory Segments -------- >>>>>>> key shmid owner perms bytes nattch status >>>>>>> 0x0000000a 11 praveen 666 240 6 >>>>>>> >>>>>>> and I am observing same error as below. >>>>>>> >>>>>>> Thanks >>>>>>> praveen >>>>>>> >>>>>>>> On 23 Oct 2024, at 7:08?PM, Barry Smith wrote: >>>>>>>> >>>>>>>> >>>>>>>> Please take a look at the notes in https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/PetscShmgetAllocateArray/__;!!G_uCfscf7eWS!a83YNiE6vfZTxyhWxnJxmh95Z63M4HapLLI6b0Q_hFwekNKgjv2j3t61lo38kvy2gLDVjWO5IScFYBVU4eqTfOA$ For some reason your program is not able to access/use the Unix shared memory; check if you are already using the shared memory (so it is not available for a new run) or the limits are too low to access enough memory. >>>>>>>> >>>>>>>> Barry >>>>>>>> >>>>>>>>> On Oct 23, 2024, at 8:23?AM, Praveen C wrote: >>>>>>>>> >>>>>>>>> Dear all >>>>>>>>> >>>>>>>>> I am not able to run the boussinesq example from geoclaw using petsc at 3.22.0 >>>>>>>>> >>>>>>>>> https://urldefense.us/v3/__https://github.com/clawpack/geoclaw/tree/3303883f46572c58130d161986b8a87a57ca7816/examples/bouss__;!!G_uCfscf7eWS!e3VQ4NHKmXGstRsQW5vtI7fmKfUT9zmJkMJcPbcvPyIjicyfJpNoMgx3wZ-qyGcKNSjIkNZkzilec8MnHN6PMw$ >>>>>>>>> >>>>>>>>> It runs with petsc at 3.21.6 >>>>>>>>> >>>>>>>>> The error I get is given below. After printing this, the code does not progress. >>>>>>>>> >>>>>>>>> I use the following petsc options >>>>>>>>> >>>>>>>>> # set min numbers of matrix rows per MPI rank (default is 10000) >>>>>>>>> -mpi_linear_solve_minimum_count_per_rank 5000 >>>>>>>>> >>>>>>>>> >>>>>>>>> # Krylov linear solver: >>>>>>>>> -mpi_linear_solver_server >>>>>>>>> -mpi_linear_solver_server_view >>>>>>>>> -ksp_type gmres >>>>>>>>> -ksp_max_it 200 >>>>>>>>> -ksp_reuse_preconditioner >>>>>>>>> -ksp_rtol 1.e-9 >>>>>>>>> >>>>>>>>> # preconditioner: >>>>>>>>> -pc_type gamg >>>>>>>>> >>>>>>>>> I installed petsc and other dependencies for clawpack using miniforge. >>>>>>>>> >>>>>>>>> Thanks >>>>>>>>> pc >>>>>>>>> >>>>>>>>> ==> Use Bouss. in water deeper than 1.0000000000000000 Using a PETSc solver >>>>>>>>> Using Bouss equations from the start >>>>>>>>> rnode allocated... >>>>>>>>> node allocated... >>>>>>>>> listOfGrids allocated... >>>>>>>>> Storage allocated... >>>>>>>>> bndList allocated... >>>>>>>>> Gridding level 1 at t = 0.000000E+00: 4 grids with 10000 cells >>>>>>>>> Setting initial dt to 2.9999999999999999E-002 >>>>>>>>> max threads set to 6 >>>>>>>>> Done reading data, starting computation ... Total zeta at initial time: 39269.907650665169 GEOCLAW: Frame 0 output files done at time t = 0.000000D+00 >>>>>>>>> >>>>>>>>> [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >>>>>>>>> [0]PETSC ERROR: Petsc has generated inconsistent data >>>>>>>>> [0]PETSC ERROR: Unable to locate PCMPI allocated shared address 0x55e6d750ae20 >>>>>>>>> [0]PETSC ERROR: WARNING! There are unused option(s) set! Could be the program crashed before usage or a spelling mistake, etc! >>>>>>>>> [0]PETSC ERROR: Option left: name:-ksp_max_it value: 200 source: file >>>>>>>>> [0]PETSC ERROR: Option left: name:-ksp_reuse_preconditioner (no value) source: file >>>>>>>>> [0]PETSC ERROR: Option left: name:-ksp_rtol value: 1.e-9 source: file >>>>>>>>> [0]PETSC ERROR: Option left: name:-ksp_type value: gmres source: file >>>>>>>>> [0]PETSC ERROR: Option left: name:-mpi_linear_solve_minimum_count_per_rank value: 5000 source: file >>>>>>>>> [0]PETSC ERROR: Option left: name:-mpi_linear_solver_server_view (no value) source: file >>>>>>>>> [0]PETSC ERROR: Option left: name:-pc_type value: gamg source: file >>>>>>>>> [0]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!e3VQ4NHKmXGstRsQW5vtI7fmKfUT9zmJkMJcPbcvPyIjicyfJpNoMgx3wZ-qyGcKNSjIkNZkzilec8MvNjNo7A$ for trouble shooting. >>>>>>>>> [0]PETSC ERROR: Petsc Release Version 3.22.0, Sep 28, 2024 [0]PETSC ERROR: /home/praveen/bouss/radial_flat/xgeoclaw with 6 MPI process(es) and PETSC_ARCH on euler by praveen Thu Oct 17 21:49:54 2024 >>>>>>>>> [0]PETSC ERROR: Configure options: AR=${PREFIX}/bin/x86_64-conda-linux-gnu-ar CC=mpicc CXX=mpicxx FC=mpifort CFLAGS="-march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /opt/miniforge/envs/claw/include " CPPFLAGS="-DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /opt/miniforge/envs/claw/include" CXXFLAGS="-fvisibility-inlines-hidden -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /opt/miniforge/envs/claw/include " FFLAGS="-march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /opt/miniforge/envs/claw/include -Wl,--no-as-needed" LDFLAGS="-pthread -fopenmp -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,--allow-shlib-undefined -Wl,-rpath,/opt/miniforge/envs/claw/lib -Wl,-rpath-link,/opt/miniforge/envs/claw/lib -L/opt/miniforge/envs/claw/lib -Wl,-rpath-link,/opt/miniforge/envs/claw/lib" LIBS="-Wl,-rpath,/opt/miniforge/envs/claw/lib -lmpi_mpifh -lgfortran" --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 --FOPTFLAGS=-O3 --with-clib-autodetect=0 --with-cxxlib-autodetect=0 --with-fortranlib-autodetect=0 --with-debugging=0 --with-blas-lib=libblas.so --with-lapack-lib=liblapack.so --with-yaml=1 --with-hdf5=1 --with-fftw=1 --with-hwloc=0 --with-hypre=1 --with-metis=1 --with-mpi=1 --with-mumps=1 --with-parmetis=1 --with-pthread=1 --with-ptscotch=1 --with-shared-libraries --with-ssl=0 --with-scalapack=1 --with-superlu=1 --with-superlu_dist=1 --with-superlu_dist-include=/opt/miniforge/envs/claw/include/superlu-dist --with-superlu_dist-lib=-lsuperlu_dist --with-suitesparse=1 --with-suitesparse-dir=/opt/miniforge/envs/claw --with-x=0 --with-scalar-type=real --with-cuda=0 --prefix=/opt/miniforge/envs/claw >>>>>>>>> [0]PETSC ERROR: #1 PetscShmgetMapAddresses() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/sys/utils/server.c:114 >>>>>>>>> [0]PETSC ERROR: #2 PCMPISetMat() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/pc/impls/mpi/pcmpi.c:269 >>>>>>>>> [0]PETSC ERROR: #3 PCSetUp_MPI() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/pc/impls/mpi/pcmpi.c:853 >>>>>>>>> [0]PETSC ERROR: #4 PCSetUp() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/pc/interface/precon.c:1071 >>>>>>>>> [0]PETSC ERROR: #5 KSPSetUp() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/ksp/interface/itfunc.c:415 >>>>>>>>> [0]PETSC ERROR: #6 KSPSolve_Private() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/ksp/interface/itfunc.c:826 >>>>>>>>> [0]PETSC ERROR: #7 KSPSolve() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/ksp/interface/itfunc.c:1075 >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cpraveen at gmail.com Thu Oct 24 11:17:31 2024 From: cpraveen at gmail.com (Praveen C) Date: Thu, 24 Oct 2024 21:47:31 +0530 Subject: [petsc-users] Problem running geoclaw example with petsc 3.22 In-Reply-To: <490B5A4A-BBD8-490A-91FD-7CEC9E01A99B@petsc.dev> References: <1C63D19F-6EBC-43A3-8B40-65EA34991AFA@gmail.com> <0136027D-C2CE-47D0-AF97-ACDE1DA0C382@gmail.com> <4726A341-54DB-4592-BBAE-843AF271F28E@petsc.dev> <27B47E2D-00D5-4326-95A9-7595DFF0E952@gmail.com> <490B5A4A-BBD8-490A-91FD-7CEC9E01A99B@petsc.dev> Message-ID: Hello Barry I use this script to install clawpack and required dependencies https://urldefense.us/v3/__https://github.com/cpraveen/cfdlab/blob/master/bin/clawpack.sh__;!!G_uCfscf7eWS!fvuzf9vr_6FD4jbeIajCoXJnAXTzstvxWHD2FAL9lFHroogk1p_mvSk5I20ogPCA3z3OI8jNMbzr3OApoFWW8w$ See lines 125-132 You can use it like this export CLAW=/path/to/where/you/want/clawpack bash clawpack.sh v5.11.0 This will git pull clawpack and creates a conda env called ?claw? and installs inside that. I have not specified petsc version in this script, but latest miniforge should install petsc at 3.22 Thank you praveen > On 24 Oct 2024, at 9:37?PM, Barry Smith wrote: > > > Good, some helpful information with the runs you made. > > In the crash below it made more progress it was able to allocate multiple regions of shared memory and access them. I don't know why it would crash later. > > Can you tell me all the steps with miniforge (which seems to be related to the failure) you use? I've never used miniforge. > > If I can get to an environment that reproduces the problem I can debug it and fix it. > > Barry > > >> On Oct 24, 2024, at 11:49?AM, Praveen C wrote: >> >> I get this >> >> $ mpiexec -n 3 ./ex89f -n 20 -mpi_linear_solver_server -mpi_linear_solver_server -mpi_linear_solver_server_ksp_view -ksp_monitor -ksp_converged_reason -ksp_view -mpi_linear_solver_server_minimum_count_per_rank 5 >> [0]PETSC ERROR: ------------------------------------------------------------------------ >> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range >> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >> [0]PETSC ERROR: or see https://urldefense.us/v3/__https://petsc.org/release/faq/*valgrind__;Iw!!G_uCfscf7eWS!fvuzf9vr_6FD4jbeIajCoXJnAXTzstvxWHD2FAL9lFHroogk1p_mvSk5I20ogPCA3z3OI8jNMbzr3OAkxkT2HQ$ and https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!fvuzf9vr_6FD4jbeIajCoXJnAXTzstvxWHD2FAL9lFHroogk1p_mvSk5I20ogPCA3z3OI8jNMbzr3ODvXnLz2A$ >> [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run >> [0]PETSC ERROR: to get more information on the crash. >> [0]PETSC ERROR: Run with -malloc_debug to check if memory corruption is causing the crash. >> -------------------------------------------------------------------------- >> MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_SELF >> Proc: [[47380,1],0] >> Errorcode: 59 >> >> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. >> You may or may not see output from other processes, depending on >> exactly when Open MPI kills them. >> -------------------------------------------------------------------------- >> -------------------------------------------------------------------------- >> prterun has exited due to process rank 0 with PID 0 on node MacMiniHome calling >> "abort". This may have caused other processes in the application to be >> terminated by signals sent by prterun (as reported here). >> ????????????????????????????????????? >> >> and I have this after the code exits >> >> $ ipcs -m >> IPC status from as of Thu Oct 24 21:17:39 IST 2024 >> T ID KEY MODE OWNER GROUP >> Shared Memory: >> m 1572864 0x0000000b --rw-rw-rw- praveen staff >> m 524289 0x0000000c --rw-rw-rw- praveen staff >> m 655362 0x0000000d --rw-rw-rw- praveen staff >> m 262147 0x0000000e --rw-rw-rw- praveen staff >> m 262148 0x0000000f --rw-rw-rw- praveen staff >> m 393221 0x0000000a --rw-rw-rw- praveen staff >> >> This is with petsc installed with miniforge, which I also use with clawpack. With spack installed petsc, I can run the ex89f example. >> >> Thanks >> praveen >> >>> On 24 Oct 2024, at 7:55?PM, Barry Smith wrote: >>> >>> >>> Ok, super strange, I've run many times on my Mac >>> >>> Can you please try to remove that allocated memory with ipcrm and then >>> >>> cd $PETSC_DIR/src/ksp/ksp/tutorials >>> make ex89f >>> mpiexec -n 3 ./ex89f -n 20 -mpi_linear_solver_server -mpi_linear_solver_server -mpi_linear_solver_server_ksp_view -ksp_monitor -ksp_converged_reason -ksp_view -mpi_linear_solver_server_minimum_count_per_rank 5 >>> >>> This does the same thing as the GeoClaw code but is much simpler. >>> >>> Barry >>> >>> >>> >>> >>>> On Oct 23, 2024, at 10:55?PM, Praveen C wrote: >>>> >>>> I get very similar error on my mac with >>>> >>>> $ gfortran -v >>>> Using built-in specs. >>>> COLLECT_GCC=gfortran >>>> COLLECT_LTO_WRAPPER=/opt/homebrew/Caskroom/miniforge/base/envs/claw/libexec/gcc/arm64-apple-darwin20.0.0/13.2.0/lto-wrapper >>>> Target: arm64-apple-darwin20.0.0 >>>> Configured with: ../configure --prefix=/opt/homebrew/Caskroom/miniforge/base/envs/claw --build=x86_64-apple-darwin13.4.0 --host=arm64-apple-darwin20.0.0 --target=arm64-apple-darwin20.0.0 --with-libiconv-prefix=/opt/homebrew/Caskroom/miniforge/base/envs/claw --enable-languages=fortran --disable-multilib --enable-checking=release --disable-bootstrap --disable-libssp --with-gmp=/opt/homebrew/Caskroom/miniforge/base/envs/claw --with-mpfr=/opt/homebrew/Caskroom/miniforge/base/envs/claw --with-mpc=/opt/homebrew/Caskroom/miniforge/base/envs/claw --with-isl=/opt/homebrew/Caskroom/miniforge/base/envs/claw --enable-darwin-at-rpath >>>> Thread model: posix >>>> Supported LTO compression algorithms: zlib >>>> gcc version 13.2.0 (GCC) >>>> >>>> Before starting >>>> >>>> $ ipcs -m >>>> IPC status from as of Thu Oct 24 08:02:11 IST 2024 >>>> T ID KEY MODE OWNER GROUP >>>> Shared Memory: >>>> >>>> and when I run the code >>>> >>>> Using a PETSc solver >>>> Using Bouss equations from the start >>>> rnode allocated... >>>> node allocated... >>>> listOfGrids allocated... >>>> Storage allocated... >>>> bndList allocated... >>>> Gridding level 1 at t = 0.000000E+00: 4 grids with 10000 cells >>>> Setting initial dt to 2.9999999999999999E-002 >>>> max threads set to 1 >>>> >>>> Done reading data, starting computation ... >>>> >>>> Total zeta at initial time: 39269.907650665169 >>>> GEOCLAW: Frame 0 output files done at time t = 0.000000D+00 >>>> >>>> [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >>>> [0]PETSC ERROR: Petsc has generated inconsistent data >>>> [0]PETSC ERROR: Unable to locate PCMPI allocated shared address 0x130698000 >>>> [0]PETSC ERROR: WARNING! There are unused option(s) set! Could be the program crashed before usage or a spelling mistake, etc! >>>> [0]PETSC ERROR: Option left: name:-ksp_type value: preonly source: file >>>> [0]PETSC ERROR: Option left: name:-mpi_ksp_max_it value: 200 source: file >>>> [0]PETSC ERROR: Option left: name:-mpi_ksp_reuse_preconditioner (no value) source: file >>>> [0]PETSC ERROR: Option left: name:-mpi_ksp_rtol value: 1.e-9 source: file >>>> [0]PETSC ERROR: Option left: name:-mpi_ksp_type value: gmres source: file >>>> [0]PETSC ERROR: Option left: name:-mpi_linear_solver_server_view (no value) source: file >>>> [0]PETSC ERROR: Option left: name:-mpi_pc_gamg_sym_graph value: true source: file >>>> [0]PETSC ERROR: Option left: name:-mpi_pc_gamg_symmetrize_graph value: true source: file >>>> [0]PETSC ERROR: Option left: name:-mpi_pc_type value: gamg source: file >>>> [0]PETSC ERROR: Option left: name:-pc_mpi_minimum_count_per_rank value: 5000 source: file >>>> [0]PETSC ERROR: Option left: name:-pc_type value: mpi source: file >>>> [0]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!fvuzf9vr_6FD4jbeIajCoXJnAXTzstvxWHD2FAL9lFHroogk1p_mvSk5I20ogPCA3z3OI8jNMbzr3ODvXnLz2A$ for trouble shooting. >>>> [0]PETSC ERROR: Petsc Release Version 3.22.0, Sep 28, 2024 >>>> [0]PETSC ERROR: /Users/praveen/work/bouss/radial_flat/xgeoclaw with 6 MPI process(es) and PETSC_ARCH on MacMiniHome.local by praveen Thu Oct 24 08:04:27 2024 >>>> [0]PETSC ERROR: Configure options: AR=arm64-apple-darwin20.0.0-ar CC=mpicc CXX=mpicxx FC=mpifort CFLAGS="-ftree-vectorize -fPIC -fstack-protector-strong -O2 -pipe -isystem /opt/homebrew/Caskroom/miniforge/base/envs/claw/include " CPPFLAGS="-D_FORTIFY_SOURCE=2 -isystem /opt/homebrew/Caskroom/miniforge/base/envs/claw/include -mmacosx-version-min=11.0 -mmacosx-version-min=11.0" CXXFLAGS="-ftree-vectorize -fPIC -fstack-protector-strong -O2 -pipe -stdlib=libc++ -fvisibility-inlines-hidden -fmessage-length=0 -isystem /opt/homebrew/Caskroom/miniforge/base/envs/claw/include " FFLAGS="-march=armv8.3-a -ftree-vectorize -fPIC -fno-stack-protector -O2 -pipe -isystem /opt/homebrew/Caskroom/miniforge/base/envs/claw/include " LDFLAGS="-Wl,-headerpad_max_install_names -Wl,-dead_strip_dylibs -Wl,-rpath,/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib -L/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib" LIBS="-Wl,-rpath,/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib -lmpi_mpifh -lgfortran" --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 --FOPTFLAGS=-O3 --with-clib-autodetect=0 --with-cxxlib-autodetect=0 --with-fortranlib-autodetect=0 --with-debugging=0 --with-blas-lib=libblas.dylib --with-lapack-lib=liblapack.dylib --with-yaml=1 --with-hdf5=1 --with-fftw=1 --with-hwloc=0 --with-hypre=1 --with-metis=1 --with-mpi=1 --with-mumps=1 --with-parmetis=1 --with-pthread=1 --with-ptscotch=1 --with-shared-libraries --with-ssl=0 --with-scalapack=1 --with-superlu=1 --with-superlu_dist=1 --with-superlu_dist-include=/opt/homebrew/Caskroom/miniforge/base/envs/claw/include/superlu-dist --with-superlu_dist-lib=-lsuperlu_dist --with-suitesparse=1 --with-suitesparse-dir=/opt/homebrew/Caskroom/miniforge/base/envs/claw --with-x=0 --with-scalar-type=real --with-cuda=0 --with-batch --prefix=/opt/homebrew/Caskroom/miniforge/base/envs/claw >>>> [0]PETSC ERROR: #1 PetscShmgetMapAddresses() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/sys/utils/server.c:114 >>>> [0]PETSC ERROR: #2 PCMPISetMat() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/ksp/pc/impls/mpi/pcmpi.c:269 >>>> [0]PETSC ERROR: #3 PCSetUp_MPI() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/ksp/pc/impls/mpi/pcmpi.c:853 >>>> [0]PETSC ERROR: #4 PCSetUp() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/ksp/pc/interface/precon.c:1071 >>>> [0]PETSC ERROR: #5 KSPSetUp() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/ksp/ksp/interface/itfunc.c:415 >>>> [0]PETSC ERROR: #6 KSPSolve_Private() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/ksp/ksp/interface/itfunc.c:826 >>>> [0]PETSC ERROR: #7 KSPSolve() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/ksp/ksp/interface/itfunc.c:1075 >>>> >>>> Code does not progress and I kill it >>>> >>>> ^CTraceback (most recent call last): >>>> File "/Users/praveen/Applications/clawpack/clawutil/src/python/clawutil/runclaw.py", line 341, in >>>> runclaw(*args) >>>> File "/Users/praveen/Applications/clawpack/clawutil/src/python/clawutil/runclaw.py", line 242, in runclaw >>>> proc = subprocess.check_call(cmd_split, >>>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ >>>> File "/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib/python3.12/subprocess.py", line 408, in check_call >>>> retcode = call(*popenargs, **kwargs) >>>> ^^^^^^^^^^^^^^^^^^^^^^^^^^ >>>> File "/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib/python3.12/subprocess.py", line 391, in call >>>> return p.wait(timeout=timeout) >>>> ^^^^^^^^^^^^^^^^^^^^^^^ >>>> File "/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib/python3.12/subprocess.py", line 1264, in wait >>>> return self._wait(timeout=timeout) >>>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^ >>>> File "/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib/python3.12/subprocess.py", line 2053, in _wait >>>> (pid, sts) = self._try_wait(0) >>>> ^^^^^^^^^^^^^^^^^ >>>> File "/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib/python3.12/subprocess.py", line 2011, in _try_wait >>>> (pid, sts) = os.waitpid(self.pid, wait_flags) >>>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ >>>> KeyboardInterrupt >>>> make[1]: *** [output] Interrupt: 2 >>>> make: *** [.output] Interrupt: 2 >>>> >>>> Now it says >>>> >>>> $ ipcs -m >>>> IPC status from as of Thu Oct 24 08:05:06 IST 2024 >>>> T ID KEY MODE OWNER GROUP >>>> Shared Memory: >>>> m 720896 0x0000000a --rw-rw-rw- praveen staff >>>> >>>> Thanks >>>> praveen >>>> >>>>> On 23 Oct 2024, at 8:26?PM, Barry Smith wrote: >>>>> >>>>> >>>>> Hmm, so it is creating the first shared memory region with ID of 10 (A in hex) puts it in a linked list in PETSc but then when it tries to find it in the linked list it cannot find it. >>>>> >>>>> I don't know how to reproduce this or debug it remotely. >>>>> >>>>> Can you build on a completely different machine or with completely different compilers? >>>>> >>>>> Barry >>>>> >>>>> >>>>> >>>>> >>>>>> On Oct 23, 2024, at 10:31?AM, Praveen C wrote: >>>>>> >>>>>> I get same error and now it shows >>>>>> >>>>>> $ ipcs -m >>>>>> >>>>>> ------ Shared Memory Segments -------- >>>>>> key shmid owner perms bytes nattch status >>>>>> 0x0000000a 32788 praveen 666 240 6 >>>>>> >>>>>> Note that the code seems to be still running after printing those error message, but it is not printing any progress which it should do. >>>>>> >>>>>> Thanks >>>>>> praveen >>>>>> >>>>>>> On 23 Oct 2024, at 7:56?PM, Barry Smith wrote: >>>>>>> >>>>>>> >>>>>>> Try >>>>>>> >>>>>>> ipcrm -m 11 >>>>>>> >>>>>>> ipcs -m >>>>>>> >>>>>>> Try running the program again >>>>>>> >>>>>>> If failed check >>>>>>> >>>>>>> ipcs -m >>>>>>> >>>>>>> again >>>>>>> >>>>>>> >>>>>>> >>>>>>>> On Oct 23, 2024, at 10:20?AM, Praveen C wrote: >>>>>>>> >>>>>>>> Hello Barry >>>>>>>> >>>>>>>> I see this >>>>>>>> >>>>>>>> $ ipcs -m >>>>>>>> >>>>>>>> ------ Shared Memory Segments -------- >>>>>>>> key shmid owner perms bytes nattch status >>>>>>>> 0x0000000a 11 praveen 666 240 6 >>>>>>>> >>>>>>>> and I am observing same error as below. >>>>>>>> >>>>>>>> Thanks >>>>>>>> praveen >>>>>>>> >>>>>>>>> On 23 Oct 2024, at 7:08?PM, Barry Smith wrote: >>>>>>>>> >>>>>>>>> >>>>>>>>> Please take a look at the notes in https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/PetscShmgetAllocateArray/__;!!G_uCfscf7eWS!fvuzf9vr_6FD4jbeIajCoXJnAXTzstvxWHD2FAL9lFHroogk1p_mvSk5I20ogPCA3z3OI8jNMbzr3OD4MyLSQw$ For some reason your program is not able to access/use the Unix shared memory; check if you are already using the shared memory (so it is not available for a new run) or the limits are too low to access enough memory. >>>>>>>>> >>>>>>>>> Barry >>>>>>>>> >>>>>>>>>> On Oct 23, 2024, at 8:23?AM, Praveen C wrote: >>>>>>>>>> >>>>>>>>>> Dear all >>>>>>>>>> >>>>>>>>>> I am not able to run the boussinesq example from geoclaw using petsc at 3.22.0 >>>>>>>>>> >>>>>>>>>> https://urldefense.us/v3/__https://github.com/clawpack/geoclaw/tree/3303883f46572c58130d161986b8a87a57ca7816/examples/bouss__;!!G_uCfscf7eWS!e3VQ4NHKmXGstRsQW5vtI7fmKfUT9zmJkMJcPbcvPyIjicyfJpNoMgx3wZ-qyGcKNSjIkNZkzilec8MnHN6PMw$ >>>>>>>>>> >>>>>>>>>> It runs with petsc at 3.21.6 >>>>>>>>>> >>>>>>>>>> The error I get is given below. After printing this, the code does not progress. >>>>>>>>>> >>>>>>>>>> I use the following petsc options >>>>>>>>>> >>>>>>>>>> # set min numbers of matrix rows per MPI rank (default is 10000) >>>>>>>>>> -mpi_linear_solve_minimum_count_per_rank 5000 >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> # Krylov linear solver: >>>>>>>>>> -mpi_linear_solver_server >>>>>>>>>> -mpi_linear_solver_server_view >>>>>>>>>> -ksp_type gmres >>>>>>>>>> -ksp_max_it 200 >>>>>>>>>> -ksp_reuse_preconditioner >>>>>>>>>> -ksp_rtol 1.e-9 >>>>>>>>>> >>>>>>>>>> # preconditioner: >>>>>>>>>> -pc_type gamg >>>>>>>>>> >>>>>>>>>> I installed petsc and other dependencies for clawpack using miniforge. >>>>>>>>>> >>>>>>>>>> Thanks >>>>>>>>>> pc >>>>>>>>>> >>>>>>>>>> ==> Use Bouss. in water deeper than 1.0000000000000000 Using a PETSc solver >>>>>>>>>> Using Bouss equations from the start >>>>>>>>>> rnode allocated... >>>>>>>>>> node allocated... >>>>>>>>>> listOfGrids allocated... >>>>>>>>>> Storage allocated... >>>>>>>>>> bndList allocated... >>>>>>>>>> Gridding level 1 at t = 0.000000E+00: 4 grids with 10000 cells >>>>>>>>>> Setting initial dt to 2.9999999999999999E-002 >>>>>>>>>> max threads set to 6 >>>>>>>>>> Done reading data, starting computation ... Total zeta at initial time: 39269.907650665169 GEOCLAW: Frame 0 output files done at time t = 0.000000D+00 >>>>>>>>>> >>>>>>>>>> [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >>>>>>>>>> [0]PETSC ERROR: Petsc has generated inconsistent data >>>>>>>>>> [0]PETSC ERROR: Unable to locate PCMPI allocated shared address 0x55e6d750ae20 >>>>>>>>>> [0]PETSC ERROR: WARNING! There are unused option(s) set! Could be the program crashed before usage or a spelling mistake, etc! >>>>>>>>>> [0]PETSC ERROR: Option left: name:-ksp_max_it value: 200 source: file >>>>>>>>>> [0]PETSC ERROR: Option left: name:-ksp_reuse_preconditioner (no value) source: file >>>>>>>>>> [0]PETSC ERROR: Option left: name:-ksp_rtol value: 1.e-9 source: file >>>>>>>>>> [0]PETSC ERROR: Option left: name:-ksp_type value: gmres source: file >>>>>>>>>> [0]PETSC ERROR: Option left: name:-mpi_linear_solve_minimum_count_per_rank value: 5000 source: file >>>>>>>>>> [0]PETSC ERROR: Option left: name:-mpi_linear_solver_server_view (no value) source: file >>>>>>>>>> [0]PETSC ERROR: Option left: name:-pc_type value: gamg source: file >>>>>>>>>> [0]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!e3VQ4NHKmXGstRsQW5vtI7fmKfUT9zmJkMJcPbcvPyIjicyfJpNoMgx3wZ-qyGcKNSjIkNZkzilec8MvNjNo7A$ for trouble shooting. >>>>>>>>>> [0]PETSC ERROR: Petsc Release Version 3.22.0, Sep 28, 2024 [0]PETSC ERROR: /home/praveen/bouss/radial_flat/xgeoclaw with 6 MPI process(es) and PETSC_ARCH on euler by praveen Thu Oct 17 21:49:54 2024 >>>>>>>>>> [0]PETSC ERROR: Configure options: AR=${PREFIX}/bin/x86_64-conda-linux-gnu-ar CC=mpicc CXX=mpicxx FC=mpifort CFLAGS="-march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /opt/miniforge/envs/claw/include " CPPFLAGS="-DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /opt/miniforge/envs/claw/include" CXXFLAGS="-fvisibility-inlines-hidden -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /opt/miniforge/envs/claw/include " FFLAGS="-march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /opt/miniforge/envs/claw/include -Wl,--no-as-needed" LDFLAGS="-pthread -fopenmp -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,--allow-shlib-undefined -Wl,-rpath,/opt/miniforge/envs/claw/lib -Wl,-rpath-link,/opt/miniforge/envs/claw/lib -L/opt/miniforge/envs/claw/lib -Wl,-rpath-link,/opt/miniforge/envs/claw/lib" LIBS="-Wl,-rpath,/opt/miniforge/envs/claw/lib -lmpi_mpifh -lgfortran" --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 --FOPTFLAGS=-O3 --with-clib-autodetect=0 --with-cxxlib-autodetect=0 --with-fortranlib-autodetect=0 --with-debugging=0 --with-blas-lib=libblas.so --with-lapack-lib=liblapack.so --with-yaml=1 --with-hdf5=1 --with-fftw=1 --with-hwloc=0 --with-hypre=1 --with-metis=1 --with-mpi=1 --with-mumps=1 --with-parmetis=1 --with-pthread=1 --with-ptscotch=1 --with-shared-libraries --with-ssl=0 --with-scalapack=1 --with-superlu=1 --with-superlu_dist=1 --with-superlu_dist-include=/opt/miniforge/envs/claw/include/superlu-dist --with-superlu_dist-lib=-lsuperlu_dist --with-suitesparse=1 --with-suitesparse-dir=/opt/miniforge/envs/claw --with-x=0 --with-scalar-type=real --with-cuda=0 --prefix=/opt/miniforge/envs/claw >>>>>>>>>> [0]PETSC ERROR: #1 PetscShmgetMapAddresses() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/sys/utils/server.c:114 >>>>>>>>>> [0]PETSC ERROR: #2 PCMPISetMat() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/pc/impls/mpi/pcmpi.c:269 >>>>>>>>>> [0]PETSC ERROR: #3 PCSetUp_MPI() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/pc/impls/mpi/pcmpi.c:853 >>>>>>>>>> [0]PETSC ERROR: #4 PCSetUp() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/pc/interface/precon.c:1071 >>>>>>>>>> [0]PETSC ERROR: #5 KSPSetUp() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/ksp/interface/itfunc.c:415 >>>>>>>>>> [0]PETSC ERROR: #6 KSPSolve_Private() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/ksp/interface/itfunc.c:826 >>>>>>>>>> [0]PETSC ERROR: #7 KSPSolve() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/ksp/interface/itfunc.c:1075 >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From matteo.semplice at uninsubria.it Thu Oct 24 15:02:10 2024 From: matteo.semplice at uninsubria.it (Semplice Matteo) Date: Thu, 24 Oct 2024 20:02:10 +0000 Subject: [petsc-users] R: local/global DMPlex Vec output In-Reply-To: References: Message-ID: Hi, ??????I tried again today and have (re?)discovered this example https://urldefense.us/v3/__https://petsc.org/release/src/dm/impls/plex/tutorials/ex14.c.html__;!!G_uCfscf7eWS!fJpyioZNC3oN15Qm8-PTF_nxD_NYc-Ou_-moVFROjwZyZREL7e0PwP-rDwjCcynulDAeLGL1WqkN7tyuZepLZ0L7BfI88y7favtI0g$ , but I cannot understand if in my case I should call PetscSFCreateSectionSF and, if so, how should I then activate the returned SF. Matteo ________________________________ Da: Semplice Matteo Inviato: marted? 22 ottobre 2024 00:24 A: Matthew Knepley Cc: PETSc Oggetto: Re: [petsc-users] local/global DMPlex Vec output Dear Matt, I guess you're right: thresholding by rank==0 and rank==1 in paraview reveals that it is indeed the overlap cells that are appear twice in the output. The attached file is not exactly minimal but hopefully short enough. If I run it in serial, all is ok, but with mpirun -np 2 ./saveDemo it creates a 10x10 grid, but I get "output.vtu" with a total of 120 cells. However the pointSF of the DMPlex seems correct. Thanks Matteo Il 21/10/24 19:15, Matthew Knepley ha scritto: On Mon, Oct 21, 2024 at 12:22?PM Matteo Semplice via petsc-users > wrote: Dear petsc-users, I am having issues with output of parallel data attached to a DMPlex (or maybe more fundamental ones about DMPlex...). So I currently 1. create a DMPlex (DMPlexCreateGmshFromFile or DMPlexCreateBoxMesh) 2. partition it 3. and create a section for my data layout with DMPlexCreateSection(ctx.dmMesh, NULL, numComp, numDof, numBC, NULL, NULL, NULL, NULL, &sUavg) 4. DMSetLocalSection(ctx.dmMesh, sUavg) 5. create solLoc and solGlob vectors with DMCreateGlobalVector and DMCreateLocalVector 6. solve .... 7. VecView(ctx.solGlob, vtkViewer) on a .vtu file but when I load data in ParaView I get more cells than expected and it is as if the cells in the halo are put twice in output. (I could create a MWE if the above is not clear) I think we need an MWE here, because from the explanation above, it should work. However, I can try to guess the problem. When you partition the mesh, I am guessing that you have cells in the overlap. These cells must be in the point SF in order for the global section to give them a unique owner. Perhaps something has gone wrong here. Thanks, Matt I guess that the culprit is point (4), but if I replace it with DMSetGlobalSection then I cannot create the local vector at point (5). How should I handle this properly? In my code I need to create both local and global vectors, to perform at least GlobalToLocal and to save the global data. (On a side note, I tried also HDF5 but then it complains about the DM not having a DS...; really, any working solution that allows data to be explored with Paraview is fine) Thanks for any advice! Matteo Semplice -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fJpyioZNC3oN15Qm8-PTF_nxD_NYc-Ou_-moVFROjwZyZREL7e0PwP-rDwjCcynulDAeLGL1WqkN7tyuZepLZ0L7BfI88y7jE-NoBQ$ -- --- Professore Associato in Analisi Numerica Dipartimento di Scienza e Alta Tecnologia Universit? degli Studi dell'Insubria Via Valleggio, 11 - Como -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Oct 24 15:20:03 2024 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 24 Oct 2024 16:20:03 -0400 Subject: [petsc-users] local/global DMPlex Vec output In-Reply-To: References: Message-ID: I just looked at the code. The VTK code is very old, and does not check for cell overlap. We have been recommending that people use either HDF5 or CGNS, both of which work in this case I believe. I can fix VTK if that is what you want, but it might take me a little while as it is very busy at work right now. However, if you output HDF5, then you can run ./lib/petsc/bin/petsc_gen_xdmf.py mesh.h5 and it will generate an XDMF file so you can load it into ParaView. Or you can output CGNS which I think ParaView understands. Thanks, Matt On Thu, Oct 24, 2024 at 4:02?PM Semplice Matteo < matteo.semplice at uninsubria.it> wrote: > Hi, > I tried again today and have (re?)discovered this example > https://urldefense.us/v3/__https://petsc.org/release/src/dm/impls/plex/tutorials/ex14.c.html__;!!G_uCfscf7eWS!djj05CUxAKqqqzrdcdpxybBp4r8LCeoh2YHKGXWZBrw8af69K7SN2w0jfqyuP3lDKjRWhw4qzK7MlWA5PAGG$ , but I > cannot understand if in my case I should call PetscSFCreateSectionSF > and, > if so, how should I then activate the returned SF. > Matteo > ------------------------------ > *Da:* Semplice Matteo > *Inviato:* marted? 22 ottobre 2024 00:24 > *A:* Matthew Knepley > *Cc:* PETSc > *Oggetto:* Re: [petsc-users] local/global DMPlex Vec output > > > Dear Matt, > > I guess you're right: thresholding by rank==0 and rank==1 in paraview > reveals that it is indeed the overlap cells that are appear twice in the > output. > > The attached file is not exactly minimal but hopefully short enough. If I > run it in serial, all is ok, but with > > mpirun -np 2 ./saveDemo > > it creates a 10x10 grid, but I get "output.vtu" with a total of 120 cells. > However the pointSF of the DMPlex seems correct. > > Thanks > > Matteo > Il 21/10/24 19:15, Matthew Knepley ha scritto: > > On Mon, Oct 21, 2024 at 12:22?PM Matteo Semplice via petsc-users < > petsc-users at mcs.anl.gov> wrote: > > Dear petsc-users, > > I am having issues with output of parallel data attached to a DMPlex > (or maybe more fundamental ones about DMPlex...). > > So I currently > > 1. create a DMPlex (DMPlexCreateGmshFromFile or DMPlexCreateBoxMesh) > 2. partition it > 3. and create a section for my data layout with > DMPlexCreateSection(ctx.dmMesh, NULL, numComp, numDof, numBC, NULL, NULL, > NULL, NULL, &sUavg) > 4. DMSetLocalSection(ctx.dmMesh, sUavg) > 5. create solLoc and solGlob vectors with DMCreateGlobalVector and > DMCreateLocalVector > 6. solve .... > 7. VecView(ctx.solGlob, vtkViewer) on a .vtu file > > but when I load data in ParaView I get more cells than expected and it is > as if the cells in the halo are put twice in output. (I could create a MWE > if the above is not clear) > > I think we need an MWE here, because from the explanation above, it should > work. > > However, I can try to guess the problem. When you partition the mesh, I am > guessing that you have cells in the overlap. These cells > must be in the point SF in order for the global section to give them a > unique owner. Perhaps something has gone wrong here. > > Thanks, > > Matt > > I guess that the culprit is point (4), but if I replace it with > DMSetGlobalSection then I cannot create the local vector at point (5). > > How should I handle this properly? In my code I need to create both local > and global vectors, to perform at least GlobalToLocal and to save the > global data. > > (On a side note, I tried also HDF5 but then it complains about the DM not > having a DS...; really, any working solution that allows data to be > explored with Paraview is fine) > > Thanks for any advice! > > Matteo Semplice > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!djj05CUxAKqqqzrdcdpxybBp4r8LCeoh2YHKGXWZBrw8af69K7SN2w0jfqyuP3lDKjRWhw4qzK7MlTV94teN$ > > > -- > --- > Professore Associato in Analisi Numerica > Dipartimento di Scienza e Alta Tecnologia > Universit? degli Studi dell'Insubria > Via Valleggio, 11 - Como > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!djj05CUxAKqqqzrdcdpxybBp4r8LCeoh2YHKGXWZBrw8af69K7SN2w0jfqyuP3lDKjRWhw4qzK7MlTV94teN$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From matteo.semplice at uninsubria.it Thu Oct 24 17:04:52 2024 From: matteo.semplice at uninsubria.it (Matteo Semplice) Date: Fri, 25 Oct 2024 00:04:52 +0200 Subject: [petsc-users] local/global DMPlex Vec output In-Reply-To: References: Message-ID: <00e2b913-4a38-4d7c-9ae6-90c5dc86647b@uninsubria.it> Hi. The HDF5 solution looks good to me, but I now get this error $ ../src/saveDemo Creating mesh with (10,10) faces [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Object is in wrong state [0]PETSC ERROR: Need to call DMCreateDS() before calling DMGetDS() [0]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!aQGZ72ku3riSEIsR_uRzVpbrS8s0nCNB91RQiigaVSevPhI-eLawlvrtybboEPpmaKHwaeqhEaTP5EpUDnBn2BsaaYMpSbj6ii1DUw$ for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.21.5, unknown [0]PETSC ERROR: ../src/saveDemo on a ?named dentdherens by matteo Fri Oct 25 00:00:44 2024 [0]PETSC ERROR: Configure options --COPTFLAGS="-O3 -march=native -mtune=native -mavx2" --CXXOPTFLAGS="-O3 -march=native -mtune=native -mavx2" --FOPTFLAGS="-O3 -march=native -mtune=native -mavx2" --PETSC_ARCH=op t --with-strict-petscerrorcode --download-hdf5 --prefix=/home/matteo/software/petsc/3.21-opt/ --with-debugging=0 --with-gmsh --with-metis --with-parmetis --with-triangle PETSC_DIR=/home/matteo/software/petsc -- force [0]PETSC ERROR: #1 DMGetDS() at /home/matteo/software/petsc/src/dm/interface/dm.c:5525 [0]PETSC ERROR: #2 DMPlexInsertBoundaryValues_Plex() at /home/matteo/software/petsc/src/dm/impls/plex/plexfem.c:1136 [0]PETSC ERROR: #3 DMPlexInsertBoundaryValues() at /home/matteo/software/petsc/src/dm/impls/plex/plexfem.c:1274 [0]PETSC ERROR: #4 VecView_Plex_HDF5_Internal() at /home/matteo/software/petsc/src/dm/impls/plex/plexhdf5.c:477 [0]PETSC ERROR: #5 VecView_Plex() at /home/matteo/software/petsc/src/dm/impls/plex/plex.c:656 [0]PETSC ERROR: #6 VecView() at /home/matteo/software/petsc/src/vec/vec/interface/vector.c:806 [0]PETSC ERROR: #7 main() at saveDemo.cpp:123 [0]PETSC ERROR: No PETSc Option Table entries [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov---------- -------------------------------------------------------------------------- MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_SELF with errorcode 73. NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. You may or may not see output from other processes, depending on exactly when Open MPI kills them. -------------------------------------------------------------------------- I attach the modified sample code that produced the above error. Thanks ??? Matteo Il 24/10/24 22:20, Matthew Knepley ha scritto: > I just looked at the code. The VTK code is very old, and does not > check for cell overlap. > > We have been recommending that people use either HDF5 or CGNS, both of > which work in this case > I believe. I can fix VTK if that is what you want, but it might take > me a little while as it is very busy at > work right now. However, if you output HDF5, then you can run > > ? ./lib/petsc/bin/petsc_gen_xdmf.py mesh.h5 > > and it will generate an XDMF file so you can load it into ParaView. Or > you can output CGNS which I think > ParaView understands. > > ? Thanks, > > ? ? ?Matt > > On Thu, Oct 24, 2024 at 4:02?PM Semplice Matteo > wrote: > > Hi, > ??????I tried again today and have (re?)discovered this example > https://urldefense.us/v3/__https://petsc.org/release/src/dm/impls/plex/tutorials/ex14.c.html__;!!G_uCfscf7eWS!aQGZ72ku3riSEIsR_uRzVpbrS8s0nCNB91RQiigaVSevPhI-eLawlvrtybboEPpmaKHwaeqhEaTP5EpUDnBn2BsaaYMpSbiLx3Cm6A$ , > but I cannot understand if in my case I should call > PetscSFCreateSectionSF > ?and, > if so, how should I then activate the returned SF. > Matteo > ------------------------------------------------------------------------ > *Da:* Semplice Matteo > *Inviato:* marted? 22 ottobre 2024 00:24 > *A:* Matthew Knepley > *Cc:* PETSc > *Oggetto:* Re: [petsc-users] local/global DMPlex Vec output > > Dear Matt, > > ??? I guess you're right: thresholding by rank==0 and rank==1 in > paraview reveals that it is indeed the overlap cells that are > appear twice in the output. > > The attached file is not exactly minimal but hopefully short > enough. If I run it in serial, all is ok, but with > > ??? mpirun -np 2 ./saveDemo > > it creates a 10x10 grid, but I get "output.vtu" with a total of > 120 cells. However the pointSF of the DMPlex seems correct. > > Thanks > > ??? Matteo > > Il 21/10/24 19:15, Matthew Knepley ha scritto: >> On Mon, Oct 21, 2024 at 12:22?PM Matteo Semplice via petsc-users >> wrote: >> >> Dear petsc-users, >> >> ??? I am having issues with output of parallel data attached >> to a DMPlex (or maybe more fundamental ones about DMPlex...). >> >> So I currently >> >> 1. create a DMPlex (DMPlexCreateGmshFromFile or >> DMPlexCreateBoxMesh) >> 2. partition it >> 3. and create a section for my data layout with >> DMPlexCreateSection(ctx.dmMesh, NULL, numComp, numDof, >> numBC, NULL, NULL, NULL, NULL, &sUavg) >> 4. DMSetLocalSection(ctx.dmMesh, sUavg) >> 5. create solLoc and solGlob vectors with >> DMCreateGlobalVector and DMCreateLocalVector >> 6. solve .... >> 7. VecView(ctx.solGlob, vtkViewer) on a .vtu file >> >> but when I load data in ParaView I get more cells than >> expected and it is as if the cells in the halo are put twice >> in output. (I could create a MWE if the above is not clear) >> >> I think we need an MWE here, because from the explanation above, >> it should work. >> >> However, I can try to guess the problem. When you partition the >> mesh, I am guessing that you have cells in the overlap. These cells >> must be in the point SF in order for the global section to give >> them a unique owner. Perhaps something has gone wrong here. >> >> ? Thanks, >> >> ? ? ?Matt >> >> I guess that the culprit is point (4), but if I replace it >> with DMSetGlobalSection then I cannot create the local vector >> at point (5). >> >> How should I handle this properly? In my code I need to >> create both local and global vectors, to perform at least >> GlobalToLocal and to save the global data. >> >> (On a side note, I tried also HDF5 but then it complains >> about the DM not having a DS...; really, any working solution >> that allows data to be explored with Paraview is fine) >> >> Thanks for any advice! >> >> Matteo Semplice >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to >> which their experiments lead. >> -- Norbert Wiener >> >> https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!aQGZ72ku3riSEIsR_uRzVpbrS8s0nCNB91RQiigaVSevPhI-eLawlvrtybboEPpmaKHwaeqhEaTP5EpUDnBn2BsaaYMpSbixfvYF1A$ >> > > -- > --- > Professore Associato in Analisi Numerica > Dipartimento di Scienza e Alta Tecnologia > Universit? degli Studi dell'Insubria > Via Valleggio, 11 - Como > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!aQGZ72ku3riSEIsR_uRzVpbrS8s0nCNB91RQiigaVSevPhI-eLawlvrtybboEPpmaKHwaeqhEaTP5EpUDnBn2BsaaYMpSbixfvYF1A$ > -- --- Professore Associato in Analisi Numerica Dipartimento di Scienza e Alta Tecnologia Universit? degli Studi dell'Insubria Via Valleggio, 11 - Como -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: saveDemo.cpp Type: text/x-c++src Size: 5524 bytes Desc: not available URL: From knepley at gmail.com Thu Oct 24 18:02:13 2024 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 24 Oct 2024 19:02:13 -0400 Subject: [petsc-users] local/global DMPlex Vec output In-Reply-To: <00e2b913-4a38-4d7c-9ae6-90c5dc86647b@uninsubria.it> References: <00e2b913-4a38-4d7c-9ae6-90c5dc86647b@uninsubria.it> Message-ID: On Thu, Oct 24, 2024 at 6:04?PM Matteo Semplice < matteo.semplice at uninsubria.it> wrote: > Hi. The HDF5 solution looks good to me, but I now get this error > > Okay, I can make a workaround for this. Here is what is happening. When you output solutions, you really want the essential boundary conditions included in the output, and the only way I have to do that is for you to tell me about the discretization, so I require the DS. What I can do is ignore this step if there is no DS. Let me do that and mail you with the branch. Thanks! Matt > $ ../src/saveDemo > Creating mesh with (10,10) faces > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Object is in wrong state > [0]PETSC ERROR: Need to call DMCreateDS() before calling DMGetDS() > [0]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!dLtO-GGeUrd81R06hiAmfndS-cgZrVE4t8Hipjb8bHTF-vqn1VNb3pjTj5fljYzb6ejXt1QTiLZq1kEPxNH0$ for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.21.5, unknown > [0]PETSC ERROR: ../src/saveDemo on a named dentdherens by matteo Fri Oct > 25 00:00:44 2024 > [0]PETSC ERROR: Configure options --COPTFLAGS="-O3 -march=native > -mtune=native -mavx2" --CXXOPTFLAGS="-O3 -march=native -mtune=native > -mavx2" --FOPTFLAGS="-O3 -march=native -mtune=native -mavx2" --PETSC_ARCH=op > t --with-strict-petscerrorcode --download-hdf5 > --prefix=/home/matteo/software/petsc/3.21-opt/ --with-debugging=0 > --with-gmsh --with-metis --with-parmetis --with-triangle > PETSC_DIR=/home/matteo/software/petsc -- > force > [0]PETSC ERROR: #1 DMGetDS() at > /home/matteo/software/petsc/src/dm/interface/dm.c:5525 > [0]PETSC ERROR: #2 DMPlexInsertBoundaryValues_Plex() at > /home/matteo/software/petsc/src/dm/impls/plex/plexfem.c:1136 > [0]PETSC ERROR: #3 DMPlexInsertBoundaryValues() at > /home/matteo/software/petsc/src/dm/impls/plex/plexfem.c:1274 > [0]PETSC ERROR: #4 VecView_Plex_HDF5_Internal() at > /home/matteo/software/petsc/src/dm/impls/plex/plexhdf5.c:477 > [0]PETSC ERROR: #5 VecView_Plex() at > /home/matteo/software/petsc/src/dm/impls/plex/plex.c:656 > [0]PETSC ERROR: #6 VecView() at > /home/matteo/software/petsc/src/vec/vec/interface/vector.c:806 > [0]PETSC ERROR: #7 main() at saveDemo.cpp:123 > [0]PETSC ERROR: No PETSc Option Table entries > [0]PETSC ERROR: ----------------End of Error Message -------send entire > error message to petsc-maint at mcs.anl.gov---------- > -------------------------------------------------------------------------- > MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_SELF > with errorcode 73. > > NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. > You may or may not see output from other processes, depending on > exactly when Open MPI kills them. > -------------------------------------------------------------------------- > > I attach the modified sample code that produced the above error. > > Thanks > > Matteo > Il 24/10/24 22:20, Matthew Knepley ha scritto: > > I just looked at the code. The VTK code is very old, and does not check > for cell overlap. > > We have been recommending that people use either HDF5 or CGNS, both of > which work in this case > I believe. I can fix VTK if that is what you want, but it might take me a > little while as it is very busy at > work right now. However, if you output HDF5, then you can run > > ./lib/petsc/bin/petsc_gen_xdmf.py mesh.h5 > > and it will generate an XDMF file so you can load it into ParaView. Or you > can output CGNS which I think > ParaView understands. > > Thanks, > > Matt > > On Thu, Oct 24, 2024 at 4:02?PM Semplice Matteo < > matteo.semplice at uninsubria.it> wrote: > >> Hi, >> I tried again today and have (re?)discovered this example >> https://urldefense.us/v3/__https://petsc.org/release/src/dm/impls/plex/tutorials/ex14.c.html__;!!G_uCfscf7eWS!dLtO-GGeUrd81R06hiAmfndS-cgZrVE4t8Hipjb8bHTF-vqn1VNb3pjTj5fljYzb6ejXt1QTiLZq1oOKrq2I$ , but I >> cannot understand if in my case I should call PetscSFCreateSectionSF >> and, >> if so, how should I then activate the returned SF. >> Matteo >> ------------------------------ >> *Da:* Semplice Matteo >> *Inviato:* marted? 22 ottobre 2024 00:24 >> *A:* Matthew Knepley >> *Cc:* PETSc >> *Oggetto:* Re: [petsc-users] local/global DMPlex Vec output >> >> >> Dear Matt, >> >> I guess you're right: thresholding by rank==0 and rank==1 in paraview >> reveals that it is indeed the overlap cells that are appear twice in the >> output. >> >> The attached file is not exactly minimal but hopefully short enough. If I >> run it in serial, all is ok, but with >> >> mpirun -np 2 ./saveDemo >> >> it creates a 10x10 grid, but I get "output.vtu" with a total of 120 >> cells. However the pointSF of the DMPlex seems correct. >> >> Thanks >> >> Matteo >> Il 21/10/24 19:15, Matthew Knepley ha scritto: >> >> On Mon, Oct 21, 2024 at 12:22?PM Matteo Semplice via petsc-users < >> petsc-users at mcs.anl.gov> wrote: >> >> Dear petsc-users, >> >> I am having issues with output of parallel data attached to a DMPlex >> (or maybe more fundamental ones about DMPlex...). >> >> So I currently >> >> 1. create a DMPlex (DMPlexCreateGmshFromFile or DMPlexCreateBoxMesh) >> 2. partition it >> 3. and create a section for my data layout with >> DMPlexCreateSection(ctx.dmMesh, NULL, numComp, numDof, numBC, NULL, NULL, >> NULL, NULL, &sUavg) >> 4. DMSetLocalSection(ctx.dmMesh, sUavg) >> 5. create solLoc and solGlob vectors with DMCreateGlobalVector and >> DMCreateLocalVector >> 6. solve .... >> 7. VecView(ctx.solGlob, vtkViewer) on a .vtu file >> >> but when I load data in ParaView I get more cells than expected and it is >> as if the cells in the halo are put twice in output. (I could create a MWE >> if the above is not clear) >> >> I think we need an MWE here, because from the explanation above, it >> should work. >> >> However, I can try to guess the problem. When you partition the mesh, I >> am guessing that you have cells in the overlap. These cells >> must be in the point SF in order for the global section to give them a >> unique owner. Perhaps something has gone wrong here. >> >> Thanks, >> >> Matt >> >> I guess that the culprit is point (4), but if I replace it with >> DMSetGlobalSection then I cannot create the local vector at point (5). >> >> How should I handle this properly? In my code I need to create both local >> and global vectors, to perform at least GlobalToLocal and to save the >> global data. >> >> (On a side note, I tried also HDF5 but then it complains about the DM not >> having a DS...; really, any working solution that allows data to be >> explored with Paraview is fine) >> >> Thanks for any advice! >> >> Matteo Semplice >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!dLtO-GGeUrd81R06hiAmfndS-cgZrVE4t8Hipjb8bHTF-vqn1VNb3pjTj5fljYzb6ejXt1QTiLZq1j7Zbw0Z$ >> >> >> -- >> --- >> Professore Associato in Analisi Numerica >> Dipartimento di Scienza e Alta Tecnologia >> Universit? degli Studi dell'Insubria >> Via Valleggio, 11 - Como >> >> > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!dLtO-GGeUrd81R06hiAmfndS-cgZrVE4t8Hipjb8bHTF-vqn1VNb3pjTj5fljYzb6ejXt1QTiLZq1j7Zbw0Z$ > > > -- > --- > Professore Associato in Analisi Numerica > Dipartimento di Scienza e Alta Tecnologia > Universit? degli Studi dell'Insubria > Via Valleggio, 11 - Como > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!dLtO-GGeUrd81R06hiAmfndS-cgZrVE4t8Hipjb8bHTF-vqn1VNb3pjTj5fljYzb6ejXt1QTiLZq1j7Zbw0Z$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From diegomagela at usp.br Thu Oct 24 19:48:51 2024 From: diegomagela at usp.br (Diego Magela Lemos) Date: Thu, 24 Oct 2024 21:48:51 -0300 Subject: [petsc-users] Block sparse matrix Message-ID: Is there something similar to SciPy sparse block_array ( https://urldefense.us/v3/__https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.block_array.html*scipy.sparse.block_array__;Iw!!G_uCfscf7eWS!YANe4IC4s7Ki7CYmRbt0miQwrrFq7e-6_btfizr9d4vir0dzWIncCQBkH3Cbr94Y_jxGNSAhJh59aSOHAHRESglgg7A$ ) or bmat ( https://urldefense.us/v3/__https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.bmat.html__;!!G_uCfscf7eWS!YANe4IC4s7Ki7CYmRbt0miQwrrFq7e-6_btfizr9d4vir0dzWIncCQBkH3Cbr94Y_jxGNSAhJh59aSOHAHREvOCba0s$ ) in PETSc? -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Thu Oct 24 20:01:40 2024 From: bsmith at petsc.dev (Barry Smith) Date: Thu, 24 Oct 2024 21:01:40 -0400 Subject: [petsc-users] Block sparse matrix In-Reply-To: References: Message-ID: <09D52C55-5F12-4976-86B9-46431D2AAC49@petsc.dev> Perhaps this is what you are looking for https://urldefense.us/v3/__https://petsc.org/release/manualpages/Mat/MATNEST/*matnest__;Iw!!G_uCfscf7eWS!e9L1JRLW7O-w_LDeaQomJxREvfqcm0FrZtlpjeE46ux3HjMJELyJmp9jx4uTJkK7WyQc57DtJFcC3nFe6jEJpOU$ ? > On Oct 24, 2024, at 8:48?PM, Diego Magela Lemos via petsc-users wrote: > > Is there something similar to SciPy sparse block_array (https://urldefense.us/v3/__https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.block_array.html*scipy.sparse.block_array__;Iw!!G_uCfscf7eWS!e9L1JRLW7O-w_LDeaQomJxREvfqcm0FrZtlpjeE46ux3HjMJELyJmp9jx4uTJkK7WyQc57DtJFcC3nFeocXEqtM$ ) or bmat (https://urldefense.us/v3/__https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.bmat.html__;!!G_uCfscf7eWS!e9L1JRLW7O-w_LDeaQomJxREvfqcm0FrZtlpjeE46ux3HjMJELyJmp9jx4uTJkK7WyQc57DtJFcC3nFezKs5Ti0$ ) in PETSc? -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Oct 24 20:02:08 2024 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 24 Oct 2024 21:02:08 -0400 Subject: [petsc-users] Block sparse matrix In-Reply-To: References: Message-ID: On Thu, Oct 24, 2024 at 8:49?PM Diego Magela Lemos via petsc-users < petsc-users at mcs.anl.gov> wrote: > Is there something similar to SciPy sparse block_array ( > https://urldefense.us/v3/__https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.block_array.html*scipy.sparse.block_array__;Iw!!G_uCfscf7eWS!dZFqpGVfNQMRHO5CeKFJ1fh9RlQt32Q_DyPYhfwuFy6Xv40dl94COZfQKFh7xywfE8Z0FeFIBUrNHvp9rgVV$ > ) > or bmat ( > https://urldefense.us/v3/__https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.bmat.html__;!!G_uCfscf7eWS!dZFqpGVfNQMRHO5CeKFJ1fh9RlQt32Q_DyPYhfwuFy6Xv40dl94COZfQKFh7xywfE8Z0FeFIBUrNHiq0Tphm$ > ) > in PETSc? > Yes, MATNEST, but it does not convert the result to CSR. Matt -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!dZFqpGVfNQMRHO5CeKFJ1fh9RlQt32Q_DyPYhfwuFy6Xv40dl94COZfQKFh7xywfE8Z0FeFIBUrNHtvyOnr5$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Thu Oct 24 20:08:12 2024 From: bsmith at petsc.dev (Barry Smith) Date: Thu, 24 Oct 2024 21:08:12 -0400 Subject: [petsc-users] Block sparse matrix In-Reply-To: References: Message-ID: <0F1771FC-0212-48F1-A1DF-E340F09ACDF9@petsc.dev> > On Oct 24, 2024, at 9:02?PM, Matthew Knepley wrote: > > On Thu, Oct 24, 2024 at 8:49?PM Diego Magela Lemos via petsc-users > wrote: >> Is there something similar to SciPy sparse block_array (https://urldefense.us/v3/__https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.block_array.html*scipy.sparse.block_array__;Iw!!G_uCfscf7eWS!fxhgjj4X8vHGK28Zh3l6x6nmnLOI-7wwIouvNhI9v3RCPvUzL_4KWGY0ZgCaMugPhFAn7AyDI7t7nCHZJ4waON0$ ) or bmat (https://urldefense.us/v3/__https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.bmat.html__;!!G_uCfscf7eWS!fxhgjj4X8vHGK28Zh3l6x6nmnLOI-7wwIouvNhI9v3RCPvUzL_4KWGY0ZgCaMugPhFAn7AyDI7t7nCHZcQ6q2Uk$ ) in PETSc? > > Yes, MATNEST, but it does not convert the result to CSR. PetscCall(MatConvert(nest, MATAIJ, MAT_INITIAL_MATRIX, &aij)); will do a conversion if needed. > > Matt > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fxhgjj4X8vHGK28Zh3l6x6nmnLOI-7wwIouvNhI9v3RCPvUzL_4KWGY0ZgCaMugPhFAn7AyDI7t7nCHZzB0hsUo$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From vaishakprasad at gmail.com Fri Oct 25 12:58:38 2024 From: vaishakprasad at gmail.com (Vaishak Prasad) Date: Fri, 25 Oct 2024 23:28:38 +0530 Subject: [petsc-users] Requesting help with SNES non-linear example Message-ID: Dear All, Greetings from India. I am new to petsc4py. I was trying out the example on slides 18, and 19 at https://urldefense.us/v3/__https://www.bu.edu/pasi/files/2011/01/Lisandro-Dalcin-petsc4py.pdf__;!!G_uCfscf7eWS!ehNmddsrNhTg2K1fvJVZhjNXm7zt2pz25zK1Th5iKAIPiWNt4cl3DL63Z2e4zzZ0-e1Ff4Xg7-UB4QGl2eBfEVBffcVp$ However, I get a zero vector for the solution `x`. I am using petsc4py 3.21.3 on Linux. Any help would be appreciated. With regards -- Vaishak Prasad Post Doctoral Fellow Astrophysical Relativity Group International Center for Theoretical Sciences Tata Institute of Fundamental Research Survey No. 151, Shivakote, Hesaraghatta Hobli, Bengaluru, Karnataka, India, Earth, Solar system, Milky Way Galaxy, The Local Group -------------- next part -------------- An HTML attachment was scrubbed... URL: From jroman at dsic.upv.es Sat Oct 26 02:12:25 2024 From: jroman at dsic.upv.es (Jose E. Roman) Date: Sat, 26 Oct 2024 07:12:25 +0000 Subject: [petsc-users] Requesting help with SNES non-linear example In-Reply-To: References: Message-ID: <74EC49BE-4CE2-4430-916B-827366190B8F@dsic.upv.es> The examples in those old slides might be outdated. Try with the examples that come with the PETSc repo or tarball. In particular, that example can be found in src/binding/petsc4py/demo/legacy/bratu2d/bratu2d.py. Jose > El 25 oct 2024, a las 19:58, Vaishak Prasad escribi?: > > Dear All, > Greetings from India. > > I am new to petsc4py. I was trying out the example on slides 18, and 19 at > > https://urldefense.us/v3/__https://www.bu.edu/pasi/files/2011/01/Lisandro-Dalcin-petsc4py.pdf__;!!G_uCfscf7eWS!dpUYbX5YRxvaN33h0mhRCxEMkia_yIdGU6sQOpV0upkfReAJkPC8H6GX4HH6NNvgOCt1F2Ogn2963Ff_Vkub5yK7$ > > However, I get a zero vector for the solution `x`. > > I am using petsc4py 3.21.3 on Linux. > > Any help would be appreciated. > > With regards > > -- > Vaishak Prasad > Post Doctoral Fellow > Astrophysical Relativity Group > International Center for Theoretical Sciences > Tata Institute of Fundamental Research > Survey No. 151, Shivakote, Hesaraghatta Hobli, > Bengaluru, Karnataka, India, > Earth, Solar system, > Milky Way Galaxy, The Local Group From vaishakprasad at gmail.com Sat Oct 26 04:11:41 2024 From: vaishakprasad at gmail.com (Vaishak Prasad) Date: Sat, 26 Oct 2024 14:41:41 +0530 Subject: [petsc-users] Requesting help with SNES non-linear example In-Reply-To: <74EC49BE-4CE2-4430-916B-827366190B8F@dsic.upv.es> References: <74EC49BE-4CE2-4430-916B-827366190B8F@dsic.upv.es> Message-ID: Hi, Thanks for pointing me to the updated example. The example itself works fine. However, on adapting the example to a simple problem ( recovering a given 1D vector), I find that there is only one function call happening and the solution has not converged. I am attaching the modified example here. Is there something silly I am doing here? Any help would be appreciated. Thanks and Regards On Sat, Oct 26, 2024 at 12:42?PM Jose E. Roman wrote: > The examples in those old slides might be outdated. Try with the examples > that come with the PETSc repo or tarball. In particular, that example can > be found in src/binding/petsc4py/demo/legacy/bratu2d/bratu2d.py. > > Jose > > > El 25 oct 2024, a las 19:58, Vaishak Prasad > escribi?: > > > > Dear All, > > Greetings from India. > > > > I am new to petsc4py. I was trying out the example on slides 18, and 19 > at > > > > https://urldefense.us/v3/__https://www.bu.edu/pasi/files/2011/01/Lisandro-Dalcin-petsc4py.pdf__;!!G_uCfscf7eWS!crzDSnMPvGJ19hJ6wP30I5DQnV7XMkJmxebYeLV9MFjpkLL2d5VtXHhHE2si93mzUmxzKxeVo0dkoyx-Cd04hRPUqrAd$ > > > > However, I get a zero vector for the solution `x`. > > > > I am using petsc4py 3.21.3 on Linux. > > > > Any help would be appreciated. > > > > With regards > > > > -- > > Vaishak Prasad > > Post Doctoral Fellow > > Astrophysical Relativity Group > > International Center for Theoretical Sciences > > Tata Institute of Fundamental Research > > Survey No. 151, Shivakote, Hesaraghatta Hobli, > > Bengaluru, Karnataka, India, > > Earth, Solar system, > > Milky Way Galaxy, The Local Group > > -- Vaishak Prasad Post Doctoral Fellow Astrophysical Relativity Group International Center for Theoretical Sciences Tata Institute of Fundamental Research Survey No. 151, Shivakote, Hesaraghatta Hobli, Bengaluru, Karnataka, India, Earth, Solar system, Milky Way Galaxy, The Local Group -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: mwe.py Type: text/x-python Size: 1875 bytes Desc: not available URL: From jroman at dsic.upv.es Sat Oct 26 05:02:40 2024 From: jroman at dsic.upv.es (Jose E. Roman) Date: Sat, 26 Oct 2024 10:02:40 +0000 Subject: [petsc-users] Requesting help with SNES non-linear example In-Reply-To: References: <74EC49BE-4CE2-4430-916B-827366190B8F@dsic.upv.es> Message-ID: Your residue() function is computing something (f) that is not returned to the calling function. So evalFunction() returns a zero residual and the solver stops in one iteration. Mixing numpy arrays and petsc4py vectors is going to be tricky unless you know what you are doing. It is simpler to manipulate petsc4py vectors directly, see for instance bratu3d.py or other examples. Jose > El 26 oct 2024, a las 11:11, Vaishak Prasad escribi?: > > Hi, > > Thanks for pointing me to the updated example. The example itself works fine. > > However, on adapting the example to a simple problem ( recovering a given 1D vector), I find that there is only one function call happening and the solution has not converged. I am attaching the modified example here. Is there something silly I am doing here? > > Any help would be appreciated. > > Thanks and Regards > > > On Sat, Oct 26, 2024 at 12:42?PM Jose E. Roman wrote: > The examples in those old slides might be outdated. Try with the examples that come with the PETSc repo or tarball. In particular, that example can be found in src/binding/petsc4py/demo/legacy/bratu2d/bratu2d.py. > > Jose > > > El 25 oct 2024, a las 19:58, Vaishak Prasad escribi?: > > > > Dear All, > > Greetings from India. > > > > I am new to petsc4py. I was trying out the example on slides 18, and 19 at > > > > https://urldefense.us/v3/__https://www.bu.edu/pasi/files/2011/01/Lisandro-Dalcin-petsc4py.pdf__;!!G_uCfscf7eWS!d-A7MAUVTgb6BEVbhTWgL1jYExkAYjqUU0OnPOv8BSXFQ1xC2M5NuiIo5WN7D2AITF0S3Zmq77m1KKOaKZdvpQn4$ > > > > However, I get a zero vector for the solution `x`. > > > > I am using petsc4py 3.21.3 on Linux. > > > > Any help would be appreciated. > > > > With regards > > > > -- > > Vaishak Prasad > > Post Doctoral Fellow > > Astrophysical Relativity Group > > International Center for Theoretical Sciences > > Tata Institute of Fundamental Research > > Survey No. 151, Shivakote, Hesaraghatta Hobli, > > Bengaluru, Karnataka, India, > > Earth, Solar system, > > Milky Way Galaxy, The Local Group > > > > -- > Vaishak Prasad > Post Doctoral Fellow > Astrophysical Relativity Group > International Center for Theoretical Sciences > Tata Institute of Fundamental Research > Survey No. 151, Shivakote, Hesaraghatta Hobli, > Bengaluru, Karnataka, India, > Earth, Solar system, > Milky Way Galaxy, The Local Group > From vaishakprasad at gmail.com Sat Oct 26 05:39:23 2024 From: vaishakprasad at gmail.com (Vaishak Prasad) Date: Sat, 26 Oct 2024 16:09:23 +0530 Subject: [petsc-users] Requesting help with SNES non-linear example In-Reply-To: References: <74EC49BE-4CE2-4430-916B-827366190B8F@dsic.upv.es> Message-ID: Dear Jose, Thankyou very much for your response. On Sat, Oct 26, 2024 at 3:32?PM Jose E. Roman wrote: > Your residue() function is computing something (f) that is not returned to > the calling function. So evalFunction() returns a zero residual and the > solver stops in one iteration. > Apologies, I am a little confused. The example bratu2d.py imports the residue function from bratu2dnpy.py and it also does not return anything (and it works). I was under the impression that a pointer to `f` is passed to petsc, that overwrites it at each iteration. Please correct me if I am wrong. I also tried returning `f` in my residue function but it does not change anything. Is this what you suggested? > Mixing numpy arrays and petsc4py vectors is going to be tricky unless you > know what you are doing. It is simpler to manipulate petsc4py vectors > directly, see for instance bratu3d.py or other examples. > I have also tried this. I created a petsc vector for the input expected solution and then converted it into a numpy array using the `getArray` method. But noting changes. I also printed some information about the convergence itself. I find ConvergedReason is 2 (absolute tolerance) getTolerances is [1e-8, 1e-50, 1e-8, 50] (default) getErrorIfNotConverged is False getIterationNumber is 0 In contrast, the convergence reason for the bratu2d example was 3 (relative tolerance). I also tried increasing the `max_it` but to no avail. I appreciate your valuable suggestion and advice in this regard. Thanks and Regards Jose > > > > El 26 oct 2024, a las 11:11, Vaishak Prasad > escribi?: > > > > Hi, > > > > Thanks for pointing me to the updated example. The example itself works > fine. > > > > However, on adapting the example to a simple problem ( recovering a > given 1D vector), I find that there is only one function call happening and > the solution has not converged. I am attaching the modified example here. > Is there something silly I am doing here? > > > > Any help would be appreciated. > > > > Thanks and Regards > > > > > > On Sat, Oct 26, 2024 at 12:42?PM Jose E. Roman > wrote: > > The examples in those old slides might be outdated. Try with the > examples that come with the PETSc repo or tarball. In particular, that > example can be found in src/binding/petsc4py/demo/legacy/bratu2d/bratu2d.py. > > > > Jose > > > > > El 25 oct 2024, a las 19:58, Vaishak Prasad > escribi?: > > > > > > Dear All, > > > Greetings from India. > > > > > > I am new to petsc4py. I was trying out the example on slides 18, and > 19 at > > > > > > https://urldefense.us/v3/__https://www.bu.edu/pasi/files/2011/01/Lisandro-Dalcin-petsc4py.pdf__;!!G_uCfscf7eWS!bzL5mw0mqx-jt3BIhPqHNtxZUf469dbmQwxugUupECLT7CEMbk_vj-TZ_wEym_uN1O4CWKJKeODMn0a0tO72ky2PkUrV$ > > > > > > However, I get a zero vector for the solution `x`. > > > > > > I am using petsc4py 3.21.3 on Linux. > > > > > > Any help would be appreciated. > > > > > > With regards > > > > > > -- > > > Vaishak Prasad > > > Post Doctoral Fellow > > > Astrophysical Relativity Group > > > International Center for Theoretical Sciences > > > Tata Institute of Fundamental Research > > > Survey No. 151, Shivakote, Hesaraghatta Hobli, > > > Bengaluru, Karnataka, India, > > > Earth, Solar system, > > > Milky Way Galaxy, The Local Group > > > > > > > > -- > > Vaishak Prasad > > Post Doctoral Fellow > > Astrophysical Relativity Group > > International Center for Theoretical Sciences > > Tata Institute of Fundamental Research > > Survey No. 151, Shivakote, Hesaraghatta Hobli, > > Bengaluru, Karnataka, India, > > Earth, Solar system, > > Milky Way Galaxy, The Local Group > > > > -- Vaishak Prasad Post Doctoral Fellow Astrophysical Relativity Group International Center for Theoretical Sciences Tata Institute of Fundamental Research Survey No. 151, Shivakote, Hesaraghatta Hobli, Bengaluru, Karnataka, India, Earth, Solar system, Milky Way Galaxy, The Local Group -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: mwe.py Type: text/x-python Size: 2281 bytes Desc: not available URL: From jroman at dsic.upv.es Sun Oct 27 14:03:11 2024 From: jroman at dsic.upv.es (Jose E. Roman) Date: Sun, 27 Oct 2024 19:03:11 +0000 Subject: [petsc-users] Requesting help with SNES non-linear example In-Reply-To: References: <74EC49BE-4CE2-4430-916B-827366190B8F@dsic.upv.es> Message-ID: bratu2dnpy.py does not "return f", instead it modifies the argument f by manipulating its internal array. If you want to use a function that "return f" you have to receive this returned variable and then set it to the Vec object, i.e.: f = self.compute(expected_soln_vec_npy, x, f) F.setArray(f) Jose > El 26 oct 2024, a las 12:39, Vaishak Prasad escribi?: > > Dear Jose, > > Thankyou very much for your response. > > On Sat, Oct 26, 2024 at 3:32?PM Jose E. Roman wrote: > Your residue() function is computing something (f) that is not returned to the calling function. So evalFunction() returns a zero residual and the solver stops in one iteration. > Apologies, I am a little confused. The example bratu2d.py imports the residue function from bratu2dnpy.py and it also does not return anything (and it works). I was under the impression that > a pointer to `f` is passed to petsc, that overwrites it at each iteration. Please correct me if I am wrong. > > I also tried returning `f` in my residue function but it does not change anything. Is this what you suggested? > Mixing numpy arrays and petsc4py vectors is going to be tricky unless you know what you are doing. It is simpler to manipulate petsc4py vectors directly, see for instance bratu3d.py or other examples. > I have also tried this. I created a petsc vector for the input expected solution and then converted it into a numpy array using the `getArray` method. But noting changes. > > I also printed some information about the convergence itself. I find > ConvergedReason is 2 (absolute tolerance) > getTolerances is [1e-8, 1e-50, 1e-8, 50] (default) > getErrorIfNotConverged is False > getIterationNumber is 0 > > In contrast, the convergence reason for the bratu2d example was 3 (relative tolerance). > > I also tried increasing the `max_it` but to no avail. > > I appreciate your valuable suggestion and advice in this regard. > > Thanks and Regards > > Jose > > > > El 26 oct 2024, a las 11:11, Vaishak Prasad escribi?: > > > > Hi, > > > > Thanks for pointing me to the updated example. The example itself works fine. > > > > However, on adapting the example to a simple problem ( recovering a given 1D vector), I find that there is only one function call happening and the solution has not converged. I am attaching the modified example here. Is there something silly I am doing here? > > > > Any help would be appreciated. > > > > Thanks and Regards > > > > > > On Sat, Oct 26, 2024 at 12:42?PM Jose E. Roman wrote: > > The examples in those old slides might be outdated. Try with the examples that come with the PETSc repo or tarball. In particular, that example can be found in src/binding/petsc4py/demo/legacy/bratu2d/bratu2d.py. > > > > Jose > > > > > El 25 oct 2024, a las 19:58, Vaishak Prasad escribi?: > > > > > > Dear All, > > > Greetings from India. > > > > > > I am new to petsc4py. I was trying out the example on slides 18, and 19 at > > > > > > https://urldefense.us/v3/__https://www.bu.edu/pasi/files/2011/01/Lisandro-Dalcin-petsc4py.pdf__;!!G_uCfscf7eWS!ZnQ15XH0wdJTgAhkpejZ0TFFnvrfoV8_YNfEnMmNV9HbN5jMiA0BcE2ru-OzhnPQxY5rps05o84vIoA9rvUd4pK3$ > > > > > > However, I get a zero vector for the solution `x`. > > > > > > I am using petsc4py 3.21.3 on Linux. > > > > > > Any help would be appreciated. > > > > > > With regards > > > > > > -- > > > Vaishak Prasad > > > Post Doctoral Fellow > > > Astrophysical Relativity Group > > > International Center for Theoretical Sciences > > > Tata Institute of Fundamental Research > > > Survey No. 151, Shivakote, Hesaraghatta Hobli, > > > Bengaluru, Karnataka, India, > > > Earth, Solar system, > > > Milky Way Galaxy, The Local Group > > > > > > > > -- > > Vaishak Prasad > > Post Doctoral Fellow > > Astrophysical Relativity Group > > International Center for Theoretical Sciences > > Tata Institute of Fundamental Research > > Survey No. 151, Shivakote, Hesaraghatta Hobli, > > Bengaluru, Karnataka, India, > > Earth, Solar system, > > Milky Way Galaxy, The Local Group > > > > > > -- > Vaishak Prasad > Post Doctoral Fellow > Astrophysical Relativity Group > International Center for Theoretical Sciences > Tata Institute of Fundamental Research > Survey No. 151, Shivakote, Hesaraghatta Hobli, > Bengaluru, Karnataka, India, > Earth, Solar system, > Milky Way Galaxy, The Local Group From mail2amneet at gmail.com Mon Oct 28 19:05:39 2024 From: mail2amneet at gmail.com (Amneet Bhalla) Date: Mon, 28 Oct 2024 17:05:39 -0700 Subject: [petsc-users] Rigid body nullspace for Stokes operator Message-ID: Hi Folks, I am trying to solve the momentum equation in a projection preconditioner using GAMG or Hypre solver. The equation looks like for velocity variable *v* looks like: [image: Screenshot 2024-10-28 at 4.15.17 PM.png] Here, ? is spatially varying dynamic viscosity and ? is spatially varying bulk viscosity. I understand that I need to specify rigid body nullspace modes to the multigrid solver in order to accelerate its convergence. Looking into this routine MatNullSpaceCreateRigidBody() ( https://urldefense.us/v3/__https://petsc.org/release/manualpages/Mat/MatNullSpaceCreateRigidBody/__;!!G_uCfscf7eWS!eKqgIJjCdMzIU76f7X65AmGxrU_-lC7W02BMWafJ77DNf_IuQk6O1X3qU1x9Ez8NJ20vZEL-mF6T1yNmDnwv0eWa2w$ ), I see that I need to provide the coordinates of each node. I am using staggered grid discretization. Do I need to provide coordinates of staggered grid locations? Thanks, -- --Amneet -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Screenshot 2024-10-28 at 4.15.17 PM.png Type: image/png Size: 17171 bytes Desc: not available URL: From wurunjian at gmail.com Mon Oct 28 21:44:55 2024 From: wurunjian at gmail.com (Runjian Wu) Date: Tue, 29 Oct 2024 10:44:55 +0800 Subject: [petsc-users] Check the precision for real numbers Message-ID: Hi all, I have a question about how to return the precision for real number in Fortran code? Thanks, Runjian -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Tue Oct 29 07:47:13 2024 From: bsmith at petsc.dev (Barry Smith) Date: Tue, 29 Oct 2024 08:47:13 -0400 Subject: [petsc-users] Check the precision for real numbers In-Reply-To: References: Message-ID: Do you mean inside Fortran source code, you want to determine if single or double is being used? 1) The PETSc preprocess C macros are also included in Petsc's Fortran include files, so either PETSC_USE_REAL_SINGLE or PETSC_USE_REAL_DOUBLE is always defined, you can then use preprocessor macros to check: for example #if defined(PETSC_USE_REAL_SINGLE) xxxx #else xxxx #endif 2) you can apply the Fortran kind() function to PetscReal to determine what type it is. I hope this helps, Barry > On Oct 28, 2024, at 10:44?PM, Runjian Wu wrote: > > Hi all, > > I have a question about how to return the precision for real number in Fortran code? > > Thanks, > > Runjian From mfadams at lbl.gov Tue Oct 29 08:28:32 2024 From: mfadams at lbl.gov (Mark Adams) Date: Tue, 29 Oct 2024 09:28:32 -0400 Subject: [petsc-users] Rigid body nullspace for Stokes operator In-Reply-To: References: Message-ID: This coordinate interface is just a shortcut for vertex based discretizations with 3 dof per vertex, etc. (maybe works in 2D). You will need to construct the null space vectors manually and attach it to the matrix. Used by GAMG. Note, for hypre you want to use the "nodal" options and it does not use these null space vectors. That is probably the way you want to go. eg: -pc_hypre_boomeramg_nodal_coarsen I would run with hypre boomerang and -help and grep on nodal to see all the "nodal" options and use them. Thanks, Mark On Mon, Oct 28, 2024 at 8:06?PM Amneet Bhalla wrote: > Hi Folks, > > I am trying to solve the momentum equation in a projection preconditioner > using GAMG or Hypre solver. The equation looks like for velocity variable > *v* looks like: > > > [image: Screenshot 2024-10-28 at 4.15.17 PM.png] > > Here, ? is spatially varying dynamic viscosity and ? is spatially varying > bulk viscosity. I understand that I need to specify rigid body nullspace > modes to the multigrid solver in order to accelerate its convergence. > Looking into this routine MatNullSpaceCreateRigidBody() ( > https://urldefense.us/v3/__https://petsc.org/release/manualpages/Mat/MatNullSpaceCreateRigidBody/__;!!G_uCfscf7eWS!dV9NK_u6h8vxc8z14I3rAhKas06pBwOaFzCouvxeqW6uBZzjyiAz-uZwk8AQFRvQM8s_OIv76NJ6eA7dXf6NylY$ > ), > I see that I need to provide the coordinates of each node. I am using > staggered grid discretization. Do I need to provide coordinates of > staggered grid locations? > > Thanks, > -- > --Amneet > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Screenshot 2024-10-28 at 4.15.17 PM.png Type: image/png Size: 17171 bytes Desc: not available URL: From mfadams at lbl.gov Tue Oct 29 09:35:07 2024 From: mfadams at lbl.gov (Mark Adams) Date: Tue, 29 Oct 2024 10:35:07 -0400 Subject: [petsc-users] Rigid body nullspace for Stokes operator In-Reply-To: References: Message-ID: Oh my mistake. You are using staggered grids. So you don't have a block size that hypre would use for the "nodal" methods. I'm not sure what you are doing exactly, but try hypre and you could create the null space, zero energy modes, manually, attach to the matrix and try GAMG. You can run with '-info :pc' and grep on GAMG to see if GAMG is picking the null space up (send this output if you can't figure it out). Thanks, Mark On Tue, Oct 29, 2024 at 9:28?AM Mark Adams wrote: > This coordinate interface is just a shortcut for vertex based > discretizations with 3 dof per vertex, etc. (maybe works in 2D). > You will need to construct the null space vectors manually and attach it > to the matrix. Used by GAMG. > > Note, for hypre you want to use the "nodal" options and it does not use > these null space vectors. That is probably the way you want to go. > eg: -pc_hypre_boomeramg_nodal_coarsen > > I would run with hypre boomerang and -help and grep on nodal to see all > the "nodal" options and use them. > > Thanks, > Mark > > > On Mon, Oct 28, 2024 at 8:06?PM Amneet Bhalla > wrote: > >> Hi Folks, >> >> I am trying to solve the momentum equation in a projection preconditioner >> using GAMG or Hypre solver. The equation looks like for velocity variable >> *v* looks like: >> >> >> [image: Screenshot 2024-10-28 at 4.15.17 PM.png] >> >> Here, ? is spatially varying dynamic viscosity and ? is spatially >> varying bulk viscosity. I understand that I need to specify rigid body >> nullspace modes to the multigrid solver in order to accelerate its >> convergence. Looking into this routine MatNullSpaceCreateRigidBody() ( >> https://urldefense.us/v3/__https://petsc.org/release/manualpages/Mat/MatNullSpaceCreateRigidBody/__;!!G_uCfscf7eWS!fvbxApGfOnVAUcLkCZPIxbc5jwYHihkD1jmTEvL11YXyNA4lOMbAnVq07EV5NPvtjqYVtD9iNVbIoU_JvyhjM_M$ >> ), >> I see that I need to provide the coordinates of each node. I am using >> staggered grid discretization. Do I need to provide coordinates of >> staggered grid locations? >> >> Thanks, >> -- >> --Amneet >> >> >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Screenshot 2024-10-28 at 4.15.17 PM.png Type: image/png Size: 17171 bytes Desc: not available URL: From daniel.pino_munoz at mines-paristech.fr Tue Oct 29 11:37:51 2024 From: daniel.pino_munoz at mines-paristech.fr (Daniel Pino Munoz) Date: Tue, 29 Oct 2024 17:37:51 +0100 Subject: [petsc-users] Example of SNES using matrix free jacobian Message-ID: <161ca7d4-82c4-4696-b123-4d441eb903da@mines-paristech.fr> Dear all, I have a linear problem that I am currently solving with a KSP matrix-free. I would like to move on to a non linear problem, so figure I could start by solving the same linear problem using SNES. So I am setting the problem as follows: SNESCreate(PETSC_COMM_WORLD, &snes); MatCreateShell(PETSC_COMM_WORLD, n_dofs, n_dofs, PETSC_DETERMINE, PETSC_DETERMINE, &ctx, &J); MatCreateShell(PETSC_COMM_WORLD, n_dofs, n_dofs, PETSC_DETERMINE, PETSC_DETERMINE, &ctx, &B); MatShellSetOperation(J, MATOP_MULT, (void (*)(void))(Multiplication)); MatCreateVecs(J, &x_sol, &b); VecDuplicate(x_sol, &r); SNESSetFromOptions(snes); SNESSetFunction(snes, r, &(computeResidual), &ctx); SNESSetUseMatrixFree(snes, PETSC_FALSE, PETSC_TRUE); SNESGetLineSearch(snes, &linesearch); SNESGetKSP(snes, &ksp); KSPSetOperators(ksp, J, J); KSPSetInitialGuessNonzero(ksp, PETSC_TRUE); I tested it with a small problem (compiled in debug) and it works. When I compiled it in Release, it crashes with a segfault. I tried running the Debug version through valgrind, but even for a small problem, it is too slow. So I was wondering if you guys could see any rocky mistake on the lines I used above? Otherwise, is there any example that uses a SNES combined with a matrix free KSP operator? Thank you, ? Daniel From bsmith at petsc.dev Tue Oct 29 14:01:59 2024 From: bsmith at petsc.dev (Barry Smith) Date: Tue, 29 Oct 2024 15:01:59 -0400 Subject: [petsc-users] Example of SNES using matrix free jacobian In-Reply-To: <161ca7d4-82c4-4696-b123-4d441eb903da@mines-paristech.fr> References: <161ca7d4-82c4-4696-b123-4d441eb903da@mines-paristech.fr> Message-ID: Don't call KSPSetOperators(ksp, J, J); instead call SNESSetJacobian(snes,J,J, MatMFFDComputeJacobian) but I am not sure that would explain the crash. BTW: since you are applying no preconditioner if the matrix is ill-conditioned it may take many iterations or not converge. You can try something like -ksp_gmres_restart 100 or similar value to try to improve convergence (default is 30). Barry > On Oct 29, 2024, at 12:37?PM, Daniel Pino Munoz wrote: > > Dear all, > > I have a linear problem that I am currently solving with a KSP matrix-free. > > I would like to move on to a non linear problem, so figure I could start by solving the same linear problem using SNES. So I am setting the problem as follows: > > SNESCreate(PETSC_COMM_WORLD, &snes); > MatCreateShell(PETSC_COMM_WORLD, n_dofs, n_dofs, PETSC_DETERMINE, PETSC_DETERMINE, &ctx, &J); > MatCreateShell(PETSC_COMM_WORLD, n_dofs, n_dofs, PETSC_DETERMINE, PETSC_DETERMINE, &ctx, &B); > MatShellSetOperation(J, MATOP_MULT, (void (*)(void))(Multiplication)); > MatCreateVecs(J, &x_sol, &b); > VecDuplicate(x_sol, &r); > SNESSetFromOptions(snes); > SNESSetFunction(snes, r, &(computeResidual), &ctx); > SNESSetUseMatrixFree(snes, PETSC_FALSE, PETSC_TRUE); > SNESGetLineSearch(snes, &linesearch); > SNESGetKSP(snes, &ksp); > KSPSetOperators(ksp, J, J); > KSPSetInitialGuessNonzero(ksp, PETSC_TRUE); > > I tested it with a small problem (compiled in debug) and it works. > > When I compiled it in Release, it crashes with a segfault. I tried running the Debug version through valgrind, but even for a small problem, it is too slow. So I was wondering if you guys could see any rocky mistake on the lines I used above? > > Otherwise, is there any example that uses a SNES combined with a matrix free KSP operator? > > Thank you, > > Daniel > From daniel.pino_munoz at mines-paristech.fr Tue Oct 29 14:13:22 2024 From: daniel.pino_munoz at mines-paristech.fr (Daniel Pino Munoz) Date: Tue, 29 Oct 2024 20:13:22 +0100 Subject: [petsc-users] Example of SNES using matrix free jacobian In-Reply-To: References: <161ca7d4-82c4-4696-b123-4d441eb903da@mines-paristech.fr> Message-ID: Hi Barry, Thanks for getting back to me! I tried replacing KSPSetOperators(ksp, J, J); by SNESSetJacobian(snes,J,J, MatMFFDComputeJacobian) and I get the same result = It works in Debug mode but not in Release. I also ran valgrind and it did not catch any memory problem. Any ideas? PS : You are right regarding the number of iterations of the non preconditioned problem. In the previous version of the code that only used a KSP, I already had to set -ksp_gmres_restart 100. But thanks for the heads up. Best, ? Daniel On 29/10/2024 20:01, Barry Smith wrote: > Don't call > > KSPSetOperators(ksp, J, J); > > > instead call > > SNESSetJacobian(snes,J,J, MatMFFDComputeJacobian) > > but I am not sure that would explain the crash. > > BTW: since you are applying no preconditioner if the matrix is ill-conditioned it may take many iterations or not converge. You can try something like -ksp_gmres_restart 100 or similar value to try to improve convergence (default is 30). > > Barry > > > > >> On Oct 29, 2024, at 12:37?PM, Daniel Pino Munoz wrote: >> >> Dear all, >> >> I have a linear problem that I am currently solving with a KSP matrix-free. >> >> I would like to move on to a non linear problem, so figure I could start by solving the same linear problem using SNES. So I am setting the problem as follows: >> >> SNESCreate(PETSC_COMM_WORLD, &snes); >> MatCreateShell(PETSC_COMM_WORLD, n_dofs, n_dofs, PETSC_DETERMINE, PETSC_DETERMINE, &ctx, &J); >> MatCreateShell(PETSC_COMM_WORLD, n_dofs, n_dofs, PETSC_DETERMINE, PETSC_DETERMINE, &ctx, &B); >> MatShellSetOperation(J, MATOP_MULT, (void (*)(void))(Multiplication)); >> MatCreateVecs(J, &x_sol, &b); >> VecDuplicate(x_sol, &r); >> SNESSetFromOptions(snes); >> SNESSetFunction(snes, r, &(computeResidual), &ctx); >> SNESSetUseMatrixFree(snes, PETSC_FALSE, PETSC_TRUE); >> SNESGetLineSearch(snes, &linesearch); >> SNESGetKSP(snes, &ksp); >> KSPSetOperators(ksp, J, J); >> KSPSetInitialGuessNonzero(ksp, PETSC_TRUE); >> >> I tested it with a small problem (compiled in debug) and it works. >> >> When I compiled it in Release, it crashes with a segfault. I tried running the Debug version through valgrind, but even for a small problem, it is too slow. So I was wondering if you guys could see any rocky mistake on the lines I used above? >> >> Otherwise, is there any example that uses a SNES combined with a matrix free KSP operator? >> >> Thank you, >> >> Daniel >> From mail2amneet at gmail.com Tue Oct 29 14:47:26 2024 From: mail2amneet at gmail.com (Amneet Bhalla) Date: Tue, 29 Oct 2024 12:47:26 -0700 Subject: [petsc-users] Rigid body nullspace for Stokes operator In-Reply-To: References: Message-ID: Hi Mark, Thanks! I am not sure how to construct null space and zero energy modes manually for this operator. Is there some theory or documentation I can follow to figure out what the null space and zero energy modes look like for this operator? Once I know what these are in symbolic form, I think I should be able to construct them manually. Best, --Amneet On Tue, Oct 29, 2024 at 7:35?AM Mark Adams wrote: > Oh my mistake. You are using staggered grids. So you don't have a block > size that hypre would use for the "nodal" methods. > I'm not sure what you are doing exactly, but try hypre and you could > create the null space, zero energy modes, manually, attach to the matrix > and try GAMG. > You can run with '-info :pc' and grep on GAMG to see if GAMG is picking > the null space up (send this output if you can't figure it out). > > Thanks, > Mark > > On Tue, Oct 29, 2024 at 9:28?AM Mark Adams wrote: > >> This coordinate interface is just a shortcut for vertex based >> discretizations with 3 dof per vertex, etc. (maybe works in 2D). >> You will need to construct the null space vectors manually and attach it >> to the matrix. Used by GAMG. >> >> Note, for hypre you want to use the "nodal" options and it does not use >> these null space vectors. That is probably the way you want to go. >> eg: -pc_hypre_boomeramg_nodal_coarsen >> >> I would run with hypre boomerang and -help and grep on nodal to see all >> the "nodal" options and use them. >> >> Thanks, >> Mark >> >> >> On Mon, Oct 28, 2024 at 8:06?PM Amneet Bhalla >> wrote: >> >>> Hi Folks, >>> >>> I am trying to solve the momentum equation in a projection >>> preconditioner using GAMG or Hypre solver. The equation looks like for >>> velocity variable *v* looks like: >>> >>> >>> [image: Screenshot 2024-10-28 at 4.15.17 PM.png] >>> >>> Here, ? is spatially varying dynamic viscosity and ? is spatially >>> varying bulk viscosity. I understand that I need to specify rigid body >>> nullspace modes to the multigrid solver in order to accelerate its >>> convergence. Looking into this routine MatNullSpaceCreateRigidBody() ( >>> https://urldefense.us/v3/__https://petsc.org/release/manualpages/Mat/MatNullSpaceCreateRigidBody/__;!!G_uCfscf7eWS!aMSTPnVUiWdbtpI6UINZH776-nK60U2dgj1Bdi0emhPwrO4arqfcByiLjc2h5-zJ200uYPq7HgtpUtfCgZLepbk2Fw$ >>> ), >>> I see that I need to provide the coordinates of each node. I am using >>> staggered grid discretization. Do I need to provide coordinates of >>> staggered grid locations? >>> >>> Thanks, >>> -- >>> --Amneet >>> >>> >>> >>> -- --Amneet -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Screenshot 2024-10-28 at 4.15.17 PM.png Type: image/png Size: 17171 bytes Desc: not available URL: From bsmith at petsc.dev Tue Oct 29 14:17:59 2024 From: bsmith at petsc.dev (Barry Smith) Date: Tue, 29 Oct 2024 15:17:59 -0400 Subject: [petsc-users] Example of SNES using matrix free jacobian In-Reply-To: References: <161ca7d4-82c4-4696-b123-4d441eb903da@mines-paristech.fr> Message-ID: <7096B68A-67A7-41AA-9DAF-A3FE78C8679E@petsc.dev> Hmm, cut and paste the output when it crashes. Also run with -malloc_debug and see what happens Barry > On Oct 29, 2024, at 3:13?PM, Daniel Pino Munoz wrote: > > Hi Barry, > > Thanks for getting back to me! > > I tried replacing KSPSetOperators(ksp, J, J); by SNESSetJacobian(snes,J,J, MatMFFDComputeJacobian) > > and I get the same result = It works in Debug mode but not in Release. I also ran valgrind and it did not catch any memory problem. > > Any ideas? > > PS : You are right regarding the number of iterations of the non preconditioned problem. In the previous version of the code that only used a KSP, I already had to set -ksp_gmres_restart 100. But thanks for the heads up. > > Best, > > Daniel > > On 29/10/2024 20:01, Barry Smith wrote: >> Don't call >> >> KSPSetOperators(ksp, J, J); >> >> >> instead call >> >> SNESSetJacobian(snes,J,J, MatMFFDComputeJacobian) >> >> but I am not sure that would explain the crash. >> >> BTW: since you are applying no preconditioner if the matrix is ill-conditioned it may take many iterations or not converge. You can try something like -ksp_gmres_restart 100 or similar value to try to improve convergence (default is 30). >> >> Barry >> >> >> >> >>> On Oct 29, 2024, at 12:37?PM, Daniel Pino Munoz wrote: >>> >>> Dear all, >>> >>> I have a linear problem that I am currently solving with a KSP matrix-free. >>> >>> I would like to move on to a non linear problem, so figure I could start by solving the same linear problem using SNES. So I am setting the problem as follows: >>> >>> SNESCreate(PETSC_COMM_WORLD, &snes); >>> MatCreateShell(PETSC_COMM_WORLD, n_dofs, n_dofs, PETSC_DETERMINE, PETSC_DETERMINE, &ctx, &J); >>> MatCreateShell(PETSC_COMM_WORLD, n_dofs, n_dofs, PETSC_DETERMINE, PETSC_DETERMINE, &ctx, &B); >>> MatShellSetOperation(J, MATOP_MULT, (void (*)(void))(Multiplication)); >>> MatCreateVecs(J, &x_sol, &b); >>> VecDuplicate(x_sol, &r); >>> SNESSetFromOptions(snes); >>> SNESSetFunction(snes, r, &(computeResidual), &ctx); >>> SNESSetUseMatrixFree(snes, PETSC_FALSE, PETSC_TRUE); >>> SNESGetLineSearch(snes, &linesearch); >>> SNESGetKSP(snes, &ksp); >>> KSPSetOperators(ksp, J, J); >>> KSPSetInitialGuessNonzero(ksp, PETSC_TRUE); >>> >>> I tested it with a small problem (compiled in debug) and it works. >>> >>> When I compiled it in Release, it crashes with a segfault. I tried running the Debug version through valgrind, but even for a small problem, it is too slow. So I was wondering if you guys could see any rocky mistake on the lines I used above? >>> >>> Otherwise, is there any example that uses a SNES combined with a matrix free KSP operator? >>> >>> Thank you, >>> >>> Daniel >>> From daniel.pino_munoz at mines-paristech.fr Tue Oct 29 15:28:49 2024 From: daniel.pino_munoz at mines-paristech.fr (Daniel Pino Munoz) Date: Tue, 29 Oct 2024 21:28:49 +0100 Subject: [petsc-users] Example of SNES using matrix free jacobian In-Reply-To: <7096B68A-67A7-41AA-9DAF-A3FE78C8679E@petsc.dev> References: <161ca7d4-82c4-4696-b123-4d441eb903da@mines-paristech.fr> <7096B68A-67A7-41AA-9DAF-A3FE78C8679E@petsc.dev> Message-ID: <4257d3ef-9a69-4b3c-a435-f8f9fe77f50c@mines-paristech.fr> I ran it with -malloc_debug and it does not change anything. The output is the following: he absolute tolerance is 0.001 The relative tolerance is 0.001 The divergence tolerance is 10000 The maximum iterations is 10000 Initial load ! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see https://urldefense.us/v3/__https://petsc.org/release/faq/*valgrind__;Iw!!G_uCfscf7eWS!ecHZVRiYJFEV3Ru8URyFbEGJ2vXQ2KQNNyMC0E_KEclprR38CfKltYBqj_PQOatAvKs-QLz5Xh161SwNgfph8H2kFhfcV5HZZemMgh7IVw$ and https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!ecHZVRiYJFEV3Ru8URyFbEGJ2vXQ2KQNNyMC0E_KEclprR38CfKltYBqj_PQOatAvKs-QLz5Xh161SwNgfph8H2kFhfcV5HZZekrkUwXlA$ [0]PETSC ERROR: ---------------------? Stack Frames ------------------------------------ [0]PETSC ERROR: The line numbers in the error traceback are not always exact. [0]PETSC ERROR: #1 SNES callback function [0]PETSC ERROR: #2 SNESComputeFunction() at /home/daniel-pino/Software/Dependencies/petsc/src/snes/interface/snes.c:2489 [0]PETSC ERROR: #3 SNESSolve_KSPONLY() at /home/daniel-pino/Software/Dependencies/petsc/src/snes/impls/ksponly/ksponly.c:27 [0]PETSC ERROR: #4 SNESSolve() at /home/daniel-pino/Software/Dependencies/petsc/src/snes/interface/snes.c:4841 -------------------------------------------------------------------------- MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD ? Proc: [[27669,1],0] ? Errorcode: 59 NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. You may or may not see output from other processes, depending on exactly when Open MPI kills them. -------------------------------------------------------------------------- On 29/10/2024 20:17, Barry Smith wrote: > Hmm, cut and paste the output when it crashes. > > Also run with -malloc_debug and see what happens > > > Barry > > >> On Oct 29, 2024, at 3:13?PM, Daniel Pino Munoz wrote: >> >> Hi Barry, >> >> Thanks for getting back to me! >> >> I tried replacing KSPSetOperators(ksp, J, J); by SNESSetJacobian(snes,J,J, MatMFFDComputeJacobian) >> >> and I get the same result = It works in Debug mode but not in Release. I also ran valgrind and it did not catch any memory problem. >> >> Any ideas? >> >> PS : You are right regarding the number of iterations of the non preconditioned problem. In the previous version of the code that only used a KSP, I already had to set -ksp_gmres_restart 100. But thanks for the heads up. >> >> Best, >> >> Daniel >> >> On 29/10/2024 20:01, Barry Smith wrote: >>> Don't call >>> >>> KSPSetOperators(ksp, J, J); >>> >>> >>> instead call >>> >>> SNESSetJacobian(snes,J,J, MatMFFDComputeJacobian) >>> >>> but I am not sure that would explain the crash. >>> >>> BTW: since you are applying no preconditioner if the matrix is ill-conditioned it may take many iterations or not converge. You can try something like -ksp_gmres_restart 100 or similar value to try to improve convergence (default is 30). >>> >>> Barry >>> >>> >>> >>> >>>> On Oct 29, 2024, at 12:37?PM, Daniel Pino Munoz wrote: >>>> >>>> Dear all, >>>> >>>> I have a linear problem that I am currently solving with a KSP matrix-free. >>>> >>>> I would like to move on to a non linear problem, so figure I could start by solving the same linear problem using SNES. So I am setting the problem as follows: >>>> >>>> SNESCreate(PETSC_COMM_WORLD, &snes); >>>> MatCreateShell(PETSC_COMM_WORLD, n_dofs, n_dofs, PETSC_DETERMINE, PETSC_DETERMINE, &ctx, &J); >>>> MatCreateShell(PETSC_COMM_WORLD, n_dofs, n_dofs, PETSC_DETERMINE, PETSC_DETERMINE, &ctx, &B); >>>> MatShellSetOperation(J, MATOP_MULT, (void (*)(void))(Multiplication)); >>>> MatCreateVecs(J, &x_sol, &b); >>>> VecDuplicate(x_sol, &r); >>>> SNESSetFromOptions(snes); >>>> SNESSetFunction(snes, r, &(computeResidual), &ctx); >>>> SNESSetUseMatrixFree(snes, PETSC_FALSE, PETSC_TRUE); >>>> SNESGetLineSearch(snes, &linesearch); >>>> SNESGetKSP(snes, &ksp); >>>> KSPSetOperators(ksp, J, J); >>>> KSPSetInitialGuessNonzero(ksp, PETSC_TRUE); >>>> >>>> I tested it with a small problem (compiled in debug) and it works. >>>> >>>> When I compiled it in Release, it crashes with a segfault. I tried running the Debug version through valgrind, but even for a small problem, it is too slow. So I was wondering if you guys could see any rocky mistake on the lines I used above? >>>> >>>> Otherwise, is there any example that uses a SNES combined with a matrix free KSP operator? >>>> >>>> Thank you, >>>> >>>> Daniel >>>> From bsmith at petsc.dev Tue Oct 29 15:31:35 2024 From: bsmith at petsc.dev (Barry Smith) Date: Tue, 29 Oct 2024 16:31:35 -0400 Subject: [petsc-users] Example of SNES using matrix free jacobian In-Reply-To: <4257d3ef-9a69-4b3c-a435-f8f9fe77f50c@mines-paristech.fr> References: <161ca7d4-82c4-4696-b123-4d441eb903da@mines-paristech.fr> <7096B68A-67A7-41AA-9DAF-A3FE78C8679E@petsc.dev> <4257d3ef-9a69-4b3c-a435-f8f9fe77f50c@mines-paristech.fr> Message-ID: <78522DC0-7204-42C0-8A0C-47E2CA55AB80@petsc.dev> This [0]PETSC ERROR: #1 SNES callback function indicates the crash is in your computeResidual function and thus you need to debug your function Barry > On Oct 29, 2024, at 4:28?PM, Daniel Pino Munoz wrote: > > I ran it with -malloc_debug and it does not change anything. > > The output is the following: > > he absolute tolerance is 0.001 > The relative tolerance is 0.001 > The divergence tolerance is 10000 > The maximum iterations is 10000 > Initial load ! > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range > [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [0]PETSC ERROR: or see https://urldefense.us/v3/__https://petsc.org/release/faq/*valgrind__;Iw!!G_uCfscf7eWS!eQxllmSvw4WuVrXfPz5uK3h9cSjVYg1fCeWM8bhtm4NwaGhghQVhAMg18ppPqOQxe2lkX-UQyOXgJx91MV6-KqE$ and https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!eQxllmSvw4WuVrXfPz5uK3h9cSjVYg1fCeWM8bhtm4NwaGhghQVhAMg18ppPqOQxe2lkX-UQyOXgJx910VEZbZQ$ > [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ > [0]PETSC ERROR: The line numbers in the error traceback are not always exact. > [0]PETSC ERROR: #1 SNES callback function > [0]PETSC ERROR: #2 SNESComputeFunction() at /home/daniel-pino/Software/Dependencies/petsc/src/snes/interface/snes.c:2489 > [0]PETSC ERROR: #3 SNESSolve_KSPONLY() at /home/daniel-pino/Software/Dependencies/petsc/src/snes/impls/ksponly/ksponly.c:27 > [0]PETSC ERROR: #4 SNESSolve() at /home/daniel-pino/Software/Dependencies/petsc/src/snes/interface/snes.c:4841 > -------------------------------------------------------------------------- > MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD > Proc: [[27669,1],0] > Errorcode: 59 > > NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. > You may or may not see output from other processes, depending on > exactly when Open MPI kills them. > -------------------------------------------------------------------------- > > > > On 29/10/2024 20:17, Barry Smith wrote: >> Hmm, cut and paste the output when it crashes. >> >> Also run with -malloc_debug and see what happens >> >> >> Barry >> >> >>> On Oct 29, 2024, at 3:13?PM, Daniel Pino Munoz wrote: >>> >>> Hi Barry, >>> >>> Thanks for getting back to me! >>> >>> I tried replacing KSPSetOperators(ksp, J, J); by SNESSetJacobian(snes,J,J, MatMFFDComputeJacobian) >>> >>> and I get the same result = It works in Debug mode but not in Release. I also ran valgrind and it did not catch any memory problem. >>> >>> Any ideas? >>> >>> PS : You are right regarding the number of iterations of the non preconditioned problem. In the previous version of the code that only used a KSP, I already had to set -ksp_gmres_restart 100. But thanks for the heads up. >>> >>> Best, >>> >>> Daniel >>> >>> On 29/10/2024 20:01, Barry Smith wrote: >>>> Don't call >>>> >>>> KSPSetOperators(ksp, J, J); >>>> >>>> >>>> instead call >>>> >>>> SNESSetJacobian(snes,J,J, MatMFFDComputeJacobian) >>>> >>>> but I am not sure that would explain the crash. >>>> >>>> BTW: since you are applying no preconditioner if the matrix is ill-conditioned it may take many iterations or not converge. You can try something like -ksp_gmres_restart 100 or similar value to try to improve convergence (default is 30). >>>> >>>> Barry >>>> >>>> >>>> >>>> >>>>> On Oct 29, 2024, at 12:37?PM, Daniel Pino Munoz wrote: >>>>> >>>>> Dear all, >>>>> >>>>> I have a linear problem that I am currently solving with a KSP matrix-free. >>>>> >>>>> I would like to move on to a non linear problem, so figure I could start by solving the same linear problem using SNES. So I am setting the problem as follows: >>>>> >>>>> SNESCreate(PETSC_COMM_WORLD, &snes); >>>>> MatCreateShell(PETSC_COMM_WORLD, n_dofs, n_dofs, PETSC_DETERMINE, PETSC_DETERMINE, &ctx, &J); >>>>> MatCreateShell(PETSC_COMM_WORLD, n_dofs, n_dofs, PETSC_DETERMINE, PETSC_DETERMINE, &ctx, &B); >>>>> MatShellSetOperation(J, MATOP_MULT, (void (*)(void))(Multiplication)); >>>>> MatCreateVecs(J, &x_sol, &b); >>>>> VecDuplicate(x_sol, &r); >>>>> SNESSetFromOptions(snes); >>>>> SNESSetFunction(snes, r, &(computeResidual), &ctx); >>>>> SNESSetUseMatrixFree(snes, PETSC_FALSE, PETSC_TRUE); >>>>> SNESGetLineSearch(snes, &linesearch); >>>>> SNESGetKSP(snes, &ksp); >>>>> KSPSetOperators(ksp, J, J); >>>>> KSPSetInitialGuessNonzero(ksp, PETSC_TRUE); >>>>> >>>>> I tested it with a small problem (compiled in debug) and it works. >>>>> >>>>> When I compiled it in Release, it crashes with a segfault. I tried running the Debug version through valgrind, but even for a small problem, it is too slow. So I was wondering if you guys could see any rocky mistake on the lines I used above? >>>>> >>>>> Otherwise, is there any example that uses a SNES combined with a matrix free KSP operator? >>>>> >>>>> Thank you, >>>>> >>>>> Daniel >>>>> From daniel.pino_munoz at mines-paristech.fr Tue Oct 29 15:35:12 2024 From: daniel.pino_munoz at mines-paristech.fr (Daniel Pino Munoz) Date: Tue, 29 Oct 2024 21:35:12 +0100 Subject: [petsc-users] Example of SNES using matrix free jacobian In-Reply-To: <78522DC0-7204-42C0-8A0C-47E2CA55AB80@petsc.dev> References: <161ca7d4-82c4-4696-b123-4d441eb903da@mines-paristech.fr> <7096B68A-67A7-41AA-9DAF-A3FE78C8679E@petsc.dev> <4257d3ef-9a69-4b3c-a435-f8f9fe77f50c@mines-paristech.fr> <78522DC0-7204-42C0-8A0C-47E2CA55AB80@petsc.dev> Message-ID: <40d13023-ede8-4226-bdd9-a8a495ef4daa@mines-paristech.fr> That's what I thought, so I replaced its content by: VecZeroEntries(f); and the result is the same... On 29/10/2024 21:31, Barry Smith wrote: > This > [0]PETSC ERROR: #1 SNES callback function > indicates the crash is in your computeResidual function and thus you need to debug your function > > Barry > > >> On Oct 29, 2024, at 4:28?PM, Daniel Pino Munoz wrote: >> >> I ran it with -malloc_debug and it does not change anything. >> >> The output is the following: >> >> he absolute tolerance is 0.001 >> The relative tolerance is 0.001 >> The divergence tolerance is 10000 >> The maximum iterations is 10000 >> Initial load ! >> [0]PETSC ERROR: ------------------------------------------------------------------------ >> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range >> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >> [0]PETSC ERROR: or see https://urldefense.us/v3/__https://petsc.org/release/faq/*valgrind__;Iw!!G_uCfscf7eWS!Z0ifVNZaie3VDyNzpdW9w-vk0Aw1cEjqvlrJcp_-d2R06pFj_9PAlB2x6azoU8bjbqzaGXlVlziswk5eBSowDWMil2MSKYzGXNx6bzkEpw$ and https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!Z0ifVNZaie3VDyNzpdW9w-vk0Aw1cEjqvlrJcp_-d2R06pFj_9PAlB2x6azoU8bjbqzaGXlVlziswk5eBSowDWMil2MSKYzGXNxyzMy5Zw$ >> [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ >> [0]PETSC ERROR: The line numbers in the error traceback are not always exact. >> [0]PETSC ERROR: #1 SNES callback function >> [0]PETSC ERROR: #2 SNESComputeFunction() at /home/daniel-pino/Software/Dependencies/petsc/src/snes/interface/snes.c:2489 >> [0]PETSC ERROR: #3 SNESSolve_KSPONLY() at /home/daniel-pino/Software/Dependencies/petsc/src/snes/impls/ksponly/ksponly.c:27 >> [0]PETSC ERROR: #4 SNESSolve() at /home/daniel-pino/Software/Dependencies/petsc/src/snes/interface/snes.c:4841 >> -------------------------------------------------------------------------- >> MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD >> Proc: [[27669,1],0] >> Errorcode: 59 >> >> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. >> You may or may not see output from other processes, depending on >> exactly when Open MPI kills them. >> -------------------------------------------------------------------------- >> >> >> >> On 29/10/2024 20:17, Barry Smith wrote: >>> Hmm, cut and paste the output when it crashes. >>> >>> Also run with -malloc_debug and see what happens >>> >>> >>> Barry >>> >>> >>>> On Oct 29, 2024, at 3:13?PM, Daniel Pino Munoz wrote: >>>> >>>> Hi Barry, >>>> >>>> Thanks for getting back to me! >>>> >>>> I tried replacing KSPSetOperators(ksp, J, J); by SNESSetJacobian(snes,J,J, MatMFFDComputeJacobian) >>>> >>>> and I get the same result = It works in Debug mode but not in Release. I also ran valgrind and it did not catch any memory problem. >>>> >>>> Any ideas? >>>> >>>> PS : You are right regarding the number of iterations of the non preconditioned problem. In the previous version of the code that only used a KSP, I already had to set -ksp_gmres_restart 100. But thanks for the heads up. >>>> >>>> Best, >>>> >>>> Daniel >>>> >>>> On 29/10/2024 20:01, Barry Smith wrote: >>>>> Don't call >>>>> >>>>> KSPSetOperators(ksp, J, J); >>>>> >>>>> >>>>> instead call >>>>> >>>>> SNESSetJacobian(snes,J,J, MatMFFDComputeJacobian) >>>>> >>>>> but I am not sure that would explain the crash. >>>>> >>>>> BTW: since you are applying no preconditioner if the matrix is ill-conditioned it may take many iterations or not converge. You can try something like -ksp_gmres_restart 100 or similar value to try to improve convergence (default is 30). >>>>> >>>>> Barry >>>>> >>>>> >>>>> >>>>> >>>>>> On Oct 29, 2024, at 12:37?PM, Daniel Pino Munoz wrote: >>>>>> >>>>>> Dear all, >>>>>> >>>>>> I have a linear problem that I am currently solving with a KSP matrix-free. >>>>>> >>>>>> I would like to move on to a non linear problem, so figure I could start by solving the same linear problem using SNES. So I am setting the problem as follows: >>>>>> >>>>>> SNESCreate(PETSC_COMM_WORLD, &snes); >>>>>> MatCreateShell(PETSC_COMM_WORLD, n_dofs, n_dofs, PETSC_DETERMINE, PETSC_DETERMINE, &ctx, &J); >>>>>> MatCreateShell(PETSC_COMM_WORLD, n_dofs, n_dofs, PETSC_DETERMINE, PETSC_DETERMINE, &ctx, &B); >>>>>> MatShellSetOperation(J, MATOP_MULT, (void (*)(void))(Multiplication)); >>>>>> MatCreateVecs(J, &x_sol, &b); >>>>>> VecDuplicate(x_sol, &r); >>>>>> SNESSetFromOptions(snes); >>>>>> SNESSetFunction(snes, r, &(computeResidual), &ctx); >>>>>> SNESSetUseMatrixFree(snes, PETSC_FALSE, PETSC_TRUE); >>>>>> SNESGetLineSearch(snes, &linesearch); >>>>>> SNESGetKSP(snes, &ksp); >>>>>> KSPSetOperators(ksp, J, J); >>>>>> KSPSetInitialGuessNonzero(ksp, PETSC_TRUE); >>>>>> >>>>>> I tested it with a small problem (compiled in debug) and it works. >>>>>> >>>>>> When I compiled it in Release, it crashes with a segfault. I tried running the Debug version through valgrind, but even for a small problem, it is too slow. So I was wondering if you guys could see any rocky mistake on the lines I used above? >>>>>> >>>>>> Otherwise, is there any example that uses a SNES combined with a matrix free KSP operator? >>>>>> >>>>>> Thank you, >>>>>> >>>>>> Daniel >>>>>> From bsmith at petsc.dev Tue Oct 29 17:04:08 2024 From: bsmith at petsc.dev (Barry Smith) Date: Tue, 29 Oct 2024 18:04:08 -0400 Subject: [petsc-users] Example of SNES using matrix free jacobian In-Reply-To: <40d13023-ede8-4226-bdd9-a8a495ef4daa@mines-paristech.fr> References: <161ca7d4-82c4-4696-b123-4d441eb903da@mines-paristech.fr> <7096B68A-67A7-41AA-9DAF-A3FE78C8679E@petsc.dev> <4257d3ef-9a69-4b3c-a435-f8f9fe77f50c@mines-paristech.fr> <78522DC0-7204-42C0-8A0C-47E2CA55AB80@petsc.dev> <40d13023-ede8-4226-bdd9-a8a495ef4daa@mines-paristech.fr> Message-ID: Run in the debugger, even though there will not be full debugging support with the optimizations it should still provide you some information > On Oct 29, 2024, at 4:35?PM, Daniel Pino Munoz wrote: > > That's what I thought, so I replaced its content by: > > VecZeroEntries(f); > > and the result is the same... > > > On 29/10/2024 21:31, Barry Smith wrote: >> This >> [0]PETSC ERROR: #1 SNES callback function >> indicates the crash is in your computeResidual function and thus you need to debug your function >> >> Barry >> >> >>> On Oct 29, 2024, at 4:28?PM, Daniel Pino Munoz wrote: >>> >>> I ran it with -malloc_debug and it does not change anything. >>> >>> The output is the following: >>> >>> he absolute tolerance is 0.001 >>> The relative tolerance is 0.001 >>> The divergence tolerance is 10000 >>> The maximum iterations is 10000 >>> Initial load ! >>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range >>> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >>> [0]PETSC ERROR: or see https://urldefense.us/v3/__https://petsc.org/release/faq/*valgrind__;Iw!!G_uCfscf7eWS!dibDYt_ZFa-6FBchlzMndWYg1NqvsEIZOJlfOEQkpzMOqyVUFiiumr5JERY5e-jjXiKd8sI5HQxEi-65JQwqWxc$ and https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!dibDYt_ZFa-6FBchlzMndWYg1NqvsEIZOJlfOEQkpzMOqyVUFiiumr5JERY5e-jjXiKd8sI5HQxEi-65GdSV720$ >>> [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ >>> [0]PETSC ERROR: The line numbers in the error traceback are not always exact. >>> [0]PETSC ERROR: #1 SNES callback function >>> [0]PETSC ERROR: #2 SNESComputeFunction() at /home/daniel-pino/Software/Dependencies/petsc/src/snes/interface/snes.c:2489 >>> [0]PETSC ERROR: #3 SNESSolve_KSPONLY() at /home/daniel-pino/Software/Dependencies/petsc/src/snes/impls/ksponly/ksponly.c:27 >>> [0]PETSC ERROR: #4 SNESSolve() at /home/daniel-pino/Software/Dependencies/petsc/src/snes/interface/snes.c:4841 >>> -------------------------------------------------------------------------- >>> MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD >>> Proc: [[27669,1],0] >>> Errorcode: 59 >>> >>> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. >>> You may or may not see output from other processes, depending on >>> exactly when Open MPI kills them. >>> -------------------------------------------------------------------------- >>> >>> >>> >>> On 29/10/2024 20:17, Barry Smith wrote: >>>> Hmm, cut and paste the output when it crashes. >>>> >>>> Also run with -malloc_debug and see what happens >>>> >>>> >>>> Barry >>>> >>>> >>>>> On Oct 29, 2024, at 3:13?PM, Daniel Pino Munoz wrote: >>>>> >>>>> Hi Barry, >>>>> >>>>> Thanks for getting back to me! >>>>> >>>>> I tried replacing KSPSetOperators(ksp, J, J); by SNESSetJacobian(snes,J,J, MatMFFDComputeJacobian) >>>>> >>>>> and I get the same result = It works in Debug mode but not in Release. I also ran valgrind and it did not catch any memory problem. >>>>> >>>>> Any ideas? >>>>> >>>>> PS : You are right regarding the number of iterations of the non preconditioned problem. In the previous version of the code that only used a KSP, I already had to set -ksp_gmres_restart 100. But thanks for the heads up. >>>>> >>>>> Best, >>>>> >>>>> Daniel >>>>> >>>>> On 29/10/2024 20:01, Barry Smith wrote: >>>>>> Don't call >>>>>> >>>>>> KSPSetOperators(ksp, J, J); >>>>>> >>>>>> >>>>>> instead call >>>>>> >>>>>> SNESSetJacobian(snes,J,J, MatMFFDComputeJacobian) >>>>>> >>>>>> but I am not sure that would explain the crash. >>>>>> >>>>>> BTW: since you are applying no preconditioner if the matrix is ill-conditioned it may take many iterations or not converge. You can try something like -ksp_gmres_restart 100 or similar value to try to improve convergence (default is 30). >>>>>> >>>>>> Barry >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> On Oct 29, 2024, at 12:37?PM, Daniel Pino Munoz wrote: >>>>>>> >>>>>>> Dear all, >>>>>>> >>>>>>> I have a linear problem that I am currently solving with a KSP matrix-free. >>>>>>> >>>>>>> I would like to move on to a non linear problem, so figure I could start by solving the same linear problem using SNES. So I am setting the problem as follows: >>>>>>> >>>>>>> SNESCreate(PETSC_COMM_WORLD, &snes); >>>>>>> MatCreateShell(PETSC_COMM_WORLD, n_dofs, n_dofs, PETSC_DETERMINE, PETSC_DETERMINE, &ctx, &J); >>>>>>> MatCreateShell(PETSC_COMM_WORLD, n_dofs, n_dofs, PETSC_DETERMINE, PETSC_DETERMINE, &ctx, &B); >>>>>>> MatShellSetOperation(J, MATOP_MULT, (void (*)(void))(Multiplication)); >>>>>>> MatCreateVecs(J, &x_sol, &b); >>>>>>> VecDuplicate(x_sol, &r); >>>>>>> SNESSetFromOptions(snes); >>>>>>> SNESSetFunction(snes, r, &(computeResidual), &ctx); >>>>>>> SNESSetUseMatrixFree(snes, PETSC_FALSE, PETSC_TRUE); >>>>>>> SNESGetLineSearch(snes, &linesearch); >>>>>>> SNESGetKSP(snes, &ksp); >>>>>>> KSPSetOperators(ksp, J, J); >>>>>>> KSPSetInitialGuessNonzero(ksp, PETSC_TRUE); >>>>>>> >>>>>>> I tested it with a small problem (compiled in debug) and it works. >>>>>>> >>>>>>> When I compiled it in Release, it crashes with a segfault. I tried running the Debug version through valgrind, but even for a small problem, it is too slow. So I was wondering if you guys could see any rocky mistake on the lines I used above? >>>>>>> >>>>>>> Otherwise, is there any example that uses a SNES combined with a matrix free KSP operator? >>>>>>> >>>>>>> Thank you, >>>>>>> >>>>>>> Daniel >>>>>>> From bsmith at petsc.dev Tue Oct 29 17:12:55 2024 From: bsmith at petsc.dev (Barry Smith) Date: Tue, 29 Oct 2024 18:12:55 -0400 Subject: [petsc-users] Problem running geoclaw example with petsc 3.22 In-Reply-To: References: <1C63D19F-6EBC-43A3-8B40-65EA34991AFA@gmail.com> <0136027D-C2CE-47D0-AF97-ACDE1DA0C382@gmail.com> <4726A341-54DB-4592-BBAE-843AF271F28E@petsc.dev> <27B47E2D-00D5-4326-95A9-7595DFF0E952@gmail.com> <490B5A4A-BBD8-490A-91FD-7CEC9E01A99B@petsc.dev> Message-ID: <106E3A52-15DE-4618-8ADC-269A07CD01FE@petsc.dev> Sorry for the delay, I've finally made some progress. Used your script to build the Conda environment. I was then able to run src/ksp/ksp/tutorials/ex1 example with the -mpi_linear_solver_server option successfully (no crash like you reported). Next step run the radial_flat geoclaw example > On Oct 24, 2024, at 12:17?PM, Praveen C wrote: > > Hello Barry > > I use this script to install clawpack and required dependencies > > https://urldefense.us/v3/__https://github.com/cpraveen/cfdlab/blob/master/bin/clawpack.sh__;!!G_uCfscf7eWS!cYF25X5EAlaskjgwI_7h2iV9k5frPImwuNtFQBgjkAbh78iJAMPQN8tmTthDfer0eDLZdmc-yATYdcABIVv8-MM$ > > See lines 125-132 > > You can use it like this > > export CLAW=/path/to/where/you/want/clawpack > bash clawpack.sh v5.11.0 > > This will git pull clawpack and creates a conda env called ?claw? and installs inside that. > > I have not specified petsc version in this script, but latest miniforge should install petsc at 3.22 > > Thank you > praveen > >> On 24 Oct 2024, at 9:37?PM, Barry Smith wrote: >> >> >> Good, some helpful information with the runs you made. >> >> In the crash below it made more progress it was able to allocate multiple regions of shared memory and access them. I don't know why it would crash later. >> >> Can you tell me all the steps with miniforge (which seems to be related to the failure) you use? I've never used miniforge. >> >> If I can get to an environment that reproduces the problem I can debug it and fix it. >> >> Barry >> >> >>> On Oct 24, 2024, at 11:49?AM, Praveen C wrote: >>> >>> I get this >>> >>> $ mpiexec -n 3 ./ex89f -n 20 -mpi_linear_solver_server -mpi_linear_solver_server -mpi_linear_solver_server_ksp_view -ksp_monitor -ksp_converged_reason -ksp_view -mpi_linear_solver_server_minimum_count_per_rank 5 >>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range >>> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >>> [0]PETSC ERROR: or see https://urldefense.us/v3/__https://petsc.org/release/faq/*valgrind__;Iw!!G_uCfscf7eWS!cYF25X5EAlaskjgwI_7h2iV9k5frPImwuNtFQBgjkAbh78iJAMPQN8tmTthDfer0eDLZdmc-yATYdcABCUbhRTs$ and https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!cYF25X5EAlaskjgwI_7h2iV9k5frPImwuNtFQBgjkAbh78iJAMPQN8tmTthDfer0eDLZdmc-yATYdcABzeXGc5s$ >>> [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run >>> [0]PETSC ERROR: to get more information on the crash. >>> [0]PETSC ERROR: Run with -malloc_debug to check if memory corruption is causing the crash. >>> -------------------------------------------------------------------------- >>> MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_SELF >>> Proc: [[47380,1],0] >>> Errorcode: 59 >>> >>> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. >>> You may or may not see output from other processes, depending on >>> exactly when Open MPI kills them. >>> -------------------------------------------------------------------------- >>> -------------------------------------------------------------------------- >>> prterun has exited due to process rank 0 with PID 0 on node MacMiniHome calling >>> "abort". This may have caused other processes in the application to be >>> terminated by signals sent by prterun (as reported here). >>> ????????????????????????????????????? >>> >>> and I have this after the code exits >>> >>> $ ipcs -m >>> IPC status from as of Thu Oct 24 21:17:39 IST 2024 >>> T ID KEY MODE OWNER GROUP >>> Shared Memory: >>> m 1572864 0x0000000b --rw-rw-rw- praveen staff >>> m 524289 0x0000000c --rw-rw-rw- praveen staff >>> m 655362 0x0000000d --rw-rw-rw- praveen staff >>> m 262147 0x0000000e --rw-rw-rw- praveen staff >>> m 262148 0x0000000f --rw-rw-rw- praveen staff >>> m 393221 0x0000000a --rw-rw-rw- praveen staff >>> >>> This is with petsc installed with miniforge, which I also use with clawpack. With spack installed petsc, I can run the ex89f example. >>> >>> Thanks >>> praveen >>> >>>> On 24 Oct 2024, at 7:55?PM, Barry Smith wrote: >>>> >>>> >>>> Ok, super strange, I've run many times on my Mac >>>> >>>> Can you please try to remove that allocated memory with ipcrm and then >>>> >>>> cd $PETSC_DIR/src/ksp/ksp/tutorials >>>> make ex89f >>>> mpiexec -n 3 ./ex89f -n 20 -mpi_linear_solver_server -mpi_linear_solver_server -mpi_linear_solver_server_ksp_view -ksp_monitor -ksp_converged_reason -ksp_view -mpi_linear_solver_server_minimum_count_per_rank 5 >>>> >>>> This does the same thing as the GeoClaw code but is much simpler. >>>> >>>> Barry >>>> >>>> >>>> >>>> >>>>> On Oct 23, 2024, at 10:55?PM, Praveen C wrote: >>>>> >>>>> I get very similar error on my mac with >>>>> >>>>> $ gfortran -v >>>>> Using built-in specs. >>>>> COLLECT_GCC=gfortran >>>>> COLLECT_LTO_WRAPPER=/opt/homebrew/Caskroom/miniforge/base/envs/claw/libexec/gcc/arm64-apple-darwin20.0.0/13.2.0/lto-wrapper >>>>> Target: arm64-apple-darwin20.0.0 >>>>> Configured with: ../configure --prefix=/opt/homebrew/Caskroom/miniforge/base/envs/claw --build=x86_64-apple-darwin13.4.0 --host=arm64-apple-darwin20.0.0 --target=arm64-apple-darwin20.0.0 --with-libiconv-prefix=/opt/homebrew/Caskroom/miniforge/base/envs/claw --enable-languages=fortran --disable-multilib --enable-checking=release --disable-bootstrap --disable-libssp --with-gmp=/opt/homebrew/Caskroom/miniforge/base/envs/claw --with-mpfr=/opt/homebrew/Caskroom/miniforge/base/envs/claw --with-mpc=/opt/homebrew/Caskroom/miniforge/base/envs/claw --with-isl=/opt/homebrew/Caskroom/miniforge/base/envs/claw --enable-darwin-at-rpath >>>>> Thread model: posix >>>>> Supported LTO compression algorithms: zlib >>>>> gcc version 13.2.0 (GCC) >>>>> >>>>> Before starting >>>>> >>>>> $ ipcs -m >>>>> IPC status from as of Thu Oct 24 08:02:11 IST 2024 >>>>> T ID KEY MODE OWNER GROUP >>>>> Shared Memory: >>>>> >>>>> and when I run the code >>>>> >>>>> Using a PETSc solver >>>>> Using Bouss equations from the start >>>>> rnode allocated... >>>>> node allocated... >>>>> listOfGrids allocated... >>>>> Storage allocated... >>>>> bndList allocated... >>>>> Gridding level 1 at t = 0.000000E+00: 4 grids with 10000 cells >>>>> Setting initial dt to 2.9999999999999999E-002 >>>>> max threads set to 1 >>>>> >>>>> Done reading data, starting computation ... >>>>> >>>>> Total zeta at initial time: 39269.907650665169 >>>>> GEOCLAW: Frame 0 output files done at time t = 0.000000D+00 >>>>> >>>>> [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >>>>> [0]PETSC ERROR: Petsc has generated inconsistent data >>>>> [0]PETSC ERROR: Unable to locate PCMPI allocated shared address 0x130698000 >>>>> [0]PETSC ERROR: WARNING! There are unused option(s) set! Could be the program crashed before usage or a spelling mistake, etc! >>>>> [0]PETSC ERROR: Option left: name:-ksp_type value: preonly source: file >>>>> [0]PETSC ERROR: Option left: name:-mpi_ksp_max_it value: 200 source: file >>>>> [0]PETSC ERROR: Option left: name:-mpi_ksp_reuse_preconditioner (no value) source: file >>>>> [0]PETSC ERROR: Option left: name:-mpi_ksp_rtol value: 1.e-9 source: file >>>>> [0]PETSC ERROR: Option left: name:-mpi_ksp_type value: gmres source: file >>>>> [0]PETSC ERROR: Option left: name:-mpi_linear_solver_server_view (no value) source: file >>>>> [0]PETSC ERROR: Option left: name:-mpi_pc_gamg_sym_graph value: true source: file >>>>> [0]PETSC ERROR: Option left: name:-mpi_pc_gamg_symmetrize_graph value: true source: file >>>>> [0]PETSC ERROR: Option left: name:-mpi_pc_type value: gamg source: file >>>>> [0]PETSC ERROR: Option left: name:-pc_mpi_minimum_count_per_rank value: 5000 source: file >>>>> [0]PETSC ERROR: Option left: name:-pc_type value: mpi source: file >>>>> [0]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!cYF25X5EAlaskjgwI_7h2iV9k5frPImwuNtFQBgjkAbh78iJAMPQN8tmTthDfer0eDLZdmc-yATYdcABzeXGc5s$ for trouble shooting. >>>>> [0]PETSC ERROR: Petsc Release Version 3.22.0, Sep 28, 2024 >>>>> [0]PETSC ERROR: /Users/praveen/work/bouss/radial_flat/xgeoclaw with 6 MPI process(es) and PETSC_ARCH on MacMiniHome.local by praveen Thu Oct 24 08:04:27 2024 >>>>> [0]PETSC ERROR: Configure options: AR=arm64-apple-darwin20.0.0-ar CC=mpicc CXX=mpicxx FC=mpifort CFLAGS="-ftree-vectorize -fPIC -fstack-protector-strong -O2 -pipe -isystem /opt/homebrew/Caskroom/miniforge/base/envs/claw/include " CPPFLAGS="-D_FORTIFY_SOURCE=2 -isystem /opt/homebrew/Caskroom/miniforge/base/envs/claw/include -mmacosx-version-min=11.0 -mmacosx-version-min=11.0" CXXFLAGS="-ftree-vectorize -fPIC -fstack-protector-strong -O2 -pipe -stdlib=libc++ -fvisibility-inlines-hidden -fmessage-length=0 -isystem /opt/homebrew/Caskroom/miniforge/base/envs/claw/include " FFLAGS="-march=armv8.3-a -ftree-vectorize -fPIC -fno-stack-protector -O2 -pipe -isystem /opt/homebrew/Caskroom/miniforge/base/envs/claw/include " LDFLAGS="-Wl,-headerpad_max_install_names -Wl,-dead_strip_dylibs -Wl,-rpath,/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib -L/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib" LIBS="-Wl,-rpath,/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib -lmpi_mpifh -lgfortran" --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 --FOPTFLAGS=-O3 --with-clib-autodetect=0 --with-cxxlib-autodetect=0 --with-fortranlib-autodetect=0 --with-debugging=0 --with-blas-lib=libblas.dylib --with-lapack-lib=liblapack.dylib --with-yaml=1 --with-hdf5=1 --with-fftw=1 --with-hwloc=0 --with-hypre=1 --with-metis=1 --with-mpi=1 --with-mumps=1 --with-parmetis=1 --with-pthread=1 --with-ptscotch=1 --with-shared-libraries --with-ssl=0 --with-scalapack=1 --with-superlu=1 --with-superlu_dist=1 --with-superlu_dist-include=/opt/homebrew/Caskroom/miniforge/base/envs/claw/include/superlu-dist --with-superlu_dist-lib=-lsuperlu_dist --with-suitesparse=1 --with-suitesparse-dir=/opt/homebrew/Caskroom/miniforge/base/envs/claw --with-x=0 --with-scalar-type=real --with-cuda=0 --with-batch --prefix=/opt/homebrew/Caskroom/miniforge/base/envs/claw >>>>> [0]PETSC ERROR: #1 PetscShmgetMapAddresses() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/sys/utils/server.c:114 >>>>> [0]PETSC ERROR: #2 PCMPISetMat() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/ksp/pc/impls/mpi/pcmpi.c:269 >>>>> [0]PETSC ERROR: #3 PCSetUp_MPI() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/ksp/pc/impls/mpi/pcmpi.c:853 >>>>> [0]PETSC ERROR: #4 PCSetUp() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/ksp/pc/interface/precon.c:1071 >>>>> [0]PETSC ERROR: #5 KSPSetUp() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/ksp/ksp/interface/itfunc.c:415 >>>>> [0]PETSC ERROR: #6 KSPSolve_Private() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/ksp/ksp/interface/itfunc.c:826 >>>>> [0]PETSC ERROR: #7 KSPSolve() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/ksp/ksp/interface/itfunc.c:1075 >>>>> >>>>> Code does not progress and I kill it >>>>> >>>>> ^CTraceback (most recent call last): >>>>> File "/Users/praveen/Applications/clawpack/clawutil/src/python/clawutil/runclaw.py", line 341, in >>>>> runclaw(*args) >>>>> File "/Users/praveen/Applications/clawpack/clawutil/src/python/clawutil/runclaw.py", line 242, in runclaw >>>>> proc = subprocess.check_call(cmd_split, >>>>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ >>>>> File "/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib/python3.12/subprocess.py", line 408, in check_call >>>>> retcode = call(*popenargs, **kwargs) >>>>> ^^^^^^^^^^^^^^^^^^^^^^^^^^ >>>>> File "/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib/python3.12/subprocess.py", line 391, in call >>>>> return p.wait(timeout=timeout) >>>>> ^^^^^^^^^^^^^^^^^^^^^^^ >>>>> File "/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib/python3.12/subprocess.py", line 1264, in wait >>>>> return self._wait(timeout=timeout) >>>>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^ >>>>> File "/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib/python3.12/subprocess.py", line 2053, in _wait >>>>> (pid, sts) = self._try_wait(0) >>>>> ^^^^^^^^^^^^^^^^^ >>>>> File "/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib/python3.12/subprocess.py", line 2011, in _try_wait >>>>> (pid, sts) = os.waitpid(self.pid, wait_flags) >>>>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ >>>>> KeyboardInterrupt >>>>> make[1]: *** [output] Interrupt: 2 >>>>> make: *** [.output] Interrupt: 2 >>>>> >>>>> Now it says >>>>> >>>>> $ ipcs -m >>>>> IPC status from as of Thu Oct 24 08:05:06 IST 2024 >>>>> T ID KEY MODE OWNER GROUP >>>>> Shared Memory: >>>>> m 720896 0x0000000a --rw-rw-rw- praveen staff >>>>> >>>>> Thanks >>>>> praveen >>>>> >>>>>> On 23 Oct 2024, at 8:26?PM, Barry Smith wrote: >>>>>> >>>>>> >>>>>> Hmm, so it is creating the first shared memory region with ID of 10 (A in hex) puts it in a linked list in PETSc but then when it tries to find it in the linked list it cannot find it. >>>>>> >>>>>> I don't know how to reproduce this or debug it remotely. >>>>>> >>>>>> Can you build on a completely different machine or with completely different compilers? >>>>>> >>>>>> Barry >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> On Oct 23, 2024, at 10:31?AM, Praveen C wrote: >>>>>>> >>>>>>> I get same error and now it shows >>>>>>> >>>>>>> $ ipcs -m >>>>>>> >>>>>>> ------ Shared Memory Segments -------- >>>>>>> key shmid owner perms bytes nattch status >>>>>>> 0x0000000a 32788 praveen 666 240 6 >>>>>>> >>>>>>> Note that the code seems to be still running after printing those error message, but it is not printing any progress which it should do. >>>>>>> >>>>>>> Thanks >>>>>>> praveen >>>>>>> >>>>>>>> On 23 Oct 2024, at 7:56?PM, Barry Smith wrote: >>>>>>>> >>>>>>>> >>>>>>>> Try >>>>>>>> >>>>>>>> ipcrm -m 11 >>>>>>>> >>>>>>>> ipcs -m >>>>>>>> >>>>>>>> Try running the program again >>>>>>>> >>>>>>>> If failed check >>>>>>>> >>>>>>>> ipcs -m >>>>>>>> >>>>>>>> again >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> On Oct 23, 2024, at 10:20?AM, Praveen C wrote: >>>>>>>>> >>>>>>>>> Hello Barry >>>>>>>>> >>>>>>>>> I see this >>>>>>>>> >>>>>>>>> $ ipcs -m >>>>>>>>> >>>>>>>>> ------ Shared Memory Segments -------- >>>>>>>>> key shmid owner perms bytes nattch status >>>>>>>>> 0x0000000a 11 praveen 666 240 6 >>>>>>>>> >>>>>>>>> and I am observing same error as below. >>>>>>>>> >>>>>>>>> Thanks >>>>>>>>> praveen >>>>>>>>> >>>>>>>>>> On 23 Oct 2024, at 7:08?PM, Barry Smith wrote: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Please take a look at the notes in https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/PetscShmgetAllocateArray/__;!!G_uCfscf7eWS!cYF25X5EAlaskjgwI_7h2iV9k5frPImwuNtFQBgjkAbh78iJAMPQN8tmTthDfer0eDLZdmc-yATYdcABFss4_0M$ For some reason your program is not able to access/use the Unix shared memory; check if you are already using the shared memory (so it is not available for a new run) or the limits are too low to access enough memory. >>>>>>>>>> >>>>>>>>>> Barry >>>>>>>>>> >>>>>>>>>>> On Oct 23, 2024, at 8:23?AM, Praveen C wrote: >>>>>>>>>>> >>>>>>>>>>> Dear all >>>>>>>>>>> >>>>>>>>>>> I am not able to run the boussinesq example from geoclaw using petsc at 3.22.0 >>>>>>>>>>> >>>>>>>>>>> https://urldefense.us/v3/__https://github.com/clawpack/geoclaw/tree/3303883f46572c58130d161986b8a87a57ca7816/examples/bouss__;!!G_uCfscf7eWS!e3VQ4NHKmXGstRsQW5vtI7fmKfUT9zmJkMJcPbcvPyIjicyfJpNoMgx3wZ-qyGcKNSjIkNZkzilec8MnHN6PMw$ >>>>>>>>>>> >>>>>>>>>>> It runs with petsc at 3.21.6 >>>>>>>>>>> >>>>>>>>>>> The error I get is given below. After printing this, the code does not progress. >>>>>>>>>>> >>>>>>>>>>> I use the following petsc options >>>>>>>>>>> >>>>>>>>>>> # set min numbers of matrix rows per MPI rank (default is 10000) >>>>>>>>>>> -mpi_linear_solve_minimum_count_per_rank 5000 >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> # Krylov linear solver: >>>>>>>>>>> -mpi_linear_solver_server >>>>>>>>>>> -mpi_linear_solver_server_view >>>>>>>>>>> -ksp_type gmres >>>>>>>>>>> -ksp_max_it 200 >>>>>>>>>>> -ksp_reuse_preconditioner >>>>>>>>>>> -ksp_rtol 1.e-9 >>>>>>>>>>> >>>>>>>>>>> # preconditioner: >>>>>>>>>>> -pc_type gamg >>>>>>>>>>> >>>>>>>>>>> I installed petsc and other dependencies for clawpack using miniforge. >>>>>>>>>>> >>>>>>>>>>> Thanks >>>>>>>>>>> pc >>>>>>>>>>> >>>>>>>>>>> ==> Use Bouss. in water deeper than 1.0000000000000000 Using a PETSc solver >>>>>>>>>>> Using Bouss equations from the start >>>>>>>>>>> rnode allocated... >>>>>>>>>>> node allocated... >>>>>>>>>>> listOfGrids allocated... >>>>>>>>>>> Storage allocated... >>>>>>>>>>> bndList allocated... >>>>>>>>>>> Gridding level 1 at t = 0.000000E+00: 4 grids with 10000 cells >>>>>>>>>>> Setting initial dt to 2.9999999999999999E-002 >>>>>>>>>>> max threads set to 6 >>>>>>>>>>> Done reading data, starting computation ... Total zeta at initial time: 39269.907650665169 GEOCLAW: Frame 0 output files done at time t = 0.000000D+00 >>>>>>>>>>> >>>>>>>>>>> [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >>>>>>>>>>> [0]PETSC ERROR: Petsc has generated inconsistent data >>>>>>>>>>> [0]PETSC ERROR: Unable to locate PCMPI allocated shared address 0x55e6d750ae20 >>>>>>>>>>> [0]PETSC ERROR: WARNING! There are unused option(s) set! Could be the program crashed before usage or a spelling mistake, etc! >>>>>>>>>>> [0]PETSC ERROR: Option left: name:-ksp_max_it value: 200 source: file >>>>>>>>>>> [0]PETSC ERROR: Option left: name:-ksp_reuse_preconditioner (no value) source: file >>>>>>>>>>> [0]PETSC ERROR: Option left: name:-ksp_rtol value: 1.e-9 source: file >>>>>>>>>>> [0]PETSC ERROR: Option left: name:-ksp_type value: gmres source: file >>>>>>>>>>> [0]PETSC ERROR: Option left: name:-mpi_linear_solve_minimum_count_per_rank value: 5000 source: file >>>>>>>>>>> [0]PETSC ERROR: Option left: name:-mpi_linear_solver_server_view (no value) source: file >>>>>>>>>>> [0]PETSC ERROR: Option left: name:-pc_type value: gamg source: file >>>>>>>>>>> [0]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!e3VQ4NHKmXGstRsQW5vtI7fmKfUT9zmJkMJcPbcvPyIjicyfJpNoMgx3wZ-qyGcKNSjIkNZkzilec8MvNjNo7A$ for trouble shooting. >>>>>>>>>>> [0]PETSC ERROR: Petsc Release Version 3.22.0, Sep 28, 2024 [0]PETSC ERROR: /home/praveen/bouss/radial_flat/xgeoclaw with 6 MPI process(es) and PETSC_ARCH on euler by praveen Thu Oct 17 21:49:54 2024 >>>>>>>>>>> [0]PETSC ERROR: Configure options: AR=${PREFIX}/bin/x86_64-conda-linux-gnu-ar CC=mpicc CXX=mpicxx FC=mpifort CFLAGS="-march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /opt/miniforge/envs/claw/include " CPPFLAGS="-DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /opt/miniforge/envs/claw/include" CXXFLAGS="-fvisibility-inlines-hidden -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /opt/miniforge/envs/claw/include " FFLAGS="-march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /opt/miniforge/envs/claw/include -Wl,--no-as-needed" LDFLAGS="-pthread -fopenmp -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,--allow-shlib-undefined -Wl,-rpath,/opt/miniforge/envs/claw/lib -Wl,-rpath-link,/opt/miniforge/envs/claw/lib -L/opt/miniforge/envs/claw/lib -Wl,-rpath-link,/opt/miniforge/envs/claw/lib" LIBS="-Wl,-rpath,/opt/miniforge/envs/claw/lib -lmpi_mpifh -lgfortran" --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 --FOPTFLAGS=-O3 --with-clib-autodetect=0 --with-cxxlib-autodetect=0 --with-fortranlib-autodetect=0 --with-debugging=0 --with-blas-lib=libblas.so --with-lapack-lib=liblapack.so --with-yaml=1 --with-hdf5=1 --with-fftw=1 --with-hwloc=0 --with-hypre=1 --with-metis=1 --with-mpi=1 --with-mumps=1 --with-parmetis=1 --with-pthread=1 --with-ptscotch=1 --with-shared-libraries --with-ssl=0 --with-scalapack=1 --with-superlu=1 --with-superlu_dist=1 --with-superlu_dist-include=/opt/miniforge/envs/claw/include/superlu-dist --with-superlu_dist-lib=-lsuperlu_dist --with-suitesparse=1 --with-suitesparse-dir=/opt/miniforge/envs/claw --with-x=0 --with-scalar-type=real --with-cuda=0 --prefix=/opt/miniforge/envs/claw >>>>>>>>>>> [0]PETSC ERROR: #1 PetscShmgetMapAddresses() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/sys/utils/server.c:114 >>>>>>>>>>> [0]PETSC ERROR: #2 PCMPISetMat() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/pc/impls/mpi/pcmpi.c:269 >>>>>>>>>>> [0]PETSC ERROR: #3 PCSetUp_MPI() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/pc/impls/mpi/pcmpi.c:853 >>>>>>>>>>> [0]PETSC ERROR: #4 PCSetUp() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/pc/interface/precon.c:1071 >>>>>>>>>>> [0]PETSC ERROR: #5 KSPSetUp() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/ksp/interface/itfunc.c:415 >>>>>>>>>>> [0]PETSC ERROR: #6 KSPSolve_Private() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/ksp/interface/itfunc.c:826 >>>>>>>>>>> [0]PETSC ERROR: #7 KSPSolve() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/ksp/interface/itfunc.c:1075 >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Tue Oct 29 21:00:41 2024 From: mfadams at lbl.gov (Mark Adams) Date: Tue, 29 Oct 2024 22:00:41 -0400 Subject: [petsc-users] Rigid body nullspace for Stokes operator In-Reply-To: References: Message-ID: This is linear elasticity and there are 6 "null" vectors (if you removed Dirichlet boundary conditions these are eigenvectors with zero eigenvalue): 3 translations, x, y, z, and three rotatiions xx, yy ,zz. x = (1,0,0,1,0,0,1,0 ...) and xx is something like (0, z_1, -y_1, 0, z_2, -y_2, ...) where z_1 is the z coordinate of the first vertex, etc. Mark On Tue, Oct 29, 2024 at 3:47?PM Amneet Bhalla wrote: > Hi Mark, > > Thanks! I am not sure how to construct null space and zero energy modes > manually for this operator. Is there some theory or documentation I can > follow to figure out what the null space and zero energy modes look like > for this operator? Once I know what these are in symbolic form, I think I > should be able to construct them manually. > > Best, > --Amneet > > On Tue, Oct 29, 2024 at 7:35?AM Mark Adams wrote: > >> Oh my mistake. You are using staggered grids. So you don't have a block >> size that hypre would use for the "nodal" methods. >> I'm not sure what you are doing exactly, but try hypre and you could >> create the null space, zero energy modes, manually, attach to the matrix >> and try GAMG. >> You can run with '-info :pc' and grep on GAMG to see if GAMG is picking >> the null space up (send this output if you can't figure it out). >> >> Thanks, >> Mark >> >> On Tue, Oct 29, 2024 at 9:28?AM Mark Adams wrote: >> >>> This coordinate interface is just a shortcut for vertex based >>> discretizations with 3 dof per vertex, etc. (maybe works in 2D). >>> You will need to construct the null space vectors manually and attach it >>> to the matrix. Used by GAMG. >>> >>> Note, for hypre you want to use the "nodal" options and it does not use >>> these null space vectors. That is probably the way you want to go. >>> eg: -pc_hypre_boomeramg_nodal_coarsen >>> >>> I would run with hypre boomerang and -help and grep on nodal to see all >>> the "nodal" options and use them. >>> >>> Thanks, >>> Mark >>> >>> >>> On Mon, Oct 28, 2024 at 8:06?PM Amneet Bhalla >>> wrote: >>> >>>> Hi Folks, >>>> >>>> I am trying to solve the momentum equation in a projection >>>> preconditioner using GAMG or Hypre solver. The equation looks like for >>>> velocity variable *v* looks like: >>>> >>>> >>>> [image: Screenshot 2024-10-28 at 4.15.17 PM.png] >>>> >>>> Here, ? is spatially varying dynamic viscosity and ? is spatially >>>> varying bulk viscosity. I understand that I need to specify rigid body >>>> nullspace modes to the multigrid solver in order to accelerate its >>>> convergence. Looking into this routine MatNullSpaceCreateRigidBody() ( >>>> https://urldefense.us/v3/__https://petsc.org/release/manualpages/Mat/MatNullSpaceCreateRigidBody/__;!!G_uCfscf7eWS!bVR6duCoDqPhZrWS-sm1c5qxsFPjZMhdT86AqLpPzWgVy5qoRhd4_Jue2LJOIS6LRrtV2cHGrqger1Yvb-Y5f-0$ >>>> ), >>>> I see that I need to provide the coordinates of each node. I am using >>>> staggered grid discretization. Do I need to provide coordinates of >>>> staggered grid locations? >>>> >>>> Thanks, >>>> -- >>>> --Amneet >>>> >>>> >>>> >>>> > > -- > --Amneet > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Screenshot 2024-10-28 at 4.15.17 PM.png Type: image/png Size: 17171 bytes Desc: not available URL: From jed at jedbrown.org Tue Oct 29 22:03:30 2024 From: jed at jedbrown.org (Jed Brown) Date: Tue, 29 Oct 2024 21:03:30 -0600 Subject: [petsc-users] Rigid body nullspace for Stokes operator In-Reply-To: References: Message-ID: <875xpas35p.fsf@jedbrown.org> And to be clear, we recommend using fieldsplit Schur to separate the pressure and velocity part (there are many examples). Applying AMG directly to the saddle point problem will not be a good solver because the heuristics assume positivity and do not preserve inf-sup stability (nor do standard smoothers). Mark Adams writes: > This is linear elasticity and there are 6 "null" vectors (if you removed > Dirichlet boundary conditions these are eigenvectors with zero eigenvalue): > 3 translations, x, y, z, and three rotatiions xx, yy ,zz. > x = (1,0,0,1,0,0,1,0 ...) > and xx is something like (0, z_1, -y_1, 0, z_2, -y_2, ...) where z_1 is the > z coordinate of the first vertex, etc. > > Mark > > On Tue, Oct 29, 2024 at 3:47?PM Amneet Bhalla wrote: > >> Hi Mark, >> >> Thanks! I am not sure how to construct null space and zero energy modes >> manually for this operator. Is there some theory or documentation I can >> follow to figure out what the null space and zero energy modes look like >> for this operator? Once I know what these are in symbolic form, I think I >> should be able to construct them manually. >> >> Best, >> --Amneet >> >> On Tue, Oct 29, 2024 at 7:35?AM Mark Adams wrote: >> >>> Oh my mistake. You are using staggered grids. So you don't have a block >>> size that hypre would use for the "nodal" methods. >>> I'm not sure what you are doing exactly, but try hypre and you could >>> create the null space, zero energy modes, manually, attach to the matrix >>> and try GAMG. >>> You can run with '-info :pc' and grep on GAMG to see if GAMG is picking >>> the null space up (send this output if you can't figure it out). >>> >>> Thanks, >>> Mark >>> >>> On Tue, Oct 29, 2024 at 9:28?AM Mark Adams wrote: >>> >>>> This coordinate interface is just a shortcut for vertex based >>>> discretizations with 3 dof per vertex, etc. (maybe works in 2D). >>>> You will need to construct the null space vectors manually and attach it >>>> to the matrix. Used by GAMG. >>>> >>>> Note, for hypre you want to use the "nodal" options and it does not use >>>> these null space vectors. That is probably the way you want to go. >>>> eg: -pc_hypre_boomeramg_nodal_coarsen >>>> >>>> I would run with hypre boomerang and -help and grep on nodal to see all >>>> the "nodal" options and use them. >>>> >>>> Thanks, >>>> Mark >>>> >>>> >>>> On Mon, Oct 28, 2024 at 8:06?PM Amneet Bhalla >>>> wrote: >>>> >>>>> Hi Folks, >>>>> >>>>> I am trying to solve the momentum equation in a projection >>>>> preconditioner using GAMG or Hypre solver. The equation looks like for >>>>> velocity variable *v* looks like: >>>>> >>>>> >>>>> [image: Screenshot 2024-10-28 at 4.15.17 PM.png] >>>>> >>>>> Here, ? is spatially varying dynamic viscosity and ? is spatially >>>>> varying bulk viscosity. I understand that I need to specify rigid body >>>>> nullspace modes to the multigrid solver in order to accelerate its >>>>> convergence. Looking into this routine MatNullSpaceCreateRigidBody() ( >>>>> https://urldefense.us/v3/__https://petsc.org/release/manualpages/Mat/MatNullSpaceCreateRigidBody/__;!!G_uCfscf7eWS!bVR6duCoDqPhZrWS-sm1c5qxsFPjZMhdT86AqLpPzWgVy5qoRhd4_Jue2LJOIS6LRrtV2cHGrqger1Yvb-Y5f-0$ >>>>> ), >>>>> I see that I need to provide the coordinates of each node. I am using >>>>> staggered grid discretization. Do I need to provide coordinates of >>>>> staggered grid locations? >>>>> >>>>> Thanks, >>>>> -- >>>>> --Amneet >>>>> >>>>> >>>>> >>>>> >> >> -- >> --Amneet >> >> >> >> From bsmith at petsc.dev Wed Oct 30 07:58:51 2024 From: bsmith at petsc.dev (Barry Smith) Date: Wed, 30 Oct 2024 08:58:51 -0400 Subject: [petsc-users] Problem running geoclaw example with petsc 3.22 In-Reply-To: References: <1C63D19F-6EBC-43A3-8B40-65EA34991AFA@gmail.com> <0136027D-C2CE-47D0-AF97-ACDE1DA0C382@gmail.com> <4726A341-54DB-4592-BBAE-843AF271F28E@petsc.dev> <27B47E2D-00D5-4326-95A9-7595DFF0E952@gmail.com> <490B5A4A-BBD8-490A-91FD-7CEC9E01A99B@petsc.dev> Message-ID: <96674E24-9EDA-42DE-8459-81D46A260CDE@petsc.dev> Please send me exact instructions on how you are running the geoclaw example, when I run geoclaw/examples/bouss/radial_flat it only runs with one level, but never errors, I changed setrun.py to amrdata.amr_levels_max = 5 but it made no difference. Thanks Barry > On Oct 24, 2024, at 12:17?PM, Praveen C wrote: > > Hello Barry > > I use this script to install clawpack and required dependencies > > https://urldefense.us/v3/__https://github.com/cpraveen/cfdlab/blob/master/bin/clawpack.sh__;!!G_uCfscf7eWS!YrqSkST0JSk-fVtltvkqECwG-Dvudk3Ba8qjPKEcTmL340jOuxTjqeQk2w6UmNMF9yUGd7H-E4YG1DtZVuDoN_E$ > > See lines 125-132 > > You can use it like this > > export CLAW=/path/to/where/you/want/clawpack > bash clawpack.sh v5.11.0 > > This will git pull clawpack and creates a conda env called ?claw? and installs inside that. > > I have not specified petsc version in this script, but latest miniforge should install petsc at 3.22 > > Thank you > praveen > >> On 24 Oct 2024, at 9:37?PM, Barry Smith wrote: >> >> >> Good, some helpful information with the runs you made. >> >> In the crash below it made more progress it was able to allocate multiple regions of shared memory and access them. I don't know why it would crash later. >> >> Can you tell me all the steps with miniforge (which seems to be related to the failure) you use? I've never used miniforge. >> >> If I can get to an environment that reproduces the problem I can debug it and fix it. >> >> Barry >> >> >>> On Oct 24, 2024, at 11:49?AM, Praveen C wrote: >>> >>> I get this >>> >>> $ mpiexec -n 3 ./ex89f -n 20 -mpi_linear_solver_server -mpi_linear_solver_server -mpi_linear_solver_server_ksp_view -ksp_monitor -ksp_converged_reason -ksp_view -mpi_linear_solver_server_minimum_count_per_rank 5 >>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range >>> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >>> [0]PETSC ERROR: or see https://urldefense.us/v3/__https://petsc.org/release/faq/*valgrind__;Iw!!G_uCfscf7eWS!YrqSkST0JSk-fVtltvkqECwG-Dvudk3Ba8qjPKEcTmL340jOuxTjqeQk2w6UmNMF9yUGd7H-E4YG1DtZ_1I3Lsc$ and https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!YrqSkST0JSk-fVtltvkqECwG-Dvudk3Ba8qjPKEcTmL340jOuxTjqeQk2w6UmNMF9yUGd7H-E4YG1DtZNJ6EACo$ >>> [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run >>> [0]PETSC ERROR: to get more information on the crash. >>> [0]PETSC ERROR: Run with -malloc_debug to check if memory corruption is causing the crash. >>> -------------------------------------------------------------------------- >>> MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_SELF >>> Proc: [[47380,1],0] >>> Errorcode: 59 >>> >>> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. >>> You may or may not see output from other processes, depending on >>> exactly when Open MPI kills them. >>> -------------------------------------------------------------------------- >>> -------------------------------------------------------------------------- >>> prterun has exited due to process rank 0 with PID 0 on node MacMiniHome calling >>> "abort". This may have caused other processes in the application to be >>> terminated by signals sent by prterun (as reported here). >>> ????????????????????????????????????? >>> >>> and I have this after the code exits >>> >>> $ ipcs -m >>> IPC status from as of Thu Oct 24 21:17:39 IST 2024 >>> T ID KEY MODE OWNER GROUP >>> Shared Memory: >>> m 1572864 0x0000000b --rw-rw-rw- praveen staff >>> m 524289 0x0000000c --rw-rw-rw- praveen staff >>> m 655362 0x0000000d --rw-rw-rw- praveen staff >>> m 262147 0x0000000e --rw-rw-rw- praveen staff >>> m 262148 0x0000000f --rw-rw-rw- praveen staff >>> m 393221 0x0000000a --rw-rw-rw- praveen staff >>> >>> This is with petsc installed with miniforge, which I also use with clawpack. With spack installed petsc, I can run the ex89f example. >>> >>> Thanks >>> praveen >>> >>>> On 24 Oct 2024, at 7:55?PM, Barry Smith wrote: >>>> >>>> >>>> Ok, super strange, I've run many times on my Mac >>>> >>>> Can you please try to remove that allocated memory with ipcrm and then >>>> >>>> cd $PETSC_DIR/src/ksp/ksp/tutorials >>>> make ex89f >>>> mpiexec -n 3 ./ex89f -n 20 -mpi_linear_solver_server -mpi_linear_solver_server -mpi_linear_solver_server_ksp_view -ksp_monitor -ksp_converged_reason -ksp_view -mpi_linear_solver_server_minimum_count_per_rank 5 >>>> >>>> This does the same thing as the GeoClaw code but is much simpler. >>>> >>>> Barry >>>> >>>> >>>> >>>> >>>>> On Oct 23, 2024, at 10:55?PM, Praveen C wrote: >>>>> >>>>> I get very similar error on my mac with >>>>> >>>>> $ gfortran -v >>>>> Using built-in specs. >>>>> COLLECT_GCC=gfortran >>>>> COLLECT_LTO_WRAPPER=/opt/homebrew/Caskroom/miniforge/base/envs/claw/libexec/gcc/arm64-apple-darwin20.0.0/13.2.0/lto-wrapper >>>>> Target: arm64-apple-darwin20.0.0 >>>>> Configured with: ../configure --prefix=/opt/homebrew/Caskroom/miniforge/base/envs/claw --build=x86_64-apple-darwin13.4.0 --host=arm64-apple-darwin20.0.0 --target=arm64-apple-darwin20.0.0 --with-libiconv-prefix=/opt/homebrew/Caskroom/miniforge/base/envs/claw --enable-languages=fortran --disable-multilib --enable-checking=release --disable-bootstrap --disable-libssp --with-gmp=/opt/homebrew/Caskroom/miniforge/base/envs/claw --with-mpfr=/opt/homebrew/Caskroom/miniforge/base/envs/claw --with-mpc=/opt/homebrew/Caskroom/miniforge/base/envs/claw --with-isl=/opt/homebrew/Caskroom/miniforge/base/envs/claw --enable-darwin-at-rpath >>>>> Thread model: posix >>>>> Supported LTO compression algorithms: zlib >>>>> gcc version 13.2.0 (GCC) >>>>> >>>>> Before starting >>>>> >>>>> $ ipcs -m >>>>> IPC status from as of Thu Oct 24 08:02:11 IST 2024 >>>>> T ID KEY MODE OWNER GROUP >>>>> Shared Memory: >>>>> >>>>> and when I run the code >>>>> >>>>> Using a PETSc solver >>>>> Using Bouss equations from the start >>>>> rnode allocated... >>>>> node allocated... >>>>> listOfGrids allocated... >>>>> Storage allocated... >>>>> bndList allocated... >>>>> Gridding level 1 at t = 0.000000E+00: 4 grids with 10000 cells >>>>> Setting initial dt to 2.9999999999999999E-002 >>>>> max threads set to 1 >>>>> >>>>> Done reading data, starting computation ... >>>>> >>>>> Total zeta at initial time: 39269.907650665169 >>>>> GEOCLAW: Frame 0 output files done at time t = 0.000000D+00 >>>>> >>>>> [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >>>>> [0]PETSC ERROR: Petsc has generated inconsistent data >>>>> [0]PETSC ERROR: Unable to locate PCMPI allocated shared address 0x130698000 >>>>> [0]PETSC ERROR: WARNING! There are unused option(s) set! Could be the program crashed before usage or a spelling mistake, etc! >>>>> [0]PETSC ERROR: Option left: name:-ksp_type value: preonly source: file >>>>> [0]PETSC ERROR: Option left: name:-mpi_ksp_max_it value: 200 source: file >>>>> [0]PETSC ERROR: Option left: name:-mpi_ksp_reuse_preconditioner (no value) source: file >>>>> [0]PETSC ERROR: Option left: name:-mpi_ksp_rtol value: 1.e-9 source: file >>>>> [0]PETSC ERROR: Option left: name:-mpi_ksp_type value: gmres source: file >>>>> [0]PETSC ERROR: Option left: name:-mpi_linear_solver_server_view (no value) source: file >>>>> [0]PETSC ERROR: Option left: name:-mpi_pc_gamg_sym_graph value: true source: file >>>>> [0]PETSC ERROR: Option left: name:-mpi_pc_gamg_symmetrize_graph value: true source: file >>>>> [0]PETSC ERROR: Option left: name:-mpi_pc_type value: gamg source: file >>>>> [0]PETSC ERROR: Option left: name:-pc_mpi_minimum_count_per_rank value: 5000 source: file >>>>> [0]PETSC ERROR: Option left: name:-pc_type value: mpi source: file >>>>> [0]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!YrqSkST0JSk-fVtltvkqECwG-Dvudk3Ba8qjPKEcTmL340jOuxTjqeQk2w6UmNMF9yUGd7H-E4YG1DtZNJ6EACo$ for trouble shooting. >>>>> [0]PETSC ERROR: Petsc Release Version 3.22.0, Sep 28, 2024 >>>>> [0]PETSC ERROR: /Users/praveen/work/bouss/radial_flat/xgeoclaw with 6 MPI process(es) and PETSC_ARCH on MacMiniHome.local by praveen Thu Oct 24 08:04:27 2024 >>>>> [0]PETSC ERROR: Configure options: AR=arm64-apple-darwin20.0.0-ar CC=mpicc CXX=mpicxx FC=mpifort CFLAGS="-ftree-vectorize -fPIC -fstack-protector-strong -O2 -pipe -isystem /opt/homebrew/Caskroom/miniforge/base/envs/claw/include " CPPFLAGS="-D_FORTIFY_SOURCE=2 -isystem /opt/homebrew/Caskroom/miniforge/base/envs/claw/include -mmacosx-version-min=11.0 -mmacosx-version-min=11.0" CXXFLAGS="-ftree-vectorize -fPIC -fstack-protector-strong -O2 -pipe -stdlib=libc++ -fvisibility-inlines-hidden -fmessage-length=0 -isystem /opt/homebrew/Caskroom/miniforge/base/envs/claw/include " FFLAGS="-march=armv8.3-a -ftree-vectorize -fPIC -fno-stack-protector -O2 -pipe -isystem /opt/homebrew/Caskroom/miniforge/base/envs/claw/include " LDFLAGS="-Wl,-headerpad_max_install_names -Wl,-dead_strip_dylibs -Wl,-rpath,/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib -L/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib" LIBS="-Wl,-rpath,/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib -lmpi_mpifh -lgfortran" --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 --FOPTFLAGS=-O3 --with-clib-autodetect=0 --with-cxxlib-autodetect=0 --with-fortranlib-autodetect=0 --with-debugging=0 --with-blas-lib=libblas.dylib --with-lapack-lib=liblapack.dylib --with-yaml=1 --with-hdf5=1 --with-fftw=1 --with-hwloc=0 --with-hypre=1 --with-metis=1 --with-mpi=1 --with-mumps=1 --with-parmetis=1 --with-pthread=1 --with-ptscotch=1 --with-shared-libraries --with-ssl=0 --with-scalapack=1 --with-superlu=1 --with-superlu_dist=1 --with-superlu_dist-include=/opt/homebrew/Caskroom/miniforge/base/envs/claw/include/superlu-dist --with-superlu_dist-lib=-lsuperlu_dist --with-suitesparse=1 --with-suitesparse-dir=/opt/homebrew/Caskroom/miniforge/base/envs/claw --with-x=0 --with-scalar-type=real --with-cuda=0 --with-batch --prefix=/opt/homebrew/Caskroom/miniforge/base/envs/claw >>>>> [0]PETSC ERROR: #1 PetscShmgetMapAddresses() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/sys/utils/server.c:114 >>>>> [0]PETSC ERROR: #2 PCMPISetMat() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/ksp/pc/impls/mpi/pcmpi.c:269 >>>>> [0]PETSC ERROR: #3 PCSetUp_MPI() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/ksp/pc/impls/mpi/pcmpi.c:853 >>>>> [0]PETSC ERROR: #4 PCSetUp() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/ksp/pc/interface/precon.c:1071 >>>>> [0]PETSC ERROR: #5 KSPSetUp() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/ksp/ksp/interface/itfunc.c:415 >>>>> [0]PETSC ERROR: #6 KSPSolve_Private() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/ksp/ksp/interface/itfunc.c:826 >>>>> [0]PETSC ERROR: #7 KSPSolve() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/ksp/ksp/interface/itfunc.c:1075 >>>>> >>>>> Code does not progress and I kill it >>>>> >>>>> ^CTraceback (most recent call last): >>>>> File "/Users/praveen/Applications/clawpack/clawutil/src/python/clawutil/runclaw.py", line 341, in >>>>> runclaw(*args) >>>>> File "/Users/praveen/Applications/clawpack/clawutil/src/python/clawutil/runclaw.py", line 242, in runclaw >>>>> proc = subprocess.check_call(cmd_split, >>>>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ >>>>> File "/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib/python3.12/subprocess.py", line 408, in check_call >>>>> retcode = call(*popenargs, **kwargs) >>>>> ^^^^^^^^^^^^^^^^^^^^^^^^^^ >>>>> File "/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib/python3.12/subprocess.py", line 391, in call >>>>> return p.wait(timeout=timeout) >>>>> ^^^^^^^^^^^^^^^^^^^^^^^ >>>>> File "/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib/python3.12/subprocess.py", line 1264, in wait >>>>> return self._wait(timeout=timeout) >>>>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^ >>>>> File "/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib/python3.12/subprocess.py", line 2053, in _wait >>>>> (pid, sts) = self._try_wait(0) >>>>> ^^^^^^^^^^^^^^^^^ >>>>> File "/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib/python3.12/subprocess.py", line 2011, in _try_wait >>>>> (pid, sts) = os.waitpid(self.pid, wait_flags) >>>>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ >>>>> KeyboardInterrupt >>>>> make[1]: *** [output] Interrupt: 2 >>>>> make: *** [.output] Interrupt: 2 >>>>> >>>>> Now it says >>>>> >>>>> $ ipcs -m >>>>> IPC status from as of Thu Oct 24 08:05:06 IST 2024 >>>>> T ID KEY MODE OWNER GROUP >>>>> Shared Memory: >>>>> m 720896 0x0000000a --rw-rw-rw- praveen staff >>>>> >>>>> Thanks >>>>> praveen >>>>> >>>>>> On 23 Oct 2024, at 8:26?PM, Barry Smith wrote: >>>>>> >>>>>> >>>>>> Hmm, so it is creating the first shared memory region with ID of 10 (A in hex) puts it in a linked list in PETSc but then when it tries to find it in the linked list it cannot find it. >>>>>> >>>>>> I don't know how to reproduce this or debug it remotely. >>>>>> >>>>>> Can you build on a completely different machine or with completely different compilers? >>>>>> >>>>>> Barry >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> On Oct 23, 2024, at 10:31?AM, Praveen C wrote: >>>>>>> >>>>>>> I get same error and now it shows >>>>>>> >>>>>>> $ ipcs -m >>>>>>> >>>>>>> ------ Shared Memory Segments -------- >>>>>>> key shmid owner perms bytes nattch status >>>>>>> 0x0000000a 32788 praveen 666 240 6 >>>>>>> >>>>>>> Note that the code seems to be still running after printing those error message, but it is not printing any progress which it should do. >>>>>>> >>>>>>> Thanks >>>>>>> praveen >>>>>>> >>>>>>>> On 23 Oct 2024, at 7:56?PM, Barry Smith wrote: >>>>>>>> >>>>>>>> >>>>>>>> Try >>>>>>>> >>>>>>>> ipcrm -m 11 >>>>>>>> >>>>>>>> ipcs -m >>>>>>>> >>>>>>>> Try running the program again >>>>>>>> >>>>>>>> If failed check >>>>>>>> >>>>>>>> ipcs -m >>>>>>>> >>>>>>>> again >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> On Oct 23, 2024, at 10:20?AM, Praveen C wrote: >>>>>>>>> >>>>>>>>> Hello Barry >>>>>>>>> >>>>>>>>> I see this >>>>>>>>> >>>>>>>>> $ ipcs -m >>>>>>>>> >>>>>>>>> ------ Shared Memory Segments -------- >>>>>>>>> key shmid owner perms bytes nattch status >>>>>>>>> 0x0000000a 11 praveen 666 240 6 >>>>>>>>> >>>>>>>>> and I am observing same error as below. >>>>>>>>> >>>>>>>>> Thanks >>>>>>>>> praveen >>>>>>>>> >>>>>>>>>> On 23 Oct 2024, at 7:08?PM, Barry Smith wrote: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Please take a look at the notes in https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/PetscShmgetAllocateArray/__;!!G_uCfscf7eWS!YrqSkST0JSk-fVtltvkqECwG-Dvudk3Ba8qjPKEcTmL340jOuxTjqeQk2w6UmNMF9yUGd7H-E4YG1DtZr_sBLR0$ For some reason your program is not able to access/use the Unix shared memory; check if you are already using the shared memory (so it is not available for a new run) or the limits are too low to access enough memory. >>>>>>>>>> >>>>>>>>>> Barry >>>>>>>>>> >>>>>>>>>>> On Oct 23, 2024, at 8:23?AM, Praveen C wrote: >>>>>>>>>>> >>>>>>>>>>> Dear all >>>>>>>>>>> >>>>>>>>>>> I am not able to run the boussinesq example from geoclaw using petsc at 3.22.0 >>>>>>>>>>> >>>>>>>>>>> https://urldefense.us/v3/__https://github.com/clawpack/geoclaw/tree/3303883f46572c58130d161986b8a87a57ca7816/examples/bouss__;!!G_uCfscf7eWS!e3VQ4NHKmXGstRsQW5vtI7fmKfUT9zmJkMJcPbcvPyIjicyfJpNoMgx3wZ-qyGcKNSjIkNZkzilec8MnHN6PMw$ >>>>>>>>>>> >>>>>>>>>>> It runs with petsc at 3.21.6 >>>>>>>>>>> >>>>>>>>>>> The error I get is given below. After printing this, the code does not progress. >>>>>>>>>>> >>>>>>>>>>> I use the following petsc options >>>>>>>>>>> >>>>>>>>>>> # set min numbers of matrix rows per MPI rank (default is 10000) >>>>>>>>>>> -mpi_linear_solve_minimum_count_per_rank 5000 >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> # Krylov linear solver: >>>>>>>>>>> -mpi_linear_solver_server >>>>>>>>>>> -mpi_linear_solver_server_view >>>>>>>>>>> -ksp_type gmres >>>>>>>>>>> -ksp_max_it 200 >>>>>>>>>>> -ksp_reuse_preconditioner >>>>>>>>>>> -ksp_rtol 1.e-9 >>>>>>>>>>> >>>>>>>>>>> # preconditioner: >>>>>>>>>>> -pc_type gamg >>>>>>>>>>> >>>>>>>>>>> I installed petsc and other dependencies for clawpack using miniforge. >>>>>>>>>>> >>>>>>>>>>> Thanks >>>>>>>>>>> pc >>>>>>>>>>> >>>>>>>>>>> ==> Use Bouss. in water deeper than 1.0000000000000000 Using a PETSc solver >>>>>>>>>>> Using Bouss equations from the start >>>>>>>>>>> rnode allocated... >>>>>>>>>>> node allocated... >>>>>>>>>>> listOfGrids allocated... >>>>>>>>>>> Storage allocated... >>>>>>>>>>> bndList allocated... >>>>>>>>>>> Gridding level 1 at t = 0.000000E+00: 4 grids with 10000 cells >>>>>>>>>>> Setting initial dt to 2.9999999999999999E-002 >>>>>>>>>>> max threads set to 6 >>>>>>>>>>> Done reading data, starting computation ... Total zeta at initial time: 39269.907650665169 GEOCLAW: Frame 0 output files done at time t = 0.000000D+00 >>>>>>>>>>> >>>>>>>>>>> [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >>>>>>>>>>> [0]PETSC ERROR: Petsc has generated inconsistent data >>>>>>>>>>> [0]PETSC ERROR: Unable to locate PCMPI allocated shared address 0x55e6d750ae20 >>>>>>>>>>> [0]PETSC ERROR: WARNING! There are unused option(s) set! Could be the program crashed before usage or a spelling mistake, etc! >>>>>>>>>>> [0]PETSC ERROR: Option left: name:-ksp_max_it value: 200 source: file >>>>>>>>>>> [0]PETSC ERROR: Option left: name:-ksp_reuse_preconditioner (no value) source: file >>>>>>>>>>> [0]PETSC ERROR: Option left: name:-ksp_rtol value: 1.e-9 source: file >>>>>>>>>>> [0]PETSC ERROR: Option left: name:-ksp_type value: gmres source: file >>>>>>>>>>> [0]PETSC ERROR: Option left: name:-mpi_linear_solve_minimum_count_per_rank value: 5000 source: file >>>>>>>>>>> [0]PETSC ERROR: Option left: name:-mpi_linear_solver_server_view (no value) source: file >>>>>>>>>>> [0]PETSC ERROR: Option left: name:-pc_type value: gamg source: file >>>>>>>>>>> [0]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!e3VQ4NHKmXGstRsQW5vtI7fmKfUT9zmJkMJcPbcvPyIjicyfJpNoMgx3wZ-qyGcKNSjIkNZkzilec8MvNjNo7A$ for trouble shooting. >>>>>>>>>>> [0]PETSC ERROR: Petsc Release Version 3.22.0, Sep 28, 2024 [0]PETSC ERROR: /home/praveen/bouss/radial_flat/xgeoclaw with 6 MPI process(es) and PETSC_ARCH on euler by praveen Thu Oct 17 21:49:54 2024 >>>>>>>>>>> [0]PETSC ERROR: Configure options: AR=${PREFIX}/bin/x86_64-conda-linux-gnu-ar CC=mpicc CXX=mpicxx FC=mpifort CFLAGS="-march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /opt/miniforge/envs/claw/include " CPPFLAGS="-DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /opt/miniforge/envs/claw/include" CXXFLAGS="-fvisibility-inlines-hidden -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /opt/miniforge/envs/claw/include " FFLAGS="-march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /opt/miniforge/envs/claw/include -Wl,--no-as-needed" LDFLAGS="-pthread -fopenmp -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,--allow-shlib-undefined -Wl,-rpath,/opt/miniforge/envs/claw/lib -Wl,-rpath-link,/opt/miniforge/envs/claw/lib -L/opt/miniforge/envs/claw/lib -Wl,-rpath-link,/opt/miniforge/envs/claw/lib" LIBS="-Wl,-rpath,/opt/miniforge/envs/claw/lib -lmpi_mpifh -lgfortran" --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 --FOPTFLAGS=-O3 --with-clib-autodetect=0 --with-cxxlib-autodetect=0 --with-fortranlib-autodetect=0 --with-debugging=0 --with-blas-lib=libblas.so --with-lapack-lib=liblapack.so --with-yaml=1 --with-hdf5=1 --with-fftw=1 --with-hwloc=0 --with-hypre=1 --with-metis=1 --with-mpi=1 --with-mumps=1 --with-parmetis=1 --with-pthread=1 --with-ptscotch=1 --with-shared-libraries --with-ssl=0 --with-scalapack=1 --with-superlu=1 --with-superlu_dist=1 --with-superlu_dist-include=/opt/miniforge/envs/claw/include/superlu-dist --with-superlu_dist-lib=-lsuperlu_dist --with-suitesparse=1 --with-suitesparse-dir=/opt/miniforge/envs/claw --with-x=0 --with-scalar-type=real --with-cuda=0 --prefix=/opt/miniforge/envs/claw >>>>>>>>>>> [0]PETSC ERROR: #1 PetscShmgetMapAddresses() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/sys/utils/server.c:114 >>>>>>>>>>> [0]PETSC ERROR: #2 PCMPISetMat() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/pc/impls/mpi/pcmpi.c:269 >>>>>>>>>>> [0]PETSC ERROR: #3 PCSetUp_MPI() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/pc/impls/mpi/pcmpi.c:853 >>>>>>>>>>> [0]PETSC ERROR: #4 PCSetUp() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/pc/interface/precon.c:1071 >>>>>>>>>>> [0]PETSC ERROR: #5 KSPSetUp() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/ksp/interface/itfunc.c:415 >>>>>>>>>>> [0]PETSC ERROR: #6 KSPSolve_Private() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/ksp/interface/itfunc.c:826 >>>>>>>>>>> [0]PETSC ERROR: #7 KSPSolve() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/ksp/interface/itfunc.c:1075 >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cpraveen at gmail.com Wed Oct 30 08:05:31 2024 From: cpraveen at gmail.com (Praveen C) Date: Wed, 30 Oct 2024 18:35:31 +0530 Subject: [petsc-users] Problem running geoclaw example with petsc 3.22 In-Reply-To: <96674E24-9EDA-42DE-8459-81D46A260CDE@petsc.dev> References: <1C63D19F-6EBC-43A3-8B40-65EA34991AFA@gmail.com> <0136027D-C2CE-47D0-AF97-ACDE1DA0C382@gmail.com> <4726A341-54DB-4592-BBAE-843AF271F28E@petsc.dev> <27B47E2D-00D5-4326-95A9-7595DFF0E952@gmail.com> <490B5A4A-BBD8-490A-91FD-7CEC9E01A99B@petsc.dev> <96674E24-9EDA-42DE-8459-81D46A260CDE@petsc.dev> Message-ID: I have attached some files for this cd bouss . setenv.sh # some settings in this file may need to be changed cd radial_flat/1d_radial make .output cd .. make .output Thanks praveen ?? > On 30 Oct 2024, at 6:28?PM, Barry Smith wrote: > > > Please send me exact instructions on how you are running the geoclaw example, when I run > > geoclaw/examples/bouss/radial_flat > > > it only runs with one level, but never errors, I changed setrun.py to amrdata.amr_levels_max = 5 but it made no difference. > > Thanks > > Barry > > > >> On Oct 24, 2024, at 12:17?PM, Praveen C wrote: >> >> Hello Barry >> >> I use this script to install clawpack and required dependencies >> >> https://urldefense.us/v3/__https://github.com/cpraveen/cfdlab/blob/master/bin/clawpack.sh__;!!G_uCfscf7eWS!fM-tM-j46TWI2-dEM5dWqIZEgpVdFBhYKU4M3RXKtOogG10VbWUtHGAQv6q7qO0TV_YQ1yP5pgPAY1_xD38Dbw$ >> >> See lines 125-132 >> >> You can use it like this >> >> export CLAW=/path/to/where/you/want/clawpack >> bash clawpack.sh v5.11.0 >> >> This will git pull clawpack and creates a conda env called ?claw? and installs inside that. >> >> I have not specified petsc version in this script, but latest miniforge should install petsc at 3.22 >> >> Thank you >> praveen >> >>> On 24 Oct 2024, at 9:37?PM, Barry Smith wrote: >>> >>> >>> Good, some helpful information with the runs you made. >>> >>> In the crash below it made more progress it was able to allocate multiple regions of shared memory and access them. I don't know why it would crash later. >>> >>> Can you tell me all the steps with miniforge (which seems to be related to the failure) you use? I've never used miniforge. >>> >>> If I can get to an environment that reproduces the problem I can debug it and fix it. >>> >>> Barry >>> >>> >>>> On Oct 24, 2024, at 11:49?AM, Praveen C wrote: >>>> >>>> I get this >>>> >>>> $ mpiexec -n 3 ./ex89f -n 20 -mpi_linear_solver_server -mpi_linear_solver_server -mpi_linear_solver_server_ksp_view -ksp_monitor -ksp_converged_reason -ksp_view -mpi_linear_solver_server_minimum_count_per_rank 5 >>>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range >>>> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >>>> [0]PETSC ERROR: or see https://urldefense.us/v3/__https://petsc.org/release/faq/*valgrind__;Iw!!G_uCfscf7eWS!fM-tM-j46TWI2-dEM5dWqIZEgpVdFBhYKU4M3RXKtOogG10VbWUtHGAQv6q7qO0TV_YQ1yP5pgPAY19gJufvcA$ and https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!fM-tM-j46TWI2-dEM5dWqIZEgpVdFBhYKU4M3RXKtOogG10VbWUtHGAQv6q7qO0TV_YQ1yP5pgPAY186WqRS9Q$ >>>> [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run >>>> [0]PETSC ERROR: to get more information on the crash. >>>> [0]PETSC ERROR: Run with -malloc_debug to check if memory corruption is causing the crash. >>>> -------------------------------------------------------------------------- >>>> MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_SELF >>>> Proc: [[47380,1],0] >>>> Errorcode: 59 >>>> >>>> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. >>>> You may or may not see output from other processes, depending on >>>> exactly when Open MPI kills them. >>>> -------------------------------------------------------------------------- >>>> -------------------------------------------------------------------------- >>>> prterun has exited due to process rank 0 with PID 0 on node MacMiniHome calling >>>> "abort". This may have caused other processes in the application to be >>>> terminated by signals sent by prterun (as reported here). >>>> ????????????????????????????????????? >>>> >>>> and I have this after the code exits >>>> >>>> $ ipcs -m >>>> IPC status from as of Thu Oct 24 21:17:39 IST 2024 >>>> T ID KEY MODE OWNER GROUP >>>> Shared Memory: >>>> m 1572864 0x0000000b --rw-rw-rw- praveen staff >>>> m 524289 0x0000000c --rw-rw-rw- praveen staff >>>> m 655362 0x0000000d --rw-rw-rw- praveen staff >>>> m 262147 0x0000000e --rw-rw-rw- praveen staff >>>> m 262148 0x0000000f --rw-rw-rw- praveen staff >>>> m 393221 0x0000000a --rw-rw-rw- praveen staff >>>> >>>> This is with petsc installed with miniforge, which I also use with clawpack. With spack installed petsc, I can run the ex89f example. >>>> >>>> Thanks >>>> praveen >>>> >>>>> On 24 Oct 2024, at 7:55?PM, Barry Smith wrote: >>>>> >>>>> >>>>> Ok, super strange, I've run many times on my Mac >>>>> >>>>> Can you please try to remove that allocated memory with ipcrm and then >>>>> >>>>> cd $PETSC_DIR/src/ksp/ksp/tutorials >>>>> make ex89f >>>>> mpiexec -n 3 ./ex89f -n 20 -mpi_linear_solver_server -mpi_linear_solver_server -mpi_linear_solver_server_ksp_view -ksp_monitor -ksp_converged_reason -ksp_view -mpi_linear_solver_server_minimum_count_per_rank 5 >>>>> >>>>> This does the same thing as the GeoClaw code but is much simpler. >>>>> >>>>> Barry >>>>> >>>>> >>>>> >>>>> >>>>>> On Oct 23, 2024, at 10:55?PM, Praveen C wrote: >>>>>> >>>>>> I get very similar error on my mac with >>>>>> >>>>>> $ gfortran -v >>>>>> Using built-in specs. >>>>>> COLLECT_GCC=gfortran >>>>>> COLLECT_LTO_WRAPPER=/opt/homebrew/Caskroom/miniforge/base/envs/claw/libexec/gcc/arm64-apple-darwin20.0.0/13.2.0/lto-wrapper >>>>>> Target: arm64-apple-darwin20.0.0 >>>>>> Configured with: ../configure --prefix=/opt/homebrew/Caskroom/miniforge/base/envs/claw --build=x86_64-apple-darwin13.4.0 --host=arm64-apple-darwin20.0.0 --target=arm64-apple-darwin20.0.0 --with-libiconv-prefix=/opt/homebrew/Caskroom/miniforge/base/envs/claw --enable-languages=fortran --disable-multilib --enable-checking=release --disable-bootstrap --disable-libssp --with-gmp=/opt/homebrew/Caskroom/miniforge/base/envs/claw --with-mpfr=/opt/homebrew/Caskroom/miniforge/base/envs/claw --with-mpc=/opt/homebrew/Caskroom/miniforge/base/envs/claw --with-isl=/opt/homebrew/Caskroom/miniforge/base/envs/claw --enable-darwin-at-rpath >>>>>> Thread model: posix >>>>>> Supported LTO compression algorithms: zlib >>>>>> gcc version 13.2.0 (GCC) >>>>>> >>>>>> Before starting >>>>>> >>>>>> $ ipcs -m >>>>>> IPC status from as of Thu Oct 24 08:02:11 IST 2024 >>>>>> T ID KEY MODE OWNER GROUP >>>>>> Shared Memory: >>>>>> >>>>>> and when I run the code >>>>>> >>>>>> Using a PETSc solver >>>>>> Using Bouss equations from the start >>>>>> rnode allocated... >>>>>> node allocated... >>>>>> listOfGrids allocated... >>>>>> Storage allocated... >>>>>> bndList allocated... >>>>>> Gridding level 1 at t = 0.000000E+00: 4 grids with 10000 cells >>>>>> Setting initial dt to 2.9999999999999999E-002 >>>>>> max threads set to 1 >>>>>> >>>>>> Done reading data, starting computation ... >>>>>> >>>>>> Total zeta at initial time: 39269.907650665169 >>>>>> GEOCLAW: Frame 0 output files done at time t = 0.000000D+00 >>>>>> >>>>>> [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >>>>>> [0]PETSC ERROR: Petsc has generated inconsistent data >>>>>> [0]PETSC ERROR: Unable to locate PCMPI allocated shared address 0x130698000 >>>>>> [0]PETSC ERROR: WARNING! There are unused option(s) set! Could be the program crashed before usage or a spelling mistake, etc! >>>>>> [0]PETSC ERROR: Option left: name:-ksp_type value: preonly source: file >>>>>> [0]PETSC ERROR: Option left: name:-mpi_ksp_max_it value: 200 source: file >>>>>> [0]PETSC ERROR: Option left: name:-mpi_ksp_reuse_preconditioner (no value) source: file >>>>>> [0]PETSC ERROR: Option left: name:-mpi_ksp_rtol value: 1.e-9 source: file >>>>>> [0]PETSC ERROR: Option left: name:-mpi_ksp_type value: gmres source: file >>>>>> [0]PETSC ERROR: Option left: name:-mpi_linear_solver_server_view (no value) source: file >>>>>> [0]PETSC ERROR: Option left: name:-mpi_pc_gamg_sym_graph value: true source: file >>>>>> [0]PETSC ERROR: Option left: name:-mpi_pc_gamg_symmetrize_graph value: true source: file >>>>>> [0]PETSC ERROR: Option left: name:-mpi_pc_type value: gamg source: file >>>>>> [0]PETSC ERROR: Option left: name:-pc_mpi_minimum_count_per_rank value: 5000 source: file >>>>>> [0]PETSC ERROR: Option left: name:-pc_type value: mpi source: file >>>>>> [0]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!fM-tM-j46TWI2-dEM5dWqIZEgpVdFBhYKU4M3RXKtOogG10VbWUtHGAQv6q7qO0TV_YQ1yP5pgPAY186WqRS9Q$ for trouble shooting. >>>>>> [0]PETSC ERROR: Petsc Release Version 3.22.0, Sep 28, 2024 >>>>>> [0]PETSC ERROR: /Users/praveen/work/bouss/radial_flat/xgeoclaw with 6 MPI process(es) and PETSC_ARCH on MacMiniHome.local by praveen Thu Oct 24 08:04:27 2024 >>>>>> [0]PETSC ERROR: Configure options: AR=arm64-apple-darwin20.0.0-ar CC=mpicc CXX=mpicxx FC=mpifort CFLAGS="-ftree-vectorize -fPIC -fstack-protector-strong -O2 -pipe -isystem /opt/homebrew/Caskroom/miniforge/base/envs/claw/include " CPPFLAGS="-D_FORTIFY_SOURCE=2 -isystem /opt/homebrew/Caskroom/miniforge/base/envs/claw/include -mmacosx-version-min=11.0 -mmacosx-version-min=11.0" CXXFLAGS="-ftree-vectorize -fPIC -fstack-protector-strong -O2 -pipe -stdlib=libc++ -fvisibility-inlines-hidden -fmessage-length=0 -isystem /opt/homebrew/Caskroom/miniforge/base/envs/claw/include " FFLAGS="-march=armv8.3-a -ftree-vectorize -fPIC -fno-stack-protector -O2 -pipe -isystem /opt/homebrew/Caskroom/miniforge/base/envs/claw/include " LDFLAGS="-Wl,-headerpad_max_install_names -Wl,-dead_strip_dylibs -Wl,-rpath,/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib -L/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib" LIBS="-Wl,-rpath,/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib -lmpi_mpifh -lgfortran" --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 --FOPTFLAGS=-O3 --with-clib-autodetect=0 --with-cxxlib-autodetect=0 --with-fortranlib-autodetect=0 --with-debugging=0 --with-blas-lib=libblas.dylib --with-lapack-lib=liblapack.dylib --with-yaml=1 --with-hdf5=1 --with-fftw=1 --with-hwloc=0 --with-hypre=1 --with-metis=1 --with-mpi=1 --with-mumps=1 --with-parmetis=1 --with-pthread=1 --with-ptscotch=1 --with-shared-libraries --with-ssl=0 --with-scalapack=1 --with-superlu=1 --with-superlu_dist=1 --with-superlu_dist-include=/opt/homebrew/Caskroom/miniforge/base/envs/claw/include/superlu-dist --with-superlu_dist-lib=-lsuperlu_dist --with-suitesparse=1 --with-suitesparse-dir=/opt/homebrew/Caskroom/miniforge/base/envs/claw --with-x=0 --with-scalar-type=real --with-cuda=0 --with-batch --prefix=/opt/homebrew/Caskroom/miniforge/base/envs/claw >>>>>> [0]PETSC ERROR: #1 PetscShmgetMapAddresses() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/sys/utils/server.c:114 >>>>>> [0]PETSC ERROR: #2 PCMPISetMat() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/ksp/pc/impls/mpi/pcmpi.c:269 >>>>>> [0]PETSC ERROR: #3 PCSetUp_MPI() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/ksp/pc/impls/mpi/pcmpi.c:853 >>>>>> [0]PETSC ERROR: #4 PCSetUp() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/ksp/pc/interface/precon.c:1071 >>>>>> [0]PETSC ERROR: #5 KSPSetUp() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/ksp/ksp/interface/itfunc.c:415 >>>>>> [0]PETSC ERROR: #6 KSPSolve_Private() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/ksp/ksp/interface/itfunc.c:826 >>>>>> [0]PETSC ERROR: #7 KSPSolve() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/ksp/ksp/interface/itfunc.c:1075 >>>>>> >>>>>> Code does not progress and I kill it >>>>>> >>>>>> ^CTraceback (most recent call last): >>>>>> File "/Users/praveen/Applications/clawpack/clawutil/src/python/clawutil/runclaw.py", line 341, in >>>>>> runclaw(*args) >>>>>> File "/Users/praveen/Applications/clawpack/clawutil/src/python/clawutil/runclaw.py", line 242, in runclaw >>>>>> proc = subprocess.check_call(cmd_split, >>>>>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ >>>>>> File "/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib/python3.12/subprocess.py", line 408, in check_call >>>>>> retcode = call(*popenargs, **kwargs) >>>>>> ^^^^^^^^^^^^^^^^^^^^^^^^^^ >>>>>> File "/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib/python3.12/subprocess.py", line 391, in call >>>>>> return p.wait(timeout=timeout) >>>>>> ^^^^^^^^^^^^^^^^^^^^^^^ >>>>>> File "/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib/python3.12/subprocess.py", line 1264, in wait >>>>>> return self._wait(timeout=timeout) >>>>>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^ >>>>>> File "/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib/python3.12/subprocess.py", line 2053, in _wait >>>>>> (pid, sts) = self._try_wait(0) >>>>>> ^^^^^^^^^^^^^^^^^ >>>>>> File "/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib/python3.12/subprocess.py", line 2011, in _try_wait >>>>>> (pid, sts) = os.waitpid(self.pid, wait_flags) >>>>>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ >>>>>> KeyboardInterrupt >>>>>> make[1]: *** [output] Interrupt: 2 >>>>>> make: *** [.output] Interrupt: 2 >>>>>> >>>>>> Now it says >>>>>> >>>>>> $ ipcs -m >>>>>> IPC status from as of Thu Oct 24 08:05:06 IST 2024 >>>>>> T ID KEY MODE OWNER GROUP >>>>>> Shared Memory: >>>>>> m 720896 0x0000000a --rw-rw-rw- praveen staff >>>>>> >>>>>> Thanks >>>>>> praveen >>>>>> >>>>>>> On 23 Oct 2024, at 8:26?PM, Barry Smith wrote: >>>>>>> >>>>>>> >>>>>>> Hmm, so it is creating the first shared memory region with ID of 10 (A in hex) puts it in a linked list in PETSc but then when it tries to find it in the linked list it cannot find it. >>>>>>> >>>>>>> I don't know how to reproduce this or debug it remotely. >>>>>>> >>>>>>> Can you build on a completely different machine or with completely different compilers? >>>>>>> >>>>>>> Barry >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> On Oct 23, 2024, at 10:31?AM, Praveen C wrote: >>>>>>>> >>>>>>>> I get same error and now it shows >>>>>>>> >>>>>>>> $ ipcs -m >>>>>>>> >>>>>>>> ------ Shared Memory Segments -------- >>>>>>>> key shmid owner perms bytes nattch status >>>>>>>> 0x0000000a 32788 praveen 666 240 6 >>>>>>>> >>>>>>>> Note that the code seems to be still running after printing those error message, but it is not printing any progress which it should do. >>>>>>>> >>>>>>>> Thanks >>>>>>>> praveen >>>>>>>> >>>>>>>>> On 23 Oct 2024, at 7:56?PM, Barry Smith wrote: >>>>>>>>> >>>>>>>>> >>>>>>>>> Try >>>>>>>>> >>>>>>>>> ipcrm -m 11 >>>>>>>>> >>>>>>>>> ipcs -m >>>>>>>>> >>>>>>>>> Try running the program again >>>>>>>>> >>>>>>>>> If failed check >>>>>>>>> >>>>>>>>> ipcs -m >>>>>>>>> >>>>>>>>> again >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> On Oct 23, 2024, at 10:20?AM, Praveen C wrote: >>>>>>>>>> >>>>>>>>>> Hello Barry >>>>>>>>>> >>>>>>>>>> I see this >>>>>>>>>> >>>>>>>>>> $ ipcs -m >>>>>>>>>> >>>>>>>>>> ------ Shared Memory Segments -------- >>>>>>>>>> key shmid owner perms bytes nattch status >>>>>>>>>> 0x0000000a 11 praveen 666 240 6 >>>>>>>>>> >>>>>>>>>> and I am observing same error as below. >>>>>>>>>> >>>>>>>>>> Thanks >>>>>>>>>> praveen >>>>>>>>>> >>>>>>>>>>> On 23 Oct 2024, at 7:08?PM, Barry Smith wrote: >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Please take a look at the notes in https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/PetscShmgetAllocateArray/__;!!G_uCfscf7eWS!fM-tM-j46TWI2-dEM5dWqIZEgpVdFBhYKU4M3RXKtOogG10VbWUtHGAQv6q7qO0TV_YQ1yP5pgPAY1_RqN86qA$ For some reason your program is not able to access/use the Unix shared memory; check if you are already using the shared memory (so it is not available for a new run) or the limits are too low to access enough memory. >>>>>>>>>>> >>>>>>>>>>> Barry >>>>>>>>>>> >>>>>>>>>>>> On Oct 23, 2024, at 8:23?AM, Praveen C wrote: >>>>>>>>>>>> >>>>>>>>>>>> Dear all >>>>>>>>>>>> >>>>>>>>>>>> I am not able to run the boussinesq example from geoclaw using petsc at 3.22.0 >>>>>>>>>>>> >>>>>>>>>>>> https://urldefense.us/v3/__https://github.com/clawpack/geoclaw/tree/3303883f46572c58130d161986b8a87a57ca7816/examples/bouss__;!!G_uCfscf7eWS!e3VQ4NHKmXGstRsQW5vtI7fmKfUT9zmJkMJcPbcvPyIjicyfJpNoMgx3wZ-qyGcKNSjIkNZkzilec8MnHN6PMw$ >>>>>>>>>>>> >>>>>>>>>>>> It runs with petsc at 3.21.6 >>>>>>>>>>>> >>>>>>>>>>>> The error I get is given below. After printing this, the code does not progress. >>>>>>>>>>>> >>>>>>>>>>>> I use the following petsc options >>>>>>>>>>>> >>>>>>>>>>>> # set min numbers of matrix rows per MPI rank (default is 10000) >>>>>>>>>>>> -mpi_linear_solve_minimum_count_per_rank 5000 >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> # Krylov linear solver: >>>>>>>>>>>> -mpi_linear_solver_server >>>>>>>>>>>> -mpi_linear_solver_server_view >>>>>>>>>>>> -ksp_type gmres >>>>>>>>>>>> -ksp_max_it 200 >>>>>>>>>>>> -ksp_reuse_preconditioner >>>>>>>>>>>> -ksp_rtol 1.e-9 >>>>>>>>>>>> >>>>>>>>>>>> # preconditioner: >>>>>>>>>>>> -pc_type gamg >>>>>>>>>>>> >>>>>>>>>>>> I installed petsc and other dependencies for clawpack using miniforge. >>>>>>>>>>>> >>>>>>>>>>>> Thanks >>>>>>>>>>>> pc >>>>>>>>>>>> >>>>>>>>>>>> ==> Use Bouss. in water deeper than 1.0000000000000000 Using a PETSc solver >>>>>>>>>>>> Using Bouss equations from the start >>>>>>>>>>>> rnode allocated... >>>>>>>>>>>> node allocated... >>>>>>>>>>>> listOfGrids allocated... >>>>>>>>>>>> Storage allocated... >>>>>>>>>>>> bndList allocated... >>>>>>>>>>>> Gridding level 1 at t = 0.000000E+00: 4 grids with 10000 cells >>>>>>>>>>>> Setting initial dt to 2.9999999999999999E-002 >>>>>>>>>>>> max threads set to 6 >>>>>>>>>>>> Done reading data, starting computation ... Total zeta at initial time: 39269.907650665169 GEOCLAW: Frame 0 output files done at time t = 0.000000D+00 >>>>>>>>>>>> >>>>>>>>>>>> [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >>>>>>>>>>>> [0]PETSC ERROR: Petsc has generated inconsistent data >>>>>>>>>>>> [0]PETSC ERROR: Unable to locate PCMPI allocated shared address 0x55e6d750ae20 >>>>>>>>>>>> [0]PETSC ERROR: WARNING! There are unused option(s) set! Could be the program crashed before usage or a spelling mistake, etc! >>>>>>>>>>>> [0]PETSC ERROR: Option left: name:-ksp_max_it value: 200 source: file >>>>>>>>>>>> [0]PETSC ERROR: Option left: name:-ksp_reuse_preconditioner (no value) source: file >>>>>>>>>>>> [0]PETSC ERROR: Option left: name:-ksp_rtol value: 1.e-9 source: file >>>>>>>>>>>> [0]PETSC ERROR: Option left: name:-ksp_type value: gmres source: file >>>>>>>>>>>> [0]PETSC ERROR: Option left: name:-mpi_linear_solve_minimum_count_per_rank value: 5000 source: file >>>>>>>>>>>> [0]PETSC ERROR: Option left: name:-mpi_linear_solver_server_view (no value) source: file >>>>>>>>>>>> [0]PETSC ERROR: Option left: name:-pc_type value: gamg source: file >>>>>>>>>>>> [0]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!e3VQ4NHKmXGstRsQW5vtI7fmKfUT9zmJkMJcPbcvPyIjicyfJpNoMgx3wZ-qyGcKNSjIkNZkzilec8MvNjNo7A$ for trouble shooting. >>>>>>>>>>>> [0]PETSC ERROR: Petsc Release Version 3.22.0, Sep 28, 2024 [0]PETSC ERROR: /home/praveen/bouss/radial_flat/xgeoclaw with 6 MPI process(es) and PETSC_ARCH on euler by praveen Thu Oct 17 21:49:54 2024 >>>>>>>>>>>> [0]PETSC ERROR: Configure options: AR=${PREFIX}/bin/x86_64-conda-linux-gnu-ar CC=mpicc CXX=mpicxx FC=mpifort CFLAGS="-march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /opt/miniforge/envs/claw/include " CPPFLAGS="-DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /opt/miniforge/envs/claw/include" CXXFLAGS="-fvisibility-inlines-hidden -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /opt/miniforge/envs/claw/include " FFLAGS="-march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /opt/miniforge/envs/claw/include -Wl,--no-as-needed" LDFLAGS="-pthread -fopenmp -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,--allow-shlib-undefined -Wl,-rpath,/opt/miniforge/envs/claw/lib -Wl,-rpath-link,/opt/miniforge/envs/claw/lib -L/opt/miniforge/envs/claw/lib -Wl,-rpath-link,/opt/miniforge/envs/claw/lib" LIBS="-Wl,-rpath,/opt/miniforge/envs/claw/lib -lmpi_mpifh -lgfortran" --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 --FOPTFLAGS=-O3 --with-clib-autodetect=0 --with-cxxlib-autodetect=0 --with-fortranlib-autodetect=0 --with-debugging=0 --with-blas-lib=libblas.so --with-lapack-lib=liblapack.so --with-yaml=1 --with-hdf5=1 --with-fftw=1 --with-hwloc=0 --with-hypre=1 --with-metis=1 --with-mpi=1 --with-mumps=1 --with-parmetis=1 --with-pthread=1 --with-ptscotch=1 --with-shared-libraries --with-ssl=0 --with-scalapack=1 --with-superlu=1 --with-superlu_dist=1 --with-superlu_dist-include=/opt/miniforge/envs/claw/include/superlu-dist --with-superlu_dist-lib=-lsuperlu_dist --with-suitesparse=1 --with-suitesparse-dir=/opt/miniforge/envs/claw --with-x=0 --with-scalar-type=real --with-cuda=0 --prefix=/opt/miniforge/envs/claw >>>>>>>>>>>> [0]PETSC ERROR: #1 PetscShmgetMapAddresses() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/sys/utils/server.c:114 >>>>>>>>>>>> [0]PETSC ERROR: #2 PCMPISetMat() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/pc/impls/mpi/pcmpi.c:269 >>>>>>>>>>>> [0]PETSC ERROR: #3 PCSetUp_MPI() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/pc/impls/mpi/pcmpi.c:853 >>>>>>>>>>>> [0]PETSC ERROR: #4 PCSetUp() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/pc/interface/precon.c:1071 >>>>>>>>>>>> [0]PETSC ERROR: #5 KSPSetUp() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/ksp/interface/itfunc.c:415 >>>>>>>>>>>> [0]PETSC ERROR: #6 KSPSolve_Private() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/ksp/interface/itfunc.c:826 >>>>>>>>>>>> [0]PETSC ERROR: #7 KSPSolve() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/ksp/interface/itfunc.c:1075 >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: petscMPIoptions Type: application/octet-stream Size: 1166 bytes Desc: not available URL: -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: setenv.sh Type: application/octet-stream Size: 566 bytes Desc: not available URL: -------------- next part -------------- An HTML attachment was scrubbed... URL: From mail2amneet at gmail.com Wed Oct 30 09:53:09 2024 From: mail2amneet at gmail.com (Amneet Bhalla) Date: Wed, 30 Oct 2024 07:53:09 -0700 Subject: [petsc-users] Rigid body nullspace for Stokes operator In-Reply-To: <875xpas35p.fsf@jedbrown.org> References: <875xpas35p.fsf@jedbrown.org> Message-ID: @Mark: Is there some document/paper that I can follow to check the algebra of these zero eigenvectors/null space modes? @Jed : We use a projection method preconditioner to solve the coupled velocity pressure system as described here ( https://urldefense.us/v3/__https://www.sciencedirect.com/science/article/pii/S0021999123004205__;!!G_uCfscf7eWS!ZRrdNNCSkRL0WcX9v_6J47NL0alKPWI62lpOHMD1McnE_Mj0VpDWMYMqnlAYtYcpBM9gXe_PJCQpsqHNGDw-c3gVcg$ ). It is an approximation of the Schur complement. As a part of projection preconditioner, we need to solve just the momentum equation separately without considering the pressure part. On Tue, Oct 29, 2024 at 8:03?PM Jed Brown wrote: > And to be clear, we recommend using fieldsplit Schur to separate the > pressure and velocity part (there are many examples). Applying AMG directly > to the saddle point problem will not be a good solver because the > heuristics assume positivity and do not preserve inf-sup stability (nor do > standard smoothers). > > Mark Adams writes: > > > This is linear elasticity and there are 6 "null" vectors (if you removed > > Dirichlet boundary conditions these are eigenvectors with zero > eigenvalue): > > 3 translations, x, y, z, and three rotatiions xx, yy ,zz. > > x = (1,0,0,1,0,0,1,0 ...) > > and xx is something like (0, z_1, -y_1, 0, z_2, -y_2, ...) where z_1 is > the > > z coordinate of the first vertex, etc. > > > > Mark > > > > On Tue, Oct 29, 2024 at 3:47?PM Amneet Bhalla > wrote: > > > >> Hi Mark, > >> > >> Thanks! I am not sure how to construct null space and zero energy modes > >> manually for this operator. Is there some theory or documentation I can > >> follow to figure out what the null space and zero energy modes look like > >> for this operator? Once I know what these are in symbolic form, I think > I > >> should be able to construct them manually. > >> > >> Best, > >> --Amneet > >> > >> On Tue, Oct 29, 2024 at 7:35?AM Mark Adams wrote: > >> > >>> Oh my mistake. You are using staggered grids. So you don't have a block > >>> size that hypre would use for the "nodal" methods. > >>> I'm not sure what you are doing exactly, but try hypre and you could > >>> create the null space, zero energy modes, manually, attach to the > matrix > >>> and try GAMG. > >>> You can run with '-info :pc' and grep on GAMG to see if GAMG is picking > >>> the null space up (send this output if you can't figure it out). > >>> > >>> Thanks, > >>> Mark > >>> > >>> On Tue, Oct 29, 2024 at 9:28?AM Mark Adams wrote: > >>> > >>>> This coordinate interface is just a shortcut for vertex based > >>>> discretizations with 3 dof per vertex, etc. (maybe works in 2D). > >>>> You will need to construct the null space vectors manually and attach > it > >>>> to the matrix. Used by GAMG. > >>>> > >>>> Note, for hypre you want to use the "nodal" options and it does not > use > >>>> these null space vectors. That is probably the way you want to go. > >>>> eg: -pc_hypre_boomeramg_nodal_coarsen > >>>> > >>>> I would run with hypre boomerang and -help and grep on nodal to see > all > >>>> the "nodal" options and use them. > >>>> > >>>> Thanks, > >>>> Mark > >>>> > >>>> > >>>> On Mon, Oct 28, 2024 at 8:06?PM Amneet Bhalla > >>>> wrote: > >>>> > >>>>> Hi Folks, > >>>>> > >>>>> I am trying to solve the momentum equation in a projection > >>>>> preconditioner using GAMG or Hypre solver. The equation looks like > for > >>>>> velocity variable *v* looks like: > >>>>> > >>>>> > >>>>> [image: Screenshot 2024-10-28 at 4.15.17 PM.png] > >>>>> > >>>>> Here, ? is spatially varying dynamic viscosity and ? is spatially > >>>>> varying bulk viscosity. I understand that I need to specify rigid > body > >>>>> nullspace modes to the multigrid solver in order to accelerate its > >>>>> convergence. Looking into this routine MatNullSpaceCreateRigidBody() > ( > >>>>> > https://urldefense.us/v3/__https://petsc.org/release/manualpages/Mat/MatNullSpaceCreateRigidBody/__;!!G_uCfscf7eWS!bVR6duCoDqPhZrWS-sm1c5qxsFPjZMhdT86AqLpPzWgVy5qoRhd4_Jue2LJOIS6LRrtV2cHGrqger1Yvb-Y5f-0$ > >>>>> < > https://urldefense.us/v3/__https://petsc.org/release/manualpages/Mat/MatNullSpaceCreateRigidBody/__;!!G_uCfscf7eWS!eKqgIJjCdMzIU76f7X65AmGxrU_-lC7W02BMWafJ77DNf_IuQk6O1X3qU1x9Ez8NJ20vZEL-mF6T1yNmDnwv0eWa2w$ > >), > >>>>> I see that I need to provide the coordinates of each node. I am using > >>>>> staggered grid discretization. Do I need to provide coordinates of > >>>>> staggered grid locations? > >>>>> > >>>>> Thanks, > >>>>> -- > >>>>> --Amneet > >>>>> > >>>>> > >>>>> > >>>>> > >> > >> -- > >> --Amneet > >> > >> > >> > >> > -- --Amneet -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Wed Oct 30 10:57:38 2024 From: bsmith at petsc.dev (Barry Smith) Date: Wed, 30 Oct 2024 11:57:38 -0400 Subject: [petsc-users] Problem running geoclaw example with petsc 3.22 In-Reply-To: References: <1C63D19F-6EBC-43A3-8B40-65EA34991AFA@gmail.com> <0136027D-C2CE-47D0-AF97-ACDE1DA0C382@gmail.com> <4726A341-54DB-4592-BBAE-843AF271F28E@petsc.dev> <27B47E2D-00D5-4326-95A9-7595DFF0E952@gmail.com> <490B5A4A-BBD8-490A-91FD-7CEC9E01A99B@petsc.dev> <96674E24-9EDA-42DE-8459-81D46A260CDE@petsc.dev> Message-ID: <61A25E4A-998E-4D1B-AFE9-3074D5D807C4@petsc.dev> ok, I get what is below, no crash, but I don't understand why it never uses more than 1 level. Barry ==> Running with command: /opt/homebrew/Caskroom/miniconda/base/envs/claw//bin/mpiexec -n 1 /Users/barrysmith/Src/clawpack/geoclaw/examples/bouss/radial_flat/xgeoclaw Reading data file: claw.data first 5 lines are comments and will be skipped Reading data file: amr.data first 5 lines are comments and will be skipped Running amrclaw ... Reading data file: geoclaw.data first 5 lines are comments and will be skipped Reading data file: refinement.data first 5 lines are comments and will be skipped Reading data file: dtopo.data first 5 lines are comments and will be skipped Reading data file: topo.data first 5 lines are comments and will be skipped converting to topotype > 1 might reduce file size python tools for converting files are provided Reading topography file /Users/barrysmith/Src/clawpack/geoclaw/examples/bouss/radial_flat/flat100.tt1 Reading data file: qinit.data first 5 lines are comments and will be skipped qinit_type = 0, no perturbation Reading data file: fgout_grids.data first 5 lines are comments and will be skipped Reading data file: friction.data first 5 lines are comments and will be skipped Reading data file: multilayer.data first 5 lines are comments and will be skipped Reading data file: surge.data first 5 lines are comments and will be skipped Reading data file: regions.data first 5 lines are comments and will be skipped Reading data file: flagregions.data first 5 lines are comments and will be skipped +++ rregion bounding box: 0.0000000000000000 5000.0000000000000 0.0000000000000000 5000.0000000000000 +++ i, rr%s(1), rr%ds: 1 0.0000000000000000 5000.0000000000000 +++ Ruled region name: Region_diagonal +++ Ruled region file_name: /Users/barrysmith/Src/clawpack/geoclaw/examples/bouss/radial_flat/RuledRectangle_Diagonal.data +++ rregion bounding box: 0.0000000000000000 5000.0000000000000 -1000.0000000000000 6000.0000000000000 +++ i, rr%s(1), rr%ds: 2 0.0000000000000000 5000.0000000000000 +++ rregion bounding box: 0.0000000000000000 1000.0000000000000 0.0000000000000000 1000.0000000000000 +++ i, rr%s(1), rr%ds: 3 0.0000000000000000 1000.0000000000000 Reading data file: gauges.data first 5 lines are comments and will be skipped Reading data file: fgmax_grids.data first 5 lines are comments and will be skipped Reading data file: adjoint.data first 5 lines are comments and will be skipped Reading data file: bouss.data first 5 lines are comments and will be skipped Using SGN equations ==> Applying Bouss equations to selected grids between levels 1 and 10 ==> Use Bouss. in water deeper than 1.0000000000000000 Using a PETSc solver Using Bouss equations from the start rnode allocated... node allocated... listOfGrids allocated... Storage allocated... bndList allocated... Gridding level 1 at t = 0.000000E+00: 4 grids with 10000 cells Setting initial dt to 2.9999999999999999E-002 max threads set to 1 Done reading data, starting computation ... Total zeta at initial time: 39269.785700799781 GEOCLAW: Frame 0 output files done at time t = 0.000000D+00 AMRCLAW: level 1 CFL = .192E-01 dt = 0.3000E-01 final t = 0.300000E-01 AMRCLAW: level 1 CFL = .752E+00 dt = 0.1172E+01 final t = 0.120248E+01 AMRCLAW: level 1 CFL = .749E+00 dt = 0.1169E+01 final t = 0.237133E+01 AMRCLAW: level 1 CFL = .748E+00 dt = 0.1170E+01 final t = 0.354179E+01 AMRCLAW: level 1 CFL = .748E+00 dt = 0.1174E+01 final t = 0.471569E+01 AMRCLAW: level 1 CFL = .748E+00 dt = 0.1177E+01 final t = 0.589266E+01 AMRCLAW: level 1 CFL = .750E+00 dt = 0.1179E+01 final t = 0.707205E+01 AMRCLAW: level 1 CFL = .750E+00 dt = 0.1180E+01 final t = 0.825165E+01 AMRCLAW: level 1 CFL = .749E+00 dt = 0.1180E+01 final t = 0.943177E+01 AMRCLAW: level 1 CFL = .360E+00 dt = 0.5682E+00 final t = 0.100000E+02 GEOCLAW: Frame 1 output files done at time t = 0.100000D+02 AMRCLAW: level 1 CFL > On Oct 30, 2024, at 9:05?AM, Praveen C wrote: > > I have attached some files for this > > cd bouss > . setenv.sh # some settings in this file may need to be changed > > cd radial_flat/1d_radial > make .output > cd .. > make .output > > Thanks > praveen > > > > >> On 30 Oct 2024, at 6:28?PM, Barry Smith wrote: >> >> >> Please send me exact instructions on how you are running the geoclaw example, when I run >> >> geoclaw/examples/bouss/radial_flat >> >> >> it only runs with one level, but never errors, I changed setrun.py to amrdata.amr_levels_max = 5 but it made no difference. >> >> Thanks >> >> Barry >> >> >> >>> On Oct 24, 2024, at 12:17?PM, Praveen C wrote: >>> >>> Hello Barry >>> >>> I use this script to install clawpack and required dependencies >>> >>> https://urldefense.us/v3/__https://github.com/cpraveen/cfdlab/blob/master/bin/clawpack.sh__;!!G_uCfscf7eWS!f1DiQ8G6UmANFjslNZcmJqVvHN6inDzfMhpk0lw9SBjhLwHsJZQkGKO-AqOKFLhvsgX82tniWEhpMyhJ_yn90KY$ >>> >>> See lines 125-132 >>> >>> You can use it like this >>> >>> export CLAW=/path/to/where/you/want/clawpack >>> bash clawpack.sh v5.11.0 >>> >>> This will git pull clawpack and creates a conda env called ?claw? and installs inside that. >>> >>> I have not specified petsc version in this script, but latest miniforge should install petsc at 3.22 >>> >>> Thank you >>> praveen >>> >>>> On 24 Oct 2024, at 9:37?PM, Barry Smith wrote: >>>> >>>> >>>> Good, some helpful information with the runs you made. >>>> >>>> In the crash below it made more progress it was able to allocate multiple regions of shared memory and access them. I don't know why it would crash later. >>>> >>>> Can you tell me all the steps with miniforge (which seems to be related to the failure) you use? I've never used miniforge. >>>> >>>> If I can get to an environment that reproduces the problem I can debug it and fix it. >>>> >>>> Barry >>>> >>>> >>>>> On Oct 24, 2024, at 11:49?AM, Praveen C wrote: >>>>> >>>>> I get this >>>>> >>>>> $ mpiexec -n 3 ./ex89f -n 20 -mpi_linear_solver_server -mpi_linear_solver_server -mpi_linear_solver_server_ksp_view -ksp_monitor -ksp_converged_reason -ksp_view -mpi_linear_solver_server_minimum_count_per_rank 5 >>>>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>>>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range >>>>> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >>>>> [0]PETSC ERROR: or see https://urldefense.us/v3/__https://petsc.org/release/faq/*valgrind__;Iw!!G_uCfscf7eWS!f1DiQ8G6UmANFjslNZcmJqVvHN6inDzfMhpk0lw9SBjhLwHsJZQkGKO-AqOKFLhvsgX82tniWEhpMyhJs2ihDgI$ and https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!f1DiQ8G6UmANFjslNZcmJqVvHN6inDzfMhpk0lw9SBjhLwHsJZQkGKO-AqOKFLhvsgX82tniWEhpMyhJJxW1N9o$ >>>>> [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run >>>>> [0]PETSC ERROR: to get more information on the crash. >>>>> [0]PETSC ERROR: Run with -malloc_debug to check if memory corruption is causing the crash. >>>>> -------------------------------------------------------------------------- >>>>> MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_SELF >>>>> Proc: [[47380,1],0] >>>>> Errorcode: 59 >>>>> >>>>> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. >>>>> You may or may not see output from other processes, depending on >>>>> exactly when Open MPI kills them. >>>>> -------------------------------------------------------------------------- >>>>> -------------------------------------------------------------------------- >>>>> prterun has exited due to process rank 0 with PID 0 on node MacMiniHome calling >>>>> "abort". This may have caused other processes in the application to be >>>>> terminated by signals sent by prterun (as reported here). >>>>> ????????????????????????????????????? >>>>> >>>>> and I have this after the code exits >>>>> >>>>> $ ipcs -m >>>>> IPC status from as of Thu Oct 24 21:17:39 IST 2024 >>>>> T ID KEY MODE OWNER GROUP >>>>> Shared Memory: >>>>> m 1572864 0x0000000b --rw-rw-rw- praveen staff >>>>> m 524289 0x0000000c --rw-rw-rw- praveen staff >>>>> m 655362 0x0000000d --rw-rw-rw- praveen staff >>>>> m 262147 0x0000000e --rw-rw-rw- praveen staff >>>>> m 262148 0x0000000f --rw-rw-rw- praveen staff >>>>> m 393221 0x0000000a --rw-rw-rw- praveen staff >>>>> >>>>> This is with petsc installed with miniforge, which I also use with clawpack. With spack installed petsc, I can run the ex89f example. >>>>> >>>>> Thanks >>>>> praveen >>>>> >>>>>> On 24 Oct 2024, at 7:55?PM, Barry Smith wrote: >>>>>> >>>>>> >>>>>> Ok, super strange, I've run many times on my Mac >>>>>> >>>>>> Can you please try to remove that allocated memory with ipcrm and then >>>>>> >>>>>> cd $PETSC_DIR/src/ksp/ksp/tutorials >>>>>> make ex89f >>>>>> mpiexec -n 3 ./ex89f -n 20 -mpi_linear_solver_server -mpi_linear_solver_server -mpi_linear_solver_server_ksp_view -ksp_monitor -ksp_converged_reason -ksp_view -mpi_linear_solver_server_minimum_count_per_rank 5 >>>>>> >>>>>> This does the same thing as the GeoClaw code but is much simpler. >>>>>> >>>>>> Barry >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> On Oct 23, 2024, at 10:55?PM, Praveen C wrote: >>>>>>> >>>>>>> I get very similar error on my mac with >>>>>>> >>>>>>> $ gfortran -v >>>>>>> Using built-in specs. >>>>>>> COLLECT_GCC=gfortran >>>>>>> COLLECT_LTO_WRAPPER=/opt/homebrew/Caskroom/miniforge/base/envs/claw/libexec/gcc/arm64-apple-darwin20.0.0/13.2.0/lto-wrapper >>>>>>> Target: arm64-apple-darwin20.0.0 >>>>>>> Configured with: ../configure --prefix=/opt/homebrew/Caskroom/miniforge/base/envs/claw --build=x86_64-apple-darwin13.4.0 --host=arm64-apple-darwin20.0.0 --target=arm64-apple-darwin20.0.0 --with-libiconv-prefix=/opt/homebrew/Caskroom/miniforge/base/envs/claw --enable-languages=fortran --disable-multilib --enable-checking=release --disable-bootstrap --disable-libssp --with-gmp=/opt/homebrew/Caskroom/miniforge/base/envs/claw --with-mpfr=/opt/homebrew/Caskroom/miniforge/base/envs/claw --with-mpc=/opt/homebrew/Caskroom/miniforge/base/envs/claw --with-isl=/opt/homebrew/Caskroom/miniforge/base/envs/claw --enable-darwin-at-rpath >>>>>>> Thread model: posix >>>>>>> Supported LTO compression algorithms: zlib >>>>>>> gcc version 13.2.0 (GCC) >>>>>>> >>>>>>> Before starting >>>>>>> >>>>>>> $ ipcs -m >>>>>>> IPC status from as of Thu Oct 24 08:02:11 IST 2024 >>>>>>> T ID KEY MODE OWNER GROUP >>>>>>> Shared Memory: >>>>>>> >>>>>>> and when I run the code >>>>>>> >>>>>>> Using a PETSc solver >>>>>>> Using Bouss equations from the start >>>>>>> rnode allocated... >>>>>>> node allocated... >>>>>>> listOfGrids allocated... >>>>>>> Storage allocated... >>>>>>> bndList allocated... >>>>>>> Gridding level 1 at t = 0.000000E+00: 4 grids with 10000 cells >>>>>>> Setting initial dt to 2.9999999999999999E-002 >>>>>>> max threads set to 1 >>>>>>> >>>>>>> Done reading data, starting computation ... >>>>>>> >>>>>>> Total zeta at initial time: 39269.907650665169 >>>>>>> GEOCLAW: Frame 0 output files done at time t = 0.000000D+00 >>>>>>> >>>>>>> [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >>>>>>> [0]PETSC ERROR: Petsc has generated inconsistent data >>>>>>> [0]PETSC ERROR: Unable to locate PCMPI allocated shared address 0x130698000 >>>>>>> [0]PETSC ERROR: WARNING! There are unused option(s) set! Could be the program crashed before usage or a spelling mistake, etc! >>>>>>> [0]PETSC ERROR: Option left: name:-ksp_type value: preonly source: file >>>>>>> [0]PETSC ERROR: Option left: name:-mpi_ksp_max_it value: 200 source: file >>>>>>> [0]PETSC ERROR: Option left: name:-mpi_ksp_reuse_preconditioner (no value) source: file >>>>>>> [0]PETSC ERROR: Option left: name:-mpi_ksp_rtol value: 1.e-9 source: file >>>>>>> [0]PETSC ERROR: Option left: name:-mpi_ksp_type value: gmres source: file >>>>>>> [0]PETSC ERROR: Option left: name:-mpi_linear_solver_server_view (no value) source: file >>>>>>> [0]PETSC ERROR: Option left: name:-mpi_pc_gamg_sym_graph value: true source: file >>>>>>> [0]PETSC ERROR: Option left: name:-mpi_pc_gamg_symmetrize_graph value: true source: file >>>>>>> [0]PETSC ERROR: Option left: name:-mpi_pc_type value: gamg source: file >>>>>>> [0]PETSC ERROR: Option left: name:-pc_mpi_minimum_count_per_rank value: 5000 source: file >>>>>>> [0]PETSC ERROR: Option left: name:-pc_type value: mpi source: file >>>>>>> [0]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!f1DiQ8G6UmANFjslNZcmJqVvHN6inDzfMhpk0lw9SBjhLwHsJZQkGKO-AqOKFLhvsgX82tniWEhpMyhJJxW1N9o$ for trouble shooting. >>>>>>> [0]PETSC ERROR: Petsc Release Version 3.22.0, Sep 28, 2024 >>>>>>> [0]PETSC ERROR: /Users/praveen/work/bouss/radial_flat/xgeoclaw with 6 MPI process(es) and PETSC_ARCH on MacMiniHome.local by praveen Thu Oct 24 08:04:27 2024 >>>>>>> [0]PETSC ERROR: Configure options: AR=arm64-apple-darwin20.0.0-ar CC=mpicc CXX=mpicxx FC=mpifort CFLAGS="-ftree-vectorize -fPIC -fstack-protector-strong -O2 -pipe -isystem /opt/homebrew/Caskroom/miniforge/base/envs/claw/include " CPPFLAGS="-D_FORTIFY_SOURCE=2 -isystem /opt/homebrew/Caskroom/miniforge/base/envs/claw/include -mmacosx-version-min=11.0 -mmacosx-version-min=11.0" CXXFLAGS="-ftree-vectorize -fPIC -fstack-protector-strong -O2 -pipe -stdlib=libc++ -fvisibility-inlines-hidden -fmessage-length=0 -isystem /opt/homebrew/Caskroom/miniforge/base/envs/claw/include " FFLAGS="-march=armv8.3-a -ftree-vectorize -fPIC -fno-stack-protector -O2 -pipe -isystem /opt/homebrew/Caskroom/miniforge/base/envs/claw/include " LDFLAGS="-Wl,-headerpad_max_install_names -Wl,-dead_strip_dylibs -Wl,-rpath,/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib -L/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib" LIBS="-Wl,-rpath,/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib -lmpi_mpifh -lgfortran" --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 --FOPTFLAGS=-O3 --with-clib-autodetect=0 --with-cxxlib-autodetect=0 --with-fortranlib-autodetect=0 --with-debugging=0 --with-blas-lib=libblas.dylib --with-lapack-lib=liblapack.dylib --with-yaml=1 --with-hdf5=1 --with-fftw=1 --with-hwloc=0 --with-hypre=1 --with-metis=1 --with-mpi=1 --with-mumps=1 --with-parmetis=1 --with-pthread=1 --with-ptscotch=1 --with-shared-libraries --with-ssl=0 --with-scalapack=1 --with-superlu=1 --with-superlu_dist=1 --with-superlu_dist-include=/opt/homebrew/Caskroom/miniforge/base/envs/claw/include/superlu-dist --with-superlu_dist-lib=-lsuperlu_dist --with-suitesparse=1 --with-suitesparse-dir=/opt/homebrew/Caskroom/miniforge/base/envs/claw --with-x=0 --with-scalar-type=real --with-cuda=0 --with-batch --prefix=/opt/homebrew/Caskroom/miniforge/base/envs/claw >>>>>>> [0]PETSC ERROR: #1 PetscShmgetMapAddresses() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/sys/utils/server.c:114 >>>>>>> [0]PETSC ERROR: #2 PCMPISetMat() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/ksp/pc/impls/mpi/pcmpi.c:269 >>>>>>> [0]PETSC ERROR: #3 PCSetUp_MPI() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/ksp/pc/impls/mpi/pcmpi.c:853 >>>>>>> [0]PETSC ERROR: #4 PCSetUp() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/ksp/pc/interface/precon.c:1071 >>>>>>> [0]PETSC ERROR: #5 KSPSetUp() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/ksp/ksp/interface/itfunc.c:415 >>>>>>> [0]PETSC ERROR: #6 KSPSolve_Private() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/ksp/ksp/interface/itfunc.c:826 >>>>>>> [0]PETSC ERROR: #7 KSPSolve() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/ksp/ksp/interface/itfunc.c:1075 >>>>>>> >>>>>>> Code does not progress and I kill it >>>>>>> >>>>>>> ^CTraceback (most recent call last): >>>>>>> File "/Users/praveen/Applications/clawpack/clawutil/src/python/clawutil/runclaw.py", line 341, in >>>>>>> runclaw(*args) >>>>>>> File "/Users/praveen/Applications/clawpack/clawutil/src/python/clawutil/runclaw.py", line 242, in runclaw >>>>>>> proc = subprocess.check_call(cmd_split, >>>>>>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ >>>>>>> File "/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib/python3.12/subprocess.py", line 408, in check_call >>>>>>> retcode = call(*popenargs, **kwargs) >>>>>>> ^^^^^^^^^^^^^^^^^^^^^^^^^^ >>>>>>> File "/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib/python3.12/subprocess.py", line 391, in call >>>>>>> return p.wait(timeout=timeout) >>>>>>> ^^^^^^^^^^^^^^^^^^^^^^^ >>>>>>> File "/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib/python3.12/subprocess.py", line 1264, in wait >>>>>>> return self._wait(timeout=timeout) >>>>>>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^ >>>>>>> File "/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib/python3.12/subprocess.py", line 2053, in _wait >>>>>>> (pid, sts) = self._try_wait(0) >>>>>>> ^^^^^^^^^^^^^^^^^ >>>>>>> File "/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib/python3.12/subprocess.py", line 2011, in _try_wait >>>>>>> (pid, sts) = os.waitpid(self.pid, wait_flags) >>>>>>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ >>>>>>> KeyboardInterrupt >>>>>>> make[1]: *** [output] Interrupt: 2 >>>>>>> make: *** [.output] Interrupt: 2 >>>>>>> >>>>>>> Now it says >>>>>>> >>>>>>> $ ipcs -m >>>>>>> IPC status from as of Thu Oct 24 08:05:06 IST 2024 >>>>>>> T ID KEY MODE OWNER GROUP >>>>>>> Shared Memory: >>>>>>> m 720896 0x0000000a --rw-rw-rw- praveen staff >>>>>>> >>>>>>> Thanks >>>>>>> praveen >>>>>>> >>>>>>>> On 23 Oct 2024, at 8:26?PM, Barry Smith wrote: >>>>>>>> >>>>>>>> >>>>>>>> Hmm, so it is creating the first shared memory region with ID of 10 (A in hex) puts it in a linked list in PETSc but then when it tries to find it in the linked list it cannot find it. >>>>>>>> >>>>>>>> I don't know how to reproduce this or debug it remotely. >>>>>>>> >>>>>>>> Can you build on a completely different machine or with completely different compilers? >>>>>>>> >>>>>>>> Barry >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> On Oct 23, 2024, at 10:31?AM, Praveen C wrote: >>>>>>>>> >>>>>>>>> I get same error and now it shows >>>>>>>>> >>>>>>>>> $ ipcs -m >>>>>>>>> >>>>>>>>> ------ Shared Memory Segments -------- >>>>>>>>> key shmid owner perms bytes nattch status >>>>>>>>> 0x0000000a 32788 praveen 666 240 6 >>>>>>>>> >>>>>>>>> Note that the code seems to be still running after printing those error message, but it is not printing any progress which it should do. >>>>>>>>> >>>>>>>>> Thanks >>>>>>>>> praveen >>>>>>>>> >>>>>>>>>> On 23 Oct 2024, at 7:56?PM, Barry Smith wrote: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Try >>>>>>>>>> >>>>>>>>>> ipcrm -m 11 >>>>>>>>>> >>>>>>>>>> ipcs -m >>>>>>>>>> >>>>>>>>>> Try running the program again >>>>>>>>>> >>>>>>>>>> If failed check >>>>>>>>>> >>>>>>>>>> ipcs -m >>>>>>>>>> >>>>>>>>>> again >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> On Oct 23, 2024, at 10:20?AM, Praveen C wrote: >>>>>>>>>>> >>>>>>>>>>> Hello Barry >>>>>>>>>>> >>>>>>>>>>> I see this >>>>>>>>>>> >>>>>>>>>>> $ ipcs -m >>>>>>>>>>> >>>>>>>>>>> ------ Shared Memory Segments -------- >>>>>>>>>>> key shmid owner perms bytes nattch status >>>>>>>>>>> 0x0000000a 11 praveen 666 240 6 >>>>>>>>>>> >>>>>>>>>>> and I am observing same error as below. >>>>>>>>>>> >>>>>>>>>>> Thanks >>>>>>>>>>> praveen >>>>>>>>>>> >>>>>>>>>>>> On 23 Oct 2024, at 7:08?PM, Barry Smith wrote: >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Please take a look at the notes in https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/PetscShmgetAllocateArray/__;!!G_uCfscf7eWS!f1DiQ8G6UmANFjslNZcmJqVvHN6inDzfMhpk0lw9SBjhLwHsJZQkGKO-AqOKFLhvsgX82tniWEhpMyhJOVJl36A$ For some reason your program is not able to access/use the Unix shared memory; check if you are already using the shared memory (so it is not available for a new run) or the limits are too low to access enough memory. >>>>>>>>>>>> >>>>>>>>>>>> Barry >>>>>>>>>>>> >>>>>>>>>>>>> On Oct 23, 2024, at 8:23?AM, Praveen C wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> Dear all >>>>>>>>>>>>> >>>>>>>>>>>>> I am not able to run the boussinesq example from geoclaw using petsc at 3.22.0 >>>>>>>>>>>>> >>>>>>>>>>>>> https://urldefense.us/v3/__https://github.com/clawpack/geoclaw/tree/3303883f46572c58130d161986b8a87a57ca7816/examples/bouss__;!!G_uCfscf7eWS!e3VQ4NHKmXGstRsQW5vtI7fmKfUT9zmJkMJcPbcvPyIjicyfJpNoMgx3wZ-qyGcKNSjIkNZkzilec8MnHN6PMw$ >>>>>>>>>>>>> >>>>>>>>>>>>> It runs with petsc at 3.21.6 >>>>>>>>>>>>> >>>>>>>>>>>>> The error I get is given below. After printing this, the code does not progress. >>>>>>>>>>>>> >>>>>>>>>>>>> I use the following petsc options >>>>>>>>>>>>> >>>>>>>>>>>>> # set min numbers of matrix rows per MPI rank (default is 10000) >>>>>>>>>>>>> -mpi_linear_solve_minimum_count_per_rank 5000 >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> # Krylov linear solver: >>>>>>>>>>>>> -mpi_linear_solver_server >>>>>>>>>>>>> -mpi_linear_solver_server_view >>>>>>>>>>>>> -ksp_type gmres >>>>>>>>>>>>> -ksp_max_it 200 >>>>>>>>>>>>> -ksp_reuse_preconditioner >>>>>>>>>>>>> -ksp_rtol 1.e-9 >>>>>>>>>>>>> >>>>>>>>>>>>> # preconditioner: >>>>>>>>>>>>> -pc_type gamg >>>>>>>>>>>>> >>>>>>>>>>>>> I installed petsc and other dependencies for clawpack using miniforge. >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks >>>>>>>>>>>>> pc >>>>>>>>>>>>> >>>>>>>>>>>>> ==> Use Bouss. in water deeper than 1.0000000000000000 Using a PETSc solver >>>>>>>>>>>>> Using Bouss equations from the start >>>>>>>>>>>>> rnode allocated... >>>>>>>>>>>>> node allocated... >>>>>>>>>>>>> listOfGrids allocated... >>>>>>>>>>>>> Storage allocated... >>>>>>>>>>>>> bndList allocated... >>>>>>>>>>>>> Gridding level 1 at t = 0.000000E+00: 4 grids with 10000 cells >>>>>>>>>>>>> Setting initial dt to 2.9999999999999999E-002 >>>>>>>>>>>>> max threads set to 6 >>>>>>>>>>>>> Done reading data, starting computation ... Total zeta at initial time: 39269.907650665169 GEOCLAW: Frame 0 output files done at time t = 0.000000D+00 >>>>>>>>>>>>> >>>>>>>>>>>>> [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >>>>>>>>>>>>> [0]PETSC ERROR: Petsc has generated inconsistent data >>>>>>>>>>>>> [0]PETSC ERROR: Unable to locate PCMPI allocated shared address 0x55e6d750ae20 >>>>>>>>>>>>> [0]PETSC ERROR: WARNING! There are unused option(s) set! Could be the program crashed before usage or a spelling mistake, etc! >>>>>>>>>>>>> [0]PETSC ERROR: Option left: name:-ksp_max_it value: 200 source: file >>>>>>>>>>>>> [0]PETSC ERROR: Option left: name:-ksp_reuse_preconditioner (no value) source: file >>>>>>>>>>>>> [0]PETSC ERROR: Option left: name:-ksp_rtol value: 1.e-9 source: file >>>>>>>>>>>>> [0]PETSC ERROR: Option left: name:-ksp_type value: gmres source: file >>>>>>>>>>>>> [0]PETSC ERROR: Option left: name:-mpi_linear_solve_minimum_count_per_rank value: 5000 source: file >>>>>>>>>>>>> [0]PETSC ERROR: Option left: name:-mpi_linear_solver_server_view (no value) source: file >>>>>>>>>>>>> [0]PETSC ERROR: Option left: name:-pc_type value: gamg source: file >>>>>>>>>>>>> [0]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!e3VQ4NHKmXGstRsQW5vtI7fmKfUT9zmJkMJcPbcvPyIjicyfJpNoMgx3wZ-qyGcKNSjIkNZkzilec8MvNjNo7A$ for trouble shooting. >>>>>>>>>>>>> [0]PETSC ERROR: Petsc Release Version 3.22.0, Sep 28, 2024 [0]PETSC ERROR: /home/praveen/bouss/radial_flat/xgeoclaw with 6 MPI process(es) and PETSC_ARCH on euler by praveen Thu Oct 17 21:49:54 2024 >>>>>>>>>>>>> [0]PETSC ERROR: Configure options: AR=${PREFIX}/bin/x86_64-conda-linux-gnu-ar CC=mpicc CXX=mpicxx FC=mpifort CFLAGS="-march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /opt/miniforge/envs/claw/include " CPPFLAGS="-DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /opt/miniforge/envs/claw/include" CXXFLAGS="-fvisibility-inlines-hidden -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /opt/miniforge/envs/claw/include " FFLAGS="-march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /opt/miniforge/envs/claw/include -Wl,--no-as-needed" LDFLAGS="-pthread -fopenmp -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,--allow-shlib-undefined -Wl,-rpath,/opt/miniforge/envs/claw/lib -Wl,-rpath-link,/opt/miniforge/envs/claw/lib -L/opt/miniforge/envs/claw/lib -Wl,-rpath-link,/opt/miniforge/envs/claw/lib" LIBS="-Wl,-rpath,/opt/miniforge/envs/claw/lib -lmpi_mpifh -lgfortran" --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 --FOPTFLAGS=-O3 --with-clib-autodetect=0 --with-cxxlib-autodetect=0 --with-fortranlib-autodetect=0 --with-debugging=0 --with-blas-lib=libblas.so --with-lapack-lib=liblapack.so --with-yaml=1 --with-hdf5=1 --with-fftw=1 --with-hwloc=0 --with-hypre=1 --with-metis=1 --with-mpi=1 --with-mumps=1 --with-parmetis=1 --with-pthread=1 --with-ptscotch=1 --with-shared-libraries --with-ssl=0 --with-scalapack=1 --with-superlu=1 --with-superlu_dist=1 --with-superlu_dist-include=/opt/miniforge/envs/claw/include/superlu-dist --with-superlu_dist-lib=-lsuperlu_dist --with-suitesparse=1 --with-suitesparse-dir=/opt/miniforge/envs/claw --with-x=0 --with-scalar-type=real --with-cuda=0 --prefix=/opt/miniforge/envs/claw >>>>>>>>>>>>> [0]PETSC ERROR: #1 PetscShmgetMapAddresses() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/sys/utils/server.c:114 >>>>>>>>>>>>> [0]PETSC ERROR: #2 PCMPISetMat() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/pc/impls/mpi/pcmpi.c:269 >>>>>>>>>>>>> [0]PETSC ERROR: #3 PCSetUp_MPI() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/pc/impls/mpi/pcmpi.c:853 >>>>>>>>>>>>> [0]PETSC ERROR: #4 PCSetUp() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/pc/interface/precon.c:1071 >>>>>>>>>>>>> [0]PETSC ERROR: #5 KSPSetUp() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/ksp/interface/itfunc.c:415 >>>>>>>>>>>>> [0]PETSC ERROR: #6 KSPSolve_Private() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/ksp/interface/itfunc.c:826 >>>>>>>>>>>>> [0]PETSC ERROR: #7 KSPSolve() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/ksp/interface/itfunc.c:1075 >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Wed Oct 30 11:03:12 2024 From: bsmith at petsc.dev (Barry Smith) Date: Wed, 30 Oct 2024 12:03:12 -0400 Subject: [petsc-users] Problem running geoclaw example with petsc 3.22 In-Reply-To: References: <1C63D19F-6EBC-43A3-8B40-65EA34991AFA@gmail.com> <0136027D-C2CE-47D0-AF97-ACDE1DA0C382@gmail.com> <4726A341-54DB-4592-BBAE-843AF271F28E@petsc.dev> <27B47E2D-00D5-4326-95A9-7595DFF0E952@gmail.com> <490B5A4A-BBD8-490A-91FD-7CEC9E01A99B@petsc.dev> <96674E24-9EDA-42DE-8459-81D46A260CDE@petsc.dev> Message-ID: Please put -mpi_linear_solver_server_use_shared_memory false into the petscMPIOptions file and see if that changes anything. Barry > On Oct 30, 2024, at 9:05?AM, Praveen C wrote: > > I have attached some files for this > > cd bouss > . setenv.sh # some settings in this file may need to be changed > > cd radial_flat/1d_radial > make .output > cd .. > make .output > > Thanks > praveen > > > > >> On 30 Oct 2024, at 6:28?PM, Barry Smith wrote: >> >> >> Please send me exact instructions on how you are running the geoclaw example, when I run >> >> geoclaw/examples/bouss/radial_flat >> >> >> it only runs with one level, but never errors, I changed setrun.py to amrdata.amr_levels_max = 5 but it made no difference. >> >> Thanks >> >> Barry >> >> >> >>> On Oct 24, 2024, at 12:17?PM, Praveen C wrote: >>> >>> Hello Barry >>> >>> I use this script to install clawpack and required dependencies >>> >>> https://urldefense.us/v3/__https://github.com/cpraveen/cfdlab/blob/master/bin/clawpack.sh__;!!G_uCfscf7eWS!Yu4Xq_TulaXyaW5-F8HQsKBbyqOIu4yt2cGboLDLOsJd6JDUdZg7ZDD6qBhQ61zQH71h_nhuxTPKGswlUpmus7o$ >>> >>> See lines 125-132 >>> >>> You can use it like this >>> >>> export CLAW=/path/to/where/you/want/clawpack >>> bash clawpack.sh v5.11.0 >>> >>> This will git pull clawpack and creates a conda env called ?claw? and installs inside that. >>> >>> I have not specified petsc version in this script, but latest miniforge should install petsc at 3.22 >>> >>> Thank you >>> praveen >>> >>>> On 24 Oct 2024, at 9:37?PM, Barry Smith wrote: >>>> >>>> >>>> Good, some helpful information with the runs you made. >>>> >>>> In the crash below it made more progress it was able to allocate multiple regions of shared memory and access them. I don't know why it would crash later. >>>> >>>> Can you tell me all the steps with miniforge (which seems to be related to the failure) you use? I've never used miniforge. >>>> >>>> If I can get to an environment that reproduces the problem I can debug it and fix it. >>>> >>>> Barry >>>> >>>> >>>>> On Oct 24, 2024, at 11:49?AM, Praveen C wrote: >>>>> >>>>> I get this >>>>> >>>>> $ mpiexec -n 3 ./ex89f -n 20 -mpi_linear_solver_server -mpi_linear_solver_server -mpi_linear_solver_server_ksp_view -ksp_monitor -ksp_converged_reason -ksp_view -mpi_linear_solver_server_minimum_count_per_rank 5 >>>>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>>>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range >>>>> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >>>>> [0]PETSC ERROR: or see https://urldefense.us/v3/__https://petsc.org/release/faq/*valgrind__;Iw!!G_uCfscf7eWS!Yu4Xq_TulaXyaW5-F8HQsKBbyqOIu4yt2cGboLDLOsJd6JDUdZg7ZDD6qBhQ61zQH71h_nhuxTPKGswlQZYihyk$ and https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!Yu4Xq_TulaXyaW5-F8HQsKBbyqOIu4yt2cGboLDLOsJd6JDUdZg7ZDD6qBhQ61zQH71h_nhuxTPKGswlf-KHakI$ >>>>> [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run >>>>> [0]PETSC ERROR: to get more information on the crash. >>>>> [0]PETSC ERROR: Run with -malloc_debug to check if memory corruption is causing the crash. >>>>> -------------------------------------------------------------------------- >>>>> MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_SELF >>>>> Proc: [[47380,1],0] >>>>> Errorcode: 59 >>>>> >>>>> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. >>>>> You may or may not see output from other processes, depending on >>>>> exactly when Open MPI kills them. >>>>> -------------------------------------------------------------------------- >>>>> -------------------------------------------------------------------------- >>>>> prterun has exited due to process rank 0 with PID 0 on node MacMiniHome calling >>>>> "abort". This may have caused other processes in the application to be >>>>> terminated by signals sent by prterun (as reported here). >>>>> ????????????????????????????????????? >>>>> >>>>> and I have this after the code exits >>>>> >>>>> $ ipcs -m >>>>> IPC status from as of Thu Oct 24 21:17:39 IST 2024 >>>>> T ID KEY MODE OWNER GROUP >>>>> Shared Memory: >>>>> m 1572864 0x0000000b --rw-rw-rw- praveen staff >>>>> m 524289 0x0000000c --rw-rw-rw- praveen staff >>>>> m 655362 0x0000000d --rw-rw-rw- praveen staff >>>>> m 262147 0x0000000e --rw-rw-rw- praveen staff >>>>> m 262148 0x0000000f --rw-rw-rw- praveen staff >>>>> m 393221 0x0000000a --rw-rw-rw- praveen staff >>>>> >>>>> This is with petsc installed with miniforge, which I also use with clawpack. With spack installed petsc, I can run the ex89f example. >>>>> >>>>> Thanks >>>>> praveen >>>>> >>>>>> On 24 Oct 2024, at 7:55?PM, Barry Smith wrote: >>>>>> >>>>>> >>>>>> Ok, super strange, I've run many times on my Mac >>>>>> >>>>>> Can you please try to remove that allocated memory with ipcrm and then >>>>>> >>>>>> cd $PETSC_DIR/src/ksp/ksp/tutorials >>>>>> make ex89f >>>>>> mpiexec -n 3 ./ex89f -n 20 -mpi_linear_solver_server -mpi_linear_solver_server -mpi_linear_solver_server_ksp_view -ksp_monitor -ksp_converged_reason -ksp_view -mpi_linear_solver_server_minimum_count_per_rank 5 >>>>>> >>>>>> This does the same thing as the GeoClaw code but is much simpler. >>>>>> >>>>>> Barry >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> On Oct 23, 2024, at 10:55?PM, Praveen C wrote: >>>>>>> >>>>>>> I get very similar error on my mac with >>>>>>> >>>>>>> $ gfortran -v >>>>>>> Using built-in specs. >>>>>>> COLLECT_GCC=gfortran >>>>>>> COLLECT_LTO_WRAPPER=/opt/homebrew/Caskroom/miniforge/base/envs/claw/libexec/gcc/arm64-apple-darwin20.0.0/13.2.0/lto-wrapper >>>>>>> Target: arm64-apple-darwin20.0.0 >>>>>>> Configured with: ../configure --prefix=/opt/homebrew/Caskroom/miniforge/base/envs/claw --build=x86_64-apple-darwin13.4.0 --host=arm64-apple-darwin20.0.0 --target=arm64-apple-darwin20.0.0 --with-libiconv-prefix=/opt/homebrew/Caskroom/miniforge/base/envs/claw --enable-languages=fortran --disable-multilib --enable-checking=release --disable-bootstrap --disable-libssp --with-gmp=/opt/homebrew/Caskroom/miniforge/base/envs/claw --with-mpfr=/opt/homebrew/Caskroom/miniforge/base/envs/claw --with-mpc=/opt/homebrew/Caskroom/miniforge/base/envs/claw --with-isl=/opt/homebrew/Caskroom/miniforge/base/envs/claw --enable-darwin-at-rpath >>>>>>> Thread model: posix >>>>>>> Supported LTO compression algorithms: zlib >>>>>>> gcc version 13.2.0 (GCC) >>>>>>> >>>>>>> Before starting >>>>>>> >>>>>>> $ ipcs -m >>>>>>> IPC status from as of Thu Oct 24 08:02:11 IST 2024 >>>>>>> T ID KEY MODE OWNER GROUP >>>>>>> Shared Memory: >>>>>>> >>>>>>> and when I run the code >>>>>>> >>>>>>> Using a PETSc solver >>>>>>> Using Bouss equations from the start >>>>>>> rnode allocated... >>>>>>> node allocated... >>>>>>> listOfGrids allocated... >>>>>>> Storage allocated... >>>>>>> bndList allocated... >>>>>>> Gridding level 1 at t = 0.000000E+00: 4 grids with 10000 cells >>>>>>> Setting initial dt to 2.9999999999999999E-002 >>>>>>> max threads set to 1 >>>>>>> >>>>>>> Done reading data, starting computation ... >>>>>>> >>>>>>> Total zeta at initial time: 39269.907650665169 >>>>>>> GEOCLAW: Frame 0 output files done at time t = 0.000000D+00 >>>>>>> >>>>>>> [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >>>>>>> [0]PETSC ERROR: Petsc has generated inconsistent data >>>>>>> [0]PETSC ERROR: Unable to locate PCMPI allocated shared address 0x130698000 >>>>>>> [0]PETSC ERROR: WARNING! There are unused option(s) set! Could be the program crashed before usage or a spelling mistake, etc! >>>>>>> [0]PETSC ERROR: Option left: name:-ksp_type value: preonly source: file >>>>>>> [0]PETSC ERROR: Option left: name:-mpi_ksp_max_it value: 200 source: file >>>>>>> [0]PETSC ERROR: Option left: name:-mpi_ksp_reuse_preconditioner (no value) source: file >>>>>>> [0]PETSC ERROR: Option left: name:-mpi_ksp_rtol value: 1.e-9 source: file >>>>>>> [0]PETSC ERROR: Option left: name:-mpi_ksp_type value: gmres source: file >>>>>>> [0]PETSC ERROR: Option left: name:-mpi_linear_solver_server_view (no value) source: file >>>>>>> [0]PETSC ERROR: Option left: name:-mpi_pc_gamg_sym_graph value: true source: file >>>>>>> [0]PETSC ERROR: Option left: name:-mpi_pc_gamg_symmetrize_graph value: true source: file >>>>>>> [0]PETSC ERROR: Option left: name:-mpi_pc_type value: gamg source: file >>>>>>> [0]PETSC ERROR: Option left: name:-pc_mpi_minimum_count_per_rank value: 5000 source: file >>>>>>> [0]PETSC ERROR: Option left: name:-pc_type value: mpi source: file >>>>>>> [0]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!Yu4Xq_TulaXyaW5-F8HQsKBbyqOIu4yt2cGboLDLOsJd6JDUdZg7ZDD6qBhQ61zQH71h_nhuxTPKGswlf-KHakI$ for trouble shooting. >>>>>>> [0]PETSC ERROR: Petsc Release Version 3.22.0, Sep 28, 2024 >>>>>>> [0]PETSC ERROR: /Users/praveen/work/bouss/radial_flat/xgeoclaw with 6 MPI process(es) and PETSC_ARCH on MacMiniHome.local by praveen Thu Oct 24 08:04:27 2024 >>>>>>> [0]PETSC ERROR: Configure options: AR=arm64-apple-darwin20.0.0-ar CC=mpicc CXX=mpicxx FC=mpifort CFLAGS="-ftree-vectorize -fPIC -fstack-protector-strong -O2 -pipe -isystem /opt/homebrew/Caskroom/miniforge/base/envs/claw/include " CPPFLAGS="-D_FORTIFY_SOURCE=2 -isystem /opt/homebrew/Caskroom/miniforge/base/envs/claw/include -mmacosx-version-min=11.0 -mmacosx-version-min=11.0" CXXFLAGS="-ftree-vectorize -fPIC -fstack-protector-strong -O2 -pipe -stdlib=libc++ -fvisibility-inlines-hidden -fmessage-length=0 -isystem /opt/homebrew/Caskroom/miniforge/base/envs/claw/include " FFLAGS="-march=armv8.3-a -ftree-vectorize -fPIC -fno-stack-protector -O2 -pipe -isystem /opt/homebrew/Caskroom/miniforge/base/envs/claw/include " LDFLAGS="-Wl,-headerpad_max_install_names -Wl,-dead_strip_dylibs -Wl,-rpath,/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib -L/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib" LIBS="-Wl,-rpath,/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib -lmpi_mpifh -lgfortran" --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 --FOPTFLAGS=-O3 --with-clib-autodetect=0 --with-cxxlib-autodetect=0 --with-fortranlib-autodetect=0 --with-debugging=0 --with-blas-lib=libblas.dylib --with-lapack-lib=liblapack.dylib --with-yaml=1 --with-hdf5=1 --with-fftw=1 --with-hwloc=0 --with-hypre=1 --with-metis=1 --with-mpi=1 --with-mumps=1 --with-parmetis=1 --with-pthread=1 --with-ptscotch=1 --with-shared-libraries --with-ssl=0 --with-scalapack=1 --with-superlu=1 --with-superlu_dist=1 --with-superlu_dist-include=/opt/homebrew/Caskroom/miniforge/base/envs/claw/include/superlu-dist --with-superlu_dist-lib=-lsuperlu_dist --with-suitesparse=1 --with-suitesparse-dir=/opt/homebrew/Caskroom/miniforge/base/envs/claw --with-x=0 --with-scalar-type=real --with-cuda=0 --with-batch --prefix=/opt/homebrew/Caskroom/miniforge/base/envs/claw >>>>>>> [0]PETSC ERROR: #1 PetscShmgetMapAddresses() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/sys/utils/server.c:114 >>>>>>> [0]PETSC ERROR: #2 PCMPISetMat() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/ksp/pc/impls/mpi/pcmpi.c:269 >>>>>>> [0]PETSC ERROR: #3 PCSetUp_MPI() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/ksp/pc/impls/mpi/pcmpi.c:853 >>>>>>> [0]PETSC ERROR: #4 PCSetUp() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/ksp/pc/interface/precon.c:1071 >>>>>>> [0]PETSC ERROR: #5 KSPSetUp() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/ksp/ksp/interface/itfunc.c:415 >>>>>>> [0]PETSC ERROR: #6 KSPSolve_Private() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/ksp/ksp/interface/itfunc.c:826 >>>>>>> [0]PETSC ERROR: #7 KSPSolve() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/ksp/ksp/interface/itfunc.c:1075 >>>>>>> >>>>>>> Code does not progress and I kill it >>>>>>> >>>>>>> ^CTraceback (most recent call last): >>>>>>> File "/Users/praveen/Applications/clawpack/clawutil/src/python/clawutil/runclaw.py", line 341, in >>>>>>> runclaw(*args) >>>>>>> File "/Users/praveen/Applications/clawpack/clawutil/src/python/clawutil/runclaw.py", line 242, in runclaw >>>>>>> proc = subprocess.check_call(cmd_split, >>>>>>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ >>>>>>> File "/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib/python3.12/subprocess.py", line 408, in check_call >>>>>>> retcode = call(*popenargs, **kwargs) >>>>>>> ^^^^^^^^^^^^^^^^^^^^^^^^^^ >>>>>>> File "/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib/python3.12/subprocess.py", line 391, in call >>>>>>> return p.wait(timeout=timeout) >>>>>>> ^^^^^^^^^^^^^^^^^^^^^^^ >>>>>>> File "/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib/python3.12/subprocess.py", line 1264, in wait >>>>>>> return self._wait(timeout=timeout) >>>>>>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^ >>>>>>> File "/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib/python3.12/subprocess.py", line 2053, in _wait >>>>>>> (pid, sts) = self._try_wait(0) >>>>>>> ^^^^^^^^^^^^^^^^^ >>>>>>> File "/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib/python3.12/subprocess.py", line 2011, in _try_wait >>>>>>> (pid, sts) = os.waitpid(self.pid, wait_flags) >>>>>>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ >>>>>>> KeyboardInterrupt >>>>>>> make[1]: *** [output] Interrupt: 2 >>>>>>> make: *** [.output] Interrupt: 2 >>>>>>> >>>>>>> Now it says >>>>>>> >>>>>>> $ ipcs -m >>>>>>> IPC status from as of Thu Oct 24 08:05:06 IST 2024 >>>>>>> T ID KEY MODE OWNER GROUP >>>>>>> Shared Memory: >>>>>>> m 720896 0x0000000a --rw-rw-rw- praveen staff >>>>>>> >>>>>>> Thanks >>>>>>> praveen >>>>>>> >>>>>>>> On 23 Oct 2024, at 8:26?PM, Barry Smith wrote: >>>>>>>> >>>>>>>> >>>>>>>> Hmm, so it is creating the first shared memory region with ID of 10 (A in hex) puts it in a linked list in PETSc but then when it tries to find it in the linked list it cannot find it. >>>>>>>> >>>>>>>> I don't know how to reproduce this or debug it remotely. >>>>>>>> >>>>>>>> Can you build on a completely different machine or with completely different compilers? >>>>>>>> >>>>>>>> Barry >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> On Oct 23, 2024, at 10:31?AM, Praveen C wrote: >>>>>>>>> >>>>>>>>> I get same error and now it shows >>>>>>>>> >>>>>>>>> $ ipcs -m >>>>>>>>> >>>>>>>>> ------ Shared Memory Segments -------- >>>>>>>>> key shmid owner perms bytes nattch status >>>>>>>>> 0x0000000a 32788 praveen 666 240 6 >>>>>>>>> >>>>>>>>> Note that the code seems to be still running after printing those error message, but it is not printing any progress which it should do. >>>>>>>>> >>>>>>>>> Thanks >>>>>>>>> praveen >>>>>>>>> >>>>>>>>>> On 23 Oct 2024, at 7:56?PM, Barry Smith wrote: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Try >>>>>>>>>> >>>>>>>>>> ipcrm -m 11 >>>>>>>>>> >>>>>>>>>> ipcs -m >>>>>>>>>> >>>>>>>>>> Try running the program again >>>>>>>>>> >>>>>>>>>> If failed check >>>>>>>>>> >>>>>>>>>> ipcs -m >>>>>>>>>> >>>>>>>>>> again >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> On Oct 23, 2024, at 10:20?AM, Praveen C wrote: >>>>>>>>>>> >>>>>>>>>>> Hello Barry >>>>>>>>>>> >>>>>>>>>>> I see this >>>>>>>>>>> >>>>>>>>>>> $ ipcs -m >>>>>>>>>>> >>>>>>>>>>> ------ Shared Memory Segments -------- >>>>>>>>>>> key shmid owner perms bytes nattch status >>>>>>>>>>> 0x0000000a 11 praveen 666 240 6 >>>>>>>>>>> >>>>>>>>>>> and I am observing same error as below. >>>>>>>>>>> >>>>>>>>>>> Thanks >>>>>>>>>>> praveen >>>>>>>>>>> >>>>>>>>>>>> On 23 Oct 2024, at 7:08?PM, Barry Smith wrote: >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Please take a look at the notes in https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/PetscShmgetAllocateArray/__;!!G_uCfscf7eWS!Yu4Xq_TulaXyaW5-F8HQsKBbyqOIu4yt2cGboLDLOsJd6JDUdZg7ZDD6qBhQ61zQH71h_nhuxTPKGswlOQeAPuk$ For some reason your program is not able to access/use the Unix shared memory; check if you are already using the shared memory (so it is not available for a new run) or the limits are too low to access enough memory. >>>>>>>>>>>> >>>>>>>>>>>> Barry >>>>>>>>>>>> >>>>>>>>>>>>> On Oct 23, 2024, at 8:23?AM, Praveen C wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> Dear all >>>>>>>>>>>>> >>>>>>>>>>>>> I am not able to run the boussinesq example from geoclaw using petsc at 3.22.0 >>>>>>>>>>>>> >>>>>>>>>>>>> https://urldefense.us/v3/__https://github.com/clawpack/geoclaw/tree/3303883f46572c58130d161986b8a87a57ca7816/examples/bouss__;!!G_uCfscf7eWS!e3VQ4NHKmXGstRsQW5vtI7fmKfUT9zmJkMJcPbcvPyIjicyfJpNoMgx3wZ-qyGcKNSjIkNZkzilec8MnHN6PMw$ >>>>>>>>>>>>> >>>>>>>>>>>>> It runs with petsc at 3.21.6 >>>>>>>>>>>>> >>>>>>>>>>>>> The error I get is given below. After printing this, the code does not progress. >>>>>>>>>>>>> >>>>>>>>>>>>> I use the following petsc options >>>>>>>>>>>>> >>>>>>>>>>>>> # set min numbers of matrix rows per MPI rank (default is 10000) >>>>>>>>>>>>> -mpi_linear_solve_minimum_count_per_rank 5000 >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> # Krylov linear solver: >>>>>>>>>>>>> -mpi_linear_solver_server >>>>>>>>>>>>> -mpi_linear_solver_server_view >>>>>>>>>>>>> -ksp_type gmres >>>>>>>>>>>>> -ksp_max_it 200 >>>>>>>>>>>>> -ksp_reuse_preconditioner >>>>>>>>>>>>> -ksp_rtol 1.e-9 >>>>>>>>>>>>> >>>>>>>>>>>>> # preconditioner: >>>>>>>>>>>>> -pc_type gamg >>>>>>>>>>>>> >>>>>>>>>>>>> I installed petsc and other dependencies for clawpack using miniforge. >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks >>>>>>>>>>>>> pc >>>>>>>>>>>>> >>>>>>>>>>>>> ==> Use Bouss. in water deeper than 1.0000000000000000 Using a PETSc solver >>>>>>>>>>>>> Using Bouss equations from the start >>>>>>>>>>>>> rnode allocated... >>>>>>>>>>>>> node allocated... >>>>>>>>>>>>> listOfGrids allocated... >>>>>>>>>>>>> Storage allocated... >>>>>>>>>>>>> bndList allocated... >>>>>>>>>>>>> Gridding level 1 at t = 0.000000E+00: 4 grids with 10000 cells >>>>>>>>>>>>> Setting initial dt to 2.9999999999999999E-002 >>>>>>>>>>>>> max threads set to 6 >>>>>>>>>>>>> Done reading data, starting computation ... Total zeta at initial time: 39269.907650665169 GEOCLAW: Frame 0 output files done at time t = 0.000000D+00 >>>>>>>>>>>>> >>>>>>>>>>>>> [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >>>>>>>>>>>>> [0]PETSC ERROR: Petsc has generated inconsistent data >>>>>>>>>>>>> [0]PETSC ERROR: Unable to locate PCMPI allocated shared address 0x55e6d750ae20 >>>>>>>>>>>>> [0]PETSC ERROR: WARNING! There are unused option(s) set! Could be the program crashed before usage or a spelling mistake, etc! >>>>>>>>>>>>> [0]PETSC ERROR: Option left: name:-ksp_max_it value: 200 source: file >>>>>>>>>>>>> [0]PETSC ERROR: Option left: name:-ksp_reuse_preconditioner (no value) source: file >>>>>>>>>>>>> [0]PETSC ERROR: Option left: name:-ksp_rtol value: 1.e-9 source: file >>>>>>>>>>>>> [0]PETSC ERROR: Option left: name:-ksp_type value: gmres source: file >>>>>>>>>>>>> [0]PETSC ERROR: Option left: name:-mpi_linear_solve_minimum_count_per_rank value: 5000 source: file >>>>>>>>>>>>> [0]PETSC ERROR: Option left: name:-mpi_linear_solver_server_view (no value) source: file >>>>>>>>>>>>> [0]PETSC ERROR: Option left: name:-pc_type value: gamg source: file >>>>>>>>>>>>> [0]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!e3VQ4NHKmXGstRsQW5vtI7fmKfUT9zmJkMJcPbcvPyIjicyfJpNoMgx3wZ-qyGcKNSjIkNZkzilec8MvNjNo7A$ for trouble shooting. >>>>>>>>>>>>> [0]PETSC ERROR: Petsc Release Version 3.22.0, Sep 28, 2024 [0]PETSC ERROR: /home/praveen/bouss/radial_flat/xgeoclaw with 6 MPI process(es) and PETSC_ARCH on euler by praveen Thu Oct 17 21:49:54 2024 >>>>>>>>>>>>> [0]PETSC ERROR: Configure options: AR=${PREFIX}/bin/x86_64-conda-linux-gnu-ar CC=mpicc CXX=mpicxx FC=mpifort CFLAGS="-march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /opt/miniforge/envs/claw/include " CPPFLAGS="-DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /opt/miniforge/envs/claw/include" CXXFLAGS="-fvisibility-inlines-hidden -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /opt/miniforge/envs/claw/include " FFLAGS="-march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /opt/miniforge/envs/claw/include -Wl,--no-as-needed" LDFLAGS="-pthread -fopenmp -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,--allow-shlib-undefined -Wl,-rpath,/opt/miniforge/envs/claw/lib -Wl,-rpath-link,/opt/miniforge/envs/claw/lib -L/opt/miniforge/envs/claw/lib -Wl,-rpath-link,/opt/miniforge/envs/claw/lib" LIBS="-Wl,-rpath,/opt/miniforge/envs/claw/lib -lmpi_mpifh -lgfortran" --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 --FOPTFLAGS=-O3 --with-clib-autodetect=0 --with-cxxlib-autodetect=0 --with-fortranlib-autodetect=0 --with-debugging=0 --with-blas-lib=libblas.so --with-lapack-lib=liblapack.so --with-yaml=1 --with-hdf5=1 --with-fftw=1 --with-hwloc=0 --with-hypre=1 --with-metis=1 --with-mpi=1 --with-mumps=1 --with-parmetis=1 --with-pthread=1 --with-ptscotch=1 --with-shared-libraries --with-ssl=0 --with-scalapack=1 --with-superlu=1 --with-superlu_dist=1 --with-superlu_dist-include=/opt/miniforge/envs/claw/include/superlu-dist --with-superlu_dist-lib=-lsuperlu_dist --with-suitesparse=1 --with-suitesparse-dir=/opt/miniforge/envs/claw --with-x=0 --with-scalar-type=real --with-cuda=0 --prefix=/opt/miniforge/envs/claw >>>>>>>>>>>>> [0]PETSC ERROR: #1 PetscShmgetMapAddresses() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/sys/utils/server.c:114 >>>>>>>>>>>>> [0]PETSC ERROR: #2 PCMPISetMat() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/pc/impls/mpi/pcmpi.c:269 >>>>>>>>>>>>> [0]PETSC ERROR: #3 PCSetUp_MPI() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/pc/impls/mpi/pcmpi.c:853 >>>>>>>>>>>>> [0]PETSC ERROR: #4 PCSetUp() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/pc/interface/precon.c:1071 >>>>>>>>>>>>> [0]PETSC ERROR: #5 KSPSetUp() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/ksp/interface/itfunc.c:415 >>>>>>>>>>>>> [0]PETSC ERROR: #6 KSPSolve_Private() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/ksp/interface/itfunc.c:826 >>>>>>>>>>>>> [0]PETSC ERROR: #7 KSPSolve() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/ksp/interface/itfunc.c:1075 >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mail2amneet at gmail.com Wed Oct 30 12:48:58 2024 From: mail2amneet at gmail.com (Amneet Bhalla) Date: Wed, 30 Oct 2024 10:48:58 -0700 Subject: [petsc-users] Rigid body nullspace for Stokes operator In-Reply-To: References: <875xpas35p.fsf@jedbrown.org> Message-ID: I think the nullspace for the velocity operator is of the form vnull = U + ? ? r in which U is a rigid body velocity and ? is the rigid body rotational velocity, and r is the radius vector from the center of mass. I believe I need to construct 6 nullspace vectors in 3D and 3 nullspace vectors in 2D. Sounds correct? Also does the center of mass coordinates matter when defining r? On Wed, Oct 30, 2024 at 7:53?AM Amneet Bhalla wrote: > @Mark: Is there some document/paper that I can follow to check the algebra > of these zero eigenvectors/null space modes? > > @Jed : We use a projection method preconditioner to solve the coupled > velocity pressure system as described here ( > https://urldefense.us/v3/__https://www.sciencedirect.com/science/article/pii/S0021999123004205__;!!G_uCfscf7eWS!b_U_d3oTZckny15inoAStJf_DkxTjYjj_yN7UJJPIs82UiIP-TYpBuEhMzrLsOWU6-Zv4dvrTgU9VJWe_WVwhigemQ$ ). It > is an approximation of the Schur complement. As a part of projection > preconditioner, we need to solve just the momentum equation separately > without considering the pressure part. > > On Tue, Oct 29, 2024 at 8:03?PM Jed Brown wrote: > >> And to be clear, we recommend using fieldsplit Schur to separate the >> pressure and velocity part (there are many examples). Applying AMG directly >> to the saddle point problem will not be a good solver because the >> heuristics assume positivity and do not preserve inf-sup stability (nor do >> standard smoothers). >> >> Mark Adams writes: >> >> > This is linear elasticity and there are 6 "null" vectors (if you removed >> > Dirichlet boundary conditions these are eigenvectors with zero >> eigenvalue): >> > 3 translations, x, y, z, and three rotatiions xx, yy ,zz. >> > x = (1,0,0,1,0,0,1,0 ...) >> > and xx is something like (0, z_1, -y_1, 0, z_2, -y_2, ...) where z_1 is >> the >> > z coordinate of the first vertex, etc. >> > >> > Mark >> > >> > On Tue, Oct 29, 2024 at 3:47?PM Amneet Bhalla >> wrote: >> > >> >> Hi Mark, >> >> >> >> Thanks! I am not sure how to construct null space and zero energy modes >> >> manually for this operator. Is there some theory or documentation I can >> >> follow to figure out what the null space and zero energy modes look >> like >> >> for this operator? Once I know what these are in symbolic form, I >> think I >> >> should be able to construct them manually. >> >> >> >> Best, >> >> --Amneet >> >> >> >> On Tue, Oct 29, 2024 at 7:35?AM Mark Adams wrote: >> >> >> >>> Oh my mistake. You are using staggered grids. So you don't have a >> block >> >>> size that hypre would use for the "nodal" methods. >> >>> I'm not sure what you are doing exactly, but try hypre and you could >> >>> create the null space, zero energy modes, manually, attach to the >> matrix >> >>> and try GAMG. >> >>> You can run with '-info :pc' and grep on GAMG to see if GAMG is >> picking >> >>> the null space up (send this output if you can't figure it out). >> >>> >> >>> Thanks, >> >>> Mark >> >>> >> >>> On Tue, Oct 29, 2024 at 9:28?AM Mark Adams wrote: >> >>> >> >>>> This coordinate interface is just a shortcut for vertex based >> >>>> discretizations with 3 dof per vertex, etc. (maybe works in 2D). >> >>>> You will need to construct the null space vectors manually and >> attach it >> >>>> to the matrix. Used by GAMG. >> >>>> >> >>>> Note, for hypre you want to use the "nodal" options and it does not >> use >> >>>> these null space vectors. That is probably the way you want to go. >> >>>> eg: -pc_hypre_boomeramg_nodal_coarsen >> >>>> >> >>>> I would run with hypre boomerang and -help and grep on nodal to see >> all >> >>>> the "nodal" options and use them. >> >>>> >> >>>> Thanks, >> >>>> Mark >> >>>> >> >>>> >> >>>> On Mon, Oct 28, 2024 at 8:06?PM Amneet Bhalla > > >> >>>> wrote: >> >>>> >> >>>>> Hi Folks, >> >>>>> >> >>>>> I am trying to solve the momentum equation in a projection >> >>>>> preconditioner using GAMG or Hypre solver. The equation looks like >> for >> >>>>> velocity variable *v* looks like: >> >>>>> >> >>>>> >> >>>>> [image: Screenshot 2024-10-28 at 4.15.17 PM.png] >> >>>>> >> >>>>> Here, ? is spatially varying dynamic viscosity and ? is spatially >> >>>>> varying bulk viscosity. I understand that I need to specify rigid >> body >> >>>>> nullspace modes to the multigrid solver in order to accelerate its >> >>>>> convergence. Looking into this routine >> MatNullSpaceCreateRigidBody() ( >> >>>>> >> https://urldefense.us/v3/__https://petsc.org/release/manualpages/Mat/MatNullSpaceCreateRigidBody/__;!!G_uCfscf7eWS!bVR6duCoDqPhZrWS-sm1c5qxsFPjZMhdT86AqLpPzWgVy5qoRhd4_Jue2LJOIS6LRrtV2cHGrqger1Yvb-Y5f-0$ >> >>>>> < >> https://urldefense.us/v3/__https://petsc.org/release/manualpages/Mat/MatNullSpaceCreateRigidBody/__;!!G_uCfscf7eWS!eKqgIJjCdMzIU76f7X65AmGxrU_-lC7W02BMWafJ77DNf_IuQk6O1X3qU1x9Ez8NJ20vZEL-mF6T1yNmDnwv0eWa2w$ >> >), >> >>>>> I see that I need to provide the coordinates of each node. I am >> using >> >>>>> staggered grid discretization. Do I need to provide coordinates of >> >>>>> staggered grid locations? >> >>>>> >> >>>>> Thanks, >> >>>>> -- >> >>>>> --Amneet >> >>>>> >> >>>>> >> >>>>> >> >>>>> >> >> >> >> -- >> >> --Amneet >> >> >> >> >> >> >> >> >> > > > -- > --Amneet > > > > -- --Amneet -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Wed Oct 30 14:25:51 2024 From: mfadams at lbl.gov (Mark Adams) Date: Wed, 30 Oct 2024 15:25:51 -0400 Subject: [petsc-users] Rigid body nullspace for Stokes operator In-Reply-To: References: <875xpas35p.fsf@jedbrown.org> Message-ID: On Wed, Oct 30, 2024 at 1:49?PM Amneet Bhalla wrote: > I think the nullspace for the velocity operator is of the form > > vnull = U + ? ? r > in which U is a rigid body velocity and ? is the rigid body rotational > velocity, and r is the radius vector from the center of mass. I believe I > need to construct 6 nullspace vectors in 3D and 3 nullspace vectors in 2D. > Sounds correct? > yes > Also does the center of mass coordinates matter when defining r? > no, but this would be good for conditioning, which I've never seen being an issue. You can just coordinate as r. That is the origin is your "center of mass". > > On Wed, Oct 30, 2024 at 7:53?AM Amneet Bhalla > wrote: > >> @Mark: Is there some document/paper that I can follow to check the >> algebra of these zero eigenvectors/null space modes? >> >> @Jed : We use a projection method preconditioner to solve the coupled >> velocity pressure system as described here ( >> https://urldefense.us/v3/__https://www.sciencedirect.com/science/article/pii/S0021999123004205__;!!G_uCfscf7eWS!dQPSeiutdub0FrPQ-gswdIQg7lOPRtrx-WocvHMu2ULBMuYLTKOxD_tE709brtLs8GY5uJVTOApLX6lrX3ZWoA8$ ). It >> is an approximation of the Schur complement. As a part of projection >> preconditioner, we need to solve just the momentum equation separately >> without considering the pressure part. >> >> On Tue, Oct 29, 2024 at 8:03?PM Jed Brown wrote: >> >>> And to be clear, we recommend using fieldsplit Schur to separate the >>> pressure and velocity part (there are many examples). Applying AMG directly >>> to the saddle point problem will not be a good solver because the >>> heuristics assume positivity and do not preserve inf-sup stability (nor do >>> standard smoothers). >>> >>> Mark Adams writes: >>> >>> > This is linear elasticity and there are 6 "null" vectors (if you >>> removed >>> > Dirichlet boundary conditions these are eigenvectors with zero >>> eigenvalue): >>> > 3 translations, x, y, z, and three rotatiions xx, yy ,zz. >>> > x = (1,0,0,1,0,0,1,0 ...) >>> > and xx is something like (0, z_1, -y_1, 0, z_2, -y_2, ...) where z_1 >>> is the >>> > z coordinate of the first vertex, etc. >>> > >>> > Mark >>> > >>> > On Tue, Oct 29, 2024 at 3:47?PM Amneet Bhalla >>> wrote: >>> > >>> >> Hi Mark, >>> >> >>> >> Thanks! I am not sure how to construct null space and zero energy >>> modes >>> >> manually for this operator. Is there some theory or documentation I >>> can >>> >> follow to figure out what the null space and zero energy modes look >>> like >>> >> for this operator? Once I know what these are in symbolic form, I >>> think I >>> >> should be able to construct them manually. >>> >> >>> >> Best, >>> >> --Amneet >>> >> >>> >> On Tue, Oct 29, 2024 at 7:35?AM Mark Adams wrote: >>> >> >>> >>> Oh my mistake. You are using staggered grids. So you don't have a >>> block >>> >>> size that hypre would use for the "nodal" methods. >>> >>> I'm not sure what you are doing exactly, but try hypre and you could >>> >>> create the null space, zero energy modes, manually, attach to the >>> matrix >>> >>> and try GAMG. >>> >>> You can run with '-info :pc' and grep on GAMG to see if GAMG is >>> picking >>> >>> the null space up (send this output if you can't figure it out). >>> >>> >>> >>> Thanks, >>> >>> Mark >>> >>> >>> >>> On Tue, Oct 29, 2024 at 9:28?AM Mark Adams wrote: >>> >>> >>> >>>> This coordinate interface is just a shortcut for vertex based >>> >>>> discretizations with 3 dof per vertex, etc. (maybe works in 2D). >>> >>>> You will need to construct the null space vectors manually and >>> attach it >>> >>>> to the matrix. Used by GAMG. >>> >>>> >>> >>>> Note, for hypre you want to use the "nodal" options and it does not >>> use >>> >>>> these null space vectors. That is probably the way you want to go. >>> >>>> eg: -pc_hypre_boomeramg_nodal_coarsen >>> >>>> >>> >>>> I would run with hypre boomerang and -help and grep on nodal to see >>> all >>> >>>> the "nodal" options and use them. >>> >>>> >>> >>>> Thanks, >>> >>>> Mark >>> >>>> >>> >>>> >>> >>>> On Mon, Oct 28, 2024 at 8:06?PM Amneet Bhalla < >>> mail2amneet at gmail.com> >>> >>>> wrote: >>> >>>> >>> >>>>> Hi Folks, >>> >>>>> >>> >>>>> I am trying to solve the momentum equation in a projection >>> >>>>> preconditioner using GAMG or Hypre solver. The equation looks like >>> for >>> >>>>> velocity variable *v* looks like: >>> >>>>> >>> >>>>> >>> >>>>> [image: Screenshot 2024-10-28 at 4.15.17 PM.png] >>> >>>>> >>> >>>>> Here, ? is spatially varying dynamic viscosity and ? is spatially >>> >>>>> varying bulk viscosity. I understand that I need to specify rigid >>> body >>> >>>>> nullspace modes to the multigrid solver in order to accelerate its >>> >>>>> convergence. Looking into this routine >>> MatNullSpaceCreateRigidBody() ( >>> >>>>> >>> https://urldefense.us/v3/__https://petsc.org/release/manualpages/Mat/MatNullSpaceCreateRigidBody/__;!!G_uCfscf7eWS!bVR6duCoDqPhZrWS-sm1c5qxsFPjZMhdT86AqLpPzWgVy5qoRhd4_Jue2LJOIS6LRrtV2cHGrqger1Yvb-Y5f-0$ >>> >>>>> < >>> https://urldefense.us/v3/__https://petsc.org/release/manualpages/Mat/MatNullSpaceCreateRigidBody/__;!!G_uCfscf7eWS!eKqgIJjCdMzIU76f7X65AmGxrU_-lC7W02BMWafJ77DNf_IuQk6O1X3qU1x9Ez8NJ20vZEL-mF6T1yNmDnwv0eWa2w$ >>> >), >>> >>>>> I see that I need to provide the coordinates of each node. I am >>> using >>> >>>>> staggered grid discretization. Do I need to provide coordinates of >>> >>>>> staggered grid locations? >>> >>>>> >>> >>>>> Thanks, >>> >>>>> -- >>> >>>>> --Amneet >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> >> >>> >> -- >>> >> --Amneet >>> >> >>> >> >>> >> >>> >> >>> >> >> >> -- >> --Amneet >> >> >> >> > > -- > --Amneet > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Wed Oct 30 14:28:25 2024 From: jed at jedbrown.org (Jed Brown) Date: Wed, 30 Oct 2024 13:28:25 -0600 Subject: [petsc-users] Rigid body nullspace for Stokes operator In-Reply-To: References: <875xpas35p.fsf@jedbrown.org> Message-ID: <87bjz1qtk6.fsf@jedbrown.org> Yes to 6 null vectors in 3D, 3 null vectors in 2D. The center of mass does not need to be identified because you can algebraically orthogonalize (lines 411-420 here). https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/blob/main/src/dm/impls/plex/plexfem.c?ref_type=heads*L377-425__;Iw!!G_uCfscf7eWS!bc0A2Vc5CGCuDQuOVZRyBUNTEbK3NZvHNmZUECNwUBwJPj0Pvv0CW2rGfEzeSxVXdWkjCE-D5jl2sonSQN8$ See also this implementation with raw coordinates. GAMG orthogonalizes within each aggregate (in a later phase of the algorithm) so global orthogonalization is not necessary. https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/blob/main/src/ksp/pc/impls/gamg/agg.c?ref_type=heads*L387__;Iw!!G_uCfscf7eWS!bc0A2Vc5CGCuDQuOVZRyBUNTEbK3NZvHNmZUECNwUBwJPj0Pvv0CW2rGfEzeSxVXdWkjCE-D5jl20dv2S0M$ Amneet Bhalla writes: > I think the nullspace for the velocity operator is of the form > > vnull = U + ? ? r > in which U is a rigid body velocity and ? is the rigid body rotational > velocity, and r is the radius vector from the center of mass. I believe I > need to construct 6 nullspace vectors in 3D and 3 nullspace vectors in 2D. > Sounds correct? Also does the center of mass coordinates matter when > defining r? > > On Wed, Oct 30, 2024 at 7:53?AM Amneet Bhalla wrote: > >> @Mark: Is there some document/paper that I can follow to check the algebra >> of these zero eigenvectors/null space modes? >> >> @Jed : We use a projection method preconditioner to solve the coupled >> velocity pressure system as described here ( >> https://urldefense.us/v3/__https://www.sciencedirect.com/science/article/pii/S0021999123004205__;!!G_uCfscf7eWS!bc0A2Vc5CGCuDQuOVZRyBUNTEbK3NZvHNmZUECNwUBwJPj0Pvv0CW2rGfEzeSxVXdWkjCE-D5jl2BQDymOU$ ). It >> is an approximation of the Schur complement. As a part of projection >> preconditioner, we need to solve just the momentum equation separately >> without considering the pressure part. >> >> On Tue, Oct 29, 2024 at 8:03?PM Jed Brown wrote: >> >>> And to be clear, we recommend using fieldsplit Schur to separate the >>> pressure and velocity part (there are many examples). Applying AMG directly >>> to the saddle point problem will not be a good solver because the >>> heuristics assume positivity and do not preserve inf-sup stability (nor do >>> standard smoothers). >>> >>> Mark Adams writes: >>> >>> > This is linear elasticity and there are 6 "null" vectors (if you removed >>> > Dirichlet boundary conditions these are eigenvectors with zero >>> eigenvalue): >>> > 3 translations, x, y, z, and three rotatiions xx, yy ,zz. >>> > x = (1,0,0,1,0,0,1,0 ...) >>> > and xx is something like (0, z_1, -y_1, 0, z_2, -y_2, ...) where z_1 is >>> the >>> > z coordinate of the first vertex, etc. >>> > >>> > Mark >>> > >>> > On Tue, Oct 29, 2024 at 3:47?PM Amneet Bhalla >>> wrote: >>> > >>> >> Hi Mark, >>> >> >>> >> Thanks! I am not sure how to construct null space and zero energy modes >>> >> manually for this operator. Is there some theory or documentation I can >>> >> follow to figure out what the null space and zero energy modes look >>> like >>> >> for this operator? Once I know what these are in symbolic form, I >>> think I >>> >> should be able to construct them manually. >>> >> >>> >> Best, >>> >> --Amneet >>> >> >>> >> On Tue, Oct 29, 2024 at 7:35?AM Mark Adams wrote: >>> >> >>> >>> Oh my mistake. You are using staggered grids. So you don't have a >>> block >>> >>> size that hypre would use for the "nodal" methods. >>> >>> I'm not sure what you are doing exactly, but try hypre and you could >>> >>> create the null space, zero energy modes, manually, attach to the >>> matrix >>> >>> and try GAMG. >>> >>> You can run with '-info :pc' and grep on GAMG to see if GAMG is >>> picking >>> >>> the null space up (send this output if you can't figure it out). >>> >>> >>> >>> Thanks, >>> >>> Mark >>> >>> >>> >>> On Tue, Oct 29, 2024 at 9:28?AM Mark Adams wrote: >>> >>> >>> >>>> This coordinate interface is just a shortcut for vertex based >>> >>>> discretizations with 3 dof per vertex, etc. (maybe works in 2D). >>> >>>> You will need to construct the null space vectors manually and >>> attach it >>> >>>> to the matrix. Used by GAMG. >>> >>>> >>> >>>> Note, for hypre you want to use the "nodal" options and it does not >>> use >>> >>>> these null space vectors. That is probably the way you want to go. >>> >>>> eg: -pc_hypre_boomeramg_nodal_coarsen >>> >>>> >>> >>>> I would run with hypre boomerang and -help and grep on nodal to see >>> all >>> >>>> the "nodal" options and use them. >>> >>>> >>> >>>> Thanks, >>> >>>> Mark >>> >>>> >>> >>>> >>> >>>> On Mon, Oct 28, 2024 at 8:06?PM Amneet Bhalla >> > >>> >>>> wrote: >>> >>>> >>> >>>>> Hi Folks, >>> >>>>> >>> >>>>> I am trying to solve the momentum equation in a projection >>> >>>>> preconditioner using GAMG or Hypre solver. The equation looks like >>> for >>> >>>>> velocity variable *v* looks like: >>> >>>>> >>> >>>>> >>> >>>>> [image: Screenshot 2024-10-28 at 4.15.17 PM.png] >>> >>>>> >>> >>>>> Here, ? is spatially varying dynamic viscosity and ? is spatially >>> >>>>> varying bulk viscosity. I understand that I need to specify rigid >>> body >>> >>>>> nullspace modes to the multigrid solver in order to accelerate its >>> >>>>> convergence. Looking into this routine >>> MatNullSpaceCreateRigidBody() ( >>> >>>>> >>> https://urldefense.us/v3/__https://petsc.org/release/manualpages/Mat/MatNullSpaceCreateRigidBody/__;!!G_uCfscf7eWS!bVR6duCoDqPhZrWS-sm1c5qxsFPjZMhdT86AqLpPzWgVy5qoRhd4_Jue2LJOIS6LRrtV2cHGrqger1Yvb-Y5f-0$ >>> >>>>> < >>> https://urldefense.us/v3/__https://petsc.org/release/manualpages/Mat/MatNullSpaceCreateRigidBody/__;!!G_uCfscf7eWS!eKqgIJjCdMzIU76f7X65AmGxrU_-lC7W02BMWafJ77DNf_IuQk6O1X3qU1x9Ez8NJ20vZEL-mF6T1yNmDnwv0eWa2w$ >>> >), >>> >>>>> I see that I need to provide the coordinates of each node. I am >>> using >>> >>>>> staggered grid discretization. Do I need to provide coordinates of >>> >>>>> staggered grid locations? >>> >>>>> >>> >>>>> Thanks, >>> >>>>> -- >>> >>>>> --Amneet >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> >> >>> >> -- >>> >> --Amneet >>> >> >>> >> >>> >> >>> >> >>> >> >> >> -- >> --Amneet >> >> >> >> > > -- > --Amneet From p.khurana22 at imperial.ac.uk Wed Oct 30 15:13:18 2024 From: p.khurana22 at imperial.ac.uk (Khurana, Parv) Date: Wed, 30 Oct 2024 20:13:18 +0000 Subject: [petsc-users] Expected weak scaling behaviour for AMG libraries? Message-ID: Hello PETSc Community, I am trying to understand the scaling behaviour of AMG methods in PETSc (Hypre for now) and how many DOFs/Rank are needed for a performant AMG solve. I?m currently conducting weak scaling tests using src/snes/tutorials/ex12.c in 3D, applying Dirichlet BCs with FEM at P=1. The tests keep DOFs per processor constant while increasing the mesh size and processor count, specifically: * 20000 and 80000 DOF/RANK configurations. * Running SNES twice, using GMRES with a tolerance of 1e-5 and preconditioning with Hypre-BoomerAMG. Unfortunately, parallel efficiency degrades noticeably with increased processor counts. Are there any insights or rules of thumb for using AMG more effectively? I have been looking at this issue for a while now and would love to engage in a further discussion. Please find below the weak scaling results and the options I use to run the tests. [cid:b3265a6e-baed-4fc1-a58d-0eebbb7de5a4] #Run type -run_type full -petscpartitioner_type simple #Mesh settings -dm_plex_dim 3 -dm_plex_simplex 1 -dm_refine 5 #Varied this -dm_plex_box_faces 6,6,6 #BCs and FEM space -bc_type dirichlet -petscspace_degree 1 #Solver settings -snes_max_it 2 -ksp_type gmres -ksp_rtol 1.0e-5 #Same settings as what we use for LOR -pc_type hypre -pc_hypre_type boomeramg -pc_hypre_boomeramg_coarsen_type hmis -pc_hypre_boomeramg_relax_type_all symmetric-sor/jacobi -pc_hypre_boomeramg_strong_threshold 0.7 -pc_hypre_boomeramg_interp_type ext+i -pc_hypre_boomeramg_P_max 2 -pc_hypre_boomeramg_truncfactor 0.3 Best, Parv -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image.png Type: image/png Size: 119488 bytes Desc: image.png URL: From knepley at gmail.com Wed Oct 30 22:02:14 2024 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 30 Oct 2024 23:02:14 -0400 Subject: [petsc-users] Expected weak scaling behaviour for AMG libraries? In-Reply-To: References: Message-ID: On Wed, Oct 30, 2024 at 4:13?PM Khurana, Parv wrote: > Hello PETSc Community, > I am trying to understand the scaling behaviour of AMG methods in PETSc > (Hypre for now) and how many DOFs/Rank are needed for a performant AMG > solve. > I?m currently conducting weak scaling tests using > src/snes/tutorials/ex12.c in 3D, applying Dirichlet BCs with FEM at P=1. > The tests keep DOFs per processor constant while increasing the mesh size > and processor count, specifically: > > - *20000 and 80000 DOF/RANK* configurations. > - Running SNES twice, using GMRES with a tolerance of 1e-5 and > preconditioning with Hypre-BoomerAMG. > > A couple of quick points in order to make sure that there is no confusion: 1) Partitioner type "simple" is for the CI. It is a very bad partition, and should not be used for timing. The default is ParMetis which should be good enough. 2) You start out with 6^3 = 216 elements, distribute that, and then refine it. This will be _really_ bad load balance on all arrangement except the divisors of 216. You usually want to start out with something bigger at the later stages. You can use -dm_refine_pre to refine before distribution. 3) It is not clear you are using the timing for just the solver (SNESSolve). It could be that extraneous things are taking time. When asking questions like this, please always send the output of -log_view for timing, and at least -ksp_monitor_true_residial for convergence. 4) SNES ex56 is the example we use for GAMG scalability testing Thanks, Matt > Unfortunately, parallel efficiency degrades noticeably with increased > processor counts. Are there any insights or rules of thumb for using AMG > more effectively? I have been looking at this issue for a while > now and would love to engage in a further discussion. Please find below the > weak scaling results and the options I use to run the tests. > *#Run type* > -run_type full > -petscpartitioner_type simple > > *#Mesh settings* > -dm_plex_dim 3 > -dm_plex_simplex 1 > -dm_refine 5 #Varied this > -dm_plex_box_faces 6,6,6 > > *#BCs and FEM space* > -bc_type dirichlet > -petscspace_degree 1 > > *#Solver settings* > -snes_max_it 2 > -ksp_type gmres > -ksp_rtol 1.0e-5 > #Same settings as what we use for LOR > -pc_type hypre > -pc_hypre_type boomeramg > -pc_hypre_boomeramg_coarsen_type hmis > -pc_hypre_boomeramg_relax_type_all symmetric-sor/jacobi > -pc_hypre_boomeramg_strong_threshold 0.7 > -pc_hypre_boomeramg_interp_type ext+i > -pc_hypre_boomeramg_P_max 2 > -pc_hypre_boomeramg_truncfactor 0.3 > > Best, > Parv > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!YXR-8qKioRYS0fNOHacYGkm6WaIuKge2zoTiW1n0vLsWQUBiyLM48cg58pRLtNm0QjVigIZYftn2x-1bo13Z$ -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image.png Type: image/png Size: 119488 bytes Desc: not available URL: From mail2amneet at gmail.com Wed Oct 30 22:20:36 2024 From: mail2amneet at gmail.com (Amneet Bhalla) Date: Wed, 30 Oct 2024 20:20:36 -0700 Subject: [petsc-users] Rigid body nullspace for Stokes operator In-Reply-To: <87bjz1qtk6.fsf@jedbrown.org> References: <875xpas35p.fsf@jedbrown.org> <87bjz1qtk6.fsf@jedbrown.org> Message-ID: I think Mark mentioned this earlier, but I want to make sure that the rigid body null vectors should be specified only when Neumann boundary conditions are used on all boundaries of the domain, correct? Alternatively, if a Dirichlet boundary condition is used (on any part of the domain boundary) then there is no null space, i.e., the operator is a full rank matrix? If the above is true, then I think I do not need to specify the rigid body null modes because I am using Dirichlet boundary conditions for the velocity solver. On Wed, Oct 30, 2024 at 12:28?PM Jed Brown wrote: > Yes to 6 null vectors in 3D, 3 null vectors in 2D. The center of mass does > not need to be identified because you can algebraically orthogonalize > (lines 411-420 here). > > > https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/blob/main/src/dm/impls/plex/plexfem.c?ref_type=heads*L377-425__;Iw!!G_uCfscf7eWS!ezw7zWJYwCHHUvoAmPZ4JT-HJ8l8V8aC3pq3VbTRke4EO6i5Z_AxjqmaJ1M_6BS6b99zSzfvl8FQIhZEjNHJ84-m6w$ > > See also this implementation with raw coordinates. GAMG orthogonalizes > within each aggregate (in a later phase of the algorithm) so global > orthogonalization is not necessary. > > > https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/blob/main/src/ksp/pc/impls/gamg/agg.c?ref_type=heads*L387__;Iw!!G_uCfscf7eWS!ezw7zWJYwCHHUvoAmPZ4JT-HJ8l8V8aC3pq3VbTRke4EO6i5Z_AxjqmaJ1M_6BS6b99zSzfvl8FQIhZEjNFr-J8G0w$ > > Amneet Bhalla writes: > > > I think the nullspace for the velocity operator is of the form > > > > vnull = U + ? ? r > > in which U is a rigid body velocity and ? is the rigid body rotational > > velocity, and r is the radius vector from the center of mass. I believe I > > need to construct 6 nullspace vectors in 3D and 3 nullspace vectors in > 2D. > > Sounds correct? Also does the center of mass coordinates matter when > > defining r? > > > > On Wed, Oct 30, 2024 at 7:53?AM Amneet Bhalla > wrote: > > > >> @Mark: Is there some document/paper that I can follow to check the > algebra > >> of these zero eigenvectors/null space modes? > >> > >> @Jed : We use a projection method preconditioner to solve the coupled > >> velocity pressure system as described here ( > >> https://urldefense.us/v3/__https://www.sciencedirect.com/science/article/pii/S0021999123004205__;!!G_uCfscf7eWS!ezw7zWJYwCHHUvoAmPZ4JT-HJ8l8V8aC3pq3VbTRke4EO6i5Z_AxjqmaJ1M_6BS6b99zSzfvl8FQIhZEjNFHB5gzxg$ ). > It > >> is an approximation of the Schur complement. As a part of projection > >> preconditioner, we need to solve just the momentum equation separately > >> without considering the pressure part. > >> > >> On Tue, Oct 29, 2024 at 8:03?PM Jed Brown wrote: > >> > >>> And to be clear, we recommend using fieldsplit Schur to separate the > >>> pressure and velocity part (there are many examples). Applying AMG > directly > >>> to the saddle point problem will not be a good solver because the > >>> heuristics assume positivity and do not preserve inf-sup stability > (nor do > >>> standard smoothers). > >>> > >>> Mark Adams writes: > >>> > >>> > This is linear elasticity and there are 6 "null" vectors (if you > removed > >>> > Dirichlet boundary conditions these are eigenvectors with zero > >>> eigenvalue): > >>> > 3 translations, x, y, z, and three rotatiions xx, yy ,zz. > >>> > x = (1,0,0,1,0,0,1,0 ...) > >>> > and xx is something like (0, z_1, -y_1, 0, z_2, -y_2, ...) where z_1 > is > >>> the > >>> > z coordinate of the first vertex, etc. > >>> > > >>> > Mark > >>> > > >>> > On Tue, Oct 29, 2024 at 3:47?PM Amneet Bhalla > > >>> wrote: > >>> > > >>> >> Hi Mark, > >>> >> > >>> >> Thanks! I am not sure how to construct null space and zero energy > modes > >>> >> manually for this operator. Is there some theory or documentation I > can > >>> >> follow to figure out what the null space and zero energy modes look > >>> like > >>> >> for this operator? Once I know what these are in symbolic form, I > >>> think I > >>> >> should be able to construct them manually. > >>> >> > >>> >> Best, > >>> >> --Amneet > >>> >> > >>> >> On Tue, Oct 29, 2024 at 7:35?AM Mark Adams wrote: > >>> >> > >>> >>> Oh my mistake. You are using staggered grids. So you don't have a > >>> block > >>> >>> size that hypre would use for the "nodal" methods. > >>> >>> I'm not sure what you are doing exactly, but try hypre and you > could > >>> >>> create the null space, zero energy modes, manually, attach to the > >>> matrix > >>> >>> and try GAMG. > >>> >>> You can run with '-info :pc' and grep on GAMG to see if GAMG is > >>> picking > >>> >>> the null space up (send this output if you can't figure it out). > >>> >>> > >>> >>> Thanks, > >>> >>> Mark > >>> >>> > >>> >>> On Tue, Oct 29, 2024 at 9:28?AM Mark Adams > wrote: > >>> >>> > >>> >>>> This coordinate interface is just a shortcut for vertex based > >>> >>>> discretizations with 3 dof per vertex, etc. (maybe works in 2D). > >>> >>>> You will need to construct the null space vectors manually and > >>> attach it > >>> >>>> to the matrix. Used by GAMG. > >>> >>>> > >>> >>>> Note, for hypre you want to use the "nodal" options and it does > not > >>> use > >>> >>>> these null space vectors. That is probably the way you want to go. > >>> >>>> eg: -pc_hypre_boomeramg_nodal_coarsen > >>> >>>> > >>> >>>> I would run with hypre boomerang and -help and grep on nodal to > see > >>> all > >>> >>>> the "nodal" options and use them. > >>> >>>> > >>> >>>> Thanks, > >>> >>>> Mark > >>> >>>> > >>> >>>> > >>> >>>> On Mon, Oct 28, 2024 at 8:06?PM Amneet Bhalla < > mail2amneet at gmail.com > >>> > > >>> >>>> wrote: > >>> >>>> > >>> >>>>> Hi Folks, > >>> >>>>> > >>> >>>>> I am trying to solve the momentum equation in a projection > >>> >>>>> preconditioner using GAMG or Hypre solver. The equation looks > like > >>> for > >>> >>>>> velocity variable *v* looks like: > >>> >>>>> > >>> >>>>> > >>> >>>>> [image: Screenshot 2024-10-28 at 4.15.17 PM.png] > >>> >>>>> > >>> >>>>> Here, ? is spatially varying dynamic viscosity and ? is spatially > >>> >>>>> varying bulk viscosity. I understand that I need to specify rigid > >>> body > >>> >>>>> nullspace modes to the multigrid solver in order to accelerate > its > >>> >>>>> convergence. Looking into this routine > >>> MatNullSpaceCreateRigidBody() ( > >>> >>>>> > >>> > https://urldefense.us/v3/__https://petsc.org/release/manualpages/Mat/MatNullSpaceCreateRigidBody/__;!!G_uCfscf7eWS!bVR6duCoDqPhZrWS-sm1c5qxsFPjZMhdT86AqLpPzWgVy5qoRhd4_Jue2LJOIS6LRrtV2cHGrqger1Yvb-Y5f-0$ > >>> >>>>> < > >>> > https://urldefense.us/v3/__https://petsc.org/release/manualpages/Mat/MatNullSpaceCreateRigidBody/__;!!G_uCfscf7eWS!eKqgIJjCdMzIU76f7X65AmGxrU_-lC7W02BMWafJ77DNf_IuQk6O1X3qU1x9Ez8NJ20vZEL-mF6T1yNmDnwv0eWa2w$ > >>> >), > >>> >>>>> I see that I need to provide the coordinates of each node. I am > >>> using > >>> >>>>> staggered grid discretization. Do I need to provide coordinates > of > >>> >>>>> staggered grid locations? > >>> >>>>> > >>> >>>>> Thanks, > >>> >>>>> -- > >>> >>>>> --Amneet > >>> >>>>> > >>> >>>>> > >>> >>>>> > >>> >>>>> > >>> >> > >>> >> -- > >>> >> --Amneet > >>> >> > >>> >> > >>> >> > >>> >> > >>> > >> > >> > >> -- > >> --Amneet > >> > >> > >> > >> > > > > -- > > --Amneet > -- --Amneet -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Thu Oct 31 00:01:08 2024 From: jed at jedbrown.org (Jed Brown) Date: Wed, 30 Oct 2024 23:01:08 -0600 Subject: [petsc-users] Rigid body nullspace for Stokes operator In-Reply-To: References: <875xpas35p.fsf@jedbrown.org> <87bjz1qtk6.fsf@jedbrown.org> Message-ID: <87y124q31n.fsf@jedbrown.org> There is MatSetNearNullSpace, which is used to ensure an approximation property in coarse levels of multigrid. That should always be set when dealing with problems like this, regardless of boundary conditions. Separately, there is MatSetNullSpace, which you should only use if you are solving a singular system with that null space. Note that you need at least three non-colinear points to constrain the null space so if you, for example, have a Dirichlet condition at only one point or along one straight line, there will still be a null space. Amneet Bhalla writes: > I think Mark mentioned this earlier, but I want to make sure that the rigid > body null vectors should be specified only when Neumann boundary conditions > are used on all boundaries of the domain, correct? Alternatively, if a > Dirichlet boundary condition is used (on any part of the domain boundary) > then there is no null space, i.e., the operator is a full rank matrix? > > If the above is true, then I think I do not need to specify the rigid body > null modes because I am using Dirichlet boundary conditions for the > velocity solver. > > On Wed, Oct 30, 2024 at 12:28?PM Jed Brown wrote: > >> Yes to 6 null vectors in 3D, 3 null vectors in 2D. The center of mass does >> not need to be identified because you can algebraically orthogonalize >> (lines 411-420 here). >> >> >> https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/blob/main/src/dm/impls/plex/plexfem.c?ref_type=heads*L377-425__;Iw!!G_uCfscf7eWS!cUe-WWTFiWh16JyZXhgGBBJ9kXbQpDaKFqY17p826i1tQk6dtu_ccEOtB9z_w_rXWK8hYv5SqNXhXSmZGcc$ >> >> See also this implementation with raw coordinates. GAMG orthogonalizes >> within each aggregate (in a later phase of the algorithm) so global >> orthogonalization is not necessary. >> >> >> https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/blob/main/src/ksp/pc/impls/gamg/agg.c?ref_type=heads*L387__;Iw!!G_uCfscf7eWS!cUe-WWTFiWh16JyZXhgGBBJ9kXbQpDaKFqY17p826i1tQk6dtu_ccEOtB9z_w_rXWK8hYv5SqNXhcaBBq7s$ >> >> Amneet Bhalla writes: >> >> > I think the nullspace for the velocity operator is of the form >> > >> > vnull = U + ? ? r >> > in which U is a rigid body velocity and ? is the rigid body rotational >> > velocity, and r is the radius vector from the center of mass. I believe I >> > need to construct 6 nullspace vectors in 3D and 3 nullspace vectors in >> 2D. >> > Sounds correct? Also does the center of mass coordinates matter when >> > defining r? >> > >> > On Wed, Oct 30, 2024 at 7:53?AM Amneet Bhalla >> wrote: >> > >> >> @Mark: Is there some document/paper that I can follow to check the >> algebra >> >> of these zero eigenvectors/null space modes? >> >> >> >> @Jed : We use a projection method preconditioner to solve the coupled >> >> velocity pressure system as described here ( >> >> https://urldefense.us/v3/__https://www.sciencedirect.com/science/article/pii/S0021999123004205__;!!G_uCfscf7eWS!cUe-WWTFiWh16JyZXhgGBBJ9kXbQpDaKFqY17p826i1tQk6dtu_ccEOtB9z_w_rXWK8hYv5SqNXhOpVmAEw$ ). >> It >> >> is an approximation of the Schur complement. As a part of projection >> >> preconditioner, we need to solve just the momentum equation separately >> >> without considering the pressure part. >> >> >> >> On Tue, Oct 29, 2024 at 8:03?PM Jed Brown wrote: >> >> >> >>> And to be clear, we recommend using fieldsplit Schur to separate the >> >>> pressure and velocity part (there are many examples). Applying AMG >> directly >> >>> to the saddle point problem will not be a good solver because the >> >>> heuristics assume positivity and do not preserve inf-sup stability >> (nor do >> >>> standard smoothers). >> >>> >> >>> Mark Adams writes: >> >>> >> >>> > This is linear elasticity and there are 6 "null" vectors (if you >> removed >> >>> > Dirichlet boundary conditions these are eigenvectors with zero >> >>> eigenvalue): >> >>> > 3 translations, x, y, z, and three rotatiions xx, yy ,zz. >> >>> > x = (1,0,0,1,0,0,1,0 ...) >> >>> > and xx is something like (0, z_1, -y_1, 0, z_2, -y_2, ...) where z_1 >> is >> >>> the >> >>> > z coordinate of the first vertex, etc. >> >>> > >> >>> > Mark >> >>> > >> >>> > On Tue, Oct 29, 2024 at 3:47?PM Amneet Bhalla > > >> >>> wrote: >> >>> > >> >>> >> Hi Mark, >> >>> >> >> >>> >> Thanks! I am not sure how to construct null space and zero energy >> modes >> >>> >> manually for this operator. Is there some theory or documentation I >> can >> >>> >> follow to figure out what the null space and zero energy modes look >> >>> like >> >>> >> for this operator? Once I know what these are in symbolic form, I >> >>> think I >> >>> >> should be able to construct them manually. >> >>> >> >> >>> >> Best, >> >>> >> --Amneet >> >>> >> >> >>> >> On Tue, Oct 29, 2024 at 7:35?AM Mark Adams wrote: >> >>> >> >> >>> >>> Oh my mistake. You are using staggered grids. So you don't have a >> >>> block >> >>> >>> size that hypre would use for the "nodal" methods. >> >>> >>> I'm not sure what you are doing exactly, but try hypre and you >> could >> >>> >>> create the null space, zero energy modes, manually, attach to the >> >>> matrix >> >>> >>> and try GAMG. >> >>> >>> You can run with '-info :pc' and grep on GAMG to see if GAMG is >> >>> picking >> >>> >>> the null space up (send this output if you can't figure it out). >> >>> >>> >> >>> >>> Thanks, >> >>> >>> Mark >> >>> >>> >> >>> >>> On Tue, Oct 29, 2024 at 9:28?AM Mark Adams >> wrote: >> >>> >>> >> >>> >>>> This coordinate interface is just a shortcut for vertex based >> >>> >>>> discretizations with 3 dof per vertex, etc. (maybe works in 2D). >> >>> >>>> You will need to construct the null space vectors manually and >> >>> attach it >> >>> >>>> to the matrix. Used by GAMG. >> >>> >>>> >> >>> >>>> Note, for hypre you want to use the "nodal" options and it does >> not >> >>> use >> >>> >>>> these null space vectors. That is probably the way you want to go. >> >>> >>>> eg: -pc_hypre_boomeramg_nodal_coarsen >> >>> >>>> >> >>> >>>> I would run with hypre boomerang and -help and grep on nodal to >> see >> >>> all >> >>> >>>> the "nodal" options and use them. >> >>> >>>> >> >>> >>>> Thanks, >> >>> >>>> Mark >> >>> >>>> >> >>> >>>> >> >>> >>>> On Mon, Oct 28, 2024 at 8:06?PM Amneet Bhalla < >> mail2amneet at gmail.com >> >>> > >> >>> >>>> wrote: >> >>> >>>> >> >>> >>>>> Hi Folks, >> >>> >>>>> >> >>> >>>>> I am trying to solve the momentum equation in a projection >> >>> >>>>> preconditioner using GAMG or Hypre solver. The equation looks >> like >> >>> for >> >>> >>>>> velocity variable *v* looks like: >> >>> >>>>> >> >>> >>>>> >> >>> >>>>> [image: Screenshot 2024-10-28 at 4.15.17 PM.png] >> >>> >>>>> >> >>> >>>>> Here, ? is spatially varying dynamic viscosity and ? is spatially >> >>> >>>>> varying bulk viscosity. I understand that I need to specify rigid >> >>> body >> >>> >>>>> nullspace modes to the multigrid solver in order to accelerate >> its >> >>> >>>>> convergence. Looking into this routine >> >>> MatNullSpaceCreateRigidBody() ( >> >>> >>>>> >> >>> >> https://urldefense.us/v3/__https://petsc.org/release/manualpages/Mat/MatNullSpaceCreateRigidBody/__;!!G_uCfscf7eWS!bVR6duCoDqPhZrWS-sm1c5qxsFPjZMhdT86AqLpPzWgVy5qoRhd4_Jue2LJOIS6LRrtV2cHGrqger1Yvb-Y5f-0$ >> >>> >>>>> < >> >>> >> https://urldefense.us/v3/__https://petsc.org/release/manualpages/Mat/MatNullSpaceCreateRigidBody/__;!!G_uCfscf7eWS!eKqgIJjCdMzIU76f7X65AmGxrU_-lC7W02BMWafJ77DNf_IuQk6O1X3qU1x9Ez8NJ20vZEL-mF6T1yNmDnwv0eWa2w$ >> >>> >), >> >>> >>>>> I see that I need to provide the coordinates of each node. I am >> >>> using >> >>> >>>>> staggered grid discretization. Do I need to provide coordinates >> of >> >>> >>>>> staggered grid locations? >> >>> >>>>> >> >>> >>>>> Thanks, >> >>> >>>>> -- >> >>> >>>>> --Amneet >> >>> >>>>> >> >>> >>>>> >> >>> >>>>> >> >>> >>>>> >> >>> >> >> >>> >> -- >> >>> >> --Amneet >> >>> >> >> >>> >> >> >>> >> >> >>> >> >> >>> >> >> >> >> >> >> -- >> >> --Amneet >> >> >> >> >> >> >> >> >> > >> > -- >> > --Amneet >> > > > -- > --Amneet From cpraveen at gmail.com Thu Oct 31 00:24:29 2024 From: cpraveen at gmail.com (Praveen C) Date: Thu, 31 Oct 2024 10:54:29 +0530 Subject: [petsc-users] Problem running geoclaw example with petsc 3.22 In-Reply-To: References: <1C63D19F-6EBC-43A3-8B40-65EA34991AFA@gmail.com> <0136027D-C2CE-47D0-AF97-ACDE1DA0C382@gmail.com> <4726A341-54DB-4592-BBAE-843AF271F28E@petsc.dev> <27B47E2D-00D5-4326-95A9-7595DFF0E952@gmail.com> <490B5A4A-BBD8-490A-91FD-7CEC9E01A99B@petsc.dev> <96674E24-9EDA-42DE-8459-81D46A260CDE@petsc.dev> Message-ID: Hello Barry With the extra option > -mpi_linear_solver_server_use_shared_memory false I get different error Thanks praveen $ make .output /Library/Developer/CommandLineTools/usr/bin/make output -f Makefile /Users/praveen/Applications/clawpack/geoclaw/src/2d/bouss/Makefile.bouss /Users/praveen/Applications/clawpack/clawutil/src/Makefile.common rm -f .output python /Users/praveen/Applications/clawpack/clawutil/src/python/clawutil/runclaw.py /Users/praveen/Work/bouss/radial_flat/xgeoclaw _output \ True None . False False None "/opt/homebrew/Caskroom/miniforge/base/envs/claw/./bin/mpiexec -n 6" ==> runclaw: Will take data from /Volumes/Samsung_T5/Work/bouss/radial_flat ==> runclaw: Will write output to /Volumes/Samsung_T5/Work/bouss/radial_flat/_output ==> runclaw: Removing all old fort/gauge files in /Volumes/Samsung_T5/Work/bouss/radial_flat/_output ==> Running with command: /opt/homebrew/Caskroom/miniforge/base/envs/claw/./bin/mpiexec -n 6 /Users/praveen/Work/bouss/radial_flat/xgeoclaw Reading data file: claw.data first 5 lines are comments and will be skipped Reading data file: amr.data first 5 lines are comments and will be skipped Running amrclaw ... Reading data file: geoclaw.data first 5 lines are comments and will be skipped Reading data file: refinement.data first 5 lines are comments and will be skipped Reading data file: dtopo.data first 5 lines are comments and will be skipped Reading data file: topo.data first 5 lines are comments and will be skipped converting to topotype > 1 might reduce file size python tools for converting files are provided Reading topography file /Volumes/Samsung_T5/Work/bouss/radial_flat/flat100.tt1 Reading data file: qinit.data first 5 lines are comments and will be skipped qinit_type = 0, no perturbation Reading data file: fgout_grids.data first 5 lines are comments and will be skipped Reading data file: friction.data first 5 lines are comments and will be skipped Reading data file: multilayer.data first 5 lines are comments and will be skipped Reading data file: surge.data first 5 lines are comments and will be skipped Reading data file: regions.data first 5 lines are comments and will be skipped Reading data file: flagregions.data first 5 lines are comments and will be skipped +++ rregion bounding box: 0.0000000000000000 5000.0000000000000 0.0000000000000000 5000.0000000000000 +++ i, rr%s(1), rr%ds: 1 0.0000000000000000 5000.0000000000000 +++ Ruled region name: Region_diagonal +++ Ruled region file_name: /Volumes/Samsung_T5/Work/bouss/radial_flat/RuledRectangle_Diagonal.data +++ rregion bounding box: 0.0000000000000000 5000.0000000000000 -1000.0000000000000 6000.0000000000000 +++ i, rr%s(1), rr%ds: 2 0.0000000000000000 5000.0000000000000 +++ rregion bounding box: 0.0000000000000000 1000.0000000000000 0.0000000000000000 1000.0000000000000 +++ i, rr%s(1), rr%ds: 3 0.0000000000000000 1000.0000000000000 Reading data file: gauges.data first 5 lines are comments and will be skipped Reading data file: fgmax_grids.data first 5 lines are comments and will be skipped Reading data file: adjoint.data first 5 lines are comments and will be skipped Reading data file: bouss.data first 5 lines are comments and will be skipped Using SGN equations ==> Applying Bouss equations to selected grids between levels 1 and 10 ==> Use Bouss. in water deeper than 1.0000000000000000 Using a PETSc solver Using Bouss equations from the start rnode allocated... node allocated... listOfGrids allocated... Storage allocated... bndList allocated... Gridding level 1 at t = 0.000000E+00: 4 grids with 10000 cells Setting initial dt to 2.9999999999999999E-002 max threads set to 1 Done reading data, starting computation ... Total zeta at initial time: 39269.907650665169 GEOCLAW: Frame 0 output files done at time t = 0.000000D+00 [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see https://urldefense.us/v3/__https://petsc.org/release/faq/*valgrind__;Iw!!G_uCfscf7eWS!ZcKI6rl1C9IIPiAD2VOcI1E8tTTVXaFy7T2RGbUz5E43yWaFQlh4FVtReJGsyAD1Ao2TnT98cs2aAt5V3vICfQ$ and https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!ZcKI6rl1C9IIPiAD2VOcI1E8tTTVXaFy7T2RGbUz5E43yWaFQlh4FVtReJGsyAD1Ao2TnT98cs2aAt6ZkxL0UA$ [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [0]PETSC ERROR: to get more information on the crash. [0]PETSC ERROR: Run with -malloc_debug to check if memory corruption is causing the crash. -------------------------------------------------------------------------- MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_SELF Proc: [[5779,1],0] Errorcode: 59 NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. You may or may not see output from other processes, depending on exactly when Open MPI kills them. -------------------------------------------------------------------------- -------------------------------------------------------------------------- prterun has exited due to process rank 0 with PID 0 on node chandra calling "abort". This may have caused other processes in the application to be terminated by signals sent by prterun (as reported here). -------------------------------------------------------------------------- Traceback (most recent call last): File "/Users/praveen/Applications/clawpack/clawutil/src/python/clawutil/runclaw.py", line 242, in runclaw proc = subprocess.check_call(cmd_split, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib/python3.12/subprocess.py", line 413, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '['/opt/homebrew/Caskroom/miniforge/base/envs/claw/./bin/mpiexec', '-n', '6', '/Users/praveen/Work/bouss/radial_flat/xgeoclaw']' returned non-zero exit status 59. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/Users/praveen/Applications/clawpack/clawutil/src/python/clawutil/runclaw.py", line 341, in runclaw(*args) File "/Users/praveen/Applications/clawpack/clawutil/src/python/clawutil/runclaw.py", line 249, in runclaw raise ClawExeError(exe_error_str, cpe.returncode, cpe.cmd, ClawExeError: *** FORTRAN EXE FAILED *** make[1]: *** [output] Error 1 make: *** [.output] Error 2 > On 30 Oct 2024, at 9:33?PM, Barry Smith wrote: > > > Please put > > -mpi_linear_solver_server_use_shared_memory false > > into the petscMPIOptions file and see if that changes anything. > > Barry > > >> On Oct 30, 2024, at 9:05?AM, Praveen C wrote: >> >> I have attached some files for this >> >> cd bouss >> . setenv.sh # some settings in this file may need to be changed >> >> cd radial_flat/1d_radial >> make .output >> cd .. >> make .output >> >> Thanks >> praveen >> >> >> >> >>> On 30 Oct 2024, at 6:28?PM, Barry Smith wrote: >>> >>> >>> Please send me exact instructions on how you are running the geoclaw example, when I run >>> >>> geoclaw/examples/bouss/radial_flat >>> >>> >>> it only runs with one level, but never errors, I changed setrun.py to amrdata.amr_levels_max = 5 but it made no difference. >>> >>> Thanks >>> >>> Barry >>> >>> >>> >>>> On Oct 24, 2024, at 12:17?PM, Praveen C wrote: >>>> >>>> Hello Barry >>>> >>>> I use this script to install clawpack and required dependencies >>>> >>>> https://urldefense.us/v3/__https://github.com/cpraveen/cfdlab/blob/master/bin/clawpack.sh__;!!G_uCfscf7eWS!ZcKI6rl1C9IIPiAD2VOcI1E8tTTVXaFy7T2RGbUz5E43yWaFQlh4FVtReJGsyAD1Ao2TnT98cs2aAt7bQr0S6A$ >>>> >>>> See lines 125-132 >>>> >>>> You can use it like this >>>> >>>> export CLAW=/path/to/where/you/want/clawpack >>>> bash clawpack.sh v5.11.0 >>>> >>>> This will git pull clawpack and creates a conda env called ?claw? and installs inside that. >>>> >>>> I have not specified petsc version in this script, but latest miniforge should install petsc at 3.22 >>>> >>>> Thank you >>>> praveen >>>> >>>>> On 24 Oct 2024, at 9:37?PM, Barry Smith wrote: >>>>> >>>>> >>>>> Good, some helpful information with the runs you made. >>>>> >>>>> In the crash below it made more progress it was able to allocate multiple regions of shared memory and access them. I don't know why it would crash later. >>>>> >>>>> Can you tell me all the steps with miniforge (which seems to be related to the failure) you use? I've never used miniforge. >>>>> >>>>> If I can get to an environment that reproduces the problem I can debug it and fix it. >>>>> >>>>> Barry >>>>> >>>>> >>>>>> On Oct 24, 2024, at 11:49?AM, Praveen C wrote: >>>>>> >>>>>> I get this >>>>>> >>>>>> $ mpiexec -n 3 ./ex89f -n 20 -mpi_linear_solver_server -mpi_linear_solver_server -mpi_linear_solver_server_ksp_view -ksp_monitor -ksp_converged_reason -ksp_view -mpi_linear_solver_server_minimum_count_per_rank 5 >>>>>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>>>>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range >>>>>> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >>>>>> [0]PETSC ERROR: or see https://urldefense.us/v3/__https://petsc.org/release/faq/*valgrind__;Iw!!G_uCfscf7eWS!ZcKI6rl1C9IIPiAD2VOcI1E8tTTVXaFy7T2RGbUz5E43yWaFQlh4FVtReJGsyAD1Ao2TnT98cs2aAt5V3vICfQ$ and https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!ZcKI6rl1C9IIPiAD2VOcI1E8tTTVXaFy7T2RGbUz5E43yWaFQlh4FVtReJGsyAD1Ao2TnT98cs2aAt6ZkxL0UA$ >>>>>> [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run >>>>>> [0]PETSC ERROR: to get more information on the crash. >>>>>> [0]PETSC ERROR: Run with -malloc_debug to check if memory corruption is causing the crash. >>>>>> -------------------------------------------------------------------------- >>>>>> MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_SELF >>>>>> Proc: [[47380,1],0] >>>>>> Errorcode: 59 >>>>>> >>>>>> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. >>>>>> You may or may not see output from other processes, depending on >>>>>> exactly when Open MPI kills them. >>>>>> -------------------------------------------------------------------------- >>>>>> -------------------------------------------------------------------------- >>>>>> prterun has exited due to process rank 0 with PID 0 on node MacMiniHome calling >>>>>> "abort". This may have caused other processes in the application to be >>>>>> terminated by signals sent by prterun (as reported here). >>>>>> ????????????????????????????????????? >>>>>> >>>>>> and I have this after the code exits >>>>>> >>>>>> $ ipcs -m >>>>>> IPC status from as of Thu Oct 24 21:17:39 IST 2024 >>>>>> T ID KEY MODE OWNER GROUP >>>>>> Shared Memory: >>>>>> m 1572864 0x0000000b --rw-rw-rw- praveen staff >>>>>> m 524289 0x0000000c --rw-rw-rw- praveen staff >>>>>> m 655362 0x0000000d --rw-rw-rw- praveen staff >>>>>> m 262147 0x0000000e --rw-rw-rw- praveen staff >>>>>> m 262148 0x0000000f --rw-rw-rw- praveen staff >>>>>> m 393221 0x0000000a --rw-rw-rw- praveen staff >>>>>> >>>>>> This is with petsc installed with miniforge, which I also use with clawpack. With spack installed petsc, I can run the ex89f example. >>>>>> >>>>>> Thanks >>>>>> praveen >>>>>> >>>>>>> On 24 Oct 2024, at 7:55?PM, Barry Smith wrote: >>>>>>> >>>>>>> >>>>>>> Ok, super strange, I've run many times on my Mac >>>>>>> >>>>>>> Can you please try to remove that allocated memory with ipcrm and then >>>>>>> >>>>>>> cd $PETSC_DIR/src/ksp/ksp/tutorials >>>>>>> make ex89f >>>>>>> mpiexec -n 3 ./ex89f -n 20 -mpi_linear_solver_server -mpi_linear_solver_server -mpi_linear_solver_server_ksp_view -ksp_monitor -ksp_converged_reason -ksp_view -mpi_linear_solver_server_minimum_count_per_rank 5 >>>>>>> >>>>>>> This does the same thing as the GeoClaw code but is much simpler. >>>>>>> >>>>>>> Barry >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> On Oct 23, 2024, at 10:55?PM, Praveen C wrote: >>>>>>>> >>>>>>>> I get very similar error on my mac with >>>>>>>> >>>>>>>> $ gfortran -v >>>>>>>> Using built-in specs. >>>>>>>> COLLECT_GCC=gfortran >>>>>>>> COLLECT_LTO_WRAPPER=/opt/homebrew/Caskroom/miniforge/base/envs/claw/libexec/gcc/arm64-apple-darwin20.0.0/13.2.0/lto-wrapper >>>>>>>> Target: arm64-apple-darwin20.0.0 >>>>>>>> Configured with: ../configure --prefix=/opt/homebrew/Caskroom/miniforge/base/envs/claw --build=x86_64-apple-darwin13.4.0 --host=arm64-apple-darwin20.0.0 --target=arm64-apple-darwin20.0.0 --with-libiconv-prefix=/opt/homebrew/Caskroom/miniforge/base/envs/claw --enable-languages=fortran --disable-multilib --enable-checking=release --disable-bootstrap --disable-libssp --with-gmp=/opt/homebrew/Caskroom/miniforge/base/envs/claw --with-mpfr=/opt/homebrew/Caskroom/miniforge/base/envs/claw --with-mpc=/opt/homebrew/Caskroom/miniforge/base/envs/claw --with-isl=/opt/homebrew/Caskroom/miniforge/base/envs/claw --enable-darwin-at-rpath >>>>>>>> Thread model: posix >>>>>>>> Supported LTO compression algorithms: zlib >>>>>>>> gcc version 13.2.0 (GCC) >>>>>>>> >>>>>>>> Before starting >>>>>>>> >>>>>>>> $ ipcs -m >>>>>>>> IPC status from as of Thu Oct 24 08:02:11 IST 2024 >>>>>>>> T ID KEY MODE OWNER GROUP >>>>>>>> Shared Memory: >>>>>>>> >>>>>>>> and when I run the code >>>>>>>> >>>>>>>> Using a PETSc solver >>>>>>>> Using Bouss equations from the start >>>>>>>> rnode allocated... >>>>>>>> node allocated... >>>>>>>> listOfGrids allocated... >>>>>>>> Storage allocated... >>>>>>>> bndList allocated... >>>>>>>> Gridding level 1 at t = 0.000000E+00: 4 grids with 10000 cells >>>>>>>> Setting initial dt to 2.9999999999999999E-002 >>>>>>>> max threads set to 1 >>>>>>>> >>>>>>>> Done reading data, starting computation ... >>>>>>>> >>>>>>>> Total zeta at initial time: 39269.907650665169 >>>>>>>> GEOCLAW: Frame 0 output files done at time t = 0.000000D+00 >>>>>>>> >>>>>>>> [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >>>>>>>> [0]PETSC ERROR: Petsc has generated inconsistent data >>>>>>>> [0]PETSC ERROR: Unable to locate PCMPI allocated shared address 0x130698000 >>>>>>>> [0]PETSC ERROR: WARNING! There are unused option(s) set! Could be the program crashed before usage or a spelling mistake, etc! >>>>>>>> [0]PETSC ERROR: Option left: name:-ksp_type value: preonly source: file >>>>>>>> [0]PETSC ERROR: Option left: name:-mpi_ksp_max_it value: 200 source: file >>>>>>>> [0]PETSC ERROR: Option left: name:-mpi_ksp_reuse_preconditioner (no value) source: file >>>>>>>> [0]PETSC ERROR: Option left: name:-mpi_ksp_rtol value: 1.e-9 source: file >>>>>>>> [0]PETSC ERROR: Option left: name:-mpi_ksp_type value: gmres source: file >>>>>>>> [0]PETSC ERROR: Option left: name:-mpi_linear_solver_server_view (no value) source: file >>>>>>>> [0]PETSC ERROR: Option left: name:-mpi_pc_gamg_sym_graph value: true source: file >>>>>>>> [0]PETSC ERROR: Option left: name:-mpi_pc_gamg_symmetrize_graph value: true source: file >>>>>>>> [0]PETSC ERROR: Option left: name:-mpi_pc_type value: gamg source: file >>>>>>>> [0]PETSC ERROR: Option left: name:-pc_mpi_minimum_count_per_rank value: 5000 source: file >>>>>>>> [0]PETSC ERROR: Option left: name:-pc_type value: mpi source: file >>>>>>>> [0]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!ZcKI6rl1C9IIPiAD2VOcI1E8tTTVXaFy7T2RGbUz5E43yWaFQlh4FVtReJGsyAD1Ao2TnT98cs2aAt6ZkxL0UA$ for trouble shooting. >>>>>>>> [0]PETSC ERROR: Petsc Release Version 3.22.0, Sep 28, 2024 >>>>>>>> [0]PETSC ERROR: /Users/praveen/work/bouss/radial_flat/xgeoclaw with 6 MPI process(es) and PETSC_ARCH on MacMiniHome.local by praveen Thu Oct 24 08:04:27 2024 >>>>>>>> [0]PETSC ERROR: Configure options: AR=arm64-apple-darwin20.0.0-ar CC=mpicc CXX=mpicxx FC=mpifort CFLAGS="-ftree-vectorize -fPIC -fstack-protector-strong -O2 -pipe -isystem /opt/homebrew/Caskroom/miniforge/base/envs/claw/include " CPPFLAGS="-D_FORTIFY_SOURCE=2 -isystem /opt/homebrew/Caskroom/miniforge/base/envs/claw/include -mmacosx-version-min=11.0 -mmacosx-version-min=11.0" CXXFLAGS="-ftree-vectorize -fPIC -fstack-protector-strong -O2 -pipe -stdlib=libc++ -fvisibility-inlines-hidden -fmessage-length=0 -isystem /opt/homebrew/Caskroom/miniforge/base/envs/claw/include " FFLAGS="-march=armv8.3-a -ftree-vectorize -fPIC -fno-stack-protector -O2 -pipe -isystem /opt/homebrew/Caskroom/miniforge/base/envs/claw/include " LDFLAGS="-Wl,-headerpad_max_install_names -Wl,-dead_strip_dylibs -Wl,-rpath,/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib -L/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib" LIBS="-Wl,-rpath,/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib -lmpi_mpifh -lgfortran" --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 --FOPTFLAGS=-O3 --with-clib-autodetect=0 --with-cxxlib-autodetect=0 --with-fortranlib-autodetect=0 --with-debugging=0 --with-blas-lib=libblas.dylib --with-lapack-lib=liblapack.dylib --with-yaml=1 --with-hdf5=1 --with-fftw=1 --with-hwloc=0 --with-hypre=1 --with-metis=1 --with-mpi=1 --with-mumps=1 --with-parmetis=1 --with-pthread=1 --with-ptscotch=1 --with-shared-libraries --with-ssl=0 --with-scalapack=1 --with-superlu=1 --with-superlu_dist=1 --with-superlu_dist-include=/opt/homebrew/Caskroom/miniforge/base/envs/claw/include/superlu-dist --with-superlu_dist-lib=-lsuperlu_dist --with-suitesparse=1 --with-suitesparse-dir=/opt/homebrew/Caskroom/miniforge/base/envs/claw --with-x=0 --with-scalar-type=real --with-cuda=0 --with-batch --prefix=/opt/homebrew/Caskroom/miniforge/base/envs/claw >>>>>>>> [0]PETSC ERROR: #1 PetscShmgetMapAddresses() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/sys/utils/server.c:114 >>>>>>>> [0]PETSC ERROR: #2 PCMPISetMat() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/ksp/pc/impls/mpi/pcmpi.c:269 >>>>>>>> [0]PETSC ERROR: #3 PCSetUp_MPI() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/ksp/pc/impls/mpi/pcmpi.c:853 >>>>>>>> [0]PETSC ERROR: #4 PCSetUp() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/ksp/pc/interface/precon.c:1071 >>>>>>>> [0]PETSC ERROR: #5 KSPSetUp() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/ksp/ksp/interface/itfunc.c:415 >>>>>>>> [0]PETSC ERROR: #6 KSPSolve_Private() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/ksp/ksp/interface/itfunc.c:826 >>>>>>>> [0]PETSC ERROR: #7 KSPSolve() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/ksp/ksp/interface/itfunc.c:1075 >>>>>>>> >>>>>>>> Code does not progress and I kill it >>>>>>>> >>>>>>>> ^CTraceback (most recent call last): >>>>>>>> File "/Users/praveen/Applications/clawpack/clawutil/src/python/clawutil/runclaw.py", line 341, in >>>>>>>> runclaw(*args) >>>>>>>> File "/Users/praveen/Applications/clawpack/clawutil/src/python/clawutil/runclaw.py", line 242, in runclaw >>>>>>>> proc = subprocess.check_call(cmd_split, >>>>>>>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ >>>>>>>> File "/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib/python3.12/subprocess.py", line 408, in check_call >>>>>>>> retcode = call(*popenargs, **kwargs) >>>>>>>> ^^^^^^^^^^^^^^^^^^^^^^^^^^ >>>>>>>> File "/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib/python3.12/subprocess.py", line 391, in call >>>>>>>> return p.wait(timeout=timeout) >>>>>>>> ^^^^^^^^^^^^^^^^^^^^^^^ >>>>>>>> File "/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib/python3.12/subprocess.py", line 1264, in wait >>>>>>>> return self._wait(timeout=timeout) >>>>>>>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^ >>>>>>>> File "/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib/python3.12/subprocess.py", line 2053, in _wait >>>>>>>> (pid, sts) = self._try_wait(0) >>>>>>>> ^^^^^^^^^^^^^^^^^ >>>>>>>> File "/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib/python3.12/subprocess.py", line 2011, in _try_wait >>>>>>>> (pid, sts) = os.waitpid(self.pid, wait_flags) >>>>>>>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ >>>>>>>> KeyboardInterrupt >>>>>>>> make[1]: *** [output] Interrupt: 2 >>>>>>>> make: *** [.output] Interrupt: 2 >>>>>>>> >>>>>>>> Now it says >>>>>>>> >>>>>>>> $ ipcs -m >>>>>>>> IPC status from as of Thu Oct 24 08:05:06 IST 2024 >>>>>>>> T ID KEY MODE OWNER GROUP >>>>>>>> Shared Memory: >>>>>>>> m 720896 0x0000000a --rw-rw-rw- praveen staff >>>>>>>> >>>>>>>> Thanks >>>>>>>> praveen >>>>>>>> >>>>>>>>> On 23 Oct 2024, at 8:26?PM, Barry Smith wrote: >>>>>>>>> >>>>>>>>> >>>>>>>>> Hmm, so it is creating the first shared memory region with ID of 10 (A in hex) puts it in a linked list in PETSc but then when it tries to find it in the linked list it cannot find it. >>>>>>>>> >>>>>>>>> I don't know how to reproduce this or debug it remotely. >>>>>>>>> >>>>>>>>> Can you build on a completely different machine or with completely different compilers? >>>>>>>>> >>>>>>>>> Barry >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> On Oct 23, 2024, at 10:31?AM, Praveen C wrote: >>>>>>>>>> >>>>>>>>>> I get same error and now it shows >>>>>>>>>> >>>>>>>>>> $ ipcs -m >>>>>>>>>> >>>>>>>>>> ------ Shared Memory Segments -------- >>>>>>>>>> key shmid owner perms bytes nattch status >>>>>>>>>> 0x0000000a 32788 praveen 666 240 6 >>>>>>>>>> >>>>>>>>>> Note that the code seems to be still running after printing those error message, but it is not printing any progress which it should do. >>>>>>>>>> >>>>>>>>>> Thanks >>>>>>>>>> praveen >>>>>>>>>> >>>>>>>>>>> On 23 Oct 2024, at 7:56?PM, Barry Smith wrote: >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Try >>>>>>>>>>> >>>>>>>>>>> ipcrm -m 11 >>>>>>>>>>> >>>>>>>>>>> ipcs -m >>>>>>>>>>> >>>>>>>>>>> Try running the program again >>>>>>>>>>> >>>>>>>>>>> If failed check >>>>>>>>>>> >>>>>>>>>>> ipcs -m >>>>>>>>>>> >>>>>>>>>>> again >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> On Oct 23, 2024, at 10:20?AM, Praveen C wrote: >>>>>>>>>>>> >>>>>>>>>>>> Hello Barry >>>>>>>>>>>> >>>>>>>>>>>> I see this >>>>>>>>>>>> >>>>>>>>>>>> $ ipcs -m >>>>>>>>>>>> >>>>>>>>>>>> ------ Shared Memory Segments -------- >>>>>>>>>>>> key shmid owner perms bytes nattch status >>>>>>>>>>>> 0x0000000a 11 praveen 666 240 6 >>>>>>>>>>>> >>>>>>>>>>>> and I am observing same error as below. >>>>>>>>>>>> >>>>>>>>>>>> Thanks >>>>>>>>>>>> praveen >>>>>>>>>>>> >>>>>>>>>>>>> On 23 Oct 2024, at 7:08?PM, Barry Smith wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Please take a look at the notes in https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/PetscShmgetAllocateArray/__;!!G_uCfscf7eWS!ZcKI6rl1C9IIPiAD2VOcI1E8tTTVXaFy7T2RGbUz5E43yWaFQlh4FVtReJGsyAD1Ao2TnT98cs2aAt6cqcy0-g$ For some reason your program is not able to access/use the Unix shared memory; check if you are already using the shared memory (so it is not available for a new run) or the limits are too low to access enough memory. >>>>>>>>>>>>> >>>>>>>>>>>>> Barry >>>>>>>>>>>>> >>>>>>>>>>>>>> On Oct 23, 2024, at 8:23?AM, Praveen C wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> Dear all >>>>>>>>>>>>>> >>>>>>>>>>>>>> I am not able to run the boussinesq example from geoclaw using petsc at 3.22.0 >>>>>>>>>>>>>> >>>>>>>>>>>>>> https://urldefense.us/v3/__https://github.com/clawpack/geoclaw/tree/3303883f46572c58130d161986b8a87a57ca7816/examples/bouss__;!!G_uCfscf7eWS!e3VQ4NHKmXGstRsQW5vtI7fmKfUT9zmJkMJcPbcvPyIjicyfJpNoMgx3wZ-qyGcKNSjIkNZkzilec8MnHN6PMw$ >>>>>>>>>>>>>> >>>>>>>>>>>>>> It runs with petsc at 3.21.6 >>>>>>>>>>>>>> >>>>>>>>>>>>>> The error I get is given below. After printing this, the code does not progress. >>>>>>>>>>>>>> >>>>>>>>>>>>>> I use the following petsc options >>>>>>>>>>>>>> >>>>>>>>>>>>>> # set min numbers of matrix rows per MPI rank (default is 10000) >>>>>>>>>>>>>> -mpi_linear_solve_minimum_count_per_rank 5000 >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> # Krylov linear solver: >>>>>>>>>>>>>> -mpi_linear_solver_server >>>>>>>>>>>>>> -mpi_linear_solver_server_view >>>>>>>>>>>>>> -ksp_type gmres >>>>>>>>>>>>>> -ksp_max_it 200 >>>>>>>>>>>>>> -ksp_reuse_preconditioner >>>>>>>>>>>>>> -ksp_rtol 1.e-9 >>>>>>>>>>>>>> >>>>>>>>>>>>>> # preconditioner: >>>>>>>>>>>>>> -pc_type gamg >>>>>>>>>>>>>> >>>>>>>>>>>>>> I installed petsc and other dependencies for clawpack using miniforge. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks >>>>>>>>>>>>>> pc >>>>>>>>>>>>>> >>>>>>>>>>>>>> ==> Use Bouss. in water deeper than 1.0000000000000000 Using a PETSc solver >>>>>>>>>>>>>> Using Bouss equations from the start >>>>>>>>>>>>>> rnode allocated... >>>>>>>>>>>>>> node allocated... >>>>>>>>>>>>>> listOfGrids allocated... >>>>>>>>>>>>>> Storage allocated... >>>>>>>>>>>>>> bndList allocated... >>>>>>>>>>>>>> Gridding level 1 at t = 0.000000E+00: 4 grids with 10000 cells >>>>>>>>>>>>>> Setting initial dt to 2.9999999999999999E-002 >>>>>>>>>>>>>> max threads set to 6 >>>>>>>>>>>>>> Done reading data, starting computation ... Total zeta at initial time: 39269.907650665169 GEOCLAW: Frame 0 output files done at time t = 0.000000D+00 >>>>>>>>>>>>>> >>>>>>>>>>>>>> [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >>>>>>>>>>>>>> [0]PETSC ERROR: Petsc has generated inconsistent data >>>>>>>>>>>>>> [0]PETSC ERROR: Unable to locate PCMPI allocated shared address 0x55e6d750ae20 >>>>>>>>>>>>>> [0]PETSC ERROR: WARNING! There are unused option(s) set! Could be the program crashed before usage or a spelling mistake, etc! >>>>>>>>>>>>>> [0]PETSC ERROR: Option left: name:-ksp_max_it value: 200 source: file >>>>>>>>>>>>>> [0]PETSC ERROR: Option left: name:-ksp_reuse_preconditioner (no value) source: file >>>>>>>>>>>>>> [0]PETSC ERROR: Option left: name:-ksp_rtol value: 1.e-9 source: file >>>>>>>>>>>>>> [0]PETSC ERROR: Option left: name:-ksp_type value: gmres source: file >>>>>>>>>>>>>> [0]PETSC ERROR: Option left: name:-mpi_linear_solve_minimum_count_per_rank value: 5000 source: file >>>>>>>>>>>>>> [0]PETSC ERROR: Option left: name:-mpi_linear_solver_server_view (no value) source: file >>>>>>>>>>>>>> [0]PETSC ERROR: Option left: name:-pc_type value: gamg source: file >>>>>>>>>>>>>> [0]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!e3VQ4NHKmXGstRsQW5vtI7fmKfUT9zmJkMJcPbcvPyIjicyfJpNoMgx3wZ-qyGcKNSjIkNZkzilec8MvNjNo7A$ for trouble shooting. >>>>>>>>>>>>>> [0]PETSC ERROR: Petsc Release Version 3.22.0, Sep 28, 2024 [0]PETSC ERROR: /home/praveen/bouss/radial_flat/xgeoclaw with 6 MPI process(es) and PETSC_ARCH on euler by praveen Thu Oct 17 21:49:54 2024 >>>>>>>>>>>>>> [0]PETSC ERROR: Configure options: AR=${PREFIX}/bin/x86_64-conda-linux-gnu-ar CC=mpicc CXX=mpicxx FC=mpifort CFLAGS="-march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /opt/miniforge/envs/claw/include " CPPFLAGS="-DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /opt/miniforge/envs/claw/include" CXXFLAGS="-fvisibility-inlines-hidden -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /opt/miniforge/envs/claw/include " FFLAGS="-march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /opt/miniforge/envs/claw/include -Wl,--no-as-needed" LDFLAGS="-pthread -fopenmp -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,--allow-shlib-undefined -Wl,-rpath,/opt/miniforge/envs/claw/lib -Wl,-rpath-link,/opt/miniforge/envs/claw/lib -L/opt/miniforge/envs/claw/lib -Wl,-rpath-link,/opt/miniforge/envs/claw/lib" LIBS="-Wl,-rpath,/opt/miniforge/envs/claw/lib -lmpi_mpifh -lgfortran" --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 --FOPTFLAGS=-O3 --with-clib-autodetect=0 --with-cxxlib-autodetect=0 --with-fortranlib-autodetect=0 --with-debugging=0 --with-blas-lib=libblas.so --with-lapack-lib=liblapack.so --with-yaml=1 --with-hdf5=1 --with-fftw=1 --with-hwloc=0 --with-hypre=1 --with-metis=1 --with-mpi=1 --with-mumps=1 --with-parmetis=1 --with-pthread=1 --with-ptscotch=1 --with-shared-libraries --with-ssl=0 --with-scalapack=1 --with-superlu=1 --with-superlu_dist=1 --with-superlu_dist-include=/opt/miniforge/envs/claw/include/superlu-dist --with-superlu_dist-lib=-lsuperlu_dist --with-suitesparse=1 --with-suitesparse-dir=/opt/miniforge/envs/claw --with-x=0 --with-scalar-type=real --with-cuda=0 --prefix=/opt/miniforge/envs/claw >>>>>>>>>>>>>> [0]PETSC ERROR: #1 PetscShmgetMapAddresses() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/sys/utils/server.c:114 >>>>>>>>>>>>>> [0]PETSC ERROR: #2 PCMPISetMat() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/pc/impls/mpi/pcmpi.c:269 >>>>>>>>>>>>>> [0]PETSC ERROR: #3 PCSetUp_MPI() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/pc/impls/mpi/pcmpi.c:853 >>>>>>>>>>>>>> [0]PETSC ERROR: #4 PCSetUp() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/pc/interface/precon.c:1071 >>>>>>>>>>>>>> [0]PETSC ERROR: #5 KSPSetUp() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/ksp/interface/itfunc.c:415 >>>>>>>>>>>>>> [0]PETSC ERROR: #6 KSPSolve_Private() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/ksp/interface/itfunc.c:826 >>>>>>>>>>>>>> [0]PETSC ERROR: #7 KSPSolve() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/ksp/interface/itfunc.c:1075 >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cpraveen at gmail.com Thu Oct 31 00:24:29 2024 From: cpraveen at gmail.com (Praveen C) Date: Thu, 31 Oct 2024 10:54:29 +0530 Subject: [petsc-users] Problem running geoclaw example with petsc 3.22 In-Reply-To: References: <1C63D19F-6EBC-43A3-8B40-65EA34991AFA@gmail.com> <0136027D-C2CE-47D0-AF97-ACDE1DA0C382@gmail.com> <4726A341-54DB-4592-BBAE-843AF271F28E@petsc.dev> <27B47E2D-00D5-4326-95A9-7595DFF0E952@gmail.com> <490B5A4A-BBD8-490A-91FD-7CEC9E01A99B@petsc.dev> <96674E24-9EDA-42DE-8459-81D46A260CDE@petsc.dev> Message-ID: Hello Barry With the extra option > -mpi_linear_solver_server_use_shared_memory false I get different error Thanks praveen $ make .output /Library/Developer/CommandLineTools/usr/bin/make output -f Makefile /Users/praveen/Applications/clawpack/geoclaw/src/2d/bouss/Makefile.bouss /Users/praveen/Applications/clawpack/clawutil/src/Makefile.common rm -f .output python /Users/praveen/Applications/clawpack/clawutil/src/python/clawutil/runclaw.py /Users/praveen/Work/bouss/radial_flat/xgeoclaw _output \ True None . False False None "/opt/homebrew/Caskroom/miniforge/base/envs/claw/./bin/mpiexec -n 6" ==> runclaw: Will take data from /Volumes/Samsung_T5/Work/bouss/radial_flat ==> runclaw: Will write output to /Volumes/Samsung_T5/Work/bouss/radial_flat/_output ==> runclaw: Removing all old fort/gauge files in /Volumes/Samsung_T5/Work/bouss/radial_flat/_output ==> Running with command: /opt/homebrew/Caskroom/miniforge/base/envs/claw/./bin/mpiexec -n 6 /Users/praveen/Work/bouss/radial_flat/xgeoclaw Reading data file: claw.data first 5 lines are comments and will be skipped Reading data file: amr.data first 5 lines are comments and will be skipped Running amrclaw ... Reading data file: geoclaw.data first 5 lines are comments and will be skipped Reading data file: refinement.data first 5 lines are comments and will be skipped Reading data file: dtopo.data first 5 lines are comments and will be skipped Reading data file: topo.data first 5 lines are comments and will be skipped converting to topotype > 1 might reduce file size python tools for converting files are provided Reading topography file /Volumes/Samsung_T5/Work/bouss/radial_flat/flat100.tt1 Reading data file: qinit.data first 5 lines are comments and will be skipped qinit_type = 0, no perturbation Reading data file: fgout_grids.data first 5 lines are comments and will be skipped Reading data file: friction.data first 5 lines are comments and will be skipped Reading data file: multilayer.data first 5 lines are comments and will be skipped Reading data file: surge.data first 5 lines are comments and will be skipped Reading data file: regions.data first 5 lines are comments and will be skipped Reading data file: flagregions.data first 5 lines are comments and will be skipped +++ rregion bounding box: 0.0000000000000000 5000.0000000000000 0.0000000000000000 5000.0000000000000 +++ i, rr%s(1), rr%ds: 1 0.0000000000000000 5000.0000000000000 +++ Ruled region name: Region_diagonal +++ Ruled region file_name: /Volumes/Samsung_T5/Work/bouss/radial_flat/RuledRectangle_Diagonal.data +++ rregion bounding box: 0.0000000000000000 5000.0000000000000 -1000.0000000000000 6000.0000000000000 +++ i, rr%s(1), rr%ds: 2 0.0000000000000000 5000.0000000000000 +++ rregion bounding box: 0.0000000000000000 1000.0000000000000 0.0000000000000000 1000.0000000000000 +++ i, rr%s(1), rr%ds: 3 0.0000000000000000 1000.0000000000000 Reading data file: gauges.data first 5 lines are comments and will be skipped Reading data file: fgmax_grids.data first 5 lines are comments and will be skipped Reading data file: adjoint.data first 5 lines are comments and will be skipped Reading data file: bouss.data first 5 lines are comments and will be skipped Using SGN equations ==> Applying Bouss equations to selected grids between levels 1 and 10 ==> Use Bouss. in water deeper than 1.0000000000000000 Using a PETSc solver Using Bouss equations from the start rnode allocated... node allocated... listOfGrids allocated... Storage allocated... bndList allocated... Gridding level 1 at t = 0.000000E+00: 4 grids with 10000 cells Setting initial dt to 2.9999999999999999E-002 max threads set to 1 Done reading data, starting computation ... Total zeta at initial time: 39269.907650665169 GEOCLAW: Frame 0 output files done at time t = 0.000000D+00 [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see https://urldefense.us/v3/__https://petsc.org/release/faq/*valgrind__;Iw!!G_uCfscf7eWS!YfrSA6JCkK1gfGgtTWPw-8jHlRI4KhOjRVRmncpA01ImURbJtFEQuwPz9Y6AYUfTMVo0CPQxqXWVhulpyMltnQ$ and https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!YfrSA6JCkK1gfGgtTWPw-8jHlRI4KhOjRVRmncpA01ImURbJtFEQuwPz9Y6AYUfTMVo0CPQxqXWVhukefFfCQA$ [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [0]PETSC ERROR: to get more information on the crash. [0]PETSC ERROR: Run with -malloc_debug to check if memory corruption is causing the crash. -------------------------------------------------------------------------- MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_SELF Proc: [[5779,1],0] Errorcode: 59 NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. You may or may not see output from other processes, depending on exactly when Open MPI kills them. -------------------------------------------------------------------------- -------------------------------------------------------------------------- prterun has exited due to process rank 0 with PID 0 on node chandra calling "abort". This may have caused other processes in the application to be terminated by signals sent by prterun (as reported here). -------------------------------------------------------------------------- Traceback (most recent call last): File "/Users/praveen/Applications/clawpack/clawutil/src/python/clawutil/runclaw.py", line 242, in runclaw proc = subprocess.check_call(cmd_split, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib/python3.12/subprocess.py", line 413, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '['/opt/homebrew/Caskroom/miniforge/base/envs/claw/./bin/mpiexec', '-n', '6', '/Users/praveen/Work/bouss/radial_flat/xgeoclaw']' returned non-zero exit status 59. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/Users/praveen/Applications/clawpack/clawutil/src/python/clawutil/runclaw.py", line 341, in runclaw(*args) File "/Users/praveen/Applications/clawpack/clawutil/src/python/clawutil/runclaw.py", line 249, in runclaw raise ClawExeError(exe_error_str, cpe.returncode, cpe.cmd, ClawExeError: *** FORTRAN EXE FAILED *** make[1]: *** [output] Error 1 make: *** [.output] Error 2 > On 30 Oct 2024, at 9:33?PM, Barry Smith wrote: > > > Please put > > -mpi_linear_solver_server_use_shared_memory false > > into the petscMPIOptions file and see if that changes anything. > > Barry > > >> On Oct 30, 2024, at 9:05?AM, Praveen C wrote: >> >> I have attached some files for this >> >> cd bouss >> . setenv.sh # some settings in this file may need to be changed >> >> cd radial_flat/1d_radial >> make .output >> cd .. >> make .output >> >> Thanks >> praveen >> >> >> >> >>> On 30 Oct 2024, at 6:28?PM, Barry Smith wrote: >>> >>> >>> Please send me exact instructions on how you are running the geoclaw example, when I run >>> >>> geoclaw/examples/bouss/radial_flat >>> >>> >>> it only runs with one level, but never errors, I changed setrun.py to amrdata.amr_levels_max = 5 but it made no difference. >>> >>> Thanks >>> >>> Barry >>> >>> >>> >>>> On Oct 24, 2024, at 12:17?PM, Praveen C wrote: >>>> >>>> Hello Barry >>>> >>>> I use this script to install clawpack and required dependencies >>>> >>>> https://urldefense.us/v3/__https://github.com/cpraveen/cfdlab/blob/master/bin/clawpack.sh__;!!G_uCfscf7eWS!YfrSA6JCkK1gfGgtTWPw-8jHlRI4KhOjRVRmncpA01ImURbJtFEQuwPz9Y6AYUfTMVo0CPQxqXWVhunZ-SMX1Q$ >>>> >>>> See lines 125-132 >>>> >>>> You can use it like this >>>> >>>> export CLAW=/path/to/where/you/want/clawpack >>>> bash clawpack.sh v5.11.0 >>>> >>>> This will git pull clawpack and creates a conda env called ?claw? and installs inside that. >>>> >>>> I have not specified petsc version in this script, but latest miniforge should install petsc at 3.22 >>>> >>>> Thank you >>>> praveen >>>> >>>>> On 24 Oct 2024, at 9:37?PM, Barry Smith wrote: >>>>> >>>>> >>>>> Good, some helpful information with the runs you made. >>>>> >>>>> In the crash below it made more progress it was able to allocate multiple regions of shared memory and access them. I don't know why it would crash later. >>>>> >>>>> Can you tell me all the steps with miniforge (which seems to be related to the failure) you use? I've never used miniforge. >>>>> >>>>> If I can get to an environment that reproduces the problem I can debug it and fix it. >>>>> >>>>> Barry >>>>> >>>>> >>>>>> On Oct 24, 2024, at 11:49?AM, Praveen C wrote: >>>>>> >>>>>> I get this >>>>>> >>>>>> $ mpiexec -n 3 ./ex89f -n 20 -mpi_linear_solver_server -mpi_linear_solver_server -mpi_linear_solver_server_ksp_view -ksp_monitor -ksp_converged_reason -ksp_view -mpi_linear_solver_server_minimum_count_per_rank 5 >>>>>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>>>>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range >>>>>> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >>>>>> [0]PETSC ERROR: or see https://urldefense.us/v3/__https://petsc.org/release/faq/*valgrind__;Iw!!G_uCfscf7eWS!YfrSA6JCkK1gfGgtTWPw-8jHlRI4KhOjRVRmncpA01ImURbJtFEQuwPz9Y6AYUfTMVo0CPQxqXWVhulpyMltnQ$ and https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!YfrSA6JCkK1gfGgtTWPw-8jHlRI4KhOjRVRmncpA01ImURbJtFEQuwPz9Y6AYUfTMVo0CPQxqXWVhukefFfCQA$ >>>>>> [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run >>>>>> [0]PETSC ERROR: to get more information on the crash. >>>>>> [0]PETSC ERROR: Run with -malloc_debug to check if memory corruption is causing the crash. >>>>>> -------------------------------------------------------------------------- >>>>>> MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_SELF >>>>>> Proc: [[47380,1],0] >>>>>> Errorcode: 59 >>>>>> >>>>>> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. >>>>>> You may or may not see output from other processes, depending on >>>>>> exactly when Open MPI kills them. >>>>>> -------------------------------------------------------------------------- >>>>>> -------------------------------------------------------------------------- >>>>>> prterun has exited due to process rank 0 with PID 0 on node MacMiniHome calling >>>>>> "abort". This may have caused other processes in the application to be >>>>>> terminated by signals sent by prterun (as reported here). >>>>>> ????????????????????????????????????? >>>>>> >>>>>> and I have this after the code exits >>>>>> >>>>>> $ ipcs -m >>>>>> IPC status from as of Thu Oct 24 21:17:39 IST 2024 >>>>>> T ID KEY MODE OWNER GROUP >>>>>> Shared Memory: >>>>>> m 1572864 0x0000000b --rw-rw-rw- praveen staff >>>>>> m 524289 0x0000000c --rw-rw-rw- praveen staff >>>>>> m 655362 0x0000000d --rw-rw-rw- praveen staff >>>>>> m 262147 0x0000000e --rw-rw-rw- praveen staff >>>>>> m 262148 0x0000000f --rw-rw-rw- praveen staff >>>>>> m 393221 0x0000000a --rw-rw-rw- praveen staff >>>>>> >>>>>> This is with petsc installed with miniforge, which I also use with clawpack. With spack installed petsc, I can run the ex89f example. >>>>>> >>>>>> Thanks >>>>>> praveen >>>>>> >>>>>>> On 24 Oct 2024, at 7:55?PM, Barry Smith wrote: >>>>>>> >>>>>>> >>>>>>> Ok, super strange, I've run many times on my Mac >>>>>>> >>>>>>> Can you please try to remove that allocated memory with ipcrm and then >>>>>>> >>>>>>> cd $PETSC_DIR/src/ksp/ksp/tutorials >>>>>>> make ex89f >>>>>>> mpiexec -n 3 ./ex89f -n 20 -mpi_linear_solver_server -mpi_linear_solver_server -mpi_linear_solver_server_ksp_view -ksp_monitor -ksp_converged_reason -ksp_view -mpi_linear_solver_server_minimum_count_per_rank 5 >>>>>>> >>>>>>> This does the same thing as the GeoClaw code but is much simpler. >>>>>>> >>>>>>> Barry >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> On Oct 23, 2024, at 10:55?PM, Praveen C wrote: >>>>>>>> >>>>>>>> I get very similar error on my mac with >>>>>>>> >>>>>>>> $ gfortran -v >>>>>>>> Using built-in specs. >>>>>>>> COLLECT_GCC=gfortran >>>>>>>> COLLECT_LTO_WRAPPER=/opt/homebrew/Caskroom/miniforge/base/envs/claw/libexec/gcc/arm64-apple-darwin20.0.0/13.2.0/lto-wrapper >>>>>>>> Target: arm64-apple-darwin20.0.0 >>>>>>>> Configured with: ../configure --prefix=/opt/homebrew/Caskroom/miniforge/base/envs/claw --build=x86_64-apple-darwin13.4.0 --host=arm64-apple-darwin20.0.0 --target=arm64-apple-darwin20.0.0 --with-libiconv-prefix=/opt/homebrew/Caskroom/miniforge/base/envs/claw --enable-languages=fortran --disable-multilib --enable-checking=release --disable-bootstrap --disable-libssp --with-gmp=/opt/homebrew/Caskroom/miniforge/base/envs/claw --with-mpfr=/opt/homebrew/Caskroom/miniforge/base/envs/claw --with-mpc=/opt/homebrew/Caskroom/miniforge/base/envs/claw --with-isl=/opt/homebrew/Caskroom/miniforge/base/envs/claw --enable-darwin-at-rpath >>>>>>>> Thread model: posix >>>>>>>> Supported LTO compression algorithms: zlib >>>>>>>> gcc version 13.2.0 (GCC) >>>>>>>> >>>>>>>> Before starting >>>>>>>> >>>>>>>> $ ipcs -m >>>>>>>> IPC status from as of Thu Oct 24 08:02:11 IST 2024 >>>>>>>> T ID KEY MODE OWNER GROUP >>>>>>>> Shared Memory: >>>>>>>> >>>>>>>> and when I run the code >>>>>>>> >>>>>>>> Using a PETSc solver >>>>>>>> Using Bouss equations from the start >>>>>>>> rnode allocated... >>>>>>>> node allocated... >>>>>>>> listOfGrids allocated... >>>>>>>> Storage allocated... >>>>>>>> bndList allocated... >>>>>>>> Gridding level 1 at t = 0.000000E+00: 4 grids with 10000 cells >>>>>>>> Setting initial dt to 2.9999999999999999E-002 >>>>>>>> max threads set to 1 >>>>>>>> >>>>>>>> Done reading data, starting computation ... >>>>>>>> >>>>>>>> Total zeta at initial time: 39269.907650665169 >>>>>>>> GEOCLAW: Frame 0 output files done at time t = 0.000000D+00 >>>>>>>> >>>>>>>> [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >>>>>>>> [0]PETSC ERROR: Petsc has generated inconsistent data >>>>>>>> [0]PETSC ERROR: Unable to locate PCMPI allocated shared address 0x130698000 >>>>>>>> [0]PETSC ERROR: WARNING! There are unused option(s) set! Could be the program crashed before usage or a spelling mistake, etc! >>>>>>>> [0]PETSC ERROR: Option left: name:-ksp_type value: preonly source: file >>>>>>>> [0]PETSC ERROR: Option left: name:-mpi_ksp_max_it value: 200 source: file >>>>>>>> [0]PETSC ERROR: Option left: name:-mpi_ksp_reuse_preconditioner (no value) source: file >>>>>>>> [0]PETSC ERROR: Option left: name:-mpi_ksp_rtol value: 1.e-9 source: file >>>>>>>> [0]PETSC ERROR: Option left: name:-mpi_ksp_type value: gmres source: file >>>>>>>> [0]PETSC ERROR: Option left: name:-mpi_linear_solver_server_view (no value) source: file >>>>>>>> [0]PETSC ERROR: Option left: name:-mpi_pc_gamg_sym_graph value: true source: file >>>>>>>> [0]PETSC ERROR: Option left: name:-mpi_pc_gamg_symmetrize_graph value: true source: file >>>>>>>> [0]PETSC ERROR: Option left: name:-mpi_pc_type value: gamg source: file >>>>>>>> [0]PETSC ERROR: Option left: name:-pc_mpi_minimum_count_per_rank value: 5000 source: file >>>>>>>> [0]PETSC ERROR: Option left: name:-pc_type value: mpi source: file >>>>>>>> [0]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!YfrSA6JCkK1gfGgtTWPw-8jHlRI4KhOjRVRmncpA01ImURbJtFEQuwPz9Y6AYUfTMVo0CPQxqXWVhukefFfCQA$ for trouble shooting. >>>>>>>> [0]PETSC ERROR: Petsc Release Version 3.22.0, Sep 28, 2024 >>>>>>>> [0]PETSC ERROR: /Users/praveen/work/bouss/radial_flat/xgeoclaw with 6 MPI process(es) and PETSC_ARCH on MacMiniHome.local by praveen Thu Oct 24 08:04:27 2024 >>>>>>>> [0]PETSC ERROR: Configure options: AR=arm64-apple-darwin20.0.0-ar CC=mpicc CXX=mpicxx FC=mpifort CFLAGS="-ftree-vectorize -fPIC -fstack-protector-strong -O2 -pipe -isystem /opt/homebrew/Caskroom/miniforge/base/envs/claw/include " CPPFLAGS="-D_FORTIFY_SOURCE=2 -isystem /opt/homebrew/Caskroom/miniforge/base/envs/claw/include -mmacosx-version-min=11.0 -mmacosx-version-min=11.0" CXXFLAGS="-ftree-vectorize -fPIC -fstack-protector-strong -O2 -pipe -stdlib=libc++ -fvisibility-inlines-hidden -fmessage-length=0 -isystem /opt/homebrew/Caskroom/miniforge/base/envs/claw/include " FFLAGS="-march=armv8.3-a -ftree-vectorize -fPIC -fno-stack-protector -O2 -pipe -isystem /opt/homebrew/Caskroom/miniforge/base/envs/claw/include " LDFLAGS="-Wl,-headerpad_max_install_names -Wl,-dead_strip_dylibs -Wl,-rpath,/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib -L/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib" LIBS="-Wl,-rpath,/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib -lmpi_mpifh -lgfortran" --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 --FOPTFLAGS=-O3 --with-clib-autodetect=0 --with-cxxlib-autodetect=0 --with-fortranlib-autodetect=0 --with-debugging=0 --with-blas-lib=libblas.dylib --with-lapack-lib=liblapack.dylib --with-yaml=1 --with-hdf5=1 --with-fftw=1 --with-hwloc=0 --with-hypre=1 --with-metis=1 --with-mpi=1 --with-mumps=1 --with-parmetis=1 --with-pthread=1 --with-ptscotch=1 --with-shared-libraries --with-ssl=0 --with-scalapack=1 --with-superlu=1 --with-superlu_dist=1 --with-superlu_dist-include=/opt/homebrew/Caskroom/miniforge/base/envs/claw/include/superlu-dist --with-superlu_dist-lib=-lsuperlu_dist --with-suitesparse=1 --with-suitesparse-dir=/opt/homebrew/Caskroom/miniforge/base/envs/claw --with-x=0 --with-scalar-type=real --with-cuda=0 --with-batch --prefix=/opt/homebrew/Caskroom/miniforge/base/envs/claw >>>>>>>> [0]PETSC ERROR: #1 PetscShmgetMapAddresses() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/sys/utils/server.c:114 >>>>>>>> [0]PETSC ERROR: #2 PCMPISetMat() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/ksp/pc/impls/mpi/pcmpi.c:269 >>>>>>>> [0]PETSC ERROR: #3 PCSetUp_MPI() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/ksp/pc/impls/mpi/pcmpi.c:853 >>>>>>>> [0]PETSC ERROR: #4 PCSetUp() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/ksp/pc/interface/precon.c:1071 >>>>>>>> [0]PETSC ERROR: #5 KSPSetUp() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/ksp/ksp/interface/itfunc.c:415 >>>>>>>> [0]PETSC ERROR: #6 KSPSolve_Private() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/ksp/ksp/interface/itfunc.c:826 >>>>>>>> [0]PETSC ERROR: #7 KSPSolve() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/ksp/ksp/interface/itfunc.c:1075 >>>>>>>> >>>>>>>> Code does not progress and I kill it >>>>>>>> >>>>>>>> ^CTraceback (most recent call last): >>>>>>>> File "/Users/praveen/Applications/clawpack/clawutil/src/python/clawutil/runclaw.py", line 341, in >>>>>>>> runclaw(*args) >>>>>>>> File "/Users/praveen/Applications/clawpack/clawutil/src/python/clawutil/runclaw.py", line 242, in runclaw >>>>>>>> proc = subprocess.check_call(cmd_split, >>>>>>>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ >>>>>>>> File "/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib/python3.12/subprocess.py", line 408, in check_call >>>>>>>> retcode = call(*popenargs, **kwargs) >>>>>>>> ^^^^^^^^^^^^^^^^^^^^^^^^^^ >>>>>>>> File "/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib/python3.12/subprocess.py", line 391, in call >>>>>>>> return p.wait(timeout=timeout) >>>>>>>> ^^^^^^^^^^^^^^^^^^^^^^^ >>>>>>>> File "/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib/python3.12/subprocess.py", line 1264, in wait >>>>>>>> return self._wait(timeout=timeout) >>>>>>>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^ >>>>>>>> File "/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib/python3.12/subprocess.py", line 2053, in _wait >>>>>>>> (pid, sts) = self._try_wait(0) >>>>>>>> ^^^^^^^^^^^^^^^^^ >>>>>>>> File "/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib/python3.12/subprocess.py", line 2011, in _try_wait >>>>>>>> (pid, sts) = os.waitpid(self.pid, wait_flags) >>>>>>>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ >>>>>>>> KeyboardInterrupt >>>>>>>> make[1]: *** [output] Interrupt: 2 >>>>>>>> make: *** [.output] Interrupt: 2 >>>>>>>> >>>>>>>> Now it says >>>>>>>> >>>>>>>> $ ipcs -m >>>>>>>> IPC status from as of Thu Oct 24 08:05:06 IST 2024 >>>>>>>> T ID KEY MODE OWNER GROUP >>>>>>>> Shared Memory: >>>>>>>> m 720896 0x0000000a --rw-rw-rw- praveen staff >>>>>>>> >>>>>>>> Thanks >>>>>>>> praveen >>>>>>>> >>>>>>>>> On 23 Oct 2024, at 8:26?PM, Barry Smith wrote: >>>>>>>>> >>>>>>>>> >>>>>>>>> Hmm, so it is creating the first shared memory region with ID of 10 (A in hex) puts it in a linked list in PETSc but then when it tries to find it in the linked list it cannot find it. >>>>>>>>> >>>>>>>>> I don't know how to reproduce this or debug it remotely. >>>>>>>>> >>>>>>>>> Can you build on a completely different machine or with completely different compilers? >>>>>>>>> >>>>>>>>> Barry >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> On Oct 23, 2024, at 10:31?AM, Praveen C wrote: >>>>>>>>>> >>>>>>>>>> I get same error and now it shows >>>>>>>>>> >>>>>>>>>> $ ipcs -m >>>>>>>>>> >>>>>>>>>> ------ Shared Memory Segments -------- >>>>>>>>>> key shmid owner perms bytes nattch status >>>>>>>>>> 0x0000000a 32788 praveen 666 240 6 >>>>>>>>>> >>>>>>>>>> Note that the code seems to be still running after printing those error message, but it is not printing any progress which it should do. >>>>>>>>>> >>>>>>>>>> Thanks >>>>>>>>>> praveen >>>>>>>>>> >>>>>>>>>>> On 23 Oct 2024, at 7:56?PM, Barry Smith wrote: >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Try >>>>>>>>>>> >>>>>>>>>>> ipcrm -m 11 >>>>>>>>>>> >>>>>>>>>>> ipcs -m >>>>>>>>>>> >>>>>>>>>>> Try running the program again >>>>>>>>>>> >>>>>>>>>>> If failed check >>>>>>>>>>> >>>>>>>>>>> ipcs -m >>>>>>>>>>> >>>>>>>>>>> again >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> On Oct 23, 2024, at 10:20?AM, Praveen C wrote: >>>>>>>>>>>> >>>>>>>>>>>> Hello Barry >>>>>>>>>>>> >>>>>>>>>>>> I see this >>>>>>>>>>>> >>>>>>>>>>>> $ ipcs -m >>>>>>>>>>>> >>>>>>>>>>>> ------ Shared Memory Segments -------- >>>>>>>>>>>> key shmid owner perms bytes nattch status >>>>>>>>>>>> 0x0000000a 11 praveen 666 240 6 >>>>>>>>>>>> >>>>>>>>>>>> and I am observing same error as below. >>>>>>>>>>>> >>>>>>>>>>>> Thanks >>>>>>>>>>>> praveen >>>>>>>>>>>> >>>>>>>>>>>>> On 23 Oct 2024, at 7:08?PM, Barry Smith wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Please take a look at the notes in https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/PetscShmgetAllocateArray/__;!!G_uCfscf7eWS!YfrSA6JCkK1gfGgtTWPw-8jHlRI4KhOjRVRmncpA01ImURbJtFEQuwPz9Y6AYUfTMVo0CPQxqXWVhumk7U9-wA$ For some reason your program is not able to access/use the Unix shared memory; check if you are already using the shared memory (so it is not available for a new run) or the limits are too low to access enough memory. >>>>>>>>>>>>> >>>>>>>>>>>>> Barry >>>>>>>>>>>>> >>>>>>>>>>>>>> On Oct 23, 2024, at 8:23?AM, Praveen C wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> Dear all >>>>>>>>>>>>>> >>>>>>>>>>>>>> I am not able to run the boussinesq example from geoclaw using petsc at 3.22.0 >>>>>>>>>>>>>> >>>>>>>>>>>>>> https://urldefense.us/v3/__https://github.com/clawpack/geoclaw/tree/3303883f46572c58130d161986b8a87a57ca7816/examples/bouss__;!!G_uCfscf7eWS!e3VQ4NHKmXGstRsQW5vtI7fmKfUT9zmJkMJcPbcvPyIjicyfJpNoMgx3wZ-qyGcKNSjIkNZkzilec8MnHN6PMw$ >>>>>>>>>>>>>> >>>>>>>>>>>>>> It runs with petsc at 3.21.6 >>>>>>>>>>>>>> >>>>>>>>>>>>>> The error I get is given below. After printing this, the code does not progress. >>>>>>>>>>>>>> >>>>>>>>>>>>>> I use the following petsc options >>>>>>>>>>>>>> >>>>>>>>>>>>>> # set min numbers of matrix rows per MPI rank (default is 10000) >>>>>>>>>>>>>> -mpi_linear_solve_minimum_count_per_rank 5000 >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> # Krylov linear solver: >>>>>>>>>>>>>> -mpi_linear_solver_server >>>>>>>>>>>>>> -mpi_linear_solver_server_view >>>>>>>>>>>>>> -ksp_type gmres >>>>>>>>>>>>>> -ksp_max_it 200 >>>>>>>>>>>>>> -ksp_reuse_preconditioner >>>>>>>>>>>>>> -ksp_rtol 1.e-9 >>>>>>>>>>>>>> >>>>>>>>>>>>>> # preconditioner: >>>>>>>>>>>>>> -pc_type gamg >>>>>>>>>>>>>> >>>>>>>>>>>>>> I installed petsc and other dependencies for clawpack using miniforge. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks >>>>>>>>>>>>>> pc >>>>>>>>>>>>>> >>>>>>>>>>>>>> ==> Use Bouss. in water deeper than 1.0000000000000000 Using a PETSc solver >>>>>>>>>>>>>> Using Bouss equations from the start >>>>>>>>>>>>>> rnode allocated... >>>>>>>>>>>>>> node allocated... >>>>>>>>>>>>>> listOfGrids allocated... >>>>>>>>>>>>>> Storage allocated... >>>>>>>>>>>>>> bndList allocated... >>>>>>>>>>>>>> Gridding level 1 at t = 0.000000E+00: 4 grids with 10000 cells >>>>>>>>>>>>>> Setting initial dt to 2.9999999999999999E-002 >>>>>>>>>>>>>> max threads set to 6 >>>>>>>>>>>>>> Done reading data, starting computation ... Total zeta at initial time: 39269.907650665169 GEOCLAW: Frame 0 output files done at time t = 0.000000D+00 >>>>>>>>>>>>>> >>>>>>>>>>>>>> [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >>>>>>>>>>>>>> [0]PETSC ERROR: Petsc has generated inconsistent data >>>>>>>>>>>>>> [0]PETSC ERROR: Unable to locate PCMPI allocated shared address 0x55e6d750ae20 >>>>>>>>>>>>>> [0]PETSC ERROR: WARNING! There are unused option(s) set! Could be the program crashed before usage or a spelling mistake, etc! >>>>>>>>>>>>>> [0]PETSC ERROR: Option left: name:-ksp_max_it value: 200 source: file >>>>>>>>>>>>>> [0]PETSC ERROR: Option left: name:-ksp_reuse_preconditioner (no value) source: file >>>>>>>>>>>>>> [0]PETSC ERROR: Option left: name:-ksp_rtol value: 1.e-9 source: file >>>>>>>>>>>>>> [0]PETSC ERROR: Option left: name:-ksp_type value: gmres source: file >>>>>>>>>>>>>> [0]PETSC ERROR: Option left: name:-mpi_linear_solve_minimum_count_per_rank value: 5000 source: file >>>>>>>>>>>>>> [0]PETSC ERROR: Option left: name:-mpi_linear_solver_server_view (no value) source: file >>>>>>>>>>>>>> [0]PETSC ERROR: Option left: name:-pc_type value: gamg source: file >>>>>>>>>>>>>> [0]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!e3VQ4NHKmXGstRsQW5vtI7fmKfUT9zmJkMJcPbcvPyIjicyfJpNoMgx3wZ-qyGcKNSjIkNZkzilec8MvNjNo7A$ for trouble shooting. >>>>>>>>>>>>>> [0]PETSC ERROR: Petsc Release Version 3.22.0, Sep 28, 2024 [0]PETSC ERROR: /home/praveen/bouss/radial_flat/xgeoclaw with 6 MPI process(es) and PETSC_ARCH on euler by praveen Thu Oct 17 21:49:54 2024 >>>>>>>>>>>>>> [0]PETSC ERROR: Configure options: AR=${PREFIX}/bin/x86_64-conda-linux-gnu-ar CC=mpicc CXX=mpicxx FC=mpifort CFLAGS="-march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /opt/miniforge/envs/claw/include " CPPFLAGS="-DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /opt/miniforge/envs/claw/include" CXXFLAGS="-fvisibility-inlines-hidden -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /opt/miniforge/envs/claw/include " FFLAGS="-march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /opt/miniforge/envs/claw/include -Wl,--no-as-needed" LDFLAGS="-pthread -fopenmp -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,--allow-shlib-undefined -Wl,-rpath,/opt/miniforge/envs/claw/lib -Wl,-rpath-link,/opt/miniforge/envs/claw/lib -L/opt/miniforge/envs/claw/lib -Wl,-rpath-link,/opt/miniforge/envs/claw/lib" LIBS="-Wl,-rpath,/opt/miniforge/envs/claw/lib -lmpi_mpifh -lgfortran" --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 --FOPTFLAGS=-O3 --with-clib-autodetect=0 --with-cxxlib-autodetect=0 --with-fortranlib-autodetect=0 --with-debugging=0 --with-blas-lib=libblas.so --with-lapack-lib=liblapack.so --with-yaml=1 --with-hdf5=1 --with-fftw=1 --with-hwloc=0 --with-hypre=1 --with-metis=1 --with-mpi=1 --with-mumps=1 --with-parmetis=1 --with-pthread=1 --with-ptscotch=1 --with-shared-libraries --with-ssl=0 --with-scalapack=1 --with-superlu=1 --with-superlu_dist=1 --with-superlu_dist-include=/opt/miniforge/envs/claw/include/superlu-dist --with-superlu_dist-lib=-lsuperlu_dist --with-suitesparse=1 --with-suitesparse-dir=/opt/miniforge/envs/claw --with-x=0 --with-scalar-type=real --with-cuda=0 --prefix=/opt/miniforge/envs/claw >>>>>>>>>>>>>> [0]PETSC ERROR: #1 PetscShmgetMapAddresses() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/sys/utils/server.c:114 >>>>>>>>>>>>>> [0]PETSC ERROR: #2 PCMPISetMat() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/pc/impls/mpi/pcmpi.c:269 >>>>>>>>>>>>>> [0]PETSC ERROR: #3 PCSetUp_MPI() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/pc/impls/mpi/pcmpi.c:853 >>>>>>>>>>>>>> [0]PETSC ERROR: #4 PCSetUp() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/pc/interface/precon.c:1071 >>>>>>>>>>>>>> [0]PETSC ERROR: #5 KSPSetUp() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/ksp/interface/itfunc.c:415 >>>>>>>>>>>>>> [0]PETSC ERROR: #6 KSPSolve_Private() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/ksp/interface/itfunc.c:826 >>>>>>>>>>>>>> [0]PETSC ERROR: #7 KSPSolve() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/ksp/interface/itfunc.c:1075 >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Thu Oct 31 06:30:33 2024 From: mfadams at lbl.gov (Mark Adams) Date: Thu, 31 Oct 2024 07:30:33 -0400 Subject: [petsc-users] Expected weak scaling behaviour for AMG libraries? In-Reply-To: References: Message-ID: As Matt said snes ex56 is better because it does a convergence test that refines the grid. You need/want these two parameters to have the same arg (eg, 2,2,1): -dm_plex_box_faces 2,2,1 -petscpartitioner_simple_process_grid 2,2,1. This will put one cell per process. Then you use: -max_conv_its N, to specify the N levels of refinement to do. It will run the 2,2,1 first then a 4,4,2, etc., N times. /src/snes/tests/ex13.c is designed for benchmarking and it uses '-petscpartitioner_simple_node_grid 1,1,1 [default]' to give you a two level partitioner. You need to have dm_plex_box_faces_i = petscpartitioner_simple_process_grid_i * petscpartitioner_simple_node_grid_i Again, you should put one cell per process (NP = product of dm_plex_box_faces args) and use -dm_refine N to get a single solve. Mark On Wed, Oct 30, 2024 at 11:02?PM Matthew Knepley wrote: > On Wed, Oct 30, 2024 at 4:13?PM Khurana, Parv > wrote: > >> Hello PETSc Community, >> I am trying to understand the scaling behaviour of AMG methods in PETSc >> (Hypre for now) and how many DOFs/Rank are needed for a performant AMG >> solve. >> I?m currently conducting weak scaling tests using >> src/snes/tutorials/ex12.c in 3D, applying Dirichlet BCs with FEM at P=1. >> The tests keep DOFs per processor constant while increasing the mesh size >> and processor count, specifically: >> >> - *20000 and 80000 DOF/RANK* configurations. >> - Running SNES twice, using GMRES with a tolerance of 1e-5 and >> preconditioning with Hypre-BoomerAMG. >> >> A couple of quick points in order to make sure that there is no > confusion: > > 1) Partitioner type "simple" is for the CI. It is a very bad partition, > and should not be used for timing. The default is ParMetis which should be > good enough. > > 2) You start out with 6^3 = 216 elements, distribute that, and then refine > it. This will be _really_ bad load balance on all arrangement except the > divisors of 216. You usually want to start out with something bigger at the > later stages. You can use -dm_refine_pre to refine before distribution. > > 3) It is not clear you are using the timing for just the solver > (SNESSolve). It could be that extraneous things are taking time. When > asking questions like this, please always send the output of -log_view for > timing, and at least -ksp_monitor_true_residial for convergence. > > 4) SNES ex56 is the example we use for GAMG scalability testing > > Thanks, > > Matt > >> Unfortunately, parallel efficiency degrades noticeably with increased >> processor counts. Are there any insights or rules of thumb for using AMG >> more effectively? I have been looking at this issue for a while >> now and would love to engage in a further discussion. Please find below the >> weak scaling results and the options I use to run the tests. >> *#Run type* >> -run_type full >> -petscpartitioner_type simple >> >> *#Mesh settings* >> -dm_plex_dim 3 >> -dm_plex_simplex 1 >> -dm_refine 5 #Varied this >> -dm_plex_box_faces 6,6,6 >> >> *#BCs and FEM space* >> -bc_type dirichlet >> -petscspace_degree 1 >> >> *#Solver settings* >> -snes_max_it 2 >> -ksp_type gmres >> -ksp_rtol 1.0e-5 >> #Same settings as what we use for LOR >> -pc_type hypre >> -pc_hypre_type boomeramg >> -pc_hypre_boomeramg_coarsen_type hmis >> -pc_hypre_boomeramg_relax_type_all symmetric-sor/jacobi >> -pc_hypre_boomeramg_strong_threshold 0.7 >> -pc_hypre_boomeramg_interp_type ext+i >> -pc_hypre_boomeramg_P_max 2 >> -pc_hypre_boomeramg_truncfactor 0.3 >> >> Best, >> Parv >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!cMOkz8-C-z7mPu1_ju5K0omHBE-cuHEm-yHUPTRQIOlOoAPQGVBhPIB-FD1VU_y4lJ1LMbJE6S3RyDw3TSCCdDg$ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image.png Type: image/png Size: 119488 bytes Desc: not available URL: From mfadams at lbl.gov Thu Oct 31 06:41:20 2024 From: mfadams at lbl.gov (Mark Adams) Date: Thu, 31 Oct 2024 07:41:20 -0400 Subject: [petsc-users] Rigid body nullspace for Stokes operator In-Reply-To: <87y124q31n.fsf@jedbrown.org> References: <875xpas35p.fsf@jedbrown.org> <87bjz1qtk6.fsf@jedbrown.org> <87y124q31n.fsf@jedbrown.org> Message-ID: Just note, GAMG uses these near null space vectors but hypre does not as far as I know. Hypre does do elasticity so I would just stick with hypre if you just want to solve your problem (and move on) On Thu, Oct 31, 2024 at 1:01?AM Jed Brown wrote: > There is MatSetNearNullSpace, which is used to ensure an approximation > property in coarse levels of multigrid. That should always be set when > dealing with problems like this, regardless of boundary conditions. > Separately, there is MatSetNullSpace, which you should only use if you are > solving a singular system with that null space. > > Note that you need at least three non-colinear points to constrain the > null space so if you, for example, have a Dirichlet condition at only one > point or along one straight line, there will still be a null space. > > Amneet Bhalla writes: > > > I think Mark mentioned this earlier, but I want to make sure that the > rigid > > body null vectors should be specified only when Neumann boundary > conditions > > are used on all boundaries of the domain, correct? Alternatively, if a > > Dirichlet boundary condition is used (on any part of the domain boundary) > > then there is no null space, i.e., the operator is a full rank matrix? > > > > If the above is true, then I think I do not need to specify the rigid > body > > null modes because I am using Dirichlet boundary conditions for the > > velocity solver. > > > > On Wed, Oct 30, 2024 at 12:28?PM Jed Brown wrote: > > > >> Yes to 6 null vectors in 3D, 3 null vectors in 2D. The center of mass > does > >> not need to be identified because you can algebraically orthogonalize > >> (lines 411-420 here). > >> > >> > >> > https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/blob/main/src/dm/impls/plex/plexfem.c?ref_type=heads*L377-425__;Iw!!G_uCfscf7eWS!b0T54ExhpyMjL9O-B63FmKcfUHpYoH0-ixjUnroOoB0PZPSEVH2-LEMb_Yw57sybhA0rknBisFM2erfwfjN2syg$ > >> > >> See also this implementation with raw coordinates. GAMG orthogonalizes > >> within each aggregate (in a later phase of the algorithm) so global > >> orthogonalization is not necessary. > >> > >> > >> > https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/blob/main/src/ksp/pc/impls/gamg/agg.c?ref_type=heads*L387__;Iw!!G_uCfscf7eWS!b0T54ExhpyMjL9O-B63FmKcfUHpYoH0-ixjUnroOoB0PZPSEVH2-LEMb_Yw57sybhA0rknBisFM2erfw3-yhJ68$ > >> > >> Amneet Bhalla writes: > >> > >> > I think the nullspace for the velocity operator is of the form > >> > > >> > vnull = U + ? ? r > >> > in which U is a rigid body velocity and ? is the rigid body rotational > >> > velocity, and r is the radius vector from the center of mass. I > believe I > >> > need to construct 6 nullspace vectors in 3D and 3 nullspace vectors in > >> 2D. > >> > Sounds correct? Also does the center of mass coordinates matter when > >> > defining r? > >> > > >> > On Wed, Oct 30, 2024 at 7:53?AM Amneet Bhalla > >> wrote: > >> > > >> >> @Mark: Is there some document/paper that I can follow to check the > >> algebra > >> >> of these zero eigenvectors/null space modes? > >> >> > >> >> @Jed : We use a projection method preconditioner to solve the coupled > >> >> velocity pressure system as described here ( > >> >> https://urldefense.us/v3/__https://www.sciencedirect.com/science/article/pii/S0021999123004205__;!!G_uCfscf7eWS!b0T54ExhpyMjL9O-B63FmKcfUHpYoH0-ixjUnroOoB0PZPSEVH2-LEMb_Yw57sybhA0rknBisFM2erfwvAPghoI$ > ). > >> It > >> >> is an approximation of the Schur complement. As a part of projection > >> >> preconditioner, we need to solve just the momentum equation > separately > >> >> without considering the pressure part. > >> >> > >> >> On Tue, Oct 29, 2024 at 8:03?PM Jed Brown wrote: > >> >> > >> >>> And to be clear, we recommend using fieldsplit Schur to separate the > >> >>> pressure and velocity part (there are many examples). Applying AMG > >> directly > >> >>> to the saddle point problem will not be a good solver because the > >> >>> heuristics assume positivity and do not preserve inf-sup stability > >> (nor do > >> >>> standard smoothers). > >> >>> > >> >>> Mark Adams writes: > >> >>> > >> >>> > This is linear elasticity and there are 6 "null" vectors (if you > >> removed > >> >>> > Dirichlet boundary conditions these are eigenvectors with zero > >> >>> eigenvalue): > >> >>> > 3 translations, x, y, z, and three rotatiions xx, yy ,zz. > >> >>> > x = (1,0,0,1,0,0,1,0 ...) > >> >>> > and xx is something like (0, z_1, -y_1, 0, z_2, -y_2, ...) where > z_1 > >> is > >> >>> the > >> >>> > z coordinate of the first vertex, etc. > >> >>> > > >> >>> > Mark > >> >>> > > >> >>> > On Tue, Oct 29, 2024 at 3:47?PM Amneet Bhalla < > mail2amneet at gmail.com > >> > > >> >>> wrote: > >> >>> > > >> >>> >> Hi Mark, > >> >>> >> > >> >>> >> Thanks! I am not sure how to construct null space and zero energy > >> modes > >> >>> >> manually for this operator. Is there some theory or > documentation I > >> can > >> >>> >> follow to figure out what the null space and zero energy modes > look > >> >>> like > >> >>> >> for this operator? Once I know what these are in symbolic form, I > >> >>> think I > >> >>> >> should be able to construct them manually. > >> >>> >> > >> >>> >> Best, > >> >>> >> --Amneet > >> >>> >> > >> >>> >> On Tue, Oct 29, 2024 at 7:35?AM Mark Adams > wrote: > >> >>> >> > >> >>> >>> Oh my mistake. You are using staggered grids. So you don't have > a > >> >>> block > >> >>> >>> size that hypre would use for the "nodal" methods. > >> >>> >>> I'm not sure what you are doing exactly, but try hypre and you > >> could > >> >>> >>> create the null space, zero energy modes, manually, attach to > the > >> >>> matrix > >> >>> >>> and try GAMG. > >> >>> >>> You can run with '-info :pc' and grep on GAMG to see if GAMG is > >> >>> picking > >> >>> >>> the null space up (send this output if you can't figure it out). > >> >>> >>> > >> >>> >>> Thanks, > >> >>> >>> Mark > >> >>> >>> > >> >>> >>> On Tue, Oct 29, 2024 at 9:28?AM Mark Adams > >> wrote: > >> >>> >>> > >> >>> >>>> This coordinate interface is just a shortcut for vertex based > >> >>> >>>> discretizations with 3 dof per vertex, etc. (maybe works in > 2D). > >> >>> >>>> You will need to construct the null space vectors manually and > >> >>> attach it > >> >>> >>>> to the matrix. Used by GAMG. > >> >>> >>>> > >> >>> >>>> Note, for hypre you want to use the "nodal" options and it does > >> not > >> >>> use > >> >>> >>>> these null space vectors. That is probably the way you want to > go. > >> >>> >>>> eg: -pc_hypre_boomeramg_nodal_coarsen > >> >>> >>>> > >> >>> >>>> I would run with hypre boomerang and -help and grep on nodal to > >> see > >> >>> all > >> >>> >>>> the "nodal" options and use them. > >> >>> >>>> > >> >>> >>>> Thanks, > >> >>> >>>> Mark > >> >>> >>>> > >> >>> >>>> > >> >>> >>>> On Mon, Oct 28, 2024 at 8:06?PM Amneet Bhalla < > >> mail2amneet at gmail.com > >> >>> > > >> >>> >>>> wrote: > >> >>> >>>> > >> >>> >>>>> Hi Folks, > >> >>> >>>>> > >> >>> >>>>> I am trying to solve the momentum equation in a projection > >> >>> >>>>> preconditioner using GAMG or Hypre solver. The equation looks > >> like > >> >>> for > >> >>> >>>>> velocity variable *v* looks like: > >> >>> >>>>> > >> >>> >>>>> > >> >>> >>>>> [image: Screenshot 2024-10-28 at 4.15.17 PM.png] > >> >>> >>>>> > >> >>> >>>>> Here, ? is spatially varying dynamic viscosity and ? is > spatially > >> >>> >>>>> varying bulk viscosity. I understand that I need to specify > rigid > >> >>> body > >> >>> >>>>> nullspace modes to the multigrid solver in order to accelerate > >> its > >> >>> >>>>> convergence. Looking into this routine > >> >>> MatNullSpaceCreateRigidBody() ( > >> >>> >>>>> > >> >>> > >> > https://urldefense.us/v3/__https://petsc.org/release/manualpages/Mat/MatNullSpaceCreateRigidBody/__;!!G_uCfscf7eWS!bVR6duCoDqPhZrWS-sm1c5qxsFPjZMhdT86AqLpPzWgVy5qoRhd4_Jue2LJOIS6LRrtV2cHGrqger1Yvb-Y5f-0$ > >> >>> >>>>> < > >> >>> > >> > https://urldefense.us/v3/__https://petsc.org/release/manualpages/Mat/MatNullSpaceCreateRigidBody/__;!!G_uCfscf7eWS!eKqgIJjCdMzIU76f7X65AmGxrU_-lC7W02BMWafJ77DNf_IuQk6O1X3qU1x9Ez8NJ20vZEL-mF6T1yNmDnwv0eWa2w$ > >> >>> >), > >> >>> >>>>> I see that I need to provide the coordinates of each node. I > am > >> >>> using > >> >>> >>>>> staggered grid discretization. Do I need to provide > coordinates > >> of > >> >>> >>>>> staggered grid locations? > >> >>> >>>>> > >> >>> >>>>> Thanks, > >> >>> >>>>> -- > >> >>> >>>>> --Amneet > >> >>> >>>>> > >> >>> >>>>> > >> >>> >>>>> > >> >>> >>>>> > >> >>> >> > >> >>> >> -- > >> >>> >> --Amneet > >> >>> >> > >> >>> >> > >> >>> >> > >> >>> >> > >> >>> > >> >> > >> >> > >> >> -- > >> >> --Amneet > >> >> > >> >> > >> >> > >> >> > >> > > >> > -- > >> > --Amneet > >> > > > > > > -- > > --Amneet > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pierre at joliv.et Thu Oct 31 07:32:04 2024 From: pierre at joliv.et (Pierre Jolivet) Date: Thu, 31 Oct 2024 13:32:04 +0100 Subject: [petsc-users] Rigid body nullspace for Stokes operator In-Reply-To: References: <875xpas35p.fsf@jedbrown.org> <87bjz1qtk6.fsf@jedbrown.org> <87y124q31n.fsf@jedbrown.org> Message-ID: > On 31 Oct 2024, at 12:41?PM, Mark Adams wrote: > > Just note, GAMG uses these near null space vectors but hypre does not as far as I know. They do: https://urldefense.us/v3/__https://petsc.org/release/src/ksp/pc/impls/hypre/hypre.c.html*line266__;Iw!!G_uCfscf7eWS!fN2P5tyzz7MnVVKuwdqFol0wnv9A2u6R9B5cIqWr_vYjn31baodY7EqpFqUYLcNSC84acCNBS8cOQxgVTQzDkw$ Thanks, Pierre > Hypre does do elasticity so I would just stick with hypre if you just want to solve your problem (and move on) > > On Thu, Oct 31, 2024 at 1:01?AM Jed Brown > wrote: >> There is MatSetNearNullSpace, which is used to ensure an approximation property in coarse levels of multigrid. That should always be set when dealing with problems like this, regardless of boundary conditions. Separately, there is MatSetNullSpace, which you should only use if you are solving a singular system with that null space. >> >> Note that you need at least three non-colinear points to constrain the null space so if you, for example, have a Dirichlet condition at only one point or along one straight line, there will still be a null space. >> >> Amneet Bhalla > writes: >> >> > I think Mark mentioned this earlier, but I want to make sure that the rigid >> > body null vectors should be specified only when Neumann boundary conditions >> > are used on all boundaries of the domain, correct? Alternatively, if a >> > Dirichlet boundary condition is used (on any part of the domain boundary) >> > then there is no null space, i.e., the operator is a full rank matrix? >> > >> > If the above is true, then I think I do not need to specify the rigid body >> > null modes because I am using Dirichlet boundary conditions for the >> > velocity solver. >> > >> > On Wed, Oct 30, 2024 at 12:28?PM Jed Brown > wrote: >> > >> >> Yes to 6 null vectors in 3D, 3 null vectors in 2D. The center of mass does >> >> not need to be identified because you can algebraically orthogonalize >> >> (lines 411-420 here). >> >> >> >> >> >> https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/blob/main/src/dm/impls/plex/plexfem.c?ref_type=heads*L377-425__;Iw!!G_uCfscf7eWS!fN2P5tyzz7MnVVKuwdqFol0wnv9A2u6R9B5cIqWr_vYjn31baodY7EqpFqUYLcNSC84acCNBS8cOQxjIjp05SQ$ >> >> >> >> See also this implementation with raw coordinates. GAMG orthogonalizes >> >> within each aggregate (in a later phase of the algorithm) so global >> >> orthogonalization is not necessary. >> >> >> >> >> >> https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/blob/main/src/ksp/pc/impls/gamg/agg.c?ref_type=heads*L387__;Iw!!G_uCfscf7eWS!fN2P5tyzz7MnVVKuwdqFol0wnv9A2u6R9B5cIqWr_vYjn31baodY7EqpFqUYLcNSC84acCNBS8cOQxhOaixdSg$ >> >> >> >> Amneet Bhalla > writes: >> >> >> >> > I think the nullspace for the velocity operator is of the form >> >> > >> >> > vnull = U + ? ? r >> >> > in which U is a rigid body velocity and ? is the rigid body rotational >> >> > velocity, and r is the radius vector from the center of mass. I believe I >> >> > need to construct 6 nullspace vectors in 3D and 3 nullspace vectors in >> >> 2D. >> >> > Sounds correct? Also does the center of mass coordinates matter when >> >> > defining r? >> >> > >> >> > On Wed, Oct 30, 2024 at 7:53?AM Amneet Bhalla > >> >> wrote: >> >> > >> >> >> @Mark: Is there some document/paper that I can follow to check the >> >> algebra >> >> >> of these zero eigenvectors/null space modes? >> >> >> >> >> >> @Jed : We use a projection method preconditioner to solve the coupled >> >> >> velocity pressure system as described here ( >> >> >> https://urldefense.us/v3/__https://www.sciencedirect.com/science/article/pii/S0021999123004205__;!!G_uCfscf7eWS!fN2P5tyzz7MnVVKuwdqFol0wnv9A2u6R9B5cIqWr_vYjn31baodY7EqpFqUYLcNSC84acCNBS8cOQxgzS7uh6w$ ). >> >> It >> >> >> is an approximation of the Schur complement. As a part of projection >> >> >> preconditioner, we need to solve just the momentum equation separately >> >> >> without considering the pressure part. >> >> >> >> >> >> On Tue, Oct 29, 2024 at 8:03?PM Jed Brown > wrote: >> >> >> >> >> >>> And to be clear, we recommend using fieldsplit Schur to separate the >> >> >>> pressure and velocity part (there are many examples). Applying AMG >> >> directly >> >> >>> to the saddle point problem will not be a good solver because the >> >> >>> heuristics assume positivity and do not preserve inf-sup stability >> >> (nor do >> >> >>> standard smoothers). >> >> >>> >> >> >>> Mark Adams > writes: >> >> >>> >> >> >>> > This is linear elasticity and there are 6 "null" vectors (if you >> >> removed >> >> >>> > Dirichlet boundary conditions these are eigenvectors with zero >> >> >>> eigenvalue): >> >> >>> > 3 translations, x, y, z, and three rotatiions xx, yy ,zz. >> >> >>> > x = (1,0,0,1,0,0,1,0 ...) >> >> >>> > and xx is something like (0, z_1, -y_1, 0, z_2, -y_2, ...) where z_1 >> >> is >> >> >>> the >> >> >>> > z coordinate of the first vertex, etc. >> >> >>> > >> >> >>> > Mark >> >> >>> > >> >> >>> > On Tue, Oct 29, 2024 at 3:47?PM Amneet Bhalla >> >> > >> >> >>> wrote: >> >> >>> > >> >> >>> >> Hi Mark, >> >> >>> >> >> >> >>> >> Thanks! I am not sure how to construct null space and zero energy >> >> modes >> >> >>> >> manually for this operator. Is there some theory or documentation I >> >> can >> >> >>> >> follow to figure out what the null space and zero energy modes look >> >> >>> like >> >> >>> >> for this operator? Once I know what these are in symbolic form, I >> >> >>> think I >> >> >>> >> should be able to construct them manually. >> >> >>> >> >> >> >>> >> Best, >> >> >>> >> --Amneet >> >> >>> >> >> >> >>> >> On Tue, Oct 29, 2024 at 7:35?AM Mark Adams > wrote: >> >> >>> >> >> >> >>> >>> Oh my mistake. You are using staggered grids. So you don't have a >> >> >>> block >> >> >>> >>> size that hypre would use for the "nodal" methods. >> >> >>> >>> I'm not sure what you are doing exactly, but try hypre and you >> >> could >> >> >>> >>> create the null space, zero energy modes, manually, attach to the >> >> >>> matrix >> >> >>> >>> and try GAMG. >> >> >>> >>> You can run with '-info :pc' and grep on GAMG to see if GAMG is >> >> >>> picking >> >> >>> >>> the null space up (send this output if you can't figure it out). >> >> >>> >>> >> >> >>> >>> Thanks, >> >> >>> >>> Mark >> >> >>> >>> >> >> >>> >>> On Tue, Oct 29, 2024 at 9:28?AM Mark Adams > >> >> wrote: >> >> >>> >>> >> >> >>> >>>> This coordinate interface is just a shortcut for vertex based >> >> >>> >>>> discretizations with 3 dof per vertex, etc. (maybe works in 2D). >> >> >>> >>>> You will need to construct the null space vectors manually and >> >> >>> attach it >> >> >>> >>>> to the matrix. Used by GAMG. >> >> >>> >>>> >> >> >>> >>>> Note, for hypre you want to use the "nodal" options and it does >> >> not >> >> >>> use >> >> >>> >>>> these null space vectors. That is probably the way you want to go. >> >> >>> >>>> eg: -pc_hypre_boomeramg_nodal_coarsen >> >> >>> >>>> >> >> >>> >>>> I would run with hypre boomerang and -help and grep on nodal to >> >> see >> >> >>> all >> >> >>> >>>> the "nodal" options and use them. >> >> >>> >>>> >> >> >>> >>>> Thanks, >> >> >>> >>>> Mark >> >> >>> >>>> >> >> >>> >>>> >> >> >>> >>>> On Mon, Oct 28, 2024 at 8:06?PM Amneet Bhalla < >> >> mail2amneet at gmail.com >> >> >>> > >> >> >>> >>>> wrote: >> >> >>> >>>> >> >> >>> >>>>> Hi Folks, >> >> >>> >>>>> >> >> >>> >>>>> I am trying to solve the momentum equation in a projection >> >> >>> >>>>> preconditioner using GAMG or Hypre solver. The equation looks >> >> like >> >> >>> for >> >> >>> >>>>> velocity variable *v* looks like: >> >> >>> >>>>> >> >> >>> >>>>> >> >> >>> >>>>> [image: Screenshot 2024-10-28 at 4.15.17 PM.png] >> >> >>> >>>>> >> >> >>> >>>>> Here, ? is spatially varying dynamic viscosity and ? is spatially >> >> >>> >>>>> varying bulk viscosity. I understand that I need to specify rigid >> >> >>> body >> >> >>> >>>>> nullspace modes to the multigrid solver in order to accelerate >> >> its >> >> >>> >>>>> convergence. Looking into this routine >> >> >>> MatNullSpaceCreateRigidBody() ( >> >> >>> >>>>> >> >> >>> >> >> https://urldefense.us/v3/__https://petsc.org/release/manualpages/Mat/MatNullSpaceCreateRigidBody/__;!!G_uCfscf7eWS!bVR6duCoDqPhZrWS-sm1c5qxsFPjZMhdT86AqLpPzWgVy5qoRhd4_Jue2LJOIS6LRrtV2cHGrqger1Yvb-Y5f-0$ >> >> >>> >>>>> < >> >> >>> >> >> https://urldefense.us/v3/__https://petsc.org/release/manualpages/Mat/MatNullSpaceCreateRigidBody/__;!!G_uCfscf7eWS!eKqgIJjCdMzIU76f7X65AmGxrU_-lC7W02BMWafJ77DNf_IuQk6O1X3qU1x9Ez8NJ20vZEL-mF6T1yNmDnwv0eWa2w$ >> >> >>> >), >> >> >>> >>>>> I see that I need to provide the coordinates of each node. I am >> >> >>> using >> >> >>> >>>>> staggered grid discretization. Do I need to provide coordinates >> >> of >> >> >>> >>>>> staggered grid locations? >> >> >>> >>>>> >> >> >>> >>>>> Thanks, >> >> >>> >>>>> -- >> >> >>> >>>>> --Amneet >> >> >>> >>>>> >> >> >>> >>>>> >> >> >>> >>>>> >> >> >>> >>>>> >> >> >>> >> >> >> >>> >> -- >> >> >>> >> --Amneet >> >> >>> >> >> >> >>> >> >> >> >>> >> >> >> >>> >> >> >> >>> >> >> >> >> >> >> >> >> >> -- >> >> >> --Amneet >> >> >> >> >> >> >> >> >> >> >> >> >> >> > >> >> > -- >> >> > --Amneet >> >> >> > >> > >> > -- >> > --Amneet -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Thu Oct 31 08:47:54 2024 From: mfadams at lbl.gov (Mark Adams) Date: Thu, 31 Oct 2024 09:47:54 -0400 Subject: [petsc-users] Rigid body nullspace for Stokes operator In-Reply-To: References: <875xpas35p.fsf@jedbrown.org> <87bjz1qtk6.fsf@jedbrown.org> <87y124q31n.fsf@jedbrown.org> Message-ID: Interesting. I have seen hypre do fine on elasticity, but do you know if boomeramg (classical) uses these vectors or is there a smoothed aggregation solver in hypre? On Thu, Oct 31, 2024 at 8:32?AM Pierre Jolivet wrote: > > > On 31 Oct 2024, at 12:41?PM, Mark Adams wrote: > > Just note, GAMG uses these near null space vectors but hypre does not as > far as I know. > > > They do: > https://urldefense.us/v3/__https://petsc.org/release/src/ksp/pc/impls/hypre/hypre.c.html*line266__;Iw!!G_uCfscf7eWS!fgZ28yXAXjJBIqd4SxJehY7SSaYVe_YNHPW4QnnTewSSOxFsPPcBZFPv6Y2IT8PkI3SDEARmixD85cmswwWL0Vk$ > > Thanks, > Pierre > > Hypre does do elasticity so I would just stick with hypre if you just want > to solve your problem (and move on) > > On Thu, Oct 31, 2024 at 1:01?AM Jed Brown wrote: > >> There is MatSetNearNullSpace, which is used to ensure an approximation >> property in coarse levels of multigrid. That should always be set when >> dealing with problems like this, regardless of boundary conditions. >> Separately, there is MatSetNullSpace, which you should only use if you are >> solving a singular system with that null space. >> >> Note that you need at least three non-colinear points to constrain the >> null space so if you, for example, have a Dirichlet condition at only one >> point or along one straight line, there will still be a null space. >> >> Amneet Bhalla writes: >> >> > I think Mark mentioned this earlier, but I want to make sure that the >> rigid >> > body null vectors should be specified only when Neumann boundary >> conditions >> > are used on all boundaries of the domain, correct? Alternatively, if a >> > Dirichlet boundary condition is used (on any part of the domain >> boundary) >> > then there is no null space, i.e., the operator is a full rank matrix? >> > >> > If the above is true, then I think I do not need to specify the rigid >> body >> > null modes because I am using Dirichlet boundary conditions for the >> > velocity solver. >> > >> > On Wed, Oct 30, 2024 at 12:28?PM Jed Brown wrote: >> > >> >> Yes to 6 null vectors in 3D, 3 null vectors in 2D. The center of mass >> does >> >> not need to be identified because you can algebraically orthogonalize >> >> (lines 411-420 here). >> >> >> >> >> >> >> https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/blob/main/src/dm/impls/plex/plexfem.c?ref_type=heads*L377-425__;Iw!!G_uCfscf7eWS!fgZ28yXAXjJBIqd4SxJehY7SSaYVe_YNHPW4QnnTewSSOxFsPPcBZFPv6Y2IT8PkI3SDEARmixD85cms_bElxqQ$ >> >> >> >> >> See also this implementation with raw coordinates. GAMG orthogonalizes >> >> within each aggregate (in a later phase of the algorithm) so global >> >> orthogonalization is not necessary. >> >> >> >> >> >> >> https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/blob/main/src/ksp/pc/impls/gamg/agg.c?ref_type=heads*L387__;Iw!!G_uCfscf7eWS!fgZ28yXAXjJBIqd4SxJehY7SSaYVe_YNHPW4QnnTewSSOxFsPPcBZFPv6Y2IT8PkI3SDEARmixD85cmsTsohckA$ >> >> >> >> >> Amneet Bhalla writes: >> >> >> >> > I think the nullspace for the velocity operator is of the form >> >> > >> >> > vnull = U + ? ? r >> >> > in which U is a rigid body velocity and ? is the rigid body >> rotational >> >> > velocity, and r is the radius vector from the center of mass. I >> believe I >> >> > need to construct 6 nullspace vectors in 3D and 3 nullspace vectors >> in >> >> 2D. >> >> > Sounds correct? Also does the center of mass coordinates matter when >> >> > defining r? >> >> > >> >> > On Wed, Oct 30, 2024 at 7:53?AM Amneet Bhalla > > >> >> wrote: >> >> > >> >> >> @Mark: Is there some document/paper that I can follow to check the >> >> algebra >> >> >> of these zero eigenvectors/null space modes? >> >> >> >> >> >> @Jed : We use a projection method preconditioner to solve the >> coupled >> >> >> velocity pressure system as described here ( >> >> >> https://urldefense.us/v3/__https://www.sciencedirect.com/science/article/pii/S0021999123004205__;!!G_uCfscf7eWS!fgZ28yXAXjJBIqd4SxJehY7SSaYVe_YNHPW4QnnTewSSOxFsPPcBZFPv6Y2IT8PkI3SDEARmixD85cmswf1ygKo$ >> >> ). >> >> It >> >> >> is an approximation of the Schur complement. As a part of projection >> >> >> preconditioner, we need to solve just the momentum equation >> separately >> >> >> without considering the pressure part. >> >> >> >> >> >> On Tue, Oct 29, 2024 at 8:03?PM Jed Brown wrote: >> >> >> >> >> >>> And to be clear, we recommend using fieldsplit Schur to separate >> the >> >> >>> pressure and velocity part (there are many examples). Applying AMG >> >> directly >> >> >>> to the saddle point problem will not be a good solver because the >> >> >>> heuristics assume positivity and do not preserve inf-sup stability >> >> (nor do >> >> >>> standard smoothers). >> >> >>> >> >> >>> Mark Adams writes: >> >> >>> >> >> >>> > This is linear elasticity and there are 6 "null" vectors (if you >> >> removed >> >> >>> > Dirichlet boundary conditions these are eigenvectors with zero >> >> >>> eigenvalue): >> >> >>> > 3 translations, x, y, z, and three rotatiions xx, yy ,zz. >> >> >>> > x = (1,0,0,1,0,0,1,0 ...) >> >> >>> > and xx is something like (0, z_1, -y_1, 0, z_2, -y_2, ...) where >> z_1 >> >> is >> >> >>> the >> >> >>> > z coordinate of the first vertex, etc. >> >> >>> > >> >> >>> > Mark >> >> >>> > >> >> >>> > On Tue, Oct 29, 2024 at 3:47?PM Amneet Bhalla < >> mail2amneet at gmail.com >> >> > >> >> >>> wrote: >> >> >>> > >> >> >>> >> Hi Mark, >> >> >>> >> >> >> >>> >> Thanks! I am not sure how to construct null space and zero >> energy >> >> modes >> >> >>> >> manually for this operator. Is there some theory or >> documentation I >> >> can >> >> >>> >> follow to figure out what the null space and zero energy modes >> look >> >> >>> like >> >> >>> >> for this operator? Once I know what these are in symbolic form, >> I >> >> >>> think I >> >> >>> >> should be able to construct them manually. >> >> >>> >> >> >> >>> >> Best, >> >> >>> >> --Amneet >> >> >>> >> >> >> >>> >> On Tue, Oct 29, 2024 at 7:35?AM Mark Adams >> wrote: >> >> >>> >> >> >> >>> >>> Oh my mistake. You are using staggered grids. So you don't >> have a >> >> >>> block >> >> >>> >>> size that hypre would use for the "nodal" methods. >> >> >>> >>> I'm not sure what you are doing exactly, but try hypre and you >> >> could >> >> >>> >>> create the null space, zero energy modes, manually, attach to >> the >> >> >>> matrix >> >> >>> >>> and try GAMG. >> >> >>> >>> You can run with '-info :pc' and grep on GAMG to see if GAMG is >> >> >>> picking >> >> >>> >>> the null space up (send this output if you can't figure it >> out). >> >> >>> >>> >> >> >>> >>> Thanks, >> >> >>> >>> Mark >> >> >>> >>> >> >> >>> >>> On Tue, Oct 29, 2024 at 9:28?AM Mark Adams >> >> wrote: >> >> >>> >>> >> >> >>> >>>> This coordinate interface is just a shortcut for vertex based >> >> >>> >>>> discretizations with 3 dof per vertex, etc. (maybe works in >> 2D). >> >> >>> >>>> You will need to construct the null space vectors manually and >> >> >>> attach it >> >> >>> >>>> to the matrix. Used by GAMG. >> >> >>> >>>> >> >> >>> >>>> Note, for hypre you want to use the "nodal" options and it >> does >> >> not >> >> >>> use >> >> >>> >>>> these null space vectors. That is probably the way you want >> to go. >> >> >>> >>>> eg: -pc_hypre_boomeramg_nodal_coarsen >> >> >>> >>>> >> >> >>> >>>> I would run with hypre boomerang and -help and grep on nodal >> to >> >> see >> >> >>> all >> >> >>> >>>> the "nodal" options and use them. >> >> >>> >>>> >> >> >>> >>>> Thanks, >> >> >>> >>>> Mark >> >> >>> >>>> >> >> >>> >>>> >> >> >>> >>>> On Mon, Oct 28, 2024 at 8:06?PM Amneet Bhalla < >> >> mail2amneet at gmail.com >> >> >>> > >> >> >>> >>>> wrote: >> >> >>> >>>> >> >> >>> >>>>> Hi Folks, >> >> >>> >>>>> >> >> >>> >>>>> I am trying to solve the momentum equation in a projection >> >> >>> >>>>> preconditioner using GAMG or Hypre solver. The equation looks >> >> like >> >> >>> for >> >> >>> >>>>> velocity variable *v* looks like: >> >> >>> >>>>> >> >> >>> >>>>> >> >> >>> >>>>> [image: Screenshot 2024-10-28 at 4.15.17 PM.png] >> >> >>> >>>>> >> >> >>> >>>>> Here, ? is spatially varying dynamic viscosity and ? is >> spatially >> >> >>> >>>>> varying bulk viscosity. I understand that I need to specify >> rigid >> >> >>> body >> >> >>> >>>>> nullspace modes to the multigrid solver in order to >> accelerate >> >> its >> >> >>> >>>>> convergence. Looking into this routine >> >> >>> MatNullSpaceCreateRigidBody() ( >> >> >>> >>>>> >> >> >>> >> >> >> https://urldefense.us/v3/__https://petsc.org/release/manualpages/Mat/MatNullSpaceCreateRigidBody/__;!!G_uCfscf7eWS!bVR6duCoDqPhZrWS-sm1c5qxsFPjZMhdT86AqLpPzWgVy5qoRhd4_Jue2LJOIS6LRrtV2cHGrqger1Yvb-Y5f-0$ >> >> >>> >>>>> < >> >> >>> >> >> >> https://urldefense.us/v3/__https://petsc.org/release/manualpages/Mat/MatNullSpaceCreateRigidBody/__;!!G_uCfscf7eWS!eKqgIJjCdMzIU76f7X65AmGxrU_-lC7W02BMWafJ77DNf_IuQk6O1X3qU1x9Ez8NJ20vZEL-mF6T1yNmDnwv0eWa2w$ >> >> >>> >), >> >> >>> >>>>> I see that I need to provide the coordinates of each node. I >> am >> >> >>> using >> >> >>> >>>>> staggered grid discretization. Do I need to provide >> coordinates >> >> of >> >> >>> >>>>> staggered grid locations? >> >> >>> >>>>> >> >> >>> >>>>> Thanks, >> >> >>> >>>>> -- >> >> >>> >>>>> --Amneet >> >> >>> >>>>> >> >> >>> >>>>> >> >> >>> >>>>> >> >> >>> >>>>> >> >> >>> >> >> >> >>> >> -- >> >> >>> >> --Amneet >> >> >>> >> >> >> >>> >> >> >> >>> >> >> >> >>> >> >> >> >>> >> >> >> >> >> >> >> >> >> -- >> >> >> --Amneet >> >> >> >> >> >> >> >> >> >> >> >> >> >> > >> >> > -- >> >> > --Amneet >> >> >> > >> > >> > -- >> > --Amneet > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pierre at joliv.et Thu Oct 31 09:01:39 2024 From: pierre at joliv.et (Pierre Jolivet) Date: Thu, 31 Oct 2024 15:01:39 +0100 Subject: [petsc-users] Rigid body nullspace for Stokes operator In-Reply-To: References: <875xpas35p.fsf@jedbrown.org> <87bjz1qtk6.fsf@jedbrown.org> <87y124q31n.fsf@jedbrown.org> Message-ID: <96BE4263-8C34-4A2E-91B4-305F94FFCAB4@joliv.et> > On 31 Oct 2024, at 2:47?PM, Mark Adams wrote: > > Interesting. I have seen hypre do fine on elasticity, but do you know if boomeramg (classical) uses these vectors or is there a smoothed aggregation solver in hypre? I?m not sure it is precisely ?standard? smoothed aggregation, see bottom paragraph of https://urldefense.us/v3/__https://hypre.readthedocs.io/en/latest/solvers-boomeramg.html*amg-for-systems-of-pdes__;Iw!!G_uCfscf7eWS!fdj4AzKhAcDmM1x_ZQ8gxqWeX6BAKY9urnvATMpT7hC8lw77ak7tqxqXGIX3PMg2wYA5PGu7EzyCW0yzixutBg$ I?ve never made it to work, but I know some do. A while back, Stefano gave me this pointer as well: https://urldefense.us/v3/__https://github.com/mfem/mfem/blob/17955e114020af340e9a06a66ebef43e05012d9c/linalg/hypre.cpp*L5245__;Iw!!G_uCfscf7eWS!fdj4AzKhAcDmM1x_ZQ8gxqWeX6BAKY9urnvATMpT7hC8lw77ak7tqxqXGIX3PMg2wYA5PGu7EzyCW0wVEi33Pw$ Thanks, Pierre > On Thu, Oct 31, 2024 at 8:32?AM Pierre Jolivet > wrote: >> >> >>> On 31 Oct 2024, at 12:41?PM, Mark Adams > wrote: >>> >>> Just note, GAMG uses these near null space vectors but hypre does not as far as I know. >> >> They do: https://urldefense.us/v3/__https://petsc.org/release/src/ksp/pc/impls/hypre/hypre.c.html*line266__;Iw!!G_uCfscf7eWS!fdj4AzKhAcDmM1x_ZQ8gxqWeX6BAKY9urnvATMpT7hC8lw77ak7tqxqXGIX3PMg2wYA5PGu7EzyCW0wJINgbAg$ >> >> Thanks, >> Pierre >> >>> Hypre does do elasticity so I would just stick with hypre if you just want to solve your problem (and move on) >>> >>> On Thu, Oct 31, 2024 at 1:01?AM Jed Brown > wrote: >>>> There is MatSetNearNullSpace, which is used to ensure an approximation property in coarse levels of multigrid. That should always be set when dealing with problems like this, regardless of boundary conditions. Separately, there is MatSetNullSpace, which you should only use if you are solving a singular system with that null space. >>>> >>>> Note that you need at least three non-colinear points to constrain the null space so if you, for example, have a Dirichlet condition at only one point or along one straight line, there will still be a null space. >>>> >>>> Amneet Bhalla > writes: >>>> >>>> > I think Mark mentioned this earlier, but I want to make sure that the rigid >>>> > body null vectors should be specified only when Neumann boundary conditions >>>> > are used on all boundaries of the domain, correct? Alternatively, if a >>>> > Dirichlet boundary condition is used (on any part of the domain boundary) >>>> > then there is no null space, i.e., the operator is a full rank matrix? >>>> > >>>> > If the above is true, then I think I do not need to specify the rigid body >>>> > null modes because I am using Dirichlet boundary conditions for the >>>> > velocity solver. >>>> > >>>> > On Wed, Oct 30, 2024 at 12:28?PM Jed Brown > wrote: >>>> > >>>> >> Yes to 6 null vectors in 3D, 3 null vectors in 2D. The center of mass does >>>> >> not need to be identified because you can algebraically orthogonalize >>>> >> (lines 411-420 here). >>>> >> >>>> >> >>>> >> https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/blob/main/src/dm/impls/plex/plexfem.c?ref_type=heads*L377-425__;Iw!!G_uCfscf7eWS!fdj4AzKhAcDmM1x_ZQ8gxqWeX6BAKY9urnvATMpT7hC8lw77ak7tqxqXGIX3PMg2wYA5PGu7EzyCW0wNnvPLMA$ >>>> >> >>>> >> See also this implementation with raw coordinates. GAMG orthogonalizes >>>> >> within each aggregate (in a later phase of the algorithm) so global >>>> >> orthogonalization is not necessary. >>>> >> >>>> >> >>>> >> https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/blob/main/src/ksp/pc/impls/gamg/agg.c?ref_type=heads*L387__;Iw!!G_uCfscf7eWS!fdj4AzKhAcDmM1x_ZQ8gxqWeX6BAKY9urnvATMpT7hC8lw77ak7tqxqXGIX3PMg2wYA5PGu7EzyCW0yNqXfRKQ$ >>>> >> >>>> >> Amneet Bhalla > writes: >>>> >> >>>> >> > I think the nullspace for the velocity operator is of the form >>>> >> > >>>> >> > vnull = U + ? ? r >>>> >> > in which U is a rigid body velocity and ? is the rigid body rotational >>>> >> > velocity, and r is the radius vector from the center of mass. I believe I >>>> >> > need to construct 6 nullspace vectors in 3D and 3 nullspace vectors in >>>> >> 2D. >>>> >> > Sounds correct? Also does the center of mass coordinates matter when >>>> >> > defining r? >>>> >> > >>>> >> > On Wed, Oct 30, 2024 at 7:53?AM Amneet Bhalla > >>>> >> wrote: >>>> >> > >>>> >> >> @Mark: Is there some document/paper that I can follow to check the >>>> >> algebra >>>> >> >> of these zero eigenvectors/null space modes? >>>> >> >> >>>> >> >> @Jed : We use a projection method preconditioner to solve the coupled >>>> >> >> velocity pressure system as described here ( >>>> >> >> https://urldefense.us/v3/__https://www.sciencedirect.com/science/article/pii/S0021999123004205__;!!G_uCfscf7eWS!fdj4AzKhAcDmM1x_ZQ8gxqWeX6BAKY9urnvATMpT7hC8lw77ak7tqxqXGIX3PMg2wYA5PGu7EzyCW0xFZnni7g$ ). >>>> >> It >>>> >> >> is an approximation of the Schur complement. As a part of projection >>>> >> >> preconditioner, we need to solve just the momentum equation separately >>>> >> >> without considering the pressure part. >>>> >> >> >>>> >> >> On Tue, Oct 29, 2024 at 8:03?PM Jed Brown > wrote: >>>> >> >> >>>> >> >>> And to be clear, we recommend using fieldsplit Schur to separate the >>>> >> >>> pressure and velocity part (there are many examples). Applying AMG >>>> >> directly >>>> >> >>> to the saddle point problem will not be a good solver because the >>>> >> >>> heuristics assume positivity and do not preserve inf-sup stability >>>> >> (nor do >>>> >> >>> standard smoothers). >>>> >> >>> >>>> >> >>> Mark Adams > writes: >>>> >> >>> >>>> >> >>> > This is linear elasticity and there are 6 "null" vectors (if you >>>> >> removed >>>> >> >>> > Dirichlet boundary conditions these are eigenvectors with zero >>>> >> >>> eigenvalue): >>>> >> >>> > 3 translations, x, y, z, and three rotatiions xx, yy ,zz. >>>> >> >>> > x = (1,0,0,1,0,0,1,0 ...) >>>> >> >>> > and xx is something like (0, z_1, -y_1, 0, z_2, -y_2, ...) where z_1 >>>> >> is >>>> >> >>> the >>>> >> >>> > z coordinate of the first vertex, etc. >>>> >> >>> > >>>> >> >>> > Mark >>>> >> >>> > >>>> >> >>> > On Tue, Oct 29, 2024 at 3:47?PM Amneet Bhalla >>>> >> > >>>> >> >>> wrote: >>>> >> >>> > >>>> >> >>> >> Hi Mark, >>>> >> >>> >> >>>> >> >>> >> Thanks! I am not sure how to construct null space and zero energy >>>> >> modes >>>> >> >>> >> manually for this operator. Is there some theory or documentation I >>>> >> can >>>> >> >>> >> follow to figure out what the null space and zero energy modes look >>>> >> >>> like >>>> >> >>> >> for this operator? Once I know what these are in symbolic form, I >>>> >> >>> think I >>>> >> >>> >> should be able to construct them manually. >>>> >> >>> >> >>>> >> >>> >> Best, >>>> >> >>> >> --Amneet >>>> >> >>> >> >>>> >> >>> >> On Tue, Oct 29, 2024 at 7:35?AM Mark Adams > wrote: >>>> >> >>> >> >>>> >> >>> >>> Oh my mistake. You are using staggered grids. So you don't have a >>>> >> >>> block >>>> >> >>> >>> size that hypre would use for the "nodal" methods. >>>> >> >>> >>> I'm not sure what you are doing exactly, but try hypre and you >>>> >> could >>>> >> >>> >>> create the null space, zero energy modes, manually, attach to the >>>> >> >>> matrix >>>> >> >>> >>> and try GAMG. >>>> >> >>> >>> You can run with '-info :pc' and grep on GAMG to see if GAMG is >>>> >> >>> picking >>>> >> >>> >>> the null space up (send this output if you can't figure it out). >>>> >> >>> >>> >>>> >> >>> >>> Thanks, >>>> >> >>> >>> Mark >>>> >> >>> >>> >>>> >> >>> >>> On Tue, Oct 29, 2024 at 9:28?AM Mark Adams > >>>> >> wrote: >>>> >> >>> >>> >>>> >> >>> >>>> This coordinate interface is just a shortcut for vertex based >>>> >> >>> >>>> discretizations with 3 dof per vertex, etc. (maybe works in 2D). >>>> >> >>> >>>> You will need to construct the null space vectors manually and >>>> >> >>> attach it >>>> >> >>> >>>> to the matrix. Used by GAMG. >>>> >> >>> >>>> >>>> >> >>> >>>> Note, for hypre you want to use the "nodal" options and it does >>>> >> not >>>> >> >>> use >>>> >> >>> >>>> these null space vectors. That is probably the way you want to go. >>>> >> >>> >>>> eg: -pc_hypre_boomeramg_nodal_coarsen >>>> >> >>> >>>> >>>> >> >>> >>>> I would run with hypre boomerang and -help and grep on nodal to >>>> >> see >>>> >> >>> all >>>> >> >>> >>>> the "nodal" options and use them. >>>> >> >>> >>>> >>>> >> >>> >>>> Thanks, >>>> >> >>> >>>> Mark >>>> >> >>> >>>> >>>> >> >>> >>>> >>>> >> >>> >>>> On Mon, Oct 28, 2024 at 8:06?PM Amneet Bhalla < >>>> >> mail2amneet at gmail.com >>>> >> >>> > >>>> >> >>> >>>> wrote: >>>> >> >>> >>>> >>>> >> >>> >>>>> Hi Folks, >>>> >> >>> >>>>> >>>> >> >>> >>>>> I am trying to solve the momentum equation in a projection >>>> >> >>> >>>>> preconditioner using GAMG or Hypre solver. The equation looks >>>> >> like >>>> >> >>> for >>>> >> >>> >>>>> velocity variable *v* looks like: >>>> >> >>> >>>>> >>>> >> >>> >>>>> >>>> >> >>> >>>>> [image: Screenshot 2024-10-28 at 4.15.17 PM.png] >>>> >> >>> >>>>> >>>> >> >>> >>>>> Here, ? is spatially varying dynamic viscosity and ? is spatially >>>> >> >>> >>>>> varying bulk viscosity. I understand that I need to specify rigid >>>> >> >>> body >>>> >> >>> >>>>> nullspace modes to the multigrid solver in order to accelerate >>>> >> its >>>> >> >>> >>>>> convergence. Looking into this routine >>>> >> >>> MatNullSpaceCreateRigidBody() ( >>>> >> >>> >>>>> >>>> >> >>> >>>> >> https://urldefense.us/v3/__https://petsc.org/release/manualpages/Mat/MatNullSpaceCreateRigidBody/__;!!G_uCfscf7eWS!bVR6duCoDqPhZrWS-sm1c5qxsFPjZMhdT86AqLpPzWgVy5qoRhd4_Jue2LJOIS6LRrtV2cHGrqger1Yvb-Y5f-0$ >>>> >> >>> >>>>> < >>>> >> >>> >>>> >> https://urldefense.us/v3/__https://petsc.org/release/manualpages/Mat/MatNullSpaceCreateRigidBody/__;!!G_uCfscf7eWS!eKqgIJjCdMzIU76f7X65AmGxrU_-lC7W02BMWafJ77DNf_IuQk6O1X3qU1x9Ez8NJ20vZEL-mF6T1yNmDnwv0eWa2w$ >>>> >> >>> >), >>>> >> >>> >>>>> I see that I need to provide the coordinates of each node. I am >>>> >> >>> using >>>> >> >>> >>>>> staggered grid discretization. Do I need to provide coordinates >>>> >> of >>>> >> >>> >>>>> staggered grid locations? >>>> >> >>> >>>>> >>>> >> >>> >>>>> Thanks, >>>> >> >>> >>>>> -- >>>> >> >>> >>>>> --Amneet >>>> >> >>> >>>>> >>>> >> >>> >>>>> >>>> >> >>> >>>>> >>>> >> >>> >>>>> >>>> >> >>> >> >>>> >> >>> >> -- >>>> >> >>> >> --Amneet >>>> >> >>> >> >>>> >> >>> >> >>>> >> >>> >> >>>> >> >>> >> >>>> >> >>> >>>> >> >> >>>> >> >> >>>> >> >> -- >>>> >> >> --Amneet >>>> >> >> >>>> >> >> >>>> >> >> >>>> >> >> >>>> >> > >>>> >> > -- >>>> >> > --Amneet >>>> >> >>>> > >>>> > >>>> > -- >>>> > --Amneet >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Thu Oct 31 09:18:40 2024 From: bsmith at petsc.dev (Barry Smith) Date: Thu, 31 Oct 2024 10:18:40 -0400 Subject: [petsc-users] Problem running geoclaw example with petsc 3.22 In-Reply-To: References: <1C63D19F-6EBC-43A3-8B40-65EA34991AFA@gmail.com> <0136027D-C2CE-47D0-AF97-ACDE1DA0C382@gmail.com> <4726A341-54DB-4592-BBAE-843AF271F28E@petsc.dev> <27B47E2D-00D5-4326-95A9-7595DFF0E952@gmail.com> <490B5A4A-BBD8-490A-91FD-7CEC9E01A99B@petsc.dev> <96674E24-9EDA-42DE-8459-81D46A260CDE@petsc.dev> Message-ID: Thanks, this is progress :-) You can add to petscMPIOptions the lines -start_in_debugger -debugger_nodes 0 then run as usual, a window will pop up with the debugger type c (for continue) then when it craches type bt (for backtrace) and it should print out the stack frames, cut and paste them all and send that back Barry > On Oct 31, 2024, at 1:24?AM, Praveen C wrote: > > Hello Barry > > With the extra option > >> -mpi_linear_solver_server_use_shared_memory false > > > I get different error > > Thanks > praveen > > $ make .output > /Library/Developer/CommandLineTools/usr/bin/make output -f Makefile /Users/praveen/Applications/clawpack/geoclaw/src/2d/bouss/Makefile.bouss /Users/praveen/Applications/clawpack/clawutil/src/Makefile.common > rm -f .output > python /Users/praveen/Applications/clawpack/clawutil/src/python/clawutil/runclaw.py /Users/praveen/Work/bouss/radial_flat/xgeoclaw _output \ > True None . False False None "/opt/homebrew/Caskroom/miniforge/base/envs/claw/./bin/mpiexec -n 6" > ==> runclaw: Will take data from /Volumes/Samsung_T5/Work/bouss/radial_flat > ==> runclaw: Will write output to /Volumes/Samsung_T5/Work/bouss/radial_flat/_output > ==> runclaw: Removing all old fort/gauge files in /Volumes/Samsung_T5/Work/bouss/radial_flat/_output > > ==> Running with command: > /opt/homebrew/Caskroom/miniforge/base/envs/claw/./bin/mpiexec -n 6 /Users/praveen/Work/bouss/radial_flat/xgeoclaw > Reading data file: claw.data > first 5 lines are comments and will be skipped > Reading data file: amr.data > first 5 lines are comments and will be skipped > > Running amrclaw ... > > Reading data file: geoclaw.data > first 5 lines are comments and will be skipped > Reading data file: refinement.data > first 5 lines are comments and will be skipped > Reading data file: dtopo.data > first 5 lines are comments and will be skipped > Reading data file: topo.data > first 5 lines are comments and will be skipped > converting to topotype > 1 might reduce file size > python tools for converting files are provided > > Reading topography file /Volumes/Samsung_T5/Work/bouss/radial_flat/flat100.tt1 > Reading data file: qinit.data > first 5 lines are comments and will be skipped > qinit_type = 0, no perturbation > Reading data file: fgout_grids.data > first 5 lines are comments and will be skipped > Reading data file: friction.data > first 5 lines are comments and will be skipped > Reading data file: multilayer.data > first 5 lines are comments and will be skipped > Reading data file: surge.data > first 5 lines are comments and will be skipped > Reading data file: regions.data > first 5 lines are comments and will be skipped > Reading data file: flagregions.data > first 5 lines are comments and will be skipped > +++ rregion bounding box: > 0.0000000000000000 5000.0000000000000 0.0000000000000000 5000.0000000000000 > +++ i, rr%s(1), rr%ds: 1 0.0000000000000000 5000.0000000000000 > +++ Ruled region name: Region_diagonal > +++ Ruled region file_name: /Volumes/Samsung_T5/Work/bouss/radial_flat/RuledRectangle_Diagonal.data > +++ rregion bounding box: > 0.0000000000000000 5000.0000000000000 -1000.0000000000000 6000.0000000000000 > +++ i, rr%s(1), rr%ds: 2 0.0000000000000000 5000.0000000000000 > +++ rregion bounding box: > 0.0000000000000000 1000.0000000000000 0.0000000000000000 1000.0000000000000 > +++ i, rr%s(1), rr%ds: 3 0.0000000000000000 1000.0000000000000 > Reading data file: gauges.data > first 5 lines are comments and will be skipped > Reading data file: fgmax_grids.data > first 5 lines are comments and will be skipped > Reading data file: adjoint.data > first 5 lines are comments and will be skipped > Reading data file: bouss.data > first 5 lines are comments and will be skipped > Using SGN equations > ==> Applying Bouss equations to selected grids between levels 1 and 10 > ==> Use Bouss. in water deeper than 1.0000000000000000 > Using a PETSc solver > Using Bouss equations from the start > rnode allocated... > node allocated... > listOfGrids allocated... > Storage allocated... > bndList allocated... > Gridding level 1 at t = 0.000000E+00: 4 grids with 10000 cells > Setting initial dt to 2.9999999999999999E-002 > max threads set to 1 > > Done reading data, starting computation ... > > Total zeta at initial time: 39269.907650665169 > GEOCLAW: Frame 0 output files done at time t = 0.000000D+00 > > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range > [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [0]PETSC ERROR: or see https://urldefense.us/v3/__https://petsc.org/release/faq/*valgrind__;Iw!!G_uCfscf7eWS!eiByYM9OWRuz3FQcylxVZQPskjJAYk5n9fzr1ZlHLo8_rF0ZbemCaV8fPWmYHH3mmUMBSJyaYkhhxsJBVvg65ic$ and https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!eiByYM9OWRuz3FQcylxVZQPskjJAYk5n9fzr1ZlHLo8_rF0ZbemCaV8fPWmYHH3mmUMBSJyaYkhhxsJB3S85n4s$ > [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run > [0]PETSC ERROR: to get more information on the crash. > [0]PETSC ERROR: Run with -malloc_debug to check if memory corruption is causing the crash. > -------------------------------------------------------------------------- > MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_SELF > Proc: [[5779,1],0] > Errorcode: 59 > > NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. > You may or may not see output from other processes, depending on > exactly when Open MPI kills them. > -------------------------------------------------------------------------- > -------------------------------------------------------------------------- > prterun has exited due to process rank 0 with PID 0 on node chandra calling > "abort". This may have caused other processes in the application to be > terminated by signals sent by prterun (as reported here). > -------------------------------------------------------------------------- > Traceback (most recent call last): > File "/Users/praveen/Applications/clawpack/clawutil/src/python/clawutil/runclaw.py", line 242, in runclaw > proc = subprocess.check_call(cmd_split, > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > File "/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib/python3.12/subprocess.py", line 413, in check_call > raise CalledProcessError(retcode, cmd) > subprocess.CalledProcessError: Command '['/opt/homebrew/Caskroom/miniforge/base/envs/claw/./bin/mpiexec', '-n', '6', '/Users/praveen/Work/bouss/radial_flat/xgeoclaw']' returned non-zero exit status 59. > > During handling of the above exception, another exception occurred: > > Traceback (most recent call last): > File "/Users/praveen/Applications/clawpack/clawutil/src/python/clawutil/runclaw.py", line 341, in > runclaw(*args) > File "/Users/praveen/Applications/clawpack/clawutil/src/python/clawutil/runclaw.py", line 249, in runclaw > raise ClawExeError(exe_error_str, cpe.returncode, cpe.cmd, > ClawExeError: > > *** FORTRAN EXE FAILED *** > > make[1]: *** [output] Error 1 > make: *** [.output] Error 2 > >> On 30 Oct 2024, at 9:33?PM, Barry Smith wrote: >> >> >> Please put >> >> -mpi_linear_solver_server_use_shared_memory false >> >> into the petscMPIOptions file and see if that changes anything. >> >> Barry >> >> >>> On Oct 30, 2024, at 9:05?AM, Praveen C wrote: >>> >>> I have attached some files for this >>> >>> cd bouss >>> . setenv.sh # some settings in this file may need to be changed >>> >>> cd radial_flat/1d_radial >>> make .output >>> cd .. >>> make .output >>> >>> Thanks >>> praveen >>> >>> >>> >>> >>>> On 30 Oct 2024, at 6:28?PM, Barry Smith wrote: >>>> >>>> >>>> Please send me exact instructions on how you are running the geoclaw example, when I run >>>> >>>> geoclaw/examples/bouss/radial_flat >>>> >>>> >>>> it only runs with one level, but never errors, I changed setrun.py to amrdata.amr_levels_max = 5 but it made no difference. >>>> >>>> Thanks >>>> >>>> Barry >>>> >>>> >>>> >>>>> On Oct 24, 2024, at 12:17?PM, Praveen C wrote: >>>>> >>>>> Hello Barry >>>>> >>>>> I use this script to install clawpack and required dependencies >>>>> >>>>> https://urldefense.us/v3/__https://github.com/cpraveen/cfdlab/blob/master/bin/clawpack.sh__;!!G_uCfscf7eWS!eiByYM9OWRuz3FQcylxVZQPskjJAYk5n9fzr1ZlHLo8_rF0ZbemCaV8fPWmYHH3mmUMBSJyaYkhhxsJBDAG1OOo$ >>>>> >>>>> See lines 125-132 >>>>> >>>>> You can use it like this >>>>> >>>>> export CLAW=/path/to/where/you/want/clawpack >>>>> bash clawpack.sh v5.11.0 >>>>> >>>>> This will git pull clawpack and creates a conda env called ?claw? and installs inside that. >>>>> >>>>> I have not specified petsc version in this script, but latest miniforge should install petsc at 3.22 >>>>> >>>>> Thank you >>>>> praveen >>>>> >>>>>> On 24 Oct 2024, at 9:37?PM, Barry Smith wrote: >>>>>> >>>>>> >>>>>> Good, some helpful information with the runs you made. >>>>>> >>>>>> In the crash below it made more progress it was able to allocate multiple regions of shared memory and access them. I don't know why it would crash later. >>>>>> >>>>>> Can you tell me all the steps with miniforge (which seems to be related to the failure) you use? I've never used miniforge. >>>>>> >>>>>> If I can get to an environment that reproduces the problem I can debug it and fix it. >>>>>> >>>>>> Barry >>>>>> >>>>>> >>>>>>> On Oct 24, 2024, at 11:49?AM, Praveen C wrote: >>>>>>> >>>>>>> I get this >>>>>>> >>>>>>> $ mpiexec -n 3 ./ex89f -n 20 -mpi_linear_solver_server -mpi_linear_solver_server -mpi_linear_solver_server_ksp_view -ksp_monitor -ksp_converged_reason -ksp_view -mpi_linear_solver_server_minimum_count_per_rank 5 >>>>>>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>>>>>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range >>>>>>> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >>>>>>> [0]PETSC ERROR: or see https://urldefense.us/v3/__https://petsc.org/release/faq/*valgrind__;Iw!!G_uCfscf7eWS!eiByYM9OWRuz3FQcylxVZQPskjJAYk5n9fzr1ZlHLo8_rF0ZbemCaV8fPWmYHH3mmUMBSJyaYkhhxsJBVvg65ic$ and https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!eiByYM9OWRuz3FQcylxVZQPskjJAYk5n9fzr1ZlHLo8_rF0ZbemCaV8fPWmYHH3mmUMBSJyaYkhhxsJB3S85n4s$ >>>>>>> [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run >>>>>>> [0]PETSC ERROR: to get more information on the crash. >>>>>>> [0]PETSC ERROR: Run with -malloc_debug to check if memory corruption is causing the crash. >>>>>>> -------------------------------------------------------------------------- >>>>>>> MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_SELF >>>>>>> Proc: [[47380,1],0] >>>>>>> Errorcode: 59 >>>>>>> >>>>>>> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. >>>>>>> You may or may not see output from other processes, depending on >>>>>>> exactly when Open MPI kills them. >>>>>>> -------------------------------------------------------------------------- >>>>>>> -------------------------------------------------------------------------- >>>>>>> prterun has exited due to process rank 0 with PID 0 on node MacMiniHome calling >>>>>>> "abort". This may have caused other processes in the application to be >>>>>>> terminated by signals sent by prterun (as reported here). >>>>>>> ????????????????????????????????????? >>>>>>> >>>>>>> and I have this after the code exits >>>>>>> >>>>>>> $ ipcs -m >>>>>>> IPC status from as of Thu Oct 24 21:17:39 IST 2024 >>>>>>> T ID KEY MODE OWNER GROUP >>>>>>> Shared Memory: >>>>>>> m 1572864 0x0000000b --rw-rw-rw- praveen staff >>>>>>> m 524289 0x0000000c --rw-rw-rw- praveen staff >>>>>>> m 655362 0x0000000d --rw-rw-rw- praveen staff >>>>>>> m 262147 0x0000000e --rw-rw-rw- praveen staff >>>>>>> m 262148 0x0000000f --rw-rw-rw- praveen staff >>>>>>> m 393221 0x0000000a --rw-rw-rw- praveen staff >>>>>>> >>>>>>> This is with petsc installed with miniforge, which I also use with clawpack. With spack installed petsc, I can run the ex89f example. >>>>>>> >>>>>>> Thanks >>>>>>> praveen >>>>>>> >>>>>>>> On 24 Oct 2024, at 7:55?PM, Barry Smith wrote: >>>>>>>> >>>>>>>> >>>>>>>> Ok, super strange, I've run many times on my Mac >>>>>>>> >>>>>>>> Can you please try to remove that allocated memory with ipcrm and then >>>>>>>> >>>>>>>> cd $PETSC_DIR/src/ksp/ksp/tutorials >>>>>>>> make ex89f >>>>>>>> mpiexec -n 3 ./ex89f -n 20 -mpi_linear_solver_server -mpi_linear_solver_server -mpi_linear_solver_server_ksp_view -ksp_monitor -ksp_converged_reason -ksp_view -mpi_linear_solver_server_minimum_count_per_rank 5 >>>>>>>> >>>>>>>> This does the same thing as the GeoClaw code but is much simpler. >>>>>>>> >>>>>>>> Barry >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> On Oct 23, 2024, at 10:55?PM, Praveen C wrote: >>>>>>>>> >>>>>>>>> I get very similar error on my mac with >>>>>>>>> >>>>>>>>> $ gfortran -v >>>>>>>>> Using built-in specs. >>>>>>>>> COLLECT_GCC=gfortran >>>>>>>>> COLLECT_LTO_WRAPPER=/opt/homebrew/Caskroom/miniforge/base/envs/claw/libexec/gcc/arm64-apple-darwin20.0.0/13.2.0/lto-wrapper >>>>>>>>> Target: arm64-apple-darwin20.0.0 >>>>>>>>> Configured with: ../configure --prefix=/opt/homebrew/Caskroom/miniforge/base/envs/claw --build=x86_64-apple-darwin13.4.0 --host=arm64-apple-darwin20.0.0 --target=arm64-apple-darwin20.0.0 --with-libiconv-prefix=/opt/homebrew/Caskroom/miniforge/base/envs/claw --enable-languages=fortran --disable-multilib --enable-checking=release --disable-bootstrap --disable-libssp --with-gmp=/opt/homebrew/Caskroom/miniforge/base/envs/claw --with-mpfr=/opt/homebrew/Caskroom/miniforge/base/envs/claw --with-mpc=/opt/homebrew/Caskroom/miniforge/base/envs/claw --with-isl=/opt/homebrew/Caskroom/miniforge/base/envs/claw --enable-darwin-at-rpath >>>>>>>>> Thread model: posix >>>>>>>>> Supported LTO compression algorithms: zlib >>>>>>>>> gcc version 13.2.0 (GCC) >>>>>>>>> >>>>>>>>> Before starting >>>>>>>>> >>>>>>>>> $ ipcs -m >>>>>>>>> IPC status from as of Thu Oct 24 08:02:11 IST 2024 >>>>>>>>> T ID KEY MODE OWNER GROUP >>>>>>>>> Shared Memory: >>>>>>>>> >>>>>>>>> and when I run the code >>>>>>>>> >>>>>>>>> Using a PETSc solver >>>>>>>>> Using Bouss equations from the start >>>>>>>>> rnode allocated... >>>>>>>>> node allocated... >>>>>>>>> listOfGrids allocated... >>>>>>>>> Storage allocated... >>>>>>>>> bndList allocated... >>>>>>>>> Gridding level 1 at t = 0.000000E+00: 4 grids with 10000 cells >>>>>>>>> Setting initial dt to 2.9999999999999999E-002 >>>>>>>>> max threads set to 1 >>>>>>>>> >>>>>>>>> Done reading data, starting computation ... >>>>>>>>> >>>>>>>>> Total zeta at initial time: 39269.907650665169 >>>>>>>>> GEOCLAW: Frame 0 output files done at time t = 0.000000D+00 >>>>>>>>> >>>>>>>>> [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >>>>>>>>> [0]PETSC ERROR: Petsc has generated inconsistent data >>>>>>>>> [0]PETSC ERROR: Unable to locate PCMPI allocated shared address 0x130698000 >>>>>>>>> [0]PETSC ERROR: WARNING! There are unused option(s) set! Could be the program crashed before usage or a spelling mistake, etc! >>>>>>>>> [0]PETSC ERROR: Option left: name:-ksp_type value: preonly source: file >>>>>>>>> [0]PETSC ERROR: Option left: name:-mpi_ksp_max_it value: 200 source: file >>>>>>>>> [0]PETSC ERROR: Option left: name:-mpi_ksp_reuse_preconditioner (no value) source: file >>>>>>>>> [0]PETSC ERROR: Option left: name:-mpi_ksp_rtol value: 1.e-9 source: file >>>>>>>>> [0]PETSC ERROR: Option left: name:-mpi_ksp_type value: gmres source: file >>>>>>>>> [0]PETSC ERROR: Option left: name:-mpi_linear_solver_server_view (no value) source: file >>>>>>>>> [0]PETSC ERROR: Option left: name:-mpi_pc_gamg_sym_graph value: true source: file >>>>>>>>> [0]PETSC ERROR: Option left: name:-mpi_pc_gamg_symmetrize_graph value: true source: file >>>>>>>>> [0]PETSC ERROR: Option left: name:-mpi_pc_type value: gamg source: file >>>>>>>>> [0]PETSC ERROR: Option left: name:-pc_mpi_minimum_count_per_rank value: 5000 source: file >>>>>>>>> [0]PETSC ERROR: Option left: name:-pc_type value: mpi source: file >>>>>>>>> [0]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!eiByYM9OWRuz3FQcylxVZQPskjJAYk5n9fzr1ZlHLo8_rF0ZbemCaV8fPWmYHH3mmUMBSJyaYkhhxsJB3S85n4s$ for trouble shooting. >>>>>>>>> [0]PETSC ERROR: Petsc Release Version 3.22.0, Sep 28, 2024 >>>>>>>>> [0]PETSC ERROR: /Users/praveen/work/bouss/radial_flat/xgeoclaw with 6 MPI process(es) and PETSC_ARCH on MacMiniHome.local by praveen Thu Oct 24 08:04:27 2024 >>>>>>>>> [0]PETSC ERROR: Configure options: AR=arm64-apple-darwin20.0.0-ar CC=mpicc CXX=mpicxx FC=mpifort CFLAGS="-ftree-vectorize -fPIC -fstack-protector-strong -O2 -pipe -isystem /opt/homebrew/Caskroom/miniforge/base/envs/claw/include " CPPFLAGS="-D_FORTIFY_SOURCE=2 -isystem /opt/homebrew/Caskroom/miniforge/base/envs/claw/include -mmacosx-version-min=11.0 -mmacosx-version-min=11.0" CXXFLAGS="-ftree-vectorize -fPIC -fstack-protector-strong -O2 -pipe -stdlib=libc++ -fvisibility-inlines-hidden -fmessage-length=0 -isystem /opt/homebrew/Caskroom/miniforge/base/envs/claw/include " FFLAGS="-march=armv8.3-a -ftree-vectorize -fPIC -fno-stack-protector -O2 -pipe -isystem /opt/homebrew/Caskroom/miniforge/base/envs/claw/include " LDFLAGS="-Wl,-headerpad_max_install_names -Wl,-dead_strip_dylibs -Wl,-rpath,/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib -L/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib" LIBS="-Wl,-rpath,/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib -lmpi_mpifh -lgfortran" --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 --FOPTFLAGS=-O3 --with-clib-autodetect=0 --with-cxxlib-autodetect=0 --with-fortranlib-autodetect=0 --with-debugging=0 --with-blas-lib=libblas.dylib --with-lapack-lib=liblapack.dylib --with-yaml=1 --with-hdf5=1 --with-fftw=1 --with-hwloc=0 --with-hypre=1 --with-metis=1 --with-mpi=1 --with-mumps=1 --with-parmetis=1 --with-pthread=1 --with-ptscotch=1 --with-shared-libraries --with-ssl=0 --with-scalapack=1 --with-superlu=1 --with-superlu_dist=1 --with-superlu_dist-include=/opt/homebrew/Caskroom/miniforge/base/envs/claw/include/superlu-dist --with-superlu_dist-lib=-lsuperlu_dist --with-suitesparse=1 --with-suitesparse-dir=/opt/homebrew/Caskroom/miniforge/base/envs/claw --with-x=0 --with-scalar-type=real --with-cuda=0 --with-batch --prefix=/opt/homebrew/Caskroom/miniforge/base/envs/claw >>>>>>>>> [0]PETSC ERROR: #1 PetscShmgetMapAddresses() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/sys/utils/server.c:114 >>>>>>>>> [0]PETSC ERROR: #2 PCMPISetMat() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/ksp/pc/impls/mpi/pcmpi.c:269 >>>>>>>>> [0]PETSC ERROR: #3 PCSetUp_MPI() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/ksp/pc/impls/mpi/pcmpi.c:853 >>>>>>>>> [0]PETSC ERROR: #4 PCSetUp() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/ksp/pc/interface/precon.c:1071 >>>>>>>>> [0]PETSC ERROR: #5 KSPSetUp() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/ksp/ksp/interface/itfunc.c:415 >>>>>>>>> [0]PETSC ERROR: #6 KSPSolve_Private() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/ksp/ksp/interface/itfunc.c:826 >>>>>>>>> [0]PETSC ERROR: #7 KSPSolve() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/ksp/ksp/interface/itfunc.c:1075 >>>>>>>>> >>>>>>>>> Code does not progress and I kill it >>>>>>>>> >>>>>>>>> ^CTraceback (most recent call last): >>>>>>>>> File "/Users/praveen/Applications/clawpack/clawutil/src/python/clawutil/runclaw.py", line 341, in >>>>>>>>> runclaw(*args) >>>>>>>>> File "/Users/praveen/Applications/clawpack/clawutil/src/python/clawutil/runclaw.py", line 242, in runclaw >>>>>>>>> proc = subprocess.check_call(cmd_split, >>>>>>>>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ >>>>>>>>> File "/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib/python3.12/subprocess.py", line 408, in check_call >>>>>>>>> retcode = call(*popenargs, **kwargs) >>>>>>>>> ^^^^^^^^^^^^^^^^^^^^^^^^^^ >>>>>>>>> File "/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib/python3.12/subprocess.py", line 391, in call >>>>>>>>> return p.wait(timeout=timeout) >>>>>>>>> ^^^^^^^^^^^^^^^^^^^^^^^ >>>>>>>>> File "/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib/python3.12/subprocess.py", line 1264, in wait >>>>>>>>> return self._wait(timeout=timeout) >>>>>>>>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^ >>>>>>>>> File "/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib/python3.12/subprocess.py", line 2053, in _wait >>>>>>>>> (pid, sts) = self._try_wait(0) >>>>>>>>> ^^^^^^^^^^^^^^^^^ >>>>>>>>> File "/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib/python3.12/subprocess.py", line 2011, in _try_wait >>>>>>>>> (pid, sts) = os.waitpid(self.pid, wait_flags) >>>>>>>>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ >>>>>>>>> KeyboardInterrupt >>>>>>>>> make[1]: *** [output] Interrupt: 2 >>>>>>>>> make: *** [.output] Interrupt: 2 >>>>>>>>> >>>>>>>>> Now it says >>>>>>>>> >>>>>>>>> $ ipcs -m >>>>>>>>> IPC status from as of Thu Oct 24 08:05:06 IST 2024 >>>>>>>>> T ID KEY MODE OWNER GROUP >>>>>>>>> Shared Memory: >>>>>>>>> m 720896 0x0000000a --rw-rw-rw- praveen staff >>>>>>>>> >>>>>>>>> Thanks >>>>>>>>> praveen >>>>>>>>> >>>>>>>>>> On 23 Oct 2024, at 8:26?PM, Barry Smith wrote: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Hmm, so it is creating the first shared memory region with ID of 10 (A in hex) puts it in a linked list in PETSc but then when it tries to find it in the linked list it cannot find it. >>>>>>>>>> >>>>>>>>>> I don't know how to reproduce this or debug it remotely. >>>>>>>>>> >>>>>>>>>> Can you build on a completely different machine or with completely different compilers? >>>>>>>>>> >>>>>>>>>> Barry >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> On Oct 23, 2024, at 10:31?AM, Praveen C wrote: >>>>>>>>>>> >>>>>>>>>>> I get same error and now it shows >>>>>>>>>>> >>>>>>>>>>> $ ipcs -m >>>>>>>>>>> >>>>>>>>>>> ------ Shared Memory Segments -------- >>>>>>>>>>> key shmid owner perms bytes nattch status >>>>>>>>>>> 0x0000000a 32788 praveen 666 240 6 >>>>>>>>>>> >>>>>>>>>>> Note that the code seems to be still running after printing those error message, but it is not printing any progress which it should do. >>>>>>>>>>> >>>>>>>>>>> Thanks >>>>>>>>>>> praveen >>>>>>>>>>> >>>>>>>>>>>> On 23 Oct 2024, at 7:56?PM, Barry Smith wrote: >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Try >>>>>>>>>>>> >>>>>>>>>>>> ipcrm -m 11 >>>>>>>>>>>> >>>>>>>>>>>> ipcs -m >>>>>>>>>>>> >>>>>>>>>>>> Try running the program again >>>>>>>>>>>> >>>>>>>>>>>> If failed check >>>>>>>>>>>> >>>>>>>>>>>> ipcs -m >>>>>>>>>>>> >>>>>>>>>>>> again >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> On Oct 23, 2024, at 10:20?AM, Praveen C wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> Hello Barry >>>>>>>>>>>>> >>>>>>>>>>>>> I see this >>>>>>>>>>>>> >>>>>>>>>>>>> $ ipcs -m >>>>>>>>>>>>> >>>>>>>>>>>>> ------ Shared Memory Segments -------- >>>>>>>>>>>>> key shmid owner perms bytes nattch status >>>>>>>>>>>>> 0x0000000a 11 praveen 666 240 6 >>>>>>>>>>>>> >>>>>>>>>>>>> and I am observing same error as below. >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks >>>>>>>>>>>>> praveen >>>>>>>>>>>>> >>>>>>>>>>>>>> On 23 Oct 2024, at 7:08?PM, Barry Smith wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Please take a look at the notes in https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/PetscShmgetAllocateArray/__;!!G_uCfscf7eWS!eiByYM9OWRuz3FQcylxVZQPskjJAYk5n9fzr1ZlHLo8_rF0ZbemCaV8fPWmYHH3mmUMBSJyaYkhhxsJB5aYdAxo$ For some reason your program is not able to access/use the Unix shared memory; check if you are already using the shared memory (so it is not available for a new run) or the limits are too low to access enough memory. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Barry >>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Oct 23, 2024, at 8:23?AM, Praveen C wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Dear all >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I am not able to run the boussinesq example from geoclaw using petsc at 3.22.0 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> https://urldefense.us/v3/__https://github.com/clawpack/geoclaw/tree/3303883f46572c58130d161986b8a87a57ca7816/examples/bouss__;!!G_uCfscf7eWS!e3VQ4NHKmXGstRsQW5vtI7fmKfUT9zmJkMJcPbcvPyIjicyfJpNoMgx3wZ-qyGcKNSjIkNZkzilec8MnHN6PMw$ >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> It runs with petsc at 3.21.6 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> The error I get is given below. After printing this, the code does not progress. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I use the following petsc options >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> # set min numbers of matrix rows per MPI rank (default is 10000) >>>>>>>>>>>>>>> -mpi_linear_solve_minimum_count_per_rank 5000 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> # Krylov linear solver: >>>>>>>>>>>>>>> -mpi_linear_solver_server >>>>>>>>>>>>>>> -mpi_linear_solver_server_view >>>>>>>>>>>>>>> -ksp_type gmres >>>>>>>>>>>>>>> -ksp_max_it 200 >>>>>>>>>>>>>>> -ksp_reuse_preconditioner >>>>>>>>>>>>>>> -ksp_rtol 1.e-9 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> # preconditioner: >>>>>>>>>>>>>>> -pc_type gamg >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I installed petsc and other dependencies for clawpack using miniforge. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks >>>>>>>>>>>>>>> pc >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> ==> Use Bouss. in water deeper than 1.0000000000000000 Using a PETSc solver >>>>>>>>>>>>>>> Using Bouss equations from the start >>>>>>>>>>>>>>> rnode allocated... >>>>>>>>>>>>>>> node allocated... >>>>>>>>>>>>>>> listOfGrids allocated... >>>>>>>>>>>>>>> Storage allocated... >>>>>>>>>>>>>>> bndList allocated... >>>>>>>>>>>>>>> Gridding level 1 at t = 0.000000E+00: 4 grids with 10000 cells >>>>>>>>>>>>>>> Setting initial dt to 2.9999999999999999E-002 >>>>>>>>>>>>>>> max threads set to 6 >>>>>>>>>>>>>>> Done reading data, starting computation ... Total zeta at initial time: 39269.907650665169 GEOCLAW: Frame 0 output files done at time t = 0.000000D+00 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >>>>>>>>>>>>>>> [0]PETSC ERROR: Petsc has generated inconsistent data >>>>>>>>>>>>>>> [0]PETSC ERROR: Unable to locate PCMPI allocated shared address 0x55e6d750ae20 >>>>>>>>>>>>>>> [0]PETSC ERROR: WARNING! There are unused option(s) set! Could be the program crashed before usage or a spelling mistake, etc! >>>>>>>>>>>>>>> [0]PETSC ERROR: Option left: name:-ksp_max_it value: 200 source: file >>>>>>>>>>>>>>> [0]PETSC ERROR: Option left: name:-ksp_reuse_preconditioner (no value) source: file >>>>>>>>>>>>>>> [0]PETSC ERROR: Option left: name:-ksp_rtol value: 1.e-9 source: file >>>>>>>>>>>>>>> [0]PETSC ERROR: Option left: name:-ksp_type value: gmres source: file >>>>>>>>>>>>>>> [0]PETSC ERROR: Option left: name:-mpi_linear_solve_minimum_count_per_rank value: 5000 source: file >>>>>>>>>>>>>>> [0]PETSC ERROR: Option left: name:-mpi_linear_solver_server_view (no value) source: file >>>>>>>>>>>>>>> [0]PETSC ERROR: Option left: name:-pc_type value: gamg source: file >>>>>>>>>>>>>>> [0]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!e3VQ4NHKmXGstRsQW5vtI7fmKfUT9zmJkMJcPbcvPyIjicyfJpNoMgx3wZ-qyGcKNSjIkNZkzilec8MvNjNo7A$ for trouble shooting. >>>>>>>>>>>>>>> [0]PETSC ERROR: Petsc Release Version 3.22.0, Sep 28, 2024 [0]PETSC ERROR: /home/praveen/bouss/radial_flat/xgeoclaw with 6 MPI process(es) and PETSC_ARCH on euler by praveen Thu Oct 17 21:49:54 2024 >>>>>>>>>>>>>>> [0]PETSC ERROR: Configure options: AR=${PREFIX}/bin/x86_64-conda-linux-gnu-ar CC=mpicc CXX=mpicxx FC=mpifort CFLAGS="-march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /opt/miniforge/envs/claw/include " CPPFLAGS="-DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /opt/miniforge/envs/claw/include" CXXFLAGS="-fvisibility-inlines-hidden -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /opt/miniforge/envs/claw/include " FFLAGS="-march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /opt/miniforge/envs/claw/include -Wl,--no-as-needed" LDFLAGS="-pthread -fopenmp -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,--allow-shlib-undefined -Wl,-rpath,/opt/miniforge/envs/claw/lib -Wl,-rpath-link,/opt/miniforge/envs/claw/lib -L/opt/miniforge/envs/claw/lib -Wl,-rpath-link,/opt/miniforge/envs/claw/lib" LIBS="-Wl,-rpath,/opt/miniforge/envs/claw/lib -lmpi_mpifh -lgfortran" --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 --FOPTFLAGS=-O3 --with-clib-autodetect=0 --with-cxxlib-autodetect=0 --with-fortranlib-autodetect=0 --with-debugging=0 --with-blas-lib=libblas.so --with-lapack-lib=liblapack.so --with-yaml=1 --with-hdf5=1 --with-fftw=1 --with-hwloc=0 --with-hypre=1 --with-metis=1 --with-mpi=1 --with-mumps=1 --with-parmetis=1 --with-pthread=1 --with-ptscotch=1 --with-shared-libraries --with-ssl=0 --with-scalapack=1 --with-superlu=1 --with-superlu_dist=1 --with-superlu_dist-include=/opt/miniforge/envs/claw/include/superlu-dist --with-superlu_dist-lib=-lsuperlu_dist --with-suitesparse=1 --with-suitesparse-dir=/opt/miniforge/envs/claw --with-x=0 --with-scalar-type=real --with-cuda=0 --prefix=/opt/miniforge/envs/claw >>>>>>>>>>>>>>> [0]PETSC ERROR: #1 PetscShmgetMapAddresses() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/sys/utils/server.c:114 >>>>>>>>>>>>>>> [0]PETSC ERROR: #2 PCMPISetMat() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/pc/impls/mpi/pcmpi.c:269 >>>>>>>>>>>>>>> [0]PETSC ERROR: #3 PCSetUp_MPI() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/pc/impls/mpi/pcmpi.c:853 >>>>>>>>>>>>>>> [0]PETSC ERROR: #4 PCSetUp() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/pc/interface/precon.c:1071 >>>>>>>>>>>>>>> [0]PETSC ERROR: #5 KSPSetUp() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/ksp/interface/itfunc.c:415 >>>>>>>>>>>>>>> [0]PETSC ERROR: #6 KSPSolve_Private() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/ksp/interface/itfunc.c:826 >>>>>>>>>>>>>>> [0]PETSC ERROR: #7 KSPSolve() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/ksp/interface/itfunc.c:1075 >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Thu Oct 31 10:18:39 2024 From: jed at jedbrown.org (Jed Brown) Date: Thu, 31 Oct 2024 09:18:39 -0600 Subject: [petsc-users] Rigid body nullspace for Stokes operator In-Reply-To: <96BE4263-8C34-4A2E-91B4-305F94FFCAB4@joliv.et> References: <875xpas35p.fsf@jedbrown.org> <87bjz1qtk6.fsf@jedbrown.org> <87y124q31n.fsf@jedbrown.org> <96BE4263-8C34-4A2E-91B4-305F94FFCAB4@joliv.et> Message-ID: <87v7x8pagg.fsf@jedbrown.org> Pierre Jolivet writes: >> On 31 Oct 2024, at 2:47?PM, Mark Adams wrote: >> >> Interesting. I have seen hypre do fine on elasticity, but do you know if boomeramg (classical) uses these vectors or is there a smoothed aggregation solver in hypre? > > I?m not sure it is precisely ?standard? smoothed aggregation, see bottom paragraph of https://urldefense.us/v3/__https://hypre.readthedocs.io/en/latest/solvers-boomeramg.html*amg-for-systems-of-pdes__;Iw!!G_uCfscf7eWS!fdj4AzKhAcDmM1x_ZQ8gxqWeX6BAKY9urnvATMpT7hC8lw77ak7tqxqXGIX3PMg2wYA5PGu7EzyCW0yzixutBg$ > I?ve never made it to work, but I know some do. > A while back, Stefano gave me this pointer as well: https://urldefense.us/v3/__https://github.com/mfem/mfem/blob/17955e114020af340e9a06a66ebef43e05012d9c/linalg/hypre.cpp*L5245__;Iw!!G_uCfscf7eWS!fdj4AzKhAcDmM1x_ZQ8gxqWeX6BAKY9urnvATMpT7hC8lw77ak7tqxqXGIX3PMg2wYA5PGu7EzyCW0wVEi33Pw$ It's still classical AMG, and in my experience, struggles on very thin structures (e.g., aspect ratio 1000 cantilever beams) when compared to SA. However, it can be quite competitive for many structures. I found that the "MFEM elasticity suite", which is based on Baker et al 2010, gave rather poor results. This is a configuration that works on GPUs and gives good convergence and performance for elasticity: https://urldefense.us/v3/__https://github.com/hypre-space/hypre/issues/601*issuecomment-1069426997__;Iw!!G_uCfscf7eWS!arUVBVKKcYs1M5OhNqqRZl2b2o0NIUkG7fV_22qBbg-ssHhhHazhkpMbYNjCOTN66Sfbk-VZilfox9bxDf0$ In the above issue, I was only using BoomerAMG as a coarse level for p-MG so all the options have a `-mg_coarse_` prefix; here are those options without the prefix: -pc_hypre_boomeramg_coarsen_type pmis -pc_hypre_boomeramg_interp_type ext+i -pc_hypre_boomeramg_no_CF -pc_hypre_boomeramg_P_max 6 -pc_hypre_boomeramg_print_statistics 1 -pc_hypre_boomeramg_relax_type_down Chebyshev -pc_hypre_boomeramg_relax_type_up Chebyshev -pc_hypre_boomeramg_strong_threshold 0.5 -pc_type hypre From bsmith at petsc.dev Thu Oct 31 11:12:42 2024 From: bsmith at petsc.dev (Barry Smith) Date: Thu, 31 Oct 2024 12:12:42 -0400 Subject: [petsc-users] Problem running geoclaw example with petsc 3.22 In-Reply-To: References: <1C63D19F-6EBC-43A3-8B40-65EA34991AFA@gmail.com> <0136027D-C2CE-47D0-AF97-ACDE1DA0C382@gmail.com> <4726A341-54DB-4592-BBAE-843AF271F28E@petsc.dev> <27B47E2D-00D5-4326-95A9-7595DFF0E952@gmail.com> <490B5A4A-BBD8-490A-91FD-7CEC9E01A99B@petsc.dev> <96674E24-9EDA-42DE-8459-81D46A260CDE@petsc.dev> Message-ID: <954823D2-4DBA-459E-87C2-219F2AB0025D@petsc.dev> I finally got geoclaw build and running your conda environment and it runs without crashing so I am not sure why it crashes for you below. We'll need to see wht the debugger says. Barry > On Oct 31, 2024, at 1:24?AM, Praveen C wrote: > > Hello Barry > > With the extra option > >> -mpi_linear_solver_server_use_shared_memory false > > > I get different error > > Thanks > praveen > > $ make .output > /Library/Developer/CommandLineTools/usr/bin/make output -f Makefile /Users/praveen/Applications/clawpack/geoclaw/src/2d/bouss/Makefile.bouss /Users/praveen/Applications/clawpack/clawutil/src/Makefile.common > rm -f .output > python /Users/praveen/Applications/clawpack/clawutil/src/python/clawutil/runclaw.py /Users/praveen/Work/bouss/radial_flat/xgeoclaw _output \ > True None . False False None "/opt/homebrew/Caskroom/miniforge/base/envs/claw/./bin/mpiexec -n 6" > ==> runclaw: Will take data from /Volumes/Samsung_T5/Work/bouss/radial_flat > ==> runclaw: Will write output to /Volumes/Samsung_T5/Work/bouss/radial_flat/_output > ==> runclaw: Removing all old fort/gauge files in /Volumes/Samsung_T5/Work/bouss/radial_flat/_output > > ==> Running with command: > /opt/homebrew/Caskroom/miniforge/base/envs/claw/./bin/mpiexec -n 6 /Users/praveen/Work/bouss/radial_flat/xgeoclaw > Reading data file: claw.data > first 5 lines are comments and will be skipped > Reading data file: amr.data > first 5 lines are comments and will be skipped > > Running amrclaw ... > > Reading data file: geoclaw.data > first 5 lines are comments and will be skipped > Reading data file: refinement.data > first 5 lines are comments and will be skipped > Reading data file: dtopo.data > first 5 lines are comments and will be skipped > Reading data file: topo.data > first 5 lines are comments and will be skipped > converting to topotype > 1 might reduce file size > python tools for converting files are provided > > Reading topography file /Volumes/Samsung_T5/Work/bouss/radial_flat/flat100.tt1 > Reading data file: qinit.data > first 5 lines are comments and will be skipped > qinit_type = 0, no perturbation > Reading data file: fgout_grids.data > first 5 lines are comments and will be skipped > Reading data file: friction.data > first 5 lines are comments and will be skipped > Reading data file: multilayer.data > first 5 lines are comments and will be skipped > Reading data file: surge.data > first 5 lines are comments and will be skipped > Reading data file: regions.data > first 5 lines are comments and will be skipped > Reading data file: flagregions.data > first 5 lines are comments and will be skipped > +++ rregion bounding box: > 0.0000000000000000 5000.0000000000000 0.0000000000000000 5000.0000000000000 > +++ i, rr%s(1), rr%ds: 1 0.0000000000000000 5000.0000000000000 > +++ Ruled region name: Region_diagonal > +++ Ruled region file_name: /Volumes/Samsung_T5/Work/bouss/radial_flat/RuledRectangle_Diagonal.data > +++ rregion bounding box: > 0.0000000000000000 5000.0000000000000 -1000.0000000000000 6000.0000000000000 > +++ i, rr%s(1), rr%ds: 2 0.0000000000000000 5000.0000000000000 > +++ rregion bounding box: > 0.0000000000000000 1000.0000000000000 0.0000000000000000 1000.0000000000000 > +++ i, rr%s(1), rr%ds: 3 0.0000000000000000 1000.0000000000000 > Reading data file: gauges.data > first 5 lines are comments and will be skipped > Reading data file: fgmax_grids.data > first 5 lines are comments and will be skipped > Reading data file: adjoint.data > first 5 lines are comments and will be skipped > Reading data file: bouss.data > first 5 lines are comments and will be skipped > Using SGN equations > ==> Applying Bouss equations to selected grids between levels 1 and 10 > ==> Use Bouss. in water deeper than 1.0000000000000000 > Using a PETSc solver > Using Bouss equations from the start > rnode allocated... > node allocated... > listOfGrids allocated... > Storage allocated... > bndList allocated... > Gridding level 1 at t = 0.000000E+00: 4 grids with 10000 cells > Setting initial dt to 2.9999999999999999E-002 > max threads set to 1 > > Done reading data, starting computation ... > > Total zeta at initial time: 39269.907650665169 > GEOCLAW: Frame 0 output files done at time t = 0.000000D+00 > > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range > [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [0]PETSC ERROR: or see https://urldefense.us/v3/__https://petsc.org/release/faq/*valgrind__;Iw!!G_uCfscf7eWS!YE7EmIAS98p3V1aFvP_9zM7XUxOaR8mpeYeE6-_kpR_YcQyL0xdzNUR-lLIKHvMEn3eMgNIWysJZCLeHNaTdvgE$ and https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!YE7EmIAS98p3V1aFvP_9zM7XUxOaR8mpeYeE6-_kpR_YcQyL0xdzNUR-lLIKHvMEn3eMgNIWysJZCLeHIcTqVdE$ > [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run > [0]PETSC ERROR: to get more information on the crash. > [0]PETSC ERROR: Run with -malloc_debug to check if memory corruption is causing the crash. > -------------------------------------------------------------------------- > MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_SELF > Proc: [[5779,1],0] > Errorcode: 59 > > NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. > You may or may not see output from other processes, depending on > exactly when Open MPI kills them. > -------------------------------------------------------------------------- > -------------------------------------------------------------------------- > prterun has exited due to process rank 0 with PID 0 on node chandra calling > "abort". This may have caused other processes in the application to be > terminated by signals sent by prterun (as reported here). > -------------------------------------------------------------------------- > Traceback (most recent call last): > File "/Users/praveen/Applications/clawpack/clawutil/src/python/clawutil/runclaw.py", line 242, in runclaw > proc = subprocess.check_call(cmd_split, > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > File "/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib/python3.12/subprocess.py", line 413, in check_call > raise CalledProcessError(retcode, cmd) > subprocess.CalledProcessError: Command '['/opt/homebrew/Caskroom/miniforge/base/envs/claw/./bin/mpiexec', '-n', '6', '/Users/praveen/Work/bouss/radial_flat/xgeoclaw']' returned non-zero exit status 59. > > During handling of the above exception, another exception occurred: > > Traceback (most recent call last): > File "/Users/praveen/Applications/clawpack/clawutil/src/python/clawutil/runclaw.py", line 341, in > runclaw(*args) > File "/Users/praveen/Applications/clawpack/clawutil/src/python/clawutil/runclaw.py", line 249, in runclaw > raise ClawExeError(exe_error_str, cpe.returncode, cpe.cmd, > ClawExeError: > > *** FORTRAN EXE FAILED *** > > make[1]: *** [output] Error 1 > make: *** [.output] Error 2 > >> On 30 Oct 2024, at 9:33?PM, Barry Smith wrote: >> >> >> Please put >> >> -mpi_linear_solver_server_use_shared_memory false >> >> into the petscMPIOptions file and see if that changes anything. >> >> Barry >> >> >>> On Oct 30, 2024, at 9:05?AM, Praveen C wrote: >>> >>> I have attached some files for this >>> >>> cd bouss >>> . setenv.sh # some settings in this file may need to be changed >>> >>> cd radial_flat/1d_radial >>> make .output >>> cd .. >>> make .output >>> >>> Thanks >>> praveen >>> >>> >>> >>> >>>> On 30 Oct 2024, at 6:28?PM, Barry Smith wrote: >>>> >>>> >>>> Please send me exact instructions on how you are running the geoclaw example, when I run >>>> >>>> geoclaw/examples/bouss/radial_flat >>>> >>>> >>>> it only runs with one level, but never errors, I changed setrun.py to amrdata.amr_levels_max = 5 but it made no difference. >>>> >>>> Thanks >>>> >>>> Barry >>>> >>>> >>>> >>>>> On Oct 24, 2024, at 12:17?PM, Praveen C wrote: >>>>> >>>>> Hello Barry >>>>> >>>>> I use this script to install clawpack and required dependencies >>>>> >>>>> https://urldefense.us/v3/__https://github.com/cpraveen/cfdlab/blob/master/bin/clawpack.sh__;!!G_uCfscf7eWS!YE7EmIAS98p3V1aFvP_9zM7XUxOaR8mpeYeE6-_kpR_YcQyL0xdzNUR-lLIKHvMEn3eMgNIWysJZCLeHDBTfkMQ$ >>>>> >>>>> See lines 125-132 >>>>> >>>>> You can use it like this >>>>> >>>>> export CLAW=/path/to/where/you/want/clawpack >>>>> bash clawpack.sh v5.11.0 >>>>> >>>>> This will git pull clawpack and creates a conda env called ?claw? and installs inside that. >>>>> >>>>> I have not specified petsc version in this script, but latest miniforge should install petsc at 3.22 >>>>> >>>>> Thank you >>>>> praveen >>>>> >>>>>> On 24 Oct 2024, at 9:37?PM, Barry Smith wrote: >>>>>> >>>>>> >>>>>> Good, some helpful information with the runs you made. >>>>>> >>>>>> In the crash below it made more progress it was able to allocate multiple regions of shared memory and access them. I don't know why it would crash later. >>>>>> >>>>>> Can you tell me all the steps with miniforge (which seems to be related to the failure) you use? I've never used miniforge. >>>>>> >>>>>> If I can get to an environment that reproduces the problem I can debug it and fix it. >>>>>> >>>>>> Barry >>>>>> >>>>>> >>>>>>> On Oct 24, 2024, at 11:49?AM, Praveen C wrote: >>>>>>> >>>>>>> I get this >>>>>>> >>>>>>> $ mpiexec -n 3 ./ex89f -n 20 -mpi_linear_solver_server -mpi_linear_solver_server -mpi_linear_solver_server_ksp_view -ksp_monitor -ksp_converged_reason -ksp_view -mpi_linear_solver_server_minimum_count_per_rank 5 >>>>>>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>>>>>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range >>>>>>> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >>>>>>> [0]PETSC ERROR: or see https://urldefense.us/v3/__https://petsc.org/release/faq/*valgrind__;Iw!!G_uCfscf7eWS!YE7EmIAS98p3V1aFvP_9zM7XUxOaR8mpeYeE6-_kpR_YcQyL0xdzNUR-lLIKHvMEn3eMgNIWysJZCLeHNaTdvgE$ and https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!YE7EmIAS98p3V1aFvP_9zM7XUxOaR8mpeYeE6-_kpR_YcQyL0xdzNUR-lLIKHvMEn3eMgNIWysJZCLeHIcTqVdE$ >>>>>>> [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run >>>>>>> [0]PETSC ERROR: to get more information on the crash. >>>>>>> [0]PETSC ERROR: Run with -malloc_debug to check if memory corruption is causing the crash. >>>>>>> -------------------------------------------------------------------------- >>>>>>> MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_SELF >>>>>>> Proc: [[47380,1],0] >>>>>>> Errorcode: 59 >>>>>>> >>>>>>> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. >>>>>>> You may or may not see output from other processes, depending on >>>>>>> exactly when Open MPI kills them. >>>>>>> -------------------------------------------------------------------------- >>>>>>> -------------------------------------------------------------------------- >>>>>>> prterun has exited due to process rank 0 with PID 0 on node MacMiniHome calling >>>>>>> "abort". This may have caused other processes in the application to be >>>>>>> terminated by signals sent by prterun (as reported here). >>>>>>> ????????????????????????????????????? >>>>>>> >>>>>>> and I have this after the code exits >>>>>>> >>>>>>> $ ipcs -m >>>>>>> IPC status from as of Thu Oct 24 21:17:39 IST 2024 >>>>>>> T ID KEY MODE OWNER GROUP >>>>>>> Shared Memory: >>>>>>> m 1572864 0x0000000b --rw-rw-rw- praveen staff >>>>>>> m 524289 0x0000000c --rw-rw-rw- praveen staff >>>>>>> m 655362 0x0000000d --rw-rw-rw- praveen staff >>>>>>> m 262147 0x0000000e --rw-rw-rw- praveen staff >>>>>>> m 262148 0x0000000f --rw-rw-rw- praveen staff >>>>>>> m 393221 0x0000000a --rw-rw-rw- praveen staff >>>>>>> >>>>>>> This is with petsc installed with miniforge, which I also use with clawpack. With spack installed petsc, I can run the ex89f example. >>>>>>> >>>>>>> Thanks >>>>>>> praveen >>>>>>> >>>>>>>> On 24 Oct 2024, at 7:55?PM, Barry Smith wrote: >>>>>>>> >>>>>>>> >>>>>>>> Ok, super strange, I've run many times on my Mac >>>>>>>> >>>>>>>> Can you please try to remove that allocated memory with ipcrm and then >>>>>>>> >>>>>>>> cd $PETSC_DIR/src/ksp/ksp/tutorials >>>>>>>> make ex89f >>>>>>>> mpiexec -n 3 ./ex89f -n 20 -mpi_linear_solver_server -mpi_linear_solver_server -mpi_linear_solver_server_ksp_view -ksp_monitor -ksp_converged_reason -ksp_view -mpi_linear_solver_server_minimum_count_per_rank 5 >>>>>>>> >>>>>>>> This does the same thing as the GeoClaw code but is much simpler. >>>>>>>> >>>>>>>> Barry >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> On Oct 23, 2024, at 10:55?PM, Praveen C wrote: >>>>>>>>> >>>>>>>>> I get very similar error on my mac with >>>>>>>>> >>>>>>>>> $ gfortran -v >>>>>>>>> Using built-in specs. >>>>>>>>> COLLECT_GCC=gfortran >>>>>>>>> COLLECT_LTO_WRAPPER=/opt/homebrew/Caskroom/miniforge/base/envs/claw/libexec/gcc/arm64-apple-darwin20.0.0/13.2.0/lto-wrapper >>>>>>>>> Target: arm64-apple-darwin20.0.0 >>>>>>>>> Configured with: ../configure --prefix=/opt/homebrew/Caskroom/miniforge/base/envs/claw --build=x86_64-apple-darwin13.4.0 --host=arm64-apple-darwin20.0.0 --target=arm64-apple-darwin20.0.0 --with-libiconv-prefix=/opt/homebrew/Caskroom/miniforge/base/envs/claw --enable-languages=fortran --disable-multilib --enable-checking=release --disable-bootstrap --disable-libssp --with-gmp=/opt/homebrew/Caskroom/miniforge/base/envs/claw --with-mpfr=/opt/homebrew/Caskroom/miniforge/base/envs/claw --with-mpc=/opt/homebrew/Caskroom/miniforge/base/envs/claw --with-isl=/opt/homebrew/Caskroom/miniforge/base/envs/claw --enable-darwin-at-rpath >>>>>>>>> Thread model: posix >>>>>>>>> Supported LTO compression algorithms: zlib >>>>>>>>> gcc version 13.2.0 (GCC) >>>>>>>>> >>>>>>>>> Before starting >>>>>>>>> >>>>>>>>> $ ipcs -m >>>>>>>>> IPC status from as of Thu Oct 24 08:02:11 IST 2024 >>>>>>>>> T ID KEY MODE OWNER GROUP >>>>>>>>> Shared Memory: >>>>>>>>> >>>>>>>>> and when I run the code >>>>>>>>> >>>>>>>>> Using a PETSc solver >>>>>>>>> Using Bouss equations from the start >>>>>>>>> rnode allocated... >>>>>>>>> node allocated... >>>>>>>>> listOfGrids allocated... >>>>>>>>> Storage allocated... >>>>>>>>> bndList allocated... >>>>>>>>> Gridding level 1 at t = 0.000000E+00: 4 grids with 10000 cells >>>>>>>>> Setting initial dt to 2.9999999999999999E-002 >>>>>>>>> max threads set to 1 >>>>>>>>> >>>>>>>>> Done reading data, starting computation ... >>>>>>>>> >>>>>>>>> Total zeta at initial time: 39269.907650665169 >>>>>>>>> GEOCLAW: Frame 0 output files done at time t = 0.000000D+00 >>>>>>>>> >>>>>>>>> [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >>>>>>>>> [0]PETSC ERROR: Petsc has generated inconsistent data >>>>>>>>> [0]PETSC ERROR: Unable to locate PCMPI allocated shared address 0x130698000 >>>>>>>>> [0]PETSC ERROR: WARNING! There are unused option(s) set! Could be the program crashed before usage or a spelling mistake, etc! >>>>>>>>> [0]PETSC ERROR: Option left: name:-ksp_type value: preonly source: file >>>>>>>>> [0]PETSC ERROR: Option left: name:-mpi_ksp_max_it value: 200 source: file >>>>>>>>> [0]PETSC ERROR: Option left: name:-mpi_ksp_reuse_preconditioner (no value) source: file >>>>>>>>> [0]PETSC ERROR: Option left: name:-mpi_ksp_rtol value: 1.e-9 source: file >>>>>>>>> [0]PETSC ERROR: Option left: name:-mpi_ksp_type value: gmres source: file >>>>>>>>> [0]PETSC ERROR: Option left: name:-mpi_linear_solver_server_view (no value) source: file >>>>>>>>> [0]PETSC ERROR: Option left: name:-mpi_pc_gamg_sym_graph value: true source: file >>>>>>>>> [0]PETSC ERROR: Option left: name:-mpi_pc_gamg_symmetrize_graph value: true source: file >>>>>>>>> [0]PETSC ERROR: Option left: name:-mpi_pc_type value: gamg source: file >>>>>>>>> [0]PETSC ERROR: Option left: name:-pc_mpi_minimum_count_per_rank value: 5000 source: file >>>>>>>>> [0]PETSC ERROR: Option left: name:-pc_type value: mpi source: file >>>>>>>>> [0]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!YE7EmIAS98p3V1aFvP_9zM7XUxOaR8mpeYeE6-_kpR_YcQyL0xdzNUR-lLIKHvMEn3eMgNIWysJZCLeHIcTqVdE$ for trouble shooting. >>>>>>>>> [0]PETSC ERROR: Petsc Release Version 3.22.0, Sep 28, 2024 >>>>>>>>> [0]PETSC ERROR: /Users/praveen/work/bouss/radial_flat/xgeoclaw with 6 MPI process(es) and PETSC_ARCH on MacMiniHome.local by praveen Thu Oct 24 08:04:27 2024 >>>>>>>>> [0]PETSC ERROR: Configure options: AR=arm64-apple-darwin20.0.0-ar CC=mpicc CXX=mpicxx FC=mpifort CFLAGS="-ftree-vectorize -fPIC -fstack-protector-strong -O2 -pipe -isystem /opt/homebrew/Caskroom/miniforge/base/envs/claw/include " CPPFLAGS="-D_FORTIFY_SOURCE=2 -isystem /opt/homebrew/Caskroom/miniforge/base/envs/claw/include -mmacosx-version-min=11.0 -mmacosx-version-min=11.0" CXXFLAGS="-ftree-vectorize -fPIC -fstack-protector-strong -O2 -pipe -stdlib=libc++ -fvisibility-inlines-hidden -fmessage-length=0 -isystem /opt/homebrew/Caskroom/miniforge/base/envs/claw/include " FFLAGS="-march=armv8.3-a -ftree-vectorize -fPIC -fno-stack-protector -O2 -pipe -isystem /opt/homebrew/Caskroom/miniforge/base/envs/claw/include " LDFLAGS="-Wl,-headerpad_max_install_names -Wl,-dead_strip_dylibs -Wl,-rpath,/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib -L/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib" LIBS="-Wl,-rpath,/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib -lmpi_mpifh -lgfortran" --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 --FOPTFLAGS=-O3 --with-clib-autodetect=0 --with-cxxlib-autodetect=0 --with-fortranlib-autodetect=0 --with-debugging=0 --with-blas-lib=libblas.dylib --with-lapack-lib=liblapack.dylib --with-yaml=1 --with-hdf5=1 --with-fftw=1 --with-hwloc=0 --with-hypre=1 --with-metis=1 --with-mpi=1 --with-mumps=1 --with-parmetis=1 --with-pthread=1 --with-ptscotch=1 --with-shared-libraries --with-ssl=0 --with-scalapack=1 --with-superlu=1 --with-superlu_dist=1 --with-superlu_dist-include=/opt/homebrew/Caskroom/miniforge/base/envs/claw/include/superlu-dist --with-superlu_dist-lib=-lsuperlu_dist --with-suitesparse=1 --with-suitesparse-dir=/opt/homebrew/Caskroom/miniforge/base/envs/claw --with-x=0 --with-scalar-type=real --with-cuda=0 --with-batch --prefix=/opt/homebrew/Caskroom/miniforge/base/envs/claw >>>>>>>>> [0]PETSC ERROR: #1 PetscShmgetMapAddresses() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/sys/utils/server.c:114 >>>>>>>>> [0]PETSC ERROR: #2 PCMPISetMat() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/ksp/pc/impls/mpi/pcmpi.c:269 >>>>>>>>> [0]PETSC ERROR: #3 PCSetUp_MPI() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/ksp/pc/impls/mpi/pcmpi.c:853 >>>>>>>>> [0]PETSC ERROR: #4 PCSetUp() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/ksp/pc/interface/precon.c:1071 >>>>>>>>> [0]PETSC ERROR: #5 KSPSetUp() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/ksp/ksp/interface/itfunc.c:415 >>>>>>>>> [0]PETSC ERROR: #6 KSPSolve_Private() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/ksp/ksp/interface/itfunc.c:826 >>>>>>>>> [0]PETSC ERROR: #7 KSPSolve() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/ksp/ksp/interface/itfunc.c:1075 >>>>>>>>> >>>>>>>>> Code does not progress and I kill it >>>>>>>>> >>>>>>>>> ^CTraceback (most recent call last): >>>>>>>>> File "/Users/praveen/Applications/clawpack/clawutil/src/python/clawutil/runclaw.py", line 341, in >>>>>>>>> runclaw(*args) >>>>>>>>> File "/Users/praveen/Applications/clawpack/clawutil/src/python/clawutil/runclaw.py", line 242, in runclaw >>>>>>>>> proc = subprocess.check_call(cmd_split, >>>>>>>>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ >>>>>>>>> File "/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib/python3.12/subprocess.py", line 408, in check_call >>>>>>>>> retcode = call(*popenargs, **kwargs) >>>>>>>>> ^^^^^^^^^^^^^^^^^^^^^^^^^^ >>>>>>>>> File "/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib/python3.12/subprocess.py", line 391, in call >>>>>>>>> return p.wait(timeout=timeout) >>>>>>>>> ^^^^^^^^^^^^^^^^^^^^^^^ >>>>>>>>> File "/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib/python3.12/subprocess.py", line 1264, in wait >>>>>>>>> return self._wait(timeout=timeout) >>>>>>>>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^ >>>>>>>>> File "/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib/python3.12/subprocess.py", line 2053, in _wait >>>>>>>>> (pid, sts) = self._try_wait(0) >>>>>>>>> ^^^^^^^^^^^^^^^^^ >>>>>>>>> File "/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib/python3.12/subprocess.py", line 2011, in _try_wait >>>>>>>>> (pid, sts) = os.waitpid(self.pid, wait_flags) >>>>>>>>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ >>>>>>>>> KeyboardInterrupt >>>>>>>>> make[1]: *** [output] Interrupt: 2 >>>>>>>>> make: *** [.output] Interrupt: 2 >>>>>>>>> >>>>>>>>> Now it says >>>>>>>>> >>>>>>>>> $ ipcs -m >>>>>>>>> IPC status from as of Thu Oct 24 08:05:06 IST 2024 >>>>>>>>> T ID KEY MODE OWNER GROUP >>>>>>>>> Shared Memory: >>>>>>>>> m 720896 0x0000000a --rw-rw-rw- praveen staff >>>>>>>>> >>>>>>>>> Thanks >>>>>>>>> praveen >>>>>>>>> >>>>>>>>>> On 23 Oct 2024, at 8:26?PM, Barry Smith wrote: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Hmm, so it is creating the first shared memory region with ID of 10 (A in hex) puts it in a linked list in PETSc but then when it tries to find it in the linked list it cannot find it. >>>>>>>>>> >>>>>>>>>> I don't know how to reproduce this or debug it remotely. >>>>>>>>>> >>>>>>>>>> Can you build on a completely different machine or with completely different compilers? >>>>>>>>>> >>>>>>>>>> Barry >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> On Oct 23, 2024, at 10:31?AM, Praveen C wrote: >>>>>>>>>>> >>>>>>>>>>> I get same error and now it shows >>>>>>>>>>> >>>>>>>>>>> $ ipcs -m >>>>>>>>>>> >>>>>>>>>>> ------ Shared Memory Segments -------- >>>>>>>>>>> key shmid owner perms bytes nattch status >>>>>>>>>>> 0x0000000a 32788 praveen 666 240 6 >>>>>>>>>>> >>>>>>>>>>> Note that the code seems to be still running after printing those error message, but it is not printing any progress which it should do. >>>>>>>>>>> >>>>>>>>>>> Thanks >>>>>>>>>>> praveen >>>>>>>>>>> >>>>>>>>>>>> On 23 Oct 2024, at 7:56?PM, Barry Smith wrote: >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Try >>>>>>>>>>>> >>>>>>>>>>>> ipcrm -m 11 >>>>>>>>>>>> >>>>>>>>>>>> ipcs -m >>>>>>>>>>>> >>>>>>>>>>>> Try running the program again >>>>>>>>>>>> >>>>>>>>>>>> If failed check >>>>>>>>>>>> >>>>>>>>>>>> ipcs -m >>>>>>>>>>>> >>>>>>>>>>>> again >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> On Oct 23, 2024, at 10:20?AM, Praveen C wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> Hello Barry >>>>>>>>>>>>> >>>>>>>>>>>>> I see this >>>>>>>>>>>>> >>>>>>>>>>>>> $ ipcs -m >>>>>>>>>>>>> >>>>>>>>>>>>> ------ Shared Memory Segments -------- >>>>>>>>>>>>> key shmid owner perms bytes nattch status >>>>>>>>>>>>> 0x0000000a 11 praveen 666 240 6 >>>>>>>>>>>>> >>>>>>>>>>>>> and I am observing same error as below. >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks >>>>>>>>>>>>> praveen >>>>>>>>>>>>> >>>>>>>>>>>>>> On 23 Oct 2024, at 7:08?PM, Barry Smith wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Please take a look at the notes in https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/PetscShmgetAllocateArray/__;!!G_uCfscf7eWS!YE7EmIAS98p3V1aFvP_9zM7XUxOaR8mpeYeE6-_kpR_YcQyL0xdzNUR-lLIKHvMEn3eMgNIWysJZCLeHVF9gqZE$ For some reason your program is not able to access/use the Unix shared memory; check if you are already using the shared memory (so it is not available for a new run) or the limits are too low to access enough memory. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Barry >>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Oct 23, 2024, at 8:23?AM, Praveen C wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Dear all >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I am not able to run the boussinesq example from geoclaw using petsc at 3.22.0 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> https://urldefense.us/v3/__https://github.com/clawpack/geoclaw/tree/3303883f46572c58130d161986b8a87a57ca7816/examples/bouss__;!!G_uCfscf7eWS!e3VQ4NHKmXGstRsQW5vtI7fmKfUT9zmJkMJcPbcvPyIjicyfJpNoMgx3wZ-qyGcKNSjIkNZkzilec8MnHN6PMw$ >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> It runs with petsc at 3.21.6 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> The error I get is given below. After printing this, the code does not progress. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I use the following petsc options >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> # set min numbers of matrix rows per MPI rank (default is 10000) >>>>>>>>>>>>>>> -mpi_linear_solve_minimum_count_per_rank 5000 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> # Krylov linear solver: >>>>>>>>>>>>>>> -mpi_linear_solver_server >>>>>>>>>>>>>>> -mpi_linear_solver_server_view >>>>>>>>>>>>>>> -ksp_type gmres >>>>>>>>>>>>>>> -ksp_max_it 200 >>>>>>>>>>>>>>> -ksp_reuse_preconditioner >>>>>>>>>>>>>>> -ksp_rtol 1.e-9 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> # preconditioner: >>>>>>>>>>>>>>> -pc_type gamg >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I installed petsc and other dependencies for clawpack using miniforge. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks >>>>>>>>>>>>>>> pc >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> ==> Use Bouss. in water deeper than 1.0000000000000000 Using a PETSc solver >>>>>>>>>>>>>>> Using Bouss equations from the start >>>>>>>>>>>>>>> rnode allocated... >>>>>>>>>>>>>>> node allocated... >>>>>>>>>>>>>>> listOfGrids allocated... >>>>>>>>>>>>>>> Storage allocated... >>>>>>>>>>>>>>> bndList allocated... >>>>>>>>>>>>>>> Gridding level 1 at t = 0.000000E+00: 4 grids with 10000 cells >>>>>>>>>>>>>>> Setting initial dt to 2.9999999999999999E-002 >>>>>>>>>>>>>>> max threads set to 6 >>>>>>>>>>>>>>> Done reading data, starting computation ... Total zeta at initial time: 39269.907650665169 GEOCLAW: Frame 0 output files done at time t = 0.000000D+00 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >>>>>>>>>>>>>>> [0]PETSC ERROR: Petsc has generated inconsistent data >>>>>>>>>>>>>>> [0]PETSC ERROR: Unable to locate PCMPI allocated shared address 0x55e6d750ae20 >>>>>>>>>>>>>>> [0]PETSC ERROR: WARNING! There are unused option(s) set! Could be the program crashed before usage or a spelling mistake, etc! >>>>>>>>>>>>>>> [0]PETSC ERROR: Option left: name:-ksp_max_it value: 200 source: file >>>>>>>>>>>>>>> [0]PETSC ERROR: Option left: name:-ksp_reuse_preconditioner (no value) source: file >>>>>>>>>>>>>>> [0]PETSC ERROR: Option left: name:-ksp_rtol value: 1.e-9 source: file >>>>>>>>>>>>>>> [0]PETSC ERROR: Option left: name:-ksp_type value: gmres source: file >>>>>>>>>>>>>>> [0]PETSC ERROR: Option left: name:-mpi_linear_solve_minimum_count_per_rank value: 5000 source: file >>>>>>>>>>>>>>> [0]PETSC ERROR: Option left: name:-mpi_linear_solver_server_view (no value) source: file >>>>>>>>>>>>>>> [0]PETSC ERROR: Option left: name:-pc_type value: gamg source: file >>>>>>>>>>>>>>> [0]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!e3VQ4NHKmXGstRsQW5vtI7fmKfUT9zmJkMJcPbcvPyIjicyfJpNoMgx3wZ-qyGcKNSjIkNZkzilec8MvNjNo7A$ for trouble shooting. >>>>>>>>>>>>>>> [0]PETSC ERROR: Petsc Release Version 3.22.0, Sep 28, 2024 [0]PETSC ERROR: /home/praveen/bouss/radial_flat/xgeoclaw with 6 MPI process(es) and PETSC_ARCH on euler by praveen Thu Oct 17 21:49:54 2024 >>>>>>>>>>>>>>> [0]PETSC ERROR: Configure options: AR=${PREFIX}/bin/x86_64-conda-linux-gnu-ar CC=mpicc CXX=mpicxx FC=mpifort CFLAGS="-march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /opt/miniforge/envs/claw/include " CPPFLAGS="-DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /opt/miniforge/envs/claw/include" CXXFLAGS="-fvisibility-inlines-hidden -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /opt/miniforge/envs/claw/include " FFLAGS="-march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /opt/miniforge/envs/claw/include -Wl,--no-as-needed" LDFLAGS="-pthread -fopenmp -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,--allow-shlib-undefined -Wl,-rpath,/opt/miniforge/envs/claw/lib -Wl,-rpath-link,/opt/miniforge/envs/claw/lib -L/opt/miniforge/envs/claw/lib -Wl,-rpath-link,/opt/miniforge/envs/claw/lib" LIBS="-Wl,-rpath,/opt/miniforge/envs/claw/lib -lmpi_mpifh -lgfortran" --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 --FOPTFLAGS=-O3 --with-clib-autodetect=0 --with-cxxlib-autodetect=0 --with-fortranlib-autodetect=0 --with-debugging=0 --with-blas-lib=libblas.so --with-lapack-lib=liblapack.so --with-yaml=1 --with-hdf5=1 --with-fftw=1 --with-hwloc=0 --with-hypre=1 --with-metis=1 --with-mpi=1 --with-mumps=1 --with-parmetis=1 --with-pthread=1 --with-ptscotch=1 --with-shared-libraries --with-ssl=0 --with-scalapack=1 --with-superlu=1 --with-superlu_dist=1 --with-superlu_dist-include=/opt/miniforge/envs/claw/include/superlu-dist --with-superlu_dist-lib=-lsuperlu_dist --with-suitesparse=1 --with-suitesparse-dir=/opt/miniforge/envs/claw --with-x=0 --with-scalar-type=real --with-cuda=0 --prefix=/opt/miniforge/envs/claw >>>>>>>>>>>>>>> [0]PETSC ERROR: #1 PetscShmgetMapAddresses() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/sys/utils/server.c:114 >>>>>>>>>>>>>>> [0]PETSC ERROR: #2 PCMPISetMat() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/pc/impls/mpi/pcmpi.c:269 >>>>>>>>>>>>>>> [0]PETSC ERROR: #3 PCSetUp_MPI() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/pc/impls/mpi/pcmpi.c:853 >>>>>>>>>>>>>>> [0]PETSC ERROR: #4 PCSetUp() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/pc/interface/precon.c:1071 >>>>>>>>>>>>>>> [0]PETSC ERROR: #5 KSPSetUp() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/ksp/interface/itfunc.c:415 >>>>>>>>>>>>>>> [0]PETSC ERROR: #6 KSPSolve_Private() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/ksp/interface/itfunc.c:826 >>>>>>>>>>>>>>> [0]PETSC ERROR: #7 KSPSolve() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/ksp/interface/itfunc.c:1075 >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: