From danyang.su at gmail.com Mon Jun 1 15:10:36 2020 From: danyang.su at gmail.com (Danyang Su) Date: Mon, 01 Jun 2020 13:10:36 -0700 Subject: [petsc-users] Agglomeration for Multigrid on Unstructured Meshes Message-ID: <4BDDF627-6CA6-4146-AD76-25BD566CCAB7@gmail.com> Dear All, I recalled there was a presentation ?Extreme-scale multigrid components with PETSc? taling about agglomeration in parallel multigrid, with future plan to extend to support unstructured meshes. Is this under development or to be added? Thanks and regards, Danyang -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Mon Jun 1 15:47:14 2020 From: jed at jedbrown.org (Jed Brown) Date: Mon, 01 Jun 2020 14:47:14 -0600 Subject: [petsc-users] Agglomeration for Multigrid on Unstructured Meshes In-Reply-To: <4BDDF627-6CA6-4146-AD76-25BD566CCAB7@gmail.com> References: <4BDDF627-6CA6-4146-AD76-25BD566CCAB7@gmail.com> Message-ID: <87eeqycy3x.fsf@jedbrown.org> I assume you're talking about repartitioning of coarse grids in geometric multigrid -- that hasn't been implemented. https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCTELESCOPE.html But you can use an algebraic multigrid that does similar communicator reduction, and can be applied to the original global problem or just on the "coarse" problem of an initial geometric hierarchy. Danyang Su writes: > Dear All, > > > > I recalled there was a presentation ?Extreme-scale multigrid components with PETSc? taling about agglomeration in parallel multigrid, with future plan to extend to support unstructured meshes. Is this under development or to be added? > > > > Thanks and regards, > > > > Danyang From danyang.su at gmail.com Mon Jun 1 18:02:21 2020 From: danyang.su at gmail.com (Danyang Su) Date: Mon, 01 Jun 2020 16:02:21 -0700 Subject: [petsc-users] Agglomeration for Multigrid on Unstructured Meshes In-Reply-To: <87eeqycy3x.fsf@jedbrown.org> References: <4BDDF627-6CA6-4146-AD76-25BD566CCAB7@gmail.com> <87eeqycy3x.fsf@jedbrown.org> Message-ID: <9E53C33F-5BA1-414E-989D-EBDF03AAA218@gmail.com> Thanks Jed for the quick response. Yes I am asking about the repartitioning of coarse grids in geometric multigrid for unstructured mesh. I am happy with AMG. Thanks for letting me know. Danyang ?On 2020-06-01, 1:47 PM, "Jed Brown" wrote: I assume you're talking about repartitioning of coarse grids in geometric multigrid -- that hasn't been implemented. https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCTELESCOPE.html But you can use an algebraic multigrid that does similar communicator reduction, and can be applied to the original global problem or just on the "coarse" problem of an initial geometric hierarchy. Danyang Su writes: > Dear All, > > > > I recalled there was a presentation ?Extreme-scale multigrid components with PETSc? taling about agglomeration in parallel multigrid, with future plan to extend to support unstructured meshes. Is this under development or to be added? > > > > Thanks and regards, > > > > Danyang From knepley at gmail.com Mon Jun 1 20:29:07 2020 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 1 Jun 2020 21:29:07 -0400 Subject: [petsc-users] Agglomeration for Multigrid on Unstructured Meshes In-Reply-To: <9E53C33F-5BA1-414E-989D-EBDF03AAA218@gmail.com> References: <4BDDF627-6CA6-4146-AD76-25BD566CCAB7@gmail.com> <87eeqycy3x.fsf@jedbrown.org> <9E53C33F-5BA1-414E-989D-EBDF03AAA218@gmail.com> Message-ID: On Mon, Jun 1, 2020 at 7:03 PM Danyang Su wrote: > Thanks Jed for the quick response. Yes I am asking about the > repartitioning of coarse grids in geometric multigrid for unstructured > mesh. I am happy with AMG. Thanks for letting me know. > All the pieces are there, we just have not had users asking for this, and it will take some work to put together. Thanks, Matt > Danyang > > ?On 2020-06-01, 1:47 PM, "Jed Brown" wrote: > > I assume you're talking about repartitioning of coarse grids in > geometric multigrid -- that hasn't been implemented. > > > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCTELESCOPE.html > > But you can use an algebraic multigrid that does similar communicator > reduction, and can be applied to the original global problem or just on > the "coarse" problem of an initial geometric hierarchy. > > Danyang Su writes: > > > Dear All, > > > > > > > > I recalled there was a presentation ?Extreme-scale multigrid > components with PETSc? taling about agglomeration in parallel multigrid, > with future plan to extend to support unstructured meshes. Is this under > development or to be added? > > > > > > > > Thanks and regards, > > > > > > > > Danyang > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From gautam.bisht at pnnl.gov Mon Jun 1 22:55:18 2020 From: gautam.bisht at pnnl.gov (Bisht, Gautam) Date: Tue, 2 Jun 2020 03:55:18 +0000 Subject: [petsc-users] DMPlex Exodus II transient variable Message-ID: Hi Matt, I came across ex26.c in DMPlex that writes out transient variables in an exodus II format. While VisIt can display the mesh, I can't figure out why the transient variables aren't showing up for display (see the screenshot). Do you know what is going wrong here? -Gautam. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Screen Shot 2020-06-01 at 8.42.07 PM.png Type: image/png Size: 1244152 bytes Desc: Screen Shot 2020-06-01 at 8.42.07 PM.png URL: From dave.mayhem23 at gmail.com Mon Jun 1 23:43:24 2020 From: dave.mayhem23 at gmail.com (Dave May) Date: Tue, 2 Jun 2020 06:43:24 +0200 Subject: [petsc-users] Agglomeration for Multigrid on Unstructured Meshes In-Reply-To: References: <4BDDF627-6CA6-4146-AD76-25BD566CCAB7@gmail.com> <87eeqycy3x.fsf@jedbrown.org> <9E53C33F-5BA1-414E-989D-EBDF03AAA218@gmail.com> Message-ID: On Tue 2. Jun 2020 at 03:30, Matthew Knepley wrote: > On Mon, Jun 1, 2020 at 7:03 PM Danyang Su wrote: > >> Thanks Jed for the quick response. Yes I am asking about the >> repartitioning of coarse grids in geometric multigrid for unstructured >> mesh. I am happy with AMG. Thanks for letting me know. >> > > All the pieces are there, we just have not had users asking for this, and > it will take some work to put together. > Matt - I created a branch for you and Lawrence last year which added full support for PLEX within Telescope. This implementation was not a fully automated algmoeration strategy - it utilized the partition associated with the DM returned from DMGetCoarseDM. Hence the job of building the distributed coarse hierarchy was let to the user. I?m pretty sure that code got merged into master as the branch also contained several bug mixes for Telescope. Or am I mistaken? Cheers Dave > Thanks, > > Matt > > >> Danyang >> >> ?On 2020-06-01, 1:47 PM, "Jed Brown" wrote: >> >> I assume you're talking about repartitioning of coarse grids in >> geometric multigrid -- that hasn't been implemented. >> >> >> https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCTELESCOPE.html >> >> But you can use an algebraic multigrid that does similar communicator >> reduction, and can be applied to the original global problem or just >> on >> the "coarse" problem of an initial geometric hierarchy. >> >> Danyang Su writes: >> >> > Dear All, >> > >> > >> > >> > I recalled there was a presentation ?Extreme-scale multigrid >> components with PETSc? taling about agglomeration in parallel multigrid, >> with future plan to extend to support unstructured meshes. Is this under >> development or to be added? >> > >> > >> > >> > Thanks and regards, >> > >> > >> > >> > Danyang >> >> >> > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wencel at gmail.com Tue Jun 2 03:25:42 2020 From: wencel at gmail.com (Lawrence Mitchell) Date: Tue, 2 Jun 2020 09:25:42 +0100 Subject: [petsc-users] Agglomeration for Multigrid on Unstructured Meshes In-Reply-To: References: <4BDDF627-6CA6-4146-AD76-25BD566CCAB7@gmail.com> <87eeqycy3x.fsf@jedbrown.org> <9E53C33F-5BA1-414E-989D-EBDF03AAA218@gmail.com> Message-ID: Hi Dave, > On 2 Jun 2020, at 05:43, Dave May wrote: > > > > On Tue 2. Jun 2020 at 03:30, Matthew Knepley wrote: > On Mon, Jun 1, 2020 at 7:03 PM Danyang Su wrote: > Thanks Jed for the quick response. Yes I am asking about the repartitioning of coarse grids in geometric multigrid for unstructured mesh. I am happy with AMG. Thanks for letting me know. > > All the pieces are there, we just have not had users asking for this, and it will take some work to put together. > > Matt - I created a branch for you and Lawrence last year which added full support for PLEX within Telescope. This implementation was not a fully automated algmoeration strategy - it utilized the partition associated with the DM returned from DMGetCoarseDM. Hence the job of building the distributed coarse hierarchy was let to the user. > > I?m pretty sure that code got merged into master as the branch also contained several bug mixes for Telescope. Or am I mistaken? I think you're right. I didn't manage to get the redistribution of the DMPlex object done last summer (it's bubbling up again). As I see it, for redistributed geometric multigrid on plexes, the missing piece is a function: DMPlexRedistributeOntoComm(DM old, MPI_Comm comm, DM *new) I went down a rabbit hole of trying to do this, since I actually think this replaced the current interface to DMPlexDistribute, which is DMPlexDistribute(DM old, PetscInt overlap, PetscSF *pointDistSF, DM *new) Where the new DM comes out on the same communicator as the old DM, just with a different partition. This has lots of follow-on consequences, for example, one can't easily load on P processes and then compute on Q. Unfortunately, collectiveness over MPI_Comm(old) is baked into the redistribution routines everywhere, and I didn't manage to finish things. Lawrence From knepley at gmail.com Tue Jun 2 03:35:57 2020 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 2 Jun 2020 04:35:57 -0400 Subject: [petsc-users] Agglomeration for Multigrid on Unstructured Meshes In-Reply-To: References: <4BDDF627-6CA6-4146-AD76-25BD566CCAB7@gmail.com> <87eeqycy3x.fsf@jedbrown.org> <9E53C33F-5BA1-414E-989D-EBDF03AAA218@gmail.com> Message-ID: On Tue, Jun 2, 2020 at 4:25 AM Lawrence Mitchell wrote: > Hi Dave, > > > On 2 Jun 2020, at 05:43, Dave May wrote: > > > > > > > > On Tue 2. Jun 2020 at 03:30, Matthew Knepley wrote: > > On Mon, Jun 1, 2020 at 7:03 PM Danyang Su wrote: > > Thanks Jed for the quick response. Yes I am asking about the > repartitioning of coarse grids in geometric multigrid for unstructured > mesh. I am happy with AMG. Thanks for letting me know. > > > > All the pieces are there, we just have not had users asking for this, > and it will take some work to put together. > > > > Matt - I created a branch for you and Lawrence last year which added > full support for PLEX within Telescope. This implementation was not a fully > automated algmoeration strategy - it utilized the partition associated with > the DM returned from DMGetCoarseDM. Hence the job of building the > distributed coarse hierarchy was let to the user. > > > > I?m pretty sure that code got merged into master as the branch also > contained several bug mixes for Telescope. Or am I mistaken? > > I think you're right. I didn't manage to get the redistribution of the > DMPlex object done last summer (it's bubbling up again). > > As I see it, for redistributed geometric multigrid on plexes, the missing > piece is a function: > > DMPlexRedistributeOntoComm(DM old, MPI_Comm comm, DM *new) > > I went down a rabbit hole of trying to do this, since I actually think > this replaced the current interface to DMPlexDistribute, which is > > DMPlexDistribute(DM old, PetscInt overlap, PetscSF *pointDistSF, DM *new) > > Where the new DM comes out on the same communicator as the old DM, just > with a different partition. > > This has lots of follow-on consequences, for example, one can't easily > load on P processes and then compute on Q. > > Unfortunately, collectiveness over MPI_Comm(old) is baked into the > redistribution routines everywhere, and I didn't manage to finish things. > Yes, I remember thinking this out. I believe the conclusion was that redistribution should happen on the large comm, which some fraction of processes getting no cells. Then at the end we call one new function which copies that DM onto the smaller comm. Matt > Lawrence -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From wence at gmx.li Tue Jun 2 03:49:51 2020 From: wence at gmx.li (Lawrence Mitchell) Date: Tue, 2 Jun 2020 09:49:51 +0100 Subject: [petsc-users] Agglomeration for Multigrid on Unstructured Meshes In-Reply-To: References: <4BDDF627-6CA6-4146-AD76-25BD566CCAB7@gmail.com> <87eeqycy3x.fsf@jedbrown.org> <9E53C33F-5BA1-414E-989D-EBDF03AAA218@gmail.com> Message-ID: <8ADFE4F6-7F2D-456C-8154-9DB18BB36FD0@gmx.li> > On 2 Jun 2020, at 09:35, Matthew Knepley wrote: > > On Tue, Jun 2, 2020 at 4:25 AM Lawrence Mitchell wrote: > Hi Dave, > > > On 2 Jun 2020, at 05:43, Dave May wrote: > > > > > > > > On Tue 2. Jun 2020 at 03:30, Matthew Knepley wrote: > > On Mon, Jun 1, 2020 at 7:03 PM Danyang Su wrote: > > Thanks Jed for the quick response. Yes I am asking about the repartitioning of coarse grids in geometric multigrid for unstructured mesh. I am happy with AMG. Thanks for letting me know. > > > > All the pieces are there, we just have not had users asking for this, and it will take some work to put together. > > > > Matt - I created a branch for you and Lawrence last year which added full support for PLEX within Telescope. This implementation was not a fully automated algmoeration strategy - it utilized the partition associated with the DM returned from DMGetCoarseDM. Hence the job of building the distributed coarse hierarchy was let to the user. > > > > I?m pretty sure that code got merged into master as the branch also contained several bug mixes for Telescope. Or am I mistaken? > > I think you're right. I didn't manage to get the redistribution of the DMPlex object done last summer (it's bubbling up again). > > As I see it, for redistributed geometric multigrid on plexes, the missing piece is a function: > > DMPlexRedistributeOntoComm(DM old, MPI_Comm comm, DM *new) > > I went down a rabbit hole of trying to do this, since I actually think this replaced the current interface to DMPlexDistribute, which is > > DMPlexDistribute(DM old, PetscInt overlap, PetscSF *pointDistSF, DM *new) > > Where the new DM comes out on the same communicator as the old DM, just with a different partition. > > This has lots of follow-on consequences, for example, one can't easily load on P processes and then compute on Q. > > Unfortunately, collectiveness over MPI_Comm(old) is baked into the redistribution routines everywhere, and I didn't manage to finish things. > > Yes, I remember thinking this out. I believe the conclusion was that redistribution should happen on the large comm, which > some fraction of processes getting no cells. Then at the end we call one new function which copies that DM onto the smaller comm. I think this kind of works, but I worry that it makes the following usecase really painful. 1. Parallel load on P processes 2. Redistribute onto Q > P processes ... Compute 3. (Occasionally), checkpoint by mapping onto P processes and dumping Perhaps I don't understand your proposal properly, but I think you're saying that the _input_ DM in redistribute defines the processes that participate in the redistribution (and the _output_ DM just controls the size). This would then require me in my example to load on Q processes (not P) or else do some futzing around. Or do I have the wrong end of the stick? I think I would prefer an interface where I go from commOld -> commNew with the number of partitions determined by size(commNew) Then the restriction you have on the two communicators is one of: i. commOld \subseteq commNew ii. commOld \supseteq commNew I guess if you wanted to be really fancy you could make an intercommunicator of commOld \cap commNew, but now you have to carry another communicator around with you for data transfer etc... Perhaps internally you do this by making a DM on whichever the smaller of the two communicators is that is empty on a bunch of ranks and then copying in (or copying out) appropriately. I think a first step here is to break the assumption everywhere in the redistribution interface that the size of commOld determines the number of partitions you want, and instead move to using size(commNew). Then for now we just pass commNew=commOld. Lawrence From knepley at gmail.com Tue Jun 2 03:50:28 2020 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 2 Jun 2020 04:50:28 -0400 Subject: [petsc-users] DMPlex Exodus II transient variable In-Reply-To: References: Message-ID: On Mon, Jun 1, 2020 at 11:56 PM Bisht, Gautam via petsc-users < petsc-users at mcs.anl.gov> wrote: > Hi Matt, > > I came across ex26.c > in > DMPlex that writes out transient variables in an exodus II format. While > VisIt can display the mesh, I can't figure out why the transient variables > aren't showing up for display (see the screenshot). Do you know what is > going wrong here? > Are they in the "vector" data? Matt > -Gautam. > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Jun 2 03:54:32 2020 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 2 Jun 2020 04:54:32 -0400 Subject: [petsc-users] Agglomeration for Multigrid on Unstructured Meshes In-Reply-To: <8ADFE4F6-7F2D-456C-8154-9DB18BB36FD0@gmx.li> References: <4BDDF627-6CA6-4146-AD76-25BD566CCAB7@gmail.com> <87eeqycy3x.fsf@jedbrown.org> <9E53C33F-5BA1-414E-989D-EBDF03AAA218@gmail.com> <8ADFE4F6-7F2D-456C-8154-9DB18BB36FD0@gmx.li> Message-ID: On Tue, Jun 2, 2020 at 4:49 AM Lawrence Mitchell wrote: > > On 2 Jun 2020, at 09:35, Matthew Knepley wrote: > > > > On Tue, Jun 2, 2020 at 4:25 AM Lawrence Mitchell > wrote: > > Hi Dave, > > > > > On 2 Jun 2020, at 05:43, Dave May wrote: > > > > > > > > > > > > On Tue 2. Jun 2020 at 03:30, Matthew Knepley > wrote: > > > On Mon, Jun 1, 2020 at 7:03 PM Danyang Su > wrote: > > > Thanks Jed for the quick response. Yes I am asking about the > repartitioning of coarse grids in geometric multigrid for unstructured > mesh. I am happy with AMG. Thanks for letting me know. > > > > > > All the pieces are there, we just have not had users asking for this, > and it will take some work to put together. > > > > > > Matt - I created a branch for you and Lawrence last year which added > full support for PLEX within Telescope. This implementation was not a fully > automated algmoeration strategy - it utilized the partition associated with > the DM returned from DMGetCoarseDM. Hence the job of building the > distributed coarse hierarchy was let to the user. > > > > > > I?m pretty sure that code got merged into master as the branch also > contained several bug mixes for Telescope. Or am I mistaken? > > > > I think you're right. I didn't manage to get the redistribution of the > DMPlex object done last summer (it's bubbling up again). > > > > As I see it, for redistributed geometric multigrid on plexes, the > missing piece is a function: > > > > DMPlexRedistributeOntoComm(DM old, MPI_Comm comm, DM *new) > > > > I went down a rabbit hole of trying to do this, since I actually think > this replaced the current interface to DMPlexDistribute, which is > > > > DMPlexDistribute(DM old, PetscInt overlap, PetscSF *pointDistSF, DM *new) > > > > Where the new DM comes out on the same communicator as the old DM, just > with a different partition. > > > > This has lots of follow-on consequences, for example, one can't easily > load on P processes and then compute on Q. > > > > Unfortunately, collectiveness over MPI_Comm(old) is baked into the > redistribution routines everywhere, and I didn't manage to finish things. > > > > Yes, I remember thinking this out. I believe the conclusion was that > redistribution should happen on the large comm, which > > some fraction of processes getting no cells. Then at the end we call one > new function which copies that DM onto the smaller comm. > > I think this kind of works, but I worry that it makes the following > usecase really painful. > > 1. Parallel load on P processes > 2. Redistribute onto Q > P processes > ... Compute > > 3. (Occasionally), checkpoint by mapping onto P processes and dumping > > Perhaps I don't understand your proposal properly, but I think you're > saying that the _input_ DM in redistribute defines the processes that > participate in the redistribution (and the _output_ DM just controls the > size). > > This would then require me in my example to load on Q processes (not P) or > else do some futzing around. > > Or do I have the wrong end of the stick? > > I think I would prefer an interface where I go from > > commOld -> commNew > > with the number of partitions determined by size(commNew) > > Then the restriction you have on the two communicators is one of: > > i. commOld \subseteq commNew > > ii. commOld \supseteq commNew > > I guess if you wanted to be really fancy you could make an > intercommunicator of commOld \cap commNew, but now you have to carry > another communicator around with you for data transfer etc... > > Perhaps internally you do this by making a DM on whichever the smaller of > the two communicators is that is empty on a bunch of ranks and then copying > in (or copying out) appropriately. > > I think a first step here is to break the assumption everywhere in the > redistribution interface that the size of commOld determines the number of > partitions you want, and instead move to using size(commNew). Then for now > we just pass commNew=commOld. > I almost agree. I still think we do not change Distribute(), since it is really convenient, but we do check sizes on input as you say. We either, 1) Copy the DM to a larger comm with empty slots on input or 2) Copy the DM to a smaller comm eliminating empty slots on output depending on P_in < P_out, or the reverse. THanks, Matt > Lawrence > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From wence at gmx.li Tue Jun 2 04:15:52 2020 From: wence at gmx.li (Lawrence Mitchell) Date: Tue, 2 Jun 2020 10:15:52 +0100 Subject: [petsc-users] Agglomeration for Multigrid on Unstructured Meshes In-Reply-To: References: <4BDDF627-6CA6-4146-AD76-25BD566CCAB7@gmail.com> <87eeqycy3x.fsf@jedbrown.org> <9E53C33F-5BA1-414E-989D-EBDF03AAA218@gmail.com> <8ADFE4F6-7F2D-456C-8154-9DB18BB36FD0@gmx.li> Message-ID: <119DEEE4-EB3A-441D-8BEE-56691DF47520@gmx.li> > On 2 Jun 2020, at 09:54, Matthew Knepley wrote: > > I almost agree. I still think we do not change Distribute(), since it is really convenient, but we do check sizes on input as you say. If we only want Distribute(), we have to change it a bit, because right now there's only one communicator involved. So one could go to DMPlexDistribute(DM old, PetscInt overlap, MPI_Comm commNew, PetscSF *sf, DM *new) and commNew may be MPI_COMM_NULL, meaning we pick it up from old->comm. > We either, > > 1) Copy the DM to a larger comm with empty slots on input > > or > > 2) Copy the DM to a smaller comm eliminating empty slots on output > > depending on P_in < P_out, or the reverse. So we now also need to decide on the semantics of the migrationSF. I guess it lives on commNew, but is collective over commNew \cap commOld. I can't think through if this breaks anything. Lawrence From knepley at gmail.com Tue Jun 2 05:13:43 2020 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 2 Jun 2020 06:13:43 -0400 Subject: [petsc-users] Agglomeration for Multigrid on Unstructured Meshes In-Reply-To: <119DEEE4-EB3A-441D-8BEE-56691DF47520@gmx.li> References: <4BDDF627-6CA6-4146-AD76-25BD566CCAB7@gmail.com> <87eeqycy3x.fsf@jedbrown.org> <9E53C33F-5BA1-414E-989D-EBDF03AAA218@gmail.com> <8ADFE4F6-7F2D-456C-8154-9DB18BB36FD0@gmx.li> <119DEEE4-EB3A-441D-8BEE-56691DF47520@gmx.li> Message-ID: On Tue, Jun 2, 2020 at 5:15 AM Lawrence Mitchell wrote: > > On 2 Jun 2020, at 09:54, Matthew Knepley wrote: > > > > I almost agree. I still think we do not change Distribute(), since it is > really convenient, but we do check sizes on input as you say. > > If we only want Distribute(), we have to change it a bit, because right > now there's only one communicator involved. > > So one could go to > > DMPlexDistribute(DM old, PetscInt overlap, MPI_Comm commNew, PetscSF *sf, > DM *new) > > and commNew may be MPI_COMM_NULL, meaning we pick it up from old->comm. > This seems reasonable. > > We either, > > > > 1) Copy the DM to a larger comm with empty slots on input > > > > or > > > > 2) Copy the DM to a smaller comm eliminating empty slots on output > > > > depending on P_in < P_out, or the reverse. > > So we now also need to decide on the semantics of the migrationSF. I guess > it lives on commNew, but is collective over commNew \cap commOld. I can't > think through if this breaks anything. > I don't think 0-size participants should break anything. Matt > Lawrence > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From gautam.bisht at pnnl.gov Tue Jun 2 08:32:37 2020 From: gautam.bisht at pnnl.gov (Bisht, Gautam) Date: Tue, 2 Jun 2020 13:32:37 +0000 Subject: [petsc-users] DMPlex Exodus II transient variable In-Reply-To: References: , Message-ID: The transient variables don't show up under "vector" data either. -Gautam ________________________________ From: Matthew Knepley Sent: Tuesday, June 2, 2020 1:50 AM To: Bisht, Gautam Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] DMPlex Exodus II transient variable On Mon, Jun 1, 2020 at 11:56 PM Bisht, Gautam via petsc-users > wrote: Hi Matt, I came across ex26.c in DMPlex that writes out transient variables in an exodus II format. While VisIt can display the mesh, I can't figure out why the transient variables aren't showing up for display (see the screenshot). Do you know what is going wrong here? Are they in the "vector" data? Matt -Gautam. -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Screen Shot 2020-06-02 at 6.22.57 AM.png Type: image/png Size: 402766 bytes Desc: Screen Shot 2020-06-02 at 6.22.57 AM.png URL: From bourdin at lsu.edu Tue Jun 2 13:54:42 2020 From: bourdin at lsu.edu (Blaise A Bourdin) Date: Tue, 2 Jun 2020 18:54:42 +0000 Subject: [petsc-users] DMPlex Exodus II transient variable In-Reply-To: References: Message-ID: <40D61F5B-B677-47FB-9AAE-95C5058A6BE7@lsu.edu> Hi, There is a bug in visit 3.x x < 1.2 where fields do not show up unless QA records are written in the exodus file. The bug comes from the decision of the visit team to implement an exodus reader using netcdf instead of the official API, and a bug in their implementation. Regards, Blaise On Jun 2, 2020, at 8:32 AM, Bisht, Gautam via petsc-users > wrote: The transient variables don't show up under "vector" data either. -Gautam ________________________________ From: Matthew Knepley > Sent: Tuesday, June 2, 2020 1:50 AM To: Bisht, Gautam > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] DMPlex Exodus II transient variable On Mon, Jun 1, 2020 at 11:56 PM Bisht, Gautam via petsc-users > wrote: Hi Matt, I came across ex26.c in DMPlex that writes out transient variables in an exodus II format. While VisIt can display the mesh, I can't figure out why the transient variables aren't showing up for display (see the screenshot). Do you know what is going wrong here? Are they in the "vector" data? Matt -Gautam. -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -- A.K. & Shirley Barton Professor of Mathematics Adjunct Professor of Mechanical Engineering Adjunct of the Center for Computation & Technology Louisiana State University, Lockett Hall Room 344, Baton Rouge, LA 70803, USA Tel. +1 (225) 578 1612, Fax +1 (225) 578 4276 Web http://www.math.lsu.edu/~bourdin -------------- next part -------------- An HTML attachment was scrubbed... URL: From salazardetro1 at llnl.gov Tue Jun 2 14:36:10 2020 From: salazardetro1 at llnl.gov (Salazar De Troya, Miguel) Date: Tue, 2 Jun 2020 19:36:10 +0000 Subject: [petsc-users] TSAdjoint not working correctly when using TSThetaSetEndpoint(true) Message-ID: Hello, I am attaching a simple example that uses TSAdjoint to calculate the sensitivity of a 1 degree of freedom ODE. When using theta methods, it returns the right sensitivity as given by the analytical solution. When I set `ts.setThetaEndpoint(True)`, it does not work. Is there a theoretical reason why? I was using this option because using the options ts.setType(ts.Type.THETA) ts.setTheta(0.5) ts.setThetaEndpoint(True) is equivalent to using ts.Type.CN Thanks Miguel Miguel A. Salazar de Troya Postdoctoral Researcher, Lawrence Livermore National Laboratory B141 Rm: 1085-5 Ph: 1(925) 422-6411 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: petsc_question_2.py Type: text/x-python-script Size: 5451 bytes Desc: petsc_question_2.py URL: From evanum at gmail.com Tue Jun 2 15:42:59 2020 From: evanum at gmail.com (Evan Um) Date: Tue, 2 Jun 2020 13:42:59 -0700 Subject: [petsc-users] Using "long int" variable in MatSetValues Message-ID: Dear PETSC users, Using C++, I tried to build a very large sparse matrix and had a problem shown below. Indices of the matrix should be "long int" rather than "int" but PETSC does not allow this. How could I dodge this problem? Thanks for your comments. Regards, Evan fe.cpp(2009): error: argument of type "long *" is incompatible with parameter of type "const PetscInt={int} *" ierr=MatSetValues(Bseq, 1, &mat_b_i_partitioned[i], 1, &mat_b_j_partitioned[i], &mat_b_val_partitioned[i], ADD_VALUES); -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Tue Jun 2 15:47:27 2020 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 2 Jun 2020 15:47:27 -0500 (CDT) Subject: [petsc-users] Using "long int" variable in MatSetValues In-Reply-To: References: Message-ID: You need PetscInt to be 64-bit-integer - so rebuild PETSc with the option: --with-64-bit-indices=1 Satish On Tue, 2 Jun 2020, Evan Um wrote: > Dear PETSC users, > > Using C++, I tried to build a very large sparse matrix and had a problem > shown below. Indices of the matrix should be "long int" rather than "int" > but PETSC does not allow this. How could I dodge this problem? Thanks for > your comments. > > Regards, > Evan > > fe.cpp(2009): error: argument of type "long *" is incompatible with > parameter of type "const PetscInt={int} *" > ierr=MatSetValues(Bseq, 1, &mat_b_i_partitioned[i], 1, > &mat_b_j_partitioned[i], &mat_b_val_partitioned[i], ADD_VALUES); > From gautam.bisht at pnnl.gov Tue Jun 2 16:08:14 2020 From: gautam.bisht at pnnl.gov (Bisht, Gautam) Date: Tue, 2 Jun 2020 21:08:14 +0000 Subject: [petsc-users] DMPlex Exodus II transient variable In-Reply-To: <40D61F5B-B677-47FB-9AAE-95C5058A6BE7@lsu.edu> References: <40D61F5B-B677-47FB-9AAE-95C5058A6BE7@lsu.edu> Message-ID: <34B3536E-0349-4CE0-AD43-A6D2DBB5C1C5@pnnl.gov> Thanks, Blaise. After updating VisIt to 3.1.2, the transient variables are available. -Gautam. On Jun 2, 2020, at 11:54 AM, Blaise A Bourdin > wrote: Hi, There is a bug in visit 3.x x < 1.2 where fields do not show up unless QA records are written in the exodus file. The bug comes from the decision of the visit team to implement an exodus reader using netcdf instead of the official API, and a bug in their implementation. Regards, Blaise On Jun 2, 2020, at 8:32 AM, Bisht, Gautam via petsc-users > wrote: The transient variables don't show up under "vector" data either. -Gautam ________________________________ From: Matthew Knepley > Sent: Tuesday, June 2, 2020 1:50 AM To: Bisht, Gautam > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] DMPlex Exodus II transient variable On Mon, Jun 1, 2020 at 11:56 PM Bisht, Gautam via petsc-users > wrote: Hi Matt, I came across ex26.c in DMPlex that writes out transient variables in an exodus II format. While VisIt can display the mesh, I can't figure out why the transient variables aren't showing up for display (see the screenshot). Do you know what is going wrong here? Are they in the "vector" data? Matt -Gautam. -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -- A.K. & Shirley Barton Professor of Mathematics Adjunct Professor of Mechanical Engineering Adjunct of the Center for Computation & Technology Louisiana State University, Lockett Hall Room 344, Baton Rouge, LA 70803, USA Tel. +1 (225) 578 1612, Fax +1 (225) 578 4276 Web http://www.math.lsu.edu/~bourdin -------------- next part -------------- An HTML attachment was scrubbed... URL: From hongzhang at anl.gov Tue Jun 2 17:31:42 2020 From: hongzhang at anl.gov (Zhang, Hong) Date: Tue, 2 Jun 2020 22:31:42 +0000 Subject: [petsc-users] TSAdjoint not working correctly when using TSThetaSetEndpoint(true) In-Reply-To: References: Message-ID: <8343092D-2A3E-490F-942E-41400F0B1DA4@anl.gov> Miguel, After I uncommented ts.setThetaEndpoint(True) and removed ts.setProblemType(False.ProblemType.LINEAR) in your code, I got the following result: hongzhang at Hongs-MacBook-Pro$ python3 petsc_question_2.py Cost function Vec Object: 1 MPI processes type: seq 127.781 Exact value: 127.78112 Numerical sensitivity w.r.t. a 12.778122049945212 Real sensitivity w.r.t a 12.7781121978613 Numerical sensitivity w.r.t. b 83.8907580310942 Real sensitivity w.r.t b 211.67168296791954 -ts_type cn gives the same result. What was the problem you encountered? Thanks, Hong (Mr.) On Jun 2, 2020, at 2:36 PM, Salazar De Troya, Miguel via petsc-users > wrote: Hello, I am attaching a simple example that uses TSAdjoint to calculate the sensitivity of a 1 degree of freedom ODE. When using theta methods, it returns the right sensitivity as given by the analytical solution. When I set `ts.setThetaEndpoint(True)`, it does not work. Is there a theoretical reason why? I was using this option because using the options ts.setType(ts.Type.THETA) ts.setTheta(0.5) ts.setThetaEndpoint(True) is equivalent to using ts.Type.CN Thanks Miguel Miguel A. Salazar de Troya Postdoctoral Researcher, Lawrence Livermore National Laboratory B141 Rm: 1085-5 Ph: 1(925) 422-6411 -------------- next part -------------- An HTML attachment was scrubbed... URL: From salazardetro1 at llnl.gov Tue Jun 2 18:26:44 2020 From: salazardetro1 at llnl.gov (Salazar De Troya, Miguel) Date: Tue, 2 Jun 2020 23:26:44 +0000 Subject: [petsc-users] TSAdjoint not working correctly when using TSThetaSetEndpoint(true) In-Reply-To: <8343092D-2A3E-490F-942E-41400F0B1DA4@anl.gov> References: <8343092D-2A3E-490F-942E-41400F0B1DA4@anl.gov> Message-ID: Hong, That is not the correct result, however, we can obtain the correct result if we comment ts.setThetaEndpoint(True). Why is it so? Thanks Miguel From: "Zhang, Hong" Date: Tuesday, June 2, 2020 at 3:31 PM To: "Salazar De Troya, Miguel" Cc: "petsc-users at mcs.anl.gov" Subject: Re: [petsc-users] TSAdjoint not working correctly when using TSThetaSetEndpoint(true) Miguel, After I uncommented ts.setThetaEndpoint(True) and removed ts.setProblemType(False.ProblemType.LINEAR) in your code, I got the following result: hongzhang at Hongs-MacBook-Pro$ python3 petsc_question_2.py Cost function Vec Object: 1 MPI processes type: seq 127.781 Exact value: 127.78112 Numerical sensitivity w.r.t. a 12.778122049945212 Real sensitivity w.r.t a 12.7781121978613 Numerical sensitivity w.r.t. b 83.8907580310942 Real sensitivity w.r.t b 211.67168296791954 -ts_type cn gives the same result. What was the problem you encountered? Thanks, Hong (Mr.) On Jun 2, 2020, at 2:36 PM, Salazar De Troya, Miguel via petsc-users > wrote: Hello, I am attaching a simple example that uses TSAdjoint to calculate the sensitivity of a 1 degree of freedom ODE. When using theta methods, it returns the right sensitivity as given by the analytical solution. When I set `ts.setThetaEndpoint(True)`, it does not work. Is there a theoretical reason why? I was using this option because using the options ts.setType(ts.Type.THETA) ts.setTheta(0.5) ts.setThetaEndpoint(True) is equivalent to using ts.Type.CN Thanks Miguel Miguel A. Salazar de Troya Postdoctoral Researcher, Lawrence Livermore National Laboratory B141 Rm: 1085-5 Ph: 1(925) 422-6411 -------------- next part -------------- An HTML attachment was scrubbed... URL: From hongzhang at anl.gov Tue Jun 2 22:51:43 2020 From: hongzhang at anl.gov (Zhang, Hong) Date: Wed, 3 Jun 2020 03:51:43 +0000 Subject: [petsc-users] TSAdjoint not working correctly when using TSThetaSetEndpoint(true) In-Reply-To: References: <8343092D-2A3E-490F-942E-41400F0B1DA4@anl.gov> Message-ID: <371514B8-A0A7-4A81-90E0-042D91C8B7FB@anl.gov> I checked the results again and can confirm that it is a bug on our side. A merge request has been created to fix it: https://gitlab.com/petsc/petsc/-/merge_requests/2830 Thanks, Hong (Mr.) On Jun 2, 2020, at 6:26 PM, Salazar De Troya, Miguel > wrote: Hong, That is not the correct result, however, we can obtain the correct result if we comment ts.setThetaEndpoint(True). Why is it so? Thanks Miguel From: "Zhang, Hong" > Date: Tuesday, June 2, 2020 at 3:31 PM To: "Salazar De Troya, Miguel" > Cc: "petsc-users at mcs.anl.gov" > Subject: Re: [petsc-users] TSAdjoint not working correctly when using TSThetaSetEndpoint(true) Miguel, After I uncommented ts.setThetaEndpoint(True) and removed ts.setProblemType(False.ProblemType.LINEAR) in your code, I got the following result: hongzhang at Hongs-MacBook-Pro$ python3 petsc_question_2.py Cost function Vec Object: 1 MPI processes type: seq 127.781 Exact value: 127.78112 Numerical sensitivity w.r.t. a 12.778122049945212 Real sensitivity w.r.t a 12.7781121978613 Numerical sensitivity w.r.t. b 83.8907580310942 Real sensitivity w.r.t b 211.67168296791954 -ts_type cn gives the same result. What was the problem you encountered? Thanks, Hong (Mr.) On Jun 2, 2020, at 2:36 PM, Salazar De Troya, Miguel via petsc-users > wrote: Hello, I am attaching a simple example that uses TSAdjoint to calculate the sensitivity of a 1 degree of freedom ODE. When using theta methods, it returns the right sensitivity as given by the analytical solution. When I set `ts.setThetaEndpoint(True)`, it does not work. Is there a theoretical reason why? I was using this option because using the options ts.setType(ts.Type.THETA) ts.setTheta(0.5) ts.setThetaEndpoint(True) is equivalent to using ts.Type.CN Thanks Miguel Miguel A. Salazar de Troya Postdoctoral Researcher, Lawrence Livermore National Laboratory B141 Rm: 1085-5 Ph: 1(925) 422-6411 -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Tue Jun 2 23:22:55 2020 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 2 Jun 2020 23:22:55 -0500 (CDT) Subject: [petsc-users] petsc-3.13.2.tar.gz now available Message-ID: Dear PETSc users, The patch release petsc-3.13.2 is now available for download, with change list at 'PETSc-3.13 Changelog' http://www.mcs.anl.gov/petsc/download/index.html Satish From evanum at gmail.com Wed Jun 3 01:09:18 2020 From: evanum at gmail.com (Evan Um) Date: Tue, 2 Jun 2020 23:09:18 -0700 Subject: [petsc-users] Using "long int" variable in MatSetValues In-Reply-To: References: Message-ID: Hi Satish, Thanks for your reply. When the code is compiled using the petsc library built with the option, an error message suggests that I need to use 'long long' rather than 'int'. I can replace variable types with long long in my codes, but I am wondering if there is any better way. If I replace 'int' with 'long long', it means that I have to maintain two versions of PETSC application codes for small and big problems. Best, Evan On Tue, Jun 2, 2020 at 1:47 PM Satish Balay wrote: > You need PetscInt to be 64-bit-integer - so rebuild PETSc with the option: > --with-64-bit-indices=1 > > Satish > > On Tue, 2 Jun 2020, Evan Um wrote: > > > Dear PETSC users, > > > > Using C++, I tried to build a very large sparse matrix and had a problem > > shown below. Indices of the matrix should be "long int" rather than "int" > > but PETSC does not allow this. How could I dodge this problem? Thanks for > > your comments. > > > > Regards, > > Evan > > > > fe.cpp(2009): error: argument of type "long *" is incompatible with > > parameter of type "const PetscInt={int} *" > > ierr=MatSetValues(Bseq, 1, &mat_b_i_partitioned[i], 1, > > &mat_b_j_partitioned[i], &mat_b_val_partitioned[i], ADD_VALUES); > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Wed Jun 3 02:22:34 2020 From: balay at mcs.anl.gov (Satish Balay) Date: Wed, 3 Jun 2020 02:22:34 -0500 (CDT) Subject: [petsc-users] Using "long int" variable in MatSetValues In-Reply-To: References: Message-ID: Its best to use PetscInt here. Satish On Tue, 2 Jun 2020, Evan Um wrote: > Hi Satish, > > Thanks for your reply. > When the code is compiled using the petsc library built with the option, an > error message suggests that I need to use 'long long' rather than 'int'. I > can replace variable types with long long in my codes, but I am wondering > if there is any better way. If I replace 'int' with 'long long', it means > that I have to maintain two versions of PETSC application codes for small > and big problems. > > Best, > Evan > > On Tue, Jun 2, 2020 at 1:47 PM Satish Balay wrote: > > > You need PetscInt to be 64-bit-integer - so rebuild PETSc with the option: > > --with-64-bit-indices=1 > > > > Satish > > > > On Tue, 2 Jun 2020, Evan Um wrote: > > > > > Dear PETSC users, > > > > > > Using C++, I tried to build a very large sparse matrix and had a problem > > > shown below. Indices of the matrix should be "long int" rather than "int" > > > but PETSC does not allow this. How could I dodge this problem? Thanks for > > > your comments. > > > > > > Regards, > > > Evan > > > > > > fe.cpp(2009): error: argument of type "long *" is incompatible with > > > parameter of type "const PetscInt={int} *" > > > ierr=MatSetValues(Bseq, 1, &mat_b_i_partitioned[i], 1, > > > &mat_b_j_partitioned[i], &mat_b_val_partitioned[i], ADD_VALUES); > > > > > > > > From salazardetroya at gmail.com Wed Jun 3 09:26:03 2020 From: salazardetroya at gmail.com (Miguel Angel Salazar de Troya) Date: Wed, 3 Jun 2020 07:26:03 -0700 Subject: [petsc-users] TSAdjoint not working correctly when using TSThetaSetEndpoint(true) In-Reply-To: <371514B8-A0A7-4A81-90E0-042D91C8B7FB@anl.gov> References: <8343092D-2A3E-490F-942E-41400F0B1DA4@anl.gov> <371514B8-A0A7-4A81-90E0-042D91C8B7FB@anl.gov> Message-ID: Thanks! On Tue, Jun 2, 2020, 20:51 Zhang, Hong via petsc-users < petsc-users at mcs.anl.gov> wrote: > I checked the results again and can confirm that it is a bug on our side. > A merge request has been created to fix it: > https://gitlab.com/petsc/petsc/-/merge_requests/2830 > > Thanks, > Hong (Mr.) > > On Jun 2, 2020, at 6:26 PM, Salazar De Troya, Miguel < > salazardetro1 at llnl.gov> wrote: > > Hong, > > That is not the correct result, however, we can obtain the correct result > if we comment ts.setThetaEndpoint(True). Why is it so? > > Thanks > Miguel > > *From: *"Zhang, Hong" > *Date: *Tuesday, June 2, 2020 at 3:31 PM > *To: *"Salazar De Troya, Miguel" > *Cc: *"petsc-users at mcs.anl.gov" > *Subject: *Re: [petsc-users] TSAdjoint not working correctly when using > TSThetaSetEndpoint(true) > > Miguel, > > After I uncommented ts.setThetaEndpoint(True) and > removed ts.setProblemType(False.ProblemType.LINEAR) in your code, I got the > following result: > > hongzhang at Hongs-MacBook-Pro$ python3 petsc_question_2.py > Cost function > Vec Object: 1 MPI processes > type: seq > 127.781 > Exact value: 127.78112 > > Numerical sensitivity w.r.t. a > 12.778122049945212 > Real sensitivity w.r.t a > 12.7781121978613 > > Numerical sensitivity w.r.t. b > 83.8907580310942 > Real sensitivity w.r.t b > 211.67168296791954 > > -ts_type cn gives the same result. What was the problem you encountered? > > Thanks, > Hong (Mr.) > > > On Jun 2, 2020, at 2:36 PM, Salazar De Troya, Miguel via petsc-users < > petsc-users at mcs.anl.gov> wrote: > > Hello, > > I am attaching a simple example that uses TSAdjoint to calculate the > sensitivity of a 1 degree of freedom ODE. When using theta methods, it > returns the right sensitivity as given by the analytical solution. When I > set `ts.setThetaEndpoint(True)`, it does not work. Is there a theoretical > reason why? I was using this option because using the options > > ts.setType(ts.Type.THETA) > ts.setTheta(0.5) > ts.setThetaEndpoint(True) > > is equivalent to using ts.Type.CN > > Thanks > Miguel > > Miguel A. Salazar de Troya > Postdoctoral Researcher, Lawrence Livermore National Laboratory > B141 > Rm: 1085-5 > Ph: 1(925) 422-6411 > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From danyang.su at gmail.com Wed Jun 3 16:04:01 2020 From: danyang.su at gmail.com (Danyang Su) Date: Wed, 03 Jun 2020 14:04:01 -0700 Subject: [petsc-users] Error configuring EXODUSII with cmake in petsc-3.13.2 Message-ID: <03097C85-C3FF-47F7-9DD8-21755570C5E9@gmail.com> Hi All, I got following errors compiling PETSc-3.13.2 version. =============================================================================== Trying to download git://https://github.com/gsjaardema/seacas.git for EXODUSII =============================================================================== =============================================================================== Configuring EXODUSII with cmake, this may take several minutes =============================================================================== ******************************************************************************* UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): ------------------------------------------------------------------------------- Error configuring EXODUSII with cmake ******************************************************************************* Thanks, Danyang ?On 2020-06-02, 9:23 PM, "petsc-users on behalf of Satish Balay via petsc-users" wrote: Dear PETSc users, The patch release petsc-3.13.2 is now available for download, with change list at 'PETSc-3.13 Changelog' http://www.mcs.anl.gov/petsc/download/index.html Satish -------------- next part -------------- A non-text attachment was scrubbed... Name: configure.log Type: application/octet-stream Size: 3842855 bytes Desc: not available URL: From jacob.fai at gmail.com Wed Jun 3 16:08:36 2020 From: jacob.fai at gmail.com (Jacob Faibussowitsch) Date: Wed, 3 Jun 2020 16:08:36 -0500 Subject: [petsc-users] Error configuring EXODUSII with cmake in petsc-3.13.2 In-Reply-To: <03097C85-C3FF-47F7-9DD8-21755570C5E9@gmail.com> References: <03097C85-C3FF-47F7-9DD8-21755570C5E9@gmail.com> Message-ID: From the configure.log: > -- Configuring incomplete, errors occurred!CMake Error at CMakeLists.txt:16 (CMAKE_MINIMUM_REQUIRED): > CMake 3.10.0 or higher is required. You are running version 3.5.1 Best regards, Jacob Faibussowitsch (Jacob Fai - booss - oh - vitch) Cell: (312) 694-3391 > On Jun 3, 2020, at 4:04 PM, Danyang Su wrote: > > Hi All, > > I got following errors compiling PETSc-3.13.2 version. > =============================================================================== > Trying to download git://https://github.com/gsjaardema/seacas.git for EXODUSII > =============================================================================== > =============================================================================== > Configuring EXODUSII with cmake, this may take several minutes > =============================================================================== ******************************************************************************* > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): > ------------------------------------------------------------------------------- > Error configuring EXODUSII with cmake > ******************************************************************************* > > Thanks, > > Danyang > > ?On 2020-06-02, 9:23 PM, "petsc-users on behalf of Satish Balay via petsc-users" wrote: > > Dear PETSc users, > > The patch release petsc-3.13.2 is now available for download, > with change list at 'PETSc-3.13 Changelog' > > http://www.mcs.anl.gov/petsc/download/index.html > > Satish > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Y.Juntao at hotmail.com Thu Jun 4 04:01:33 2020 From: Y.Juntao at hotmail.com (Yang Juntao) Date: Thu, 4 Jun 2020 09:01:33 +0000 Subject: [petsc-users] Slow speed to SetValues for matrix created by PETSC by mat_initial_matrix Message-ID: Hello, I computed a preconditioned matrix by MatMatMulti(), and I generated the new matrix with option MAT_INITIAL_MATRIX. However I need to change the generated matrix by insert value 1 to half of the diagonal. MatSetValue() is used but the speed a much slower than MatSetValue with matrix setup by MatCreate() and MatSeqAIJPreallocation(). I would like to ask is it due to the preallocation setting of MAT_INITIAL_MATRIX generated matrix and how I can change it? Thanks in advance JT From huq2090 at gmail.com Thu Jun 4 07:51:46 2020 From: huq2090 at gmail.com (Fazlul Huq) Date: Thu, 4 Jun 2020 07:51:46 -0500 Subject: [petsc-users] Running example problem Message-ID: Hello, I have a very preliminary question! I think I have installed PETSc correctly cause I got following on the terminal: Command: make PETSC_DIR=/home/huq2090/petsc-3.10.2 PETSC_ARCH=linux-gnu check Response: Running check examples to verify correct installation Using PETSC_DIR=/home/huq2090/petsc-3.10.2 and PETSC_ARCH=linux-gnu C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI process C/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI processes Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI process Completed test examples Now, I am trying to run an example from the list of provided examples. Can you please help me out, how to run an example problem from the list of examples? I mean, how to make executable and run the executable? Thank you. Sincerely, Huq -- Fazlul Huq Graduate Research Assistant Department of Nuclear, Plasma & Radiological Engineering (NPRE) University of Illinois at Urbana-Champaign (UIUC) E-mail: huq2090 at gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Jun 4 07:56:48 2020 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 4 Jun 2020 08:56:48 -0400 Subject: [petsc-users] Running example problem In-Reply-To: References: Message-ID: On Thu, Jun 4, 2020 at 8:53 AM Fazlul Huq wrote: > Hello, > > I have a very preliminary question! > I think I have installed PETSc correctly cause I got following on the > terminal: > > Command: > make PETSC_DIR=/home/huq2090/petsc-3.10.2 PETSC_ARCH=linux-gnu check > Response: > Running check examples to verify correct installation > Using PETSC_DIR=/home/huq2090/petsc-3.10.2 and PETSC_ARCH=linux-gnu > C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI process > C/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI processes > Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI process > Completed test examples > > Now, I am trying to run an example from the list of provided examples. > Can you please help me out, how to run an example problem from the list of > examples? > I mean, how to make executable and run the executable? > cd $PETSC_DIR cd src/snes/tutorials make ex5 ./ex5 -snes_monitor Thanks, Matt > Thank you. > Sincerely, > Huq > -- > > Fazlul Huq > Graduate Research Assistant > Department of Nuclear, Plasma & Radiological Engineering (NPRE) > University of Illinois at Urbana-Champaign (UIUC) > E-mail: huq2090 at gmail.com > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From huq2090 at gmail.com Thu Jun 4 08:12:07 2020 From: huq2090 at gmail.com (Fazlul Huq) Date: Thu, 4 Jun 2020 08:12:07 -0500 Subject: [petsc-users] Running example problem In-Reply-To: References: Message-ID: Somehow, make is not working. Please find the attachment herewith for the terminal readout. Thank you. Sincerely, Huq On Thu, Jun 4, 2020 at 7:57 AM Matthew Knepley wrote: > On Thu, Jun 4, 2020 at 8:53 AM Fazlul Huq wrote: > >> Hello, >> >> I have a very preliminary question! >> I think I have installed PETSc correctly cause I got following on the >> terminal: >> >> Command: >> make PETSC_DIR=/home/huq2090/petsc-3.10.2 PETSC_ARCH=linux-gnu check >> Response: >> Running check examples to verify correct installation >> Using PETSC_DIR=/home/huq2090/petsc-3.10.2 and PETSC_ARCH=linux-gnu >> C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI process >> C/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI >> processes >> Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI >> process >> Completed test examples >> >> Now, I am trying to run an example from the list of provided examples. >> Can you please help me out, how to run an example problem from the list >> of examples? >> I mean, how to make executable and run the executable? >> > > cd $PETSC_DIR > cd src/snes/tutorials > make ex5 > ./ex5 -snes_monitor > > Thanks, > > Matt > > >> Thank you. >> Sincerely, >> Huq >> -- >> >> Fazlul Huq >> Graduate Research Assistant >> Department of Nuclear, Plasma & Radiological Engineering (NPRE) >> University of Illinois at Urbana-Champaign (UIUC) >> E-mail: huq2090 at gmail.com >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -- Fazlul Huq Graduate Research Assistant Department of Nuclear, Plasma & Radiological Engineering (NPRE) University of Illinois at Urbana-Champaign (UIUC) E-mail: huq2090 at gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Screenshot from 2020-06-04 08-10-33.png Type: image/png Size: 38670 bytes Desc: not available URL: From knepley at gmail.com Thu Jun 4 08:13:41 2020 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 4 Jun 2020 09:13:41 -0400 Subject: [petsc-users] Running example problem In-Reply-To: References: Message-ID: On Thu, Jun 4, 2020 at 9:12 AM Fazlul Huq wrote: > Somehow, make is not working. > Please find the attachment herewith for the terminal readout. > Since you built with PETSC_ARCH=linux-gnu, you need that in your environment. Thanks, Matt > Thank you. > > Sincerely, > Huq > > On Thu, Jun 4, 2020 at 7:57 AM Matthew Knepley wrote: > >> On Thu, Jun 4, 2020 at 8:53 AM Fazlul Huq wrote: >> >>> Hello, >>> >>> I have a very preliminary question! >>> I think I have installed PETSc correctly cause I got following on the >>> terminal: >>> >>> Command: >>> make PETSC_DIR=/home/huq2090/petsc-3.10.2 PETSC_ARCH=linux-gnu check >>> Response: >>> Running check examples to verify correct installation >>> Using PETSC_DIR=/home/huq2090/petsc-3.10.2 and PETSC_ARCH=linux-gnu >>> C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI process >>> C/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI >>> processes >>> Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI >>> process >>> Completed test examples >>> >>> Now, I am trying to run an example from the list of provided examples. >>> Can you please help me out, how to run an example problem from the list >>> of examples? >>> I mean, how to make executable and run the executable? >>> >> >> cd $PETSC_DIR >> cd src/snes/tutorials >> make ex5 >> ./ex5 -snes_monitor >> >> Thanks, >> >> Matt >> >> >>> Thank you. >>> Sincerely, >>> Huq >>> -- >>> >>> Fazlul Huq >>> Graduate Research Assistant >>> Department of Nuclear, Plasma & Radiological Engineering (NPRE) >>> University of Illinois at Urbana-Champaign (UIUC) >>> E-mail: huq2090 at gmail.com >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> >> > > > -- > > Fazlul Huq > Graduate Research Assistant > Department of Nuclear, Plasma & Radiological Engineering (NPRE) > University of Illinois at Urbana-Champaign (UIUC) > E-mail: huq2090 at gmail.com > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave.mayhem23 at gmail.com Thu Jun 4 08:17:45 2020 From: dave.mayhem23 at gmail.com (Dave May) Date: Thu, 4 Jun 2020 14:17:45 +0100 Subject: [petsc-users] Running example problem In-Reply-To: References: Message-ID: On Thu, 4 Jun 2020 at 14:15, Matthew Knepley wrote: > On Thu, Jun 4, 2020 at 9:12 AM Fazlul Huq wrote: > >> Somehow, make is not working. >> Please find the attachment herewith for the terminal readout. >> > > Since you built with PETSC_ARCH=linux-gnu, you need that in your > environment. > Or just do make ex5 PETSC_ARCH=linux-gnu > > Thanks, > > Matt > > >> Thank you. >> >> Sincerely, >> Huq >> >> On Thu, Jun 4, 2020 at 7:57 AM Matthew Knepley wrote: >> >>> On Thu, Jun 4, 2020 at 8:53 AM Fazlul Huq wrote: >>> >>>> Hello, >>>> >>>> I have a very preliminary question! >>>> I think I have installed PETSc correctly cause I got following on the >>>> terminal: >>>> >>>> Command: >>>> make PETSC_DIR=/home/huq2090/petsc-3.10.2 PETSC_ARCH=linux-gnu check >>>> Response: >>>> Running check examples to verify correct installation >>>> Using PETSC_DIR=/home/huq2090/petsc-3.10.2 and PETSC_ARCH=linux-gnu >>>> C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI >>>> process >>>> C/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI >>>> processes >>>> Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI >>>> process >>>> Completed test examples >>>> >>>> Now, I am trying to run an example from the list of provided examples. >>>> Can you please help me out, how to run an example problem from the list >>>> of examples? >>>> I mean, how to make executable and run the executable? >>>> >>> >>> cd $PETSC_DIR >>> cd src/snes/tutorials >>> make ex5 >>> ./ex5 -snes_monitor >>> >>> Thanks, >>> >>> Matt >>> >>> >>>> Thank you. >>>> Sincerely, >>>> Huq >>>> -- >>>> >>>> Fazlul Huq >>>> Graduate Research Assistant >>>> Department of Nuclear, Plasma & Radiological Engineering (NPRE) >>>> University of Illinois at Urbana-Champaign (UIUC) >>>> E-mail: huq2090 at gmail.com >>>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> https://www.cse.buffalo.edu/~knepley/ >>> >>> >> >> >> -- >> >> Fazlul Huq >> Graduate Research Assistant >> Department of Nuclear, Plasma & Radiological Engineering (NPRE) >> University of Illinois at Urbana-Champaign (UIUC) >> E-mail: huq2090 at gmail.com >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave.mayhem23 at gmail.com Thu Jun 4 08:19:17 2020 From: dave.mayhem23 at gmail.com (Dave May) Date: Thu, 4 Jun 2020 14:19:17 +0100 Subject: [petsc-users] Running example problem In-Reply-To: References: Message-ID: On Thu, 4 Jun 2020 at 14:17, Dave May wrote: > > > On Thu, 4 Jun 2020 at 14:15, Matthew Knepley wrote: > >> On Thu, Jun 4, 2020 at 9:12 AM Fazlul Huq wrote: >> >>> Somehow, make is not working. >>> Please find the attachment herewith for the terminal readout. >>> >> >> Since you built with PETSC_ARCH=linux-gnu, you need that in your >> environment. >> > > Or just do > > make ex5 PETSC_ARCH=linux-gnu > sorry I hit send without checking your png) The command should be make ex5 PETSC_ARCH=arch-linux2-c-debug > > > > >> >> Thanks, >> >> Matt >> >> >>> Thank you. >>> >>> Sincerely, >>> Huq >>> >>> On Thu, Jun 4, 2020 at 7:57 AM Matthew Knepley >>> wrote: >>> >>>> On Thu, Jun 4, 2020 at 8:53 AM Fazlul Huq wrote: >>>> >>>>> Hello, >>>>> >>>>> I have a very preliminary question! >>>>> I think I have installed PETSc correctly cause I got following on the >>>>> terminal: >>>>> >>>>> Command: >>>>> make PETSC_DIR=/home/huq2090/petsc-3.10.2 PETSC_ARCH=linux-gnu check >>>>> Response: >>>>> Running check examples to verify correct installation >>>>> Using PETSC_DIR=/home/huq2090/petsc-3.10.2 and PETSC_ARCH=linux-gnu >>>>> C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI >>>>> process >>>>> C/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI >>>>> processes >>>>> Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI >>>>> process >>>>> Completed test examples >>>>> >>>>> Now, I am trying to run an example from the list of provided examples. >>>>> Can you please help me out, how to run an example problem from the >>>>> list of examples? >>>>> I mean, how to make executable and run the executable? >>>>> >>>> >>>> cd $PETSC_DIR >>>> cd src/snes/tutorials >>>> make ex5 >>>> ./ex5 -snes_monitor >>>> >>>> Thanks, >>>> >>>> Matt >>>> >>>> >>>>> Thank you. >>>>> Sincerely, >>>>> Huq >>>>> -- >>>>> >>>>> Fazlul Huq >>>>> Graduate Research Assistant >>>>> Department of Nuclear, Plasma & Radiological Engineering (NPRE) >>>>> University of Illinois at Urbana-Champaign (UIUC) >>>>> E-mail: huq2090 at gmail.com >>>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>>> https://www.cse.buffalo.edu/~knepley/ >>>> >>>> >>> >>> >>> -- >>> >>> Fazlul Huq >>> Graduate Research Assistant >>> Department of Nuclear, Plasma & Radiological Engineering (NPRE) >>> University of Illinois at Urbana-Champaign (UIUC) >>> E-mail: huq2090 at gmail.com >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Jun 4 08:34:31 2020 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 4 Jun 2020 09:34:31 -0400 Subject: [petsc-users] Running example problem In-Reply-To: References: Message-ID: On Thu, Jun 4, 2020 at 9:19 AM Dave May wrote: > > > On Thu, 4 Jun 2020 at 14:17, Dave May wrote: > >> >> >> On Thu, 4 Jun 2020 at 14:15, Matthew Knepley wrote: >> >>> On Thu, Jun 4, 2020 at 9:12 AM Fazlul Huq wrote: >>> >>>> Somehow, make is not working. >>>> Please find the attachment herewith for the terminal readout. >>>> >>> >>> Since you built with PETSC_ARCH=linux-gnu, you need that in your >>> environment. >>> >> >> Or just do >> >> make ex5 PETSC_ARCH=linux-gnu >> > > sorry I hit send without checking your png) > The command should be > > make ex5 PETSC_ARCH=arch-linux2-c-debug > No, your first one was right. Matt > >> >> >>> >>> Thanks, >>> >>> Matt >>> >>> >>>> Thank you. >>>> >>>> Sincerely, >>>> Huq >>>> >>>> On Thu, Jun 4, 2020 at 7:57 AM Matthew Knepley >>>> wrote: >>>> >>>>> On Thu, Jun 4, 2020 at 8:53 AM Fazlul Huq wrote: >>>>> >>>>>> Hello, >>>>>> >>>>>> I have a very preliminary question! >>>>>> I think I have installed PETSc correctly cause I got following on the >>>>>> terminal: >>>>>> >>>>>> Command: >>>>>> make PETSC_DIR=/home/huq2090/petsc-3.10.2 PETSC_ARCH=linux-gnu check >>>>>> Response: >>>>>> Running check examples to verify correct installation >>>>>> Using PETSC_DIR=/home/huq2090/petsc-3.10.2 and PETSC_ARCH=linux-gnu >>>>>> C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI >>>>>> process >>>>>> C/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI >>>>>> processes >>>>>> Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI >>>>>> process >>>>>> Completed test examples >>>>>> >>>>>> Now, I am trying to run an example from the list of provided >>>>>> examples. >>>>>> Can you please help me out, how to run an example problem from the >>>>>> list of examples? >>>>>> I mean, how to make executable and run the executable? >>>>>> >>>>> >>>>> cd $PETSC_DIR >>>>> cd src/snes/tutorials >>>>> make ex5 >>>>> ./ex5 -snes_monitor >>>>> >>>>> Thanks, >>>>> >>>>> Matt >>>>> >>>>> >>>>>> Thank you. >>>>>> Sincerely, >>>>>> Huq >>>>>> -- >>>>>> >>>>>> Fazlul Huq >>>>>> Graduate Research Assistant >>>>>> Department of Nuclear, Plasma & Radiological Engineering (NPRE) >>>>>> University of Illinois at Urbana-Champaign (UIUC) >>>>>> E-mail: huq2090 at gmail.com >>>>>> >>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their >>>>> experiments is infinitely more interesting than any results to which their >>>>> experiments lead. >>>>> -- Norbert Wiener >>>>> >>>>> https://www.cse.buffalo.edu/~knepley/ >>>>> >>>>> >>>> >>>> >>>> -- >>>> >>>> Fazlul Huq >>>> Graduate Research Assistant >>>> Department of Nuclear, Plasma & Radiological Engineering (NPRE) >>>> University of Illinois at Urbana-Champaign (UIUC) >>>> E-mail: huq2090 at gmail.com >>>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> https://www.cse.buffalo.edu/~knepley/ >>> >>> >> -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From huq2090 at gmail.com Thu Jun 4 09:28:35 2020 From: huq2090 at gmail.com (Fazlul Huq) Date: Thu, 4 Jun 2020 09:28:35 -0500 Subject: [petsc-users] Running example problem In-Reply-To: References: Message-ID: Thank you. Working now. I think to run the following command I need to configure PETSc with HYPRE. mpiexec -np 1 ./poisson_m -n 100000 -pc_type hypre -pc_hypre_type boomerang If I am right, can you please guide me how to configure with HYPRE and run the above command? Thank you. Sincerely, Huq On Thu, Jun 4, 2020 at 8:34 AM Matthew Knepley wrote: > > On Thu, Jun 4, 2020 at 9:19 AM Dave May wrote: > >> >> >> On Thu, 4 Jun 2020 at 14:17, Dave May wrote: >> >>> >>> >>> On Thu, 4 Jun 2020 at 14:15, Matthew Knepley wrote: >>> >>>> On Thu, Jun 4, 2020 at 9:12 AM Fazlul Huq wrote: >>>> >>>>> Somehow, make is not working. >>>>> Please find the attachment herewith for the terminal readout. >>>>> >>>> >>>> Since you built with PETSC_ARCH=linux-gnu, you need that in your >>>> environment. >>>> >>> >>> Or just do >>> >>> make ex5 PETSC_ARCH=linux-gnu >>> >> >> sorry I hit send without checking your png) >> The command should be >> >> make ex5 PETSC_ARCH=arch-linux2-c-debug >> > > No, your first one was right. > > Matt > > >> >>> >>> >>>> >>>> Thanks, >>>> >>>> Matt >>>> >>>> >>>>> Thank you. >>>>> >>>>> Sincerely, >>>>> Huq >>>>> >>>>> On Thu, Jun 4, 2020 at 7:57 AM Matthew Knepley >>>>> wrote: >>>>> >>>>>> On Thu, Jun 4, 2020 at 8:53 AM Fazlul Huq wrote: >>>>>> >>>>>>> Hello, >>>>>>> >>>>>>> I have a very preliminary question! >>>>>>> I think I have installed PETSc correctly cause I got following on >>>>>>> the terminal: >>>>>>> >>>>>>> Command: >>>>>>> make PETSC_DIR=/home/huq2090/petsc-3.10.2 PETSC_ARCH=linux-gnu check >>>>>>> Response: >>>>>>> Running check examples to verify correct installation >>>>>>> Using PETSC_DIR=/home/huq2090/petsc-3.10.2 and PETSC_ARCH=linux-gnu >>>>>>> C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI >>>>>>> process >>>>>>> C/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI >>>>>>> processes >>>>>>> Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI >>>>>>> process >>>>>>> Completed test examples >>>>>>> >>>>>>> Now, I am trying to run an example from the list of provided >>>>>>> examples. >>>>>>> Can you please help me out, how to run an example problem from the >>>>>>> list of examples? >>>>>>> I mean, how to make executable and run the executable? >>>>>>> >>>>>> >>>>>> cd $PETSC_DIR >>>>>> cd src/snes/tutorials >>>>>> make ex5 >>>>>> ./ex5 -snes_monitor >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Matt >>>>>> >>>>>> >>>>>>> Thank you. >>>>>>> Sincerely, >>>>>>> Huq >>>>>>> -- >>>>>>> >>>>>>> Fazlul Huq >>>>>>> Graduate Research Assistant >>>>>>> Department of Nuclear, Plasma & Radiological Engineering (NPRE) >>>>>>> University of Illinois at Urbana-Champaign (UIUC) >>>>>>> E-mail: huq2090 at gmail.com >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> What most experimenters take for granted before they begin their >>>>>> experiments is infinitely more interesting than any results to which their >>>>>> experiments lead. >>>>>> -- Norbert Wiener >>>>>> >>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> >>>>> Fazlul Huq >>>>> Graduate Research Assistant >>>>> Department of Nuclear, Plasma & Radiological Engineering (NPRE) >>>>> University of Illinois at Urbana-Champaign (UIUC) >>>>> E-mail: huq2090 at gmail.com >>>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>>> https://www.cse.buffalo.edu/~knepley/ >>>> >>>> >>> > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -- Fazlul Huq Graduate Research Assistant Department of Nuclear, Plasma & Radiological Engineering (NPRE) University of Illinois at Urbana-Champaign (UIUC) E-mail: huq2090 at gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From hzhang at mcs.anl.gov Thu Jun 4 09:45:06 2020 From: hzhang at mcs.anl.gov (Zhang, Hong) Date: Thu, 4 Jun 2020 14:45:06 +0000 Subject: [petsc-users] Slow speed to SetValues for matrix created by PETSC by mat_initial_matrix In-Reply-To: References: Message-ID: Check the output of MatMatMult(). If your matrix product has all diagonal entries? How large is your matrix? Hong ________________________________ From: petsc-users on behalf of Yang Juntao Sent: Thursday, June 4, 2020 4:01 AM To: petsc-users at mcs.anl.gov Subject: [petsc-users] Slow speed to SetValues for matrix created by PETSC by mat_initial_matrix Hello, I computed a preconditioned matrix by MatMatMulti(), and I generated the new matrix with option MAT_INITIAL_MATRIX. However I need to change the generated matrix by insert value 1 to half of the diagonal. MatSetValue() is used but the speed a much slower than MatSetValue with matrix setup by MatCreate() and MatSeqAIJPreallocation(). I would like to ask is it due to the preallocation setting of MAT_INITIAL_MATRIX generated matrix and how I can change it? Thanks in advance JT -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Jun 4 10:13:02 2020 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 4 Jun 2020 11:13:02 -0400 Subject: [petsc-users] Running example problem In-Reply-To: References: Message-ID: On Thu, Jun 4, 2020 at 10:28 AM Fazlul Huq wrote: > Thank you. > Working now. > > I think to run the following command I need to configure PETSc with HYPRE. > mpiexec -np 1 ./poisson_m -n 100000 -pc_type hypre -pc_hypre_type boomerang > > If I am right, can you please guide me how to configure with HYPRE and run > the above command? > --download-hypre Thanks, Matt > Thank you. > Sincerely, > Huq > > On Thu, Jun 4, 2020 at 8:34 AM Matthew Knepley wrote: > >> >> On Thu, Jun 4, 2020 at 9:19 AM Dave May wrote: >> >>> >>> >>> On Thu, 4 Jun 2020 at 14:17, Dave May wrote: >>> >>>> >>>> >>>> On Thu, 4 Jun 2020 at 14:15, Matthew Knepley wrote: >>>> >>>>> On Thu, Jun 4, 2020 at 9:12 AM Fazlul Huq wrote: >>>>> >>>>>> Somehow, make is not working. >>>>>> Please find the attachment herewith for the terminal readout. >>>>>> >>>>> >>>>> Since you built with PETSC_ARCH=linux-gnu, you need that in your >>>>> environment. >>>>> >>>> >>>> Or just do >>>> >>>> make ex5 PETSC_ARCH=linux-gnu >>>> >>> >>> sorry I hit send without checking your png) >>> The command should be >>> >>> make ex5 PETSC_ARCH=arch-linux2-c-debug >>> >> >> No, your first one was right. >> >> Matt >> >> >>> >>>> >>>> >>>>> >>>>> Thanks, >>>>> >>>>> Matt >>>>> >>>>> >>>>>> Thank you. >>>>>> >>>>>> Sincerely, >>>>>> Huq >>>>>> >>>>>> On Thu, Jun 4, 2020 at 7:57 AM Matthew Knepley >>>>>> wrote: >>>>>> >>>>>>> On Thu, Jun 4, 2020 at 8:53 AM Fazlul Huq wrote: >>>>>>> >>>>>>>> Hello, >>>>>>>> >>>>>>>> I have a very preliminary question! >>>>>>>> I think I have installed PETSc correctly cause I got following on >>>>>>>> the terminal: >>>>>>>> >>>>>>>> Command: >>>>>>>> make PETSC_DIR=/home/huq2090/petsc-3.10.2 PETSC_ARCH=linux-gnu check >>>>>>>> Response: >>>>>>>> Running check examples to verify correct installation >>>>>>>> Using PETSC_DIR=/home/huq2090/petsc-3.10.2 and PETSC_ARCH=linux-gnu >>>>>>>> C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI >>>>>>>> process >>>>>>>> C/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI >>>>>>>> processes >>>>>>>> Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI >>>>>>>> process >>>>>>>> Completed test examples >>>>>>>> >>>>>>>> Now, I am trying to run an example from the list of provided >>>>>>>> examples. >>>>>>>> Can you please help me out, how to run an example problem from the >>>>>>>> list of examples? >>>>>>>> I mean, how to make executable and run the executable? >>>>>>>> >>>>>>> >>>>>>> cd $PETSC_DIR >>>>>>> cd src/snes/tutorials >>>>>>> make ex5 >>>>>>> ./ex5 -snes_monitor >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Matt >>>>>>> >>>>>>> >>>>>>>> Thank you. >>>>>>>> Sincerely, >>>>>>>> Huq >>>>>>>> -- >>>>>>>> >>>>>>>> Fazlul Huq >>>>>>>> Graduate Research Assistant >>>>>>>> Department of Nuclear, Plasma & Radiological Engineering (NPRE) >>>>>>>> University of Illinois at Urbana-Champaign (UIUC) >>>>>>>> E-mail: huq2090 at gmail.com >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> What most experimenters take for granted before they begin their >>>>>>> experiments is infinitely more interesting than any results to which their >>>>>>> experiments lead. >>>>>>> -- Norbert Wiener >>>>>>> >>>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> >>>>>> Fazlul Huq >>>>>> Graduate Research Assistant >>>>>> Department of Nuclear, Plasma & Radiological Engineering (NPRE) >>>>>> University of Illinois at Urbana-Champaign (UIUC) >>>>>> E-mail: huq2090 at gmail.com >>>>>> >>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their >>>>> experiments is infinitely more interesting than any results to which their >>>>> experiments lead. >>>>> -- Norbert Wiener >>>>> >>>>> https://www.cse.buffalo.edu/~knepley/ >>>>> >>>>> >>>> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> >> > > > -- > > Fazlul Huq > Graduate Research Assistant > Department of Nuclear, Plasma & Radiological Engineering (NPRE) > University of Illinois at Urbana-Champaign (UIUC) > E-mail: huq2090 at gmail.com > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From huq2090 at gmail.com Thu Jun 4 10:55:56 2020 From: huq2090 at gmail.com (Fazlul Huq) Date: Thu, 4 Jun 2020 10:55:56 -0500 Subject: [petsc-users] Running example problem In-Reply-To: References: Message-ID: I have HYPRE installed in my machine. Ithink, I need to compile PETSc with HYPRE. I'm not clear about how to do that? Thank you. Sincerely, Huq On Thu, Jun 4, 2020 at 10:13 AM Matthew Knepley wrote: > On Thu, Jun 4, 2020 at 10:28 AM Fazlul Huq wrote: > >> Thank you. >> Working now. >> >> I think to run the following command I need to configure PETSc with HYPRE. >> mpiexec -np 1 ./poisson_m -n 100000 -pc_type hypre -pc_hypre_type >> boomerang >> >> If I am right, can you please guide me how to configure with HYPRE and >> run the above command? >> > > --download-hypre > > Thanks, > > Matt > > >> Thank you. >> Sincerely, >> Huq >> >> On Thu, Jun 4, 2020 at 8:34 AM Matthew Knepley wrote: >> >>> >>> On Thu, Jun 4, 2020 at 9:19 AM Dave May wrote: >>> >>>> >>>> >>>> On Thu, 4 Jun 2020 at 14:17, Dave May wrote: >>>> >>>>> >>>>> >>>>> On Thu, 4 Jun 2020 at 14:15, Matthew Knepley >>>>> wrote: >>>>> >>>>>> On Thu, Jun 4, 2020 at 9:12 AM Fazlul Huq wrote: >>>>>> >>>>>>> Somehow, make is not working. >>>>>>> Please find the attachment herewith for the terminal readout. >>>>>>> >>>>>> >>>>>> Since you built with PETSC_ARCH=linux-gnu, you need that in your >>>>>> environment. >>>>>> >>>>> >>>>> Or just do >>>>> >>>>> make ex5 PETSC_ARCH=linux-gnu >>>>> >>>> >>>> sorry I hit send without checking your png) >>>> The command should be >>>> >>>> make ex5 PETSC_ARCH=arch-linux2-c-debug >>>> >>> >>> No, your first one was right. >>> >>> Matt >>> >>> >>>> >>>>> >>>>> >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Matt >>>>>> >>>>>> >>>>>>> Thank you. >>>>>>> >>>>>>> Sincerely, >>>>>>> Huq >>>>>>> >>>>>>> On Thu, Jun 4, 2020 at 7:57 AM Matthew Knepley >>>>>>> wrote: >>>>>>> >>>>>>>> On Thu, Jun 4, 2020 at 8:53 AM Fazlul Huq >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hello, >>>>>>>>> >>>>>>>>> I have a very preliminary question! >>>>>>>>> I think I have installed PETSc correctly cause I got following on >>>>>>>>> the terminal: >>>>>>>>> >>>>>>>>> Command: >>>>>>>>> make PETSC_DIR=/home/huq2090/petsc-3.10.2 PETSC_ARCH=linux-gnu >>>>>>>>> check >>>>>>>>> Response: >>>>>>>>> Running check examples to verify correct installation >>>>>>>>> Using PETSC_DIR=/home/huq2090/petsc-3.10.2 and PETSC_ARCH=linux-gnu >>>>>>>>> C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI >>>>>>>>> process >>>>>>>>> C/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI >>>>>>>>> processes >>>>>>>>> Fortran example src/snes/tutorials/ex5f run successfully with 1 >>>>>>>>> MPI process >>>>>>>>> Completed test examples >>>>>>>>> >>>>>>>>> Now, I am trying to run an example from the list of provided >>>>>>>>> examples. >>>>>>>>> Can you please help me out, how to run an example problem from the >>>>>>>>> list of examples? >>>>>>>>> I mean, how to make executable and run the executable? >>>>>>>>> >>>>>>>> >>>>>>>> cd $PETSC_DIR >>>>>>>> cd src/snes/tutorials >>>>>>>> make ex5 >>>>>>>> ./ex5 -snes_monitor >>>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>>> Matt >>>>>>>> >>>>>>>> >>>>>>>>> Thank you. >>>>>>>>> Sincerely, >>>>>>>>> Huq >>>>>>>>> -- >>>>>>>>> >>>>>>>>> Fazlul Huq >>>>>>>>> Graduate Research Assistant >>>>>>>>> Department of Nuclear, Plasma & Radiological Engineering (NPRE) >>>>>>>>> University of Illinois at Urbana-Champaign (UIUC) >>>>>>>>> E-mail: huq2090 at gmail.com >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> What most experimenters take for granted before they begin their >>>>>>>> experiments is infinitely more interesting than any results to which their >>>>>>>> experiments lead. >>>>>>>> -- Norbert Wiener >>>>>>>> >>>>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> >>>>>>> Fazlul Huq >>>>>>> Graduate Research Assistant >>>>>>> Department of Nuclear, Plasma & Radiological Engineering (NPRE) >>>>>>> University of Illinois at Urbana-Champaign (UIUC) >>>>>>> E-mail: huq2090 at gmail.com >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> What most experimenters take for granted before they begin their >>>>>> experiments is infinitely more interesting than any results to which their >>>>>> experiments lead. >>>>>> -- Norbert Wiener >>>>>> >>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>> >>>>>> >>>>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> https://www.cse.buffalo.edu/~knepley/ >>> >>> >> >> >> -- >> >> Fazlul Huq >> Graduate Research Assistant >> Department of Nuclear, Plasma & Radiological Engineering (NPRE) >> University of Illinois at Urbana-Champaign (UIUC) >> E-mail: huq2090 at gmail.com >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -- Fazlul Huq Graduate Research Assistant Department of Nuclear, Plasma & Radiological Engineering (NPRE) University of Illinois at Urbana-Champaign (UIUC) E-mail: huq2090 at gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Jun 4 11:02:12 2020 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 4 Jun 2020 12:02:12 -0400 Subject: [petsc-users] Running example problem In-Reply-To: References: Message-ID: On Thu, Jun 4, 2020 at 11:56 AM Fazlul Huq wrote: > I have HYPRE installed in my machine. > Ithink, I need to compile PETSc with HYPRE. > I'm not clear about how to do that? > 1) Make sure it is the correct version 2) --with-hypre-dir= Thanks, Matt > Thank you. > Sincerely, > Huq > > On Thu, Jun 4, 2020 at 10:13 AM Matthew Knepley wrote: > >> On Thu, Jun 4, 2020 at 10:28 AM Fazlul Huq wrote: >> >>> Thank you. >>> Working now. >>> >>> I think to run the following command I need to configure PETSc with >>> HYPRE. >>> mpiexec -np 1 ./poisson_m -n 100000 -pc_type hypre -pc_hypre_type >>> boomerang >>> >>> If I am right, can you please guide me how to configure with HYPRE and >>> run the above command? >>> >> >> --download-hypre >> >> Thanks, >> >> Matt >> >> >>> Thank you. >>> Sincerely, >>> Huq >>> >>> On Thu, Jun 4, 2020 at 8:34 AM Matthew Knepley >>> wrote: >>> >>>> >>>> On Thu, Jun 4, 2020 at 9:19 AM Dave May >>>> wrote: >>>> >>>>> >>>>> >>>>> On Thu, 4 Jun 2020 at 14:17, Dave May wrote: >>>>> >>>>>> >>>>>> >>>>>> On Thu, 4 Jun 2020 at 14:15, Matthew Knepley >>>>>> wrote: >>>>>> >>>>>>> On Thu, Jun 4, 2020 at 9:12 AM Fazlul Huq wrote: >>>>>>> >>>>>>>> Somehow, make is not working. >>>>>>>> Please find the attachment herewith for the terminal readout. >>>>>>>> >>>>>>> >>>>>>> Since you built with PETSC_ARCH=linux-gnu, you need that in your >>>>>>> environment. >>>>>>> >>>>>> >>>>>> Or just do >>>>>> >>>>>> make ex5 PETSC_ARCH=linux-gnu >>>>>> >>>>> >>>>> sorry I hit send without checking your png) >>>>> The command should be >>>>> >>>>> make ex5 PETSC_ARCH=arch-linux2-c-debug >>>>> >>>> >>>> No, your first one was right. >>>> >>>> Matt >>>> >>>> >>>>> >>>>>> >>>>>> >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Matt >>>>>>> >>>>>>> >>>>>>>> Thank you. >>>>>>>> >>>>>>>> Sincerely, >>>>>>>> Huq >>>>>>>> >>>>>>>> On Thu, Jun 4, 2020 at 7:57 AM Matthew Knepley >>>>>>>> wrote: >>>>>>>> >>>>>>>>> On Thu, Jun 4, 2020 at 8:53 AM Fazlul Huq >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hello, >>>>>>>>>> >>>>>>>>>> I have a very preliminary question! >>>>>>>>>> I think I have installed PETSc correctly cause I got following on >>>>>>>>>> the terminal: >>>>>>>>>> >>>>>>>>>> Command: >>>>>>>>>> make PETSC_DIR=/home/huq2090/petsc-3.10.2 PETSC_ARCH=linux-gnu >>>>>>>>>> check >>>>>>>>>> Response: >>>>>>>>>> Running check examples to verify correct installation >>>>>>>>>> Using PETSC_DIR=/home/huq2090/petsc-3.10.2 and >>>>>>>>>> PETSC_ARCH=linux-gnu >>>>>>>>>> C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI >>>>>>>>>> process >>>>>>>>>> C/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI >>>>>>>>>> processes >>>>>>>>>> Fortran example src/snes/tutorials/ex5f run successfully with 1 >>>>>>>>>> MPI process >>>>>>>>>> Completed test examples >>>>>>>>>> >>>>>>>>>> Now, I am trying to run an example from the list of provided >>>>>>>>>> examples. >>>>>>>>>> Can you please help me out, how to run an example problem from >>>>>>>>>> the list of examples? >>>>>>>>>> I mean, how to make executable and run the executable? >>>>>>>>>> >>>>>>>>> >>>>>>>>> cd $PETSC_DIR >>>>>>>>> cd src/snes/tutorials >>>>>>>>> make ex5 >>>>>>>>> ./ex5 -snes_monitor >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> >>>>>>>>> Matt >>>>>>>>> >>>>>>>>> >>>>>>>>>> Thank you. >>>>>>>>>> Sincerely, >>>>>>>>>> Huq >>>>>>>>>> -- >>>>>>>>>> >>>>>>>>>> Fazlul Huq >>>>>>>>>> Graduate Research Assistant >>>>>>>>>> Department of Nuclear, Plasma & Radiological Engineering (NPRE) >>>>>>>>>> University of Illinois at Urbana-Champaign (UIUC) >>>>>>>>>> E-mail: huq2090 at gmail.com >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> What most experimenters take for granted before they begin their >>>>>>>>> experiments is infinitely more interesting than any results to which their >>>>>>>>> experiments lead. >>>>>>>>> -- Norbert Wiener >>>>>>>>> >>>>>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> >>>>>>>> Fazlul Huq >>>>>>>> Graduate Research Assistant >>>>>>>> Department of Nuclear, Plasma & Radiological Engineering (NPRE) >>>>>>>> University of Illinois at Urbana-Champaign (UIUC) >>>>>>>> E-mail: huq2090 at gmail.com >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> What most experimenters take for granted before they begin their >>>>>>> experiments is infinitely more interesting than any results to which their >>>>>>> experiments lead. >>>>>>> -- Norbert Wiener >>>>>>> >>>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>>> >>>>>>> >>>>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>>> https://www.cse.buffalo.edu/~knepley/ >>>> >>>> >>> >>> >>> -- >>> >>> Fazlul Huq >>> Graduate Research Assistant >>> Department of Nuclear, Plasma & Radiological Engineering (NPRE) >>> University of Illinois at Urbana-Champaign (UIUC) >>> E-mail: huq2090 at gmail.com >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> >> > > > -- > > Fazlul Huq > Graduate Research Assistant > Department of Nuclear, Plasma & Radiological Engineering (NPRE) > University of Illinois at Urbana-Champaign (UIUC) > E-mail: huq2090 at gmail.com > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Thu Jun 4 11:04:36 2020 From: balay at mcs.anl.gov (Satish Balay) Date: Thu, 4 Jun 2020 11:04:36 -0500 (CDT) Subject: [petsc-users] Running example problem In-Reply-To: References: Message-ID: Its best to use --download-hypre with PETSc configure - it installs a compatible version of hypre. If you have a pre-installed hypre, you'l have to: - make sure its compatible version - its built with the same MPI, compilers etc as PETSc - you can use --with-hypre-dir option with PETSc configure Satish On Thu, 4 Jun 2020, Fazlul Huq wrote: > I have HYPRE installed in my machine. > Ithink, I need to compile PETSc with HYPRE. > I'm not clear about how to do that? > > Thank you. > Sincerely, > Huq > > On Thu, Jun 4, 2020 at 10:13 AM Matthew Knepley wrote: > > > On Thu, Jun 4, 2020 at 10:28 AM Fazlul Huq wrote: > > > >> Thank you. > >> Working now. > >> > >> I think to run the following command I need to configure PETSc with HYPRE. > >> mpiexec -np 1 ./poisson_m -n 100000 -pc_type hypre -pc_hypre_type > >> boomerang > >> > >> If I am right, can you please guide me how to configure with HYPRE and > >> run the above command? > >> > > > > --download-hypre > > > > Thanks, > > > > Matt > > > > > >> Thank you. > >> Sincerely, > >> Huq > >> > >> On Thu, Jun 4, 2020 at 8:34 AM Matthew Knepley wrote: > >> > >>> > >>> On Thu, Jun 4, 2020 at 9:19 AM Dave May wrote: > >>> > >>>> > >>>> > >>>> On Thu, 4 Jun 2020 at 14:17, Dave May wrote: > >>>> > >>>>> > >>>>> > >>>>> On Thu, 4 Jun 2020 at 14:15, Matthew Knepley > >>>>> wrote: > >>>>> > >>>>>> On Thu, Jun 4, 2020 at 9:12 AM Fazlul Huq wrote: > >>>>>> > >>>>>>> Somehow, make is not working. > >>>>>>> Please find the attachment herewith for the terminal readout. > >>>>>>> > >>>>>> > >>>>>> Since you built with PETSC_ARCH=linux-gnu, you need that in your > >>>>>> environment. > >>>>>> > >>>>> > >>>>> Or just do > >>>>> > >>>>> make ex5 PETSC_ARCH=linux-gnu > >>>>> > >>>> > >>>> sorry I hit send without checking your png) > >>>> The command should be > >>>> > >>>> make ex5 PETSC_ARCH=arch-linux2-c-debug > >>>> > >>> > >>> No, your first one was right. > >>> > >>> Matt > >>> > >>> > >>>> > >>>>> > >>>>> > >>>>>> > >>>>>> Thanks, > >>>>>> > >>>>>> Matt > >>>>>> > >>>>>> > >>>>>>> Thank you. > >>>>>>> > >>>>>>> Sincerely, > >>>>>>> Huq > >>>>>>> > >>>>>>> On Thu, Jun 4, 2020 at 7:57 AM Matthew Knepley > >>>>>>> wrote: > >>>>>>> > >>>>>>>> On Thu, Jun 4, 2020 at 8:53 AM Fazlul Huq > >>>>>>>> wrote: > >>>>>>>> > >>>>>>>>> Hello, > >>>>>>>>> > >>>>>>>>> I have a very preliminary question! > >>>>>>>>> I think I have installed PETSc correctly cause I got following on > >>>>>>>>> the terminal: > >>>>>>>>> > >>>>>>>>> Command: > >>>>>>>>> make PETSC_DIR=/home/huq2090/petsc-3.10.2 PETSC_ARCH=linux-gnu > >>>>>>>>> check > >>>>>>>>> Response: > >>>>>>>>> Running check examples to verify correct installation > >>>>>>>>> Using PETSC_DIR=/home/huq2090/petsc-3.10.2 and PETSC_ARCH=linux-gnu > >>>>>>>>> C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI > >>>>>>>>> process > >>>>>>>>> C/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI > >>>>>>>>> processes > >>>>>>>>> Fortran example src/snes/tutorials/ex5f run successfully with 1 > >>>>>>>>> MPI process > >>>>>>>>> Completed test examples > >>>>>>>>> > >>>>>>>>> Now, I am trying to run an example from the list of provided > >>>>>>>>> examples. > >>>>>>>>> Can you please help me out, how to run an example problem from the > >>>>>>>>> list of examples? > >>>>>>>>> I mean, how to make executable and run the executable? > >>>>>>>>> > >>>>>>>> > >>>>>>>> cd $PETSC_DIR > >>>>>>>> cd src/snes/tutorials > >>>>>>>> make ex5 > >>>>>>>> ./ex5 -snes_monitor > >>>>>>>> > >>>>>>>> Thanks, > >>>>>>>> > >>>>>>>> Matt > >>>>>>>> > >>>>>>>> > >>>>>>>>> Thank you. > >>>>>>>>> Sincerely, > >>>>>>>>> Huq > >>>>>>>>> -- > >>>>>>>>> > >>>>>>>>> Fazlul Huq > >>>>>>>>> Graduate Research Assistant > >>>>>>>>> Department of Nuclear, Plasma & Radiological Engineering (NPRE) > >>>>>>>>> University of Illinois at Urbana-Champaign (UIUC) > >>>>>>>>> E-mail: huq2090 at gmail.com > >>>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> -- > >>>>>>>> What most experimenters take for granted before they begin their > >>>>>>>> experiments is infinitely more interesting than any results to which their > >>>>>>>> experiments lead. > >>>>>>>> -- Norbert Wiener > >>>>>>>> > >>>>>>>> https://www.cse.buffalo.edu/~knepley/ > >>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> -- > >>>>>>> > >>>>>>> Fazlul Huq > >>>>>>> Graduate Research Assistant > >>>>>>> Department of Nuclear, Plasma & Radiological Engineering (NPRE) > >>>>>>> University of Illinois at Urbana-Champaign (UIUC) > >>>>>>> E-mail: huq2090 at gmail.com > >>>>>>> > >>>>>> > >>>>>> > >>>>>> -- > >>>>>> What most experimenters take for granted before they begin their > >>>>>> experiments is infinitely more interesting than any results to which their > >>>>>> experiments lead. > >>>>>> -- Norbert Wiener > >>>>>> > >>>>>> https://www.cse.buffalo.edu/~knepley/ > >>>>>> > >>>>>> > >>>>> > >>> > >>> -- > >>> What most experimenters take for granted before they begin their > >>> experiments is infinitely more interesting than any results to which their > >>> experiments lead. > >>> -- Norbert Wiener > >>> > >>> https://www.cse.buffalo.edu/~knepley/ > >>> > >>> > >> > >> > >> -- > >> > >> Fazlul Huq > >> Graduate Research Assistant > >> Department of Nuclear, Plasma & Radiological Engineering (NPRE) > >> University of Illinois at Urbana-Champaign (UIUC) > >> E-mail: huq2090 at gmail.com > >> > > > > > > -- > > What most experimenters take for granted before they begin their > > experiments is infinitely more interesting than any results to which their > > experiments lead. > > -- Norbert Wiener > > > > https://www.cse.buffalo.edu/~knepley/ > > > > > > > From huq2090 at gmail.com Thu Jun 4 12:18:21 2020 From: huq2090 at gmail.com (Fazlul Huq) Date: Thu, 4 Jun 2020 12:18:21 -0500 Subject: [petsc-users] Running example problem In-Reply-To: References: Message-ID: I think it's okay to have a separate version of hypre for PETSc. So, what I understand is I have to do, ./configure --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-mpich --download-fblaslapack --download-hypre make all test Is that all? Am I missing something? Thank you. Sincerely, Huq On Thu, Jun 4, 2020 at 11:04 AM Satish Balay wrote: > Its best to use --download-hypre with PETSc configure - it installs a > compatible version of hypre. > > If you have a pre-installed hypre, you'l have to: > - make sure its compatible version > - its built with the same MPI, compilers etc as PETSc > - you can use --with-hypre-dir option with PETSc configure > > Satish > > On Thu, 4 Jun 2020, Fazlul Huq wrote: > > > I have HYPRE installed in my machine. > > Ithink, I need to compile PETSc with HYPRE. > > I'm not clear about how to do that? > > > > Thank you. > > Sincerely, > > Huq > > > > On Thu, Jun 4, 2020 at 10:13 AM Matthew Knepley > wrote: > > > > > On Thu, Jun 4, 2020 at 10:28 AM Fazlul Huq wrote: > > > > > >> Thank you. > > >> Working now. > > >> > > >> I think to run the following command I need to configure PETSc with > HYPRE. > > >> mpiexec -np 1 ./poisson_m -n 100000 -pc_type hypre -pc_hypre_type > > >> boomerang > > >> > > >> If I am right, can you please guide me how to configure with HYPRE and > > >> run the above command? > > >> > > > > > > --download-hypre > > > > > > Thanks, > > > > > > Matt > > > > > > > > >> Thank you. > > >> Sincerely, > > >> Huq > > >> > > >> On Thu, Jun 4, 2020 at 8:34 AM Matthew Knepley > wrote: > > >> > > >>> > > >>> On Thu, Jun 4, 2020 at 9:19 AM Dave May > wrote: > > >>> > > >>>> > > >>>> > > >>>> On Thu, 4 Jun 2020 at 14:17, Dave May > wrote: > > >>>> > > >>>>> > > >>>>> > > >>>>> On Thu, 4 Jun 2020 at 14:15, Matthew Knepley > > >>>>> wrote: > > >>>>> > > >>>>>> On Thu, Jun 4, 2020 at 9:12 AM Fazlul Huq > wrote: > > >>>>>> > > >>>>>>> Somehow, make is not working. > > >>>>>>> Please find the attachment herewith for the terminal readout. > > >>>>>>> > > >>>>>> > > >>>>>> Since you built with PETSC_ARCH=linux-gnu, you need that in your > > >>>>>> environment. > > >>>>>> > > >>>>> > > >>>>> Or just do > > >>>>> > > >>>>> make ex5 PETSC_ARCH=linux-gnu > > >>>>> > > >>>> > > >>>> sorry I hit send without checking your png) > > >>>> The command should be > > >>>> > > >>>> make ex5 PETSC_ARCH=arch-linux2-c-debug > > >>>> > > >>> > > >>> No, your first one was right. > > >>> > > >>> Matt > > >>> > > >>> > > >>>> > > >>>>> > > >>>>> > > >>>>>> > > >>>>>> Thanks, > > >>>>>> > > >>>>>> Matt > > >>>>>> > > >>>>>> > > >>>>>>> Thank you. > > >>>>>>> > > >>>>>>> Sincerely, > > >>>>>>> Huq > > >>>>>>> > > >>>>>>> On Thu, Jun 4, 2020 at 7:57 AM Matthew Knepley < > knepley at gmail.com> > > >>>>>>> wrote: > > >>>>>>> > > >>>>>>>> On Thu, Jun 4, 2020 at 8:53 AM Fazlul Huq > > >>>>>>>> wrote: > > >>>>>>>> > > >>>>>>>>> Hello, > > >>>>>>>>> > > >>>>>>>>> I have a very preliminary question! > > >>>>>>>>> I think I have installed PETSc correctly cause I got following > on > > >>>>>>>>> the terminal: > > >>>>>>>>> > > >>>>>>>>> Command: > > >>>>>>>>> make PETSC_DIR=/home/huq2090/petsc-3.10.2 PETSC_ARCH=linux-gnu > > >>>>>>>>> check > > >>>>>>>>> Response: > > >>>>>>>>> Running check examples to verify correct installation > > >>>>>>>>> Using PETSC_DIR=/home/huq2090/petsc-3.10.2 and > PETSC_ARCH=linux-gnu > > >>>>>>>>> C/C++ example src/snes/tutorials/ex19 run successfully with 1 > MPI > > >>>>>>>>> process > > >>>>>>>>> C/C++ example src/snes/tutorials/ex19 run successfully with 2 > MPI > > >>>>>>>>> processes > > >>>>>>>>> Fortran example src/snes/tutorials/ex5f run successfully with 1 > > >>>>>>>>> MPI process > > >>>>>>>>> Completed test examples > > >>>>>>>>> > > >>>>>>>>> Now, I am trying to run an example from the list of provided > > >>>>>>>>> examples. > > >>>>>>>>> Can you please help me out, how to run an example problem from > the > > >>>>>>>>> list of examples? > > >>>>>>>>> I mean, how to make executable and run the executable? > > >>>>>>>>> > > >>>>>>>> > > >>>>>>>> cd $PETSC_DIR > > >>>>>>>> cd src/snes/tutorials > > >>>>>>>> make ex5 > > >>>>>>>> ./ex5 -snes_monitor > > >>>>>>>> > > >>>>>>>> Thanks, > > >>>>>>>> > > >>>>>>>> Matt > > >>>>>>>> > > >>>>>>>> > > >>>>>>>>> Thank you. > > >>>>>>>>> Sincerely, > > >>>>>>>>> Huq > > >>>>>>>>> -- > > >>>>>>>>> > > >>>>>>>>> Fazlul Huq > > >>>>>>>>> Graduate Research Assistant > > >>>>>>>>> Department of Nuclear, Plasma & Radiological Engineering (NPRE) > > >>>>>>>>> University of Illinois at Urbana-Champaign (UIUC) > > >>>>>>>>> E-mail: huq2090 at gmail.com > > >>>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> -- > > >>>>>>>> What most experimenters take for granted before they begin their > > >>>>>>>> experiments is infinitely more interesting than any results to > which their > > >>>>>>>> experiments lead. > > >>>>>>>> -- Norbert Wiener > > >>>>>>>> > > >>>>>>>> https://www.cse.buffalo.edu/~knepley/ > > >>>>>>>> > > >>>>>>>> > > >>>>>>> > > >>>>>>> > > >>>>>>> -- > > >>>>>>> > > >>>>>>> Fazlul Huq > > >>>>>>> Graduate Research Assistant > > >>>>>>> Department of Nuclear, Plasma & Radiological Engineering (NPRE) > > >>>>>>> University of Illinois at Urbana-Champaign (UIUC) > > >>>>>>> E-mail: huq2090 at gmail.com > > >>>>>>> > > >>>>>> > > >>>>>> > > >>>>>> -- > > >>>>>> What most experimenters take for granted before they begin their > > >>>>>> experiments is infinitely more interesting than any results to > which their > > >>>>>> experiments lead. > > >>>>>> -- Norbert Wiener > > >>>>>> > > >>>>>> https://www.cse.buffalo.edu/~knepley/ > > >>>>>> > > >>>>>> > > >>>>> > > >>> > > >>> -- > > >>> What most experimenters take for granted before they begin their > > >>> experiments is infinitely more interesting than any results to which > their > > >>> experiments lead. > > >>> -- Norbert Wiener > > >>> > > >>> https://www.cse.buffalo.edu/~knepley/ > > >>> > > >>> > > >> > > >> > > >> -- > > >> > > >> Fazlul Huq > > >> Graduate Research Assistant > > >> Department of Nuclear, Plasma & Radiological Engineering (NPRE) > > >> University of Illinois at Urbana-Champaign (UIUC) > > >> E-mail: huq2090 at gmail.com > > >> > > > > > > > > > -- > > > What most experimenters take for granted before they begin their > > > experiments is infinitely more interesting than any results to which > their > > > experiments lead. > > > -- Norbert Wiener > > > > > > https://www.cse.buffalo.edu/~knepley/ > > > > > > > > > > > > > > -- Fazlul Huq Graduate Research Assistant Department of Nuclear, Plasma & Radiological Engineering (NPRE) University of Illinois at Urbana-Champaign (UIUC) E-mail: huq2090 at gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Thu Jun 4 12:27:06 2020 From: balay at mcs.anl.gov (Satish Balay) Date: Thu, 4 Jun 2020 12:27:06 -0500 (CDT) Subject: [petsc-users] Running example problem In-Reply-To: References: Message-ID: Should work If it fails - send us corresponding configure.log Satish On Thu, 4 Jun 2020, Fazlul Huq wrote: > I think it's okay to have a separate version of hypre for PETSc. > So, what I understand is I have to do, > > ./configure --with-cc=gcc --with-cxx=g++ --with-fc=gfortran > --download-mpich --download-fblaslapack --download-hypre > make all test > > Is that all? > Am I missing something? > > Thank you. > Sincerely, > Huq > > > On Thu, Jun 4, 2020 at 11:04 AM Satish Balay wrote: > > > Its best to use --download-hypre with PETSc configure - it installs a > > compatible version of hypre. > > > > If you have a pre-installed hypre, you'l have to: > > - make sure its compatible version > > - its built with the same MPI, compilers etc as PETSc > > - you can use --with-hypre-dir option with PETSc configure > > > > Satish > > > > On Thu, 4 Jun 2020, Fazlul Huq wrote: > > > > > I have HYPRE installed in my machine. > > > Ithink, I need to compile PETSc with HYPRE. > > > I'm not clear about how to do that? > > > > > > Thank you. > > > Sincerely, > > > Huq > > > > > > On Thu, Jun 4, 2020 at 10:13 AM Matthew Knepley > > wrote: > > > > > > > On Thu, Jun 4, 2020 at 10:28 AM Fazlul Huq wrote: > > > > > > > >> Thank you. > > > >> Working now. > > > >> > > > >> I think to run the following command I need to configure PETSc with > > HYPRE. > > > >> mpiexec -np 1 ./poisson_m -n 100000 -pc_type hypre -pc_hypre_type > > > >> boomerang > > > >> > > > >> If I am right, can you please guide me how to configure with HYPRE and > > > >> run the above command? > > > >> > > > > > > > > --download-hypre > > > > > > > > Thanks, > > > > > > > > Matt > > > > > > > > > > > >> Thank you. > > > >> Sincerely, > > > >> Huq > > > >> > > > >> On Thu, Jun 4, 2020 at 8:34 AM Matthew Knepley > > wrote: > > > >> > > > >>> > > > >>> On Thu, Jun 4, 2020 at 9:19 AM Dave May > > wrote: > > > >>> > > > >>>> > > > >>>> > > > >>>> On Thu, 4 Jun 2020 at 14:17, Dave May > > wrote: > > > >>>> > > > >>>>> > > > >>>>> > > > >>>>> On Thu, 4 Jun 2020 at 14:15, Matthew Knepley > > > >>>>> wrote: > > > >>>>> > > > >>>>>> On Thu, Jun 4, 2020 at 9:12 AM Fazlul Huq > > wrote: > > > >>>>>> > > > >>>>>>> Somehow, make is not working. > > > >>>>>>> Please find the attachment herewith for the terminal readout. > > > >>>>>>> > > > >>>>>> > > > >>>>>> Since you built with PETSC_ARCH=linux-gnu, you need that in your > > > >>>>>> environment. > > > >>>>>> > > > >>>>> > > > >>>>> Or just do > > > >>>>> > > > >>>>> make ex5 PETSC_ARCH=linux-gnu > > > >>>>> > > > >>>> > > > >>>> sorry I hit send without checking your png) > > > >>>> The command should be > > > >>>> > > > >>>> make ex5 PETSC_ARCH=arch-linux2-c-debug > > > >>>> > > > >>> > > > >>> No, your first one was right. > > > >>> > > > >>> Matt > > > >>> > > > >>> > > > >>>> > > > >>>>> > > > >>>>> > > > >>>>>> > > > >>>>>> Thanks, > > > >>>>>> > > > >>>>>> Matt > > > >>>>>> > > > >>>>>> > > > >>>>>>> Thank you. > > > >>>>>>> > > > >>>>>>> Sincerely, > > > >>>>>>> Huq > > > >>>>>>> > > > >>>>>>> On Thu, Jun 4, 2020 at 7:57 AM Matthew Knepley < > > knepley at gmail.com> > > > >>>>>>> wrote: > > > >>>>>>> > > > >>>>>>>> On Thu, Jun 4, 2020 at 8:53 AM Fazlul Huq > > > >>>>>>>> wrote: > > > >>>>>>>> > > > >>>>>>>>> Hello, > > > >>>>>>>>> > > > >>>>>>>>> I have a very preliminary question! > > > >>>>>>>>> I think I have installed PETSc correctly cause I got following > > on > > > >>>>>>>>> the terminal: > > > >>>>>>>>> > > > >>>>>>>>> Command: > > > >>>>>>>>> make PETSC_DIR=/home/huq2090/petsc-3.10.2 PETSC_ARCH=linux-gnu > > > >>>>>>>>> check > > > >>>>>>>>> Response: > > > >>>>>>>>> Running check examples to verify correct installation > > > >>>>>>>>> Using PETSC_DIR=/home/huq2090/petsc-3.10.2 and > > PETSC_ARCH=linux-gnu > > > >>>>>>>>> C/C++ example src/snes/tutorials/ex19 run successfully with 1 > > MPI > > > >>>>>>>>> process > > > >>>>>>>>> C/C++ example src/snes/tutorials/ex19 run successfully with 2 > > MPI > > > >>>>>>>>> processes > > > >>>>>>>>> Fortran example src/snes/tutorials/ex5f run successfully with 1 > > > >>>>>>>>> MPI process > > > >>>>>>>>> Completed test examples > > > >>>>>>>>> > > > >>>>>>>>> Now, I am trying to run an example from the list of provided > > > >>>>>>>>> examples. > > > >>>>>>>>> Can you please help me out, how to run an example problem from > > the > > > >>>>>>>>> list of examples? > > > >>>>>>>>> I mean, how to make executable and run the executable? > > > >>>>>>>>> > > > >>>>>>>> > > > >>>>>>>> cd $PETSC_DIR > > > >>>>>>>> cd src/snes/tutorials > > > >>>>>>>> make ex5 > > > >>>>>>>> ./ex5 -snes_monitor > > > >>>>>>>> > > > >>>>>>>> Thanks, > > > >>>>>>>> > > > >>>>>>>> Matt > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>>> Thank you. > > > >>>>>>>>> Sincerely, > > > >>>>>>>>> Huq > > > >>>>>>>>> -- > > > >>>>>>>>> > > > >>>>>>>>> Fazlul Huq > > > >>>>>>>>> Graduate Research Assistant > > > >>>>>>>>> Department of Nuclear, Plasma & Radiological Engineering (NPRE) > > > >>>>>>>>> University of Illinois at Urbana-Champaign (UIUC) > > > >>>>>>>>> E-mail: huq2090 at gmail.com > > > >>>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> -- > > > >>>>>>>> What most experimenters take for granted before they begin their > > > >>>>>>>> experiments is infinitely more interesting than any results to > > which their > > > >>>>>>>> experiments lead. > > > >>>>>>>> -- Norbert Wiener > > > >>>>>>>> > > > >>>>>>>> https://www.cse.buffalo.edu/~knepley/ > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>> > > > >>>>>>> > > > >>>>>>> -- > > > >>>>>>> > > > >>>>>>> Fazlul Huq > > > >>>>>>> Graduate Research Assistant > > > >>>>>>> Department of Nuclear, Plasma & Radiological Engineering (NPRE) > > > >>>>>>> University of Illinois at Urbana-Champaign (UIUC) > > > >>>>>>> E-mail: huq2090 at gmail.com > > > >>>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> -- > > > >>>>>> What most experimenters take for granted before they begin their > > > >>>>>> experiments is infinitely more interesting than any results to > > which their > > > >>>>>> experiments lead. > > > >>>>>> -- Norbert Wiener > > > >>>>>> > > > >>>>>> https://www.cse.buffalo.edu/~knepley/ > > > >>>>>> > > > >>>>>> > > > >>>>> > > > >>> > > > >>> -- > > > >>> What most experimenters take for granted before they begin their > > > >>> experiments is infinitely more interesting than any results to which > > their > > > >>> experiments lead. > > > >>> -- Norbert Wiener > > > >>> > > > >>> https://www.cse.buffalo.edu/~knepley/ > > > >>> > > > >>> > > > >> > > > >> > > > >> -- > > > >> > > > >> Fazlul Huq > > > >> Graduate Research Assistant > > > >> Department of Nuclear, Plasma & Radiological Engineering (NPRE) > > > >> University of Illinois at Urbana-Champaign (UIUC) > > > >> E-mail: huq2090 at gmail.com > > > >> > > > > > > > > > > > > -- > > > > What most experimenters take for granted before they begin their > > > > experiments is infinitely more interesting than any results to which > > their > > > > experiments lead. > > > > -- Norbert Wiener > > > > > > > > https://www.cse.buffalo.edu/~knepley/ > > > > > > > > > > > > > > > > > > > > > > > From balay at mcs.anl.gov Thu Jun 4 14:32:31 2020 From: balay at mcs.anl.gov (Satish Balay) Date: Thu, 4 Jun 2020 14:32:31 -0500 (CDT) Subject: [petsc-users] Terminating a process running petsc via petsc4py without mpi_abort In-Reply-To: References: Message-ID: I don't completely understand the issue here. How is sequential run different than parallel run? In both cases - a PetscErrorHandler is likely getting invoked. One can change this behavior with: https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscPushErrorHandler.html And there are a few default error handlers to choose PETSC_EXTERN PetscErrorCode PetscTraceBackErrorHandler(MPI_Comm,int,const char*,const char*,PetscErrorCode,PetscErrorType,const char*,void*); PETSC_EXTERN PetscErrorCode PetscIgnoreErrorHandler(MPI_Comm,int,const char*,const char*,PetscErrorCode,PetscErrorType,const char*,void*); PETSC_EXTERN PetscErrorCode PetscEmacsClientErrorHandler(MPI_Comm,int,const char*,const char*,PetscErrorCode,PetscErrorType,const char*,void*); PETSC_EXTERN PetscErrorCode PetscMPIAbortErrorHandler(MPI_Comm,int,const char*,const char*,PetscErrorCode,PetscErrorType,const char*,void*); PETSC_EXTERN PetscErrorCode PetscAbortErrorHandler(MPI_Comm,int,const char*,const char*,PetscErrorCode,PetscErrorType,const char*,void*); PETSC_EXTERN PetscErrorCode PetscAttachDebuggerErrorHandler(MPI_Comm,int,const char*,const char*,PetscErrorCode,PetscErrorType,const char*,void*); PETSC_EXTERN PetscErrorCode PetscReturnErrorHandler(MPI_Comm,int,const char*,const char*,PetscErrorCode,PetscErrorType,const char*,void*); Some of the are accessible via command line option. for ex: -on_error_abort or -on_error_mpiabort Or perhaps you want to completely disable error handler with: -no_signal_handler cc: petsc-users Satish On Thu, 4 Jun 2020, Hudson, Stephen Tobias P wrote: > Satish, > > We are having issues caused by MPI_abort getting called when we try to terminate a sub-process running petsc4py. Ideally we would always use a serial build of petsc/petsc4py in this mode, but many users will have a parallel build. We need to be able to send a terminate signal that just kills the process. > > Is there a way to turn off the mpi_abort? > > Thanks, > > Steve > > From shudson at anl.gov Thu Jun 4 14:47:03 2020 From: shudson at anl.gov (Hudson, Stephen Tobias P) Date: Thu, 4 Jun 2020 19:47:03 +0000 Subject: [petsc-users] Terminating a process running petsc via petsc4py without mpi_abort In-Reply-To: References: , Message-ID: Sounds good. I will have a look at how to set this through petsc4py. Thanks Steve ________________________________ From: Satish Balay Sent: Thursday, June 4, 2020 2:32 PM To: Hudson, Stephen Tobias P Cc: petsc-users at mcs.anl.gov ; Lisandro Dalcin Subject: Re: Terminating a process running petsc via petsc4py without mpi_abort I don't completely understand the issue here. How is sequential run different than parallel run? In both cases - a PetscErrorHandler is likely getting invoked. One can change this behavior with: https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscPushErrorHandler.html And there are a few default error handlers to choose PETSC_EXTERN PetscErrorCode PetscTraceBackErrorHandler(MPI_Comm,int,const char*,const char*,PetscErrorCode,PetscErrorType,const char*,void*); PETSC_EXTERN PetscErrorCode PetscIgnoreErrorHandler(MPI_Comm,int,const char*,const char*,PetscErrorCode,PetscErrorType,const char*,void*); PETSC_EXTERN PetscErrorCode PetscEmacsClientErrorHandler(MPI_Comm,int,const char*,const char*,PetscErrorCode,PetscErrorType,const char*,void*); PETSC_EXTERN PetscErrorCode PetscMPIAbortErrorHandler(MPI_Comm,int,const char*,const char*,PetscErrorCode,PetscErrorType,const char*,void*); PETSC_EXTERN PetscErrorCode PetscAbortErrorHandler(MPI_Comm,int,const char*,const char*,PetscErrorCode,PetscErrorType,const char*,void*); PETSC_EXTERN PetscErrorCode PetscAttachDebuggerErrorHandler(MPI_Comm,int,const char*,const char*,PetscErrorCode,PetscErrorType,const char*,void*); PETSC_EXTERN PetscErrorCode PetscReturnErrorHandler(MPI_Comm,int,const char*,const char*,PetscErrorCode,PetscErrorType,const char*,void*); Some of the are accessible via command line option. for ex: -on_error_abort or -on_error_mpiabort Or perhaps you want to completely disable error handler with: -no_signal_handler cc: petsc-users Satish On Thu, 4 Jun 2020, Hudson, Stephen Tobias P wrote: > Satish, > > We are having issues caused by MPI_abort getting called when we try to terminate a sub-process running petsc4py. Ideally we would always use a serial build of petsc/petsc4py in this mode, but many users will have a parallel build. We need to be able to send a terminate signal that just kills the process. > > Is there a way to turn off the mpi_abort? > > Thanks, > > Steve > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From shudson at anl.gov Thu Jun 4 15:39:50 2020 From: shudson at anl.gov (Hudson, Stephen Tobias P) Date: Thu, 4 Jun 2020 20:39:50 +0000 Subject: [petsc-users] Terminating a process running petsc via petsc4py without mpi_abort In-Reply-To: References: , , Message-ID: Lisandro, I don't see an interface to set this through petsc4py. Is it possible? Thanks, Steve ________________________________ From: Hudson, Stephen Tobias P Sent: Thursday, June 4, 2020 2:47 PM To: Balay, Satish Cc: petsc-users at mcs.anl.gov ; Lisandro Dalcin Subject: Re: Terminating a process running petsc via petsc4py without mpi_abort Sounds good. I will have a look at how to set this through petsc4py. Thanks Steve ________________________________ From: Satish Balay Sent: Thursday, June 4, 2020 2:32 PM To: Hudson, Stephen Tobias P Cc: petsc-users at mcs.anl.gov ; Lisandro Dalcin Subject: Re: Terminating a process running petsc via petsc4py without mpi_abort I don't completely understand the issue here. How is sequential run different than parallel run? In both cases - a PetscErrorHandler is likely getting invoked. One can change this behavior with: https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscPushErrorHandler.html And there are a few default error handlers to choose PETSC_EXTERN PetscErrorCode PetscTraceBackErrorHandler(MPI_Comm,int,const char*,const char*,PetscErrorCode,PetscErrorType,const char*,void*); PETSC_EXTERN PetscErrorCode PetscIgnoreErrorHandler(MPI_Comm,int,const char*,const char*,PetscErrorCode,PetscErrorType,const char*,void*); PETSC_EXTERN PetscErrorCode PetscEmacsClientErrorHandler(MPI_Comm,int,const char*,const char*,PetscErrorCode,PetscErrorType,const char*,void*); PETSC_EXTERN PetscErrorCode PetscMPIAbortErrorHandler(MPI_Comm,int,const char*,const char*,PetscErrorCode,PetscErrorType,const char*,void*); PETSC_EXTERN PetscErrorCode PetscAbortErrorHandler(MPI_Comm,int,const char*,const char*,PetscErrorCode,PetscErrorType,const char*,void*); PETSC_EXTERN PetscErrorCode PetscAttachDebuggerErrorHandler(MPI_Comm,int,const char*,const char*,PetscErrorCode,PetscErrorType,const char*,void*); PETSC_EXTERN PetscErrorCode PetscReturnErrorHandler(MPI_Comm,int,const char*,const char*,PetscErrorCode,PetscErrorType,const char*,void*); Some of the are accessible via command line option. for ex: -on_error_abort or -on_error_mpiabort Or perhaps you want to completely disable error handler with: -no_signal_handler cc: petsc-users Satish On Thu, 4 Jun 2020, Hudson, Stephen Tobias P wrote: > Satish, > > We are having issues caused by MPI_abort getting called when we try to terminate a sub-process running petsc4py. Ideally we would always use a serial build of petsc/petsc4py in this mode, but many users will have a parallel build. We need to be able to send a terminate signal that just kills the process. > > Is there a way to turn off the mpi_abort? > > Thanks, > > Steve > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From junchao.zhang at gmail.com Thu Jun 4 15:41:09 2020 From: junchao.zhang at gmail.com (Junchao Zhang) Date: Thu, 4 Jun 2020 15:41:09 -0500 Subject: [petsc-users] Terminating a process running petsc via petsc4py without mpi_abort In-Reply-To: References: Message-ID: John Peterson sent me an email (Thanks!) > Regarding our past discussion on PETSc signal handlers, there was a recent > thread on petsc-users: https://lists.mcs.anl.gov/pipermail/petsc-users/2020-June/041215.html > I think this poster's question may be related to signal handlers rather > than error handlers as Satish replied... unfortunately I am not subscribed > to petsc-users so I am not able to reply directly, but it may be useful to > point out the open MR on this topic: > https://gitlab.com/petsc/petsc/-/merge_requests/2745 I think we should approve this MR as it exactly addresses the Steve's problem. PetscXxxxErrorHandler handles errors, not signals, such as SIGTERM. On Thu, Jun 4, 2020 at 2:32 PM Satish Balay via petsc-users < petsc-users at mcs.anl.gov> wrote: > I don't completely understand the issue here. How is sequential run > different than parallel run? > > In both cases - a PetscErrorHandler is likely getting invoked. One can > change this behavior with: > > > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscPushErrorHandler.html > > And there are a few default error handlers to choose > > > PETSC_EXTERN PetscErrorCode PetscTraceBackErrorHandler(MPI_Comm,int,const > char*,const char*,PetscErrorCode,PetscErrorType,const char*,void*); > PETSC_EXTERN PetscErrorCode PetscIgnoreErrorHandler(MPI_Comm,int,const > char*,const char*,PetscErrorCode,PetscErrorType,const char*,void*); > PETSC_EXTERN PetscErrorCode > PetscEmacsClientErrorHandler(MPI_Comm,int,const char*,const > char*,PetscErrorCode,PetscErrorType,const char*,void*); > PETSC_EXTERN PetscErrorCode PetscMPIAbortErrorHandler(MPI_Comm,int,const > char*,const char*,PetscErrorCode,PetscErrorType,const char*,void*); > PETSC_EXTERN PetscErrorCode PetscAbortErrorHandler(MPI_Comm,int,const > char*,const char*,PetscErrorCode,PetscErrorType,const char*,void*); > PETSC_EXTERN PetscErrorCode > PetscAttachDebuggerErrorHandler(MPI_Comm,int,const char*,const > char*,PetscErrorCode,PetscErrorType,const char*,void*); > PETSC_EXTERN PetscErrorCode PetscReturnErrorHandler(MPI_Comm,int,const > char*,const char*,PetscErrorCode,PetscErrorType,const char*,void*); > > Some of the are accessible via command line option. for ex: > -on_error_abort or -on_error_mpiabort > > Or perhaps you want to completely disable error handler with: > -no_signal_handler > > cc: petsc-users > > Satish > > On Thu, 4 Jun 2020, Hudson, Stephen Tobias P wrote: > > > Satish, > > > > We are having issues caused by MPI_abort getting called when we try to > terminate a sub-process running petsc4py. Ideally we would always use a > serial build of petsc/petsc4py in this mode, but many users will have a > parallel build. We need to be able to send a terminate signal that just > kills the process. > > > > Is there a way to turn off the mpi_abort? > > > > Thanks, > > > > Steve > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From junchao.zhang at gmail.com Thu Jun 4 15:50:16 2020 From: junchao.zhang at gmail.com (Junchao Zhang) Date: Thu, 4 Jun 2020 15:50:16 -0500 Subject: [petsc-users] Terminating a process running petsc via petsc4py without mpi_abort In-Reply-To: References: Message-ID: Hi, Steve, Try this branch jczhang/let-signal-handler-call-exit to see if it fixed your problem. If yes, vote for it at https://gitlab.com/petsc/petsc/-/merge_requests/2745 :) --Junchao Zhang On Thu, Jun 4, 2020 at 3:40 PM Hudson, Stephen Tobias P via petsc-users < petsc-users at mcs.anl.gov> wrote: > Lisandro, > > I don't see an interface to set this through petsc4py. Is it possible? > > Thanks, > Steve > ------------------------------ > *From:* Hudson, Stephen Tobias P > *Sent:* Thursday, June 4, 2020 2:47 PM > *To:* Balay, Satish > *Cc:* petsc-users at mcs.anl.gov ; Lisandro Dalcin < > dalcinl at gmail.com> > *Subject:* Re: Terminating a process running petsc via petsc4py without > mpi_abort > > Sounds good. I will have a look at how to set this through petsc4py. > > Thanks > Steve > ------------------------------ > *From:* Satish Balay > *Sent:* Thursday, June 4, 2020 2:32 PM > *To:* Hudson, Stephen Tobias P > *Cc:* petsc-users at mcs.anl.gov ; Lisandro Dalcin < > dalcinl at gmail.com> > *Subject:* Re: Terminating a process running petsc via petsc4py without > mpi_abort > > I don't completely understand the issue here. How is sequential run > different than parallel run? > > In both cases - a PetscErrorHandler is likely getting invoked. One can > change this behavior with: > > > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscPushErrorHandler.html > > And there are a few default error handlers to choose > > > PETSC_EXTERN PetscErrorCode PetscTraceBackErrorHandler(MPI_Comm,int,const > char*,const char*,PetscErrorCode,PetscErrorType,const char*,void*); > PETSC_EXTERN PetscErrorCode PetscIgnoreErrorHandler(MPI_Comm,int,const > char*,const char*,PetscErrorCode,PetscErrorType,const char*,void*); > PETSC_EXTERN PetscErrorCode > PetscEmacsClientErrorHandler(MPI_Comm,int,const char*,const > char*,PetscErrorCode,PetscErrorType,const char*,void*); > PETSC_EXTERN PetscErrorCode PetscMPIAbortErrorHandler(MPI_Comm,int,const > char*,const char*,PetscErrorCode,PetscErrorType,const char*,void*); > PETSC_EXTERN PetscErrorCode PetscAbortErrorHandler(MPI_Comm,int,const > char*,const char*,PetscErrorCode,PetscErrorType,const char*,void*); > PETSC_EXTERN PetscErrorCode > PetscAttachDebuggerErrorHandler(MPI_Comm,int,const char*,const > char*,PetscErrorCode,PetscErrorType,const char*,void*); > PETSC_EXTERN PetscErrorCode PetscReturnErrorHandler(MPI_Comm,int,const > char*,const char*,PetscErrorCode,PetscErrorType,const char*,void*); > > Some of the are accessible via command line option. for ex: > -on_error_abort or -on_error_mpiabort > > Or perhaps you want to completely disable error handler with: > -no_signal_handler > > cc: petsc-users > > Satish > > On Thu, 4 Jun 2020, Hudson, Stephen Tobias P wrote: > > > Satish, > > > > We are having issues caused by MPI_abort getting called when we try to > terminate a sub-process running petsc4py. Ideally we would always use a > serial build of petsc/petsc4py in this mode, but many users will have a > parallel build. We need to be able to send a terminate signal that just > kills the process. > > > > Is there a way to turn off the mpi_abort? > > > > Thanks, > > > > Steve > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From shudson at anl.gov Thu Jun 4 15:53:38 2020 From: shudson at anl.gov (Hudson, Stephen Tobias P) Date: Thu, 4 Jun 2020 20:53:38 +0000 Subject: [petsc-users] Terminating a process running petsc via petsc4py without mpi_abort In-Reply-To: References: , Message-ID: I will try this, thanks! Steve ________________________________ From: Junchao Zhang Sent: Thursday, June 4, 2020 3:50 PM To: Hudson, Stephen Tobias P Cc: Balay, Satish ; petsc-users at mcs.anl.gov Subject: Re: [petsc-users] Terminating a process running petsc via petsc4py without mpi_abort Hi, Steve, Try this branch jczhang/let-signal-handler-call-exit to see if it fixed your problem. If yes, vote for it at https://gitlab.com/petsc/petsc/-/merge_requests/2745 :) --Junchao Zhang On Thu, Jun 4, 2020 at 3:40 PM Hudson, Stephen Tobias P via petsc-users > wrote: Lisandro, I don't see an interface to set this through petsc4py. Is it possible? Thanks, Steve ________________________________ From: Hudson, Stephen Tobias P > Sent: Thursday, June 4, 2020 2:47 PM To: Balay, Satish > Cc: petsc-users at mcs.anl.gov >; Lisandro Dalcin > Subject: Re: Terminating a process running petsc via petsc4py without mpi_abort Sounds good. I will have a look at how to set this through petsc4py. Thanks Steve ________________________________ From: Satish Balay > Sent: Thursday, June 4, 2020 2:32 PM To: Hudson, Stephen Tobias P > Cc: petsc-users at mcs.anl.gov >; Lisandro Dalcin > Subject: Re: Terminating a process running petsc via petsc4py without mpi_abort I don't completely understand the issue here. How is sequential run different than parallel run? In both cases - a PetscErrorHandler is likely getting invoked. One can change this behavior with: https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscPushErrorHandler.html And there are a few default error handlers to choose PETSC_EXTERN PetscErrorCode PetscTraceBackErrorHandler(MPI_Comm,int,const char*,const char*,PetscErrorCode,PetscErrorType,const char*,void*); PETSC_EXTERN PetscErrorCode PetscIgnoreErrorHandler(MPI_Comm,int,const char*,const char*,PetscErrorCode,PetscErrorType,const char*,void*); PETSC_EXTERN PetscErrorCode PetscEmacsClientErrorHandler(MPI_Comm,int,const char*,const char*,PetscErrorCode,PetscErrorType,const char*,void*); PETSC_EXTERN PetscErrorCode PetscMPIAbortErrorHandler(MPI_Comm,int,const char*,const char*,PetscErrorCode,PetscErrorType,const char*,void*); PETSC_EXTERN PetscErrorCode PetscAbortErrorHandler(MPI_Comm,int,const char*,const char*,PetscErrorCode,PetscErrorType,const char*,void*); PETSC_EXTERN PetscErrorCode PetscAttachDebuggerErrorHandler(MPI_Comm,int,const char*,const char*,PetscErrorCode,PetscErrorType,const char*,void*); PETSC_EXTERN PetscErrorCode PetscReturnErrorHandler(MPI_Comm,int,const char*,const char*,PetscErrorCode,PetscErrorType,const char*,void*); Some of the are accessible via command line option. for ex: -on_error_abort or -on_error_mpiabort Or perhaps you want to completely disable error handler with: -no_signal_handler cc: petsc-users Satish On Thu, 4 Jun 2020, Hudson, Stephen Tobias P wrote: > Satish, > > We are having issues caused by MPI_abort getting called when we try to terminate a sub-process running petsc4py. Ideally we would always use a serial build of petsc/petsc4py in this mode, but many users will have a parallel build. We need to be able to send a terminate signal that just kills the process. > > Is there a way to turn off the mpi_abort? > > Thanks, > > Steve > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From s_g at berkeley.edu Thu Jun 4 16:21:20 2020 From: s_g at berkeley.edu (Sanjay Govindjee) Date: Thu, 4 Jun 2020 14:21:20 -0700 Subject: [petsc-users] MatSeqBAIJSetPreallocation Message-ID: <0dabdcd1-ccc3-de58-5926-732f96b1a98e@berkeley.edu> I'm moving from 3.10 to 3.13 and ran into a compilation problem with? MatSeqBAIJSetPreallocation( ). The manual page shows: PetscErrorCode MatSeqBAIJSetPreallocation (Mat B,PetscInt bs,PetscInt nz,const PetscInt nnz[]) Which I think is the way it was before but my code is now tossing a compile error ?? 65 |????? & PETSC_NULL_INTEGER,mr(np(246)), ????? |???????????????????????????????????? 1 Error: Rank mismatch in argument 'c' at (1) (scalar and rank-1) The full line looks like: ??? ??? call MatSeqBAIJSetPreallocation(Mmat,nsbk, ??? & PETSC_NULL_INTEGER,mr(np(246)), ??? & ierr) Any suggestions on what I've messed up? -sanjay -------------- next part -------------- An HTML attachment was scrubbed... URL: From dalcinl at gmail.com Thu Jun 4 16:54:44 2020 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Fri, 5 Jun 2020 00:54:44 +0300 Subject: [petsc-users] Terminating a process running petsc via petsc4py without mpi_abort In-Reply-To: References: Message-ID: (1) You can use PETSc.Sys.pushErrorHandler("abort"), but it will not help you. What you really need is to override PETSc's default signal handling (2) While it is true that PETSc overrides the signal handler, you can override it again from python after from petsc4py import PETSc. For implementing (2), maybe you should try sending SIGINT and not SIGTERM, such that you can do the following. from petsc4py import PETSc import signal signal.signal(signal.SIGINT, signal.default_int_handler) ... if __name__ == "__main__": try: main() except KeyboardInterrupt: # Triggered if Ctrl+C or signaled with SIGINT ... # do cleanup if needed Otherwise, you just need signal.signal(signal.SIGINT, signal.SIG_DFL) PS: I'm not in favor of changing current PETSc's signal handling behavior. This particular issue is fixable with two lines of Python code: from signal import signal, SIGINT, SIG_DFL signal(SIGINT, SIG_DFL) On Thu, 4 Jun 2020 at 23:39, Hudson, Stephen Tobias P wrote: > Lisandro, > > I don't see an interface to set this through petsc4py. Is it possible? > > Thanks, > Steve > ------------------------------ > *From:* Hudson, Stephen Tobias P > *Sent:* Thursday, June 4, 2020 2:47 PM > *To:* Balay, Satish > *Cc:* petsc-users at mcs.anl.gov ; Lisandro Dalcin < > dalcinl at gmail.com> > *Subject:* Re: Terminating a process running petsc via petsc4py without > mpi_abort > > Sounds good. I will have a look at how to set this through petsc4py. > > Thanks > Steve > ------------------------------ > *From:* Satish Balay > *Sent:* Thursday, June 4, 2020 2:32 PM > *To:* Hudson, Stephen Tobias P > *Cc:* petsc-users at mcs.anl.gov ; Lisandro Dalcin < > dalcinl at gmail.com> > *Subject:* Re: Terminating a process running petsc via petsc4py without > mpi_abort > > I don't completely understand the issue here. How is sequential run > different than parallel run? > > In both cases - a PetscErrorHandler is likely getting invoked. One can > change this behavior with: > > > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscPushErrorHandler.html > > And there are a few default error handlers to choose > > > PETSC_EXTERN PetscErrorCode PetscTraceBackErrorHandler(MPI_Comm,int,const > char*,const char*,PetscErrorCode,PetscErrorType,const char*,void*); > PETSC_EXTERN PetscErrorCode PetscIgnoreErrorHandler(MPI_Comm,int,const > char*,const char*,PetscErrorCode,PetscErrorType,const char*,void*); > PETSC_EXTERN PetscErrorCode > PetscEmacsClientErrorHandler(MPI_Comm,int,const char*,const > char*,PetscErrorCode,PetscErrorType,const char*,void*); > PETSC_EXTERN PetscErrorCode PetscMPIAbortErrorHandler(MPI_Comm,int,const > char*,const char*,PetscErrorCode,PetscErrorType,const char*,void*); > PETSC_EXTERN PetscErrorCode PetscAbortErrorHandler(MPI_Comm,int,const > char*,const char*,PetscErrorCode,PetscErrorType,const char*,void*); > PETSC_EXTERN PetscErrorCode > PetscAttachDebuggerErrorHandler(MPI_Comm,int,const char*,const > char*,PetscErrorCode,PetscErrorType,const char*,void*); > PETSC_EXTERN PetscErrorCode PetscReturnErrorHandler(MPI_Comm,int,const > char*,const char*,PetscErrorCode,PetscErrorType,const char*,void*); > > Some of the are accessible via command line option. for ex: > -on_error_abort or -on_error_mpiabort > > Or perhaps you want to completely disable error handler with: > -no_signal_handler > > cc: petsc-users > > Satish > > On Thu, 4 Jun 2020, Hudson, Stephen Tobias P wrote: > > > Satish, > > > > We are having issues caused by MPI_abort getting called when we try to > terminate a sub-process running petsc4py. Ideally we would always use a > serial build of petsc/petsc4py in this mode, but many users will have a > parallel build. We need to be able to send a terminate signal that just > kills the process. > > > > Is there a way to turn off the mpi_abort? > > > > Thanks, > > > > Steve > > > > > > -- Lisandro Dalcin ============ Research Scientist Extreme Computing Research Center (ECRC) King Abdullah University of Science and Technology (KAUST) http://ecrc.kaust.edu.sa/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Thu Jun 4 17:16:12 2020 From: balay at mcs.anl.gov (Satish Balay) Date: Thu, 4 Jun 2020 17:16:12 -0500 (CDT) Subject: [petsc-users] MatSeqBAIJSetPreallocation In-Reply-To: <0dabdcd1-ccc3-de58-5926-732f96b1a98e@berkeley.edu> References: <0dabdcd1-ccc3-de58-5926-732f96b1a98e@berkeley.edu> Message-ID: On Thu, 4 Jun 2020, Sanjay Govindjee wrote: > I'm moving from 3.10 to 3.13 and ran into a compilation problem with? > MatSeqBAIJSetPreallocation( ). > The manual page shows: > > PetscErrorCode > > MatSeqBAIJSetPreallocation > (Mat > > B,PetscInt > > bs,PetscInt > > nz,const PetscInt > > nnz[]) > > > Which I think is the way it was before but my code is now tossing a compile > error > > > ?? 65 |????? & PETSC_NULL_INTEGER,mr(np(246)), > ????? |???????????????????????????????????? 1 > Error: Rank mismatch in argument 'c' at (1) (scalar and rank-1) > > > The full line looks like: > ??? ??? call MatSeqBAIJSetPreallocation(Mmat,nsbk, > ??? & PETSC_NULL_INTEGER,mr(np(246)), I think this should be PETSC_DEFAULT_INTEGER. Previously there were no Interface definitions - so no error checks by the compiler. Satish > ??? & ierr) > > Any suggestions on what I've messed up? > -sanjay > > From huq2090 at gmail.com Thu Jun 4 17:39:24 2020 From: huq2090 at gmail.com (Fazlul Huq) Date: Thu, 4 Jun 2020 17:39:24 -0500 Subject: [petsc-users] Running example problem In-Reply-To: References: Message-ID: "make all test" gives the attached summary. Looks like only one problem failed. All the others passed. Thanks. Sincerely, Huq On Thu, Jun 4, 2020 at 12:27 PM Satish Balay wrote: > Should work > > If it fails - send us corresponding configure.log > > Satish > > On Thu, 4 Jun 2020, Fazlul Huq wrote: > > > I think it's okay to have a separate version of hypre for PETSc. > > So, what I understand is I have to do, > > > > ./configure --with-cc=gcc --with-cxx=g++ --with-fc=gfortran > > --download-mpich --download-fblaslapack --download-hypre > > make all test > > > > Is that all? > > Am I missing something? > > > > Thank you. > > Sincerely, > > Huq > > > > > > On Thu, Jun 4, 2020 at 11:04 AM Satish Balay wrote: > > > > > Its best to use --download-hypre with PETSc configure - it installs a > > > compatible version of hypre. > > > > > > If you have a pre-installed hypre, you'l have to: > > > - make sure its compatible version > > > - its built with the same MPI, compilers etc as PETSc > > > - you can use --with-hypre-dir option with PETSc configure > > > > > > Satish > > > > > > On Thu, 4 Jun 2020, Fazlul Huq wrote: > > > > > > > I have HYPRE installed in my machine. > > > > Ithink, I need to compile PETSc with HYPRE. > > > > I'm not clear about how to do that? > > > > > > > > Thank you. > > > > Sincerely, > > > > Huq > > > > > > > > On Thu, Jun 4, 2020 at 10:13 AM Matthew Knepley > > > wrote: > > > > > > > > > On Thu, Jun 4, 2020 at 10:28 AM Fazlul Huq > wrote: > > > > > > > > > >> Thank you. > > > > >> Working now. > > > > >> > > > > >> I think to run the following command I need to configure PETSc > with > > > HYPRE. > > > > >> mpiexec -np 1 ./poisson_m -n 100000 -pc_type hypre -pc_hypre_type > > > > >> boomerang > > > > >> > > > > >> If I am right, can you please guide me how to configure with > HYPRE and > > > > >> run the above command? > > > > >> > > > > > > > > > > --download-hypre > > > > > > > > > > Thanks, > > > > > > > > > > Matt > > > > > > > > > > > > > > >> Thank you. > > > > >> Sincerely, > > > > >> Huq > > > > >> > > > > >> On Thu, Jun 4, 2020 at 8:34 AM Matthew Knepley > > > > wrote: > > > > >> > > > > >>> > > > > >>> On Thu, Jun 4, 2020 at 9:19 AM Dave May > > > > wrote: > > > > >>> > > > > >>>> > > > > >>>> > > > > >>>> On Thu, 4 Jun 2020 at 14:17, Dave May > > > wrote: > > > > >>>> > > > > >>>>> > > > > >>>>> > > > > >>>>> On Thu, 4 Jun 2020 at 14:15, Matthew Knepley < > knepley at gmail.com> > > > > >>>>> wrote: > > > > >>>>> > > > > >>>>>> On Thu, Jun 4, 2020 at 9:12 AM Fazlul Huq > > > wrote: > > > > >>>>>> > > > > >>>>>>> Somehow, make is not working. > > > > >>>>>>> Please find the attachment herewith for the terminal readout. > > > > >>>>>>> > > > > >>>>>> > > > > >>>>>> Since you built with PETSC_ARCH=linux-gnu, you need that in > your > > > > >>>>>> environment. > > > > >>>>>> > > > > >>>>> > > > > >>>>> Or just do > > > > >>>>> > > > > >>>>> make ex5 PETSC_ARCH=linux-gnu > > > > >>>>> > > > > >>>> > > > > >>>> sorry I hit send without checking your png) > > > > >>>> The command should be > > > > >>>> > > > > >>>> make ex5 PETSC_ARCH=arch-linux2-c-debug > > > > >>>> > > > > >>> > > > > >>> No, your first one was right. > > > > >>> > > > > >>> Matt > > > > >>> > > > > >>> > > > > >>>> > > > > >>>>> > > > > >>>>> > > > > >>>>>> > > > > >>>>>> Thanks, > > > > >>>>>> > > > > >>>>>> Matt > > > > >>>>>> > > > > >>>>>> > > > > >>>>>>> Thank you. > > > > >>>>>>> > > > > >>>>>>> Sincerely, > > > > >>>>>>> Huq > > > > >>>>>>> > > > > >>>>>>> On Thu, Jun 4, 2020 at 7:57 AM Matthew Knepley < > > > knepley at gmail.com> > > > > >>>>>>> wrote: > > > > >>>>>>> > > > > >>>>>>>> On Thu, Jun 4, 2020 at 8:53 AM Fazlul Huq < > huq2090 at gmail.com> > > > > >>>>>>>> wrote: > > > > >>>>>>>> > > > > >>>>>>>>> Hello, > > > > >>>>>>>>> > > > > >>>>>>>>> I have a very preliminary question! > > > > >>>>>>>>> I think I have installed PETSc correctly cause I got > following > > > on > > > > >>>>>>>>> the terminal: > > > > >>>>>>>>> > > > > >>>>>>>>> Command: > > > > >>>>>>>>> make PETSC_DIR=/home/huq2090/petsc-3.10.2 > PETSC_ARCH=linux-gnu > > > > >>>>>>>>> check > > > > >>>>>>>>> Response: > > > > >>>>>>>>> Running check examples to verify correct installation > > > > >>>>>>>>> Using PETSC_DIR=/home/huq2090/petsc-3.10.2 and > > > PETSC_ARCH=linux-gnu > > > > >>>>>>>>> C/C++ example src/snes/tutorials/ex19 run successfully > with 1 > > > MPI > > > > >>>>>>>>> process > > > > >>>>>>>>> C/C++ example src/snes/tutorials/ex19 run successfully > with 2 > > > MPI > > > > >>>>>>>>> processes > > > > >>>>>>>>> Fortran example src/snes/tutorials/ex5f run successfully > with 1 > > > > >>>>>>>>> MPI process > > > > >>>>>>>>> Completed test examples > > > > >>>>>>>>> > > > > >>>>>>>>> Now, I am trying to run an example from the list of > provided > > > > >>>>>>>>> examples. > > > > >>>>>>>>> Can you please help me out, how to run an example problem > from > > > the > > > > >>>>>>>>> list of examples? > > > > >>>>>>>>> I mean, how to make executable and run the executable? > > > > >>>>>>>>> > > > > >>>>>>>> > > > > >>>>>>>> cd $PETSC_DIR > > > > >>>>>>>> cd src/snes/tutorials > > > > >>>>>>>> make ex5 > > > > >>>>>>>> ./ex5 -snes_monitor > > > > >>>>>>>> > > > > >>>>>>>> Thanks, > > > > >>>>>>>> > > > > >>>>>>>> Matt > > > > >>>>>>>> > > > > >>>>>>>> > > > > >>>>>>>>> Thank you. > > > > >>>>>>>>> Sincerely, > > > > >>>>>>>>> Huq > > > > >>>>>>>>> -- > > > > >>>>>>>>> > > > > >>>>>>>>> Fazlul Huq > > > > >>>>>>>>> Graduate Research Assistant > > > > >>>>>>>>> Department of Nuclear, Plasma & Radiological Engineering > (NPRE) > > > > >>>>>>>>> University of Illinois at Urbana-Champaign (UIUC) > > > > >>>>>>>>> E-mail: huq2090 at gmail.com > > > > >>>>>>>>> > > > > >>>>>>>> > > > > >>>>>>>> > > > > >>>>>>>> -- > > > > >>>>>>>> What most experimenters take for granted before they begin > their > > > > >>>>>>>> experiments is infinitely more interesting than any results > to > > > which their > > > > >>>>>>>> experiments lead. > > > > >>>>>>>> -- Norbert Wiener > > > > >>>>>>>> > > > > >>>>>>>> https://www.cse.buffalo.edu/~knepley/ > > > > >>>>>>>> > > > > >>>>>>>> > > > > >>>>>>> > > > > >>>>>>> > > > > >>>>>>> -- > > > > >>>>>>> > > > > >>>>>>> Fazlul Huq > > > > >>>>>>> Graduate Research Assistant > > > > >>>>>>> Department of Nuclear, Plasma & Radiological Engineering > (NPRE) > > > > >>>>>>> University of Illinois at Urbana-Champaign (UIUC) > > > > >>>>>>> E-mail: huq2090 at gmail.com > > > > >>>>>>> > > > > >>>>>> > > > > >>>>>> > > > > >>>>>> -- > > > > >>>>>> What most experimenters take for granted before they begin > their > > > > >>>>>> experiments is infinitely more interesting than any results to > > > which their > > > > >>>>>> experiments lead. > > > > >>>>>> -- Norbert Wiener > > > > >>>>>> > > > > >>>>>> https://www.cse.buffalo.edu/~knepley/ > > > > >>>>>> > > > > >>>>>> > > > > >>>>> > > > > >>> > > > > >>> -- > > > > >>> What most experimenters take for granted before they begin their > > > > >>> experiments is infinitely more interesting than any results to > which > > > their > > > > >>> experiments lead. > > > > >>> -- Norbert Wiener > > > > >>> > > > > >>> https://www.cse.buffalo.edu/~knepley/ > > > > >>> > > > > >>> > > > > >> > > > > >> > > > > >> -- > > > > >> > > > > >> Fazlul Huq > > > > >> Graduate Research Assistant > > > > >> Department of Nuclear, Plasma & Radiological Engineering (NPRE) > > > > >> University of Illinois at Urbana-Champaign (UIUC) > > > > >> E-mail: huq2090 at gmail.com > > > > >> > > > > > > > > > > > > > > > -- > > > > > What most experimenters take for granted before they begin their > > > > > experiments is infinitely more interesting than any results to > which > > > their > > > > > experiments lead. > > > > > -- Norbert Wiener > > > > > > > > > > https://www.cse.buffalo.edu/~knepley/ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- Fazlul Huq Graduate Research Assistant Department of Nuclear, Plasma & Radiological Engineering (NPRE) University of Illinois at Urbana-Champaign (UIUC) E-mail: huq2090 at gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Screenshot from 2020-06-04 17-37-51.png Type: image/png Size: 72097 bytes Desc: not available URL: From s_g at berkeley.edu Thu Jun 4 17:40:21 2020 From: s_g at berkeley.edu (Sanjay Govindjee) Date: Thu, 4 Jun 2020 15:40:21 -0700 Subject: [petsc-users] MatSeqBAIJSetPreallocation In-Reply-To: References: <0dabdcd1-ccc3-de58-5926-732f96b1a98e@berkeley.edu> Message-ID: <416b027c-af6e-573a-7991-4112384bc2c6@berkeley.edu> thanks, that did it.? now I am wondering about all the other PETSC_NULL_INTEGER instances in my code.? how should I be thinking about PETSC_NULL_INTEGER and PETSC_DEFAULT_INTEGER, so that I know that I am using the correct one? On 6/4/20 3:16 PM, Satish Balay wrote: > On Thu, 4 Jun 2020, Sanjay Govindjee wrote: > >> I'm moving from 3.10 to 3.13 and ran into a compilation problem with >> MatSeqBAIJSetPreallocation( ). >> The manual page shows: >> >> PetscErrorCode >> >> MatSeqBAIJSetPreallocation >> (Mat >> >> B,PetscInt >> >> bs,PetscInt >> >> nz,const PetscInt >> >> nnz[]) >> >> >> Which I think is the way it was before but my code is now tossing a compile >> error >> >> >> ?? 65 |????? & PETSC_NULL_INTEGER,mr(np(246)), >> ????? |???????????????????????????????????? 1 >> Error: Rank mismatch in argument 'c' at (1) (scalar and rank-1) >> >> >> The full line looks like: >> ??? ??? call MatSeqBAIJSetPreallocation(Mmat,nsbk, >> ??? & PETSC_NULL_INTEGER,mr(np(246)), > I think this should be PETSC_DEFAULT_INTEGER. > > Previously there were no Interface definitions - so no error checks by the compiler. > > Satish > >> ??? & ierr) >> >> Any suggestions on what I've messed up? >> -sanjay >> >> From mfadams at lbl.gov Thu Jun 4 17:58:52 2020 From: mfadams at lbl.gov (Mark Adams) Date: Thu, 4 Jun 2020 18:58:52 -0400 Subject: [petsc-users] MatSeqBAIJSetPreallocation In-Reply-To: <416b027c-af6e-573a-7991-4112384bc2c6@berkeley.edu> References: <0dabdcd1-ccc3-de58-5926-732f96b1a98e@berkeley.edu> <416b027c-af6e-573a-7991-4112384bc2c6@berkeley.edu> Message-ID: On Thu, Jun 4, 2020 at 6:41 PM Sanjay Govindjee wrote: > thanks, that did it. now I am wondering about all the other > PETSC_NULL_INTEGER instances in > my code. how should I be thinking about PETSC_NULL_INTEGER I would think it's an integer array/pointer in C. > and > PETSC_DEFAULT_INTEGER, > and an integer in C. > so that I know that I am using the correct one? > > On 6/4/20 3:16 PM, Satish Balay wrote: > > On Thu, 4 Jun 2020, Sanjay Govindjee wrote: > > > >> I'm moving from 3.10 to 3.13 and ran into a compilation problem with > >> MatSeqBAIJSetPreallocation( ). > >> The manual page shows: > >> > >> PetscErrorCode > >> < > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscErrorCode.html#PetscErrorCode > > > >> MatSeqBAIJSetPreallocation > >> < > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatSeqBAIJSetPreallocation.html#MatSeqBAIJSetPreallocation > >(Mat > >> < > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/Mat.html#Mat > > > >> B,PetscInt > >> < > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscInt.html#PetscInt > > > >> bs,PetscInt > >> < > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscInt.html#PetscInt > > > >> nz,const PetscInt > >> < > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscInt.html#PetscInt > > > >> nnz[]) > >> > >> > >> Which I think is the way it was before but my code is now tossing a > compile > >> error > >> > >> > >> 65 | & PETSC_NULL_INTEGER,mr(np(246)), > >> | 1 > >> Error: Rank mismatch in argument 'c' at (1) (scalar and rank-1) > >> > >> > >> The full line looks like: > >> call MatSeqBAIJSetPreallocation(Mmat,nsbk, > >> & PETSC_NULL_INTEGER,mr(np(246)), > > I think this should be PETSC_DEFAULT_INTEGER. > > > > Previously there were no Interface definitions - so no error checks by > the compiler. > > > > Satish > > > >> & ierr) > >> > >> Any suggestions on what I've messed up? > >> -sanjay > >> > >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Thu Jun 4 18:04:09 2020 From: balay at mcs.anl.gov (Satish Balay) Date: Thu, 4 Jun 2020 18:04:09 -0500 (CDT) Subject: [petsc-users] Running example problem In-Reply-To: References: Message-ID: 1. its best if you can send us text [i.e copy/paste from screen, or logfiles] - instead of screenshots 2. Normally - you need to do 'make check' - and not 'make test' 3. Since I don't have the log - I don't know why this test was run mat/tests/ex78_1 needs datafiles which are not part of the distribution. [so normally it shouldn't be run in this situation] You can assume the PETSc build is fine - and use it. Satish On Thu, 4 Jun 2020, Fazlul Huq wrote: > "make all test" gives the attached summary. > > Looks like only one problem failed. > All the others passed. > > Thanks. > Sincerely, > Huq > > On Thu, Jun 4, 2020 at 12:27 PM Satish Balay wrote: > > > Should work > > > > If it fails - send us corresponding configure.log > > > > Satish > > > > On Thu, 4 Jun 2020, Fazlul Huq wrote: > > > > > I think it's okay to have a separate version of hypre for PETSc. > > > So, what I understand is I have to do, > > > > > > ./configure --with-cc=gcc --with-cxx=g++ --with-fc=gfortran > > > --download-mpich --download-fblaslapack --download-hypre > > > make all test > > > > > > Is that all? > > > Am I missing something? > > > > > > Thank you. > > > Sincerely, > > > Huq > > > > > > > > > On Thu, Jun 4, 2020 at 11:04 AM Satish Balay wrote: > > > > > > > Its best to use --download-hypre with PETSc configure - it installs a > > > > compatible version of hypre. > > > > > > > > If you have a pre-installed hypre, you'l have to: > > > > - make sure its compatible version > > > > - its built with the same MPI, compilers etc as PETSc > > > > - you can use --with-hypre-dir option with PETSc configure > > > > > > > > Satish > > > > > > > > On Thu, 4 Jun 2020, Fazlul Huq wrote: > > > > > > > > > I have HYPRE installed in my machine. > > > > > Ithink, I need to compile PETSc with HYPRE. > > > > > I'm not clear about how to do that? > > > > > > > > > > Thank you. > > > > > Sincerely, > > > > > Huq > > > > > > > > > > On Thu, Jun 4, 2020 at 10:13 AM Matthew Knepley > > > > wrote: > > > > > > > > > > > On Thu, Jun 4, 2020 at 10:28 AM Fazlul Huq > > wrote: > > > > > > > > > > > >> Thank you. > > > > > >> Working now. > > > > > >> > > > > > >> I think to run the following command I need to configure PETSc > > with > > > > HYPRE. > > > > > >> mpiexec -np 1 ./poisson_m -n 100000 -pc_type hypre -pc_hypre_type > > > > > >> boomerang > > > > > >> > > > > > >> If I am right, can you please guide me how to configure with > > HYPRE and > > > > > >> run the above command? > > > > > >> > > > > > > > > > > > > --download-hypre > > > > > > > > > > > > Thanks, > > > > > > > > > > > > Matt > > > > > > > > > > > > > > > > > >> Thank you. > > > > > >> Sincerely, > > > > > >> Huq > > > > > >> > > > > > >> On Thu, Jun 4, 2020 at 8:34 AM Matthew Knepley > > > > > > wrote: > > > > > >> > > > > > >>> > > > > > >>> On Thu, Jun 4, 2020 at 9:19 AM Dave May > > > > > > wrote: > > > > > >>> > > > > > >>>> > > > > > >>>> > > > > > >>>> On Thu, 4 Jun 2020 at 14:17, Dave May > > > > wrote: > > > > > >>>> > > > > > >>>>> > > > > > >>>>> > > > > > >>>>> On Thu, 4 Jun 2020 at 14:15, Matthew Knepley < > > knepley at gmail.com> > > > > > >>>>> wrote: > > > > > >>>>> > > > > > >>>>>> On Thu, Jun 4, 2020 at 9:12 AM Fazlul Huq > > > > wrote: > > > > > >>>>>> > > > > > >>>>>>> Somehow, make is not working. > > > > > >>>>>>> Please find the attachment herewith for the terminal readout. > > > > > >>>>>>> > > > > > >>>>>> > > > > > >>>>>> Since you built with PETSC_ARCH=linux-gnu, you need that in > > your > > > > > >>>>>> environment. > > > > > >>>>>> > > > > > >>>>> > > > > > >>>>> Or just do > > > > > >>>>> > > > > > >>>>> make ex5 PETSC_ARCH=linux-gnu > > > > > >>>>> > > > > > >>>> > > > > > >>>> sorry I hit send without checking your png) > > > > > >>>> The command should be > > > > > >>>> > > > > > >>>> make ex5 PETSC_ARCH=arch-linux2-c-debug > > > > > >>>> > > > > > >>> > > > > > >>> No, your first one was right. > > > > > >>> > > > > > >>> Matt > > > > > >>> > > > > > >>> > > > > > >>>> > > > > > >>>>> > > > > > >>>>> > > > > > >>>>>> > > > > > >>>>>> Thanks, > > > > > >>>>>> > > > > > >>>>>> Matt > > > > > >>>>>> > > > > > >>>>>> > > > > > >>>>>>> Thank you. > > > > > >>>>>>> > > > > > >>>>>>> Sincerely, > > > > > >>>>>>> Huq > > > > > >>>>>>> > > > > > >>>>>>> On Thu, Jun 4, 2020 at 7:57 AM Matthew Knepley < > > > > knepley at gmail.com> > > > > > >>>>>>> wrote: > > > > > >>>>>>> > > > > > >>>>>>>> On Thu, Jun 4, 2020 at 8:53 AM Fazlul Huq < > > huq2090 at gmail.com> > > > > > >>>>>>>> wrote: > > > > > >>>>>>>> > > > > > >>>>>>>>> Hello, > > > > > >>>>>>>>> > > > > > >>>>>>>>> I have a very preliminary question! > > > > > >>>>>>>>> I think I have installed PETSc correctly cause I got > > following > > > > on > > > > > >>>>>>>>> the terminal: > > > > > >>>>>>>>> > > > > > >>>>>>>>> Command: > > > > > >>>>>>>>> make PETSC_DIR=/home/huq2090/petsc-3.10.2 > > PETSC_ARCH=linux-gnu > > > > > >>>>>>>>> check > > > > > >>>>>>>>> Response: > > > > > >>>>>>>>> Running check examples to verify correct installation > > > > > >>>>>>>>> Using PETSC_DIR=/home/huq2090/petsc-3.10.2 and > > > > PETSC_ARCH=linux-gnu > > > > > >>>>>>>>> C/C++ example src/snes/tutorials/ex19 run successfully > > with 1 > > > > MPI > > > > > >>>>>>>>> process > > > > > >>>>>>>>> C/C++ example src/snes/tutorials/ex19 run successfully > > with 2 > > > > MPI > > > > > >>>>>>>>> processes > > > > > >>>>>>>>> Fortran example src/snes/tutorials/ex5f run successfully > > with 1 > > > > > >>>>>>>>> MPI process > > > > > >>>>>>>>> Completed test examples > > > > > >>>>>>>>> > > > > > >>>>>>>>> Now, I am trying to run an example from the list of > > provided > > > > > >>>>>>>>> examples. > > > > > >>>>>>>>> Can you please help me out, how to run an example problem > > from > > > > the > > > > > >>>>>>>>> list of examples? > > > > > >>>>>>>>> I mean, how to make executable and run the executable? > > > > > >>>>>>>>> > > > > > >>>>>>>> > > > > > >>>>>>>> cd $PETSC_DIR > > > > > >>>>>>>> cd src/snes/tutorials > > > > > >>>>>>>> make ex5 > > > > > >>>>>>>> ./ex5 -snes_monitor > > > > > >>>>>>>> > > > > > >>>>>>>> Thanks, > > > > > >>>>>>>> > > > > > >>>>>>>> Matt > > > > > >>>>>>>> > > > > > >>>>>>>> > > > > > >>>>>>>>> Thank you. > > > > > >>>>>>>>> Sincerely, > > > > > >>>>>>>>> Huq > > > > > >>>>>>>>> -- > > > > > >>>>>>>>> > > > > > >>>>>>>>> Fazlul Huq > > > > > >>>>>>>>> Graduate Research Assistant > > > > > >>>>>>>>> Department of Nuclear, Plasma & Radiological Engineering > > (NPRE) > > > > > >>>>>>>>> University of Illinois at Urbana-Champaign (UIUC) > > > > > >>>>>>>>> E-mail: huq2090 at gmail.com > > > > > >>>>>>>>> > > > > > >>>>>>>> > > > > > >>>>>>>> > > > > > >>>>>>>> -- > > > > > >>>>>>>> What most experimenters take for granted before they begin > > their > > > > > >>>>>>>> experiments is infinitely more interesting than any results > > to > > > > which their > > > > > >>>>>>>> experiments lead. > > > > > >>>>>>>> -- Norbert Wiener > > > > > >>>>>>>> > > > > > >>>>>>>> https://www.cse.buffalo.edu/~knepley/ > > > > > >>>>>>>> > > > > > >>>>>>>> > > > > > >>>>>>> > > > > > >>>>>>> > > > > > >>>>>>> -- > > > > > >>>>>>> > > > > > >>>>>>> Fazlul Huq > > > > > >>>>>>> Graduate Research Assistant > > > > > >>>>>>> Department of Nuclear, Plasma & Radiological Engineering > > (NPRE) > > > > > >>>>>>> University of Illinois at Urbana-Champaign (UIUC) > > > > > >>>>>>> E-mail: huq2090 at gmail.com > > > > > >>>>>>> > > > > > >>>>>> > > > > > >>>>>> > > > > > >>>>>> -- > > > > > >>>>>> What most experimenters take for granted before they begin > > their > > > > > >>>>>> experiments is infinitely more interesting than any results to > > > > which their > > > > > >>>>>> experiments lead. > > > > > >>>>>> -- Norbert Wiener > > > > > >>>>>> > > > > > >>>>>> https://www.cse.buffalo.edu/~knepley/ > > > > > >>>>>> > > > > > >>>>>> > > > > > >>>>> > > > > > >>> > > > > > >>> -- > > > > > >>> What most experimenters take for granted before they begin their > > > > > >>> experiments is infinitely more interesting than any results to > > which > > > > their > > > > > >>> experiments lead. > > > > > >>> -- Norbert Wiener > > > > > >>> > > > > > >>> https://www.cse.buffalo.edu/~knepley/ > > > > > >>> > > > > > >>> > > > > > >> > > > > > >> > > > > > >> -- > > > > > >> > > > > > >> Fazlul Huq > > > > > >> Graduate Research Assistant > > > > > >> Department of Nuclear, Plasma & Radiological Engineering (NPRE) > > > > > >> University of Illinois at Urbana-Champaign (UIUC) > > > > > >> E-mail: huq2090 at gmail.com > > > > > >> > > > > > > > > > > > > > > > > > > -- > > > > > > What most experimenters take for granted before they begin their > > > > > > experiments is infinitely more interesting than any results to > > which > > > > their > > > > > > experiments lead. > > > > > > -- Norbert Wiener > > > > > > > > > > > > https://www.cse.buffalo.edu/~knepley/ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From s_g at berkeley.edu Thu Jun 4 18:13:04 2020 From: s_g at berkeley.edu (Sanjay Govindjee) Date: Thu, 4 Jun 2020 16:13:04 -0700 Subject: [petsc-users] MatSeqBAIJSetPreallocation In-Reply-To: References: <0dabdcd1-ccc3-de58-5926-732f96b1a98e@berkeley.edu> <416b027c-af6e-573a-7991-4112384bc2c6@berkeley.edu> Message-ID: <9ae36828-d2c0-ba78-3c71-afc996804321@berkeley.edu> Thanks Mark. Very helpful and explains why my other PETSC_NULL_INTEGER uses are still functioning and why in some places I have PETSC_NULL_INTEGER(1).? I guess I can replace the latter with PETSC_DEFAULT_INTEGER for a cleaner look. On 6/4/20 3:58 PM, Mark Adams wrote: > > > On Thu, Jun 4, 2020 at 6:41 PM Sanjay Govindjee > wrote: > > thanks, that did it.? now I am wondering about all the other > PETSC_NULL_INTEGER instances in > my code.? how should I be thinking about PETSC_NULL_INTEGER > > > I would think it's an integer array/pointer in C. > > and > PETSC_DEFAULT_INTEGER, > > > and an integer in C. > > so that I know that I am using the correct one? > > On 6/4/20 3:16 PM, Satish Balay wrote: > > On Thu, 4 Jun 2020, Sanjay Govindjee wrote: > > > >> I'm moving from 3.10 to 3.13 and ran into a compilation problem > with > >> MatSeqBAIJSetPreallocation( ). > >> The manual page shows: > >> > >> PetscErrorCode > >> > > >> MatSeqBAIJSetPreallocation > >> > (Mat > >> > > >> B,PetscInt > >> > > >> bs,PetscInt > >> > > >> nz,const PetscInt > >> > > >> nnz[]) > >> > >> > >> Which I think is the way it was before but my code is now > tossing a compile > >> error > >> > >> > >>? ?? 65 |????? & PETSC_NULL_INTEGER,mr(np(246)), > >>? ????? |???????????????????????????????????? 1 > >> Error: Rank mismatch in argument 'c' at (1) (scalar and rank-1) > >> > >> > >> The full line looks like: > >>? ??? ??? call MatSeqBAIJSetPreallocation(Mmat,nsbk, > >>? ??? & PETSC_NULL_INTEGER,mr(np(246)), > > I think this should be PETSC_DEFAULT_INTEGER. > > > > Previously there were no Interface definitions - so no error > checks by the compiler. > > > > Satish > > > >>? ??? & ierr) > >> > >> Any suggestions on what I've messed up? > >> -sanjay > >> > >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ys453 at cam.ac.uk Fri Jun 5 05:12:33 2020 From: ys453 at cam.ac.uk (Y. Shidi) Date: Fri, 05 Jun 2020 11:12:33 +0100 Subject: [petsc-users] KSP not converge when turn on ksp_monitor_true_residual Message-ID: <9f3e24167df41df1bb9c0d8590ce8328@cam.ac.uk> Dear developers, We are using filed splitting method solve saddle point problem. Below is our ksp configuration: PetscOptionsSetValue(NULL,"-ksp_type","gmres"); PetscOptionsSetValue(NULL,"-ksp_initial_guess_nonzero",""); PetscOptionsSetValue(NULL,"-ksp_rtol","1e-20"); PetscOptionsSetValue(NULL,"-ksp_converged_reason", ""); PetscOptionsSetValue(NULL,"-pc_type", "fieldsplit"); PetscOptionsSetValue(NULL,"-pc_fieldsplit_type", "schur"); PetscOptionsSetValue(NULL,"-pc_fieldsplit_schur_fact_type", "lower"); PetscOptionsSetValue(NULL,"-pc_fieldsplit_schur_precondition", "selfp"); PetscOptionsSetValue(NULL,"-fieldsplit_0_ksp_type", "preonly"); PetscOptionsSetValue(NULL,"-fieldsplit_0_pc_type", "hypre"); PetscOptionsSetValue(NULL,"-fieldsplit_1_ksp_type", "preonly"); PetscOptionsSetValue(NULL,"-fieldsplit_1_pc_type", "hypre"); It can be converged. However, when we turn on the ksp monitor: PetscOptionsSetValue(NULL,"-ksp_monitor_true_residual", ""); It won't converge. It seems very odd, I am wondering why it happens. Thank you for your time and help. Kind regards, Shidi From knepley at gmail.com Fri Jun 5 05:48:09 2020 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 5 Jun 2020 06:48:09 -0400 Subject: [petsc-users] KSP not converge when turn on ksp_monitor_true_residual In-Reply-To: <9f3e24167df41df1bb9c0d8590ce8328@cam.ac.uk> References: <9f3e24167df41df1bb9c0d8590ce8328@cam.ac.uk> Message-ID: On Fri, Jun 5, 2020 at 6:12 AM Y. Shidi wrote: > Dear developers, > > We are using filed splitting method solve saddle point problem. > Below is our ksp configuration: > > PetscOptionsSetValue(NULL,"-ksp_type","gmres"); > PetscOptionsSetValue(NULL,"-ksp_initial_guess_nonzero",""); > PetscOptionsSetValue(NULL,"-ksp_rtol","1e-20"); > This algebraic tolerance is strange. Why put it below unit roundoff? > PetscOptionsSetValue(NULL,"-ksp_converged_reason", ""); > PetscOptionsSetValue(NULL,"-pc_type", "fieldsplit"); > PetscOptionsSetValue(NULL,"-pc_fieldsplit_type", "schur"); > PetscOptionsSetValue(NULL,"-pc_fieldsplit_schur_fact_type", "lower"); > PetscOptionsSetValue(NULL,"-pc_fieldsplit_schur_precondition", > "selfp"); > PetscOptionsSetValue(NULL,"-fieldsplit_0_ksp_type", "preonly"); > PetscOptionsSetValue(NULL,"-fieldsplit_0_pc_type", "hypre"); > PetscOptionsSetValue(NULL,"-fieldsplit_1_ksp_type", "preonly"); > PetscOptionsSetValue(NULL,"-fieldsplit_1_pc_type", "hypre"); > > It can be converged. However, when we turn on the ksp monitor: > > PetscOptionsSetValue(NULL,"-ksp_monitor_true_residual", ""); > > It won't converge. It seems very odd, I am wondering why it happens. > Can you try it with a relative tolerance of 1e-10, and send the output (with -ksp_view as well). Thanks, Matt > Thank you for your time and help. > > Kind regards, > Shidi > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From Fabian.Jakub at physik.uni-muenchen.de Fri Jun 5 06:52:20 2020 From: Fabian.Jakub at physik.uni-muenchen.de (Fabian Jakub) Date: Fri, 5 Jun 2020 13:52:20 +0200 Subject: [petsc-users] Question on reverse scatters from VecScatterCreateToAll Message-ID: <39c11da1-c092-0425-f698-f1552871aea4@physik.uni-muenchen.de> Dear Petsc list, I have a question regarding reverse vec-scatters: I have a shared memory solver that I want to use on a distributed DMDA and average its results. The shared mem solver needs some of the global state. So I want to create a full copy of a global vec on each master rank of a machine, compute some result and average the results back into the global vec. For example: the parallel layout for a global vector is 3 ranks in comm_world, each with Ndof = 6, so global size is 18 Then I create a mpi sub communicator with MPI_COMM_TYPE_SHARED Then I create a local vec on each rank, with sizes 9 on each rank_0 on the sub comm, size 0 otherwise ????? if ( sub_id == 0 ) then ??????? call VecGetSize(gvec, Nlocal, ierr); call CHKERR(ierr) ????? else ??????? Nlocal = 0 ????? endif ????? call VecCreateSeq(PETSC_COMM_SELF, Nlocal, lvec, ierr) This yields for example (first 2 ranks in a shared mem subcomm, 3rd rank on another machine): Global rank_id? |?? sub_rank_id?? |?? local_size_gvec |? local_size_lvec 0?????????????????????? |? 0??????????????????? |?? 6??????????????????????? | 18 1?????????????????????? |? 1??????????????????? |?? 6??????????????????????? | 0 2?????????????????????? |? 0??????????????????? |?? 6??????????????????????? | 18 To copy the global vec on each shared mem master, I do: ISCreateStride(PETSC_COMM_SELF, Nlocal, 0, 1, is, ierr) VecScatterCreate(gvec, is, lvec, is, ctx, ierr) VecScatterBegin(ctx, gvec, lvec, INSERT_VALUES, SCATTER_FORWARD, ierr) VecScatterEnd? (ctx, gvec, lvec, INSERT_VALUES, SCATTER_FORWARD, ierr) That works fine. Then I want to do the reverse, i.e. add all values from the local vec to the global vec on comm_world to generate the average of the results. I tried: VecSet(gvec, zero, ierr) VecScatterBegin(ctx, gvec, lvec, ADD_VALUES, SCATTER_REVERSE, ierr) I was hoping to get the sum of svec in the global vec, so that gvec / comm_size(sub_comm_id==0) gives the mean. However, I get the following error: Nonconforming object sizes Vector wrong size 18 for scatter 6 (scatter reverse and vector to != ctx from size) Going with the same approach with VecScatterCreateToAll leads to the same issue. Do you have suggestions on how I could/should achieve it? Many thanks! Fabian -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 659 bytes Desc: OpenPGP digital signature URL: From dave.mayhem23 at gmail.com Fri Jun 5 07:04:52 2020 From: dave.mayhem23 at gmail.com (Dave May) Date: Fri, 5 Jun 2020 14:04:52 +0200 Subject: [petsc-users] Question on reverse scatters from VecScatterCreateToAll In-Reply-To: <39c11da1-c092-0425-f698-f1552871aea4@physik.uni-muenchen.de> References: <39c11da1-c092-0425-f698-f1552871aea4@physik.uni-muenchen.de> Message-ID: On Fri 5. Jun 2020 at 13:52, Fabian Jakub < Fabian.Jakub at physik.uni-muenchen.de> wrote: > Dear Petsc list, > > I have a question regarding reverse vec-scatters: > > I have a shared memory solver that I want to use on a distributed DMDA and > average its results. > > The shared mem solver needs some of the global state. > > So I want to create a full copy of a global vec on each master rank of a > machine, compute some result > > and average the results back into the global vec. > > > For example: the parallel layout for a global vector is > > 3 ranks in comm_world, each with Ndof = 6, so global size is 18 > > Then I create a mpi sub communicator with MPI_COMM_TYPE_SHARED > > Then I create a local vec on each rank, with sizes 9 on each rank_0 on the > sub comm, size 0 otherwise > > if ( sub_id == 0 ) then > call VecGetSize(gvec, Nlocal, ierr); call CHKERR(ierr) > else > Nlocal = 0 > endif > call VecCreateSeq(PETSC_COMM_SELF, Nlocal, lvec, ierr) > > This yields for example (first 2 ranks in a shared mem subcomm, 3rd rank > on another machine): > > Global rank_id | sub_rank_id | local_size_gvec | local_size_lvec > > 0 | 0 | > 6 | 18 > > 1 | 1 | > 6 | 0 > > 2 | 0 | > 6 | 18 > > > To copy the global vec on each shared mem master, I do: > > ISCreateStride(PETSC_COMM_SELF, Nlocal, 0, 1, is, ierr) > > VecScatterCreate(gvec, is, lvec, is, ctx, ierr) > > VecScatterBegin(ctx, gvec, lvec, INSERT_VALUES, SCATTER_FORWARD, ierr) > > VecScatterEnd (ctx, gvec, lvec, INSERT_VALUES, SCATTER_FORWARD, ierr) > > > That works fine. > > Then I want to do the reverse, i.e. add all values from the local vec to > the global vec on comm_world to generate the average of the results. > > I tried: > > VecSet(gvec, zero, ierr) > > VecScatterBegin(ctx, gvec, lvec, ADD_VALUES, SCATTER_REVERSE, ierr) > > > I was hoping to get the sum of svec in the global vec, so that gvec / > comm_size(sub_comm_id==0) gives the mean. > > However, I get the following error: > > > Nonconforming object sizes > > Vector wrong size 18 for scatter 6 (scatter reverse and vector to != ctx > from size) > > > Going with the same approach with VecScatterCreateToAll > > leads to the same issue. > > Do you have suggestions on how I could/should achieve it? > > You have to flip the lvec, gvec args in reverse mode (as per the man pages states under Notes) Thanks Dave > Many thanks! > > Fabian > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Fabian.Jakub at physik.uni-muenchen.de Fri Jun 5 07:12:52 2020 From: Fabian.Jakub at physik.uni-muenchen.de (Fabian Jakub) Date: Fri, 5 Jun 2020 14:12:52 +0200 Subject: [petsc-users] Question on reverse scatters from VecScatterCreateToAll In-Reply-To: References: <39c11da1-c092-0425-f698-f1552871aea4@physik.uni-muenchen.de> Message-ID: <7e31f50d-2592-ebcc-8ada-0811c3578eca@physik.uni-muenchen.de> Darn, of course! This works! My sincere apologies and many thanks for the swift reply! Fabian On 05.06.20 14:04, Dave May wrote: > > > On Fri 5. Jun 2020 at 13:52, Fabian Jakub > > wrote: > > Dear Petsc list, > > I have a question regarding reverse vec-scatters: > > I have a shared memory solver that I want to use on a distributed > DMDA and average its results. > > The shared mem solver needs some of the global state. > > So I want to create a full copy of a global vec on each master > rank of a machine, compute some result > > and average the results back into the global vec. > > > For example: the parallel layout for a global vector is > > 3 ranks in comm_world, each with Ndof = 6, so global size is 18 > > Then I create a mpi sub communicator with MPI_COMM_TYPE_SHARED > > Then I create a local vec on each rank, with sizes 9 on each > rank_0 on the sub comm, size 0 otherwise > > ????? if ( sub_id == 0 ) then > ??????? call VecGetSize(gvec, Nlocal, ierr); call CHKERR(ierr) > ????? else > ??????? Nlocal = 0 > ????? endif > ????? call VecCreateSeq(PETSC_COMM_SELF, Nlocal, lvec, ierr) > > This yields for example (first 2 ranks in a shared mem subcomm, > 3rd rank on another machine): > > Global rank_id? |?? sub_rank_id?? |?? local_size_gvec |? > local_size_lvec > > 0?????????????????????? |? 0??????????????????? |?? > 6??????????????????????? | 18 > > 1?????????????????????? |? 1??????????????????? |?? > 6??????????????????????? | 0 > > 2?????????????????????? |? 0??????????????????? |?? > 6??????????????????????? | 18 > > > To copy the global vec on each shared mem master, I do: > > ISCreateStride(PETSC_COMM_SELF, Nlocal, 0, 1, is, ierr) > > VecScatterCreate(gvec, is, lvec, is, ctx, ierr) > > VecScatterBegin(ctx, gvec, lvec, INSERT_VALUES, SCATTER_FORWARD, ierr) > > VecScatterEnd? (ctx, gvec, lvec, INSERT_VALUES, SCATTER_FORWARD, ierr) > > > That works fine. > > Then I want to do the reverse, i.e. add all values from the local > vec to the global vec on comm_world to generate the average of the > results. > > I tried: > > VecSet(gvec, zero, ierr) > > VecScatterBegin(ctx, gvec, lvec, ADD_VALUES, SCATTER_REVERSE, ierr) > > > I was hoping to get the sum of svec in the global vec, so that > gvec / comm_size(sub_comm_id==0) gives the mean. > > However, I get the following error: > > > Nonconforming object sizes > > Vector wrong size 18 for scatter 6 (scatter reverse and vector to > != ctx from size) > > > Going with the same approach with VecScatterCreateToAll > > leads to the same issue. > > Do you have suggestions on how I could/should achieve it? > > > > You have to flip the lvec, gvec args in reverse mode (as per the man > pages states under Notes) > > Thanks > Dave > > > > Many thanks! > > Fabian > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 659 bytes Desc: OpenPGP digital signature URL: From ys453 at cam.ac.uk Fri Jun 5 09:17:52 2020 From: ys453 at cam.ac.uk (Y. Shidi) Date: Fri, 05 Jun 2020 15:17:52 +0100 Subject: [petsc-users] KSP not converge when turn on ksp_monitor_true_residual In-Reply-To: References: <9f3e24167df41df1bb9c0d8590ce8328@cam.ac.uk> <4858d4e12d908bd0e033e9ccb36e3848@cam.ac.uk> Message-ID: Thank you for your reply. I set the relative tolerance of 1e-10, below is the output with ksp_monitor_true_residual, step 1 (converged): 0 KSP unpreconditioned resid norm 5.853072011788e-01 true resid norm 5.853072011788e-01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 1.263467051263e-02 true resid norm 1.263467051263e-02 ||r(i)||/||b|| 2.158639170539e-02 2 KSP unpreconditioned resid norm 1.341051880494e-03 true resid norm 1.341051880494e-03 ||r(i)||/||b|| 2.291193202122e-03 3 KSP unpreconditioned resid norm 5.281711107599e-04 true resid norm 5.281711107599e-04 ||r(i)||/||b|| 9.023827311473e-04 4 KSP unpreconditioned resid norm 2.265161267248e-04 true resid norm 2.265161267248e-04 ||r(i)||/||b|| 3.870038268256e-04 5 KSP unpreconditioned resid norm 6.083108409953e-05 true resid norm 6.083108409954e-05 ||r(i)||/||b|| 1.039301822650e-04 6 KSP unpreconditioned resid norm 2.049038801905e-05 true resid norm 2.049038801906e-05 ||r(i)||/||b|| 3.500792058903e-05 7 KSP unpreconditioned resid norm 4.908575378578e-06 true resid norm 4.908575378575e-06 ||r(i)||/||b|| 8.386323231099e-06 8 KSP unpreconditioned resid norm 1.160371295964e-06 true resid norm 1.160371295960e-06 ||r(i)||/||b|| 1.982499606399e-06 9 KSP unpreconditioned resid norm 2.993349323170e-07 true resid norm 2.993349323154e-07 ||r(i)||/||b|| 5.114150854671e-07 10 KSP unpreconditioned resid norm 6.423860393220e-08 true resid norm 6.423860392241e-08 ||r(i)||/||b|| 1.097519452913e-07 11 KSP unpreconditioned resid norm 1.486343352971e-08 true resid norm 1.486343352542e-08 ||r(i)||/||b|| 2.539424339131e-08 12 KSP unpreconditioned resid norm 3.388518888745e-09 true resid norm 3.388518889406e-09 ||r(i)||/||b|| 5.789299845588e-09 13 KSP unpreconditioned resid norm 8.414456498118e-10 true resid norm 8.414456551004e-10 ||r(i)||/||b|| 1.437613706795e-09 14 KSP unpreconditioned resid norm 2.052670281766e-10 true resid norm 2.052670332352e-10 ||r(i)||/||b|| 3.506996545092e-10 15 KSP unpreconditioned resid norm 4.739342260680e-11 true resid norm 4.739342219912e-11 ||r(i)||/||b|| 8.097187614243e-11 Linear solve converged due to CONVERGED_RTOL iterations 15 KSP Object: 1 MPI processes type: fgmres restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement happy breakdown tolerance 1e-30 maximum iterations=10000, nonzero initial guess tolerances: relative=1e-10, absolute=1e-50, divergence=10000. right preconditioning using UNPRECONDITIONED norm type for convergence test PC Object: 1 MPI processes type: fieldsplit FieldSplit with Schur preconditioner, factorization FULL Preconditioner for the Schur complement formed from Sp, an assembled approximation to S, which uses A00's diagonal's inverse Split info: Split number 0 Defined by IS Split number 1 Defined by IS KSP solver for A00 block KSP Object: (fieldsplit_0_) 1 MPI processes type: preonly maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000. left preconditioning using NONE norm type for convergence test PC Object: (fieldsplit_0_) 1 MPI processes type: hypre HYPRE BoomerAMG preconditioning Cycle type V Maximum number of levels 25 Maximum number of iterations PER hypre call 1 Convergence tolerance PER hypre call 0. Threshold for strong coupling 0.25 Interpolation truncation factor 0. Interpolation: max elements per row 0 Number of levels of aggressive coarsening 0 Number of paths for aggressive coarsening 1 Maximum row sums 0.9 Sweeps down 1 Sweeps up 1 Sweeps on coarse 1 Relax down symmetric-SOR/Jacobi Relax up symmetric-SOR/Jacobi Relax on coarse Gaussian-elimination Relax weight (all) 1. Outer relax weight (all) 1. Using CF-relaxation Not using more complex smoothers. Measure type local Coarsen type Falgout Interpolation type classical linear system matrix = precond matrix: Mat Object: (fieldsplit_0_) 1 MPI processes type: seqaij rows=28578, cols=28578 total: nonzeros=653916, allocated nonzeros=653916 total number of mallocs used during MatSetValues calls =0 not using I-node routines KSP solver for S = A11 - A10 inv(A00) A01 KSP Object: (fieldsplit_1_) 1 MPI processes type: preonly maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000. left preconditioning using NONE norm type for convergence test PC Object: (fieldsplit_1_) 1 MPI processes type: hypre HYPRE BoomerAMG preconditioning Cycle type V Maximum number of levels 25 Maximum number of iterations PER hypre call 1 Convergence tolerance PER hypre call 0. Threshold for strong coupling 0.25 Interpolation truncation factor 0. Interpolation: max elements per row 0 Number of levels of aggressive coarsening 0 Number of paths for aggressive coarsening 1 Maximum row sums 0.9 Sweeps down 1 Sweeps up 1 Sweeps on coarse 1 Relax down symmetric-SOR/Jacobi Relax up symmetric-SOR/Jacobi Relax on coarse Gaussian-elimination Relax weight (all) 1. Outer relax weight (all) 1. Using CF-relaxation Not using more complex smoothers. Measure type local Coarsen type Falgout Interpolation type classical linear system matrix followed by preconditioner matrix: Mat Object: (fieldsplit_1_) 1 MPI processes type: schurcomplement rows=3874, cols=3874 Schur complement A11 - A10 inv(A00) A01 A11 Mat Object: (fieldsplit_1_) 1 MPI processes type: seqaij rows=3874, cols=3874 total: nonzeros=0, allocated nonzeros=0 total number of mallocs used during MatSetValues calls =0 using I-node routines: found 775 nodes, limit used is 5 A10 Mat Object: 1 MPI processes type: seqaij rows=3874, cols=28578 total: nonzeros=138180, allocated nonzeros=138180 total number of mallocs used during MatSetValues calls =0 not using I-node routines KSP of A00 KSP Object: (fieldsplit_0_) 1 MPI processes type: preonly maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000. left preconditioning using NONE norm type for convergence test PC Object: (fieldsplit_0_) 1 MPI processes type: hypre HYPRE BoomerAMG preconditioning Cycle type V Maximum number of levels 25 Maximum number of iterations PER hypre call 1 Convergence tolerance PER hypre call 0. Threshold for strong coupling 0.25 Interpolation truncation factor 0. Interpolation: max elements per row 0 Number of levels of aggressive coarsening 0 Number of paths for aggressive coarsening 1 Maximum row sums 0.9 Sweeps down 1 Sweeps up 1 Sweeps on coarse 1 Relax down symmetric-SOR/Jacobi Relax up symmetric-SOR/Jacobi Relax on coarse Gaussian-elimination Relax weight (all) 1. Outer relax weight (all) 1. Using CF-relaxation Not using more complex smoothers. Measure type local Coarsen type Falgout Interpolation type classical linear system matrix = precond matrix: Mat Object: (fieldsplit_0_) 1 MPI processes type: seqaij rows=28578, cols=28578 total: nonzeros=653916, allocated nonzeros=653916 total number of mallocs used during MatSetValues calls =0 not using I-node routines A01 Mat Object: 1 MPI processes type: seqaij rows=28578, cols=3874 total: nonzeros=138180, allocated nonzeros=138180 total number of mallocs used during MatSetValues calls =0 not using I-node routines Mat Object: 1 MPI processes type: seqaij rows=3874, cols=3874 total: nonzeros=77704, allocated nonzeros=77704 total number of mallocs used during MatSetValues calls =0 not using I-node routines linear system matrix = precond matrix: Mat Object: 1 MPI processes type: nest rows=32452, cols=32452 Matrix object: type=nest, rows=2, cols=2 MatNest structure: (0,0) : prefix="fieldsplit_0_", type=seqaij, rows=28578, cols=28578 (0,1) : type=seqaij, rows=28578, cols=3874 (1,0) : type=seqaij, rows=3874, cols=28578 (1,1) : prefix="fieldsplit_1_", type=seqaij, rows=3874, cols=3874 step 2 (NOT converged): Linear solve did not converge due to DIVERGED_NANORINF iterations 0 KSP Object: 1 MPI processes type: fgmres restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement happy breakdown tolerance 1e-30 maximum iterations=10000, nonzero initial guess tolerances: relative=1e-10, absolute=1e-50, divergence=10000. right preconditioning using UNPRECONDITIONED norm type for convergence test PC Object: 1 MPI processes type: fieldsplit FieldSplit with Schur preconditioner, factorization FULL Preconditioner for the Schur complement formed from Sp, an assembled approximation to S, which uses A00's diagonal's inverse Split info: Split number 0 Defined by IS Split number 1 Defined by IS KSP solver for A00 block KSP Object: (fieldsplit_0_) 1 MPI processes type: preonly maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000. left preconditioning using DEFAULT norm type for convergence test PC Object: (fieldsplit_0_) 1 MPI processes type: hypre PC has not been set up so information may be incomplete HYPRE BoomerAMG preconditioning Cycle type V Maximum number of levels 25 Maximum number of iterations PER hypre call 1 Convergence tolerance PER hypre call 0. Threshold for strong coupling 0.25 Interpolation truncation factor 0. Interpolation: max elements per row 0 Number of levels of aggressive coarsening 0 Number of paths for aggressive coarsening 1 Maximum row sums 0.9 Sweeps down 1 Sweeps up 1 Sweeps on coarse 1 Relax down symmetric-SOR/Jacobi Relax up symmetric-SOR/Jacobi Relax on coarse Gaussian-elimination Relax weight (all) 1. Outer relax weight (all) 1. Using CF-relaxation Not using more complex smoothers. Measure type local Coarsen type Falgout Interpolation type classical linear system matrix = precond matrix: Mat Object: (fieldsplit_0_) 1 MPI processes type: seqaij rows=28578, cols=28578 total: nonzeros=653916, allocated nonzeros=653916 total number of mallocs used during MatSetValues calls =0 not using I-node routines KSP solver for S = A11 - A10 inv(A00) A01 KSP Object: (fieldsplit_1_) 1 MPI processes type: preonly maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000. left preconditioning using DEFAULT norm type for convergence test PC Object: (fieldsplit_1_) 1 MPI processes type: hypre PC has not been set up so information may be incomplete HYPRE BoomerAMG preconditioning Cycle type V Maximum number of levels 25 Maximum number of iterations PER hypre call 1 Convergence tolerance PER hypre call 0. Threshold for strong coupling 0.25 Interpolation truncation factor 0. Interpolation: max elements per row 0 Number of levels of aggressive coarsening 0 Number of paths for aggressive coarsening 1 Maximum row sums 0.9 Sweeps down 1 Sweeps up 1 Sweeps on coarse 1 Relax down symmetric-SOR/Jacobi Relax up symmetric-SOR/Jacobi Relax on coarse Gaussian-elimination Relax weight (all) 1. Outer relax weight (all) 1. Using CF-relaxation Not using more complex smoothers. Measure type local Coarsen type Falgout Interpolation type classical linear system matrix followed by preconditioner matrix: Mat Object: (fieldsplit_1_) 1 MPI processes type: schurcomplement rows=3874, cols=3874 Schur complement A11 - A10 inv(A00) A01 A11 Mat Object: (fieldsplit_1_) 1 MPI processes type: seqaij rows=3874, cols=3874 total: nonzeros=0, allocated nonzeros=0 total number of mallocs used during MatSetValues calls =0 using I-node routines: found 775 nodes, limit used is 5 A10 Mat Object: 1 MPI processes type: seqaij rows=3874, cols=28578 total: nonzeros=138180, allocated nonzeros=138180 total number of mallocs used during MatSetValues calls =0 not using I-node routines KSP of A00 KSP Object: (fieldsplit_0_) 1 MPI processes type: preonly maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000. left preconditioning using DEFAULT norm type for convergence test PC Object: (fieldsplit_0_) 1 MPI processes type: hypre PC has not been set up so information may be incomplete HYPRE BoomerAMG preconditioning Cycle type V Maximum number of levels 25 Maximum number of iterations PER hypre call 1 Convergence tolerance PER hypre call 0. Threshold for strong coupling 0.25 Interpolation truncation factor 0. Interpolation: max elements per row 0 Number of levels of aggressive coarsening 0 Number of paths for aggressive coarsening 1 Maximum row sums 0.9 Sweeps down 1 Sweeps up 1 Sweeps on coarse 1 Relax down symmetric-SOR/Jacobi Relax up symmetric-SOR/Jacobi Relax on coarse Gaussian-elimination Relax weight (all) 1. Outer relax weight (all) 1. Using CF-relaxation Not using more complex smoothers. Measure type local Coarsen type Falgout Interpolation type classical linear system matrix = precond matrix: Mat Object: (fieldsplit_0_) 1 MPI processes type: seqaij rows=28578, cols=28578 total: nonzeros=653916, allocated nonzeros=653916 total number of mallocs used during MatSetValues calls =0 not using I-node routines A01 Mat Object: 1 MPI processes type: seqaij rows=28578, cols=3874 total: nonzeros=138180, allocated nonzeros=138180 total number of mallocs used during MatSetValues calls =0 not using I-node routines Mat Object: 1 MPI processes type: seqaij rows=3874, cols=3874 total: nonzeros=77704, allocated nonzeros=77704 total number of mallocs used during MatSetValues calls =0 not using I-node routines linear system matrix = precond matrix: Mat Object: 1 MPI processes type: nest rows=32452, cols=32452 Matrix object: type=nest, rows=2, cols=2 MatNest structure: (0,0) : prefix="fieldsplit_0_", type=seqaij, rows=28578, cols=28578 (0,1) : type=seqaij, rows=28578, cols=3874 (1,0) : type=seqaij, rows=3874, cols=28578 (1,1) : prefix="fieldsplit_1_", type=seqaij, rows=3874, cols=3874 HOWEVER, if I do not use -ksp_monitor_true_residual, both two steps converge. step 1: Linear solve converged due to CONVERGED_RTOL iterations 15 KSP Object: 1 MPI processes type: fgmres restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement happy breakdown tolerance 1e-30 maximum iterations=10000, nonzero initial guess tolerances: relative=1e-10, absolute=1e-50, divergence=10000. right preconditioning using UNPRECONDITIONED norm type for convergence test PC Object: 1 MPI processes type: fieldsplit FieldSplit with Schur preconditioner, factorization FULL Preconditioner for the Schur complement formed from Sp, an assembled approximation to S, which uses A00's diagonal's inverse Split info: Split number 0 Defined by IS Split number 1 Defined by IS KSP solver for A00 block KSP Object: (fieldsplit_0_) 1 MPI processes type: preonly maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000. left preconditioning using NONE norm type for convergence test PC Object: (fieldsplit_0_) 1 MPI processes type: hypre HYPRE BoomerAMG preconditioning Cycle type V Maximum number of levels 25 Maximum number of iterations PER hypre call 1 Convergence tolerance PER hypre call 0. Threshold for strong coupling 0.25 Interpolation truncation factor 0. Interpolation: max elements per row 0 Number of levels of aggressive coarsening 0 Number of paths for aggressive coarsening 1 Maximum row sums 0.9 Sweeps down 1 Sweeps up 1 Sweeps on coarse 1 Relax down symmetric-SOR/Jacobi Relax up symmetric-SOR/Jacobi Relax on coarse Gaussian-elimination Relax weight (all) 1. Outer relax weight (all) 1. Using CF-relaxation Not using more complex smoothers. Measure type local Coarsen type Falgout Interpolation type classical linear system matrix = precond matrix: Mat Object: (fieldsplit_0_) 1 MPI processes type: seqaij rows=28578, cols=28578 total: nonzeros=653916, allocated nonzeros=653916 total number of mallocs used during MatSetValues calls =0 not using I-node routines KSP solver for S = A11 - A10 inv(A00) A01 KSP Object: (fieldsplit_1_) 1 MPI processes type: preonly maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000. left preconditioning using NONE norm type for convergence test PC Object: (fieldsplit_1_) 1 MPI processes type: hypre HYPRE BoomerAMG preconditioning Cycle type V Maximum number of levels 25 Maximum number of iterations PER hypre call 1 Convergence tolerance PER hypre call 0. Threshold for strong coupling 0.25 Interpolation truncation factor 0. Interpolation: max elements per row 0 Number of levels of aggressive coarsening 0 Number of paths for aggressive coarsening 1 Maximum row sums 0.9 Sweeps down 1 Sweeps up 1 Sweeps on coarse 1 Relax down symmetric-SOR/Jacobi Relax up symmetric-SOR/Jacobi Relax on coarse Gaussian-elimination Relax weight (all) 1. Outer relax weight (all) 1. Using CF-relaxation Not using more complex smoothers. Measure type local Coarsen type Falgout Interpolation type classical linear system matrix followed by preconditioner matrix: Mat Object: (fieldsplit_1_) 1 MPI processes type: schurcomplement rows=3874, cols=3874 Schur complement A11 - A10 inv(A00) A01 A11 Mat Object: (fieldsplit_1_) 1 MPI processes type: seqaij rows=3874, cols=3874 total: nonzeros=0, allocated nonzeros=0 total number of mallocs used during MatSetValues calls =0 using I-node routines: found 775 nodes, limit used is 5 A10 Mat Object: 1 MPI processes type: seqaij rows=3874, cols=28578 total: nonzeros=138180, allocated nonzeros=138180 total number of mallocs used during MatSetValues calls =0 not using I-node routines KSP of A00 KSP Object: (fieldsplit_0_) 1 MPI processes type: preonly maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000. left preconditioning using NONE norm type for convergence test PC Object: (fieldsplit_0_) 1 MPI processes type: hypre HYPRE BoomerAMG preconditioning Cycle type V Maximum number of levels 25 Maximum number of iterations PER hypre call 1 Convergence tolerance PER hypre call 0. Threshold for strong coupling 0.25 Interpolation truncation factor 0. Interpolation: max elements per row 0 Number of levels of aggressive coarsening 0 Number of paths for aggressive coarsening 1 Maximum row sums 0.9 Sweeps down 1 Sweeps up 1 Sweeps on coarse 1 Relax down symmetric-SOR/Jacobi Relax up symmetric-SOR/Jacobi Relax on coarse Gaussian-elimination Relax weight (all) 1. Outer relax weight (all) 1. Using CF-relaxation Not using more complex smoothers. Measure type local Coarsen type Falgout Interpolation type classical linear system matrix = precond matrix: Mat Object: (fieldsplit_0_) 1 MPI processes type: seqaij rows=28578, cols=28578 total: nonzeros=653916, allocated nonzeros=653916 total number of mallocs used during MatSetValues calls =0 not using I-node routines A01 Mat Object: 1 MPI processes type: seqaij rows=28578, cols=3874 total: nonzeros=138180, allocated nonzeros=138180 total number of mallocs used during MatSetValues calls =0 not using I-node routines Mat Object: 1 MPI processes type: seqaij rows=3874, cols=3874 total: nonzeros=77704, allocated nonzeros=77704 total number of mallocs used during MatSetValues calls =0 not using I-node routines linear system matrix = precond matrix: Mat Object: 1 MPI processes type: nest rows=32452, cols=32452 Matrix object: type=nest, rows=2, cols=2 MatNest structure: (0,0) : prefix="fieldsplit_0_", type=seqaij, rows=28578, cols=28578 (0,1) : type=seqaij, rows=28578, cols=3874 (1,0) : type=seqaij, rows=3874, cols=28578 (1,1) : prefix="fieldsplit_1_", type=seqaij, rows=3874, cols=3874 step 2: Linear solve converged due to CONVERGED_RTOL iterations 1 KSP Object: 1 MPI processes type: fgmres restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement happy breakdown tolerance 1e-30 maximum iterations=10000, nonzero initial guess tolerances: relative=1e-10, absolute=1e-50, divergence=10000. right preconditioning using UNPRECONDITIONED norm type for convergence test PC Object: 1 MPI processes type: fieldsplit FieldSplit with Schur preconditioner, factorization FULL Preconditioner for the Schur complement formed from Sp, an assembled approximation to S, which uses A00's diagonal's inverse Split info: Split number 0 Defined by IS Split number 1 Defined by IS KSP solver for A00 block KSP Object: (fieldsplit_0_) 1 MPI processes type: preonly maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000. left preconditioning using NONE norm type for convergence test PC Object: (fieldsplit_0_) 1 MPI processes type: hypre HYPRE BoomerAMG preconditioning Cycle type V Maximum number of levels 25 Maximum number of iterations PER hypre call 1 Convergence tolerance PER hypre call 0. Threshold for strong coupling 0.25 Interpolation truncation factor 0. Interpolation: max elements per row 0 Number of levels of aggressive coarsening 0 Number of paths for aggressive coarsening 1 Maximum row sums 0.9 Sweeps down 1 Sweeps up 1 Sweeps on coarse 1 Relax down symmetric-SOR/Jacobi Relax up symmetric-SOR/Jacobi Relax on coarse Gaussian-elimination Relax weight (all) 1. Outer relax weight (all) 1. Using CF-relaxation Not using more complex smoothers. Measure type local Coarsen type Falgout Interpolation type classical linear system matrix = precond matrix: Mat Object: (fieldsplit_0_) 1 MPI processes type: seqaij rows=28578, cols=28578 total: nonzeros=653916, allocated nonzeros=653916 total number of mallocs used during MatSetValues calls =0 not using I-node routines KSP solver for S = A11 - A10 inv(A00) A01 KSP Object: (fieldsplit_1_) 1 MPI processes type: preonly maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000. left preconditioning using NONE norm type for convergence test PC Object: (fieldsplit_1_) 1 MPI processes type: hypre HYPRE BoomerAMG preconditioning Cycle type V Maximum number of levels 25 Maximum number of iterations PER hypre call 1 Convergence tolerance PER hypre call 0. Threshold for strong coupling 0.25 Interpolation truncation factor 0. Interpolation: max elements per row 0 Number of levels of aggressive coarsening 0 Number of paths for aggressive coarsening 1 Maximum row sums 0.9 Sweeps down 1 Sweeps up 1 Sweeps on coarse 1 Relax down symmetric-SOR/Jacobi Relax up symmetric-SOR/Jacobi Relax on coarse Gaussian-elimination Relax weight (all) 1. Outer relax weight (all) 1. Using CF-relaxation Not using more complex smoothers. Measure type local Coarsen type Falgout Interpolation type classical linear system matrix followed by preconditioner matrix: Mat Object: (fieldsplit_1_) 1 MPI processes type: schurcomplement rows=3874, cols=3874 Schur complement A11 - A10 inv(A00) A01 A11 Mat Object: (fieldsplit_1_) 1 MPI processes type: seqaij rows=3874, cols=3874 total: nonzeros=0, allocated nonzeros=0 total number of mallocs used during MatSetValues calls =0 using I-node routines: found 775 nodes, limit used is 5 A10 Mat Object: 1 MPI processes type: seqaij rows=3874, cols=28578 total: nonzeros=138180, allocated nonzeros=138180 total number of mallocs used during MatSetValues calls =0 not using I-node routines KSP of A00 KSP Object: (fieldsplit_0_) 1 MPI processes type: preonly maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000. left preconditioning using NONE norm type for convergence test PC Object: (fieldsplit_0_) 1 MPI processes type: hypre HYPRE BoomerAMG preconditioning Cycle type V Maximum number of levels 25 Maximum number of iterations PER hypre call 1 Convergence tolerance PER hypre call 0. Threshold for strong coupling 0.25 Interpolation truncation factor 0. Interpolation: max elements per row 0 Number of levels of aggressive coarsening 0 Number of paths for aggressive coarsening 1 Maximum row sums 0.9 Sweeps down 1 Sweeps up 1 Sweeps on coarse 1 Relax down symmetric-SOR/Jacobi Relax up symmetric-SOR/Jacobi Relax on coarse Gaussian-elimination Relax weight (all) 1. Outer relax weight (all) 1. Using CF-relaxation Not using more complex smoothers. Measure type local Coarsen type Falgout Interpolation type classical linear system matrix = precond matrix: Mat Object: (fieldsplit_0_) 1 MPI processes type: seqaij rows=28578, cols=28578 total: nonzeros=653916, allocated nonzeros=653916 total number of mallocs used during MatSetValues calls =0 not using I-node routines A01 Mat Object: 1 MPI processes type: seqaij rows=28578, cols=3874 total: nonzeros=138180, allocated nonzeros=138180 total number of mallocs used during MatSetValues calls =0 not using I-node routines Mat Object: 1 MPI processes type: seqaij rows=3874, cols=3874 total: nonzeros=77704, allocated nonzeros=77704 total number of mallocs used during MatSetValues calls =0 not using I-node routines linear system matrix = precond matrix: Mat Object: 1 MPI processes type: nest rows=32452, cols=32452 Matrix object: type=nest, rows=2, cols=2 MatNest structure: (0,0) : prefix="fieldsplit_0_", type=seqaij, rows=28578, cols=28578 (0,1) : type=seqaij, rows=28578, cols=3874 (1,0) : type=seqaij, rows=3874, cols=28578 (1,1) : prefix="fieldsplit_1_", type=seqaij, rows=3874, cols=3874 Thank you for your help and time. Kind regards, Shidi On 2020-06-05 13:40, Matthew Knepley wrote: > On Fri, Jun 5, 2020 at 7:42 AM Y. Shidi wrote: > >>> This algebraic tolerance is strange. Why put it below unit >> roundoff? >> >> We are testing a static bubble problem, if the tolerance is 1e-10 it >> won't approach static solution. > > I do not understand. Every computation in your problem makes errors of > order 1e-15, > since that is the roundoff. Moreover, you cannot expect a solution to > be more accurate > than (condition number * roundoff). I would be surprised if the > condition number is near 1, > since you would not need a solver. So what is going on? > > Thanks, > > Matt > >> Thanks, >> Shidi >> >> On 2020-06-05 11:48, Matthew Knepley wrote: >>> On Fri, Jun 5, 2020 at 6:12 AM Y. Shidi wrote: >>> >>>> Dear developers, >>>> >>>> We are using filed splitting method solve saddle point problem. >>>> Below is our ksp configuration: >>>> >>>> PetscOptionsSetValue(NULL,"-ksp_type","gmres"); >>>> PetscOptionsSetValue(NULL,"-ksp_initial_guess_nonzero",""); >>>> PetscOptionsSetValue(NULL,"-ksp_rtol","1e-20"); >>> >>> This algebraic tolerance is strange. Why put it below unit >> roundoff? >>> >>>> PetscOptionsSetValue(NULL,"-ksp_converged_reason", ""); >>>> PetscOptionsSetValue(NULL,"-pc_type", "fieldsplit"); >>>> PetscOptionsSetValue(NULL,"-pc_fieldsplit_type", "schur"); >>>> PetscOptionsSetValue(NULL,"-pc_fieldsplit_schur_fact_type", >>>> "lower"); >>>> PetscOptionsSetValue(NULL,"-pc_fieldsplit_schur_precondition", >>>> "selfp"); >>>> PetscOptionsSetValue(NULL,"-fieldsplit_0_ksp_type", "preonly"); >>>> PetscOptionsSetValue(NULL,"-fieldsplit_0_pc_type", "hypre"); >>>> PetscOptionsSetValue(NULL,"-fieldsplit_1_ksp_type", "preonly"); >>>> PetscOptionsSetValue(NULL,"-fieldsplit_1_pc_type", "hypre"); >>>> >>>> It can be converged. However, when we turn on the ksp monitor: >>>> >>>> PetscOptionsSetValue(NULL,"-ksp_monitor_true_residual", ""); >>>> >>>> It won't converge. It seems very odd, I am wondering why it >> happens. >>> >>> Can you try it with a relative tolerance of 1e-10, and send the >> output >>> (with -ksp_view as well). >>> >>> Thanks, >>> >>> Matt >>> >>>> Thank you for your time and help. >>>> >>>> Kind regards, >>>> Shidi >>> >>> -- >>> >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to >> which >>> their experiments lead. >>> -- Norbert Wiener >>> >>> https://www.cse.buffalo.edu/~knepley/ [1] >>> >>> >>> Links: >>> ------ >>> [1] http://www.cse.buffalo.edu/~knepley/ > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ [1] > > > Links: > ------ > [1] http://www.cse.buffalo.edu/~knepley/ From huq2090 at gmail.com Fri Jun 5 10:03:15 2020 From: huq2090 at gmail.com (Fazlul Huq) Date: Fri, 5 Jun 2020 10:03:15 -0500 Subject: [petsc-users] Norm_2 calculation Message-ID: Hello All, I'm trying to calculate the norm_2 error of a solution. Here, vector s: Analytical solution vector x: Numerical solution ierr = VecView(s,PETSC_VIEWER_STDOUT_WORLD);CHKERRQ(ierr); ierr = VecView(x,PETSC_VIEWER_STDOUT_WORLD);CHKERRQ(ierr); ierr = VecAXPY(x,-1.0,s);CHKERRQ(ierr); ierr = VecNorm(x,NORM_2,&norm);CHKERRQ(ierr); ierr = VecView(x,PETSC_VIEWER_STDOUT_WORLD);CHKERRQ(ierr); ierr = KSPGetIterationNumber(ksp,&its);CHKERRQ(ierr); if (norm > tol) { ierr = PetscPrintf(PETSC_COMM_WORLD,"Second Norm of error %g\n", double)norm);CHKERRQ(ierr); ierr = PetscPrintf(PETSC_COMM_WORLD,"Iterations %D\n",its);CHKERRQ(ierr); } Am I calculating the "Norm_2" error correctly or making any mistake? Unfortunately, for large sized matrix, say 10^6, I am getting very high value of "Norm_2" error. Thank you. Sincerely, Huq -- Fazlul Huq Graduate Research Assistant Department of Nuclear, Plasma & Radiological Engineering (NPRE) University of Illinois at Urbana-Champaign (UIUC) E-mail: huq2090 at gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Jun 5 10:13:32 2020 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 5 Jun 2020 11:13:32 -0400 Subject: [petsc-users] KSP not converge when turn on ksp_monitor_true_residual In-Reply-To: References: <9f3e24167df41df1bb9c0d8590ce8328@cam.ac.uk> <4858d4e12d908bd0e033e9ccb36e3848@cam.ac.uk> Message-ID: On Fri, Jun 5, 2020 at 10:17 AM Y. Shidi wrote: > Thank you for your reply. > I set the relative tolerance of 1e-10, below is the output with > ksp_monitor_true_residual, > > step 1 (converged): > > 0 KSP unpreconditioned resid norm 5.853072011788e-01 true resid norm > 5.853072011788e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP unpreconditioned resid norm 1.263467051263e-02 true resid norm > 1.263467051263e-02 ||r(i)||/||b|| 2.158639170539e-02 > 2 KSP unpreconditioned resid norm 1.341051880494e-03 true resid norm > 1.341051880494e-03 ||r(i)||/||b|| 2.291193202122e-03 > 3 KSP unpreconditioned resid norm 5.281711107599e-04 true resid norm > 5.281711107599e-04 ||r(i)||/||b|| 9.023827311473e-04 > 4 KSP unpreconditioned resid norm 2.265161267248e-04 true resid norm > 2.265161267248e-04 ||r(i)||/||b|| 3.870038268256e-04 > 5 KSP unpreconditioned resid norm 6.083108409953e-05 true resid norm > 6.083108409954e-05 ||r(i)||/||b|| 1.039301822650e-04 > 6 KSP unpreconditioned resid norm 2.049038801905e-05 true resid norm > 2.049038801906e-05 ||r(i)||/||b|| 3.500792058903e-05 > 7 KSP unpreconditioned resid norm 4.908575378578e-06 true resid norm > 4.908575378575e-06 ||r(i)||/||b|| 8.386323231099e-06 > 8 KSP unpreconditioned resid norm 1.160371295964e-06 true resid norm > 1.160371295960e-06 ||r(i)||/||b|| 1.982499606399e-06 > 9 KSP unpreconditioned resid norm 2.993349323170e-07 true resid norm > 2.993349323154e-07 ||r(i)||/||b|| 5.114150854671e-07 > 10 KSP unpreconditioned resid norm 6.423860393220e-08 true resid norm > 6.423860392241e-08 ||r(i)||/||b|| 1.097519452913e-07 > 11 KSP unpreconditioned resid norm 1.486343352971e-08 true resid norm > 1.486343352542e-08 ||r(i)||/||b|| 2.539424339131e-08 > 12 KSP unpreconditioned resid norm 3.388518888745e-09 true resid norm > 3.388518889406e-09 ||r(i)||/||b|| 5.789299845588e-09 > 13 KSP unpreconditioned resid norm 8.414456498118e-10 true resid norm > 8.414456551004e-10 ||r(i)||/||b|| 1.437613706795e-09 > 14 KSP unpreconditioned resid norm 2.052670281766e-10 true resid norm > 2.052670332352e-10 ||r(i)||/||b|| 3.506996545092e-10 > 15 KSP unpreconditioned resid norm 4.739342260680e-11 true resid norm > 4.739342219912e-11 ||r(i)||/||b|| 8.097187614243e-11 > Linear solve converged due to CONVERGED_RTOL iterations 15 > KSP Object: 1 MPI processes > type: fgmres > restart=30, using Classical (unmodified) Gram-Schmidt > Orthogonalization with no iterative refinement > happy breakdown tolerance 1e-30 > maximum iterations=10000, nonzero initial guess > tolerances: relative=1e-10, absolute=1e-50, divergence=10000. > right preconditioning > using UNPRECONDITIONED norm type for convergence test > PC Object: 1 MPI processes > type: fieldsplit > FieldSplit with Schur preconditioner, factorization FULL > Preconditioner for the Schur complement formed from Sp, an assembled > approximation to S, which uses A00's diagonal's inverse > Split info: > Split number 0 Defined by IS > Split number 1 Defined by IS > KSP solver for A00 block > KSP Object: (fieldsplit_0_) 1 MPI processes > type: preonly > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > left preconditioning > using NONE norm type for convergence test > PC Object: (fieldsplit_0_) 1 MPI processes > type: hypre > HYPRE BoomerAMG preconditioning > Cycle type V > Maximum number of levels 25 > Maximum number of iterations PER hypre call 1 > Convergence tolerance PER hypre call 0. > Threshold for strong coupling 0.25 > Interpolation truncation factor 0. > Interpolation: max elements per row 0 > Number of levels of aggressive coarsening 0 > Number of paths for aggressive coarsening 1 > Maximum row sums 0.9 > Sweeps down 1 > Sweeps up 1 > Sweeps on coarse 1 > Relax down symmetric-SOR/Jacobi > Relax up symmetric-SOR/Jacobi > Relax on coarse Gaussian-elimination > Relax weight (all) 1. > Outer relax weight (all) 1. > Using CF-relaxation > Not using more complex smoothers. > Measure type local > Coarsen type Falgout > Interpolation type classical > linear system matrix = precond matrix: > Mat Object: (fieldsplit_0_) 1 MPI processes > type: seqaij > rows=28578, cols=28578 > total: nonzeros=653916, allocated nonzeros=653916 > total number of mallocs used during MatSetValues calls =0 > not using I-node routines > KSP solver for S = A11 - A10 inv(A00) A01 > KSP Object: (fieldsplit_1_) 1 MPI processes > type: preonly > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > left preconditioning > using NONE norm type for convergence test > PC Object: (fieldsplit_1_) 1 MPI processes > type: hypre > HYPRE BoomerAMG preconditioning > Cycle type V > Maximum number of levels 25 > Maximum number of iterations PER hypre call 1 > Convergence tolerance PER hypre call 0. > Threshold for strong coupling 0.25 > Interpolation truncation factor 0. > Interpolation: max elements per row 0 > Number of levels of aggressive coarsening 0 > Number of paths for aggressive coarsening 1 > Maximum row sums 0.9 > Sweeps down 1 > Sweeps up 1 > Sweeps on coarse 1 > Relax down symmetric-SOR/Jacobi > Relax up symmetric-SOR/Jacobi > Relax on coarse Gaussian-elimination > Relax weight (all) 1. > Outer relax weight (all) 1. > Using CF-relaxation > Not using more complex smoothers. > Measure type local > Coarsen type Falgout > Interpolation type classical > linear system matrix followed by preconditioner matrix: > Mat Object: (fieldsplit_1_) 1 MPI processes > type: schurcomplement > rows=3874, cols=3874 > Schur complement A11 - A10 inv(A00) A01 > A11 > Mat Object: (fieldsplit_1_) 1 MPI processes > type: seqaij > rows=3874, cols=3874 > total: nonzeros=0, allocated nonzeros=0 > total number of mallocs used during MatSetValues calls > =0 > using I-node routines: found 775 nodes, limit used is > 5 > A10 > Mat Object: 1 MPI processes > type: seqaij > rows=3874, cols=28578 > total: nonzeros=138180, allocated nonzeros=138180 > total number of mallocs used during MatSetValues calls > =0 > not using I-node routines > KSP of A00 > KSP Object: (fieldsplit_0_) 1 MPI processes > type: preonly > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, > divergence=10000. > left preconditioning > using NONE norm type for convergence test > PC Object: (fieldsplit_0_) 1 MPI processes > type: hypre > HYPRE BoomerAMG preconditioning > Cycle type V > Maximum number of levels 25 > Maximum number of iterations PER hypre call 1 > Convergence tolerance PER hypre call 0. > Threshold for strong coupling 0.25 > Interpolation truncation factor 0. > Interpolation: max elements per row 0 > Number of levels of aggressive coarsening 0 > Number of paths for aggressive coarsening 1 > Maximum row sums 0.9 > Sweeps down 1 > Sweeps up 1 > Sweeps on coarse 1 > Relax down symmetric-SOR/Jacobi > Relax up symmetric-SOR/Jacobi > Relax on coarse Gaussian-elimination > Relax weight (all) 1. > Outer relax weight (all) 1. > Using CF-relaxation > Not using more complex smoothers. > Measure type local > Coarsen type Falgout > Interpolation type classical > linear system matrix = precond matrix: > Mat Object: (fieldsplit_0_) 1 MPI processes > type: seqaij > rows=28578, cols=28578 > total: nonzeros=653916, allocated nonzeros=653916 > total number of mallocs used during MatSetValues calls > =0 > not using I-node routines > A01 > Mat Object: 1 MPI processes > type: seqaij > rows=28578, cols=3874 > total: nonzeros=138180, allocated nonzeros=138180 > total number of mallocs used during MatSetValues calls > =0 > not using I-node routines > Mat Object: 1 MPI processes > type: seqaij > rows=3874, cols=3874 > total: nonzeros=77704, allocated nonzeros=77704 > total number of mallocs used during MatSetValues calls =0 > not using I-node routines > linear system matrix = precond matrix: > Mat Object: 1 MPI processes > type: nest > rows=32452, cols=32452 > Matrix object: > type=nest, rows=2, cols=2 > MatNest structure: > (0,0) : prefix="fieldsplit_0_", type=seqaij, rows=28578, > cols=28578 > (0,1) : type=seqaij, rows=28578, cols=3874 > (1,0) : type=seqaij, rows=3874, cols=28578 > (1,1) : prefix="fieldsplit_1_", type=seqaij, rows=3874, > cols=3874 > > step 2 (NOT converged): > > Linear solve did not converge due to DIVERGED_NANORINF iterations 0 > It looks like your matrix is invalid. Thanks, Matt > KSP Object: 1 MPI processes > type: fgmres > restart=30, using Classical (unmodified) Gram-Schmidt > Orthogonalization with no iterative refinement > happy breakdown tolerance 1e-30 > maximum iterations=10000, nonzero initial guess > tolerances: relative=1e-10, absolute=1e-50, divergence=10000. > right preconditioning > using UNPRECONDITIONED norm type for convergence test > PC Object: 1 MPI processes > type: fieldsplit > FieldSplit with Schur preconditioner, factorization FULL > Preconditioner for the Schur complement formed from Sp, an assembled > approximation to S, which uses A00's diagonal's inverse > Split info: > Split number 0 Defined by IS > Split number 1 Defined by IS > KSP solver for A00 block > KSP Object: (fieldsplit_0_) 1 MPI processes > type: preonly > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > left preconditioning > using DEFAULT norm type for convergence test > PC Object: (fieldsplit_0_) 1 MPI processes > type: hypre > PC has not been set up so information may be incomplete > HYPRE BoomerAMG preconditioning > Cycle type V > Maximum number of levels 25 > Maximum number of iterations PER hypre call 1 > Convergence tolerance PER hypre call 0. > Threshold for strong coupling 0.25 > Interpolation truncation factor 0. > Interpolation: max elements per row 0 > Number of levels of aggressive coarsening 0 > Number of paths for aggressive coarsening 1 > Maximum row sums 0.9 > Sweeps down 1 > Sweeps up 1 > Sweeps on coarse 1 > Relax down symmetric-SOR/Jacobi > Relax up symmetric-SOR/Jacobi > Relax on coarse Gaussian-elimination > Relax weight (all) 1. > Outer relax weight (all) 1. > Using CF-relaxation > Not using more complex smoothers. > Measure type local > Coarsen type Falgout > Interpolation type classical > linear system matrix = precond matrix: > Mat Object: (fieldsplit_0_) 1 MPI processes > type: seqaij > rows=28578, cols=28578 > total: nonzeros=653916, allocated nonzeros=653916 > total number of mallocs used during MatSetValues calls =0 > not using I-node routines > KSP solver for S = A11 - A10 inv(A00) A01 > KSP Object: (fieldsplit_1_) 1 MPI processes > type: preonly > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > left preconditioning > using DEFAULT norm type for convergence test > PC Object: (fieldsplit_1_) 1 MPI processes > type: hypre > PC has not been set up so information may be incomplete > HYPRE BoomerAMG preconditioning > Cycle type V > Maximum number of levels 25 > Maximum number of iterations PER hypre call 1 > Convergence tolerance PER hypre call 0. > Threshold for strong coupling 0.25 > Interpolation truncation factor 0. > Interpolation: max elements per row 0 > Number of levels of aggressive coarsening 0 > Number of paths for aggressive coarsening 1 > Maximum row sums 0.9 > Sweeps down 1 > Sweeps up 1 > Sweeps on coarse 1 > Relax down symmetric-SOR/Jacobi > Relax up symmetric-SOR/Jacobi > Relax on coarse Gaussian-elimination > Relax weight (all) 1. > Outer relax weight (all) 1. > Using CF-relaxation > Not using more complex smoothers. > Measure type local > Coarsen type Falgout > Interpolation type classical > linear system matrix followed by preconditioner matrix: > Mat Object: (fieldsplit_1_) 1 MPI processes > type: schurcomplement > rows=3874, cols=3874 > Schur complement A11 - A10 inv(A00) A01 > A11 > Mat Object: (fieldsplit_1_) 1 MPI processes > type: seqaij > rows=3874, cols=3874 > total: nonzeros=0, allocated nonzeros=0 > total number of mallocs used during MatSetValues calls > =0 > using I-node routines: found 775 nodes, limit used is > 5 > A10 > Mat Object: 1 MPI processes > type: seqaij > rows=3874, cols=28578 > total: nonzeros=138180, allocated nonzeros=138180 > total number of mallocs used during MatSetValues calls > =0 > not using I-node routines > KSP of A00 > KSP Object: (fieldsplit_0_) 1 MPI processes > type: preonly > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, > divergence=10000. > left preconditioning > using DEFAULT norm type for convergence test > PC Object: (fieldsplit_0_) 1 MPI processes > type: hypre > PC has not been set up so information may be incomplete > HYPRE BoomerAMG preconditioning > Cycle type V > Maximum number of levels 25 > Maximum number of iterations PER hypre call 1 > Convergence tolerance PER hypre call 0. > Threshold for strong coupling 0.25 > Interpolation truncation factor 0. > Interpolation: max elements per row 0 > Number of levels of aggressive coarsening 0 > Number of paths for aggressive coarsening 1 > Maximum row sums 0.9 > Sweeps down 1 > Sweeps up 1 > Sweeps on coarse 1 > Relax down symmetric-SOR/Jacobi > Relax up symmetric-SOR/Jacobi > Relax on coarse Gaussian-elimination > Relax weight (all) 1. > Outer relax weight (all) 1. > Using CF-relaxation > Not using more complex smoothers. > Measure type local > Coarsen type Falgout > Interpolation type classical > linear system matrix = precond matrix: > Mat Object: (fieldsplit_0_) 1 MPI processes > type: seqaij > rows=28578, cols=28578 > total: nonzeros=653916, allocated nonzeros=653916 > total number of mallocs used during MatSetValues calls > =0 > not using I-node routines > A01 > Mat Object: 1 MPI processes > type: seqaij > rows=28578, cols=3874 > total: nonzeros=138180, allocated nonzeros=138180 > total number of mallocs used during MatSetValues calls > =0 > not using I-node routines > Mat Object: 1 MPI processes > type: seqaij > rows=3874, cols=3874 > total: nonzeros=77704, allocated nonzeros=77704 > total number of mallocs used during MatSetValues calls =0 > not using I-node routines > linear system matrix = precond matrix: > Mat Object: 1 MPI processes > type: nest > rows=32452, cols=32452 > Matrix object: > type=nest, rows=2, cols=2 > MatNest structure: > (0,0) : prefix="fieldsplit_0_", type=seqaij, rows=28578, > cols=28578 > (0,1) : type=seqaij, rows=28578, cols=3874 > (1,0) : type=seqaij, rows=3874, cols=28578 > (1,1) : prefix="fieldsplit_1_", type=seqaij, rows=3874, > cols=3874 > > HOWEVER, if I do not use -ksp_monitor_true_residual, both two > steps converge. > > step 1: > Linear solve converged due to CONVERGED_RTOL iterations 15 > KSP Object: 1 MPI processes > type: fgmres > restart=30, using Classical (unmodified) Gram-Schmidt > Orthogonalization with no iterative refinement > happy breakdown tolerance 1e-30 > maximum iterations=10000, nonzero initial guess > tolerances: relative=1e-10, absolute=1e-50, divergence=10000. > right preconditioning > using UNPRECONDITIONED norm type for convergence test > PC Object: 1 MPI processes > type: fieldsplit > FieldSplit with Schur preconditioner, factorization FULL > Preconditioner for the Schur complement formed from Sp, an assembled > approximation to S, which uses A00's diagonal's inverse > Split info: > Split number 0 Defined by IS > Split number 1 Defined by IS > KSP solver for A00 block > KSP Object: (fieldsplit_0_) 1 MPI processes > type: preonly > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > left preconditioning > using NONE norm type for convergence test > PC Object: (fieldsplit_0_) 1 MPI processes > type: hypre > HYPRE BoomerAMG preconditioning > Cycle type V > Maximum number of levels 25 > Maximum number of iterations PER hypre call 1 > Convergence tolerance PER hypre call 0. > Threshold for strong coupling 0.25 > Interpolation truncation factor 0. > Interpolation: max elements per row 0 > Number of levels of aggressive coarsening 0 > Number of paths for aggressive coarsening 1 > Maximum row sums 0.9 > Sweeps down 1 > Sweeps up 1 > Sweeps on coarse 1 > Relax down symmetric-SOR/Jacobi > Relax up symmetric-SOR/Jacobi > Relax on coarse Gaussian-elimination > Relax weight (all) 1. > Outer relax weight (all) 1. > Using CF-relaxation > Not using more complex smoothers. > Measure type local > Coarsen type Falgout > Interpolation type classical > linear system matrix = precond matrix: > Mat Object: (fieldsplit_0_) 1 MPI processes > type: seqaij > rows=28578, cols=28578 > total: nonzeros=653916, allocated nonzeros=653916 > total number of mallocs used during MatSetValues calls =0 > not using I-node routines > KSP solver for S = A11 - A10 inv(A00) A01 > KSP Object: (fieldsplit_1_) 1 MPI processes > type: preonly > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > left preconditioning > using NONE norm type for convergence test > PC Object: (fieldsplit_1_) 1 MPI processes > type: hypre > HYPRE BoomerAMG preconditioning > Cycle type V > Maximum number of levels 25 > Maximum number of iterations PER hypre call 1 > Convergence tolerance PER hypre call 0. > Threshold for strong coupling 0.25 > Interpolation truncation factor 0. > Interpolation: max elements per row 0 > Number of levels of aggressive coarsening 0 > Number of paths for aggressive coarsening 1 > Maximum row sums 0.9 > Sweeps down 1 > Sweeps up 1 > Sweeps on coarse 1 > Relax down symmetric-SOR/Jacobi > Relax up symmetric-SOR/Jacobi > Relax on coarse Gaussian-elimination > Relax weight (all) 1. > Outer relax weight (all) 1. > Using CF-relaxation > Not using more complex smoothers. > Measure type local > Coarsen type Falgout > Interpolation type classical > linear system matrix followed by preconditioner matrix: > Mat Object: (fieldsplit_1_) 1 MPI processes > type: schurcomplement > rows=3874, cols=3874 > Schur complement A11 - A10 inv(A00) A01 > A11 > Mat Object: (fieldsplit_1_) 1 MPI processes > type: seqaij > rows=3874, cols=3874 > total: nonzeros=0, allocated nonzeros=0 > total number of mallocs used during MatSetValues calls > =0 > using I-node routines: found 775 nodes, limit used is > 5 > A10 > Mat Object: 1 MPI processes > type: seqaij > rows=3874, cols=28578 > total: nonzeros=138180, allocated nonzeros=138180 > total number of mallocs used during MatSetValues calls > =0 > not using I-node routines > KSP of A00 > KSP Object: (fieldsplit_0_) 1 MPI processes > type: preonly > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, > divergence=10000. > left preconditioning > using NONE norm type for convergence test > PC Object: (fieldsplit_0_) 1 MPI processes > type: hypre > HYPRE BoomerAMG preconditioning > Cycle type V > Maximum number of levels 25 > Maximum number of iterations PER hypre call 1 > Convergence tolerance PER hypre call 0. > Threshold for strong coupling 0.25 > Interpolation truncation factor 0. > Interpolation: max elements per row 0 > Number of levels of aggressive coarsening 0 > Number of paths for aggressive coarsening 1 > Maximum row sums 0.9 > Sweeps down 1 > Sweeps up 1 > Sweeps on coarse 1 > Relax down symmetric-SOR/Jacobi > Relax up symmetric-SOR/Jacobi > Relax on coarse Gaussian-elimination > Relax weight (all) 1. > Outer relax weight (all) 1. > Using CF-relaxation > Not using more complex smoothers. > Measure type local > Coarsen type Falgout > Interpolation type classical > linear system matrix = precond matrix: > Mat Object: (fieldsplit_0_) 1 MPI processes > type: seqaij > rows=28578, cols=28578 > total: nonzeros=653916, allocated nonzeros=653916 > total number of mallocs used during MatSetValues calls > =0 > not using I-node routines > A01 > Mat Object: 1 MPI processes > type: seqaij > rows=28578, cols=3874 > total: nonzeros=138180, allocated nonzeros=138180 > total number of mallocs used during MatSetValues calls > =0 > not using I-node routines > Mat Object: 1 MPI processes > type: seqaij > rows=3874, cols=3874 > total: nonzeros=77704, allocated nonzeros=77704 > total number of mallocs used during MatSetValues calls =0 > not using I-node routines > linear system matrix = precond matrix: > Mat Object: 1 MPI processes > type: nest > rows=32452, cols=32452 > Matrix object: > type=nest, rows=2, cols=2 > MatNest structure: > (0,0) : prefix="fieldsplit_0_", type=seqaij, rows=28578, > cols=28578 > (0,1) : type=seqaij, rows=28578, cols=3874 > (1,0) : type=seqaij, rows=3874, cols=28578 > (1,1) : prefix="fieldsplit_1_", type=seqaij, rows=3874, > cols=3874 > > step 2: > > Linear solve converged due to CONVERGED_RTOL iterations 1 > KSP Object: 1 MPI processes > type: fgmres > restart=30, using Classical (unmodified) Gram-Schmidt > Orthogonalization with no iterative refinement > happy breakdown tolerance 1e-30 > maximum iterations=10000, nonzero initial guess > tolerances: relative=1e-10, absolute=1e-50, divergence=10000. > right preconditioning > using UNPRECONDITIONED norm type for convergence test > PC Object: 1 MPI processes > type: fieldsplit > FieldSplit with Schur preconditioner, factorization FULL > Preconditioner for the Schur complement formed from Sp, an assembled > approximation to S, which uses A00's diagonal's inverse > Split info: > Split number 0 Defined by IS > Split number 1 Defined by IS > KSP solver for A00 block > KSP Object: (fieldsplit_0_) 1 MPI processes > type: preonly > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > left preconditioning > using NONE norm type for convergence test > PC Object: (fieldsplit_0_) 1 MPI processes > type: hypre > HYPRE BoomerAMG preconditioning > Cycle type V > Maximum number of levels 25 > Maximum number of iterations PER hypre call 1 > Convergence tolerance PER hypre call 0. > Threshold for strong coupling 0.25 > Interpolation truncation factor 0. > Interpolation: max elements per row 0 > Number of levels of aggressive coarsening 0 > Number of paths for aggressive coarsening 1 > Maximum row sums 0.9 > Sweeps down 1 > Sweeps up 1 > Sweeps on coarse 1 > Relax down symmetric-SOR/Jacobi > Relax up symmetric-SOR/Jacobi > Relax on coarse Gaussian-elimination > Relax weight (all) 1. > Outer relax weight (all) 1. > Using CF-relaxation > Not using more complex smoothers. > Measure type local > Coarsen type Falgout > Interpolation type classical > linear system matrix = precond matrix: > Mat Object: (fieldsplit_0_) 1 MPI processes > type: seqaij > rows=28578, cols=28578 > total: nonzeros=653916, allocated nonzeros=653916 > total number of mallocs used during MatSetValues calls =0 > not using I-node routines > KSP solver for S = A11 - A10 inv(A00) A01 > KSP Object: (fieldsplit_1_) 1 MPI processes > type: preonly > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > left preconditioning > using NONE norm type for convergence test > PC Object: (fieldsplit_1_) 1 MPI processes > type: hypre > HYPRE BoomerAMG preconditioning > Cycle type V > Maximum number of levels 25 > Maximum number of iterations PER hypre call 1 > Convergence tolerance PER hypre call 0. > Threshold for strong coupling 0.25 > Interpolation truncation factor 0. > Interpolation: max elements per row 0 > Number of levels of aggressive coarsening 0 > Number of paths for aggressive coarsening 1 > Maximum row sums 0.9 > Sweeps down 1 > Sweeps up 1 > Sweeps on coarse 1 > Relax down symmetric-SOR/Jacobi > Relax up symmetric-SOR/Jacobi > Relax on coarse Gaussian-elimination > Relax weight (all) 1. > Outer relax weight (all) 1. > Using CF-relaxation > Not using more complex smoothers. > Measure type local > Coarsen type Falgout > Interpolation type classical > linear system matrix followed by preconditioner matrix: > Mat Object: (fieldsplit_1_) 1 MPI processes > type: schurcomplement > rows=3874, cols=3874 > Schur complement A11 - A10 inv(A00) A01 > A11 > Mat Object: (fieldsplit_1_) 1 MPI processes > type: seqaij > rows=3874, cols=3874 > total: nonzeros=0, allocated nonzeros=0 > total number of mallocs used during MatSetValues calls > =0 > using I-node routines: found 775 nodes, limit used is > 5 > A10 > Mat Object: 1 MPI processes > type: seqaij > rows=3874, cols=28578 > total: nonzeros=138180, allocated nonzeros=138180 > total number of mallocs used during MatSetValues calls > =0 > not using I-node routines > KSP of A00 > KSP Object: (fieldsplit_0_) 1 MPI processes > type: preonly > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, > divergence=10000. > left preconditioning > using NONE norm type for convergence test > PC Object: (fieldsplit_0_) 1 MPI processes > type: hypre > HYPRE BoomerAMG preconditioning > Cycle type V > Maximum number of levels 25 > Maximum number of iterations PER hypre call 1 > Convergence tolerance PER hypre call 0. > Threshold for strong coupling 0.25 > Interpolation truncation factor 0. > Interpolation: max elements per row 0 > Number of levels of aggressive coarsening 0 > Number of paths for aggressive coarsening 1 > Maximum row sums 0.9 > Sweeps down 1 > Sweeps up 1 > Sweeps on coarse 1 > Relax down symmetric-SOR/Jacobi > Relax up symmetric-SOR/Jacobi > Relax on coarse Gaussian-elimination > Relax weight (all) 1. > Outer relax weight (all) 1. > Using CF-relaxation > Not using more complex smoothers. > Measure type local > Coarsen type Falgout > Interpolation type classical > linear system matrix = precond matrix: > Mat Object: (fieldsplit_0_) 1 MPI processes > type: seqaij > rows=28578, cols=28578 > total: nonzeros=653916, allocated nonzeros=653916 > total number of mallocs used during MatSetValues calls > =0 > not using I-node routines > A01 > Mat Object: 1 MPI processes > type: seqaij > rows=28578, cols=3874 > total: nonzeros=138180, allocated nonzeros=138180 > total number of mallocs used during MatSetValues calls > =0 > not using I-node routines > Mat Object: 1 MPI processes > type: seqaij > rows=3874, cols=3874 > total: nonzeros=77704, allocated nonzeros=77704 > total number of mallocs used during MatSetValues calls =0 > not using I-node routines > linear system matrix = precond matrix: > Mat Object: 1 MPI processes > type: nest > rows=32452, cols=32452 > Matrix object: > type=nest, rows=2, cols=2 > MatNest structure: > (0,0) : prefix="fieldsplit_0_", type=seqaij, rows=28578, > cols=28578 > (0,1) : type=seqaij, rows=28578, cols=3874 > (1,0) : type=seqaij, rows=3874, cols=28578 > (1,1) : prefix="fieldsplit_1_", type=seqaij, rows=3874, > cols=3874 > > Thank you for your help and time. > > Kind regards, > Shidi > > On 2020-06-05 13:40, Matthew Knepley wrote: > > On Fri, Jun 5, 2020 at 7:42 AM Y. Shidi wrote: > > > >>> This algebraic tolerance is strange. Why put it below unit > >> roundoff? > >> > >> We are testing a static bubble problem, if the tolerance is 1e-10 it > >> won't approach static solution. > > > > I do not understand. Every computation in your problem makes errors of > > order 1e-15, > > since that is the roundoff. Moreover, you cannot expect a solution to > > be more accurate > > than (condition number * roundoff). I would be surprised if the > > condition number is near 1, > > since you would not need a solver. So what is going on? > > > > Thanks, > > > > Matt > > > >> Thanks, > >> Shidi > >> > >> On 2020-06-05 11:48, Matthew Knepley wrote: > >>> On Fri, Jun 5, 2020 at 6:12 AM Y. Shidi wrote: > >>> > >>>> Dear developers, > >>>> > >>>> We are using filed splitting method solve saddle point problem. > >>>> Below is our ksp configuration: > >>>> > >>>> PetscOptionsSetValue(NULL,"-ksp_type","gmres"); > >>>> PetscOptionsSetValue(NULL,"-ksp_initial_guess_nonzero",""); > >>>> PetscOptionsSetValue(NULL,"-ksp_rtol","1e-20"); > >>> > >>> This algebraic tolerance is strange. Why put it below unit > >> roundoff? > >>> > >>>> PetscOptionsSetValue(NULL,"-ksp_converged_reason", ""); > >>>> PetscOptionsSetValue(NULL,"-pc_type", "fieldsplit"); > >>>> PetscOptionsSetValue(NULL,"-pc_fieldsplit_type", "schur"); > >>>> PetscOptionsSetValue(NULL,"-pc_fieldsplit_schur_fact_type", > >>>> "lower"); > >>>> PetscOptionsSetValue(NULL,"-pc_fieldsplit_schur_precondition", > >>>> "selfp"); > >>>> PetscOptionsSetValue(NULL,"-fieldsplit_0_ksp_type", "preonly"); > >>>> PetscOptionsSetValue(NULL,"-fieldsplit_0_pc_type", "hypre"); > >>>> PetscOptionsSetValue(NULL,"-fieldsplit_1_ksp_type", "preonly"); > >>>> PetscOptionsSetValue(NULL,"-fieldsplit_1_pc_type", "hypre"); > >>>> > >>>> It can be converged. However, when we turn on the ksp monitor: > >>>> > >>>> PetscOptionsSetValue(NULL,"-ksp_monitor_true_residual", ""); > >>>> > >>>> It won't converge. It seems very odd, I am wondering why it > >> happens. > >>> > >>> Can you try it with a relative tolerance of 1e-10, and send the > >> output > >>> (with -ksp_view as well). > >>> > >>> Thanks, > >>> > >>> Matt > >>> > >>>> Thank you for your time and help. > >>>> > >>>> Kind regards, > >>>> Shidi > >>> > >>> -- > >>> > >>> What most experimenters take for granted before they begin their > >>> experiments is infinitely more interesting than any results to > >> which > >>> their experiments lead. > >>> -- Norbert Wiener > >>> > >>> https://www.cse.buffalo.edu/~knepley/ [1] > >>> > >>> > >>> Links: > >>> ------ > >>> [1] http://www.cse.buffalo.edu/~knepley/ > > > > -- > > > > What most experimenters take for granted before they begin their > > experiments is infinitely more interesting than any results to which > > their experiments lead. > > -- Norbert Wiener > > > > https://www.cse.buffalo.edu/~knepley/ [1] > > > > > > Links: > > ------ > > [1] http://www.cse.buffalo.edu/~knepley/ > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Jun 5 10:16:06 2020 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 5 Jun 2020 11:16:06 -0400 Subject: [petsc-users] Norm_2 calculation In-Reply-To: References: Message-ID: On Fri, Jun 5, 2020 at 11:04 AM Fazlul Huq wrote: > Hello All, > I'm trying to calculate the norm_2 error of a solution. > Here, vector s: Analytical solution > vector x: Numerical solution > > ierr = VecView(s,PETSC_VIEWER_STDOUT_WORLD);CHKERRQ(ierr); > ierr = VecView(x,PETSC_VIEWER_STDOUT_WORLD);CHKERRQ(ierr); > > ierr = VecAXPY(x,-1.0,s);CHKERRQ(ierr); > ierr = VecNorm(x,NORM_2,&norm);CHKERRQ(ierr); > ierr = VecView(x,PETSC_VIEWER_STDOUT_WORLD);CHKERRQ(ierr); > ierr = KSPGetIterationNumber(ksp,&its);CHKERRQ(ierr); > if (norm > tol) { > ierr = PetscPrintf(PETSC_COMM_WORLD,"Second Norm of error %g\n", > double)norm);CHKERRQ(ierr); > ierr = PetscPrintf(PETSC_COMM_WORLD,"Iterations > %D\n",its);CHKERRQ(ierr); > } > > Am I calculating the "Norm_2" error correctly or making any mistake? > Unfortunately, for large sized matrix, say 10^6, I am getting very high > value of "Norm_2" error. > 1) I am guessing x comes from a KSPSolve(). It is only as accurate as your tolerance 2) This is the l_2 norm, not the L_2 norm, so if you are using a continuum method like FEM, this is likely wrong. Thanks, Matt > Thank you. > Sincerely, > Huq > > -- > > Fazlul Huq > Graduate Research Assistant > Department of Nuclear, Plasma & Radiological Engineering (NPRE) > University of Illinois at Urbana-Champaign (UIUC) > E-mail: huq2090 at gmail.com > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From huq2090 at gmail.com Fri Jun 5 11:32:44 2020 From: huq2090 at gmail.com (Fazlul Huq) Date: Fri, 5 Jun 2020 11:32:44 -0500 Subject: [petsc-users] Norm_2 calculation In-Reply-To: References: Message-ID: Input string: mpiexec -np 1 ./poisson_m -n 10000000 -pc_type cholesky -ksp_view -ksp_converged_reason -ksp_monitor_true_residual Output is attached herewith. Thank you. Sincerely, Huq On Fri, Jun 5, 2020 at 10:16 AM Matthew Knepley wrote: > On Fri, Jun 5, 2020 at 11:04 AM Fazlul Huq wrote: > >> Hello All, >> I'm trying to calculate the norm_2 error of a solution. >> Here, vector s: Analytical solution >> vector x: Numerical solution >> >> ierr = VecView(s,PETSC_VIEWER_STDOUT_WORLD);CHKERRQ(ierr); >> ierr = VecView(x,PETSC_VIEWER_STDOUT_WORLD);CHKERRQ(ierr); >> >> ierr = VecAXPY(x,-1.0,s);CHKERRQ(ierr); >> ierr = VecNorm(x,NORM_2,&norm);CHKERRQ(ierr); >> ierr = VecView(x,PETSC_VIEWER_STDOUT_WORLD);CHKERRQ(ierr); >> ierr = KSPGetIterationNumber(ksp,&its);CHKERRQ(ierr); >> if (norm > tol) { >> ierr = PetscPrintf(PETSC_COMM_WORLD,"Second Norm of error %g\n", >> double)norm);CHKERRQ(ierr); >> ierr = PetscPrintf(PETSC_COMM_WORLD,"Iterations >> %D\n",its);CHKERRQ(ierr); >> } >> >> Am I calculating the "Norm_2" error correctly or making any mistake? >> Unfortunately, for large sized matrix, say 10^6, I am getting very high >> value of "Norm_2" error. >> > > 1) I am guessing x comes from a KSPSolve(). It is only as accurate as your > tolerance > > 2) This is the l_2 norm, not the L_2 norm, so if you are using a continuum > method like FEM, this is likely wrong. > > Thanks, > > Matt > > >> Thank you. >> Sincerely, >> Huq >> >> -- >> >> Fazlul Huq >> Graduate Research Assistant >> Department of Nuclear, Plasma & Radiological Engineering (NPRE) >> University of Illinois at Urbana-Champaign (UIUC) >> E-mail: huq2090 at gmail.com >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -- Fazlul Huq Graduate Research Assistant Department of Nuclear, Plasma & Radiological Engineering (NPRE) University of Illinois at Urbana-Champaign (UIUC) E-mail: huq2090 at gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: output.log Type: text/x-log Size: 2915 bytes Desc: not available URL: From knepley at gmail.com Fri Jun 5 12:53:24 2020 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 5 Jun 2020 13:53:24 -0400 Subject: [petsc-users] Norm_2 calculation In-Reply-To: References: Message-ID: On Fri, Jun 5, 2020 at 12:32 PM Fazlul Huq wrote: > Input string: > mpiexec -np 1 ./poisson_m -n 10000000 -pc_type cholesky -ksp_view > -ksp_converged_reason -ksp_monitor_true_residual > > Output is attached herewith. > Solving the problem... 0 KSP preconditioned resid norm 9.741453255800e+07 true resid norm 1.802775638200e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 5.796040438920e+00 true resid norm 3.146946078273e-05 ||r(i)||/||b|| 1.745611606675e-06 Your Cholesky preconditioner is crap, probably because the system is either not symmetric or nearly singular. You can see this because the preconditioned residual is 6 orders of magnitude greater than the true residual. Also, if you want to evaluate convergence, you should probably use a lower tolerance like -ksp_rtol 1e-10 Thanks, Matt Thank you. > Sincerely, > Huq > > On Fri, Jun 5, 2020 at 10:16 AM Matthew Knepley wrote: > >> On Fri, Jun 5, 2020 at 11:04 AM Fazlul Huq wrote: >> >>> Hello All, >>> I'm trying to calculate the norm_2 error of a solution. >>> Here, vector s: Analytical solution >>> vector x: Numerical solution >>> >>> ierr = VecView(s,PETSC_VIEWER_STDOUT_WORLD);CHKERRQ(ierr); >>> ierr = VecView(x,PETSC_VIEWER_STDOUT_WORLD);CHKERRQ(ierr); >>> >>> ierr = VecAXPY(x,-1.0,s);CHKERRQ(ierr); >>> ierr = VecNorm(x,NORM_2,&norm);CHKERRQ(ierr); >>> ierr = VecView(x,PETSC_VIEWER_STDOUT_WORLD);CHKERRQ(ierr); >>> ierr = KSPGetIterationNumber(ksp,&its);CHKERRQ(ierr); >>> if (norm > tol) { >>> ierr = PetscPrintf(PETSC_COMM_WORLD,"Second Norm of error %g\n", >>> double)norm);CHKERRQ(ierr); >>> ierr = PetscPrintf(PETSC_COMM_WORLD,"Iterations >>> %D\n",its);CHKERRQ(ierr); >>> } >>> >>> Am I calculating the "Norm_2" error correctly or making any mistake? >>> Unfortunately, for large sized matrix, say 10^6, I am getting very high >>> value of "Norm_2" error. >>> >> >> 1) I am guessing x comes from a KSPSolve(). It is only as accurate as >> your tolerance >> >> 2) This is the l_2 norm, not the L_2 norm, so if you are using a >> continuum method like FEM, this is likely wrong. >> >> Thanks, >> >> Matt >> >> >>> Thank you. >>> Sincerely, >>> Huq >>> >>> -- >>> >>> Fazlul Huq >>> Graduate Research Assistant >>> Department of Nuclear, Plasma & Radiological Engineering (NPRE) >>> University of Illinois at Urbana-Champaign (UIUC) >>> E-mail: huq2090 at gmail.com >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> >> > > > -- > > Fazlul Huq > Graduate Research Assistant > Department of Nuclear, Plasma & Radiological Engineering (NPRE) > University of Illinois at Urbana-Champaign (UIUC) > E-mail: huq2090 at gmail.com > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From huq2090 at gmail.com Fri Jun 5 13:17:51 2020 From: huq2090 at gmail.com (Fazlul Huq) Date: Fri, 5 Jun 2020 13:17:51 -0500 Subject: [petsc-users] Norm_2 calculation In-Reply-To: References: Message-ID: The same is happening with "-pc_type hypre -pc_hypre_type boomerang" and with "-pc_type ilu". I tried with, ierr = KSPSetTolerances(ksp,1.e-10,PETSC_DEFAULT,PETSC_DEFAULT,PETSC_DEFAULT);CHKERRQ(ierr); this also. Getting higher values of error even then. Any suggestion to go over this issue? Shall I send the code? It's not a large script (single routine, less than 100 lines except comment). Thank you. Sincerely, Huq On Fri, Jun 5, 2020 at 12:53 PM Matthew Knepley wrote: > On Fri, Jun 5, 2020 at 12:32 PM Fazlul Huq wrote: > >> Input string: >> mpiexec -np 1 ./poisson_m -n 10000000 -pc_type cholesky -ksp_view >> -ksp_converged_reason -ksp_monitor_true_residual >> >> Output is attached herewith. >> > > Solving the problem... > > 0 KSP preconditioned resid norm 9.741453255800e+07 true resid norm > 1.802775638200e+01 ||r(i)||/||b|| 1.000000000000e+00 > > 1 KSP preconditioned resid norm 5.796040438920e+00 true resid norm > 3.146946078273e-05 ||r(i)||/||b|| 1.745611606675e-06 > > Your Cholesky preconditioner is crap, probably because the system is > either not symmetric or nearly singular. You can see this > because the preconditioned residual is 6 orders of magnitude greater than > the true residual. Also, if you want to evaluate convergence, you > should probably use a lower tolerance like > > -ksp_rtol 1e-10 > > Thanks, > > Matt > > Thank you. >> Sincerely, >> Huq >> >> On Fri, Jun 5, 2020 at 10:16 AM Matthew Knepley >> wrote: >> >>> On Fri, Jun 5, 2020 at 11:04 AM Fazlul Huq wrote: >>> >>>> Hello All, >>>> I'm trying to calculate the norm_2 error of a solution. >>>> Here, vector s: Analytical solution >>>> vector x: Numerical solution >>>> >>>> ierr = VecView(s,PETSC_VIEWER_STDOUT_WORLD);CHKERRQ(ierr); >>>> ierr = VecView(x,PETSC_VIEWER_STDOUT_WORLD);CHKERRQ(ierr); >>>> >>>> ierr = VecAXPY(x,-1.0,s);CHKERRQ(ierr); >>>> ierr = VecNorm(x,NORM_2,&norm);CHKERRQ(ierr); >>>> ierr = VecView(x,PETSC_VIEWER_STDOUT_WORLD);CHKERRQ(ierr); >>>> ierr = KSPGetIterationNumber(ksp,&its);CHKERRQ(ierr); >>>> if (norm > tol) { >>>> ierr = PetscPrintf(PETSC_COMM_WORLD,"Second Norm of error %g\n", >>>> double)norm);CHKERRQ(ierr); >>>> ierr = PetscPrintf(PETSC_COMM_WORLD,"Iterations >>>> %D\n",its);CHKERRQ(ierr); >>>> } >>>> >>>> Am I calculating the "Norm_2" error correctly or making any mistake? >>>> Unfortunately, for large sized matrix, say 10^6, I am getting very high >>>> value of "Norm_2" error. >>>> >>> >>> 1) I am guessing x comes from a KSPSolve(). It is only as accurate as >>> your tolerance >>> >>> 2) This is the l_2 norm, not the L_2 norm, so if you are using a >>> continuum method like FEM, this is likely wrong. >>> >>> Thanks, >>> >>> Matt >>> >>> >>>> Thank you. >>>> Sincerely, >>>> Huq >>>> >>>> -- >>>> >>>> Fazlul Huq >>>> Graduate Research Assistant >>>> Department of Nuclear, Plasma & Radiological Engineering (NPRE) >>>> University of Illinois at Urbana-Champaign (UIUC) >>>> E-mail: huq2090 at gmail.com >>>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> https://www.cse.buffalo.edu/~knepley/ >>> >>> >> >> >> -- >> >> Fazlul Huq >> Graduate Research Assistant >> Department of Nuclear, Plasma & Radiological Engineering (NPRE) >> University of Illinois at Urbana-Champaign (UIUC) >> E-mail: huq2090 at gmail.com >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -- Fazlul Huq Graduate Research Assistant Department of Nuclear, Plasma & Radiological Engineering (NPRE) University of Illinois at Urbana-Champaign (UIUC) E-mail: huq2090 at gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: output_boomeramg.log Type: text/x-log Size: 3983 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: output_ilu.log Type: text/x-log Size: 3088 bytes Desc: not available URL: From knepley at gmail.com Fri Jun 5 13:52:49 2020 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 5 Jun 2020 14:52:49 -0400 Subject: [petsc-users] Norm_2 calculation In-Reply-To: References: Message-ID: On Fri, Jun 5, 2020 at 2:18 PM Fazlul Huq wrote: > The same is happening with "-pc_type hypre -pc_hypre_type boomerang" and > with "-pc_type ilu". > > I tried with, > ierr = > KSPSetTolerances(ksp,1.e-10,PETSC_DEFAULT,PETSC_DEFAULT,PETSC_DEFAULT);CHKERRQ(ierr); > this also. > > Getting higher values of error even then. > Always check with -pc_type lu It seems likely that your matrix is flawed. Thanks, Matt > Any suggestion to go over this issue? > Shall I send the code? It's not a large script (single routine, less than > 100 lines except comment). > > Thank you. > Sincerely, > Huq > > On Fri, Jun 5, 2020 at 12:53 PM Matthew Knepley wrote: > >> On Fri, Jun 5, 2020 at 12:32 PM Fazlul Huq wrote: >> >>> Input string: >>> mpiexec -np 1 ./poisson_m -n 10000000 -pc_type cholesky -ksp_view >>> -ksp_converged_reason -ksp_monitor_true_residual >>> >>> Output is attached herewith. >>> >> >> Solving the problem... >> >> 0 KSP preconditioned resid norm 9.741453255800e+07 true resid norm >> 1.802775638200e+01 ||r(i)||/||b|| 1.000000000000e+00 >> >> 1 KSP preconditioned resid norm 5.796040438920e+00 true resid norm >> 3.146946078273e-05 ||r(i)||/||b|| 1.745611606675e-06 >> >> Your Cholesky preconditioner is crap, probably because the system is >> either not symmetric or nearly singular. You can see this >> because the preconditioned residual is 6 orders of magnitude greater than >> the true residual. Also, if you want to evaluate convergence, you >> should probably use a lower tolerance like >> >> -ksp_rtol 1e-10 >> >> Thanks, >> >> Matt >> >> Thank you. >>> Sincerely, >>> Huq >>> >>> On Fri, Jun 5, 2020 at 10:16 AM Matthew Knepley >>> wrote: >>> >>>> On Fri, Jun 5, 2020 at 11:04 AM Fazlul Huq wrote: >>>> >>>>> Hello All, >>>>> I'm trying to calculate the norm_2 error of a solution. >>>>> Here, vector s: Analytical solution >>>>> vector x: Numerical solution >>>>> >>>>> ierr = VecView(s,PETSC_VIEWER_STDOUT_WORLD);CHKERRQ(ierr); >>>>> ierr = VecView(x,PETSC_VIEWER_STDOUT_WORLD);CHKERRQ(ierr); >>>>> >>>>> ierr = VecAXPY(x,-1.0,s);CHKERRQ(ierr); >>>>> ierr = VecNorm(x,NORM_2,&norm);CHKERRQ(ierr); >>>>> ierr = VecView(x,PETSC_VIEWER_STDOUT_WORLD);CHKERRQ(ierr); >>>>> ierr = KSPGetIterationNumber(ksp,&its);CHKERRQ(ierr); >>>>> if (norm > tol) { >>>>> ierr = PetscPrintf(PETSC_COMM_WORLD,"Second Norm of error %g\n", >>>>> double)norm);CHKERRQ(ierr); >>>>> ierr = PetscPrintf(PETSC_COMM_WORLD,"Iterations >>>>> %D\n",its);CHKERRQ(ierr); >>>>> } >>>>> >>>>> Am I calculating the "Norm_2" error correctly or making any mistake? >>>>> Unfortunately, for large sized matrix, say 10^6, I am getting very >>>>> high value of "Norm_2" error. >>>>> >>>> >>>> 1) I am guessing x comes from a KSPSolve(). It is only as accurate as >>>> your tolerance >>>> >>>> 2) This is the l_2 norm, not the L_2 norm, so if you are using a >>>> continuum method like FEM, this is likely wrong. >>>> >>>> Thanks, >>>> >>>> Matt >>>> >>>> >>>>> Thank you. >>>>> Sincerely, >>>>> Huq >>>>> >>>>> -- >>>>> >>>>> Fazlul Huq >>>>> Graduate Research Assistant >>>>> Department of Nuclear, Plasma & Radiological Engineering (NPRE) >>>>> University of Illinois at Urbana-Champaign (UIUC) >>>>> E-mail: huq2090 at gmail.com >>>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>>> https://www.cse.buffalo.edu/~knepley/ >>>> >>>> >>> >>> >>> -- >>> >>> Fazlul Huq >>> Graduate Research Assistant >>> Department of Nuclear, Plasma & Radiological Engineering (NPRE) >>> University of Illinois at Urbana-Champaign (UIUC) >>> E-mail: huq2090 at gmail.com >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> >> > > > -- > > Fazlul Huq > Graduate Research Assistant > Department of Nuclear, Plasma & Radiological Engineering (NPRE) > University of Illinois at Urbana-Champaign (UIUC) > E-mail: huq2090 at gmail.com > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From adenchfi at hawk.iit.edu Fri Jun 5 13:56:45 2020 From: adenchfi at hawk.iit.edu (Adam Denchfield) Date: Fri, 5 Jun 2020 13:56:45 -0500 Subject: [petsc-users] KPM plans for slepc? Message-ID: Hello, I was wondering if there were plans to include an eigenvalue-counting/spectral density estimator into SLEPc, such as KPM or related methods. Regards, *Adam Denchfield* *Ph.D Student, Physics* University of Illinois in Chicago B.S. Applied Physics (2018) Illinois Institute of Technology Email: adenchfi at hawk.iit.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: From huq2090 at gmail.com Fri Jun 5 14:08:29 2020 From: huq2090 at gmail.com (Fazlul Huq) Date: Fri, 5 Jun 2020 14:08:29 -0500 Subject: [petsc-users] Norm_2 calculation In-Reply-To: References: Message-ID: Thanks for the hint! I will look at the matrix more closely. For reference: if (!rstart) { rstart = 1; i = 0; col[0] = 0; col[1] = 1; value[0] = 2.0; value[1] = -1.0; ierr = MatSetValues(A,1,&i,2,col,value,INSERT_VALUES);CHKERRQ(ierr); } if (rend == n) { rend = n-1; i = n-1; col[0] = n-2; col[1] = n-1; value[0] = -1.0; value[1] = 2.0; ierr = MatSetValues(A,1,&i,2,col,value,INSERT_VALUES);CHKERRQ(ierr); } /* Set entries corresponding to the mesh interior */ value[0] = -1.0; value[1] = 2.0; value[2] = -1.0; for (i=rstart; i wrote: > On Fri, Jun 5, 2020 at 2:18 PM Fazlul Huq wrote: > >> The same is happening with "-pc_type hypre -pc_hypre_type boomerang" and >> with "-pc_type ilu". >> >> I tried with, >> ierr = >> KSPSetTolerances(ksp,1.e-10,PETSC_DEFAULT,PETSC_DEFAULT,PETSC_DEFAULT);CHKERRQ(ierr); >> this also. >> >> Getting higher values of error even then. >> > > Always check with > > -pc_type lu > > It seems likely that your matrix is flawed. > > Thanks, > > Matt > > >> Any suggestion to go over this issue? >> Shall I send the code? It's not a large script (single routine, less than >> 100 lines except comment). >> >> Thank you. >> Sincerely, >> Huq >> >> On Fri, Jun 5, 2020 at 12:53 PM Matthew Knepley >> wrote: >> >>> On Fri, Jun 5, 2020 at 12:32 PM Fazlul Huq wrote: >>> >>>> Input string: >>>> mpiexec -np 1 ./poisson_m -n 10000000 -pc_type cholesky -ksp_view >>>> -ksp_converged_reason -ksp_monitor_true_residual >>>> >>>> Output is attached herewith. >>>> >>> >>> Solving the problem... >>> >>> 0 KSP preconditioned resid norm 9.741453255800e+07 true resid norm >>> 1.802775638200e+01 ||r(i)||/||b|| 1.000000000000e+00 >>> >>> 1 KSP preconditioned resid norm 5.796040438920e+00 true resid norm >>> 3.146946078273e-05 ||r(i)||/||b|| 1.745611606675e-06 >>> >>> Your Cholesky preconditioner is crap, probably because the system is >>> either not symmetric or nearly singular. You can see this >>> because the preconditioned residual is 6 orders of magnitude greater >>> than the true residual. Also, if you want to evaluate convergence, you >>> should probably use a lower tolerance like >>> >>> -ksp_rtol 1e-10 >>> >>> Thanks, >>> >>> Matt >>> >>> Thank you. >>>> Sincerely, >>>> Huq >>>> >>>> On Fri, Jun 5, 2020 at 10:16 AM Matthew Knepley >>>> wrote: >>>> >>>>> On Fri, Jun 5, 2020 at 11:04 AM Fazlul Huq wrote: >>>>> >>>>>> Hello All, >>>>>> I'm trying to calculate the norm_2 error of a solution. >>>>>> Here, vector s: Analytical solution >>>>>> vector x: Numerical solution >>>>>> >>>>>> ierr = VecView(s,PETSC_VIEWER_STDOUT_WORLD);CHKERRQ(ierr); >>>>>> ierr = VecView(x,PETSC_VIEWER_STDOUT_WORLD);CHKERRQ(ierr); >>>>>> >>>>>> ierr = VecAXPY(x,-1.0,s);CHKERRQ(ierr); >>>>>> ierr = VecNorm(x,NORM_2,&norm);CHKERRQ(ierr); >>>>>> ierr = VecView(x,PETSC_VIEWER_STDOUT_WORLD);CHKERRQ(ierr); >>>>>> ierr = KSPGetIterationNumber(ksp,&its);CHKERRQ(ierr); >>>>>> if (norm > tol) { >>>>>> ierr = PetscPrintf(PETSC_COMM_WORLD,"Second Norm of error %g\n", >>>>>> double)norm);CHKERRQ(ierr); >>>>>> ierr = PetscPrintf(PETSC_COMM_WORLD,"Iterations >>>>>> %D\n",its);CHKERRQ(ierr); >>>>>> } >>>>>> >>>>>> Am I calculating the "Norm_2" error correctly or making any mistake? >>>>>> Unfortunately, for large sized matrix, say 10^6, I am getting very >>>>>> high value of "Norm_2" error. >>>>>> >>>>> >>>>> 1) I am guessing x comes from a KSPSolve(). It is only as accurate as >>>>> your tolerance >>>>> >>>>> 2) This is the l_2 norm, not the L_2 norm, so if you are using a >>>>> continuum method like FEM, this is likely wrong. >>>>> >>>>> Thanks, >>>>> >>>>> Matt >>>>> >>>>> >>>>>> Thank you. >>>>>> Sincerely, >>>>>> Huq >>>>>> >>>>>> -- >>>>>> >>>>>> Fazlul Huq >>>>>> Graduate Research Assistant >>>>>> Department of Nuclear, Plasma & Radiological Engineering (NPRE) >>>>>> University of Illinois at Urbana-Champaign (UIUC) >>>>>> E-mail: huq2090 at gmail.com >>>>>> >>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their >>>>> experiments is infinitely more interesting than any results to which their >>>>> experiments lead. >>>>> -- Norbert Wiener >>>>> >>>>> https://www.cse.buffalo.edu/~knepley/ >>>>> >>>>> >>>> >>>> >>>> -- >>>> >>>> Fazlul Huq >>>> Graduate Research Assistant >>>> Department of Nuclear, Plasma & Radiological Engineering (NPRE) >>>> University of Illinois at Urbana-Champaign (UIUC) >>>> E-mail: huq2090 at gmail.com >>>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> https://www.cse.buffalo.edu/~knepley/ >>> >>> >> >> >> -- >> >> Fazlul Huq >> Graduate Research Assistant >> Department of Nuclear, Plasma & Radiological Engineering (NPRE) >> University of Illinois at Urbana-Champaign (UIUC) >> E-mail: huq2090 at gmail.com >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -- Fazlul Huq Graduate Research Assistant Department of Nuclear, Plasma & Radiological Engineering (NPRE) University of Illinois at Urbana-Champaign (UIUC) E-mail: huq2090 at gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: output_lu.log Type: text/x-log Size: 3334 bytes Desc: not available URL: From jroman at dsic.upv.es Fri Jun 5 14:39:39 2020 From: jroman at dsic.upv.es (Jose E. Roman) Date: Fri, 5 Jun 2020 21:39:39 +0200 Subject: [petsc-users] KPM plans for slepc? In-Reply-To: References: Message-ID: <19B60A7A-9647-4D9F-BA32-953E2D148711@dsic.upv.es> Yes, I have this in mind. But it will not happen soon because I am busy with other projects. Is this essential for your work? Jose > El 5 jun 2020, a las 20:56, Adam Denchfield via petsc-users escribi?: > > Hello, > > I was wondering if there were plans to include an eigenvalue-counting/spectral density estimator into SLEPc, such as KPM or related methods. > > Regards, > Adam Denchfield > Ph.D Student, Physics > University of Illinois in Chicago > B.S. Applied Physics (2018) > Illinois Institute of Technology > Email: adenchfi at hawk.iit.edu From shudson at anl.gov Fri Jun 5 14:41:57 2020 From: shudson at anl.gov (Hudson, Stephen Tobias P) Date: Fri, 5 Jun 2020 19:41:57 +0000 Subject: [petsc-users] Terminating a process running petsc via petsc4py without mpi_abort In-Reply-To: References: , Message-ID: Thanks, I will experiment with this. I am working through the multiprocessing interface, but I can see that the routines provided there are pretty much wrappers to the process signal functions. I guess the alternative is SIGKILL. Steve ________________________________ From: Lisandro Dalcin Sent: Thursday, June 4, 2020 4:54 PM To: Hudson, Stephen Tobias P Cc: Balay, Satish ; petsc-users at mcs.anl.gov Subject: Re: Terminating a process running petsc via petsc4py without mpi_abort (1) You can use PETSc.Sys.pushErrorHandler("abort"), but it will not help you. What you really need is to override PETSc's default signal handling (2) While it is true that PETSc overrides the signal handler, you can override it again from python after from petsc4py import PETSc. For implementing (2), maybe you should try sending SIGINT and not SIGTERM, such that you can do the following. from petsc4py import PETSc import signal signal.signal(signal.SIGINT, signal.default_int_handler) ... if __name__ == "__main__": try: main() except KeyboardInterrupt: # Triggered if Ctrl+C or signaled with SIGINT ... # do cleanup if needed Otherwise, you just need signal.signal(signal.SIGINT, signal.SIG_DFL) PS: I'm not in favor of changing current PETSc's signal handling behavior. This particular issue is fixable with two lines of Python code: from signal import signal, SIGINT, SIG_DFL signal(SIGINT, SIG_DFL) On Thu, 4 Jun 2020 at 23:39, Hudson, Stephen Tobias P > wrote: Lisandro, I don't see an interface to set this through petsc4py. Is it possible? Thanks, Steve ________________________________ From: Hudson, Stephen Tobias P > Sent: Thursday, June 4, 2020 2:47 PM To: Balay, Satish > Cc: petsc-users at mcs.anl.gov >; Lisandro Dalcin > Subject: Re: Terminating a process running petsc via petsc4py without mpi_abort Sounds good. I will have a look at how to set this through petsc4py. Thanks Steve ________________________________ From: Satish Balay > Sent: Thursday, June 4, 2020 2:32 PM To: Hudson, Stephen Tobias P > Cc: petsc-users at mcs.anl.gov >; Lisandro Dalcin > Subject: Re: Terminating a process running petsc via petsc4py without mpi_abort I don't completely understand the issue here. How is sequential run different than parallel run? In both cases - a PetscErrorHandler is likely getting invoked. One can change this behavior with: https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscPushErrorHandler.html And there are a few default error handlers to choose PETSC_EXTERN PetscErrorCode PetscTraceBackErrorHandler(MPI_Comm,int,const char*,const char*,PetscErrorCode,PetscErrorType,const char*,void*); PETSC_EXTERN PetscErrorCode PetscIgnoreErrorHandler(MPI_Comm,int,const char*,const char*,PetscErrorCode,PetscErrorType,const char*,void*); PETSC_EXTERN PetscErrorCode PetscEmacsClientErrorHandler(MPI_Comm,int,const char*,const char*,PetscErrorCode,PetscErrorType,const char*,void*); PETSC_EXTERN PetscErrorCode PetscMPIAbortErrorHandler(MPI_Comm,int,const char*,const char*,PetscErrorCode,PetscErrorType,const char*,void*); PETSC_EXTERN PetscErrorCode PetscAbortErrorHandler(MPI_Comm,int,const char*,const char*,PetscErrorCode,PetscErrorType,const char*,void*); PETSC_EXTERN PetscErrorCode PetscAttachDebuggerErrorHandler(MPI_Comm,int,const char*,const char*,PetscErrorCode,PetscErrorType,const char*,void*); PETSC_EXTERN PetscErrorCode PetscReturnErrorHandler(MPI_Comm,int,const char*,const char*,PetscErrorCode,PetscErrorType,const char*,void*); Some of the are accessible via command line option. for ex: -on_error_abort or -on_error_mpiabort Or perhaps you want to completely disable error handler with: -no_signal_handler cc: petsc-users Satish On Thu, 4 Jun 2020, Hudson, Stephen Tobias P wrote: > Satish, > > We are having issues caused by MPI_abort getting called when we try to terminate a sub-process running petsc4py. Ideally we would always use a serial build of petsc/petsc4py in this mode, but many users will have a parallel build. We need to be able to send a terminate signal that just kills the process. > > Is there a way to turn off the mpi_abort? > > Thanks, > > Steve > > -- Lisandro Dalcin ============ Research Scientist Extreme Computing Research Center (ECRC) King Abdullah University of Science and Technology (KAUST) http://ecrc.kaust.edu.sa/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From shudson at anl.gov Fri Jun 5 15:39:27 2020 From: shudson at anl.gov (Hudson, Stephen Tobias P) Date: Fri, 5 Jun 2020 20:39:27 +0000 Subject: [petsc-users] Terminating a process running petsc via petsc4py without mpi_abort In-Reply-To: References: , , Message-ID: It seems I do have to bypass Python's multiprocessing somewhat limited interface. E.g. self.process._popen._send_signal(signal.SIGINT) which works, but I am by-passing the API. I would support allowing the user to configure at run-time the signal handling for SIGTERM to exit without MPI_ABORT. I think I understand MPI_ABORT being the default, I've experienced hangs due to errors on single processes. ________________________________ From: Hudson, Stephen Tobias P Sent: Friday, June 5, 2020 2:41 PM To: Lisandro Dalcin Cc: Balay, Satish ; petsc-users at mcs.anl.gov Subject: Re: Terminating a process running petsc via petsc4py without mpi_abort Thanks, I will experiment with this. I am working through the multiprocessing interface, but I can see that the routines provided there are pretty much wrappers to the process signal functions. I guess the alternative is SIGKILL. Steve ________________________________ From: Lisandro Dalcin Sent: Thursday, June 4, 2020 4:54 PM To: Hudson, Stephen Tobias P Cc: Balay, Satish ; petsc-users at mcs.anl.gov Subject: Re: Terminating a process running petsc via petsc4py without mpi_abort (1) You can use PETSc.Sys.pushErrorHandler("abort"), but it will not help you. What you really need is to override PETSc's default signal handling (2) While it is true that PETSc overrides the signal handler, you can override it again from python after from petsc4py import PETSc. For implementing (2), maybe you should try sending SIGINT and not SIGTERM, such that you can do the following. from petsc4py import PETSc import signal signal.signal(signal.SIGINT, signal.default_int_handler) ... if __name__ == "__main__": try: main() except KeyboardInterrupt: # Triggered if Ctrl+C or signaled with SIGINT ... # do cleanup if needed Otherwise, you just need signal.signal(signal.SIGINT, signal.SIG_DFL) PS: I'm not in favor of changing current PETSc's signal handling behavior. This particular issue is fixable with two lines of Python code: from signal import signal, SIGINT, SIG_DFL signal(SIGINT, SIG_DFL) On Thu, 4 Jun 2020 at 23:39, Hudson, Stephen Tobias P > wrote: Lisandro, I don't see an interface to set this through petsc4py. Is it possible? Thanks, Steve ________________________________ From: Hudson, Stephen Tobias P > Sent: Thursday, June 4, 2020 2:47 PM To: Balay, Satish > Cc: petsc-users at mcs.anl.gov >; Lisandro Dalcin > Subject: Re: Terminating a process running petsc via petsc4py without mpi_abort Sounds good. I will have a look at how to set this through petsc4py. Thanks Steve ________________________________ From: Satish Balay > Sent: Thursday, June 4, 2020 2:32 PM To: Hudson, Stephen Tobias P > Cc: petsc-users at mcs.anl.gov >; Lisandro Dalcin > Subject: Re: Terminating a process running petsc via petsc4py without mpi_abort I don't completely understand the issue here. How is sequential run different than parallel run? In both cases - a PetscErrorHandler is likely getting invoked. One can change this behavior with: https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscPushErrorHandler.html And there are a few default error handlers to choose PETSC_EXTERN PetscErrorCode PetscTraceBackErrorHandler(MPI_Comm,int,const char*,const char*,PetscErrorCode,PetscErrorType,const char*,void*); PETSC_EXTERN PetscErrorCode PetscIgnoreErrorHandler(MPI_Comm,int,const char*,const char*,PetscErrorCode,PetscErrorType,const char*,void*); PETSC_EXTERN PetscErrorCode PetscEmacsClientErrorHandler(MPI_Comm,int,const char*,const char*,PetscErrorCode,PetscErrorType,const char*,void*); PETSC_EXTERN PetscErrorCode PetscMPIAbortErrorHandler(MPI_Comm,int,const char*,const char*,PetscErrorCode,PetscErrorType,const char*,void*); PETSC_EXTERN PetscErrorCode PetscAbortErrorHandler(MPI_Comm,int,const char*,const char*,PetscErrorCode,PetscErrorType,const char*,void*); PETSC_EXTERN PetscErrorCode PetscAttachDebuggerErrorHandler(MPI_Comm,int,const char*,const char*,PetscErrorCode,PetscErrorType,const char*,void*); PETSC_EXTERN PetscErrorCode PetscReturnErrorHandler(MPI_Comm,int,const char*,const char*,PetscErrorCode,PetscErrorType,const char*,void*); Some of the are accessible via command line option. for ex: -on_error_abort or -on_error_mpiabort Or perhaps you want to completely disable error handler with: -no_signal_handler cc: petsc-users Satish On Thu, 4 Jun 2020, Hudson, Stephen Tobias P wrote: > Satish, > > We are having issues caused by MPI_abort getting called when we try to terminate a sub-process running petsc4py. Ideally we would always use a serial build of petsc/petsc4py in this mode, but many users will have a parallel build. We need to be able to send a terminate signal that just kills the process. > > Is there a way to turn off the mpi_abort? > > Thanks, > > Steve > > -- Lisandro Dalcin ============ Research Scientist Extreme Computing Research Center (ECRC) King Abdullah University of Science and Technology (KAUST) http://ecrc.kaust.edu.sa/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From danyang.su at gmail.com Fri Jun 5 16:02:59 2020 From: danyang.su at gmail.com (Danyang Su) Date: Fri, 05 Jun 2020 14:02:59 -0700 Subject: [petsc-users] Bug in ex14f.F90 when debug flags are used? Message-ID: <3339DBD3-860E-4FB2-98D3-9AEC908C351E@gmail.com> Hi All, I have a question regarding the following example. https://www.mcs.anl.gov/petsc/petsc-current/src/ksp/ksp/tutorials/ex14f.F90.html When debug flags are used in the make file, the code crashed with following error. At line 335 of file ex14f.F90 Fortran runtime error: Index '-14450582413' of dimension 1 of array 'xx' below lower bound of 1 FFLAGS = -g -fcheck=all -fbacktrace -Wall CPPFLAGS = -g -fcheck=all -fbacktrace -Wall Does this make sense? Danyang -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Fri Jun 5 16:11:54 2020 From: balay at mcs.anl.gov (Satish Balay) Date: Fri, 5 Jun 2020 16:11:54 -0500 (CDT) Subject: [petsc-users] Bug in ex14f.F90 when debug flags are used? In-Reply-To: <3339DBD3-860E-4FB2-98D3-9AEC908C351E@gmail.com> References: <3339DBD3-860E-4FB2-98D3-9AEC908C351E@gmail.com> Message-ID: VecGetArray() is for F77 - it relies on out-of-bound access. The safer call is VecGetArrayF90() Now that PETSc requires F90 - perhaps VecGetArray() should be deprecated [and all examples fixed to use VecGetArrayF90].. Satish On Fri, 5 Jun 2020, Danyang Su wrote: > Hi All, > > > > I have a question regarding the following example. > > https://www.mcs.anl.gov/petsc/petsc-current/src/ksp/ksp/tutorials/ex14f.F90.html > > > > > > When debug flags are used in the make file, the code crashed with following error. > > > > At line 335 of file ex14f.F90 > > Fortran runtime error: Index '-14450582413' of dimension 1 of array 'xx' below lower bound of 1 > > > > FFLAGS = -g -fcheck=all -fbacktrace -Wall > > CPPFLAGS = -g -fcheck=all -fbacktrace -Wall > > > > Does this make sense? > > > > Danyang > > From danyang.su at gmail.com Fri Jun 5 16:54:48 2020 From: danyang.su at gmail.com (Danyang Su) Date: Fri, 05 Jun 2020 14:54:48 -0700 Subject: [petsc-users] Bug in ex14f.F90 when debug flags are used? In-Reply-To: References: <3339DBD3-860E-4FB2-98D3-9AEC908C351E@gmail.com> Message-ID: Thanks, Satish. Danyang ?On 2020-06-05, 2:11 PM, "Satish Balay" wrote: VecGetArray() is for F77 - it relies on out-of-bound access. The safer call is VecGetArrayF90() Now that PETSc requires F90 - perhaps VecGetArray() should be deprecated [and all examples fixed to use VecGetArrayF90].. Satish On Fri, 5 Jun 2020, Danyang Su wrote: > Hi All, > > > > I have a question regarding the following example. > > https://www.mcs.anl.gov/petsc/petsc-current/src/ksp/ksp/tutorials/ex14f.F90.html > > > > > > When debug flags are used in the make file, the code crashed with following error. > > > > At line 335 of file ex14f.F90 > > Fortran runtime error: Index '-14450582413' of dimension 1 of array 'xx' below lower bound of 1 > > > > FFLAGS = -g -fcheck=all -fbacktrace -Wall > > CPPFLAGS = -g -fcheck=all -fbacktrace -Wall > > > > Does this make sense? > > > > Danyang > > From junchao.zhang at gmail.com Fri Jun 5 19:26:34 2020 From: junchao.zhang at gmail.com (Junchao Zhang) Date: Fri, 5 Jun 2020 19:26:34 -0500 Subject: [petsc-users] Terminating a process running petsc via petsc4py without mpi_abort In-Reply-To: References: Message-ID: On Fri, Jun 5, 2020 at 3:39 PM Hudson, Stephen Tobias P via petsc-users < petsc-users at mcs.anl.gov> wrote: > It seems I do have to bypass Python's multiprocessing somewhat limited > interface. E.g. > > self.process._popen._send_signal(signal.SIGINT) > > which works, but I am by-passing the API. > > I would support allowing the user to configure at run-time the signal > handling for SIGTERM to exit without MPI_ABORT. I think I understand > MPI_ABORT being the default, I've experienced hangs due to errors on single > processes. > ?hangs due to errors on single processes". If the single processes call exit(), then there will be no hang. > ------------------------------ > *From:* Hudson, Stephen Tobias P > *Sent:* Friday, June 5, 2020 2:41 PM > *To:* Lisandro Dalcin > *Cc:* Balay, Satish ; petsc-users at mcs.anl.gov < > petsc-users at mcs.anl.gov> > *Subject:* Re: Terminating a process running petsc via petsc4py without > mpi_abort > > Thanks, I will experiment with this. > > I am working through the multiprocessing interface, but I can see that the > routines provided there are pretty much wrappers to the process signal > functions. > > I guess the alternative is SIGKILL. > > Steve > ------------------------------ > *From:* Lisandro Dalcin > *Sent:* Thursday, June 4, 2020 4:54 PM > *To:* Hudson, Stephen Tobias P > *Cc:* Balay, Satish ; petsc-users at mcs.anl.gov < > petsc-users at mcs.anl.gov> > *Subject:* Re: Terminating a process running petsc via petsc4py without > mpi_abort > > (1) You can use PETSc.Sys.pushErrorHandler("abort"), but it will not help > you. What you really need is to override PETSc's default signal handling > > (2) While it is true that PETSc overrides the signal handler, you can > override it again from python after from petsc4py import PETSc. > > For implementing (2), maybe you should try sending SIGINT and not SIGTERM, > such that you can do the following. > > from petsc4py import PETSc > > import signal > signal.signal(signal.SIGINT, signal.default_int_handler) > > ... > > if __name__ == "__main__": > try: > main() > except KeyboardInterrupt: # Triggered if Ctrl+C or signaled with > SIGINT > ... # do cleanup if needed > > Otherwise, you just need signal.signal(signal.SIGINT, signal.SIG_DFL) > > > PS: I'm not in favor of changing current PETSc's signal handling behavior. > This particular issue is fixable with two lines of Python code: > > from signal import signal, SIGINT, SIG_DFL > signal(SIGINT, SIG_DFL) > > > > On Thu, 4 Jun 2020 at 23:39, Hudson, Stephen Tobias P > wrote: > > Lisandro, > > I don't see an interface to set this through petsc4py. Is it possible? > > Thanks, > Steve > ------------------------------ > *From:* Hudson, Stephen Tobias P > *Sent:* Thursday, June 4, 2020 2:47 PM > *To:* Balay, Satish > *Cc:* petsc-users at mcs.anl.gov ; Lisandro Dalcin < > dalcinl at gmail.com> > *Subject:* Re: Terminating a process running petsc via petsc4py without > mpi_abort > > Sounds good. I will have a look at how to set this through petsc4py. > > Thanks > Steve > ------------------------------ > *From:* Satish Balay > *Sent:* Thursday, June 4, 2020 2:32 PM > *To:* Hudson, Stephen Tobias P > *Cc:* petsc-users at mcs.anl.gov ; Lisandro Dalcin < > dalcinl at gmail.com> > *Subject:* Re: Terminating a process running petsc via petsc4py without > mpi_abort > > I don't completely understand the issue here. How is sequential run > different than parallel run? > > In both cases - a PetscErrorHandler is likely getting invoked. One can > change this behavior with: > > > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscPushErrorHandler.html > > And there are a few default error handlers to choose > > > PETSC_EXTERN PetscErrorCode PetscTraceBackErrorHandler(MPI_Comm,int,const > char*,const char*,PetscErrorCode,PetscErrorType,const char*,void*); > PETSC_EXTERN PetscErrorCode PetscIgnoreErrorHandler(MPI_Comm,int,const > char*,const char*,PetscErrorCode,PetscErrorType,const char*,void*); > PETSC_EXTERN PetscErrorCode > PetscEmacsClientErrorHandler(MPI_Comm,int,const char*,const > char*,PetscErrorCode,PetscErrorType,const char*,void*); > PETSC_EXTERN PetscErrorCode PetscMPIAbortErrorHandler(MPI_Comm,int,const > char*,const char*,PetscErrorCode,PetscErrorType,const char*,void*); > PETSC_EXTERN PetscErrorCode PetscAbortErrorHandler(MPI_Comm,int,const > char*,const char*,PetscErrorCode,PetscErrorType,const char*,void*); > PETSC_EXTERN PetscErrorCode > PetscAttachDebuggerErrorHandler(MPI_Comm,int,const char*,const > char*,PetscErrorCode,PetscErrorType,const char*,void*); > PETSC_EXTERN PetscErrorCode PetscReturnErrorHandler(MPI_Comm,int,const > char*,const char*,PetscErrorCode,PetscErrorType,const char*,void*); > > Some of the are accessible via command line option. for ex: > -on_error_abort or -on_error_mpiabort > > Or perhaps you want to completely disable error handler with: > -no_signal_handler > > cc: petsc-users > > Satish > > On Thu, 4 Jun 2020, Hudson, Stephen Tobias P wrote: > > > Satish, > > > > We are having issues caused by MPI_abort getting called when we try to > terminate a sub-process running petsc4py. Ideally we would always use a > serial build of petsc/petsc4py in this mode, but many users will have a > parallel build. We need to be able to send a terminate signal that just > kills the process. > > > > Is there a way to turn off the mpi_abort? > > > > Thanks, > > > > Steve > > > > > > > > -- > Lisandro Dalcin > ============ > Research Scientist > Extreme Computing Research Center (ECRC) > King Abdullah University of Science and Technology (KAUST) > http://ecrc.kaust.edu.sa/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From adenchfi at hawk.iit.edu Fri Jun 5 19:33:31 2020 From: adenchfi at hawk.iit.edu (Adam Denchfield) Date: Fri, 5 Jun 2020 19:33:31 -0500 Subject: [petsc-users] [Ext] Re: KPM plans for slepc? In-Reply-To: <19B60A7A-9647-4D9F-BA32-953E2D148711@dsic.upv.es> References: <19B60A7A-9647-4D9F-BA32-953E2D148711@dsic.upv.es> Message-ID: No, it is not essential to my work yet. I've been entertaining the idea of building a quantum many-body physics code on top of PETSc/SLEPc, if I follow through with it I'll check back in about it. On Fri, Jun 5, 2020, 2:39 PM Jose E. Roman wrote: > Yes, I have this in mind. But it will not happen soon because I am busy > with other projects. Is this essential for your work? > Jose > > > > El 5 jun 2020, a las 20:56, Adam Denchfield via petsc-users < > petsc-users at mcs.anl.gov> escribi?: > > > > Hello, > > > > I was wondering if there were plans to include an > eigenvalue-counting/spectral density estimator into SLEPc, such as KPM or > related methods. > > > > Regards, > > Adam Denchfield > > Ph.D Student, Physics > > University of Illinois in Chicago > > B.S. Applied Physics (2018) > > Illinois Institute of Technology > > Email: adenchfi at hawk.iit.edu > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Mon Jun 8 00:30:05 2020 From: jed at jedbrown.org (Jed Brown) Date: Sun, 07 Jun 2020 23:30:05 -0600 Subject: [petsc-users] Two positions (RSE and Postdoc) at CU Boulder: Extreme-scale simulation of composite inelastic media Message-ID: <87tuzmgm5e.fsf@jedbrown.org> We have two openings with similar descriptions. For the Research Software Engineer (funded for 5-years, renewable): CU Boulder's PSAAP Multidisciplinary Simulation Center has an opening for a *Research Software Engineer* to co-lead development of robust, extensible open source software for extreme-scale simulation of large-deformation composite poro-elasto-visco-plastic media across a broad range of regimes with experimental validation and coordination with micromorphic multiscale models. Successful applicants will have strong written and verbal communication skills, experience in some of the following topics, and interest in learning the others: * collaborative software development and devops (Git, continuous integration, etc.) * maintainable, high-performance programming techniques for CPUs and GPUs * finite element and material-point discretizations * computational mechanics/inelasticity * parallel algebraic solvers such as PETSc * scalable data-intensive computing For the Postdoc: CU Boulder's PSAAP Multidisciplinary Simulation Center has an opening for a *Postdoctoral Associate* to develop algorithms for extreme-scale simulation of large-deformation composite poro-elasto-visco-plastic media across a broad range of regimes, validation and rapid calibration using experimental data, and micromorphic multiscale models with uncertainty quantification. Successful applicants will have strong written and verbal communication skills, experience in some of the following topics, and interest in learning the others: * collaborative software development and devops (Git, continuous integration, etc.) * maintainable, high-performance programming techniques for CPUs and GPUs * finite element and material-point discretizations * computational mechanics/inelasticity * multiscale modeling techniques * parallel algebraic solvers such as PETSc * scalable data-intensive computing and uncertainty quantification Both positions can start as soon as July 1 and are remote-friendly, especially during the pandemic. US citizenship or permanent residency is preferred. Minimum requirement: non-sensitive country foreign national. To apply to either position, please send a CV, cover letter, and contact information for three references to jed.brown at colorado.edu and richard.regueiro at colorado.edu. The University of Colorado Boulder is committed to building a culturally diverse community of faculty, staff, and students dedicated to contributing to an inclusive campus environment. We are an Equal Opportunity employer, including veterans and individuals with disabilities. We offer a competitive salary and a comprehensive benefits package. From shudson at anl.gov Mon Jun 8 09:39:48 2020 From: shudson at anl.gov (Hudson, Stephen Tobias P) Date: Mon, 8 Jun 2020 14:39:48 +0000 Subject: [petsc-users] Terminating a process running petsc via petsc4py without mpi_abort In-Reply-To: References: , Message-ID: Ok, having looked at this a bit more, I'm inclined to support Junchao's approach, but there seems to be concern that, even if the standards support it, there could be issues in some scenarios. I don't have enough information to dispute this. But if this was put in many years ago, I'm interested in what the other MPI libraries do now - eg. does trilinos etc use MPI_ABORT in the signal handler. If not, do users report issues hanging on terminate? ________________________________ From: Junchao Zhang Sent: Friday, June 5, 2020 7:26 PM To: Hudson, Stephen Tobias P Cc: Lisandro Dalcin ; petsc-users at mcs.anl.gov Subject: Re: [petsc-users] Terminating a process running petsc via petsc4py without mpi_abort On Fri, Jun 5, 2020 at 3:39 PM Hudson, Stephen Tobias P via petsc-users > wrote: It seems I do have to bypass Python's multiprocessing somewhat limited interface. E.g. self.process._popen._send_signal(signal.SIGINT) which works, but I am by-passing the API. I would support allowing the user to configure at run-time the signal handling for SIGTERM to exit without MPI_ABORT. I think I understand MPI_ABORT being the default, I've experienced hangs due to errors on single processes. ?hangs due to errors on single processes". If the single processes call exit(), then there will be no hang. ________________________________ From: Hudson, Stephen Tobias P > Sent: Friday, June 5, 2020 2:41 PM To: Lisandro Dalcin > Cc: Balay, Satish >; petsc-users at mcs.anl.gov > Subject: Re: Terminating a process running petsc via petsc4py without mpi_abort Thanks, I will experiment with this. I am working through the multiprocessing interface, but I can see that the routines provided there are pretty much wrappers to the process signal functions. I guess the alternative is SIGKILL. Steve ________________________________ From: Lisandro Dalcin > Sent: Thursday, June 4, 2020 4:54 PM To: Hudson, Stephen Tobias P > Cc: Balay, Satish >; petsc-users at mcs.anl.gov > Subject: Re: Terminating a process running petsc via petsc4py without mpi_abort (1) You can use PETSc.Sys.pushErrorHandler("abort"), but it will not help you. What you really need is to override PETSc's default signal handling (2) While it is true that PETSc overrides the signal handler, you can override it again from python after from petsc4py import PETSc. For implementing (2), maybe you should try sending SIGINT and not SIGTERM, such that you can do the following. from petsc4py import PETSc import signal signal.signal(signal.SIGINT, signal.default_int_handler) ... if __name__ == "__main__": try: main() except KeyboardInterrupt: # Triggered if Ctrl+C or signaled with SIGINT ... # do cleanup if needed Otherwise, you just need signal.signal(signal.SIGINT, signal.SIG_DFL) PS: I'm not in favor of changing current PETSc's signal handling behavior. This particular issue is fixable with two lines of Python code: from signal import signal, SIGINT, SIG_DFL signal(SIGINT, SIG_DFL) On Thu, 4 Jun 2020 at 23:39, Hudson, Stephen Tobias P > wrote: Lisandro, I don't see an interface to set this through petsc4py. Is it possible? Thanks, Steve ________________________________ From: Hudson, Stephen Tobias P > Sent: Thursday, June 4, 2020 2:47 PM To: Balay, Satish > Cc: petsc-users at mcs.anl.gov >; Lisandro Dalcin > Subject: Re: Terminating a process running petsc via petsc4py without mpi_abort Sounds good. I will have a look at how to set this through petsc4py. Thanks Steve ________________________________ From: Satish Balay > Sent: Thursday, June 4, 2020 2:32 PM To: Hudson, Stephen Tobias P > Cc: petsc-users at mcs.anl.gov >; Lisandro Dalcin > Subject: Re: Terminating a process running petsc via petsc4py without mpi_abort I don't completely understand the issue here. How is sequential run different than parallel run? In both cases - a PetscErrorHandler is likely getting invoked. One can change this behavior with: https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscPushErrorHandler.html And there are a few default error handlers to choose PETSC_EXTERN PetscErrorCode PetscTraceBackErrorHandler(MPI_Comm,int,const char*,const char*,PetscErrorCode,PetscErrorType,const char*,void*); PETSC_EXTERN PetscErrorCode PetscIgnoreErrorHandler(MPI_Comm,int,const char*,const char*,PetscErrorCode,PetscErrorType,const char*,void*); PETSC_EXTERN PetscErrorCode PetscEmacsClientErrorHandler(MPI_Comm,int,const char*,const char*,PetscErrorCode,PetscErrorType,const char*,void*); PETSC_EXTERN PetscErrorCode PetscMPIAbortErrorHandler(MPI_Comm,int,const char*,const char*,PetscErrorCode,PetscErrorType,const char*,void*); PETSC_EXTERN PetscErrorCode PetscAbortErrorHandler(MPI_Comm,int,const char*,const char*,PetscErrorCode,PetscErrorType,const char*,void*); PETSC_EXTERN PetscErrorCode PetscAttachDebuggerErrorHandler(MPI_Comm,int,const char*,const char*,PetscErrorCode,PetscErrorType,const char*,void*); PETSC_EXTERN PetscErrorCode PetscReturnErrorHandler(MPI_Comm,int,const char*,const char*,PetscErrorCode,PetscErrorType,const char*,void*); Some of the are accessible via command line option. for ex: -on_error_abort or -on_error_mpiabort Or perhaps you want to completely disable error handler with: -no_signal_handler cc: petsc-users Satish On Thu, 4 Jun 2020, Hudson, Stephen Tobias P wrote: > Satish, > > We are having issues caused by MPI_abort getting called when we try to terminate a sub-process running petsc4py. Ideally we would always use a serial build of petsc/petsc4py in this mode, but many users will have a parallel build. We need to be able to send a terminate signal that just kills the process. > > Is there a way to turn off the mpi_abort? > > Thanks, > > Steve > > -- Lisandro Dalcin ============ Research Scientist Extreme Computing Research Center (ECRC) King Abdullah University of Science and Technology (KAUST) http://ecrc.kaust.edu.sa/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From mukkundsunjii at gmail.com Mon Jun 8 14:42:35 2020 From: mukkundsunjii at gmail.com (MUKKUND SUNJII) Date: Mon, 8 Jun 2020 21:42:35 +0200 Subject: [petsc-users] Regarding DM Labels Message-ID: <7E91C5AA-1273-49CB-9121-54C289A94065@gmail.com> Greetings, I am working on modifying the shallow water model in ex11 for the purpose of my thesis. I wish to run a specific dam break scenario using this model. In the scenario, the velocity fields of certain cells must be constrained. That is I want to create an ?artificial wall? inside the domain by constraining the value [uh] to not change. For this purpose, I tried creating a separate label in the DM. Further, I added the cells whose velocity values I wanted to constrain to the label using the routine : DMLabelSetValue(labelWall, c, 100); Next, in the routine where the boundary conditions are set, I pass a separate command : PetscDSAddBoundary(prob, DM_BC_ESSENTIAL, "wall", "DamWall", 0, 1, &comp, (void (*)(void)) PhysicsBoundary_Dam_Wall, sizeof(damWallids), &damWallids, phys); Ofcourse, in the function PhysicsBoundary_Dam_Wall() I set the vector value of [uh] to 0. When I do run the program, I do not see any of this in play (i.e., the values are not being constrained). Is there a smarter way to go about this or would you suggest a correction in my current method? Thank you in advance for your wonderful suggestions and work! Regards, Mukkund Sunjii -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Mon Jun 8 15:02:57 2020 From: jed at jedbrown.org (Jed Brown) Date: Mon, 08 Jun 2020 14:02:57 -0600 Subject: [petsc-users] Regarding DM Labels In-Reply-To: <7E91C5AA-1273-49CB-9121-54C289A94065@gmail.com> References: <7E91C5AA-1273-49CB-9121-54C289A94065@gmail.com> Message-ID: <87a71dgwb2.fsf@jedbrown.org> MUKKUND SUNJII writes: > Greetings, > > I am working on modifying the shallow water model in ex11 for the purpose of my thesis. > > I wish to run a specific dam break scenario using this model. In the scenario, the velocity fields of certain cells must be constrained. That is I want to create an ?artificial wall? inside the domain by constraining the value [uh] to not change. Why not make your unstructured mesh be the non-wall domain? That way you don't have to constrain cell values and you can (correctly) make the (reconstructed) zero velocity occur at the cell boundary rather than the cell interior. > For this purpose, I tried creating a separate label in the DM. Further, I added the cells whose velocity values I wanted to constrain to the label using the routine : DMLabelSetValue(labelWall, c, 100); > > Next, in the routine where the boundary conditions are set, I pass a separate command : > > PetscDSAddBoundary(prob, DM_BC_ESSENTIAL, "wall", "DamWall", 0, 1, &comp, (void (*)(void)) PhysicsBoundary_Dam_Wall, sizeof(damWallids), &damWallids, phys); > > Ofcourse, in the function PhysicsBoundary_Dam_Wall() I set the vector value of [uh] to 0. When I do run the program, I do not see any of this in play (i.e., the values are not being constrained). > > Is there a smarter way to go about this or would you suggest a correction in my current method? > > Thank you in advance for your wonderful suggestions and work! > > Regards, > > Mukkund Sunjii From jed at jedbrown.org Mon Jun 8 15:18:06 2020 From: jed at jedbrown.org (Jed Brown) Date: Mon, 08 Jun 2020 14:18:06 -0600 Subject: [petsc-users] Regarding DM Labels In-Reply-To: <2C17E7B6-CE0C-4935-9AAF-A5B6433817EE@gmail.com> References: <7E91C5AA-1273-49CB-9121-54C289A94065@gmail.com> <87a71dgwb2.fsf@jedbrown.org> <16BE320E-4773-423A-A102-9EFD48BE7E34@gmail.com> <2C17E7B6-CE0C-4935-9AAF-A5B6433817EE@gmail.com> Message-ID: <877dwhgvlt.fsf@jedbrown.org> Re-adding the list. MUKKUND SUNJII writes: > Apologies, I understand your point about the non-wall domain. > > However, I am interested in what is happening on the other side of the dam (the reservoir). > > Furthermore, I suppose in the non-wall domain, the boundary condition would change with respect to the height of the water in the reservoir. Oh, you want this to be a wall that water can slosh over? I'd recommend using a formulation with variable bathymetry and looking at well-balanced schemes. Also read up on Riemann solvers capable of wetting and drying. I wouldn't treat it as a "boundary". > Regards, > > Mukkund > >> On 8 Jun 2020, at 22:05, MUKKUND SUNJII wrote: >> >> I am not familiar with the non-wall domain. >> >> Would you be kind enough to direct me to an tutorial or a test file in the repo? >> >> Regards, >> >> Mukkund >> >>> On 8 Jun 2020, at 22:02, Jed Brown wrote: >>> >>> MUKKUND SUNJII writes: >>> >>>> Greetings, >>>> >>>> I am working on modifying the shallow water model in ex11 for the purpose of my thesis. >>>> >>>> I wish to run a specific dam break scenario using this model. In the scenario, the velocity fields of certain cells must be constrained. That is I want to create an ?artificial wall? inside the domain by constraining the value [uh] to not change. >>> >>> Why not make your unstructured mesh be the non-wall domain? That way >>> you don't have to constrain cell values and you can (correctly) make the >>> (reconstructed) zero velocity occur at the cell boundary rather than the >>> cell interior. >>> >>>> For this purpose, I tried creating a separate label in the DM. Further, I added the cells whose velocity values I wanted to constrain to the label using the routine : DMLabelSetValue(labelWall, c, 100); >>>> >>>> Next, in the routine where the boundary conditions are set, I pass a separate command : >>>> >>>> PetscDSAddBoundary(prob, DM_BC_ESSENTIAL, "wall", "DamWall", 0, 1, &comp, (void (*)(void)) PhysicsBoundary_Dam_Wall, sizeof(damWallids), &damWallids, phys); >>>> >>>> Ofcourse, in the function PhysicsBoundary_Dam_Wall() I set the vector value of [uh] to 0. When I do run the program, I do not see any of this in play (i.e., the values are not being constrained). >>>> >>>> Is there a smarter way to go about this or would you suggest a correction in my current method? >>>> >>>> Thank you in advance for your wonderful suggestions and work! >>>> >>>> Regards, >>>> >>>> Mukkund Sunjii >> From mukkundsunjii at gmail.com Mon Jun 8 17:35:46 2020 From: mukkundsunjii at gmail.com (MUKKUND SUNJII) Date: Tue, 9 Jun 2020 00:35:46 +0200 Subject: [petsc-users] Regarding DM Labels In-Reply-To: <877dwhgvlt.fsf@jedbrown.org> References: <7E91C5AA-1273-49CB-9121-54C289A94065@gmail.com> <87a71dgwb2.fsf@jedbrown.org> <16BE320E-4773-423A-A102-9EFD48BE7E34@gmail.com> <2C17E7B6-CE0C-4935-9AAF-A5B6433817EE@gmail.com> <877dwhgvlt.fsf@jedbrown.org> Message-ID: Greetings, I was indeed looking at implementing a model with variable bathymetry. It appears to be trickier than it seems. Thank you for your suggestions. Regards, Mukkund > On 8 Jun 2020, at 22:18, Jed Brown wrote: > > Re-adding the list. > > MUKKUND SUNJII writes: > >> Apologies, I understand your point about the non-wall domain. >> >> However, I am interested in what is happening on the other side of the dam (the reservoir). >> >> Furthermore, I suppose in the non-wall domain, the boundary condition would change with respect to the height of the water in the reservoir. > > Oh, you want this to be a wall that water can slosh over? I'd recommend > using a formulation with variable bathymetry and looking at > well-balanced schemes. Also read up on Riemann solvers capable of > wetting and drying. I wouldn't treat it as a "boundary". > >> Regards, >> >> Mukkund >> >>> On 8 Jun 2020, at 22:05, MUKKUND SUNJII wrote: >>> >>> I am not familiar with the non-wall domain. >>> >>> Would you be kind enough to direct me to an tutorial or a test file in the repo? >>> >>> Regards, >>> >>> Mukkund >>> >>>> On 8 Jun 2020, at 22:02, Jed Brown wrote: >>>> >>>> MUKKUND SUNJII writes: >>>> >>>>> Greetings, >>>>> >>>>> I am working on modifying the shallow water model in ex11 for the purpose of my thesis. >>>>> >>>>> I wish to run a specific dam break scenario using this model. In the scenario, the velocity fields of certain cells must be constrained. That is I want to create an ?artificial wall? inside the domain by constraining the value [uh] to not change. >>>> >>>> Why not make your unstructured mesh be the non-wall domain? That way >>>> you don't have to constrain cell values and you can (correctly) make the >>>> (reconstructed) zero velocity occur at the cell boundary rather than the >>>> cell interior. >>>> >>>>> For this purpose, I tried creating a separate label in the DM. Further, I added the cells whose velocity values I wanted to constrain to the label using the routine : DMLabelSetValue(labelWall, c, 100); >>>>> >>>>> Next, in the routine where the boundary conditions are set, I pass a separate command : >>>>> >>>>> PetscDSAddBoundary(prob, DM_BC_ESSENTIAL, "wall", "DamWall", 0, 1, &comp, (void (*)(void)) PhysicsBoundary_Dam_Wall, sizeof(damWallids), &damWallids, phys); >>>>> >>>>> Ofcourse, in the function PhysicsBoundary_Dam_Wall() I set the vector value of [uh] to 0. When I do run the program, I do not see any of this in play (i.e., the values are not being constrained). >>>>> >>>>> Is there a smarter way to go about this or would you suggest a correction in my current method? >>>>> >>>>> Thank you in advance for your wonderful suggestions and work! >>>>> >>>>> Regards, >>>>> >>>>> Mukkund Sunjii >>> From mail2amneet at gmail.com Mon Jun 8 19:09:41 2020 From: mail2amneet at gmail.com (Amneet Bhalla) Date: Mon, 8 Jun 2020 17:09:41 -0700 Subject: [petsc-users] Calling MATLAB code within a parallel C/C++ code Message-ID: Hi Folks, My parallel application written in C++ (which also uses PETSc library for linear solvers) requires calling control and optimization toolbox of MATLAB at every time step to get a small number of double values. Likewise the MATLAB function will need to know some values from the parallel C++ application to return those doubles. I'm wondering if there is a recommended way of calling MATLAB from parallel C/C++/Fortran codes, and if PETSc has some interface that could be leveraged for this exchange. Thanks, -- --Amneet -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Mon Jun 8 19:51:55 2020 From: bsmith at petsc.dev (Barry Smith) Date: Mon, 8 Jun 2020 19:51:55 -0500 Subject: [petsc-users] Calling MATLAB code within a parallel C/C++ code In-Reply-To: References: Message-ID: <028B5BA9-4ECA-44B9-BE51-E24000E50185@petsc.dev> You should use the PetscMatlabEngine https://www.mcs.anl.gov/petsc/petsc-3.13/docs/manualpages/Sys/PetscMatlabEngine.html Barry > On Jun 8, 2020, at 7:09 PM, Amneet Bhalla wrote: > > Hi Folks, > > My parallel application written in C++ (which also uses PETSc library for linear solvers) requires calling control and optimization toolbox of MATLAB at every time step to get a small number of double values. Likewise the MATLAB function will need to know some values from the parallel C++ application to return those doubles. I'm wondering if there is a recommended way of calling MATLAB from parallel C/C++/Fortran codes, and if PETSc has some interface that could be leveraged for this exchange. > > Thanks, > -- > --Amneet > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mail2amneet at gmail.com Mon Jun 8 20:26:54 2020 From: mail2amneet at gmail.com (Amneet Bhalla) Date: Mon, 8 Jun 2020 18:26:54 -0700 Subject: [petsc-users] Calling MATLAB code within a parallel C/C++ code In-Reply-To: <028B5BA9-4ECA-44B9-BE51-E24000E50185@petsc.dev> References: <028B5BA9-4ECA-44B9-BE51-E24000E50185@petsc.dev> Message-ID: Great! Clicking on some of the functions pointed me to this example https://www.mcs.anl.gov/petsc/petsc-3.13/src/snes/tutorials/ex5.c.html Are there more examples, which are perhaps simpler than the above, which I can refer to? On Mon, Jun 8, 2020 at 5:51 PM Barry Smith wrote: > > You should use the PetscMatlabEngine > https://www.mcs.anl.gov/petsc/petsc-3.13/docs/manualpages/Sys/PetscMatlabEngine.html > > Barry > > > On Jun 8, 2020, at 7:09 PM, Amneet Bhalla wrote: > > Hi Folks, > > My parallel application written in C++ (which also uses PETSc library for > linear solvers) requires calling control and optimization toolbox of MATLAB > at every time step to get a small number of double values. Likewise the > MATLAB function will need to know some values from the parallel C++ > application to return those doubles. I'm wondering if there is a > recommended way of calling MATLAB from parallel C/C++/Fortran codes, and if > PETSc has some interface that could be leveraged for this exchange. > > Thanks, > -- > --Amneet > > > > > -- --Amneet -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Mon Jun 8 20:39:00 2020 From: balay at mcs.anl.gov (Satish Balay) Date: Mon, 8 Jun 2020 20:39:00 -0500 (CDT) Subject: [petsc-users] Calling MATLAB code within a parallel C/C++ code In-Reply-To: References: <028B5BA9-4ECA-44B9-BE51-E24000E50185@petsc.dev> Message-ID: there is src/tao/leastsquares/tutorials/matlab/matlab_ls_test.c also, src/vec/vec/tests/ex31.c has matlab calls - but isn't getting tested - so likely broken. Satish On Mon, 8 Jun 2020, Amneet Bhalla wrote: > Great! > > Clicking on some of the functions pointed me to this example > https://www.mcs.anl.gov/petsc/petsc-3.13/src/snes/tutorials/ex5.c.html > > Are there more examples, which are perhaps simpler than the above, which I > can refer to? > > On Mon, Jun 8, 2020 at 5:51 PM Barry Smith wrote: > > > > > You should use the PetscMatlabEngine > > https://www.mcs.anl.gov/petsc/petsc-3.13/docs/manualpages/Sys/PetscMatlabEngine.html > > > > Barry > > > > > > On Jun 8, 2020, at 7:09 PM, Amneet Bhalla wrote: > > > > Hi Folks, > > > > My parallel application written in C++ (which also uses PETSc library for > > linear solvers) requires calling control and optimization toolbox of MATLAB > > at every time step to get a small number of double values. Likewise the > > MATLAB function will need to know some values from the parallel C++ > > application to return those doubles. I'm wondering if there is a > > recommended way of calling MATLAB from parallel C/C++/Fortran codes, and if > > PETSc has some interface that could be leveraged for this exchange. > > > > Thanks, > > -- > > --Amneet > > > > > > > > > > > > From bsmith at petsc.dev Mon Jun 8 20:39:13 2020 From: bsmith at petsc.dev (Barry Smith) Date: Mon, 8 Jun 2020 20:39:13 -0500 Subject: [petsc-users] Calling MATLAB code within a parallel C/C++ code In-Reply-To: References: <028B5BA9-4ECA-44B9-BE51-E24000E50185@petsc.dev> Message-ID: <9BE0A2F9-24DF-452B-AABB-1B8FAA748181@petsc.dev> Looks like the tool to list the examples in the manual pages is broken since it didn't find these other examples, I'll take a look at it. There is also a discussion in the users manual $ git grep MatlabEngine | grep ex src/docs/tex/manual/part2.tex:PetscMatlabEngineCreate(MPI_Comm comm,char *machine,PetscMatlabEngine *e); src/docs/tex/manual/part2.tex:PetscMatlabEnginePut(PetscMatlabEngine e,PetscObject obj); src/docs/tex/manual/part2.tex:PetscMatlabEngineGet(PetscMatlabEngine e,PetscObject obj); src/docs/tex/manual/part2.tex:PetscMatlabEnginePutArray(PetscMatlabEngine e,int m,int n,PetscScalar *array,char *name); src/docs/tex/manual/part2.tex:PetscMatlabEngineGetArray(PetscMatlabEngine e,int m,int n,PetscScalar *array,char *name); src/docs/tex/manual/part2.tex:PetscMatlabEngineEvaluate(PetscMatlabEngine,"format",...); src/docs/tex/manual/part2.tex:PetscMatlabEngineEvaluate(PetscMatlabEngine,"x = \%g *y + z;",avalue); src/docs/tex/manual/part2.tex:PetscMatlabEngineGetOutput(PetscMatlabEngine,char **); src/docs/tex/manual/part2.tex:PetscMatlabEnginedPrintOutput(PetscMatlabEngine,FILE*). src/snes/tutorials/ex5.c: ierr = PetscMatlabEnginePut(PETSC_MATLAB_ENGINE_(comm),(PetscObject)localX);CHKERRQ(ierr); src/snes/tutorials/ex5.c: ierr = PetscMatlabEngineEvaluate(PETSC_MATLAB_ENGINE_(comm),"localF=ex5m(localX,%18.16e,%18.16e,%18.16e)",hx,hy,lambda);CHKERRQ(ierr); src/snes/tutorials/ex5.c: ierr = PetscMatlabEngineGet(PETSC_MATLAB_ENGINE_(comm),(PetscObject)localF);CHKERRQ(ierr); src/sys/tests/ex16.c:static char help[] = "Demonstrates PetscMatlabEngineXXX()\n"; src/sys/tests/ex16.c: ierr = PetscMatlabEngineGetOutput(PETSC_MATLAB_ENGINE_(PETSC_COMM_WORLD),&output);CHKERRQ(ierr); src/sys/tests/ex16.c: ierr = PetscMatlabEngineEvaluate(PETSC_MATLAB_ENGINE_(PETSC_COMM_WORLD),"MPI_Comm_rank");CHKERRQ(ierr); src/sys/tests/ex16.c: ierr = PetscMatlabEngineEvaluate(PETSC_MATLAB_ENGINE_(PETSC_COMM_WORLD),user);CHKERRQ(ierr); src/vec/vec/tests/ex31.c:static const char help[] = "Demonstrates PetscMatlabEngineXXX()\n"; src/vec/vec/tests/ex31.c: ierr = PetscMatlabEngineGetOutput(PETSC_MATLAB_ENGINE_WORLD,&output); src/vec/vec/tests/ex31.c: ierr = PetscMatlabEngineEvaluate(PETSC_MATLAB_ENGINE_WORLD,"MPI_Comm_rank"); src/vec/vec/tests/ex31.c: ierr = PetscMatlabEnginePut(PETSC_MATLAB_ENGINE_WORLD,(PetscObject)x);CHKERRQ(ierr); src/vec/vec/tests/ex31.c: ierr = PetscMatlabEngineEvaluate(PETSC_MATLAB_ENGINE_WORLD,"x = x + MPI_Comm_rank;\n"); src/vec/vec/tests/ex31.c: ierr = PetscMatlabEngineGet(PETSC_MATLAB_ENGINE_WORLD,(PetscObject)x);CHKERRQ(ierr); src/vec/vec/tests/ex31.c: ierr = PetscMatlabEngineEvaluate(PETSC_MATLAB_ENGINE_WORLD,"whos\n");CHKERRQ(ierr); > On Jun 8, 2020, at 8:26 PM, Amneet Bhalla wrote: > > Great! > > Clicking on some of the functions pointed me to this example > https://www.mcs.anl.gov/petsc/petsc-3.13/src/snes/tutorials/ex5.c.html > > Are there more examples, which are perhaps simpler than the above, which I can refer to? > > On Mon, Jun 8, 2020 at 5:51 PM Barry Smith > wrote: > > You should use the PetscMatlabEngine https://www.mcs.anl.gov/petsc/petsc-3.13/docs/manualpages/Sys/PetscMatlabEngine.html > > Barry > > >> On Jun 8, 2020, at 7:09 PM, Amneet Bhalla > wrote: >> >> Hi Folks, >> >> My parallel application written in C++ (which also uses PETSc library for linear solvers) requires calling control and optimization toolbox of MATLAB at every time step to get a small number of double values. Likewise the MATLAB function will need to know some values from the parallel C++ application to return those doubles. I'm wondering if there is a recommended way of calling MATLAB from parallel C/C++/Fortran codes, and if PETSc has some interface that could be leveraged for this exchange. >> >> Thanks, >> -- >> --Amneet >> >> >> > > > > -- > --Amneet > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fdkong.jd at gmail.com Tue Jun 9 15:43:31 2020 From: fdkong.jd at gmail.com (Fande Kong) Date: Tue, 9 Jun 2020 14:43:31 -0600 Subject: [petsc-users] Make stream Message-ID: Hi All, I am trying to interpret the results from "make stream" on two compute nodes, where each node has 48 cores. If my calculations are memory bandwidth limited, such as AMG, MatVec, GMRES, etc.. The best speedup I could get is 16.6938 if I start from one core?? The speedup for function evaluations and Jacobian evaluations can be better than16.6938? Thanks, Fande, Running streams with 'mpiexec ' using 'NPMAX=96' 1 19412.4570 Rate (MB/s) 2 29457.3988 Rate (MB/s) 1.51744 3 40483.9318 Rate (MB/s) 2.08546 4 51429.3431 Rate (MB/s) 2.64929 5 59849.5168 Rate (MB/s) 3.08304 6 66124.3461 Rate (MB/s) 3.40628 7 70888.1170 Rate (MB/s) 3.65167 8 73436.2374 Rate (MB/s) 3.78294 9 77441.7622 Rate (MB/s) 3.98927 10 78115.3114 Rate (MB/s) 4.02397 11 81449.3315 Rate (MB/s) 4.19572 12 82812.3471 Rate (MB/s) 4.26593 13 81442.2114 Rate (MB/s) 4.19535 14 83404.1657 Rate (MB/s) 4.29642 15 84165.8536 Rate (MB/s) 4.33565 16 83739.2910 Rate (MB/s) 4.31368 17 83724.8109 Rate (MB/s) 4.31293 18 83225.0743 Rate (MB/s) 4.28719 19 81668.2002 Rate (MB/s) 4.20699 20 83678.8007 Rate (MB/s) 4.31056 21 81400.4590 Rate (MB/s) 4.1932 22 81944.8975 Rate (MB/s) 4.22124 23 81359.8615 Rate (MB/s) 4.19111 24 80674.5064 Rate (MB/s) 4.1558 25 83761.3316 Rate (MB/s) 4.31481 26 87567.4876 Rate (MB/s) 4.51088 27 89605.4435 Rate (MB/s) 4.61586 28 94984.9755 Rate (MB/s) 4.89298 29 98260.5283 Rate (MB/s) 5.06171 30 99852.8790 Rate (MB/s) 5.14374 31 102736.3576 Rate (MB/s) 5.29228 32 108638.7488 Rate (MB/s) 5.59633 33 110431.2938 Rate (MB/s) 5.68867 34 112824.2031 Rate (MB/s) 5.81194 35 116908.3009 Rate (MB/s) 6.02232 36 121312.6574 Rate (MB/s) 6.2492 37 122507.3172 Rate (MB/s) 6.31074 38 127456.2504 Rate (MB/s) 6.56568 39 130098.7056 Rate (MB/s) 6.7018 40 134956.4461 Rate (MB/s) 6.95204 41 138309.2465 Rate (MB/s) 7.12475 42 141779.7997 Rate (MB/s) 7.30353 43 145653.3687 Rate (MB/s) 7.50307 44 149131.2087 Rate (MB/s) 7.68223 45 151611.6104 Rate (MB/s) 7.81 46 155554.6394 Rate (MB/s) 8.01312 47 159033.1938 Rate (MB/s) 8.19231 48 162216.5600 Rate (MB/s) 8.35629 49 165034.8116 Rate (MB/s) 8.50147 50 168001.4823 Rate (MB/s) 8.65429 51 170899.9045 Rate (MB/s) 8.8036 52 175687.8033 Rate (MB/s) 9.05024 53 178203.9203 Rate (MB/s) 9.17985 54 179973.3914 Rate (MB/s) 9.27101 55 182207.3495 Rate (MB/s) 9.38608 56 185712.9643 Rate (MB/s) 9.56667 57 188805.5696 Rate (MB/s) 9.72598 58 193360.9158 Rate (MB/s) 9.96064 59 198160.8016 Rate (MB/s) 10.2079 60 201297.0129 Rate (MB/s) 10.3695 61 203618.7672 Rate (MB/s) 10.4891 62 209599.2783 Rate (MB/s) 10.7971 63 211651.1587 Rate (MB/s) 10.9028 64 210254.5035 Rate (MB/s) 10.8309 65 218576.4938 Rate (MB/s) 11.2596 66 220280.0853 Rate (MB/s) 11.3473 67 221281.1867 Rate (MB/s) 11.3989 68 228941.1872 Rate (MB/s) 11.7935 69 232206.2708 Rate (MB/s) 11.9617 70 233569.5866 Rate (MB/s) 12.0319 71 238293.6355 Rate (MB/s) 12.2753 72 238987.0729 Rate (MB/s) 12.311 73 246013.4684 Rate (MB/s) 12.6729 74 248850.8942 Rate (MB/s) 12.8191 75 249355.6899 Rate (MB/s) 12.8451 76 252515.6110 Rate (MB/s) 13.0079 77 257489.4268 Rate (MB/s) 13.2641 78 260884.2771 Rate (MB/s) 13.439 79 264341.8661 Rate (MB/s) 13.6171 80 269329.1376 Rate (MB/s) 13.874 81 272286.4070 Rate (MB/s) 14.0263 82 273325.7822 Rate (MB/s) 14.0799 83 277334.6699 Rate (MB/s) 14.2864 84 280254.7286 Rate (MB/s) 14.4368 85 282219.8194 Rate (MB/s) 14.538 86 289039.2677 Rate (MB/s) 14.8893 87 291234.4715 Rate (MB/s) 15.0024 88 295941.1159 Rate (MB/s) 15.2449 89 298136.3163 Rate (MB/s) 15.358 90 302820.9080 Rate (MB/s) 15.5993 91 306387.5008 Rate (MB/s) 15.783 92 310127.0223 Rate (MB/s) 15.9756 93 310219.3643 Rate (MB/s) 15.9804 94 317089.5971 Rate (MB/s) 16.3343 95 315457.0938 Rate (MB/s) 16.2502 96 324068.8172 Rate (MB/s) 16.6938 -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Tue Jun 9 16:08:22 2020 From: bsmith at petsc.dev (Barry Smith) Date: Tue, 9 Jun 2020 16:08:22 -0500 Subject: [petsc-users] Make stream In-Reply-To: References: Message-ID: <5364BA02-DF4C-446D-844A-86EC019EB7B2@petsc.dev> You might look at the notes about MPI binding. It might give you a bit better performance. https://www.mcs.anl.gov/petsc/documentation/faq.html#computers The streams is exactly the DAXPY operation so this is the speed up you should expect for VecAXPY() which has 2 loads and 1 store per 1 multipy and 1 add VecDot() has 2 loads per 1 multiply and 1 add but also a global reduction Sparse multiply with AIJ has an integer load, 2 double loads plus 1 store per row with 1 multiply and 1 add plus communication needed for off-process portion Function evaluations often have higher arithmetic intensity so should give a bit higher speedup Jacobian evaluations often have higher arithmetic intensity but they may have MatSetValues() which is slow because no arithmetic intensity just memory motion Barry > On Jun 9, 2020, at 3:43 PM, Fande Kong wrote: > > Hi All, > > I am trying to interpret the results from "make stream" on two compute nodes, where each node has 48 cores. > > If my calculations are memory bandwidth limited, such as AMG, MatVec, GMRES, etc.. > The best speedup I could get is 16.6938 if I start from one core?? The speedup for function evaluations and Jacobian evaluations can be better than16.6938? > > Thanks, > > Fande, > > > > Running streams with 'mpiexec ' using 'NPMAX=96' > 1 19412.4570 Rate (MB/s) > 2 29457.3988 Rate (MB/s) 1.51744 > 3 40483.9318 Rate (MB/s) 2.08546 > 4 51429.3431 Rate (MB/s) 2.64929 > 5 59849.5168 Rate (MB/s) 3.08304 > 6 66124.3461 Rate (MB/s) 3.40628 > 7 70888.1170 Rate (MB/s) 3.65167 > 8 73436.2374 Rate (MB/s) 3.78294 > 9 77441.7622 Rate (MB/s) 3.98927 > 10 78115.3114 Rate (MB/s) 4.02397 > 11 81449.3315 Rate (MB/s) 4.19572 > 12 82812.3471 Rate (MB/s) 4.26593 > 13 81442.2114 Rate (MB/s) 4.19535 > 14 83404.1657 Rate (MB/s) 4.29642 > 15 84165.8536 Rate (MB/s) 4.33565 > 16 83739.2910 Rate (MB/s) 4.31368 > 17 83724.8109 Rate (MB/s) 4.31293 > 18 83225.0743 Rate (MB/s) 4.28719 > 19 81668.2002 Rate (MB/s) 4.20699 > 20 83678.8007 Rate (MB/s) 4.31056 > 21 81400.4590 Rate (MB/s) 4.1932 > 22 81944.8975 Rate (MB/s) 4.22124 > 23 81359.8615 Rate (MB/s) 4.19111 > 24 80674.5064 Rate (MB/s) 4.1558 > 25 83761.3316 Rate (MB/s) 4.31481 > 26 87567.4876 Rate (MB/s) 4.51088 > 27 89605.4435 Rate (MB/s) 4.61586 > 28 94984.9755 Rate (MB/s) 4.89298 > 29 98260.5283 Rate (MB/s) 5.06171 > 30 99852.8790 Rate (MB/s) 5.14374 > 31 102736.3576 Rate (MB/s) 5.29228 > 32 108638.7488 Rate (MB/s) 5.59633 > 33 110431.2938 Rate (MB/s) 5.68867 > 34 112824.2031 Rate (MB/s) 5.81194 > 35 116908.3009 Rate (MB/s) 6.02232 > 36 121312.6574 Rate (MB/s) 6.2492 > 37 122507.3172 Rate (MB/s) 6.31074 > 38 127456.2504 Rate (MB/s) 6.56568 > 39 130098.7056 Rate (MB/s) 6.7018 > 40 134956.4461 Rate (MB/s) 6.95204 > 41 138309.2465 Rate (MB/s) 7.12475 > 42 141779.7997 Rate (MB/s) 7.30353 > 43 145653.3687 Rate (MB/s) 7.50307 > 44 149131.2087 Rate (MB/s) 7.68223 > 45 151611.6104 Rate (MB/s) 7.81 > 46 155554.6394 Rate (MB/s) 8.01312 > 47 159033.1938 Rate (MB/s) 8.19231 > 48 162216.5600 Rate (MB/s) 8.35629 > 49 165034.8116 Rate (MB/s) 8.50147 > 50 168001.4823 Rate (MB/s) 8.65429 > 51 170899.9045 Rate (MB/s) 8.8036 > 52 175687.8033 Rate (MB/s) 9.05024 > 53 178203.9203 Rate (MB/s) 9.17985 > 54 179973.3914 Rate (MB/s) 9.27101 > 55 182207.3495 Rate (MB/s) 9.38608 > 56 185712.9643 Rate (MB/s) 9.56667 > 57 188805.5696 Rate (MB/s) 9.72598 > 58 193360.9158 Rate (MB/s) 9.96064 > 59 198160.8016 Rate (MB/s) 10.2079 > 60 201297.0129 Rate (MB/s) 10.3695 > 61 203618.7672 Rate (MB/s) 10.4891 > 62 209599.2783 Rate (MB/s) 10.7971 > 63 211651.1587 Rate (MB/s) 10.9028 > 64 210254.5035 Rate (MB/s) 10.8309 > 65 218576.4938 Rate (MB/s) 11.2596 > 66 220280.0853 Rate (MB/s) 11.3473 > 67 221281.1867 Rate (MB/s) 11.3989 > 68 228941.1872 Rate (MB/s) 11.7935 > 69 232206.2708 Rate (MB/s) 11.9617 > 70 233569.5866 Rate (MB/s) 12.0319 > 71 238293.6355 Rate (MB/s) 12.2753 > 72 238987.0729 Rate (MB/s) 12.311 > 73 246013.4684 Rate (MB/s) 12.6729 > 74 248850.8942 Rate (MB/s) 12.8191 > 75 249355.6899 Rate (MB/s) 12.8451 > 76 252515.6110 Rate (MB/s) 13.0079 > 77 257489.4268 Rate (MB/s) 13.2641 > 78 260884.2771 Rate (MB/s) 13.439 > 79 264341.8661 Rate (MB/s) 13.6171 > 80 269329.1376 Rate (MB/s) 13.874 > 81 272286.4070 Rate (MB/s) 14.0263 > 82 273325.7822 Rate (MB/s) 14.0799 > 83 277334.6699 Rate (MB/s) 14.2864 > 84 280254.7286 Rate (MB/s) 14.4368 > 85 282219.8194 Rate (MB/s) 14.538 > 86 289039.2677 Rate (MB/s) 14.8893 > 87 291234.4715 Rate (MB/s) 15.0024 > 88 295941.1159 Rate (MB/s) 15.2449 > 89 298136.3163 Rate (MB/s) 15.358 > 90 302820.9080 Rate (MB/s) 15.5993 > 91 306387.5008 Rate (MB/s) 15.783 > 92 310127.0223 Rate (MB/s) 15.9756 > 93 310219.3643 Rate (MB/s) 15.9804 > 94 317089.5971 Rate (MB/s) 16.3343 > 95 315457.0938 Rate (MB/s) 16.2502 > 96 324068.8172 Rate (MB/s) 16.6938 -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Tue Jun 9 16:19:15 2020 From: jed at jedbrown.org (Jed Brown) Date: Tue, 09 Jun 2020 15:19:15 -0600 Subject: [petsc-users] Make stream In-Reply-To: References: Message-ID: <874krkdjjg.fsf@jedbrown.org> Fande Kong writes: > Hi All, > > I am trying to interpret the results from "make stream" on two compute > nodes, where each node has 48 cores. > > If my calculations are memory bandwidth limited, such as AMG, MatVec, > GMRES, etc.. There's a lot more to AMG setup than memory bandwidth (architecture matters a lot, even between different generation CPUs). MatMult and Krylov are almost pure bandwidth. > The best speedup I could get is 16.6938 if I start from one core?? The > speedup for function evaluations and Jacobian evaluations can be better > than16.6938? Residual and Jacobians can be faster, especially if your code is slow (poorly vectorized, branchy, or has a lot of arithmetic). Are you trying to understand perf on current hardware or make decisions about new hardware? From mail2amneet at gmail.com Tue Jun 9 16:20:57 2020 From: mail2amneet at gmail.com (Amneet Bhalla) Date: Tue, 9 Jun 2020 14:20:57 -0700 Subject: [petsc-users] Calling MATLAB code within a parallel C/C++ code In-Reply-To: <9BE0A2F9-24DF-452B-AABB-1B8FAA748181@petsc.dev> References: <028B5BA9-4ECA-44B9-BE51-E24000E50185@petsc.dev> <9BE0A2F9-24DF-452B-AABB-1B8FAA748181@petsc.dev> Message-ID: Do these additional configure flags for MATLAB look OK? --with-matlab=1 --with-matlab-dir=/Applications/MATLAB_R2019b.app --with-matlab-engine=1 --with-matlab-engine-dir=/Applications/MATLAB_R2019b.app/extern/engines/ With this configuration make compiled files like: CC darwin-dbg/obj/sys/classes/viewer/impls/matlab/vmatlab.o CC darwin-dbg/obj/sys/classes/viewer/impls/matlab/ftn-custom/zvmatlabf.o CC darwin-dbg/obj/sys/classes/matlabengine/matlab.o CC darwin-dbg/obj/sys/classes/matlabengine/ftn-auto/matlabf.o CC darwin-dbg/obj/sys/classes/matlabengine/ftn-custom/zmatlabf.o CC darwin-dbg/obj/vec/pf/impls/matlab/cmatlab.o CC darwin-dbg/obj/vec/vec/utils/matlab/gcreatev.o make[2]: Leaving directory `/Users/amneetb/Softwares/PETSc-Gitlab/PETSc' BEGINNING TO COMPILE MATLAB INTERFACE Building with 'Xcode with Clang'. ld: warning: cannot export hidden symbol _mexfilerequiredapiversion from c_mexapi_version.o ld: warning: cannot export hidden symbol _mexfilerequiredapiversion from c_mexapi_version.o MEX completed successfully. Building with 'Xcode with Clang'. ld: warning: cannot export hidden symbol _mexfilerequiredapiversion from c_mexapi_version.o ld: warning: cannot export hidden symbol _mexfilerequiredapiversion from c_mexapi_version.o MEX completed successfully. Building with 'Xcode with Clang'. ld: warning: cannot export hidden symbol _mexfilerequiredapiversion from c_mexapi_version.o MEX completed successfully. Building with 'Xcode with Clang'. ld: warning: cannot export hidden symbol _mexfilerequiredapiversion from c_mexapi_version.o ld: warning: cannot export hidden symbol _mexfilerequiredapiversion from c_mexapi_version.o MEX completed successfully. However, make check gives error amneetb at Amneets-MBP:~/Softwares/PETSc-Gitlab/PETSc$ make PETSC_DIR=/Users/amneetb/Softwares/PETSc-Gitlab/PETSc PETSC_ARCH=darwin-dbg check Running test examples to verify correct installation Using PETSC_DIR=/Users/amneetb/Softwares/PETSc-Gitlab/PETSc and PETSC_ARCH=darwin-dbg Possible error running C/C++ src/snes/examples/tutorials/ex19 with 1 MPI process See http://www.mcs.anl.gov/petsc/documentation/faq.html dyld: Library not loaded: @rpath/libeng.dylib Referenced from: /Users/amneetb/Softwares/PETSc-Gitlab/PETSc/src/snes/examples/tutorials/./ex19 Reason: image not found =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = PID 14057 RUNNING AT Amneets-MBP = EXIT CODE: 6 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES =================================================================================== YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Abort trap: 6 (signal 6) This typically refers to a problem with your application. Please see the FAQ page for debugging suggestions Possible error running C/C++ src/snes/examples/tutorials/ex19 with 2 MPI processes See http://www.mcs.anl.gov/petsc/documentation/faq.html dyld: Library not loaded: @rpath/libeng.dylib Referenced from: /Users/amneetb/Softwares/PETSc-Gitlab/PETSc/src/snes/examples/tutorials/./ex19 Reason: image not found dyld: Library not loaded: @rpath/libeng.dylib Referenced from: /Users/amneetb/Softwares/PETSc-Gitlab/PETSc/src/snes/examples/tutorials/./ex19 Reason: image not found =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = PID 14062 RUNNING AT Amneets-MBP = EXIT CODE: 6 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES =================================================================================== YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Abort trap: 6 (signal 6) This typically refers to a problem with your application. Please see the FAQ page for debugging suggestions 1,5c1,17 < lid velocity = 0.0016, prandtl # = 1., grashof # = 1. < 0 SNES Function norm 0.0406612 < 1 SNES Function norm 4.12227e-06 < 2 SNES Function norm 6.098e-11 < Number of SNES iterations = 2 --- > dyld: Library not loaded: @rpath/libeng.dylib > Referenced from: /Users/amneetb/Softwares/PETSc-Gitlab/PETSc/src/snes/examples/tutorials/./ex19 > Reason: image not found > dyld: Library not loaded: @rpath/libeng.dylib > Referenced from: /Users/amneetb/Softwares/PETSc-Gitlab/PETSc/src/snes/examples/tutorials/./ex19 > Reason: image not found > > =================================================================================== > = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES > = PID 14072 RUNNING AT Amneets-MBP > = EXIT CODE: 6 > = CLEANING UP REMAINING PROCESSES > = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES > =================================================================================== > YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Abort trap: 6 (signal 6) > This typically refers to a problem with your application. > Please see the FAQ page for debugging suggestions /Users/amneetb/Softwares/PETSc-Gitlab/PETSc/src/snes/examples/tutorials Possible problem with ex19 running with hypre, diffs above ========================================= Possible error running Fortran example src/snes/examples/tutorials/ex5f with 1 MPI process See http://www.mcs.anl.gov/petsc/documentation/faq.html dyld: Library not loaded: @rpath/libeng.dylib Referenced from: /Users/amneetb/Softwares/PETSc-Gitlab/PETSc/src/snes/examples/tutorials/./ex5f Reason: image not found =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = PID 14211 RUNNING AT Amneets-MBP = EXIT CODE: 6 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES =================================================================================== YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Abort trap: 6 (signal 6) This typically refers to a problem with your application. Please see the FAQ page for debugging suggestions Completed test examples I have added the following paths in my .profile and sourced it before make all and make check export DYLD_FALLBACK_LIBRARY_PATH= /Applications/MATLAB_R2019b.app/bin/maci64:/Applications/MATLAB_R2019b.app/sys/os/maci64: $DYLD_FALLBACK_LIBRARY_PATH export DYLD_LIBRARY_PATH= /Applications/MATLAB_R2019b.app/bin/maci64:/Applications/MATLAB_R2019b.app/sys/os/maci64: $DYLD_LIBRARY_PATH On Mon, Jun 8, 2020 at 6:39 PM Barry Smith wrote: > > Looks like the tool to list the examples in the manual pages is broken > since it didn't find these other examples, I'll take a look at it. > > There is also a discussion in the users manual > > $ git grep MatlabEngine | grep ex > src/docs/tex/manual/part2.tex:PetscMatlabEngineCreate(MPI_Comm comm,char > *machine,PetscMatlabEngine *e); > src/docs/tex/manual/part2.tex:PetscMatlabEnginePut(PetscMatlabEngine > e,PetscObject obj); > src/docs/tex/manual/part2.tex:PetscMatlabEngineGet(PetscMatlabEngine > e,PetscObject obj); > src/docs/tex/manual/part2.tex:PetscMatlabEnginePutArray(PetscMatlabEngine > e,int m,int n,PetscScalar *array,char *name); > src/docs/tex/manual/part2.tex:PetscMatlabEngineGetArray(PetscMatlabEngine > e,int m,int n,PetscScalar *array,char *name); > > src/docs/tex/manual/part2.tex:PetscMatlabEngineEvaluate(PetscMatlabEngine,"format",...); > src/docs/tex/manual/part2.tex:PetscMatlabEngineEvaluate(PetscMatlabEngine,"x > = \%g *y + z;",avalue); > src/docs/tex/manual/part2.tex:PetscMatlabEngineGetOutput(PetscMatlabEngine,char > **); > > src/docs/tex/manual/part2.tex:PetscMatlabEnginedPrintOutput(PetscMatlabEngine,FILE*). > src/snes/tutorials/ex5.c: ierr = > PetscMatlabEnginePut(PETSC_MATLAB_ENGINE_(comm),(PetscObject)localX);CHKERRQ(ierr); > src/snes/tutorials/ex5.c: ierr = > PetscMatlabEngineEvaluate(PETSC_MATLAB_ENGINE_(comm),"localF=ex5m(localX,%18.16e,%18.16e,%18.16e)",hx,hy,lambda);CHKERRQ(ierr); > src/snes/tutorials/ex5.c: ierr = > PetscMatlabEngineGet(PETSC_MATLAB_ENGINE_(comm),(PetscObject)localF);CHKERRQ(ierr); > src/sys/tests/ex16.c:static char help[] = "Demonstrates > PetscMatlabEngineXXX()\n"; > src/sys/tests/ex16.c: ierr = > PetscMatlabEngineGetOutput(PETSC_MATLAB_ENGINE_(PETSC_COMM_WORLD),&output);CHKERRQ(ierr); > src/sys/tests/ex16.c: ierr = > PetscMatlabEngineEvaluate(PETSC_MATLAB_ENGINE_(PETSC_COMM_WORLD),"MPI_Comm_rank");CHKERRQ(ierr); > src/sys/tests/ex16.c: ierr = > PetscMatlabEngineEvaluate(PETSC_MATLAB_ENGINE_(PETSC_COMM_WORLD),user);CHKERRQ(ierr); > src/vec/vec/tests/ex31.c:static const char help[] = "Demonstrates > PetscMatlabEngineXXX()\n"; > src/vec/vec/tests/ex31.c: ierr = > PetscMatlabEngineGetOutput(PETSC_MATLAB_ENGINE_WORLD,&output); > src/vec/vec/tests/ex31.c: ierr = > PetscMatlabEngineEvaluate(PETSC_MATLAB_ENGINE_WORLD,"MPI_Comm_rank"); > src/vec/vec/tests/ex31.c: ierr = > PetscMatlabEnginePut(PETSC_MATLAB_ENGINE_WORLD,(PetscObject)x);CHKERRQ(ierr); > src/vec/vec/tests/ex31.c: ierr = > PetscMatlabEngineEvaluate(PETSC_MATLAB_ENGINE_WORLD,"x = x + > MPI_Comm_rank;\n"); > src/vec/vec/tests/ex31.c: ierr = > PetscMatlabEngineGet(PETSC_MATLAB_ENGINE_WORLD,(PetscObject)x);CHKERRQ(ierr); > src/vec/vec/tests/ex31.c: ierr = > PetscMatlabEngineEvaluate(PETSC_MATLAB_ENGINE_WORLD,"whos\n");CHKERRQ(ierr); > > On Jun 8, 2020, at 8:26 PM, Amneet Bhalla wrote: > > Great! > > Clicking on some of the functions pointed me to this example > https://www.mcs.anl.gov/petsc/petsc-3.13/src/snes/tutorials/ex5.c.html > > Are there more examples, which are perhaps simpler than the above, which I > can refer to? > > On Mon, Jun 8, 2020 at 5:51 PM Barry Smith wrote: > >> >> You should use the PetscMatlabEngine >> https://www.mcs.anl.gov/petsc/petsc-3.13/docs/manualpages/Sys/PetscMatlabEngine.html >> >> Barry >> >> >> On Jun 8, 2020, at 7:09 PM, Amneet Bhalla wrote: >> >> Hi Folks, >> >> My parallel application written in C++ (which also uses PETSc library for >> linear solvers) requires calling control and optimization toolbox of MATLAB >> at every time step to get a small number of double values. Likewise the >> MATLAB function will need to know some values from the parallel C++ >> application to return those doubles. I'm wondering if there is a >> recommended way of calling MATLAB from parallel C/C++/Fortran codes, and if >> PETSc has some interface that could be leveraged for this exchange. >> >> Thanks, >> -- >> --Amneet >> >> >> >> >> > > -- > --Amneet > > > > > -- --Amneet -------------- next part -------------- An HTML attachment was scrubbed... URL: From mail2amneet at gmail.com Tue Jun 9 16:26:15 2020 From: mail2amneet at gmail.com (Amneet Bhalla) Date: Tue, 9 Jun 2020 14:26:15 -0700 Subject: [petsc-users] Calling MATLAB code within a parallel C/C++ code In-Reply-To: References: <028B5BA9-4ECA-44B9-BE51-E24000E50185@petsc.dev> <9BE0A2F9-24DF-452B-AABB-1B8FAA748181@petsc.dev> Message-ID: Forgot to mention that configure found the following about MATLAB Matlab: Includes: -I/Applications/MATLAB_R2019b.app/extern/include /Applications/MATLAB_R2019b.app MatlabEngine: Library: -L/Applications/MATLAB_R2019b.app/bin/maci64 -L/Applications/MATLAB_R2019b.app/extern/lib/maci64 -leng -lmex -lmx -lmat -lut -licudata -licui18n -licuuc On Tue, Jun 9, 2020 at 2:20 PM Amneet Bhalla wrote: > Do these additional configure flags for MATLAB look OK? > > --with-matlab=1 --with-matlab-dir=/Applications/MATLAB_R2019b.app > --with-matlab-engine=1 > --with-matlab-engine-dir=/Applications/MATLAB_R2019b.app/extern/engines/ > > > > > With this configuration make compiled files like: > > > CC darwin-dbg/obj/sys/classes/viewer/impls/matlab/vmatlab.o > > CC > darwin-dbg/obj/sys/classes/viewer/impls/matlab/ftn-custom/zvmatlabf.o > > CC darwin-dbg/obj/sys/classes/matlabengine/matlab.o > > CC darwin-dbg/obj/sys/classes/matlabengine/ftn-auto/matlabf.o > > CC darwin-dbg/obj/sys/classes/matlabengine/ftn-custom/zmatlabf.o > > CC darwin-dbg/obj/vec/pf/impls/matlab/cmatlab.o > > CC darwin-dbg/obj/vec/vec/utils/matlab/gcreatev.o > > > make[2]: Leaving directory `/Users/amneetb/Softwares/PETSc-Gitlab/PETSc' > > BEGINNING TO COMPILE MATLAB INTERFACE > > Building with 'Xcode with Clang'. > > ld: warning: cannot export hidden symbol _mexfilerequiredapiversion from > c_mexapi_version.o > > > ld: warning: cannot export hidden symbol _mexfilerequiredapiversion from > c_mexapi_version.o > > > MEX completed successfully. > > Building with 'Xcode with Clang'. > > ld: warning: cannot export hidden symbol _mexfilerequiredapiversion from > c_mexapi_version.o > > > ld: warning: cannot export hidden symbol _mexfilerequiredapiversion from > c_mexapi_version.o > > > MEX completed successfully. > > Building with 'Xcode with Clang'. > > ld: warning: cannot export hidden symbol _mexfilerequiredapiversion from > c_mexapi_version.o > > > MEX completed successfully. > > Building with 'Xcode with Clang'. > > ld: warning: cannot export hidden symbol _mexfilerequiredapiversion from > c_mexapi_version.o > > > ld: warning: cannot export hidden symbol _mexfilerequiredapiversion from > c_mexapi_version.o > > > MEX completed successfully. > > > However, make check gives error > > > amneetb at Amneets-MBP:~/Softwares/PETSc-Gitlab/PETSc$ make > PETSC_DIR=/Users/amneetb/Softwares/PETSc-Gitlab/PETSc PETSC_ARCH=darwin-dbg > check > > Running test examples to verify correct installation > > Using PETSC_DIR=/Users/amneetb/Softwares/PETSc-Gitlab/PETSc and > PETSC_ARCH=darwin-dbg > > Possible error running C/C++ src/snes/examples/tutorials/ex19 with 1 MPI > process > > See http://www.mcs.anl.gov/petsc/documentation/faq.html > > dyld: Library not loaded: @rpath/libeng.dylib > > Referenced from: > /Users/amneetb/Softwares/PETSc-Gitlab/PETSc/src/snes/examples/tutorials/./ex19 > > Reason: image not found > > > > =================================================================================== > > = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES > > = PID 14057 RUNNING AT Amneets-MBP > > = EXIT CODE: 6 > > = CLEANING UP REMAINING PROCESSES > > = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES > > > =================================================================================== > > YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Abort trap: 6 (signal 6) > > This typically refers to a problem with your application. > > Please see the FAQ page for debugging suggestions > > Possible error running C/C++ src/snes/examples/tutorials/ex19 with 2 MPI > processes > > See http://www.mcs.anl.gov/petsc/documentation/faq.html > > dyld: Library not loaded: @rpath/libeng.dylib > > Referenced from: > /Users/amneetb/Softwares/PETSc-Gitlab/PETSc/src/snes/examples/tutorials/./ex19 > > Reason: image not found > > dyld: Library not loaded: @rpath/libeng.dylib > > Referenced from: > /Users/amneetb/Softwares/PETSc-Gitlab/PETSc/src/snes/examples/tutorials/./ex19 > > Reason: image not found > > > > =================================================================================== > > = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES > > = PID 14062 RUNNING AT Amneets-MBP > > = EXIT CODE: 6 > > = CLEANING UP REMAINING PROCESSES > > = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES > > > =================================================================================== > > YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Abort trap: 6 (signal 6) > > This typically refers to a problem with your application. > > Please see the FAQ page for debugging suggestions > > 1,5c1,17 > > < lid velocity = 0.0016, prandtl # = 1., grashof # = 1. > > < 0 SNES Function norm 0.0406612 > > < 1 SNES Function norm 4.12227e-06 > > < 2 SNES Function norm 6.098e-11 > > < Number of SNES iterations = 2 > > --- > > > dyld: Library not loaded: @rpath/libeng.dylib > > > Referenced from: > /Users/amneetb/Softwares/PETSc-Gitlab/PETSc/src/snes/examples/tutorials/./ex19 > > > Reason: image not found > > > dyld: Library not loaded: @rpath/libeng.dylib > > > Referenced from: > /Users/amneetb/Softwares/PETSc-Gitlab/PETSc/src/snes/examples/tutorials/./ex19 > > > Reason: image not found > > > > > > > =================================================================================== > > > = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES > > > = PID 14072 RUNNING AT Amneets-MBP > > > = EXIT CODE: 6 > > > = CLEANING UP REMAINING PROCESSES > > > = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES > > > > =================================================================================== > > > YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Abort trap: 6 (signal > 6) > > > This typically refers to a problem with your application. > > > Please see the FAQ page for debugging suggestions > > /Users/amneetb/Softwares/PETSc-Gitlab/PETSc/src/snes/examples/tutorials > > Possible problem with ex19 running with hypre, diffs above > > ========================================= > > Possible error running Fortran example src/snes/examples/tutorials/ex5f > with 1 MPI process > > See http://www.mcs.anl.gov/petsc/documentation/faq.html > > dyld: Library not loaded: @rpath/libeng.dylib > > Referenced from: > /Users/amneetb/Softwares/PETSc-Gitlab/PETSc/src/snes/examples/tutorials/./ex5f > > Reason: image not found > > > > =================================================================================== > > = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES > > = PID 14211 RUNNING AT Amneets-MBP > > = EXIT CODE: 6 > > = CLEANING UP REMAINING PROCESSES > > = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES > > > =================================================================================== > > YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Abort trap: 6 (signal 6) > > This typically refers to a problem with your application. > > Please see the FAQ page for debugging suggestions > > Completed test examples > > > I have added the following paths in my .profile and sourced it before make > all and make check > > export DYLD_FALLBACK_LIBRARY_PATH= > /Applications/MATLAB_R2019b.app/bin/maci64:/Applications/MATLAB_R2019b.app/sys/os/maci64: > $DYLD_FALLBACK_LIBRARY_PATH > > > export DYLD_LIBRARY_PATH= > /Applications/MATLAB_R2019b.app/bin/maci64:/Applications/MATLAB_R2019b.app/sys/os/maci64: > $DYLD_LIBRARY_PATH > > > > > On Mon, Jun 8, 2020 at 6:39 PM Barry Smith wrote: > >> >> Looks like the tool to list the examples in the manual pages is broken >> since it didn't find these other examples, I'll take a look at it. >> >> There is also a discussion in the users manual >> >> $ git grep MatlabEngine | grep ex >> src/docs/tex/manual/part2.tex:PetscMatlabEngineCreate(MPI_Comm comm,char >> *machine,PetscMatlabEngine *e); >> src/docs/tex/manual/part2.tex:PetscMatlabEnginePut(PetscMatlabEngine >> e,PetscObject obj); >> src/docs/tex/manual/part2.tex:PetscMatlabEngineGet(PetscMatlabEngine >> e,PetscObject obj); >> src/docs/tex/manual/part2.tex:PetscMatlabEnginePutArray(PetscMatlabEngine >> e,int m,int n,PetscScalar *array,char *name); >> src/docs/tex/manual/part2.tex:PetscMatlabEngineGetArray(PetscMatlabEngine >> e,int m,int n,PetscScalar *array,char *name); >> >> src/docs/tex/manual/part2.tex:PetscMatlabEngineEvaluate(PetscMatlabEngine,"format",...); >> src/docs/tex/manual/part2.tex:PetscMatlabEngineEvaluate(PetscMatlabEngine,"x >> = \%g *y + z;",avalue); >> src/docs/tex/manual/part2.tex:PetscMatlabEngineGetOutput(PetscMatlabEngine,char >> **); >> >> src/docs/tex/manual/part2.tex:PetscMatlabEnginedPrintOutput(PetscMatlabEngine,FILE*). >> src/snes/tutorials/ex5.c: ierr = >> PetscMatlabEnginePut(PETSC_MATLAB_ENGINE_(comm),(PetscObject)localX);CHKERRQ(ierr); >> src/snes/tutorials/ex5.c: ierr = >> PetscMatlabEngineEvaluate(PETSC_MATLAB_ENGINE_(comm),"localF=ex5m(localX,%18.16e,%18.16e,%18.16e)",hx,hy,lambda);CHKERRQ(ierr); >> src/snes/tutorials/ex5.c: ierr = >> PetscMatlabEngineGet(PETSC_MATLAB_ENGINE_(comm),(PetscObject)localF);CHKERRQ(ierr); >> src/sys/tests/ex16.c:static char help[] = "Demonstrates >> PetscMatlabEngineXXX()\n"; >> src/sys/tests/ex16.c: ierr = >> PetscMatlabEngineGetOutput(PETSC_MATLAB_ENGINE_(PETSC_COMM_WORLD),&output);CHKERRQ(ierr); >> src/sys/tests/ex16.c: ierr = >> PetscMatlabEngineEvaluate(PETSC_MATLAB_ENGINE_(PETSC_COMM_WORLD),"MPI_Comm_rank");CHKERRQ(ierr); >> src/sys/tests/ex16.c: ierr = >> PetscMatlabEngineEvaluate(PETSC_MATLAB_ENGINE_(PETSC_COMM_WORLD),user);CHKERRQ(ierr); >> src/vec/vec/tests/ex31.c:static const char help[] = "Demonstrates >> PetscMatlabEngineXXX()\n"; >> src/vec/vec/tests/ex31.c: ierr = >> PetscMatlabEngineGetOutput(PETSC_MATLAB_ENGINE_WORLD,&output); >> src/vec/vec/tests/ex31.c: ierr = >> PetscMatlabEngineEvaluate(PETSC_MATLAB_ENGINE_WORLD,"MPI_Comm_rank"); >> src/vec/vec/tests/ex31.c: ierr = >> PetscMatlabEnginePut(PETSC_MATLAB_ENGINE_WORLD,(PetscObject)x);CHKERRQ(ierr); >> src/vec/vec/tests/ex31.c: ierr = >> PetscMatlabEngineEvaluate(PETSC_MATLAB_ENGINE_WORLD,"x = x + >> MPI_Comm_rank;\n"); >> src/vec/vec/tests/ex31.c: ierr = >> PetscMatlabEngineGet(PETSC_MATLAB_ENGINE_WORLD,(PetscObject)x);CHKERRQ(ierr); >> src/vec/vec/tests/ex31.c: ierr = >> PetscMatlabEngineEvaluate(PETSC_MATLAB_ENGINE_WORLD,"whos\n");CHKERRQ(ierr); >> >> On Jun 8, 2020, at 8:26 PM, Amneet Bhalla wrote: >> >> Great! >> >> Clicking on some of the functions pointed me to this example >> https://www.mcs.anl.gov/petsc/petsc-3.13/src/snes/tutorials/ex5.c.html >> >> Are there more examples, which are perhaps simpler than the above, which >> I can refer to? >> >> On Mon, Jun 8, 2020 at 5:51 PM Barry Smith wrote: >> >>> >>> You should use the PetscMatlabEngine >>> https://www.mcs.anl.gov/petsc/petsc-3.13/docs/manualpages/Sys/PetscMatlabEngine.html >>> >>> Barry >>> >>> >>> On Jun 8, 2020, at 7:09 PM, Amneet Bhalla wrote: >>> >>> Hi Folks, >>> >>> My parallel application written in C++ (which also uses PETSc library >>> for linear solvers) requires calling control and optimization toolbox of >>> MATLAB at every time step to get a small number of double values. Likewise >>> the MATLAB function will need to know some values from the parallel C++ >>> application to return those doubles. I'm wondering if there is a >>> recommended way of calling MATLAB from parallel C/C++/Fortran codes, and if >>> PETSc has some interface that could be leveraged for this exchange. >>> >>> Thanks, >>> -- >>> --Amneet >>> >>> >>> >>> >>> >> >> -- >> --Amneet >> >> >> >> >> > > -- > --Amneet > > > > -- --Amneet -------------- next part -------------- An HTML attachment was scrubbed... URL: From junchao.zhang at gmail.com Tue Jun 9 16:47:57 2020 From: junchao.zhang at gmail.com (Junchao Zhang) Date: Tue, 9 Jun 2020 16:47:57 -0500 Subject: [petsc-users] Make stream In-Reply-To: References: Message-ID: Looks something was wrong that you need all cores to reach its bandwidth limit. Run on one node to see what happens. --Junchao Zhang On Tue, Jun 9, 2020 at 3:45 PM Fande Kong wrote: > Hi All, > > I am trying to interpret the results from "make stream" on two compute > nodes, where each node has 48 cores. > > If my calculations are memory bandwidth limited, such as AMG, MatVec, > GMRES, etc.. > The best speedup I could get is 16.6938 if I start from one core?? The > speedup for function evaluations and Jacobian evaluations can be better > than16.6938? > > Thanks, > > Fande, > > > > Running streams with 'mpiexec ' using 'NPMAX=96' > 1 19412.4570 Rate (MB/s) > 2 29457.3988 Rate (MB/s) 1.51744 > 3 40483.9318 Rate (MB/s) 2.08546 > 4 51429.3431 Rate (MB/s) 2.64929 > 5 59849.5168 Rate (MB/s) 3.08304 > 6 66124.3461 Rate (MB/s) 3.40628 > 7 70888.1170 Rate (MB/s) 3.65167 > 8 73436.2374 Rate (MB/s) 3.78294 > 9 77441.7622 Rate (MB/s) 3.98927 > 10 78115.3114 Rate (MB/s) 4.02397 > 11 81449.3315 Rate (MB/s) 4.19572 > 12 82812.3471 Rate (MB/s) 4.26593 > 13 81442.2114 Rate (MB/s) 4.19535 > 14 83404.1657 Rate (MB/s) 4.29642 > 15 84165.8536 Rate (MB/s) 4.33565 > 16 83739.2910 Rate (MB/s) 4.31368 > 17 83724.8109 Rate (MB/s) 4.31293 > 18 83225.0743 Rate (MB/s) 4.28719 > 19 81668.2002 Rate (MB/s) 4.20699 > 20 83678.8007 Rate (MB/s) 4.31056 > 21 81400.4590 Rate (MB/s) 4.1932 > 22 81944.8975 Rate (MB/s) 4.22124 > 23 81359.8615 Rate (MB/s) 4.19111 > 24 80674.5064 Rate (MB/s) 4.1558 > 25 83761.3316 Rate (MB/s) 4.31481 > 26 87567.4876 Rate (MB/s) 4.51088 > 27 89605.4435 Rate (MB/s) 4.61586 > 28 94984.9755 Rate (MB/s) 4.89298 > 29 98260.5283 Rate (MB/s) 5.06171 > 30 99852.8790 Rate (MB/s) 5.14374 > 31 102736.3576 Rate (MB/s) 5.29228 > 32 108638.7488 Rate (MB/s) 5.59633 > 33 110431.2938 Rate (MB/s) 5.68867 > 34 112824.2031 Rate (MB/s) 5.81194 > 35 116908.3009 Rate (MB/s) 6.02232 > 36 121312.6574 Rate (MB/s) 6.2492 > 37 122507.3172 Rate (MB/s) 6.31074 > 38 127456.2504 Rate (MB/s) 6.56568 > 39 130098.7056 Rate (MB/s) 6.7018 > 40 134956.4461 Rate (MB/s) 6.95204 > 41 138309.2465 Rate (MB/s) 7.12475 > 42 141779.7997 Rate (MB/s) 7.30353 > 43 145653.3687 Rate (MB/s) 7.50307 > 44 149131.2087 Rate (MB/s) 7.68223 > 45 151611.6104 Rate (MB/s) 7.81 > 46 155554.6394 Rate (MB/s) 8.01312 > 47 159033.1938 Rate (MB/s) 8.19231 > 48 162216.5600 Rate (MB/s) 8.35629 > 49 165034.8116 Rate (MB/s) 8.50147 > 50 168001.4823 Rate (MB/s) 8.65429 > 51 170899.9045 Rate (MB/s) 8.8036 > 52 175687.8033 Rate (MB/s) 9.05024 > 53 178203.9203 Rate (MB/s) 9.17985 > 54 179973.3914 Rate (MB/s) 9.27101 > 55 182207.3495 Rate (MB/s) 9.38608 > 56 185712.9643 Rate (MB/s) 9.56667 > 57 188805.5696 Rate (MB/s) 9.72598 > 58 193360.9158 Rate (MB/s) 9.96064 > 59 198160.8016 Rate (MB/s) 10.2079 > 60 201297.0129 Rate (MB/s) 10.3695 > 61 203618.7672 Rate (MB/s) 10.4891 > 62 209599.2783 Rate (MB/s) 10.7971 > 63 211651.1587 Rate (MB/s) 10.9028 > 64 210254.5035 Rate (MB/s) 10.8309 > 65 218576.4938 Rate (MB/s) 11.2596 > 66 220280.0853 Rate (MB/s) 11.3473 > 67 221281.1867 Rate (MB/s) 11.3989 > 68 228941.1872 Rate (MB/s) 11.7935 > 69 232206.2708 Rate (MB/s) 11.9617 > 70 233569.5866 Rate (MB/s) 12.0319 > 71 238293.6355 Rate (MB/s) 12.2753 > 72 238987.0729 Rate (MB/s) 12.311 > 73 246013.4684 Rate (MB/s) 12.6729 > 74 248850.8942 Rate (MB/s) 12.8191 > 75 249355.6899 Rate (MB/s) 12.8451 > 76 252515.6110 Rate (MB/s) 13.0079 > 77 257489.4268 Rate (MB/s) 13.2641 > 78 260884.2771 Rate (MB/s) 13.439 > 79 264341.8661 Rate (MB/s) 13.6171 > 80 269329.1376 Rate (MB/s) 13.874 > 81 272286.4070 Rate (MB/s) 14.0263 > 82 273325.7822 Rate (MB/s) 14.0799 > 83 277334.6699 Rate (MB/s) 14.2864 > 84 280254.7286 Rate (MB/s) 14.4368 > 85 282219.8194 Rate (MB/s) 14.538 > 86 289039.2677 Rate (MB/s) 14.8893 > 87 291234.4715 Rate (MB/s) 15.0024 > 88 295941.1159 Rate (MB/s) 15.2449 > 89 298136.3163 Rate (MB/s) 15.358 > 90 302820.9080 Rate (MB/s) 15.5993 > 91 306387.5008 Rate (MB/s) 15.783 > 92 310127.0223 Rate (MB/s) 15.9756 > 93 310219.3643 Rate (MB/s) 15.9804 > 94 317089.5971 Rate (MB/s) 16.3343 > 95 315457.0938 Rate (MB/s) 16.2502 > 96 324068.8172 Rate (MB/s) 16.6938 > -------------- next part -------------- An HTML attachment was scrubbed... URL: From junchao.zhang at gmail.com Tue Jun 9 16:51:05 2020 From: junchao.zhang at gmail.com (Junchao Zhang) Date: Tue, 9 Jun 2020 16:51:05 -0500 Subject: [petsc-users] Calling MATLAB code within a parallel C/C++ code In-Reply-To: References: <028B5BA9-4ECA-44B9-BE51-E24000E50185@petsc.dev> <9BE0A2F9-24DF-452B-AABB-1B8FAA748181@petsc.dev> Message-ID: Find recursively where is libeng.dylib under /Applications/MATLAB_R2019b.app/extern/engines/ --Junchao Zhang On Tue, Jun 9, 2020 at 4:27 PM Amneet Bhalla wrote: > Forgot to mention that configure found the following about MATLAB > > Matlab: > > Includes: -I/Applications/MATLAB_R2019b.app/extern/include > > /Applications/MATLAB_R2019b.app > > MatlabEngine: > > Library: -L/Applications/MATLAB_R2019b.app/bin/maci64 > -L/Applications/MATLAB_R2019b.app/extern/lib/maci64 -leng -lmex -lmx -lmat > -lut -licudata -licui18n -licuuc > > On Tue, Jun 9, 2020 at 2:20 PM Amneet Bhalla > wrote: > >> Do these additional configure flags for MATLAB look OK? >> >> --with-matlab=1 --with-matlab-dir=/Applications/MATLAB_R2019b.app >> --with-matlab-engine=1 >> --with-matlab-engine-dir=/Applications/MATLAB_R2019b.app/extern/engines/ >> >> >> >> >> With this configuration make compiled files like: >> >> >> CC darwin-dbg/obj/sys/classes/viewer/impls/matlab/vmatlab.o >> >> CC >> darwin-dbg/obj/sys/classes/viewer/impls/matlab/ftn-custom/zvmatlabf.o >> >> CC darwin-dbg/obj/sys/classes/matlabengine/matlab.o >> >> CC darwin-dbg/obj/sys/classes/matlabengine/ftn-auto/matlabf.o >> >> CC darwin-dbg/obj/sys/classes/matlabengine/ftn-custom/zmatlabf.o >> >> CC darwin-dbg/obj/vec/pf/impls/matlab/cmatlab.o >> >> CC darwin-dbg/obj/vec/vec/utils/matlab/gcreatev.o >> >> >> make[2]: Leaving directory `/Users/amneetb/Softwares/PETSc-Gitlab/PETSc' >> >> BEGINNING TO COMPILE MATLAB INTERFACE >> >> Building with 'Xcode with Clang'. >> >> ld: warning: cannot export hidden symbol _mexfilerequiredapiversion from >> c_mexapi_version.o >> >> >> ld: warning: cannot export hidden symbol _mexfilerequiredapiversion from >> c_mexapi_version.o >> >> >> MEX completed successfully. >> >> Building with 'Xcode with Clang'. >> >> ld: warning: cannot export hidden symbol _mexfilerequiredapiversion from >> c_mexapi_version.o >> >> >> ld: warning: cannot export hidden symbol _mexfilerequiredapiversion from >> c_mexapi_version.o >> >> >> MEX completed successfully. >> >> Building with 'Xcode with Clang'. >> >> ld: warning: cannot export hidden symbol _mexfilerequiredapiversion from >> c_mexapi_version.o >> >> >> MEX completed successfully. >> >> Building with 'Xcode with Clang'. >> >> ld: warning: cannot export hidden symbol _mexfilerequiredapiversion from >> c_mexapi_version.o >> >> >> ld: warning: cannot export hidden symbol _mexfilerequiredapiversion from >> c_mexapi_version.o >> >> >> MEX completed successfully. >> >> >> However, make check gives error >> >> >> amneetb at Amneets-MBP:~/Softwares/PETSc-Gitlab/PETSc$ make >> PETSC_DIR=/Users/amneetb/Softwares/PETSc-Gitlab/PETSc PETSC_ARCH=darwin-dbg >> check >> >> Running test examples to verify correct installation >> >> Using PETSC_DIR=/Users/amneetb/Softwares/PETSc-Gitlab/PETSc and >> PETSC_ARCH=darwin-dbg >> >> Possible error running C/C++ src/snes/examples/tutorials/ex19 with 1 MPI >> process >> >> See http://www.mcs.anl.gov/petsc/documentation/faq.html >> >> dyld: Library not loaded: @rpath/libeng.dylib >> >> Referenced from: >> /Users/amneetb/Softwares/PETSc-Gitlab/PETSc/src/snes/examples/tutorials/./ex19 >> >> Reason: image not found >> >> >> >> =================================================================================== >> >> = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES >> >> = PID 14057 RUNNING AT Amneets-MBP >> >> = EXIT CODE: 6 >> >> = CLEANING UP REMAINING PROCESSES >> >> = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES >> >> >> =================================================================================== >> >> YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Abort trap: 6 (signal 6) >> >> This typically refers to a problem with your application. >> >> Please see the FAQ page for debugging suggestions >> >> Possible error running C/C++ src/snes/examples/tutorials/ex19 with 2 MPI >> processes >> >> See http://www.mcs.anl.gov/petsc/documentation/faq.html >> >> dyld: Library not loaded: @rpath/libeng.dylib >> >> Referenced from: >> /Users/amneetb/Softwares/PETSc-Gitlab/PETSc/src/snes/examples/tutorials/./ex19 >> >> Reason: image not found >> >> dyld: Library not loaded: @rpath/libeng.dylib >> >> Referenced from: >> /Users/amneetb/Softwares/PETSc-Gitlab/PETSc/src/snes/examples/tutorials/./ex19 >> >> Reason: image not found >> >> >> >> =================================================================================== >> >> = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES >> >> = PID 14062 RUNNING AT Amneets-MBP >> >> = EXIT CODE: 6 >> >> = CLEANING UP REMAINING PROCESSES >> >> = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES >> >> >> =================================================================================== >> >> YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Abort trap: 6 (signal 6) >> >> This typically refers to a problem with your application. >> >> Please see the FAQ page for debugging suggestions >> >> 1,5c1,17 >> >> < lid velocity = 0.0016, prandtl # = 1., grashof # = 1. >> >> < 0 SNES Function norm 0.0406612 >> >> < 1 SNES Function norm 4.12227e-06 >> >> < 2 SNES Function norm 6.098e-11 >> >> < Number of SNES iterations = 2 >> >> --- >> >> > dyld: Library not loaded: @rpath/libeng.dylib >> >> > Referenced from: >> /Users/amneetb/Softwares/PETSc-Gitlab/PETSc/src/snes/examples/tutorials/./ex19 >> >> > Reason: image not found >> >> > dyld: Library not loaded: @rpath/libeng.dylib >> >> > Referenced from: >> /Users/amneetb/Softwares/PETSc-Gitlab/PETSc/src/snes/examples/tutorials/./ex19 >> >> > Reason: image not found >> >> > >> >> > >> =================================================================================== >> >> > = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES >> >> > = PID 14072 RUNNING AT Amneets-MBP >> >> > = EXIT CODE: 6 >> >> > = CLEANING UP REMAINING PROCESSES >> >> > = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES >> >> > >> =================================================================================== >> >> > YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Abort trap: 6 (signal >> 6) >> >> > This typically refers to a problem with your application. >> >> > Please see the FAQ page for debugging suggestions >> >> /Users/amneetb/Softwares/PETSc-Gitlab/PETSc/src/snes/examples/tutorials >> >> Possible problem with ex19 running with hypre, diffs above >> >> ========================================= >> >> Possible error running Fortran example src/snes/examples/tutorials/ex5f >> with 1 MPI process >> >> See http://www.mcs.anl.gov/petsc/documentation/faq.html >> >> dyld: Library not loaded: @rpath/libeng.dylib >> >> Referenced from: >> /Users/amneetb/Softwares/PETSc-Gitlab/PETSc/src/snes/examples/tutorials/./ex5f >> >> Reason: image not found >> >> >> >> =================================================================================== >> >> = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES >> >> = PID 14211 RUNNING AT Amneets-MBP >> >> = EXIT CODE: 6 >> >> = CLEANING UP REMAINING PROCESSES >> >> = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES >> >> >> =================================================================================== >> >> YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Abort trap: 6 (signal 6) >> >> This typically refers to a problem with your application. >> >> Please see the FAQ page for debugging suggestions >> >> Completed test examples >> >> >> I have added the following paths in my .profile and sourced it before >> make all and make check >> >> export DYLD_FALLBACK_LIBRARY_PATH= >> /Applications/MATLAB_R2019b.app/bin/maci64:/Applications/MATLAB_R2019b.app/sys/os/maci64: >> $DYLD_FALLBACK_LIBRARY_PATH >> >> >> export DYLD_LIBRARY_PATH= >> /Applications/MATLAB_R2019b.app/bin/maci64:/Applications/MATLAB_R2019b.app/sys/os/maci64: >> $DYLD_LIBRARY_PATH >> >> >> >> >> On Mon, Jun 8, 2020 at 6:39 PM Barry Smith wrote: >> >>> >>> Looks like the tool to list the examples in the manual pages is broken >>> since it didn't find these other examples, I'll take a look at it. >>> >>> There is also a discussion in the users manual >>> >>> $ git grep MatlabEngine | grep ex >>> src/docs/tex/manual/part2.tex:PetscMatlabEngineCreate(MPI_Comm comm,char >>> *machine,PetscMatlabEngine *e); >>> src/docs/tex/manual/part2.tex:PetscMatlabEnginePut(PetscMatlabEngine >>> e,PetscObject obj); >>> src/docs/tex/manual/part2.tex:PetscMatlabEngineGet(PetscMatlabEngine >>> e,PetscObject obj); >>> src/docs/tex/manual/part2.tex:PetscMatlabEnginePutArray(PetscMatlabEngine >>> e,int m,int n,PetscScalar *array,char *name); >>> src/docs/tex/manual/part2.tex:PetscMatlabEngineGetArray(PetscMatlabEngine >>> e,int m,int n,PetscScalar *array,char *name); >>> >>> src/docs/tex/manual/part2.tex:PetscMatlabEngineEvaluate(PetscMatlabEngine,"format",...); >>> src/docs/tex/manual/part2.tex:PetscMatlabEngineEvaluate(PetscMatlabEngine,"x >>> = \%g *y + z;",avalue); >>> src/docs/tex/manual/part2.tex:PetscMatlabEngineGetOutput(PetscMatlabEngine,char >>> **); >>> >>> src/docs/tex/manual/part2.tex:PetscMatlabEnginedPrintOutput(PetscMatlabEngine,FILE*). >>> src/snes/tutorials/ex5.c: ierr = >>> PetscMatlabEnginePut(PETSC_MATLAB_ENGINE_(comm),(PetscObject)localX);CHKERRQ(ierr); >>> src/snes/tutorials/ex5.c: ierr = >>> PetscMatlabEngineEvaluate(PETSC_MATLAB_ENGINE_(comm),"localF=ex5m(localX,%18.16e,%18.16e,%18.16e)",hx,hy,lambda);CHKERRQ(ierr); >>> src/snes/tutorials/ex5.c: ierr = >>> PetscMatlabEngineGet(PETSC_MATLAB_ENGINE_(comm),(PetscObject)localF);CHKERRQ(ierr); >>> src/sys/tests/ex16.c:static char help[] = "Demonstrates >>> PetscMatlabEngineXXX()\n"; >>> src/sys/tests/ex16.c: ierr = >>> PetscMatlabEngineGetOutput(PETSC_MATLAB_ENGINE_(PETSC_COMM_WORLD),&output);CHKERRQ(ierr); >>> src/sys/tests/ex16.c: ierr = >>> PetscMatlabEngineEvaluate(PETSC_MATLAB_ENGINE_(PETSC_COMM_WORLD),"MPI_Comm_rank");CHKERRQ(ierr); >>> src/sys/tests/ex16.c: ierr = >>> PetscMatlabEngineEvaluate(PETSC_MATLAB_ENGINE_(PETSC_COMM_WORLD),user);CHKERRQ(ierr); >>> src/vec/vec/tests/ex31.c:static const char help[] = "Demonstrates >>> PetscMatlabEngineXXX()\n"; >>> src/vec/vec/tests/ex31.c: ierr = >>> PetscMatlabEngineGetOutput(PETSC_MATLAB_ENGINE_WORLD,&output); >>> src/vec/vec/tests/ex31.c: ierr = >>> PetscMatlabEngineEvaluate(PETSC_MATLAB_ENGINE_WORLD,"MPI_Comm_rank"); >>> src/vec/vec/tests/ex31.c: ierr = >>> PetscMatlabEnginePut(PETSC_MATLAB_ENGINE_WORLD,(PetscObject)x);CHKERRQ(ierr); >>> src/vec/vec/tests/ex31.c: ierr = >>> PetscMatlabEngineEvaluate(PETSC_MATLAB_ENGINE_WORLD,"x = x + >>> MPI_Comm_rank;\n"); >>> src/vec/vec/tests/ex31.c: ierr = >>> PetscMatlabEngineGet(PETSC_MATLAB_ENGINE_WORLD,(PetscObject)x);CHKERRQ(ierr); >>> src/vec/vec/tests/ex31.c: ierr = >>> PetscMatlabEngineEvaluate(PETSC_MATLAB_ENGINE_WORLD,"whos\n");CHKERRQ(ierr); >>> >>> On Jun 8, 2020, at 8:26 PM, Amneet Bhalla wrote: >>> >>> Great! >>> >>> Clicking on some of the functions pointed me to this example >>> https://www.mcs.anl.gov/petsc/petsc-3.13/src/snes/tutorials/ex5.c.html >>> >>> Are there more examples, which are perhaps simpler than the above, which >>> I can refer to? >>> >>> On Mon, Jun 8, 2020 at 5:51 PM Barry Smith wrote: >>> >>>> >>>> You should use the PetscMatlabEngine >>>> https://www.mcs.anl.gov/petsc/petsc-3.13/docs/manualpages/Sys/PetscMatlabEngine.html >>>> >>>> Barry >>>> >>>> >>>> On Jun 8, 2020, at 7:09 PM, Amneet Bhalla >>>> wrote: >>>> >>>> Hi Folks, >>>> >>>> My parallel application written in C++ (which also uses PETSc library >>>> for linear solvers) requires calling control and optimization toolbox of >>>> MATLAB at every time step to get a small number of double values. Likewise >>>> the MATLAB function will need to know some values from the parallel C++ >>>> application to return those doubles. I'm wondering if there is a >>>> recommended way of calling MATLAB from parallel C/C++/Fortran codes, and if >>>> PETSc has some interface that could be leveraged for this exchange. >>>> >>>> Thanks, >>>> -- >>>> --Amneet >>>> >>>> >>>> >>>> >>>> >>> >>> -- >>> --Amneet >>> >>> >>> >>> >>> >> >> -- >> --Amneet >> >> >> >> > > -- > --Amneet > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From s_g at berkeley.edu Tue Jun 9 16:55:24 2020 From: s_g at berkeley.edu (Sanjay Govindjee) Date: Tue, 9 Jun 2020 14:55:24 -0700 Subject: [petsc-users] Calling MATLAB code within a parallel C/C++ code In-Reply-To: References: <028B5BA9-4ECA-44B9-BE51-E24000E50185@petsc.dev> <9BE0A2F9-24DF-452B-AABB-1B8FAA748181@petsc.dev> Message-ID: <943847d1-7b6a-8841-f2f8-91652b2f64f2@berkeley.edu> Try looking in: $(MLROOT)/bin/maci64 where MLROOT = /Applications/MATLAB_R2019a.app or the like On 6/9/20 2:51 PM, Junchao Zhang wrote: > Find recursively?where is libeng.dylib under > /Applications/MATLAB_R2019b.app/extern/engines/ > > --Junchao Zhang > > > On Tue, Jun 9, 2020 at 4:27 PM Amneet Bhalla > wrote: > > Forgot to mention that configure found the following about MATLAB > > Matlab: > > Includes: -I/Applications/MATLAB_R2019b.app/extern/include > > /Applications/MATLAB_R2019b.app > > MatlabEngine: > > Library: -L/Applications/MATLAB_R2019b.app/bin/maci64 > -L/Applications/MATLAB_R2019b.app/extern/lib/maci64 -leng -lmex > -lmx -lmat -lut -licudata -licui18n -licuuc > > > On Tue, Jun 9, 2020 at 2:20 PM Amneet Bhalla > > wrote: > > Do these additional?configure flags for MATLAB look OK? > > --with-matlab=1 > --with-matlab-dir=/Applications/MATLAB_R2019b.app > --with-matlab-engine=1 > --with-matlab-engine-dir=/Applications/MATLAB_R2019b.app/extern/engines/ > > > > > With this configuration make compiled files like: > > > ? ? ? ? ??CC > darwin-dbg/obj/sys/classes/viewer/impls/matlab/vmatlab.o > > ? ? ? ? ? CC > darwin-dbg/obj/sys/classes/viewer/impls/matlab/ftn-custom/zvmatlabf.o > > ? ? ? ? ? CC darwin-dbg/obj/sys/classes/matlabengine/matlab.o > > ? ? ? ? ? CC > darwin-dbg/obj/sys/classes/matlabengine/ftn-auto/matlabf.o > > ? ? ? ? ? CC > darwin-dbg/obj/sys/classes/matlabengine/ftn-custom/zmatlabf.o > > ? ? ? ? ? CC darwin-dbg/obj/vec/pf/impls/matlab/cmatlab.o > > ? ? ? ? ? CC darwin-dbg/obj/vec/vec/utils/matlab/gcreatev.o > > > make[2]: Leaving directory > `/Users/amneetb/Softwares/PETSc-Gitlab/PETSc' > > BEGINNING TO COMPILE MATLAB INTERFACE > > Building with 'Xcode with Clang'. > > ld: warning: cannot export hidden symbol > _mexfilerequiredapiversion from c_mexapi_version.o > > > ld: warning: cannot export hidden symbol > _mexfilerequiredapiversion from c_mexapi_version.o > > > MEX completed successfully. > > Building with 'Xcode with Clang'. > > ld: warning: cannot export hidden symbol > _mexfilerequiredapiversion from c_mexapi_version.o > > > ld: warning: cannot export hidden symbol > _mexfilerequiredapiversion from c_mexapi_version.o > > > MEX completed successfully. > > Building with 'Xcode with Clang'. > > ld: warning: cannot export hidden symbol > _mexfilerequiredapiversion from c_mexapi_version.o > > > MEX completed successfully. > > Building with 'Xcode with Clang'. > > ld: warning: cannot export hidden symbol > _mexfilerequiredapiversion from c_mexapi_version.o > > > ld: warning: cannot export hidden symbol > _mexfilerequiredapiversion from c_mexapi_version.o > > > MEX completed successfully. > > > > However, make check gives error > > > amneetb at Amneets-MBP:~/Softwares/PETSc-Gitlab/PETSc$ make > PETSC_DIR=/Users/amneetb/Softwares/PETSc-Gitlab/PETSc > PETSC_ARCH=darwin-dbg check > > Running test examples to verify correct installation > > Using PETSC_DIR=/Users/amneetb/Softwares/PETSc-Gitlab/PETSc > and PETSC_ARCH=darwin-dbg > > Possible error running C/C++ src/snes/examples/tutorials/ex19 > with 1 MPI process > > See http://www.mcs.anl.gov/petsc/documentation/faq.html > > dyld: Library not loaded: @rpath/libeng.dylib > > ? Referenced from: > /Users/amneetb/Softwares/PETSc-Gitlab/PETSc/src/snes/examples/tutorials/./ex19 > > ? Reason: image not found > > > =================================================================================== > > = ? BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES > > = ? PID 14057 RUNNING AT Amneets-MBP > > = ? EXIT CODE: 6 > > = ? CLEANING UP REMAINING PROCESSES > > = ? YOU CAN IGNORE THE BELOW CLEANUP MESSAGES > > =================================================================================== > > YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Abort trap: > 6 (signal 6) > > This typically refers to a problem with your application. > > Please see the FAQ page for debugging suggestions > > Possible error running C/C++ src/snes/examples/tutorials/ex19 > with 2 MPI processes > > See http://www.mcs.anl.gov/petsc/documentation/faq.html > > dyld: Library not loaded: @rpath/libeng.dylib > > ? Referenced from: > /Users/amneetb/Softwares/PETSc-Gitlab/PETSc/src/snes/examples/tutorials/./ex19 > > ? Reason: image not found > > dyld: Library not loaded: @rpath/libeng.dylib > > ? Referenced from: > /Users/amneetb/Softwares/PETSc-Gitlab/PETSc/src/snes/examples/tutorials/./ex19 > > ? Reason: image not found > > > =================================================================================== > > = ? BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES > > = ? PID 14062 RUNNING AT Amneets-MBP > > = ? EXIT CODE: 6 > > = ? CLEANING UP REMAINING PROCESSES > > = ? YOU CAN IGNORE THE BELOW CLEANUP MESSAGES > > =================================================================================== > > YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Abort trap: > 6 (signal 6) > > This typically refers to a problem with your application. > > Please see the FAQ page for debugging suggestions > > 1,5c1,17 > > < lid velocity = 0.0016, prandtl # = 1., grashof # = 1. > > < ? 0 SNES Function norm 0.0406612 > > < ? 1 SNES Function norm 4.12227e-06 > > < ? 2 SNES Function norm 6.098e-11 > > < Number of SNES iterations = 2 > > --- > > > dyld: Library not loaded: @rpath/libeng.dylib > > > ? Referenced from: > /Users/amneetb/Softwares/PETSc-Gitlab/PETSc/src/snes/examples/tutorials/./ex19 > > > ? Reason: image not found > > > dyld: Library not loaded: @rpath/libeng.dylib > > > ? Referenced from: > /Users/amneetb/Softwares/PETSc-Gitlab/PETSc/src/snes/examples/tutorials/./ex19 > > > ? Reason: image not found > > > > > > =================================================================================== > > > = ? BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES > > > = ? PID 14072 RUNNING AT Amneets-MBP > > > = ? EXIT CODE: 6 > > > = ? CLEANING UP REMAINING PROCESSES > > > = ? YOU CAN IGNORE THE BELOW CLEANUP MESSAGES > > > =================================================================================== > > > YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Abort trap: 6 (signal 6) > > > This typically refers to a problem with your application. > > > Please see the FAQ page for debugging suggestions > > /Users/amneetb/Softwares/PETSc-Gitlab/PETSc/src/snes/examples/tutorials > > Possible problem with ex19 running with hypre, diffs above > > ========================================= > > Possible error running Fortran example > src/snes/examples/tutorials/ex5f with 1 MPI process > > See http://www.mcs.anl.gov/petsc/documentation/faq.html > > dyld: Library not loaded: @rpath/libeng.dylib > > ? Referenced from: > /Users/amneetb/Softwares/PETSc-Gitlab/PETSc/src/snes/examples/tutorials/./ex5f > > ? Reason: image not found > > > =================================================================================== > > = ? BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES > > = ? PID 14211 RUNNING AT Amneets-MBP > > = ? EXIT CODE: 6 > > = ? CLEANING UP REMAINING PROCESSES > > = ? YOU CAN IGNORE THE BELOW CLEANUP MESSAGES > > =================================================================================== > > YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Abort trap: > 6 (signal 6) > > This typically refers to a problem with your application. > > Please see the FAQ page for debugging suggestions > > Completed test examples > > > > I have added the following paths in my .profile and sourced it > before make all and make check > > exportDYLD_FALLBACK_LIBRARY_PATH=/Applications/MATLAB_R2019b.app/bin/maci64:/Applications/MATLAB_R2019b.app/sys/os/maci64:$DYLD_FALLBACK_LIBRARY_PATH > > > exportDYLD_LIBRARY_PATH=/Applications/MATLAB_R2019b.app/bin/maci64:/Applications/MATLAB_R2019b.app/sys/os/maci64:$DYLD_LIBRARY_PATH > > > > > On Mon, Jun 8, 2020 at 6:39 PM Barry Smith > wrote: > > > ? Looks like the tool to list the examples in the manual > pages is broken since it didn't find these other examples, > I'll take a look at it. > > ? There is also a discussion in the users manual > > $ git grep MatlabEngine | grep ex > src/docs/tex/manual/part2.tex:PetscMatlabEngineCreate(MPI_Comm > comm,char *machine,PetscMatlabEngine *e); > src/docs/tex/manual/part2.tex:PetscMatlabEnginePut(PetscMatlabEngine > e,PetscObject obj); > src/docs/tex/manual/part2.tex:PetscMatlabEngineGet(PetscMatlabEngine > e,PetscObject obj); > src/docs/tex/manual/part2.tex:PetscMatlabEnginePutArray(PetscMatlabEngine > e,int m,int n,PetscScalar *array,char *name); > src/docs/tex/manual/part2.tex:PetscMatlabEngineGetArray(PetscMatlabEngine > e,int m,int n,PetscScalar *array,char *name); > src/docs/tex/manual/part2.tex:PetscMatlabEngineEvaluate(PetscMatlabEngine,"format",...); > src/docs/tex/manual/part2.tex:PetscMatlabEngineEvaluate(PetscMatlabEngine,"x > = \%g *y + z;",avalue); > src/docs/tex/manual/part2.tex:PetscMatlabEngineGetOutput(PetscMatlabEngine,char > **); > src/docs/tex/manual/part2.tex:PetscMatlabEnginedPrintOutput(PetscMatlabEngine,FILE*). > src/snes/tutorials/ex5.c: ierr = > PetscMatlabEnginePut(PETSC_MATLAB_ENGINE_(comm),(PetscObject)localX);CHKERRQ(ierr); > src/snes/tutorials/ex5.c: ierr = > PetscMatlabEngineEvaluate(PETSC_MATLAB_ENGINE_(comm),"localF=ex5m(localX,%18.16e,%18.16e,%18.16e)",hx,hy,lambda);CHKERRQ(ierr); > src/snes/tutorials/ex5.c: ierr = > PetscMatlabEngineGet(PETSC_MATLAB_ENGINE_(comm),(PetscObject)localF);CHKERRQ(ierr); > src/sys/tests/ex16.c:static char help[] = "Demonstrates > PetscMatlabEngineXXX()\n"; > src/sys/tests/ex16.c: ierr = > PetscMatlabEngineGetOutput(PETSC_MATLAB_ENGINE_(PETSC_COMM_WORLD),&output);CHKERRQ(ierr); > src/sys/tests/ex16.c: ierr = > PetscMatlabEngineEvaluate(PETSC_MATLAB_ENGINE_(PETSC_COMM_WORLD),"MPI_Comm_rank");CHKERRQ(ierr); > src/sys/tests/ex16.c: ? ierr = > PetscMatlabEngineEvaluate(PETSC_MATLAB_ENGINE_(PETSC_COMM_WORLD),user);CHKERRQ(ierr); > src/vec/vec/tests/ex31.c:static const char help[] = > "Demonstrates PetscMatlabEngineXXX()\n"; > src/vec/vec/tests/ex31.c: ierr = > PetscMatlabEngineGetOutput(PETSC_MATLAB_ENGINE_WORLD,&output); > src/vec/vec/tests/ex31.c: ierr = > PetscMatlabEngineEvaluate(PETSC_MATLAB_ENGINE_WORLD,"MPI_Comm_rank"); > src/vec/vec/tests/ex31.c: ierr = > PetscMatlabEnginePut(PETSC_MATLAB_ENGINE_WORLD,(PetscObject)x);CHKERRQ(ierr); > src/vec/vec/tests/ex31.c: ierr = > PetscMatlabEngineEvaluate(PETSC_MATLAB_ENGINE_WORLD,"x = x > + MPI_Comm_rank;\n"); > src/vec/vec/tests/ex31.c: ierr = > PetscMatlabEngineGet(PETSC_MATLAB_ENGINE_WORLD,(PetscObject)x);CHKERRQ(ierr); > src/vec/vec/tests/ex31.c: ierr = > PetscMatlabEngineEvaluate(PETSC_MATLAB_ENGINE_WORLD,"whos\n");CHKERRQ(ierr); > >> On Jun 8, 2020, at 8:26 PM, Amneet Bhalla >> > wrote: >> >> Great! >> >> Clicking on some of the functions pointed me to this example >> https://www.mcs.anl.gov/petsc/petsc-3.13/src/snes/tutorials/ex5.c.html >> >> Are there more examples, which are perhaps simpler than >> the above, which I can refer to? >> >> On Mon, Jun 8, 2020 at 5:51 PM Barry Smith >> > wrote: >> >> >> ? You should use the PetscMatlabEngine >> https://www.mcs.anl.gov/petsc/petsc-3.13/docs/manualpages/Sys/PetscMatlabEngine.html >> >> ??Barry >> >> >>> On Jun 8, 2020, at 7:09 PM, Amneet Bhalla >>> >> > wrote: >>> >>> Hi Folks, >>> >>> My parallel application written in C++ (which also >>> uses PETSc library for linear solvers) requires >>> calling control and optimization toolbox of MATLAB >>> at every time step to get a small number of double >>> values. Likewise the MATLAB function will need to >>> know some values from the parallel C++ application >>> to return those doubles. I'm wondering if there is a >>> recommended way of calling MATLAB from parallel >>> C/C++/Fortran codes, and if PETSc has some interface >>> that could be leveraged for this exchange. >>> >>> Thanks, >>> -- >>> --Amneet >>> >>> >>> >> >> >> >> -- >> --Amneet >> >> >> > > > > -- > --Amneet > > > > > > -- > --Amneet > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mail2amneet at gmail.com Tue Jun 9 16:58:59 2020 From: mail2amneet at gmail.com (Amneet Bhalla) Date: Tue, 9 Jun 2020 14:58:59 -0700 Subject: [petsc-users] Calling MATLAB code within a parallel C/C++ code In-Reply-To: References: <028B5BA9-4ECA-44B9-BE51-E24000E50185@petsc.dev> <9BE0A2F9-24DF-452B-AABB-1B8FAA748181@petsc.dev> Message-ID: It is under amneetb at Amneets-MBP:/Applications/MATLAB_R2019b.app/bin/maci64$ ls *eng.dylib libeng.dylib* On Tue, Jun 9, 2020 at 2:51 PM Junchao Zhang wrote: > Find recursively where is libeng.dylib under > /Applications/MATLAB_R2019b.app/extern/engines/ > > --Junchao Zhang > > > On Tue, Jun 9, 2020 at 4:27 PM Amneet Bhalla > wrote: > >> Forgot to mention that configure found the following about MATLAB >> >> Matlab: >> >> Includes: -I/Applications/MATLAB_R2019b.app/extern/include >> >> /Applications/MATLAB_R2019b.app >> >> MatlabEngine: >> >> Library: -L/Applications/MATLAB_R2019b.app/bin/maci64 >> -L/Applications/MATLAB_R2019b.app/extern/lib/maci64 -leng -lmex -lmx -lmat >> -lut -licudata -licui18n -licuuc >> >> On Tue, Jun 9, 2020 at 2:20 PM Amneet Bhalla >> wrote: >> >>> Do these additional configure flags for MATLAB look OK? >>> >>> --with-matlab=1 --with-matlab-dir=/Applications/MATLAB_R2019b.app >>> --with-matlab-engine=1 >>> --with-matlab-engine-dir=/Applications/MATLAB_R2019b.app/extern/engines/ >>> >>> >>> >>> >>> With this configuration make compiled files like: >>> >>> >>> CC darwin-dbg/obj/sys/classes/viewer/impls/matlab/vmatlab.o >>> >>> CC >>> darwin-dbg/obj/sys/classes/viewer/impls/matlab/ftn-custom/zvmatlabf.o >>> >>> CC darwin-dbg/obj/sys/classes/matlabengine/matlab.o >>> >>> CC darwin-dbg/obj/sys/classes/matlabengine/ftn-auto/matlabf.o >>> >>> CC >>> darwin-dbg/obj/sys/classes/matlabengine/ftn-custom/zmatlabf.o >>> >>> CC darwin-dbg/obj/vec/pf/impls/matlab/cmatlab.o >>> >>> CC darwin-dbg/obj/vec/vec/utils/matlab/gcreatev.o >>> >>> >>> make[2]: Leaving directory `/Users/amneetb/Softwares/PETSc-Gitlab/PETSc' >>> >>> BEGINNING TO COMPILE MATLAB INTERFACE >>> >>> Building with 'Xcode with Clang'. >>> >>> ld: warning: cannot export hidden symbol _mexfilerequiredapiversion from >>> c_mexapi_version.o >>> >>> >>> ld: warning: cannot export hidden symbol _mexfilerequiredapiversion from >>> c_mexapi_version.o >>> >>> >>> MEX completed successfully. >>> >>> Building with 'Xcode with Clang'. >>> >>> ld: warning: cannot export hidden symbol _mexfilerequiredapiversion from >>> c_mexapi_version.o >>> >>> >>> ld: warning: cannot export hidden symbol _mexfilerequiredapiversion from >>> c_mexapi_version.o >>> >>> >>> MEX completed successfully. >>> >>> Building with 'Xcode with Clang'. >>> >>> ld: warning: cannot export hidden symbol _mexfilerequiredapiversion from >>> c_mexapi_version.o >>> >>> >>> MEX completed successfully. >>> >>> Building with 'Xcode with Clang'. >>> >>> ld: warning: cannot export hidden symbol _mexfilerequiredapiversion from >>> c_mexapi_version.o >>> >>> >>> ld: warning: cannot export hidden symbol _mexfilerequiredapiversion from >>> c_mexapi_version.o >>> >>> >>> MEX completed successfully. >>> >>> >>> However, make check gives error >>> >>> >>> amneetb at Amneets-MBP:~/Softwares/PETSc-Gitlab/PETSc$ make >>> PETSC_DIR=/Users/amneetb/Softwares/PETSc-Gitlab/PETSc PETSC_ARCH=darwin-dbg >>> check >>> >>> Running test examples to verify correct installation >>> >>> Using PETSC_DIR=/Users/amneetb/Softwares/PETSc-Gitlab/PETSc and >>> PETSC_ARCH=darwin-dbg >>> >>> Possible error running C/C++ src/snes/examples/tutorials/ex19 with 1 MPI >>> process >>> >>> See http://www.mcs.anl.gov/petsc/documentation/faq.html >>> >>> dyld: Library not loaded: @rpath/libeng.dylib >>> >>> Referenced from: >>> /Users/amneetb/Softwares/PETSc-Gitlab/PETSc/src/snes/examples/tutorials/./ex19 >>> >>> Reason: image not found >>> >>> >>> >>> =================================================================================== >>> >>> = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES >>> >>> = PID 14057 RUNNING AT Amneets-MBP >>> >>> = EXIT CODE: 6 >>> >>> = CLEANING UP REMAINING PROCESSES >>> >>> = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES >>> >>> >>> =================================================================================== >>> >>> YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Abort trap: 6 (signal >>> 6) >>> >>> This typically refers to a problem with your application. >>> >>> Please see the FAQ page for debugging suggestions >>> >>> Possible error running C/C++ src/snes/examples/tutorials/ex19 with 2 MPI >>> processes >>> >>> See http://www.mcs.anl.gov/petsc/documentation/faq.html >>> >>> dyld: Library not loaded: @rpath/libeng.dylib >>> >>> Referenced from: >>> /Users/amneetb/Softwares/PETSc-Gitlab/PETSc/src/snes/examples/tutorials/./ex19 >>> >>> Reason: image not found >>> >>> dyld: Library not loaded: @rpath/libeng.dylib >>> >>> Referenced from: >>> /Users/amneetb/Softwares/PETSc-Gitlab/PETSc/src/snes/examples/tutorials/./ex19 >>> >>> Reason: image not found >>> >>> >>> >>> =================================================================================== >>> >>> = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES >>> >>> = PID 14062 RUNNING AT Amneets-MBP >>> >>> = EXIT CODE: 6 >>> >>> = CLEANING UP REMAINING PROCESSES >>> >>> = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES >>> >>> >>> =================================================================================== >>> >>> YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Abort trap: 6 (signal >>> 6) >>> >>> This typically refers to a problem with your application. >>> >>> Please see the FAQ page for debugging suggestions >>> >>> 1,5c1,17 >>> >>> < lid velocity = 0.0016, prandtl # = 1., grashof # = 1. >>> >>> < 0 SNES Function norm 0.0406612 >>> >>> < 1 SNES Function norm 4.12227e-06 >>> >>> < 2 SNES Function norm 6.098e-11 >>> >>> < Number of SNES iterations = 2 >>> >>> --- >>> >>> > dyld: Library not loaded: @rpath/libeng.dylib >>> >>> > Referenced from: >>> /Users/amneetb/Softwares/PETSc-Gitlab/PETSc/src/snes/examples/tutorials/./ex19 >>> >>> > Reason: image not found >>> >>> > dyld: Library not loaded: @rpath/libeng.dylib >>> >>> > Referenced from: >>> /Users/amneetb/Softwares/PETSc-Gitlab/PETSc/src/snes/examples/tutorials/./ex19 >>> >>> > Reason: image not found >>> >>> > >>> >>> > >>> =================================================================================== >>> >>> > = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES >>> >>> > = PID 14072 RUNNING AT Amneets-MBP >>> >>> > = EXIT CODE: 6 >>> >>> > = CLEANING UP REMAINING PROCESSES >>> >>> > = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES >>> >>> > >>> =================================================================================== >>> >>> > YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Abort trap: 6 >>> (signal 6) >>> >>> > This typically refers to a problem with your application. >>> >>> > Please see the FAQ page for debugging suggestions >>> >>> /Users/amneetb/Softwares/PETSc-Gitlab/PETSc/src/snes/examples/tutorials >>> >>> Possible problem with ex19 running with hypre, diffs above >>> >>> ========================================= >>> >>> Possible error running Fortran example src/snes/examples/tutorials/ex5f >>> with 1 MPI process >>> >>> See http://www.mcs.anl.gov/petsc/documentation/faq.html >>> >>> dyld: Library not loaded: @rpath/libeng.dylib >>> >>> Referenced from: >>> /Users/amneetb/Softwares/PETSc-Gitlab/PETSc/src/snes/examples/tutorials/./ex5f >>> >>> Reason: image not found >>> >>> >>> >>> =================================================================================== >>> >>> = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES >>> >>> = PID 14211 RUNNING AT Amneets-MBP >>> >>> = EXIT CODE: 6 >>> >>> = CLEANING UP REMAINING PROCESSES >>> >>> = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES >>> >>> >>> =================================================================================== >>> >>> YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Abort trap: 6 (signal >>> 6) >>> >>> This typically refers to a problem with your application. >>> >>> Please see the FAQ page for debugging suggestions >>> >>> Completed test examples >>> >>> >>> I have added the following paths in my .profile and sourced it before >>> make all and make check >>> >>> export DYLD_FALLBACK_LIBRARY_PATH= >>> /Applications/MATLAB_R2019b.app/bin/maci64:/Applications/MATLAB_R2019b.app/sys/os/maci64: >>> $DYLD_FALLBACK_LIBRARY_PATH >>> >>> >>> export DYLD_LIBRARY_PATH= >>> /Applications/MATLAB_R2019b.app/bin/maci64:/Applications/MATLAB_R2019b.app/sys/os/maci64: >>> $DYLD_LIBRARY_PATH >>> >>> >>> >>> >>> On Mon, Jun 8, 2020 at 6:39 PM Barry Smith wrote: >>> >>>> >>>> Looks like the tool to list the examples in the manual pages is >>>> broken since it didn't find these other examples, I'll take a look at it. >>>> >>>> There is also a discussion in the users manual >>>> >>>> $ git grep MatlabEngine | grep ex >>>> src/docs/tex/manual/part2.tex:PetscMatlabEngineCreate(MPI_Comm >>>> comm,char *machine,PetscMatlabEngine *e); >>>> src/docs/tex/manual/part2.tex:PetscMatlabEnginePut(PetscMatlabEngine >>>> e,PetscObject obj); >>>> src/docs/tex/manual/part2.tex:PetscMatlabEngineGet(PetscMatlabEngine >>>> e,PetscObject obj); >>>> src/docs/tex/manual/part2.tex:PetscMatlabEnginePutArray(PetscMatlabEngine >>>> e,int m,int n,PetscScalar *array,char *name); >>>> src/docs/tex/manual/part2.tex:PetscMatlabEngineGetArray(PetscMatlabEngine >>>> e,int m,int n,PetscScalar *array,char *name); >>>> >>>> src/docs/tex/manual/part2.tex:PetscMatlabEngineEvaluate(PetscMatlabEngine,"format",...); >>>> src/docs/tex/manual/part2.tex:PetscMatlabEngineEvaluate(PetscMatlabEngine,"x >>>> = \%g *y + z;",avalue); >>>> src/docs/tex/manual/part2.tex:PetscMatlabEngineGetOutput(PetscMatlabEngine,char >>>> **); >>>> >>>> src/docs/tex/manual/part2.tex:PetscMatlabEnginedPrintOutput(PetscMatlabEngine,FILE*). >>>> src/snes/tutorials/ex5.c: ierr = >>>> PetscMatlabEnginePut(PETSC_MATLAB_ENGINE_(comm),(PetscObject)localX);CHKERRQ(ierr); >>>> src/snes/tutorials/ex5.c: ierr = >>>> PetscMatlabEngineEvaluate(PETSC_MATLAB_ENGINE_(comm),"localF=ex5m(localX,%18.16e,%18.16e,%18.16e)",hx,hy,lambda);CHKERRQ(ierr); >>>> src/snes/tutorials/ex5.c: ierr = >>>> PetscMatlabEngineGet(PETSC_MATLAB_ENGINE_(comm),(PetscObject)localF);CHKERRQ(ierr); >>>> src/sys/tests/ex16.c:static char help[] = "Demonstrates >>>> PetscMatlabEngineXXX()\n"; >>>> src/sys/tests/ex16.c: ierr = >>>> PetscMatlabEngineGetOutput(PETSC_MATLAB_ENGINE_(PETSC_COMM_WORLD),&output);CHKERRQ(ierr); >>>> src/sys/tests/ex16.c: ierr = >>>> PetscMatlabEngineEvaluate(PETSC_MATLAB_ENGINE_(PETSC_COMM_WORLD),"MPI_Comm_rank");CHKERRQ(ierr); >>>> src/sys/tests/ex16.c: ierr = >>>> PetscMatlabEngineEvaluate(PETSC_MATLAB_ENGINE_(PETSC_COMM_WORLD),user);CHKERRQ(ierr); >>>> src/vec/vec/tests/ex31.c:static const char help[] = "Demonstrates >>>> PetscMatlabEngineXXX()\n"; >>>> src/vec/vec/tests/ex31.c: ierr = >>>> PetscMatlabEngineGetOutput(PETSC_MATLAB_ENGINE_WORLD,&output); >>>> src/vec/vec/tests/ex31.c: ierr = >>>> PetscMatlabEngineEvaluate(PETSC_MATLAB_ENGINE_WORLD,"MPI_Comm_rank"); >>>> src/vec/vec/tests/ex31.c: ierr = >>>> PetscMatlabEnginePut(PETSC_MATLAB_ENGINE_WORLD,(PetscObject)x);CHKERRQ(ierr); >>>> src/vec/vec/tests/ex31.c: ierr = >>>> PetscMatlabEngineEvaluate(PETSC_MATLAB_ENGINE_WORLD,"x = x + >>>> MPI_Comm_rank;\n"); >>>> src/vec/vec/tests/ex31.c: ierr = >>>> PetscMatlabEngineGet(PETSC_MATLAB_ENGINE_WORLD,(PetscObject)x);CHKERRQ(ierr); >>>> src/vec/vec/tests/ex31.c: ierr = >>>> PetscMatlabEngineEvaluate(PETSC_MATLAB_ENGINE_WORLD,"whos\n");CHKERRQ(ierr); >>>> >>>> On Jun 8, 2020, at 8:26 PM, Amneet Bhalla >>>> wrote: >>>> >>>> Great! >>>> >>>> Clicking on some of the functions pointed me to this example >>>> https://www.mcs.anl.gov/petsc/petsc-3.13/src/snes/tutorials/ex5.c.html >>>> >>>> Are there more examples, which are perhaps simpler than the above, >>>> which I can refer to? >>>> >>>> On Mon, Jun 8, 2020 at 5:51 PM Barry Smith wrote: >>>> >>>>> >>>>> You should use the PetscMatlabEngine >>>>> https://www.mcs.anl.gov/petsc/petsc-3.13/docs/manualpages/Sys/PetscMatlabEngine.html >>>>> >>>>> Barry >>>>> >>>>> >>>>> On Jun 8, 2020, at 7:09 PM, Amneet Bhalla >>>>> wrote: >>>>> >>>>> Hi Folks, >>>>> >>>>> My parallel application written in C++ (which also uses PETSc library >>>>> for linear solvers) requires calling control and optimization toolbox of >>>>> MATLAB at every time step to get a small number of double values. Likewise >>>>> the MATLAB function will need to know some values from the parallel C++ >>>>> application to return those doubles. I'm wondering if there is a >>>>> recommended way of calling MATLAB from parallel C/C++/Fortran codes, and if >>>>> PETSc has some interface that could be leveraged for this exchange. >>>>> >>>>> Thanks, >>>>> -- >>>>> --Amneet >>>>> >>>>> >>>>> >>>>> >>>>> >>>> >>>> -- >>>> --Amneet >>>> >>>> >>>> >>>> >>>> >>> >>> -- >>> --Amneet >>> >>> >>> >>> >> >> -- >> --Amneet >> >> >> >> -- --Amneet -------------- next part -------------- An HTML attachment was scrubbed... URL: From mail2amneet at gmail.com Tue Jun 9 17:09:42 2020 From: mail2amneet at gmail.com (Amneet Bhalla) Date: Tue, 9 Jun 2020 15:09:42 -0700 Subject: [petsc-users] Calling MATLAB code within a parallel C/C++ code In-Reply-To: <943847d1-7b6a-8841-f2f8-91652b2f64f2@berkeley.edu> References: <028B5BA9-4ECA-44B9-BE51-E24000E50185@petsc.dev> <9BE0A2F9-24DF-452B-AABB-1B8FAA748181@petsc.dev> <943847d1-7b6a-8841-f2f8-91652b2f64f2@berkeley.edu> Message-ID: That's where I also found it. Thanks, Sanjay! (This path is included in my .profile file export DYLD_FALLBACK_LIBRARY_PATH= /Applications/MATLAB_R2019b.app/bin/maci64:/Applications/MATLAB_R2019b.app/sys/os/maci64: $DYLD_FALLBACK_LIBRARY_PATH export DYLD_LIBRARY_PATH= /Applications/MATLAB_R2019b.app/bin/maci64:/Applications/MATLAB_R2019b.app/sys/os/maci64: $DYLD_LIBRARY_PATH ) Do I need to tell PETSc about this path as well? Something like: --with-matlab-engine-lib=/Applications/MATLAB_R2019b.app/bin/maci64/libeng.dylib On Tue, Jun 9, 2020 at 2:56 PM Sanjay Govindjee wrote: > Try looking in: > > $(MLROOT)/bin/maci64 > > where MLROOT = /Applications/MATLAB_R2019a.app or the like > > > > On 6/9/20 2:51 PM, Junchao Zhang wrote: > > Find recursively where is libeng.dylib under > /Applications/MATLAB_R2019b.app/extern/engines/ > > --Junchao Zhang > > > On Tue, Jun 9, 2020 at 4:27 PM Amneet Bhalla > wrote: > >> Forgot to mention that configure found the following about MATLAB >> >> Matlab: >> >> Includes: -I/Applications/MATLAB_R2019b.app/extern/include >> >> /Applications/MATLAB_R2019b.app >> >> MatlabEngine: >> >> Library: -L/Applications/MATLAB_R2019b.app/bin/maci64 >> -L/Applications/MATLAB_R2019b.app/extern/lib/maci64 -leng -lmex -lmx -lmat >> -lut -licudata -licui18n -licuuc >> >> On Tue, Jun 9, 2020 at 2:20 PM Amneet Bhalla >> wrote: >> >>> Do these additional configure flags for MATLAB look OK? >>> >>> --with-matlab=1 --with-matlab-dir=/Applications/MATLAB_R2019b.app >>> --with-matlab-engine=1 >>> --with-matlab-engine-dir=/Applications/MATLAB_R2019b.app/extern/engines/ >>> >>> >>> >>> >>> With this configuration make compiled files like: >>> >>> >>> CC darwin-dbg/obj/sys/classes/viewer/impls/matlab/vmatlab.o >>> >>> CC >>> darwin-dbg/obj/sys/classes/viewer/impls/matlab/ftn-custom/zvmatlabf.o >>> >>> CC darwin-dbg/obj/sys/classes/matlabengine/matlab.o >>> >>> CC darwin-dbg/obj/sys/classes/matlabengine/ftn-auto/matlabf.o >>> >>> CC >>> darwin-dbg/obj/sys/classes/matlabengine/ftn-custom/zmatlabf.o >>> >>> CC darwin-dbg/obj/vec/pf/impls/matlab/cmatlab.o >>> >>> CC darwin-dbg/obj/vec/vec/utils/matlab/gcreatev.o >>> >>> >>> make[2]: Leaving directory `/Users/amneetb/Softwares/PETSc-Gitlab/PETSc' >>> >>> BEGINNING TO COMPILE MATLAB INTERFACE >>> >>> Building with 'Xcode with Clang'. >>> >>> ld: warning: cannot export hidden symbol _mexfilerequiredapiversion from >>> c_mexapi_version.o >>> >>> >>> ld: warning: cannot export hidden symbol _mexfilerequiredapiversion from >>> c_mexapi_version.o >>> >>> >>> MEX completed successfully. >>> >>> Building with 'Xcode with Clang'. >>> >>> ld: warning: cannot export hidden symbol _mexfilerequiredapiversion from >>> c_mexapi_version.o >>> >>> >>> ld: warning: cannot export hidden symbol _mexfilerequiredapiversion from >>> c_mexapi_version.o >>> >>> >>> MEX completed successfully. >>> >>> Building with 'Xcode with Clang'. >>> >>> ld: warning: cannot export hidden symbol _mexfilerequiredapiversion from >>> c_mexapi_version.o >>> >>> >>> MEX completed successfully. >>> >>> Building with 'Xcode with Clang'. >>> >>> ld: warning: cannot export hidden symbol _mexfilerequiredapiversion from >>> c_mexapi_version.o >>> >>> >>> ld: warning: cannot export hidden symbol _mexfilerequiredapiversion from >>> c_mexapi_version.o >>> >>> >>> MEX completed successfully. >>> >>> >>> However, make check gives error >>> >>> >>> amneetb at Amneets-MBP:~/Softwares/PETSc-Gitlab/PETSc$ make >>> PETSC_DIR=/Users/amneetb/Softwares/PETSc-Gitlab/PETSc PETSC_ARCH=darwin-dbg >>> check >>> >>> Running test examples to verify correct installation >>> >>> Using PETSC_DIR=/Users/amneetb/Softwares/PETSc-Gitlab/PETSc and >>> PETSC_ARCH=darwin-dbg >>> >>> Possible error running C/C++ src/snes/examples/tutorials/ex19 with 1 MPI >>> process >>> >>> See http://www.mcs.anl.gov/petsc/documentation/faq.html >>> >>> dyld: Library not loaded: @rpath/libeng.dylib >>> >>> Referenced from: >>> /Users/amneetb/Softwares/PETSc-Gitlab/PETSc/src/snes/examples/tutorials/./ex19 >>> >>> Reason: image not found >>> >>> >>> >>> =================================================================================== >>> >>> = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES >>> >>> = PID 14057 RUNNING AT Amneets-MBP >>> >>> = EXIT CODE: 6 >>> >>> = CLEANING UP REMAINING PROCESSES >>> >>> = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES >>> >>> >>> =================================================================================== >>> >>> YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Abort trap: 6 (signal >>> 6) >>> >>> This typically refers to a problem with your application. >>> >>> Please see the FAQ page for debugging suggestions >>> >>> Possible error running C/C++ src/snes/examples/tutorials/ex19 with 2 MPI >>> processes >>> >>> See http://www.mcs.anl.gov/petsc/documentation/faq.html >>> >>> dyld: Library not loaded: @rpath/libeng.dylib >>> >>> Referenced from: >>> /Users/amneetb/Softwares/PETSc-Gitlab/PETSc/src/snes/examples/tutorials/./ex19 >>> >>> Reason: image not found >>> >>> dyld: Library not loaded: @rpath/libeng.dylib >>> >>> Referenced from: >>> /Users/amneetb/Softwares/PETSc-Gitlab/PETSc/src/snes/examples/tutorials/./ex19 >>> >>> Reason: image not found >>> >>> >>> >>> =================================================================================== >>> >>> = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES >>> >>> = PID 14062 RUNNING AT Amneets-MBP >>> >>> = EXIT CODE: 6 >>> >>> = CLEANING UP REMAINING PROCESSES >>> >>> = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES >>> >>> >>> =================================================================================== >>> >>> YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Abort trap: 6 (signal >>> 6) >>> >>> This typically refers to a problem with your application. >>> >>> Please see the FAQ page for debugging suggestions >>> >>> 1,5c1,17 >>> >>> < lid velocity = 0.0016, prandtl # = 1., grashof # = 1. >>> >>> < 0 SNES Function norm 0.0406612 >>> >>> < 1 SNES Function norm 4.12227e-06 >>> >>> < 2 SNES Function norm 6.098e-11 >>> >>> < Number of SNES iterations = 2 >>> >>> --- >>> >>> > dyld: Library not loaded: @rpath/libeng.dylib >>> >>> > Referenced from: >>> /Users/amneetb/Softwares/PETSc-Gitlab/PETSc/src/snes/examples/tutorials/./ex19 >>> >>> > Reason: image not found >>> >>> > dyld: Library not loaded: @rpath/libeng.dylib >>> >>> > Referenced from: >>> /Users/amneetb/Softwares/PETSc-Gitlab/PETSc/src/snes/examples/tutorials/./ex19 >>> >>> > Reason: image not found >>> >>> > >>> >>> > >>> =================================================================================== >>> >>> > = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES >>> >>> > = PID 14072 RUNNING AT Amneets-MBP >>> >>> > = EXIT CODE: 6 >>> >>> > = CLEANING UP REMAINING PROCESSES >>> >>> > = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES >>> >>> > >>> =================================================================================== >>> >>> > YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Abort trap: 6 >>> (signal 6) >>> >>> > This typically refers to a problem with your application. >>> >>> > Please see the FAQ page for debugging suggestions >>> >>> /Users/amneetb/Softwares/PETSc-Gitlab/PETSc/src/snes/examples/tutorials >>> >>> Possible problem with ex19 running with hypre, diffs above >>> >>> ========================================= >>> >>> Possible error running Fortran example src/snes/examples/tutorials/ex5f >>> with 1 MPI process >>> >>> See http://www.mcs.anl.gov/petsc/documentation/faq.html >>> >>> dyld: Library not loaded: @rpath/libeng.dylib >>> >>> Referenced from: >>> /Users/amneetb/Softwares/PETSc-Gitlab/PETSc/src/snes/examples/tutorials/./ex5f >>> >>> Reason: image not found >>> >>> >>> >>> =================================================================================== >>> >>> = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES >>> >>> = PID 14211 RUNNING AT Amneets-MBP >>> >>> = EXIT CODE: 6 >>> >>> = CLEANING UP REMAINING PROCESSES >>> >>> = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES >>> >>> >>> =================================================================================== >>> >>> YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Abort trap: 6 (signal >>> 6) >>> >>> This typically refers to a problem with your application. >>> >>> Please see the FAQ page for debugging suggestions >>> >>> Completed test examples >>> >>> >>> I have added the following paths in my .profile and sourced it before >>> make all and make check >>> >>> export DYLD_FALLBACK_LIBRARY_PATH= >>> /Applications/MATLAB_R2019b.app/bin/maci64:/Applications/MATLAB_R2019b.app/sys/os/maci64: >>> $DYLD_FALLBACK_LIBRARY_PATH >>> >>> >>> export DYLD_LIBRARY_PATH= >>> /Applications/MATLAB_R2019b.app/bin/maci64:/Applications/MATLAB_R2019b.app/sys/os/maci64: >>> $DYLD_LIBRARY_PATH >>> >>> >>> >>> >>> On Mon, Jun 8, 2020 at 6:39 PM Barry Smith wrote: >>> >>>> >>>> Looks like the tool to list the examples in the manual pages is >>>> broken since it didn't find these other examples, I'll take a look at it. >>>> >>>> There is also a discussion in the users manual >>>> >>>> $ git grep MatlabEngine | grep ex >>>> src/docs/tex/manual/part2.tex:PetscMatlabEngineCreate(MPI_Comm >>>> comm,char *machine,PetscMatlabEngine *e); >>>> src/docs/tex/manual/part2.tex:PetscMatlabEnginePut(PetscMatlabEngine >>>> e,PetscObject obj); >>>> src/docs/tex/manual/part2.tex:PetscMatlabEngineGet(PetscMatlabEngine >>>> e,PetscObject obj); >>>> src/docs/tex/manual/part2.tex:PetscMatlabEnginePutArray(PetscMatlabEngine >>>> e,int m,int n,PetscScalar *array,char *name); >>>> src/docs/tex/manual/part2.tex:PetscMatlabEngineGetArray(PetscMatlabEngine >>>> e,int m,int n,PetscScalar *array,char *name); >>>> >>>> src/docs/tex/manual/part2.tex:PetscMatlabEngineEvaluate(PetscMatlabEngine,"format",...); >>>> src/docs/tex/manual/part2.tex:PetscMatlabEngineEvaluate(PetscMatlabEngine,"x >>>> = \%g *y + z;",avalue); >>>> src/docs/tex/manual/part2.tex:PetscMatlabEngineGetOutput(PetscMatlabEngine,char >>>> **); >>>> >>>> src/docs/tex/manual/part2.tex:PetscMatlabEnginedPrintOutput(PetscMatlabEngine,FILE*). >>>> src/snes/tutorials/ex5.c: ierr = >>>> PetscMatlabEnginePut(PETSC_MATLAB_ENGINE_(comm),(PetscObject)localX);CHKERRQ(ierr); >>>> src/snes/tutorials/ex5.c: ierr = >>>> PetscMatlabEngineEvaluate(PETSC_MATLAB_ENGINE_(comm),"localF=ex5m(localX,%18.16e,%18.16e,%18.16e)",hx,hy,lambda);CHKERRQ(ierr); >>>> src/snes/tutorials/ex5.c: ierr = >>>> PetscMatlabEngineGet(PETSC_MATLAB_ENGINE_(comm),(PetscObject)localF);CHKERRQ(ierr); >>>> src/sys/tests/ex16.c:static char help[] = "Demonstrates >>>> PetscMatlabEngineXXX()\n"; >>>> src/sys/tests/ex16.c: ierr = >>>> PetscMatlabEngineGetOutput(PETSC_MATLAB_ENGINE_(PETSC_COMM_WORLD),&output);CHKERRQ(ierr); >>>> src/sys/tests/ex16.c: ierr = >>>> PetscMatlabEngineEvaluate(PETSC_MATLAB_ENGINE_(PETSC_COMM_WORLD),"MPI_Comm_rank");CHKERRQ(ierr); >>>> src/sys/tests/ex16.c: ierr = >>>> PetscMatlabEngineEvaluate(PETSC_MATLAB_ENGINE_(PETSC_COMM_WORLD),user);CHKERRQ(ierr); >>>> src/vec/vec/tests/ex31.c:static const char help[] = "Demonstrates >>>> PetscMatlabEngineXXX()\n"; >>>> src/vec/vec/tests/ex31.c: ierr = >>>> PetscMatlabEngineGetOutput(PETSC_MATLAB_ENGINE_WORLD,&output); >>>> src/vec/vec/tests/ex31.c: ierr = >>>> PetscMatlabEngineEvaluate(PETSC_MATLAB_ENGINE_WORLD,"MPI_Comm_rank"); >>>> src/vec/vec/tests/ex31.c: ierr = >>>> PetscMatlabEnginePut(PETSC_MATLAB_ENGINE_WORLD,(PetscObject)x);CHKERRQ(ierr); >>>> src/vec/vec/tests/ex31.c: ierr = >>>> PetscMatlabEngineEvaluate(PETSC_MATLAB_ENGINE_WORLD,"x = x + >>>> MPI_Comm_rank;\n"); >>>> src/vec/vec/tests/ex31.c: ierr = >>>> PetscMatlabEngineGet(PETSC_MATLAB_ENGINE_WORLD,(PetscObject)x);CHKERRQ(ierr); >>>> src/vec/vec/tests/ex31.c: ierr = >>>> PetscMatlabEngineEvaluate(PETSC_MATLAB_ENGINE_WORLD,"whos\n");CHKERRQ(ierr); >>>> >>>> On Jun 8, 2020, at 8:26 PM, Amneet Bhalla >>>> wrote: >>>> >>>> Great! >>>> >>>> Clicking on some of the functions pointed me to this example >>>> https://www.mcs.anl.gov/petsc/petsc-3.13/src/snes/tutorials/ex5.c.html >>>> >>>> Are there more examples, which are perhaps simpler than the above, >>>> which I can refer to? >>>> >>>> On Mon, Jun 8, 2020 at 5:51 PM Barry Smith wrote: >>>> >>>>> >>>>> You should use the PetscMatlabEngine >>>>> https://www.mcs.anl.gov/petsc/petsc-3.13/docs/manualpages/Sys/PetscMatlabEngine.html >>>>> >>>>> Barry >>>>> >>>>> >>>>> On Jun 8, 2020, at 7:09 PM, Amneet Bhalla >>>>> wrote: >>>>> >>>>> Hi Folks, >>>>> >>>>> My parallel application written in C++ (which also uses PETSc library >>>>> for linear solvers) requires calling control and optimization toolbox of >>>>> MATLAB at every time step to get a small number of double values. Likewise >>>>> the MATLAB function will need to know some values from the parallel C++ >>>>> application to return those doubles. I'm wondering if there is a >>>>> recommended way of calling MATLAB from parallel C/C++/Fortran codes, and if >>>>> PETSc has some interface that could be leveraged for this exchange. >>>>> >>>>> Thanks, >>>>> -- >>>>> --Amneet >>>>> >>>>> >>>>> >>>>> >>>>> >>>> >>>> -- >>>> --Amneet >>>> >>>> >>>> >>>> >>>> >>> >>> -- >>> --Amneet >>> >>> >>> >>> >> >> -- >> --Amneet >> >> >> >> > -- --Amneet -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Tue Jun 9 17:14:04 2020 From: bsmith at petsc.dev (Barry Smith) Date: Tue, 9 Jun 2020 17:14:04 -0500 Subject: [petsc-users] Calling MATLAB code within a parallel C/C++ code In-Reply-To: References: <028B5BA9-4ECA-44B9-BE51-E24000E50185@petsc.dev> <9BE0A2F9-24DF-452B-AABB-1B8FAA748181@petsc.dev> Message-ID: <0323F5BB-F058-4530-AE27-88172F9CA1CE@petsc.dev> The rpath is not being set properly by PETSc when it links in the Matlab engine libraries; it used to work but something else must have changed to break it. I have a fix in the branch barry/2020-06-08/fix-matlabengine-details You should be able to do git fetch git checkout barry/2020-06-08/fix-matlabengine-details ./configure ARGS .... Please let us know if this resolves the problem Unfortunately we don't have a Matlab license for our test Mac machines so cannot test this in our test suite and detect the problem as soon as it is introduced. Perhaps Satish can see if there is a spare Matlab license for Mac that MCS pays for but isn't used. Barry > On Jun 9, 2020, at 4:58 PM, Amneet Bhalla wrote: > > It is under > > amneetb at Amneets-MBP:/Applications/MATLAB_R2019b.app/bin/maci64$ ls *eng.dylib > libeng.dylib* > > On Tue, Jun 9, 2020 at 2:51 PM Junchao Zhang > wrote: > Find recursively where is libeng.dylib under /Applications/MATLAB_R2019b.app/extern/engines/ > > --Junchao Zhang > > > On Tue, Jun 9, 2020 at 4:27 PM Amneet Bhalla > wrote: > Forgot to mention that configure found the following about MATLAB > > Matlab: > Includes: -I/Applications/MATLAB_R2019b.app/extern/include > /Applications/MATLAB_R2019b.app > MatlabEngine: > Library: -L/Applications/MATLAB_R2019b.app/bin/maci64 -L/Applications/MATLAB_R2019b.app/extern/lib/maci64 -leng -lmex -lmx -lmat -lut -licudata -licui18n -licuuc > > On Tue, Jun 9, 2020 at 2:20 PM Amneet Bhalla > wrote: > Do these additional configure flags for MATLAB look OK? > > --with-matlab=1 --with-matlab-dir=/Applications/MATLAB_R2019b.app --with-matlab-engine=1 --with-matlab-engine-dir=/Applications/MATLAB_R2019b.app/extern/engines/ > > > > With this configuration make compiled files like: > > CC darwin-dbg/obj/sys/classes/viewer/impls/matlab/vmatlab.o > CC darwin-dbg/obj/sys/classes/viewer/impls/matlab/ftn-custom/zvmatlabf.o > CC darwin-dbg/obj/sys/classes/matlabengine/matlab.o > CC darwin-dbg/obj/sys/classes/matlabengine/ftn-auto/matlabf.o > CC darwin-dbg/obj/sys/classes/matlabengine/ftn-custom/zmatlabf.o > CC darwin-dbg/obj/vec/pf/impls/matlab/cmatlab.o > CC darwin-dbg/obj/vec/vec/utils/matlab/gcreatev.o > > make[2]: Leaving directory `/Users/amneetb/Softwares/PETSc-Gitlab/PETSc' > BEGINNING TO COMPILE MATLAB INTERFACE > Building with 'Xcode with Clang'. > ld: warning: cannot export hidden symbol _mexfilerequiredapiversion from c_mexapi_version.o > > ld: warning: cannot export hidden symbol _mexfilerequiredapiversion from c_mexapi_version.o > > MEX completed successfully. > Building with 'Xcode with Clang'. > ld: warning: cannot export hidden symbol _mexfilerequiredapiversion from c_mexapi_version.o > > ld: warning: cannot export hidden symbol _mexfilerequiredapiversion from c_mexapi_version.o > > MEX completed successfully. > Building with 'Xcode with Clang'. > ld: warning: cannot export hidden symbol _mexfilerequiredapiversion from c_mexapi_version.o > > MEX completed successfully. > Building with 'Xcode with Clang'. > ld: warning: cannot export hidden symbol _mexfilerequiredapiversion from c_mexapi_version.o > > ld: warning: cannot export hidden symbol _mexfilerequiredapiversion from c_mexapi_version.o > > MEX completed successfully. > > > However, make check gives error > > amneetb at Amneets-MBP:~/Softwares/PETSc-Gitlab/PETSc$ make PETSC_DIR=/Users/amneetb/Softwares/PETSc-Gitlab/PETSc PETSC_ARCH=darwin-dbg check > Running test examples to verify correct installation > Using PETSC_DIR=/Users/amneetb/Softwares/PETSc-Gitlab/PETSc and PETSC_ARCH=darwin-dbg > Possible error running C/C++ src/snes/examples/tutorials/ex19 with 1 MPI process > See http://www.mcs.anl.gov/petsc/documentation/faq.html > dyld: Library not loaded: @rpath/libeng.dylib > Referenced from: /Users/amneetb/Softwares/PETSc-Gitlab/PETSc/src/snes/examples/tutorials/./ex19 > Reason: image not found > > =================================================================================== > = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES > = PID 14057 RUNNING AT Amneets-MBP > = EXIT CODE: 6 > = CLEANING UP REMAINING PROCESSES > = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES > =================================================================================== > YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Abort trap: 6 (signal 6) > This typically refers to a problem with your application. > Please see the FAQ page for debugging suggestions > Possible error running C/C++ src/snes/examples/tutorials/ex19 with 2 MPI processes > See http://www.mcs.anl.gov/petsc/documentation/faq.html > dyld: Library not loaded: @rpath/libeng.dylib > Referenced from: /Users/amneetb/Softwares/PETSc-Gitlab/PETSc/src/snes/examples/tutorials/./ex19 > Reason: image not found > dyld: Library not loaded: @rpath/libeng.dylib > Referenced from: /Users/amneetb/Softwares/PETSc-Gitlab/PETSc/src/snes/examples/tutorials/./ex19 > Reason: image not found > > =================================================================================== > = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES > = PID 14062 RUNNING AT Amneets-MBP > = EXIT CODE: 6 > = CLEANING UP REMAINING PROCESSES > = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES > =================================================================================== > YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Abort trap: 6 (signal 6) > This typically refers to a problem with your application. > Please see the FAQ page for debugging suggestions > 1,5c1,17 > < lid velocity = 0.0016, prandtl # = 1., grashof # = 1. > < 0 SNES Function norm 0.0406612 > < 1 SNES Function norm 4.12227e-06 > < 2 SNES Function norm 6.098e-11 > < Number of SNES iterations = 2 > --- > > dyld: Library not loaded: @rpath/libeng.dylib > > Referenced from: /Users/amneetb/Softwares/PETSc-Gitlab/PETSc/src/snes/examples/tutorials/./ex19 > > Reason: image not found > > dyld: Library not loaded: @rpath/libeng.dylib > > Referenced from: /Users/amneetb/Softwares/PETSc-Gitlab/PETSc/src/snes/examples/tutorials/./ex19 > > Reason: image not found > > > > =================================================================================== > > = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES > > = PID 14072 RUNNING AT Amneets-MBP > > = EXIT CODE: 6 > > = CLEANING UP REMAINING PROCESSES > > = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES > > =================================================================================== > > YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Abort trap: 6 (signal 6) > > This typically refers to a problem with your application. > > Please see the FAQ page for debugging suggestions > /Users/amneetb/Softwares/PETSc-Gitlab/PETSc/src/snes/examples/tutorials > Possible problem with ex19 running with hypre, diffs above > ========================================= > Possible error running Fortran example src/snes/examples/tutorials/ex5f with 1 MPI process > See http://www.mcs.anl.gov/petsc/documentation/faq.html > dyld: Library not loaded: @rpath/libeng.dylib > Referenced from: /Users/amneetb/Softwares/PETSc-Gitlab/PETSc/src/snes/examples/tutorials/./ex5f > Reason: image not found > > =================================================================================== > = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES > = PID 14211 RUNNING AT Amneets-MBP > = EXIT CODE: 6 > = CLEANING UP REMAINING PROCESSES > = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES > =================================================================================== > YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Abort trap: 6 (signal 6) > This typically refers to a problem with your application. > Please see the FAQ page for debugging suggestions > Completed test examples > > > I have added the following paths in my .profile and sourced it before make all and make check > > export DYLD_FALLBACK_LIBRARY_PATH=/Applications/MATLAB_R2019b.app/bin/maci64:/Applications/MATLAB_R2019b.app/sys/os/maci64:$DYLD_FALLBACK_LIBRARY_PATH > > export DYLD_LIBRARY_PATH=/Applications/MATLAB_R2019b.app/bin/maci64:/Applications/MATLAB_R2019b.app/sys/os/maci64:$DYLD_LIBRARY_PATH > > > > On Mon, Jun 8, 2020 at 6:39 PM Barry Smith > wrote: > > Looks like the tool to list the examples in the manual pages is broken since it didn't find these other examples, I'll take a look at it. > > There is also a discussion in the users manual > > $ git grep MatlabEngine | grep ex > src/docs/tex/manual/part2.tex:PetscMatlabEngineCreate(MPI_Comm comm,char *machine,PetscMatlabEngine *e); > src/docs/tex/manual/part2.tex:PetscMatlabEnginePut(PetscMatlabEngine e,PetscObject obj); > src/docs/tex/manual/part2.tex:PetscMatlabEngineGet(PetscMatlabEngine e,PetscObject obj); > src/docs/tex/manual/part2.tex:PetscMatlabEnginePutArray(PetscMatlabEngine e,int m,int n,PetscScalar *array,char *name); > src/docs/tex/manual/part2.tex:PetscMatlabEngineGetArray(PetscMatlabEngine e,int m,int n,PetscScalar *array,char *name); > src/docs/tex/manual/part2.tex:PetscMatlabEngineEvaluate(PetscMatlabEngine,"format",...); > src/docs/tex/manual/part2.tex:PetscMatlabEngineEvaluate(PetscMatlabEngine,"x = \%g *y + z;",avalue); > src/docs/tex/manual/part2.tex:PetscMatlabEngineGetOutput(PetscMatlabEngine,char **); > src/docs/tex/manual/part2.tex:PetscMatlabEnginedPrintOutput(PetscMatlabEngine,FILE*). > src/snes/tutorials/ex5.c: ierr = PetscMatlabEnginePut(PETSC_MATLAB_ENGINE_(comm),(PetscObject)localX);CHKERRQ(ierr); > src/snes/tutorials/ex5.c: ierr = PetscMatlabEngineEvaluate(PETSC_MATLAB_ENGINE_(comm),"localF=ex5m(localX,%18.16e,%18.16e,%18.16e)",hx,hy,lambda);CHKERRQ(ierr); > src/snes/tutorials/ex5.c: ierr = PetscMatlabEngineGet(PETSC_MATLAB_ENGINE_(comm),(PetscObject)localF);CHKERRQ(ierr); > src/sys/tests/ex16.c:static char help[] = "Demonstrates PetscMatlabEngineXXX()\n"; > src/sys/tests/ex16.c: ierr = PetscMatlabEngineGetOutput(PETSC_MATLAB_ENGINE_(PETSC_COMM_WORLD),&output);CHKERRQ(ierr); > src/sys/tests/ex16.c: ierr = PetscMatlabEngineEvaluate(PETSC_MATLAB_ENGINE_(PETSC_COMM_WORLD),"MPI_Comm_rank");CHKERRQ(ierr); > src/sys/tests/ex16.c: ierr = PetscMatlabEngineEvaluate(PETSC_MATLAB_ENGINE_(PETSC_COMM_WORLD),user);CHKERRQ(ierr); > src/vec/vec/tests/ex31.c:static const char help[] = "Demonstrates PetscMatlabEngineXXX()\n"; > src/vec/vec/tests/ex31.c: ierr = PetscMatlabEngineGetOutput(PETSC_MATLAB_ENGINE_WORLD,&output); > src/vec/vec/tests/ex31.c: ierr = PetscMatlabEngineEvaluate(PETSC_MATLAB_ENGINE_WORLD,"MPI_Comm_rank"); > src/vec/vec/tests/ex31.c: ierr = PetscMatlabEnginePut(PETSC_MATLAB_ENGINE_WORLD,(PetscObject)x);CHKERRQ(ierr); > src/vec/vec/tests/ex31.c: ierr = PetscMatlabEngineEvaluate(PETSC_MATLAB_ENGINE_WORLD,"x = x + MPI_Comm_rank;\n"); > src/vec/vec/tests/ex31.c: ierr = PetscMatlabEngineGet(PETSC_MATLAB_ENGINE_WORLD,(PetscObject)x);CHKERRQ(ierr); > src/vec/vec/tests/ex31.c: ierr = PetscMatlabEngineEvaluate(PETSC_MATLAB_ENGINE_WORLD,"whos\n");CHKERRQ(ierr); > >> On Jun 8, 2020, at 8:26 PM, Amneet Bhalla > wrote: >> >> Great! >> >> Clicking on some of the functions pointed me to this example >> https://www.mcs.anl.gov/petsc/petsc-3.13/src/snes/tutorials/ex5.c.html >> >> Are there more examples, which are perhaps simpler than the above, which I can refer to? >> >> On Mon, Jun 8, 2020 at 5:51 PM Barry Smith > wrote: >> >> You should use the PetscMatlabEngine https://www.mcs.anl.gov/petsc/petsc-3.13/docs/manualpages/Sys/PetscMatlabEngine.html >> >> Barry >> >> >>> On Jun 8, 2020, at 7:09 PM, Amneet Bhalla > wrote: >>> >>> Hi Folks, >>> >>> My parallel application written in C++ (which also uses PETSc library for linear solvers) requires calling control and optimization toolbox of MATLAB at every time step to get a small number of double values. Likewise the MATLAB function will need to know some values from the parallel C++ application to return those doubles. I'm wondering if there is a recommended way of calling MATLAB from parallel C/C++/Fortran codes, and if PETSc has some interface that could be leveraged for this exchange. >>> >>> Thanks, >>> -- >>> --Amneet >>> >>> >>> >> >> >> >> -- >> --Amneet >> >> >> > > > > -- > --Amneet > > > > > > -- > --Amneet > > > > > > -- > --Amneet > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Tue Jun 9 17:17:28 2020 From: bsmith at petsc.dev (Barry Smith) Date: Tue, 9 Jun 2020 17:17:28 -0500 Subject: [petsc-users] Calling MATLAB code within a parallel C/C++ code In-Reply-To: References: <028B5BA9-4ECA-44B9-BE51-E24000E50185@petsc.dev> <9BE0A2F9-24DF-452B-AABB-1B8FAA748181@petsc.dev> <943847d1-7b6a-8841-f2f8-91652b2f64f2@berkeley.edu> Message-ID: I tried including the directories into my DYLD environmental variables and it didn't work; I am not sure why. This might be why https://stackoverflow.com/questions/35568122/why-isnt-dyld-library-path-being-propagated-here Anyways try my branch and that should work. Barry > On Jun 9, 2020, at 5:09 PM, Amneet Bhalla wrote: > > > That's where I also found it. Thanks, Sanjay! > > (This path is included in my .profile file > > export DYLD_FALLBACK_LIBRARY_PATH=/Applications/MATLAB_R2019b.app/bin/maci64:/Applications/MATLAB_R2019b.app/sys/os/maci64:$DYLD_FALLBACK_LIBRARY_PATH > > export DYLD_LIBRARY_PATH=/Applications/MATLAB_R2019b.app/bin/maci64:/Applications/MATLAB_R2019b.app/sys/os/maci64:$DYLD_LIBRARY_PATH > > > > ) > > Do I need to tell PETSc about this path as well? Something like: > > --with-matlab-engine-lib=/Applications/MATLAB_R2019b.app/bin/maci64/libeng.dylib > > > On Tue, Jun 9, 2020 at 2:56 PM Sanjay Govindjee > wrote: > Try looking in: > > $(MLROOT)/bin/maci64 > > where MLROOT = /Applications/MATLAB_R2019a.app or the like > > > On 6/9/20 2:51 PM, Junchao Zhang wrote: >> Find recursively where is libeng.dylib under /Applications/MATLAB_R2019b.app/extern/engines/ >> >> --Junchao Zhang >> >> >> On Tue, Jun 9, 2020 at 4:27 PM Amneet Bhalla > wrote: >> Forgot to mention that configure found the following about MATLAB >> >> Matlab: >> Includes: -I/Applications/MATLAB_R2019b.app/extern/include >> /Applications/MATLAB_R2019b.app >> MatlabEngine: >> Library: -L/Applications/MATLAB_R2019b.app/bin/maci64 -L/Applications/MATLAB_R2019b.app/extern/lib/maci64 -leng -lmex -lmx -lmat -lut -licudata -licui18n -licuuc >> >> On Tue, Jun 9, 2020 at 2:20 PM Amneet Bhalla > wrote: >> Do these additional configure flags for MATLAB look OK? >> >> --with-matlab=1 --with-matlab-dir=/Applications/MATLAB_R2019b.app --with-matlab-engine=1 --with-matlab-engine-dir=/Applications/MATLAB_R2019b.app/extern/engines/ >> >> >> >> With this configuration make compiled files like: >> >> CC darwin-dbg/obj/sys/classes/viewer/impls/matlab/vmatlab.o >> CC darwin-dbg/obj/sys/classes/viewer/impls/matlab/ftn-custom/zvmatlabf.o >> CC darwin-dbg/obj/sys/classes/matlabengine/matlab.o >> CC darwin-dbg/obj/sys/classes/matlabengine/ftn-auto/matlabf.o >> CC darwin-dbg/obj/sys/classes/matlabengine/ftn-custom/zmatlabf.o >> CC darwin-dbg/obj/vec/pf/impls/matlab/cmatlab.o >> CC darwin-dbg/obj/vec/vec/utils/matlab/gcreatev.o >> >> make[2]: Leaving directory `/Users/amneetb/Softwares/PETSc-Gitlab/PETSc' >> BEGINNING TO COMPILE MATLAB INTERFACE >> Building with 'Xcode with Clang'. >> ld: warning: cannot export hidden symbol _mexfilerequiredapiversion from c_mexapi_version.o >> >> ld: warning: cannot export hidden symbol _mexfilerequiredapiversion from c_mexapi_version.o >> >> MEX completed successfully. >> Building with 'Xcode with Clang'. >> ld: warning: cannot export hidden symbol _mexfilerequiredapiversion from c_mexapi_version.o >> >> ld: warning: cannot export hidden symbol _mexfilerequiredapiversion from c_mexapi_version.o >> >> MEX completed successfully. >> Building with 'Xcode with Clang'. >> ld: warning: cannot export hidden symbol _mexfilerequiredapiversion from c_mexapi_version.o >> >> MEX completed successfully. >> Building with 'Xcode with Clang'. >> ld: warning: cannot export hidden symbol _mexfilerequiredapiversion from c_mexapi_version.o >> >> ld: warning: cannot export hidden symbol _mexfilerequiredapiversion from c_mexapi_version.o >> >> MEX completed successfully. >> >> >> However, make check gives error >> >> amneetb at Amneets-MBP:~/Softwares/PETSc-Gitlab/PETSc$ make PETSC_DIR=/Users/amneetb/Softwares/PETSc-Gitlab/PETSc PETSC_ARCH=darwin-dbg check >> Running test examples to verify correct installation >> Using PETSC_DIR=/Users/amneetb/Softwares/PETSc-Gitlab/PETSc and PETSC_ARCH=darwin-dbg >> Possible error running C/C++ src/snes/examples/tutorials/ex19 with 1 MPI process >> See http://www.mcs.anl.gov/petsc/documentation/faq.html >> dyld: Library not loaded: @rpath/libeng.dylib >> Referenced from: /Users/amneetb/Softwares/PETSc-Gitlab/PETSc/src/snes/examples/tutorials/./ex19 >> Reason: image not found >> >> =================================================================================== >> = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES >> = PID 14057 RUNNING AT Amneets-MBP >> = EXIT CODE: 6 >> = CLEANING UP REMAINING PROCESSES >> = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES >> =================================================================================== >> YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Abort trap: 6 (signal 6) >> This typically refers to a problem with your application. >> Please see the FAQ page for debugging suggestions >> Possible error running C/C++ src/snes/examples/tutorials/ex19 with 2 MPI processes >> See http://www.mcs.anl.gov/petsc/documentation/faq.html >> dyld: Library not loaded: @rpath/libeng.dylib >> Referenced from: /Users/amneetb/Softwares/PETSc-Gitlab/PETSc/src/snes/examples/tutorials/./ex19 >> Reason: image not found >> dyld: Library not loaded: @rpath/libeng.dylib >> Referenced from: /Users/amneetb/Softwares/PETSc-Gitlab/PETSc/src/snes/examples/tutorials/./ex19 >> Reason: image not found >> >> =================================================================================== >> = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES >> = PID 14062 RUNNING AT Amneets-MBP >> = EXIT CODE: 6 >> = CLEANING UP REMAINING PROCESSES >> = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES >> =================================================================================== >> YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Abort trap: 6 (signal 6) >> This typically refers to a problem with your application. >> Please see the FAQ page for debugging suggestions >> 1,5c1,17 >> < lid velocity = 0.0016, prandtl # = 1., grashof # = 1. >> < 0 SNES Function norm 0.0406612 >> < 1 SNES Function norm 4.12227e-06 >> < 2 SNES Function norm 6.098e-11 >> < Number of SNES iterations = 2 >> --- >> > dyld: Library not loaded: @rpath/libeng.dylib >> > Referenced from: /Users/amneetb/Softwares/PETSc-Gitlab/PETSc/src/snes/examples/tutorials/./ex19 >> > Reason: image not found >> > dyld: Library not loaded: @rpath/libeng.dylib >> > Referenced from: /Users/amneetb/Softwares/PETSc-Gitlab/PETSc/src/snes/examples/tutorials/./ex19 >> > Reason: image not found >> > >> > =================================================================================== >> > = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES >> > = PID 14072 RUNNING AT Amneets-MBP >> > = EXIT CODE: 6 >> > = CLEANING UP REMAINING PROCESSES >> > = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES >> > =================================================================================== >> > YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Abort trap: 6 (signal 6) >> > This typically refers to a problem with your application. >> > Please see the FAQ page for debugging suggestions >> /Users/amneetb/Softwares/PETSc-Gitlab/PETSc/src/snes/examples/tutorials >> Possible problem with ex19 running with hypre, diffs above >> ========================================= >> Possible error running Fortran example src/snes/examples/tutorials/ex5f with 1 MPI process >> See http://www.mcs.anl.gov/petsc/documentation/faq.html >> dyld: Library not loaded: @rpath/libeng.dylib >> Referenced from: /Users/amneetb/Softwares/PETSc-Gitlab/PETSc/src/snes/examples/tutorials/./ex5f >> Reason: image not found >> >> =================================================================================== >> = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES >> = PID 14211 RUNNING AT Amneets-MBP >> = EXIT CODE: 6 >> = CLEANING UP REMAINING PROCESSES >> = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES >> =================================================================================== >> YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Abort trap: 6 (signal 6) >> This typically refers to a problem with your application. >> Please see the FAQ page for debugging suggestions >> Completed test examples >> >> >> I have added the following paths in my .profile and sourced it before make all and make check >> >> export DYLD_FALLBACK_LIBRARY_PATH=/Applications/MATLAB_R2019b.app/bin/maci64:/Applications/MATLAB_R2019b.app/sys/os/maci64:$DYLD_FALLBACK_LIBRARY_PATH >> >> export DYLD_LIBRARY_PATH=/Applications/MATLAB_R2019b.app/bin/maci64:/Applications/MATLAB_R2019b.app/sys/os/maci64:$DYLD_LIBRARY_PATH >> >> >> >> On Mon, Jun 8, 2020 at 6:39 PM Barry Smith > wrote: >> >> Looks like the tool to list the examples in the manual pages is broken since it didn't find these other examples, I'll take a look at it. >> >> There is also a discussion in the users manual >> >> $ git grep MatlabEngine | grep ex >> src/docs/tex/manual/part2.tex:PetscMatlabEngineCreate(MPI_Comm comm,char *machine,PetscMatlabEngine *e); >> src/docs/tex/manual/part2.tex:PetscMatlabEnginePut(PetscMatlabEngine e,PetscObject obj); >> src/docs/tex/manual/part2.tex:PetscMatlabEngineGet(PetscMatlabEngine e,PetscObject obj); >> src/docs/tex/manual/part2.tex:PetscMatlabEnginePutArray(PetscMatlabEngine e,int m,int n,PetscScalar *array,char *name); >> src/docs/tex/manual/part2.tex:PetscMatlabEngineGetArray(PetscMatlabEngine e,int m,int n,PetscScalar *array,char *name); >> src/docs/tex/manual/part2.tex:PetscMatlabEngineEvaluate(PetscMatlabEngine,"format",...); >> src/docs/tex/manual/part2.tex:PetscMatlabEngineEvaluate(PetscMatlabEngine,"x = \%g *y + z;",avalue); >> src/docs/tex/manual/part2.tex:PetscMatlabEngineGetOutput(PetscMatlabEngine,char **); >> src/docs/tex/manual/part2.tex:PetscMatlabEnginedPrintOutput(PetscMatlabEngine,FILE*). >> src/snes/tutorials/ex5.c: ierr = PetscMatlabEnginePut(PETSC_MATLAB_ENGINE_(comm),(PetscObject)localX);CHKERRQ(ierr); >> src/snes/tutorials/ex5.c: ierr = PetscMatlabEngineEvaluate(PETSC_MATLAB_ENGINE_(comm),"localF=ex5m(localX,%18.16e,%18.16e,%18.16e)",hx,hy,lambda);CHKERRQ(ierr); >> src/snes/tutorials/ex5.c: ierr = PetscMatlabEngineGet(PETSC_MATLAB_ENGINE_(comm),(PetscObject)localF);CHKERRQ(ierr); >> src/sys/tests/ex16.c:static char help[] = "Demonstrates PetscMatlabEngineXXX()\n"; >> src/sys/tests/ex16.c: ierr = PetscMatlabEngineGetOutput(PETSC_MATLAB_ENGINE_(PETSC_COMM_WORLD),&output);CHKERRQ(ierr); >> src/sys/tests/ex16.c: ierr = PetscMatlabEngineEvaluate(PETSC_MATLAB_ENGINE_(PETSC_COMM_WORLD),"MPI_Comm_rank");CHKERRQ(ierr); >> src/sys/tests/ex16.c: ierr = PetscMatlabEngineEvaluate(PETSC_MATLAB_ENGINE_(PETSC_COMM_WORLD),user);CHKERRQ(ierr); >> src/vec/vec/tests/ex31.c:static const char help[] = "Demonstrates PetscMatlabEngineXXX()\n"; >> src/vec/vec/tests/ex31.c: ierr = PetscMatlabEngineGetOutput(PETSC_MATLAB_ENGINE_WORLD,&output); >> src/vec/vec/tests/ex31.c: ierr = PetscMatlabEngineEvaluate(PETSC_MATLAB_ENGINE_WORLD,"MPI_Comm_rank"); >> src/vec/vec/tests/ex31.c: ierr = PetscMatlabEnginePut(PETSC_MATLAB_ENGINE_WORLD,(PetscObject)x);CHKERRQ(ierr); >> src/vec/vec/tests/ex31.c: ierr = PetscMatlabEngineEvaluate(PETSC_MATLAB_ENGINE_WORLD,"x = x + MPI_Comm_rank;\n"); >> src/vec/vec/tests/ex31.c: ierr = PetscMatlabEngineGet(PETSC_MATLAB_ENGINE_WORLD,(PetscObject)x);CHKERRQ(ierr); >> src/vec/vec/tests/ex31.c: ierr = PetscMatlabEngineEvaluate(PETSC_MATLAB_ENGINE_WORLD,"whos\n");CHKERRQ(ierr); >> >>> On Jun 8, 2020, at 8:26 PM, Amneet Bhalla > wrote: >>> >>> Great! >>> >>> Clicking on some of the functions pointed me to this example >>> https://www.mcs.anl.gov/petsc/petsc-3.13/src/snes/tutorials/ex5.c.html >>> >>> Are there more examples, which are perhaps simpler than the above, which I can refer to? >>> >>> On Mon, Jun 8, 2020 at 5:51 PM Barry Smith > wrote: >>> >>> You should use the PetscMatlabEngine https://www.mcs.anl.gov/petsc/petsc-3.13/docs/manualpages/Sys/PetscMatlabEngine.html >>> >>> Barry >>> >>> >>>> On Jun 8, 2020, at 7:09 PM, Amneet Bhalla > wrote: >>>> >>>> Hi Folks, >>>> >>>> My parallel application written in C++ (which also uses PETSc library for linear solvers) requires calling control and optimization toolbox of MATLAB at every time step to get a small number of double values. Likewise the MATLAB function will need to know some values from the parallel C++ application to return those doubles. I'm wondering if there is a recommended way of calling MATLAB from parallel C/C++/Fortran codes, and if PETSc has some interface that could be leveraged for this exchange. >>>> >>>> Thanks, >>>> -- >>>> --Amneet >>>> >>>> >>>> >>> >>> >>> >>> -- >>> --Amneet >>> >>> >>> >> >> >> >> -- >> --Amneet >> >> >> >> >> >> -- >> --Amneet >> >> >> > > > > -- > --Amneet > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mail2amneet at gmail.com Tue Jun 9 17:48:47 2020 From: mail2amneet at gmail.com (Amneet Bhalla) Date: Tue, 9 Jun 2020 15:48:47 -0700 Subject: [petsc-users] Calling MATLAB code within a parallel C/C++ code In-Reply-To: References: <028B5BA9-4ECA-44B9-BE51-E24000E50185@petsc.dev> <9BE0A2F9-24DF-452B-AABB-1B8FAA748181@petsc.dev> <943847d1-7b6a-8841-f2f8-91652b2f64f2@berkeley.edu> Message-ID: That worked! amneetb at Amneets-MBP:~/Softwares/PETSc-Gitlab/petsc-matlab$ make PETSC_DIR=/Users/amneetb/Softwares/PETSc-Gitlab/petsc-matlab PETSC_ARCH=darwin-dbg check Running check examples to verify correct installation Using PETSC_DIR=/Users/amneetb/Softwares/PETSc-Gitlab/petsc-matlab and PETSC_ARCH=darwin-dbg C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI process C/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI processes C/C++ example src/snes/tutorials/ex19 run successfully with hypre Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI process C/C++ example src/vec/vec/tutorials/ex31 run successfully with Matlab engine Completed test examples Should I remove DYLIB paths from .profile since these are now baked in the linker? Also, Is this branch based-off of latest master? I'm currently using PETSc v3.12.2 for IBAMR. Anything close to this version should work for me. On Tue, Jun 9, 2020 at 3:17 PM Barry Smith wrote: > > I tried including the directories into my DYLD environmental variables > and it didn't work; I am not sure why. This might be why > https://stackoverflow.com/questions/35568122/why-isnt-dyld-library-path-being-propagated-here > > Anyways try my branch and that should work. > > Barry > > > > On Jun 9, 2020, at 5:09 PM, Amneet Bhalla wrote: > > > That's where I also found it. Thanks, Sanjay! > > (This path is included in my .profile file > > export DYLD_FALLBACK_LIBRARY_PATH= > /Applications/MATLAB_R2019b.app/bin/maci64:/Applications/MATLAB_R2019b.app/sys/os/maci64: > $DYLD_FALLBACK_LIBRARY_PATH > > export DYLD_LIBRARY_PATH= > /Applications/MATLAB_R2019b.app/bin/maci64:/Applications/MATLAB_R2019b.app/sys/os/maci64: > $DYLD_LIBRARY_PATH > > > > ) > > Do I need to tell PETSc about this path as well? Something like: > > > --with-matlab-engine-lib=/Applications/MATLAB_R2019b.app/bin/maci64/libeng.dylib > > > On Tue, Jun 9, 2020 at 2:56 PM Sanjay Govindjee wrote: > >> Try looking in: >> >> $(MLROOT)/bin/maci64 >> >> where MLROOT = /Applications/MATLAB_R2019a.app or the like >> >> >> >> On 6/9/20 2:51 PM, Junchao Zhang wrote: >> >> Find recursively where is libeng.dylib under >> /Applications/MATLAB_R2019b.app/extern/engines/ >> >> --Junchao Zhang >> >> >> On Tue, Jun 9, 2020 at 4:27 PM Amneet Bhalla >> wrote: >> >>> Forgot to mention that configure found the following about MATLAB >>> >>> Matlab: >>> Includes: -I/Applications/MATLAB_R2019b.app/extern/include >>> /Applications/MATLAB_R2019b.app >>> MatlabEngine: >>> Library: -L/Applications/MATLAB_R2019b.app/bin/maci64 >>> -L/Applications/MATLAB_R2019b.app/extern/lib/maci64 -leng -lmex -lmx -lmat >>> -lut -licudata -licui18n -licuuc >>> >>> On Tue, Jun 9, 2020 at 2:20 PM Amneet Bhalla >>> wrote: >>> >>>> Do these additional configure flags for MATLAB look OK? >>>> >>>> --with-matlab=1 --with-matlab-dir=/Applications/MATLAB_R2019b.app >>>> --with-matlab-engine=1 >>>> --with-matlab-engine-dir=/Applications/MATLAB_R2019b.app/extern/engines/ >>>> >>>> >>>> >>>> With this configuration make compiled files like: >>>> >>>> CC darwin-dbg/obj/sys/classes/viewer/impls/matlab/vmatlab.o >>>> CC >>>> darwin-dbg/obj/sys/classes/viewer/impls/matlab/ftn-custom/zvmatlabf.o >>>> CC darwin-dbg/obj/sys/classes/matlabengine/matlab.o >>>> CC darwin-dbg/obj/sys/classes/matlabengine/ftn-auto/matlabf.o >>>> CC >>>> darwin-dbg/obj/sys/classes/matlabengine/ftn-custom/zmatlabf.o >>>> CC darwin-dbg/obj/vec/pf/impls/matlab/cmatlab.o >>>> CC darwin-dbg/obj/vec/vec/utils/matlab/gcreatev.o >>>> >>>> make[2]: Leaving directory `/Users/amneetb/Softwares/PETSc-Gitlab/PETSc' >>>> BEGINNING TO COMPILE MATLAB INTERFACE >>>> Building with 'Xcode with Clang'. >>>> ld: warning: cannot export hidden symbol _mexfilerequiredapiversion >>>> from c_mexapi_version.o >>>> >>>> ld: warning: cannot export hidden symbol _mexfilerequiredapiversion >>>> from c_mexapi_version.o >>>> >>>> MEX completed successfully. >>>> Building with 'Xcode with Clang'. >>>> ld: warning: cannot export hidden symbol _mexfilerequiredapiversion >>>> from c_mexapi_version.o >>>> >>>> ld: warning: cannot export hidden symbol _mexfilerequiredapiversion >>>> from c_mexapi_version.o >>>> >>>> MEX completed successfully. >>>> Building with 'Xcode with Clang'. >>>> ld: warning: cannot export hidden symbol _mexfilerequiredapiversion >>>> from c_mexapi_version.o >>>> >>>> MEX completed successfully. >>>> Building with 'Xcode with Clang'. >>>> ld: warning: cannot export hidden symbol _mexfilerequiredapiversion >>>> from c_mexapi_version.o >>>> >>>> ld: warning: cannot export hidden symbol _mexfilerequiredapiversion >>>> from c_mexapi_version.o >>>> >>>> MEX completed successfully. >>>> >>>> >>>> However, make check gives error >>>> >>>> amneetb at Amneets-MBP:~/Softwares/PETSc-Gitlab/PETSc$ make >>>> PETSC_DIR=/Users/amneetb/Softwares/PETSc-Gitlab/PETSc PETSC_ARCH=darwin-dbg >>>> check >>>> Running test examples to verify correct installation >>>> Using PETSC_DIR=/Users/amneetb/Softwares/PETSc-Gitlab/PETSc and >>>> PETSC_ARCH=darwin-dbg >>>> Possible error running C/C++ src/snes/examples/tutorials/ex19 with 1 >>>> MPI process >>>> See http://www.mcs.anl.gov/petsc/documentation/faq.html >>>> dyld: Library not loaded: @rpath/libeng.dylib >>>> Referenced from: >>>> /Users/amneetb/Softwares/PETSc-Gitlab/PETSc/src/snes/examples/tutorials/./ex19 >>>> Reason: image not found >>>> >>>> >>>> =================================================================================== >>>> = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES >>>> = PID 14057 RUNNING AT Amneets-MBP >>>> = EXIT CODE: 6 >>>> = CLEANING UP REMAINING PROCESSES >>>> = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES >>>> >>>> =================================================================================== >>>> YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Abort trap: 6 (signal >>>> 6) >>>> This typically refers to a problem with your application. >>>> Please see the FAQ page for debugging suggestions >>>> Possible error running C/C++ src/snes/examples/tutorials/ex19 with 2 >>>> MPI processes >>>> See http://www.mcs.anl.gov/petsc/documentation/faq.html >>>> dyld: Library not loaded: @rpath/libeng.dylib >>>> Referenced from: >>>> /Users/amneetb/Softwares/PETSc-Gitlab/PETSc/src/snes/examples/tutorials/./ex19 >>>> Reason: image not found >>>> dyld: Library not loaded: @rpath/libeng.dylib >>>> Referenced from: >>>> /Users/amneetb/Softwares/PETSc-Gitlab/PETSc/src/snes/examples/tutorials/./ex19 >>>> Reason: image not found >>>> >>>> >>>> =================================================================================== >>>> = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES >>>> = PID 14062 RUNNING AT Amneets-MBP >>>> = EXIT CODE: 6 >>>> = CLEANING UP REMAINING PROCESSES >>>> = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES >>>> >>>> =================================================================================== >>>> YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Abort trap: 6 (signal >>>> 6) >>>> This typically refers to a problem with your application. >>>> Please see the FAQ page for debugging suggestions >>>> 1,5c1,17 >>>> < lid velocity = 0.0016, prandtl # = 1., grashof # = 1. >>>> < 0 SNES Function norm 0.0406612 >>>> < 1 SNES Function norm 4.12227e-06 >>>> < 2 SNES Function norm 6.098e-11 >>>> < Number of SNES iterations = 2 >>>> --- >>>> > dyld: Library not loaded: @rpath/libeng.dylib >>>> > Referenced from: >>>> /Users/amneetb/Softwares/PETSc-Gitlab/PETSc/src/snes/examples/tutorials/./ex19 >>>> > Reason: image not found >>>> > dyld: Library not loaded: @rpath/libeng.dylib >>>> > Referenced from: >>>> /Users/amneetb/Softwares/PETSc-Gitlab/PETSc/src/snes/examples/tutorials/./ex19 >>>> > Reason: image not found >>>> > >>>> > >>>> =================================================================================== >>>> > = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES >>>> > = PID 14072 RUNNING AT Amneets-MBP >>>> > = EXIT CODE: 6 >>>> > = CLEANING UP REMAINING PROCESSES >>>> > = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES >>>> > >>>> =================================================================================== >>>> > YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Abort trap: 6 >>>> (signal 6) >>>> > This typically refers to a problem with your application. >>>> > Please see the FAQ page for debugging suggestions >>>> /Users/amneetb/Softwares/PETSc-Gitlab/PETSc/src/snes/examples/tutorials >>>> Possible problem with ex19 running with hypre, diffs above >>>> ========================================= >>>> Possible error running Fortran example src/snes/examples/tutorials/ex5f >>>> with 1 MPI process >>>> See http://www.mcs.anl.gov/petsc/documentation/faq.html >>>> dyld: Library not loaded: @rpath/libeng.dylib >>>> Referenced from: >>>> /Users/amneetb/Softwares/PETSc-Gitlab/PETSc/src/snes/examples/tutorials/./ex5f >>>> Reason: image not found >>>> >>>> >>>> =================================================================================== >>>> = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES >>>> = PID 14211 RUNNING AT Amneets-MBP >>>> = EXIT CODE: 6 >>>> = CLEANING UP REMAINING PROCESSES >>>> = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES >>>> >>>> =================================================================================== >>>> YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Abort trap: 6 (signal >>>> 6) >>>> This typically refers to a problem with your application. >>>> Please see the FAQ page for debugging suggestions >>>> Completed test examples >>>> >>>> >>>> I have added the following paths in my .profile and sourced it before >>>> make all and make check >>>> >>>> export DYLD_FALLBACK_LIBRARY_PATH= >>>> /Applications/MATLAB_R2019b.app/bin/maci64:/Applications/MATLAB_R2019b.app/sys/os/maci64: >>>> $DYLD_FALLBACK_LIBRARY_PATH >>>> >>>> export DYLD_LIBRARY_PATH= >>>> /Applications/MATLAB_R2019b.app/bin/maci64:/Applications/MATLAB_R2019b.app/sys/os/maci64: >>>> $DYLD_LIBRARY_PATH >>>> >>>> >>>> >>>> On Mon, Jun 8, 2020 at 6:39 PM Barry Smith wrote: >>>> >>>>> >>>>> Looks like the tool to list the examples in the manual pages is >>>>> broken since it didn't find these other examples, I'll take a look at it. >>>>> >>>>> There is also a discussion in the users manual >>>>> >>>>> $ git grep MatlabEngine | grep ex >>>>> src/docs/tex/manual/part2.tex:PetscMatlabEngineCreate(MPI_Comm >>>>> comm,char *machine,PetscMatlabEngine *e); >>>>> src/docs/tex/manual/part2.tex:PetscMatlabEnginePut(PetscMatlabEngine >>>>> e,PetscObject obj); >>>>> src/docs/tex/manual/part2.tex:PetscMatlabEngineGet(PetscMatlabEngine >>>>> e,PetscObject obj); >>>>> src/docs/tex/manual/part2.tex:PetscMatlabEnginePutArray(PetscMatlabEngine >>>>> e,int m,int n,PetscScalar *array,char *name); >>>>> src/docs/tex/manual/part2.tex:PetscMatlabEngineGetArray(PetscMatlabEngine >>>>> e,int m,int n,PetscScalar *array,char *name); >>>>> >>>>> src/docs/tex/manual/part2.tex:PetscMatlabEngineEvaluate(PetscMatlabEngine,"format",...); >>>>> src/docs/tex/manual/part2.tex:PetscMatlabEngineEvaluate(PetscMatlabEngine,"x >>>>> = \%g *y + z;",avalue); >>>>> src/docs/tex/manual/part2.tex:PetscMatlabEngineGetOutput(PetscMatlabEngine,char >>>>> **); >>>>> >>>>> src/docs/tex/manual/part2.tex:PetscMatlabEnginedPrintOutput(PetscMatlabEngine,FILE*). >>>>> src/snes/tutorials/ex5.c: ierr = >>>>> PetscMatlabEnginePut(PETSC_MATLAB_ENGINE_(comm),(PetscObject)localX);CHKERRQ(ierr); >>>>> src/snes/tutorials/ex5.c: ierr = >>>>> PetscMatlabEngineEvaluate(PETSC_MATLAB_ENGINE_(comm),"localF=ex5m(localX,%18.16e,%18.16e,%18.16e)",hx,hy,lambda);CHKERRQ(ierr); >>>>> src/snes/tutorials/ex5.c: ierr = >>>>> PetscMatlabEngineGet(PETSC_MATLAB_ENGINE_(comm),(PetscObject)localF);CHKERRQ(ierr); >>>>> src/sys/tests/ex16.c:static char help[] = "Demonstrates >>>>> PetscMatlabEngineXXX()\n"; >>>>> src/sys/tests/ex16.c: ierr = >>>>> PetscMatlabEngineGetOutput(PETSC_MATLAB_ENGINE_(PETSC_COMM_WORLD),&output);CHKERRQ(ierr); >>>>> src/sys/tests/ex16.c: ierr = >>>>> PetscMatlabEngineEvaluate(PETSC_MATLAB_ENGINE_(PETSC_COMM_WORLD),"MPI_Comm_rank");CHKERRQ(ierr); >>>>> src/sys/tests/ex16.c: ierr = >>>>> PetscMatlabEngineEvaluate(PETSC_MATLAB_ENGINE_(PETSC_COMM_WORLD),user);CHKERRQ(ierr); >>>>> src/vec/vec/tests/ex31.c:static const char help[] = "Demonstrates >>>>> PetscMatlabEngineXXX()\n"; >>>>> src/vec/vec/tests/ex31.c: ierr = >>>>> PetscMatlabEngineGetOutput(PETSC_MATLAB_ENGINE_WORLD,&output); >>>>> src/vec/vec/tests/ex31.c: ierr = >>>>> PetscMatlabEngineEvaluate(PETSC_MATLAB_ENGINE_WORLD,"MPI_Comm_rank"); >>>>> src/vec/vec/tests/ex31.c: ierr = >>>>> PetscMatlabEnginePut(PETSC_MATLAB_ENGINE_WORLD,(PetscObject)x);CHKERRQ(ierr); >>>>> src/vec/vec/tests/ex31.c: ierr = >>>>> PetscMatlabEngineEvaluate(PETSC_MATLAB_ENGINE_WORLD,"x = x + >>>>> MPI_Comm_rank;\n"); >>>>> src/vec/vec/tests/ex31.c: ierr = >>>>> PetscMatlabEngineGet(PETSC_MATLAB_ENGINE_WORLD,(PetscObject)x);CHKERRQ(ierr); >>>>> src/vec/vec/tests/ex31.c: ierr = >>>>> PetscMatlabEngineEvaluate(PETSC_MATLAB_ENGINE_WORLD,"whos\n");CHKERRQ(ierr); >>>>> >>>>> On Jun 8, 2020, at 8:26 PM, Amneet Bhalla >>>>> wrote: >>>>> >>>>> Great! >>>>> >>>>> Clicking on some of the functions pointed me to this example >>>>> https://www.mcs.anl.gov/petsc/petsc-3.13/src/snes/tutorials/ex5.c.html >>>>> >>>>> Are there more examples, which are perhaps simpler than the above, >>>>> which I can refer to? >>>>> >>>>> On Mon, Jun 8, 2020 at 5:51 PM Barry Smith wrote: >>>>> >>>>>> >>>>>> You should use the PetscMatlabEngine >>>>>> https://www.mcs.anl.gov/petsc/petsc-3.13/docs/manualpages/Sys/PetscMatlabEngine.html >>>>>> >>>>>> Barry >>>>>> >>>>>> >>>>>> On Jun 8, 2020, at 7:09 PM, Amneet Bhalla >>>>>> wrote: >>>>>> >>>>>> Hi Folks, >>>>>> >>>>>> My parallel application written in C++ (which also uses PETSc library >>>>>> for linear solvers) requires calling control and optimization toolbox of >>>>>> MATLAB at every time step to get a small number of double values. Likewise >>>>>> the MATLAB function will need to know some values from the parallel C++ >>>>>> application to return those doubles. I'm wondering if there is a >>>>>> recommended way of calling MATLAB from parallel C/C++/Fortran codes, and if >>>>>> PETSc has some interface that could be leveraged for this exchange. >>>>>> >>>>>> Thanks, >>>>>> -- >>>>>> --Amneet >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>> >>>>> -- >>>>> --Amneet >>>>> >>>>> >>>>> >>>>> >>>>> >>>> >>>> -- >>>> --Amneet >>>> >>>> >>>> >>>> >>> >>> -- >>> --Amneet >>> >>> >>> >>> >> > > -- > --Amneet > > > > > -- --Amneet -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Tue Jun 9 18:26:51 2020 From: bsmith at petsc.dev (Barry Smith) Date: Tue, 9 Jun 2020 18:26:51 -0500 Subject: [petsc-users] Calling MATLAB code within a parallel C/C++ code In-Reply-To: References: <028B5BA9-4ECA-44B9-BE51-E24000E50185@petsc.dev> <9BE0A2F9-24DF-452B-AABB-1B8FAA748181@petsc.dev> <943847d1-7b6a-8841-f2f8-91652b2f64f2@berkeley.edu> Message-ID: <5E3EB0AC-3E6E-41AB-8441-11B8941F8554@petsc.dev> Great, yes you should not need the DYLIB paths The branch is off master but what you can do is copy the config/BuildSystem/config/MatlabEngine.py to your PETSc 3.12.2 version and the configure should work (it won't automatically run the test case but you can test it manually). Barry > On Jun 9, 2020, at 5:48 PM, Amneet Bhalla wrote: > > That worked! > > amneetb at Amneets-MBP:~/Softwares/PETSc-Gitlab/petsc-matlab$ make PETSC_DIR=/Users/amneetb/Softwares/PETSc-Gitlab/petsc-matlab PETSC_ARCH=darwin-dbg check > Running check examples to verify correct installation > Using PETSC_DIR=/Users/amneetb/Softwares/PETSc-Gitlab/petsc-matlab and PETSC_ARCH=darwin-dbg > C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI process > C/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI processes > C/C++ example src/snes/tutorials/ex19 run successfully with hypre > Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI process > C/C++ example src/vec/vec/tutorials/ex31 run successfully with Matlab engine > Completed test examples > > Should I remove DYLIB paths from .profile since these are now baked in the linker? > > Also, Is this branch based-off of latest master? I'm currently using PETSc v3.12.2 for IBAMR. Anything close to this version should work for me. > > > On Tue, Jun 9, 2020 at 3:17 PM Barry Smith > wrote: > > I tried including the directories into my DYLD environmental variables and it didn't work; I am not sure why. This might be why https://stackoverflow.com/questions/35568122/why-isnt-dyld-library-path-being-propagated-here > > Anyways try my branch and that should work. > > Barry > > > >> On Jun 9, 2020, at 5:09 PM, Amneet Bhalla > wrote: >> >> >> That's where I also found it. Thanks, Sanjay! >> >> (This path is included in my .profile file >> >> export DYLD_FALLBACK_LIBRARY_PATH=/Applications/MATLAB_R2019b.app/bin/maci64:/Applications/MATLAB_R2019b.app/sys/os/maci64:$DYLD_FALLBACK_LIBRARY_PATH >> >> export DYLD_LIBRARY_PATH=/Applications/MATLAB_R2019b.app/bin/maci64:/Applications/MATLAB_R2019b.app/sys/os/maci64:$DYLD_LIBRARY_PATH >> >> >> >> ) >> >> Do I need to tell PETSc about this path as well? Something like: >> >> --with-matlab-engine-lib=/Applications/MATLAB_R2019b.app/bin/maci64/libeng.dylib >> >> >> On Tue, Jun 9, 2020 at 2:56 PM Sanjay Govindjee > wrote: >> Try looking in: >> >> $(MLROOT)/bin/maci64 >> >> where MLROOT = /Applications/MATLAB_R2019a.app or the like >> >> >> On 6/9/20 2:51 PM, Junchao Zhang wrote: >>> Find recursively where is libeng.dylib under /Applications/MATLAB_R2019b.app/extern/engines/ >>> >>> --Junchao Zhang >>> >>> >>> On Tue, Jun 9, 2020 at 4:27 PM Amneet Bhalla > wrote: >>> Forgot to mention that configure found the following about MATLAB >>> >>> Matlab: >>> Includes: -I/Applications/MATLAB_R2019b.app/extern/include >>> /Applications/MATLAB_R2019b.app >>> MatlabEngine: >>> Library: -L/Applications/MATLAB_R2019b.app/bin/maci64 -L/Applications/MATLAB_R2019b.app/extern/lib/maci64 -leng -lmex -lmx -lmat -lut -licudata -licui18n -licuuc >>> >>> On Tue, Jun 9, 2020 at 2:20 PM Amneet Bhalla > wrote: >>> Do these additional configure flags for MATLAB look OK? >>> >>> --with-matlab=1 --with-matlab-dir=/Applications/MATLAB_R2019b.app --with-matlab-engine=1 --with-matlab-engine-dir=/Applications/MATLAB_R2019b.app/extern/engines/ >>> >>> >>> >>> With this configuration make compiled files like: >>> >>> CC darwin-dbg/obj/sys/classes/viewer/impls/matlab/vmatlab.o >>> CC darwin-dbg/obj/sys/classes/viewer/impls/matlab/ftn-custom/zvmatlabf.o >>> CC darwin-dbg/obj/sys/classes/matlabengine/matlab.o >>> CC darwin-dbg/obj/sys/classes/matlabengine/ftn-auto/matlabf.o >>> CC darwin-dbg/obj/sys/classes/matlabengine/ftn-custom/zmatlabf.o >>> CC darwin-dbg/obj/vec/pf/impls/matlab/cmatlab.o >>> CC darwin-dbg/obj/vec/vec/utils/matlab/gcreatev.o >>> >>> make[2]: Leaving directory `/Users/amneetb/Softwares/PETSc-Gitlab/PETSc' >>> BEGINNING TO COMPILE MATLAB INTERFACE >>> Building with 'Xcode with Clang'. >>> ld: warning: cannot export hidden symbol _mexfilerequiredapiversion from c_mexapi_version.o >>> >>> ld: warning: cannot export hidden symbol _mexfilerequiredapiversion from c_mexapi_version.o >>> >>> MEX completed successfully. >>> Building with 'Xcode with Clang'. >>> ld: warning: cannot export hidden symbol _mexfilerequiredapiversion from c_mexapi_version.o >>> >>> ld: warning: cannot export hidden symbol _mexfilerequiredapiversion from c_mexapi_version.o >>> >>> MEX completed successfully. >>> Building with 'Xcode with Clang'. >>> ld: warning: cannot export hidden symbol _mexfilerequiredapiversion from c_mexapi_version.o >>> >>> MEX completed successfully. >>> Building with 'Xcode with Clang'. >>> ld: warning: cannot export hidden symbol _mexfilerequiredapiversion from c_mexapi_version.o >>> >>> ld: warning: cannot export hidden symbol _mexfilerequiredapiversion from c_mexapi_version.o >>> >>> MEX completed successfully. >>> >>> >>> However, make check gives error >>> >>> amneetb at Amneets-MBP:~/Softwares/PETSc-Gitlab/PETSc$ make PETSC_DIR=/Users/amneetb/Softwares/PETSc-Gitlab/PETSc PETSC_ARCH=darwin-dbg check >>> Running test examples to verify correct installation >>> Using PETSC_DIR=/Users/amneetb/Softwares/PETSc-Gitlab/PETSc and PETSC_ARCH=darwin-dbg >>> Possible error running C/C++ src/snes/examples/tutorials/ex19 with 1 MPI process >>> See http://www.mcs.anl.gov/petsc/documentation/faq.html >>> dyld: Library not loaded: @rpath/libeng.dylib >>> Referenced from: /Users/amneetb/Softwares/PETSc-Gitlab/PETSc/src/snes/examples/tutorials/./ex19 >>> Reason: image not found >>> >>> =================================================================================== >>> = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES >>> = PID 14057 RUNNING AT Amneets-MBP >>> = EXIT CODE: 6 >>> = CLEANING UP REMAINING PROCESSES >>> = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES >>> =================================================================================== >>> YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Abort trap: 6 (signal 6) >>> This typically refers to a problem with your application. >>> Please see the FAQ page for debugging suggestions >>> Possible error running C/C++ src/snes/examples/tutorials/ex19 with 2 MPI processes >>> See http://www.mcs.anl.gov/petsc/documentation/faq.html >>> dyld: Library not loaded: @rpath/libeng.dylib >>> Referenced from: /Users/amneetb/Softwares/PETSc-Gitlab/PETSc/src/snes/examples/tutorials/./ex19 >>> Reason: image not found >>> dyld: Library not loaded: @rpath/libeng.dylib >>> Referenced from: /Users/amneetb/Softwares/PETSc-Gitlab/PETSc/src/snes/examples/tutorials/./ex19 >>> Reason: image not found >>> >>> =================================================================================== >>> = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES >>> = PID 14062 RUNNING AT Amneets-MBP >>> = EXIT CODE: 6 >>> = CLEANING UP REMAINING PROCESSES >>> = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES >>> =================================================================================== >>> YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Abort trap: 6 (signal 6) >>> This typically refers to a problem with your application. >>> Please see the FAQ page for debugging suggestions >>> 1,5c1,17 >>> < lid velocity = 0.0016, prandtl # = 1., grashof # = 1. >>> < 0 SNES Function norm 0.0406612 >>> < 1 SNES Function norm 4.12227e-06 >>> < 2 SNES Function norm 6.098e-11 >>> < Number of SNES iterations = 2 >>> --- >>> > dyld: Library not loaded: @rpath/libeng.dylib >>> > Referenced from: /Users/amneetb/Softwares/PETSc-Gitlab/PETSc/src/snes/examples/tutorials/./ex19 >>> > Reason: image not found >>> > dyld: Library not loaded: @rpath/libeng.dylib >>> > Referenced from: /Users/amneetb/Softwares/PETSc-Gitlab/PETSc/src/snes/examples/tutorials/./ex19 >>> > Reason: image not found >>> > >>> > =================================================================================== >>> > = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES >>> > = PID 14072 RUNNING AT Amneets-MBP >>> > = EXIT CODE: 6 >>> > = CLEANING UP REMAINING PROCESSES >>> > = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES >>> > =================================================================================== >>> > YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Abort trap: 6 (signal 6) >>> > This typically refers to a problem with your application. >>> > Please see the FAQ page for debugging suggestions >>> /Users/amneetb/Softwares/PETSc-Gitlab/PETSc/src/snes/examples/tutorials >>> Possible problem with ex19 running with hypre, diffs above >>> ========================================= >>> Possible error running Fortran example src/snes/examples/tutorials/ex5f with 1 MPI process >>> See http://www.mcs.anl.gov/petsc/documentation/faq.html >>> dyld: Library not loaded: @rpath/libeng.dylib >>> Referenced from: /Users/amneetb/Softwares/PETSc-Gitlab/PETSc/src/snes/examples/tutorials/./ex5f >>> Reason: image not found >>> >>> =================================================================================== >>> = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES >>> = PID 14211 RUNNING AT Amneets-MBP >>> = EXIT CODE: 6 >>> = CLEANING UP REMAINING PROCESSES >>> = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES >>> =================================================================================== >>> YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Abort trap: 6 (signal 6) >>> This typically refers to a problem with your application. >>> Please see the FAQ page for debugging suggestions >>> Completed test examples >>> >>> >>> I have added the following paths in my .profile and sourced it before make all and make check >>> >>> export DYLD_FALLBACK_LIBRARY_PATH=/Applications/MATLAB_R2019b.app/bin/maci64:/Applications/MATLAB_R2019b.app/sys/os/maci64:$DYLD_FALLBACK_LIBRARY_PATH >>> >>> export DYLD_LIBRARY_PATH=/Applications/MATLAB_R2019b.app/bin/maci64:/Applications/MATLAB_R2019b.app/sys/os/maci64:$DYLD_LIBRARY_PATH >>> >>> >>> >>> On Mon, Jun 8, 2020 at 6:39 PM Barry Smith > wrote: >>> >>> Looks like the tool to list the examples in the manual pages is broken since it didn't find these other examples, I'll take a look at it. >>> >>> There is also a discussion in the users manual >>> >>> $ git grep MatlabEngine | grep ex >>> src/docs/tex/manual/part2.tex:PetscMatlabEngineCreate(MPI_Comm comm,char *machine,PetscMatlabEngine *e); >>> src/docs/tex/manual/part2.tex:PetscMatlabEnginePut(PetscMatlabEngine e,PetscObject obj); >>> src/docs/tex/manual/part2.tex:PetscMatlabEngineGet(PetscMatlabEngine e,PetscObject obj); >>> src/docs/tex/manual/part2.tex:PetscMatlabEnginePutArray(PetscMatlabEngine e,int m,int n,PetscScalar *array,char *name); >>> src/docs/tex/manual/part2.tex:PetscMatlabEngineGetArray(PetscMatlabEngine e,int m,int n,PetscScalar *array,char *name); >>> src/docs/tex/manual/part2.tex:PetscMatlabEngineEvaluate(PetscMatlabEngine,"format",...); >>> src/docs/tex/manual/part2.tex:PetscMatlabEngineEvaluate(PetscMatlabEngine,"x = \%g *y + z;",avalue); >>> src/docs/tex/manual/part2.tex:PetscMatlabEngineGetOutput(PetscMatlabEngine,char **); >>> src/docs/tex/manual/part2.tex:PetscMatlabEnginedPrintOutput(PetscMatlabEngine,FILE*). >>> src/snes/tutorials/ex5.c: ierr = PetscMatlabEnginePut(PETSC_MATLAB_ENGINE_(comm),(PetscObject)localX);CHKERRQ(ierr); >>> src/snes/tutorials/ex5.c: ierr = PetscMatlabEngineEvaluate(PETSC_MATLAB_ENGINE_(comm),"localF=ex5m(localX,%18.16e,%18.16e,%18.16e)",hx,hy,lambda);CHKERRQ(ierr); >>> src/snes/tutorials/ex5.c: ierr = PetscMatlabEngineGet(PETSC_MATLAB_ENGINE_(comm),(PetscObject)localF);CHKERRQ(ierr); >>> src/sys/tests/ex16.c:static char help[] = "Demonstrates PetscMatlabEngineXXX()\n"; >>> src/sys/tests/ex16.c: ierr = PetscMatlabEngineGetOutput(PETSC_MATLAB_ENGINE_(PETSC_COMM_WORLD),&output);CHKERRQ(ierr); >>> src/sys/tests/ex16.c: ierr = PetscMatlabEngineEvaluate(PETSC_MATLAB_ENGINE_(PETSC_COMM_WORLD),"MPI_Comm_rank");CHKERRQ(ierr); >>> src/sys/tests/ex16.c: ierr = PetscMatlabEngineEvaluate(PETSC_MATLAB_ENGINE_(PETSC_COMM_WORLD),user);CHKERRQ(ierr); >>> src/vec/vec/tests/ex31.c:static const char help[] = "Demonstrates PetscMatlabEngineXXX()\n"; >>> src/vec/vec/tests/ex31.c: ierr = PetscMatlabEngineGetOutput(PETSC_MATLAB_ENGINE_WORLD,&output); >>> src/vec/vec/tests/ex31.c: ierr = PetscMatlabEngineEvaluate(PETSC_MATLAB_ENGINE_WORLD,"MPI_Comm_rank"); >>> src/vec/vec/tests/ex31.c: ierr = PetscMatlabEnginePut(PETSC_MATLAB_ENGINE_WORLD,(PetscObject)x);CHKERRQ(ierr); >>> src/vec/vec/tests/ex31.c: ierr = PetscMatlabEngineEvaluate(PETSC_MATLAB_ENGINE_WORLD,"x = x + MPI_Comm_rank;\n"); >>> src/vec/vec/tests/ex31.c: ierr = PetscMatlabEngineGet(PETSC_MATLAB_ENGINE_WORLD,(PetscObject)x);CHKERRQ(ierr); >>> src/vec/vec/tests/ex31.c: ierr = PetscMatlabEngineEvaluate(PETSC_MATLAB_ENGINE_WORLD,"whos\n");CHKERRQ(ierr); >>> >>>> On Jun 8, 2020, at 8:26 PM, Amneet Bhalla > wrote: >>>> >>>> Great! >>>> >>>> Clicking on some of the functions pointed me to this example >>>> https://www.mcs.anl.gov/petsc/petsc-3.13/src/snes/tutorials/ex5.c.html >>>> >>>> Are there more examples, which are perhaps simpler than the above, which I can refer to? >>>> >>>> On Mon, Jun 8, 2020 at 5:51 PM Barry Smith > wrote: >>>> >>>> You should use the PetscMatlabEngine https://www.mcs.anl.gov/petsc/petsc-3.13/docs/manualpages/Sys/PetscMatlabEngine.html >>>> >>>> Barry >>>> >>>> >>>>> On Jun 8, 2020, at 7:09 PM, Amneet Bhalla > wrote: >>>>> >>>>> Hi Folks, >>>>> >>>>> My parallel application written in C++ (which also uses PETSc library for linear solvers) requires calling control and optimization toolbox of MATLAB at every time step to get a small number of double values. Likewise the MATLAB function will need to know some values from the parallel C++ application to return those doubles. I'm wondering if there is a recommended way of calling MATLAB from parallel C/C++/Fortran codes, and if PETSc has some interface that could be leveraged for this exchange. >>>>> >>>>> Thanks, >>>>> -- >>>>> --Amneet >>>>> >>>>> >>>>> >>>> >>>> >>>> >>>> -- >>>> --Amneet >>>> >>>> >>>> >>> >>> >>> >>> -- >>> --Amneet >>> >>> >>> >>> >>> >>> -- >>> --Amneet >>> >>> >>> >> >> >> >> -- >> --Amneet >> >> >> > > > > -- > --Amneet > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mail2amneet at gmail.com Tue Jun 9 18:29:55 2020 From: mail2amneet at gmail.com (Amneet Bhalla) Date: Tue, 9 Jun 2020 16:29:55 -0700 Subject: [petsc-users] Fwd: Post Doc Position in CFD In-Reply-To: References: Message-ID: FYI ---------- Forwarded message --------- From: Nils Tilton Date: Tue, Jun 9, 2020 at 11:15 AM Subject: Post Doc Position in CFD To: Amneet Pal Bhalla , Dear Colleagues, I hope this email finds you all well. Could you please help me advertise the attached post doc position opening in my group at the Colorado School of Mines? I am looking for someone experienced in CFD and High Performance Computing to help me parallelize codes developed in my group so we can expand to 3D. The codes use a combination of finite-volume, projection, and immersed boundary methods. Thanks very much for your help. Best Wishes, Nils -- --Amneet -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Announcement.pdf Type: application/pdf Size: 515246 bytes Desc: not available URL: From fdkong.jd at gmail.com Tue Jun 9 19:03:11 2020 From: fdkong.jd at gmail.com (Fande Kong) Date: Tue, 9 Jun 2020 18:03:11 -0600 Subject: [petsc-users] Make stream In-Reply-To: <5364BA02-DF4C-446D-844A-86EC019EB7B2@petsc.dev> References: <5364BA02-DF4C-446D-844A-86EC019EB7B2@petsc.dev> Message-ID: Thanks so much, Barry, On Tue, Jun 9, 2020 at 3:08 PM Barry Smith wrote: > > You might look at the notes about MPI binding. It might give you a bit > better performance. > https://www.mcs.anl.gov/petsc/documentation/faq.html#computers > I am using mvapich2, and still trying to look for which binding command lines I can use. > > The streams is exactly the DAXPY operation so this is the speed up you > should expect for VecAXPY() which has 2 loads and 1 store per 1 multipy and > 1 add > > VecDot() has 2 loads per 1 multiply and 1 add but also a global > reduction > > Sparse multiply with AIJ has an integer load, 2 double loads plus 1 > store per row with 1 multiply and 1 add plus communication needed for > off-process portion > > Function evaluations often have higher arithmetic intensity so should > give a bit higher speedup > > Jacobian evaluations often have higher arithmetic intensity but they > may have MatSetValues() which is slow because no arithmetic intensity just > memory motion > Got it. Thanks, Fande, > > Barry > > > > On Jun 9, 2020, at 3:43 PM, Fande Kong wrote: > > Hi All, > > I am trying to interpret the results from "make stream" on two compute > nodes, where each node has 48 cores. > > If my calculations are memory bandwidth limited, such as AMG, MatVec, > GMRES, etc.. > The best speedup I could get is 16.6938 if I start from one core?? The > speedup for function evaluations and Jacobian evaluations can be better > than16.6938? > > Thanks, > > Fande, > > > > Running streams with 'mpiexec ' using 'NPMAX=96' > 1 19412.4570 Rate (MB/s) > 2 29457.3988 Rate (MB/s) 1.51744 > 3 40483.9318 Rate (MB/s) 2.08546 > 4 51429.3431 Rate (MB/s) 2.64929 > 5 59849.5168 Rate (MB/s) 3.08304 > 6 66124.3461 Rate (MB/s) 3.40628 > 7 70888.1170 Rate (MB/s) 3.65167 > 8 73436.2374 Rate (MB/s) 3.78294 > 9 77441.7622 Rate (MB/s) 3.98927 > 10 78115.3114 Rate (MB/s) 4.02397 > 11 81449.3315 Rate (MB/s) 4.19572 > 12 82812.3471 Rate (MB/s) 4.26593 > 13 81442.2114 Rate (MB/s) 4.19535 > 14 83404.1657 Rate (MB/s) 4.29642 > 15 84165.8536 Rate (MB/s) 4.33565 > 16 83739.2910 Rate (MB/s) 4.31368 > 17 83724.8109 Rate (MB/s) 4.31293 > 18 83225.0743 Rate (MB/s) 4.28719 > 19 81668.2002 Rate (MB/s) 4.20699 > 20 83678.8007 Rate (MB/s) 4.31056 > 21 81400.4590 Rate (MB/s) 4.1932 > 22 81944.8975 Rate (MB/s) 4.22124 > 23 81359.8615 Rate (MB/s) 4.19111 > 24 80674.5064 Rate (MB/s) 4.1558 > 25 83761.3316 Rate (MB/s) 4.31481 > 26 87567.4876 Rate (MB/s) 4.51088 > 27 89605.4435 Rate (MB/s) 4.61586 > 28 94984.9755 Rate (MB/s) 4.89298 > 29 98260.5283 Rate (MB/s) 5.06171 > 30 99852.8790 Rate (MB/s) 5.14374 > 31 102736.3576 Rate (MB/s) 5.29228 > 32 108638.7488 Rate (MB/s) 5.59633 > 33 110431.2938 Rate (MB/s) 5.68867 > 34 112824.2031 Rate (MB/s) 5.81194 > 35 116908.3009 Rate (MB/s) 6.02232 > 36 121312.6574 Rate (MB/s) 6.2492 > 37 122507.3172 Rate (MB/s) 6.31074 > 38 127456.2504 Rate (MB/s) 6.56568 > 39 130098.7056 Rate (MB/s) 6.7018 > 40 134956.4461 Rate (MB/s) 6.95204 > 41 138309.2465 Rate (MB/s) 7.12475 > 42 141779.7997 Rate (MB/s) 7.30353 > 43 145653.3687 Rate (MB/s) 7.50307 > 44 149131.2087 Rate (MB/s) 7.68223 > 45 151611.6104 Rate (MB/s) 7.81 > 46 155554.6394 Rate (MB/s) 8.01312 > 47 159033.1938 Rate (MB/s) 8.19231 > 48 162216.5600 Rate (MB/s) 8.35629 > 49 165034.8116 Rate (MB/s) 8.50147 > 50 168001.4823 Rate (MB/s) 8.65429 > 51 170899.9045 Rate (MB/s) 8.8036 > 52 175687.8033 Rate (MB/s) 9.05024 > 53 178203.9203 Rate (MB/s) 9.17985 > 54 179973.3914 Rate (MB/s) 9.27101 > 55 182207.3495 Rate (MB/s) 9.38608 > 56 185712.9643 Rate (MB/s) 9.56667 > 57 188805.5696 Rate (MB/s) 9.72598 > 58 193360.9158 Rate (MB/s) 9.96064 > 59 198160.8016 Rate (MB/s) 10.2079 > 60 201297.0129 Rate (MB/s) 10.3695 > 61 203618.7672 Rate (MB/s) 10.4891 > 62 209599.2783 Rate (MB/s) 10.7971 > 63 211651.1587 Rate (MB/s) 10.9028 > 64 210254.5035 Rate (MB/s) 10.8309 > 65 218576.4938 Rate (MB/s) 11.2596 > 66 220280.0853 Rate (MB/s) 11.3473 > 67 221281.1867 Rate (MB/s) 11.3989 > 68 228941.1872 Rate (MB/s) 11.7935 > 69 232206.2708 Rate (MB/s) 11.9617 > 70 233569.5866 Rate (MB/s) 12.0319 > 71 238293.6355 Rate (MB/s) 12.2753 > 72 238987.0729 Rate (MB/s) 12.311 > 73 246013.4684 Rate (MB/s) 12.6729 > 74 248850.8942 Rate (MB/s) 12.8191 > 75 249355.6899 Rate (MB/s) 12.8451 > 76 252515.6110 Rate (MB/s) 13.0079 > 77 257489.4268 Rate (MB/s) 13.2641 > 78 260884.2771 Rate (MB/s) 13.439 > 79 264341.8661 Rate (MB/s) 13.6171 > 80 269329.1376 Rate (MB/s) 13.874 > 81 272286.4070 Rate (MB/s) 14.0263 > 82 273325.7822 Rate (MB/s) 14.0799 > 83 277334.6699 Rate (MB/s) 14.2864 > 84 280254.7286 Rate (MB/s) 14.4368 > 85 282219.8194 Rate (MB/s) 14.538 > 86 289039.2677 Rate (MB/s) 14.8893 > 87 291234.4715 Rate (MB/s) 15.0024 > 88 295941.1159 Rate (MB/s) 15.2449 > 89 298136.3163 Rate (MB/s) 15.358 > 90 302820.9080 Rate (MB/s) 15.5993 > 91 306387.5008 Rate (MB/s) 15.783 > 92 310127.0223 Rate (MB/s) 15.9756 > 93 310219.3643 Rate (MB/s) 15.9804 > 94 317089.5971 Rate (MB/s) 16.3343 > 95 315457.0938 Rate (MB/s) 16.2502 > 96 324068.8172 Rate (MB/s) 16.6938 > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From g.gibb at epcc.ed.ac.uk Tue Jun 9 19:11:19 2020 From: g.gibb at epcc.ed.ac.uk (GIBB Gordon) Date: Wed, 10 Jun 2020 00:11:19 +0000 Subject: [petsc-users] A series of GPU questions Message-ID: <4AD81243-3693-455E-B6D7-708BCDBFFBE5@epcc.ed.ac.uk> Hi, First of all, my apologies if this is not the appropriate list to send these questions to. I?m one of the developers of TPLS (https://sourceforge.net/projects/tpls/), a Fortran code that uses PETSc, parallelised using DM vectors. It uses a mix of our own solvers, and PETSc?s Krylov solvers. At present it has been run on up to 25,000 MPI processes, although larger problem sizes should be able to scale beyond that. With the awareness that more and more HPC machines now have one or more GPUs per node, and that upcoming machines that approach/achieve Exascale will be heterogeneous in nature, we are investigating whether it is worth using GPUs with TPLS, and if so, how to best do this. I see that in principle all we?d need to do to is set some flags as described at https://www.mcs.anl.gov/petsc/features/gpus.html to offload work onto the GPU, however I have some questions about doing this in practice: The GPU machine I have access to has nodes with two 20 core CPUs and 4 NVIDIA GPUs (so 10 cores per GPU). We could use CUDA or OpenCL, and may well explore both of them. With TPLS being an MPI application, we would wish to use many processes (and nodes), not just a single process. How would we best split this problem up? Would we have 1 MPI process per GPU (so 4 per node), and then implement our own solvers either to also work on the GPU, or use OpenMP to make use of the 10 cores per GPU? If so, how would we specify to PETSc which GPU each process would use? Would we instead just have 40 (or perhaps slightly fewer) MPI processes all sharing the GPUs? Surely this would be inefficient, and would PETSc distribute the work across all 4 GPUs, or would every process end out using a single GPU? Would the Krylov solvers be blocking whilst the GPUs are in use running the solvers, or would the host code be able to continue and carry out other calculations whilst waiting for the GPU code to finish? We may need to modify our algorithm to allow for this, but it would make sense to introduce some concurrency so that the CPUs aren?t idling whilst waiting for the GPUs to complete their work. Finally, I?m trying to get the OpenCL PETSc to work on my laptop (Macbook Pro with discrete AMD Radeon R9 M370X GPU). This is mostly because our GPU cluster is out of action until at least late June and I want to get a head start on experimenting with GPUs and TPLS. When I try to run TPLS with the ViennaCL PETSc it reports that my GPU is unable to support double precision. I confirmed that my discrete GPU does support this, however my integrated GPU (Intel Iris) does not. I suspect that ViennaCL is using my integrated GPU instead of my discrete one (it is listed as GPU 0 by OpenCL, with the AMD card is GPU 1). Is there any way of getting PETSc to report which OpenCL device is in use, or to select which device to use? I saw there was some discussion about this on the mailing list archives but I couldn?t find any conclusion. Thanks in advance for your help, Regards, Gordon ----------------------------------------------- Dr Gordon P S Gibb EPCC, The University of Edinburgh Tel: +44 131 651 3459 The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. -------------- next part -------------- An HTML attachment was scrubbed... URL: From fdkong.jd at gmail.com Tue Jun 9 19:15:16 2020 From: fdkong.jd at gmail.com (Fande Kong) Date: Tue, 9 Jun 2020 18:15:16 -0600 Subject: [petsc-users] Make stream In-Reply-To: <874krkdjjg.fsf@jedbrown.org> References: <874krkdjjg.fsf@jedbrown.org> Message-ID: Thanks, Jed, On Tue, Jun 9, 2020 at 3:19 PM Jed Brown wrote: > Fande Kong writes: > > > Hi All, > > > > I am trying to interpret the results from "make stream" on two compute > > nodes, where each node has 48 cores. > > > > If my calculations are memory bandwidth limited, such as AMG, MatVec, > > GMRES, etc.. > > There's a lot more to AMG setup than memory bandwidth (architecture > matters a lot, even between different generation CPUs). Could you elaborate a bit more on this? From my understanding, one big part of AMG SetUp is RAP that should be pretty much bandwidth. So the graph coarsening part is affected by architechniques? MatMult and > Krylov are almost pure bandwidth. > > > The best speedup I could get is 16.6938 if I start from one core?? The > > speedup for function evaluations and Jacobian evaluations can be better > > than16.6938? > > Residual and Jacobians can be faster, especially if your code is slow > (poorly vectorized, branchy, or has a lot of arithmetic). > It will be branchy when we handle complicated mutphyics. > > Are you trying to understand perf on current hardware or make decisions > about new hardware? > The nodes are INL supercomputer nodes. I am trying to understand what could be the best speedup I could get when running moose/petsc on that machine for the linear algebra part. Thanks, Fande, -------------- next part -------------- An HTML attachment was scrubbed... URL: From junchao.zhang at gmail.com Tue Jun 9 19:51:06 2020 From: junchao.zhang at gmail.com (Junchao Zhang) Date: Tue, 9 Jun 2020 19:51:06 -0500 Subject: [petsc-users] A series of GPU questions In-Reply-To: <4AD81243-3693-455E-B6D7-708BCDBFFBE5@epcc.ed.ac.uk> References: <4AD81243-3693-455E-B6D7-708BCDBFFBE5@epcc.ed.ac.uk> Message-ID: On Tue, Jun 9, 2020 at 7:11 PM GIBB Gordon wrote: > Hi, > > First of all, my apologies if this is not the appropriate list to send > these questions to. > > I?m one of the developers of TPLS (https://sourceforge.net/projects/tpls/), > a Fortran code that uses PETSc, parallelised using DM vectors. It uses a > mix of our own solvers, and PETSc?s Krylov solvers. At present it has been > run on up to 25,000 MPI processes, although larger problem sizes should be > able to scale beyond that. > > With the awareness that more and more HPC machines now have one or more > GPUs per node, and that upcoming machines that approach/achieve Exascale > will be heterogeneous in nature, we are investigating whether it is worth > using GPUs with TPLS, and if so, how to best do this. > > I see that in principle all we?d need to do to is set some flags as > described at https://www.mcs.anl.gov/petsc/features/gpus.html to offload > work onto the GPU, however I have some questions about doing this in > practice: > > The GPU machine I have access to has nodes with two 20 core CPUs and 4 > NVIDIA GPUs (so 10 cores per GPU). We could use CUDA or OpenCL, and may > well explore both of them. With TPLS being an MPI application, we would > wish to use many processes (and nodes), not just a single process. How > would we best split this problem up? > > Would we have 1 MPI process per GPU (so 4 per node), and then implement > our own solvers either to also work on the GPU, or use OpenMP to make use > of the 10 cores per GPU? If so, how would we specify to PETSc which GPU > each process would use? > > Would we instead just have 40 (or perhaps slightly fewer) MPI processes > all sharing the GPUs? Surely this would be inefficient, and would PETSc > distribute the work across all 4 GPUs, or would every process end out using > a single GPU? > See https://docs.olcf.ornl.gov/systems/summit_user_guide.html#volta-multi-process-service. In some cases, we did see better performance with multiple mpi ranks/GPU than 1 rank/GPU. The optimal configuration depends on the code. Think two extremes: One code with work done all on GPU and the other all on CPU. Probably you only need 1 mpi rank/node for the former, but full ranks for the latter. > > Would the Krylov solvers be blocking whilst the GPUs are in use running > the solvers, or would the host code be able to continue and carry out other > calculations whilst waiting for the GPU code to finish? We may need to > modify our algorithm to allow for this, but it would make sense to > introduce some concurrency so that the CPUs aren?t idling whilst waiting > for the GPUs to complete their work. > We use asynchronous kernel launch and split-phase communication (VecScatterBegin/End). As long as there is no dependency, you can overlap computations on CPU and GPU, or computations with communications. > > Finally, I?m trying to get the OpenCL PETSc to work on my laptop (Macbook > Pro with discrete AMD Radeon R9 M370X GPU). This is mostly because our GPU > cluster is out of action until at least late June and I want to get a head > start on experimenting with GPUs and TPLS. When I try to run TPLS with the > ViennaCL PETSc it reports that my GPU is unable to support double > precision. I confirmed that my discrete GPU does support this, however my > integrated GPU (Intel Iris) does not. I suspect that ViennaCL is using my > integrated GPU instead of my discrete one (it is listed as GPU 0 by OpenCL, > with the AMD card is GPU 1). Is there any way of getting PETSc to report > which OpenCL device is in use, or to select which device to use? I saw > there was some discussion about this on the mailing list archives but I > couldn?t find any conclusion. > No experience. Karl Rupp (cc'ed) might know. > > Thanks in advance for your help, > > Regards, > > Gordon > > ----------------------------------------------- > Dr Gordon P S Gibb > EPCC, The University of Edinburgh > Tel: +44 131 651 3459 > > The University of Edinburgh is a charitable body, registered in Scotland, > with registration number SC005336. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Jun 9 20:46:09 2020 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 9 Jun 2020 21:46:09 -0400 Subject: [petsc-users] A series of GPU questions In-Reply-To: References: <4AD81243-3693-455E-B6D7-708BCDBFFBE5@epcc.ed.ac.uk> Message-ID: On Tue, Jun 9, 2020 at 8:52 PM Junchao Zhang wrote: > > > On Tue, Jun 9, 2020 at 7:11 PM GIBB Gordon wrote: > >> Hi, >> >> First of all, my apologies if this is not the appropriate list to send >> these questions to. >> >> I?m one of the developers of TPLS (https://sourceforge.net/projects/tpls/), >> a Fortran code that uses PETSc, parallelised using DM vectors. It uses a >> mix of our own solvers, and PETSc?s Krylov solvers. At present it has been >> run on up to 25,000 MPI processes, although larger problem sizes should be >> able to scale beyond that. >> >> With the awareness that more and more HPC machines now have one or more >> GPUs per node, and that upcoming machines that approach/achieve Exascale >> will be heterogeneous in nature, we are investigating whether it is worth >> using GPUs with TPLS, and if so, how to best do this. >> >> I see that in principle all we?d need to do to is set some flags as >> described at https://www.mcs.anl.gov/petsc/features/gpus.html to offload >> work onto the GPU, however I have some questions about doing this in >> practice: >> >> The GPU machine I have access to has nodes with two 20 core CPUs and 4 >> NVIDIA GPUs (so 10 cores per GPU). We could use CUDA or OpenCL, and may >> well explore both of them. With TPLS being an MPI application, we would >> wish to use many processes (and nodes), not just a single process. How >> would we best split this problem up? >> >> Would we have 1 MPI process per GPU (so 4 per node), and then implement >> our own solvers either to also work on the GPU, or use OpenMP to make use >> of the 10 cores per GPU? If so, how would we specify to PETSc which GPU >> each process would use? >> > One comment here. Every experiment says that flat MPI is as efficient or more than OpenMP in this situation, so there seems to be no reason to change that. Thanks, Matt > Would we instead just have 40 (or perhaps slightly fewer) MPI processes >> all sharing the GPUs? Surely this would be inefficient, and would PETSc >> distribute the work across all 4 GPUs, or would every process end out using >> a single GPU? >> > See > https://docs.olcf.ornl.gov/systems/summit_user_guide.html#volta-multi-process-service. > In some cases, we did see better performance with multiple mpi ranks/GPU > than 1 rank/GPU. The optimal configuration depends on the code. Think two > extremes: One code with work done all on GPU and the other all on CPU. > Probably you only need 1 mpi rank/node for the former, but full ranks for > the latter. > > >> >> Would the Krylov solvers be blocking whilst the GPUs are in use running >> the solvers, or would the host code be able to continue and carry out other >> calculations whilst waiting for the GPU code to finish? We may need to >> modify our algorithm to allow for this, but it would make sense to >> introduce some concurrency so that the CPUs aren?t idling whilst waiting >> for the GPUs to complete their work. >> > We use asynchronous kernel launch and split-phase communication > (VecScatterBegin/End). As long as there is no dependency, you can overlap > computations on CPU and GPU, or computations with communications. > > >> >> Finally, I?m trying to get the OpenCL PETSc to work on my laptop (Macbook >> Pro with discrete AMD Radeon R9 M370X GPU). This is mostly because our GPU >> cluster is out of action until at least late June and I want to get a head >> start on experimenting with GPUs and TPLS. When I try to run TPLS with the >> ViennaCL PETSc it reports that my GPU is unable to support double >> precision. I confirmed that my discrete GPU does support this, however my >> integrated GPU (Intel Iris) does not. I suspect that ViennaCL is using my >> integrated GPU instead of my discrete one (it is listed as GPU 0 by OpenCL, >> with the AMD card is GPU 1). Is there any way of getting PETSc to report >> which OpenCL device is in use, or to select which device to use? I saw >> there was some discussion about this on the mailing list archives but I >> couldn?t find any conclusion. >> > No experience. Karl Rupp (cc'ed) might know. > > >> >> Thanks in advance for your help, >> >> Regards, >> >> Gordon >> >> ----------------------------------------------- >> Dr Gordon P S Gibb >> EPCC, The University of Edinburgh >> Tel: +44 131 651 3459 >> >> The University of Edinburgh is a charitable body, registered in Scotland, >> with registration number SC005336. >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Tue Jun 9 21:07:36 2020 From: bsmith at petsc.dev (Barry Smith) Date: Tue, 9 Jun 2020 21:07:36 -0500 Subject: [petsc-users] A series of GPU questions In-Reply-To: References: <4AD81243-3693-455E-B6D7-708BCDBFFBE5@epcc.ed.ac.uk> Message-ID: <5CBB3CC7-E0CA-49D7-B64F-8CDFAA1F61F1@petsc.dev> You might take a look at https://publications.anl.gov/anlpubs/2020/04/159190.pdf in the introduction there is a short discussion about some of the "gotchas" when using multi-core CPUs connected to multiple GPUs; it is focused on an IBM Power/Nvidia GPU system but the same abstract issues will arise on any similar system. 1) How many cores should share a GPU? Generally 1 but there may be exceptions 2) Special memory utilization that can be copied to/from GPUs faster, it should be turned on? 3) Is it worth doing anything with the "extra" cores that are not accessing a GPU? Probably not but there may be exceptions. 4) How to communicate between nodes with MPI, can one go directly from GPU to GPU and skip the CPU memory? Barry > On Jun 9, 2020, at 7:51 PM, Junchao Zhang wrote: > > > > On Tue, Jun 9, 2020 at 7:11 PM GIBB Gordon > wrote: > Hi, > > First of all, my apologies if this is not the appropriate list to send these questions to. > > I?m one of the developers of TPLS (https://sourceforge.net/projects/tpls/ ), a Fortran code that uses PETSc, parallelised using DM vectors. It uses a mix of our own solvers, and PETSc?s Krylov solvers. At present it has been run on up to 25,000 MPI processes, although larger problem sizes should be able to scale beyond that. > > With the awareness that more and more HPC machines now have one or more GPUs per node, and that upcoming machines that approach/achieve Exascale will be heterogeneous in nature, we are investigating whether it is worth using GPUs with TPLS, and if so, how to best do this. > > I see that in principle all we?d need to do to is set some flags as described at https://www.mcs.anl.gov/petsc/features/gpus.html to offload work onto the GPU, however I have some questions about doing this in practice: > > The GPU machine I have access to has nodes with two 20 core CPUs and 4 NVIDIA GPUs (so 10 cores per GPU). We could use CUDA or OpenCL, and may well explore both of them. With TPLS being an MPI application, we would wish to use many processes (and nodes), not just a single process. How would we best split this problem up? > > Would we have 1 MPI process per GPU (so 4 per node), and then implement our own solvers either to also work on the GPU, or use OpenMP to make use of the 10 cores per GPU? If so, how would we specify to PETSc which GPU each process would use? > > Would we instead just have 40 (or perhaps slightly fewer) MPI processes all sharing the GPUs? Surely this would be inefficient, and would PETSc distribute the work across all 4 GPUs, or would every process end out using a single GPU? > See https://docs.olcf.ornl.gov/systems/summit_user_guide.html#volta-multi-process-service . In some cases, we did see better performance with multiple mpi ranks/GPU than 1 rank/GPU. The optimal configuration depends on the code. Think two extremes: One code with work done all on GPU and the other all on CPU. Probably you only need 1 mpi rank/node for the former, but full ranks for the latter. > > > Would the Krylov solvers be blocking whilst the GPUs are in use running the solvers, or would the host code be able to continue and carry out other calculations whilst waiting for the GPU code to finish? We may need to modify our algorithm to allow for this, but it would make sense to introduce some concurrency so that the CPUs aren?t idling whilst waiting for the GPUs to complete their work. > We use asynchronous kernel launch and split-phase communication (VecScatterBegin/End). As long as there is no dependency, you can overlap computations on CPU and GPU, or computations with communications. > > > Finally, I?m trying to get the OpenCL PETSc to work on my laptop (Macbook Pro with discrete AMD Radeon R9 M370X GPU). This is mostly because our GPU cluster is out of action until at least late June and I want to get a head start on experimenting with GPUs and TPLS. When I try to run TPLS with the ViennaCL PETSc it reports that my GPU is unable to support double precision. I confirmed that my discrete GPU does support this, however my integrated GPU (Intel Iris) does not. I suspect that ViennaCL is using my integrated GPU instead of my discrete one (it is listed as GPU 0 by OpenCL, with the AMD card is GPU 1). Is there any way of getting PETSc to report which OpenCL device is in use, or to select which device to use? I saw there was some discussion about this on the mailing list archives but I couldn?t find any conclusion. > No experience. Karl Rupp (cc'ed) might know. > > > Thanks in advance for your help, > > Regards, > > Gordon > > ----------------------------------------------- > Dr Gordon P S Gibb > EPCC, The University of Edinburgh > Tel: +44 131 651 3459 > > The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. -------------- next part -------------- An HTML attachment was scrubbed... URL: From zakaryah at gmail.com Tue Jun 9 22:55:28 2020 From: zakaryah at gmail.com (zakaryah .) Date: Tue, 9 Jun 2020 23:55:28 -0400 Subject: [petsc-users] TAO STCG initial perturbation Message-ID: I am using TAO to minimize the elastic strain energy for somewhat complicated applied forces. With Newton linesearch (NLS), the STCG KSP, and several preconditioners (none, Jacobi, lmvm), the solver finds a minimum within an acceptable number of iterations (<50). Still, in the interest of performance, I am wondering about what happens after the KSP encounters an indefinite matrix (KSP_CONVERGED_CG_NEG_CURVE). After this happens, the line search performs extremely poorly, with function values at step length 1 reaching absurdly large values. As the line search tries smaller step lengths, the function values fall, then typically reach values representing a decline from the previous iteration, but only with tiny step lengths (~1e-11). The line search takes many iterations to arrive at such an improvement. Here is an example, run with -tao_type nls -tao_nls_ksp_type stcg -tao_nls_pc_type none -tao_nls_ksp_norm_type unpreconditioned: 0 TAO, Function value: 0.160612, Residual: 0.0736074 0 TAO, Function value: 0.121568, Residual: 0.0424117 Residual norms for tao_nls_ solve. 0 KSP Residual norm 4.241174778983e-02 1 KSP Residual norm 5.806891169509e-02 2 KSP Residual norm 6.492953448014e-02 3 KSP Residual norm 5.856559430984e-02 4 KSP Residual norm 5.262548559710e-02 5 KSP Residual norm 4.863633473400e-02 6 KSP Residual norm 4.725156347026e-02 7 KSP Residual norm 4.748458009319e-02 8 KSP Residual norm 4.885339641711e-02 9 KSP Residual norm 5.065071226354e-02 10 KSP Residual norm 5.085544070851e-02 11 KSP Residual norm 5.127093547976e-02 12 KSP Residual norm 5.155225884843e-02 13 KSP Residual norm 5.219895021408e-02 14 KSP Residual norm 6.480520610077e-02 15 KSP Residual norm 1.433515456621e-01 Linear tao_nls_ solve converged due to CONVERGED_CG_NEG_CURVE iterations 16 0 LS Function value: 0.121568, Step length: 0. 1 LS Function value: 5.99839e+59, Step length: 1. stx: 0., fx: 0.121568, dgx: -1.32893e+08 sty: 0., fy: 0.121568, dgy: -1.32893e+08 2 LS Function value: 1.46445e+56, Step length: 0.25 stx: 0., fx: 0.121568, dgx: -1.32893e+08 sty: 1., fy: 5.99839e+59, dgy: 3.59903e+60 3 LS Function value: 3.57532e+52, Step length: 0.0625 stx: 0., fx: 0.121568, dgx: -1.32893e+08 sty: 0.25, fy: 1.46445e+56, dgy: 3.51468e+57 4 LS Function value: 8.7288e+48, Step length: 0.015625 stx: 0., fx: 0.121568, dgx: -1.32893e+08 sty: 0.0625, fy: 3.57532e+52, dgy: 3.4323e+54 5 LS Function value: 2.13105e+45, Step length: 0.00390625 stx: 0., fx: 0.121568, dgx: -1.32893e+08 sty: 0.015625, fy: 8.7288e+48, dgy: 3.35186e+51 6 LS Function value: 5.20277e+41, Step length: 0.000976562 stx: 0., fx: 0.121568, dgx: -1.32893e+08 sty: 0.00390625, fy: 2.13105e+45, dgy: 3.2733e+48 7 LS Function value: 1.27021e+38, Step length: 0.000244141 stx: 0., fx: 0.121568, dgx: -1.32893e+08 sty: 0.000976562, fy: 5.20277e+41, dgy: 3.19658e+45 8 LS Function value: 3.1011e+34, Step length: 6.10352e-05 stx: 0., fx: 0.121568, dgx: -1.32893e+08 sty: 0.000244141, fy: 1.27021e+38, dgy: 3.12166e+42 9 LS Function value: 7.57106e+30, Step length: 1.52588e-05 stx: 0., fx: 0.121568, dgx: -1.32893e+08 sty: 6.10352e-05, fy: 3.1011e+34, dgy: 3.0485e+39 10 LS Function value: 1.84842e+27, Step length: 3.8147e-06 stx: 0., fx: 0.121568, dgx: -1.32893e+08 sty: 1.52588e-05, fy: 7.57106e+30, dgy: 2.97706e+36 11 LS Function value: 4.51295e+23, Step length: 9.53672e-07 stx: 0., fx: 0.121568, dgx: -1.32893e+08 sty: 3.8147e-06, fy: 1.84842e+27, dgy: 2.90731e+33 12 LS Function value: 1.10199e+20, Step length: 2.38417e-07 stx: 0., fx: 0.121568, dgx: -1.32893e+08 sty: 9.53672e-07, fy: 4.51295e+23, dgy: 2.83927e+30 13 LS Function value: 2.69231e+16, Step length: 5.96028e-08 stx: 0., fx: 0.121568, dgx: -1.32893e+08 sty: 2.38417e-07, fy: 1.10199e+20, dgy: 2.77314e+27 14 LS Function value: 6.59192e+12, Step length: 1.48993e-08 stx: 0., fx: 0.121568, dgx: -1.32893e+08 sty: 5.96028e-08, fy: 2.69231e+16, dgy: 2.70974e+24 15 LS Function value: 1.62897e+09, Step length: 3.72338e-09 stx: 0., fx: 0.121568, dgx: -1.32893e+08 sty: 1.48993e-08, fy: 6.59192e+12, dgy: 2.65254e+21 16 LS Function value: 421305., Step length: 9.29291e-10 stx: 0., fx: 0.121568, dgx: -1.32893e+08 sty: 3.72338e-09, fy: 1.62897e+09, dgy: 2.61626e+18 17 LS Function value: 141.939, Step length: 2.30343e-10 stx: 0., fx: 0.121568, dgx: -1.32893e+08 sty: 9.29291e-10, fy: 421305., dgy: 2.67496e+15 18 LS Function value: 0.213621, Step length: 5.45712e-11 stx: 0., fx: 0.121568, dgx: -1.32893e+08 sty: 2.30343e-10, fy: 141.939, dgy: 3.35874e+12 19 LS Function value: 0.120058, Step length: 1.32286e-11 stx: 0., fx: 0.121568, dgx: -1.32893e+08 sty: 5.45712e-11, fy: 0.213621, dgy: 8.60977e+09 1 TAO, Function value: 0.120058, Residual: 0.0484495 When the KSP does not encounter negative curvature, the linesearch performs well: 3 TAO, Function value: 0.118376, Residual: 0.0545446 Residual norms for tao_nls_ solve. 0 KSP Residual norm 5.454463134947e-02 1 KSP Residual norm 2.394277461960e-02 2 KSP Residual norm 6.674182971627e-03 3 KSP Residual norm 1.235116278289e-03 4 KSP Residual norm 1.714792759324e-04 5 KSP Residual norm 3.928769927518e-05 6 KSP Residual norm 8.464174666739e-06 7 KSP Residual norm 1.583763581407e-06 8 KSP Residual norm 3.251685842746e-07 Linear tao_nls_ solve converged due to CONVERGED_RTOL iterations 8 0 LS Function value: 0.118376, Step length: 0. 1 LS Function value: 0.117884, Step length: 1. stx: 0., fx: 0.118376, dgx: -0.000530697 sty: 0., fy: 0.118376, dgy: -0.000530697 I have attached the full log for this particular solve. The points of negative curvature do not surprise me - this is a difficult problem, with many singularities in the Jacobian/Hessian. In fact, I am very happy with how well STCG is performing, globally. My question is what to make of the poor performance of the linesearch after encountering these points. As I understand it, most of the flags for adjusting the solver after encountering an indefinite matrix involve altering the magnitude of the perturbation by which the Hessian is offset for calculating the update direction. I am not clear on how to adjust the routine for determining the initial step size. More generally, should I expect these points of negative curvature to cause difficulties for the solver, and be satisfied with a large number of iterations before finding an improvement at tiny step size? I have a loosely related question about the status of the Levenberg Marquardt solver within Tao? I remember hearing on the list that this was being worked on, but I am not sure from the branch description whether it is working or intended for general use. I would love to hear any updates on that, because I think it could be useful for my problem. Thanks! -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- 0 TAO, Function value: 0.160612, Residual: 0.0736074 0 TAO, Function value: 0.121568, Residual: 0.0424117 Residual norms for tao_nls_ solve. 0 KSP Residual norm 4.241174778983e-02 1 KSP Residual norm 5.806891169509e-02 2 KSP Residual norm 6.492953448014e-02 3 KSP Residual norm 5.856559430984e-02 4 KSP Residual norm 5.262548559710e-02 5 KSP Residual norm 4.863633473400e-02 6 KSP Residual norm 4.725156347026e-02 7 KSP Residual norm 4.748458009319e-02 8 KSP Residual norm 4.885339641711e-02 9 KSP Residual norm 5.065071226354e-02 10 KSP Residual norm 5.085544070851e-02 11 KSP Residual norm 5.127093547976e-02 12 KSP Residual norm 5.155225884843e-02 13 KSP Residual norm 5.219895021408e-02 14 KSP Residual norm 6.480520610077e-02 15 KSP Residual norm 1.433515456621e-01 Linear tao_nls_ solve converged due to CONVERGED_CG_NEG_CURVE iterations 16 0 LS Function value: 0.121568, Step length: 0. 1 LS Function value: 5.99839e+59, Step length: 1. stx: 0., fx: 0.121568, dgx: -1.32893e+08 sty: 0., fy: 0.121568, dgy: -1.32893e+08 2 LS Function value: 1.46445e+56, Step length: 0.25 stx: 0., fx: 0.121568, dgx: -1.32893e+08 sty: 1., fy: 5.99839e+59, dgy: 3.59903e+60 3 LS Function value: 3.57532e+52, Step length: 0.0625 stx: 0., fx: 0.121568, dgx: -1.32893e+08 sty: 0.25, fy: 1.46445e+56, dgy: 3.51468e+57 4 LS Function value: 8.7288e+48, Step length: 0.015625 stx: 0., fx: 0.121568, dgx: -1.32893e+08 sty: 0.0625, fy: 3.57532e+52, dgy: 3.4323e+54 5 LS Function value: 2.13105e+45, Step length: 0.00390625 stx: 0., fx: 0.121568, dgx: -1.32893e+08 sty: 0.015625, fy: 8.7288e+48, dgy: 3.35186e+51 6 LS Function value: 5.20277e+41, Step length: 0.000976562 stx: 0., fx: 0.121568, dgx: -1.32893e+08 sty: 0.00390625, fy: 2.13105e+45, dgy: 3.2733e+48 7 LS Function value: 1.27021e+38, Step length: 0.000244141 stx: 0., fx: 0.121568, dgx: -1.32893e+08 sty: 0.000976562, fy: 5.20277e+41, dgy: 3.19658e+45 8 LS Function value: 3.1011e+34, Step length: 6.10352e-05 stx: 0., fx: 0.121568, dgx: -1.32893e+08 sty: 0.000244141, fy: 1.27021e+38, dgy: 3.12166e+42 9 LS Function value: 7.57106e+30, Step length: 1.52588e-05 stx: 0., fx: 0.121568, dgx: -1.32893e+08 sty: 6.10352e-05, fy: 3.1011e+34, dgy: 3.0485e+39 10 LS Function value: 1.84842e+27, Step length: 3.8147e-06 stx: 0., fx: 0.121568, dgx: -1.32893e+08 sty: 1.52588e-05, fy: 7.57106e+30, dgy: 2.97706e+36 11 LS Function value: 4.51295e+23, Step length: 9.53672e-07 stx: 0., fx: 0.121568, dgx: -1.32893e+08 sty: 3.8147e-06, fy: 1.84842e+27, dgy: 2.90731e+33 12 LS Function value: 1.10199e+20, Step length: 2.38417e-07 stx: 0., fx: 0.121568, dgx: -1.32893e+08 sty: 9.53672e-07, fy: 4.51295e+23, dgy: 2.83927e+30 13 LS Function value: 2.69231e+16, Step length: 5.96028e-08 stx: 0., fx: 0.121568, dgx: -1.32893e+08 sty: 2.38417e-07, fy: 1.10199e+20, dgy: 2.77314e+27 14 LS Function value: 6.59192e+12, Step length: 1.48993e-08 stx: 0., fx: 0.121568, dgx: -1.32893e+08 sty: 5.96028e-08, fy: 2.69231e+16, dgy: 2.70974e+24 15 LS Function value: 1.62897e+09, Step length: 3.72338e-09 stx: 0., fx: 0.121568, dgx: -1.32893e+08 sty: 1.48993e-08, fy: 6.59192e+12, dgy: 2.65254e+21 16 LS Function value: 421305., Step length: 9.29291e-10 stx: 0., fx: 0.121568, dgx: -1.32893e+08 sty: 3.72338e-09, fy: 1.62897e+09, dgy: 2.61626e+18 17 LS Function value: 141.939, Step length: 2.30343e-10 stx: 0., fx: 0.121568, dgx: -1.32893e+08 sty: 9.29291e-10, fy: 421305., dgy: 2.67496e+15 18 LS Function value: 0.213621, Step length: 5.45712e-11 stx: 0., fx: 0.121568, dgx: -1.32893e+08 sty: 2.30343e-10, fy: 141.939, dgy: 3.35874e+12 19 LS Function value: 0.120058, Step length: 1.32286e-11 stx: 0., fx: 0.121568, dgx: -1.32893e+08 sty: 5.45712e-11, fy: 0.213621, dgy: 8.60977e+09 1 TAO, Function value: 0.120058, Residual: 0.0484495 Residual norms for tao_nls_ solve. 0 KSP Residual norm 4.844951838490e-02 1 KSP Residual norm 8.888378784357e-02 2 KSP Residual norm 1.314301325299e-01 3 KSP Residual norm 1.212791469268e-01 4 KSP Residual norm 8.542336549879e-02 5 KSP Residual norm 7.588563434299e-02 6 KSP Residual norm 8.363961352117e-02 7 KSP Residual norm 6.598030232548e-02 8 KSP Residual norm 5.204764990103e-02 9 KSP Residual norm 4.034278249686e-02 10 KSP Residual norm 3.749451801761e-02 11 KSP Residual norm 3.940541072499e-02 12 KSP Residual norm 4.626348059301e-02 13 KSP Residual norm 4.525805505460e-02 14 KSP Residual norm 4.683603033796e-02 15 KSP Residual norm 5.646783716895e-02 16 KSP Residual norm 7.654049549502e-02 17 KSP Residual norm 1.220539070594e-01 18 KSP Residual norm 3.872622341402e-01 Linear tao_nls_ solve converged due to CONVERGED_CG_NEG_CURVE iterations 19 0 LS Function value: 0.120058, Step length: 0. 1 LS Function value: 9.71204e+59, Step length: 1. stx: 0., fx: 0.120058, dgx: -1.32802e+08 sty: 0., fy: 0.120058, dgy: -1.32802e+08 2 LS Function value: 1.51751e+58, Step length: 0.5 stx: 0., fx: 0.120058, dgx: -1.32802e+08 sty: 1., fy: 9.71204e+59, dgy: 5.82723e+60 3 LS Function value: 2.3711e+56, Step length: 0.25 stx: 0., fx: 0.120058, dgx: -1.32802e+08 sty: 0.5, fy: 1.51751e+58, dgy: 1.82101e+59 4 LS Function value: 5.78883e+52, Step length: 0.0625 stx: 0., fx: 0.120058, dgx: -1.32802e+08 sty: 0.25, fy: 2.3711e+56, dgy: 5.69065e+57 5 LS Function value: 0.120058, Step length: 0. stx: 0., fx: 0.120058, dgx: -1.32802e+08 sty: 0.0625, fy: 4.49423e+307, dgy: 4.49423e+307 0 LS Function value: 0.120058, Step length: 0. 1 LS Function value: 0.118425, Step length: 1. stx: 0., fx: 0.120058, dgx: -0.00234736 sty: 0., fy: 0.120058, dgy: -0.00234736 2 TAO, Function value: 0.118425, Residual: 0.0616228 Residual norms for tao_nls_ solve. 0 KSP Residual norm 6.162283660782e-02 1 KSP Residual norm 5.025826587428e-02 2 KSP Residual norm 4.542026614329e-02 3 KSP Residual norm 2.061536850833e-02 4 KSP Residual norm 7.994088483587e-03 5 KSP Residual norm 5.455810898084e-03 6 KSP Residual norm 5.537191660058e-03 7 KSP Residual norm 4.569969520333e-03 8 KSP Residual norm 5.701507221403e-03 9 KSP Residual norm 1.108170533587e-02 10 KSP Residual norm 3.103734912773e-02 Linear tao_nls_ solve converged due to CONVERGED_CG_NEG_CURVE iterations 11 0 LS Function value: 0.118425, Step length: 0. 1 LS Function value: 3.49435e+64, Step length: 1. stx: 0., fx: 0.118425, dgx: -2.40297e+07 sty: 0., fy: 0.118425, dgy: -2.40297e+07 2 LS Function value: 8.53113e+60, Step length: 0.25 stx: 0., fx: 0.118425, dgx: -2.40297e+07 sty: 1., fy: 3.49435e+64, dgy: 2.09661e+65 3 LS Function value: 2.0828e+57, Step length: 0.0625 stx: 0., fx: 0.118425, dgx: -2.40297e+07 sty: 0.25, fy: 8.53113e+60, dgy: 2.04747e+62 4 LS Function value: 5.08495e+53, Step length: 0.015625 stx: 0., fx: 0.118425, dgx: -2.40297e+07 sty: 0.0625, fy: 2.0828e+57, dgy: 1.99948e+59 5 LS Function value: 1.24144e+50, Step length: 0.00390625 stx: 0., fx: 0.118425, dgx: -2.40297e+07 sty: 0.015625, fy: 5.08495e+53, dgy: 1.95262e+56 6 LS Function value: 3.03087e+46, Step length: 0.000976562 stx: 0., fx: 0.118425, dgx: -2.40297e+07 sty: 0.00390625, fy: 1.24144e+50, dgy: 1.90686e+53 7 LS Function value: 7.39958e+42, Step length: 0.000244141 stx: 0., fx: 0.118425, dgx: -2.40297e+07 sty: 0.000976562, fy: 3.03087e+46, dgy: 1.86217e+50 8 LS Function value: 1.80654e+39, Step length: 6.10352e-05 stx: 0., fx: 0.118425, dgx: -2.40297e+07 sty: 0.000244141, fy: 7.39958e+42, dgy: 1.81852e+47 9 LS Function value: 4.4105e+35, Step length: 1.52588e-05 stx: 0., fx: 0.118425, dgx: -2.40297e+07 sty: 6.10352e-05, fy: 1.80654e+39, dgy: 1.7759e+44 10 LS Function value: 1.07679e+32, Step length: 3.8147e-06 stx: 0., fx: 0.118425, dgx: -2.40297e+07 sty: 1.52588e-05, fy: 4.4105e+35, dgy: 1.73428e+41 11 LS Function value: 2.6289e+28, Step length: 9.53674e-07 stx: 0., fx: 0.118425, dgx: -2.40297e+07 sty: 3.8147e-06, fy: 1.07679e+32, dgy: 1.69364e+38 12 LS Function value: 6.41853e+24, Step length: 2.38418e-07 stx: 0., fx: 0.118425, dgx: -2.40297e+07 sty: 9.53674e-07, fy: 2.6289e+28, dgy: 1.65396e+35 13 LS Function value: 1.56733e+21, Step length: 5.96041e-08 stx: 0., fx: 0.118425, dgx: -2.40297e+07 sty: 2.38418e-07, fy: 6.41853e+24, dgy: 1.61526e+32 14 LS Function value: 3.82952e+17, Step length: 1.49006e-08 stx: 0., fx: 0.118425, dgx: -2.40297e+07 sty: 5.96041e-08, fy: 1.56733e+21, dgy: 1.57766e+29 15 LS Function value: 9.37896e+13, Step length: 3.72477e-09 stx: 0., fx: 0.118425, dgx: -2.40297e+07 sty: 1.49006e-08, fy: 3.82952e+17, dgy: 1.5417e+26 16 LS Function value: 2.31877e+10, Step length: 9.308e-10 stx: 0., fx: 0.118425, dgx: -2.40297e+07 sty: 3.72477e-09, fy: 9.37896e+13, dgy: 1.50953e+23 17 LS Function value: 5.94925e+06, Step length: 2.32309e-10 stx: 0., fx: 0.118425, dgx: -2.40297e+07 sty: 9.308e-10, fy: 2.31877e+10, dgy: 1.48969e+20 18 LS Function value: 1752.87, Step length: 5.76882e-11 stx: 0., fx: 0.118425, dgx: -2.40297e+07 sty: 2.32309e-10, fy: 5.94925e+06, dgy: 1.51638e+17 19 LS Function value: 0.886813, Step length: 1.4046e-11 stx: 0., fx: 0.118425, dgx: -2.40297e+07 sty: 5.76882e-11, fy: 1752.87, dgy: 1.73482e+14 20 LS Function value: 0.11866, Step length: 3.22282e-12 stx: 0., fx: 0.118425, dgx: -2.40297e+07 sty: 1.4046e-11, fy: 0.886813, dgy: 2.8477e+11 21 LS Function value: 0.118385, Step length: 1.05823e-12 stx: 0., fx: 0.118425, dgx: -2.40297e+07 sty: 3.22282e-12, fy: 0.11866, dgy: 5.41311e+08 22 LS Function value: 0.118387, Step length: 1.84626e-12 stx: 1.05823e-12, fx: 0.118385, dgx: -3.64299e+07 sty: 3.22282e-12, fy: 0.11866, dgy: 5.41311e+08 23 LS Function value: 0.118376, Step length: 1.44508e-12 stx: 1.05823e-12, fx: 0.118385, dgx: -3.64299e+07 sty: 1.84626e-12, fy: 0.118387, dgy: 7.02578e+07 3 TAO, Function value: 0.118376, Residual: 0.0545446 Residual norms for tao_nls_ solve. 0 KSP Residual norm 5.454463134947e-02 1 KSP Residual norm 2.394277461960e-02 2 KSP Residual norm 6.674182971627e-03 3 KSP Residual norm 1.235116278289e-03 4 KSP Residual norm 1.714792759324e-04 5 KSP Residual norm 3.928769927518e-05 6 KSP Residual norm 8.464174666739e-06 7 KSP Residual norm 1.583763581407e-06 8 KSP Residual norm 3.251685842746e-07 Linear tao_nls_ solve converged due to CONVERGED_RTOL iterations 8 0 LS Function value: 0.118376, Step length: 0. 1 LS Function value: 0.117884, Step length: 1. stx: 0., fx: 0.118376, dgx: -0.000530697 sty: 0., fy: 0.118376, dgy: -0.000530697 4 TAO, Function value: 0.117884, Residual: 0.0440192 Residual norms for tao_nls_ solve. 0 KSP Residual norm 4.401922906725e-02 1 KSP Residual norm 9.641051329057e-02 2 KSP Residual norm 1.583253894073e-01 3 KSP Residual norm 1.257330439674e-01 4 KSP Residual norm 7.575909045394e-02 5 KSP Residual norm 7.304083212763e-02 6 KSP Residual norm 6.995426549536e-02 7 KSP Residual norm 5.885266694346e-02 8 KSP Residual norm 4.974341074311e-02 9 KSP Residual norm 3.876927941411e-02 10 KSP Residual norm 3.495606035516e-02 11 KSP Residual norm 4.044737134240e-02 12 KSP Residual norm 4.581515301677e-02 13 KSP Residual norm 5.167427719831e-02 14 KSP Residual norm 6.363360913577e-02 15 KSP Residual norm 1.041939183912e-01 16 KSP Residual norm 5.504936234465e-01 Linear tao_nls_ solve converged due to CONVERGED_CG_NEG_CURVE iterations 17 0 LS Function value: 0.117884, Step length: 0. 1 LS Function value: 2.81045e+61, Step length: 1. stx: 0., fx: 0.117884, dgx: -1.377e+08 sty: 0., fy: 0.117884, dgy: -1.377e+08 2 LS Function value: 6.86146e+57, Step length: 0.25 stx: 0., fx: 0.117884, dgx: -1.377e+08 sty: 1., fy: 2.81045e+61, dgy: 1.68627e+62 3 LS Function value: 1.67516e+54, Step length: 0.0625 stx: 0., fx: 0.117884, dgx: -1.377e+08 sty: 0.25, fy: 6.86146e+57, dgy: 1.64675e+59 4 LS Function value: 4.08975e+50, Step length: 0.015625 stx: 0., fx: 0.117884, dgx: -1.377e+08 sty: 0.0625, fy: 1.67516e+54, dgy: 1.60815e+56 5 LS Function value: 9.98474e+46, Step length: 0.00390625 stx: 0., fx: 0.117884, dgx: -1.377e+08 sty: 0.015625, fy: 4.08975e+50, dgy: 1.57046e+53 6 LS Function value: 2.43768e+43, Step length: 0.000976562 stx: 0., fx: 0.117884, dgx: -1.377e+08 sty: 0.00390625, fy: 9.98474e+46, dgy: 1.53366e+50 7 LS Function value: 5.95137e+39, Step length: 0.000244141 stx: 0., fx: 0.117884, dgx: -1.377e+08 sty: 0.000976562, fy: 2.43768e+43, dgy: 1.49771e+47 8 LS Function value: 1.45297e+36, Step length: 6.10352e-05 stx: 0., fx: 0.117884, dgx: -1.377e+08 sty: 0.000244141, fy: 5.95137e+39, dgy: 1.46261e+44 9 LS Function value: 3.5473e+32, Step length: 1.52588e-05 stx: 0., fx: 0.117884, dgx: -1.377e+08 sty: 6.10352e-05, fy: 1.45297e+36, dgy: 1.42833e+41 10 LS Function value: 8.66048e+28, Step length: 3.8147e-06 stx: 0., fx: 0.117884, dgx: -1.377e+08 sty: 1.52588e-05, fy: 3.5473e+32, dgy: 1.39485e+38 11 LS Function value: 2.11445e+25, Step length: 9.53673e-07 stx: 0., fx: 0.117884, dgx: -1.377e+08 sty: 3.8147e-06, fy: 8.66048e+28, dgy: 1.36217e+35 12 LS Function value: 5.16294e+21, Step length: 2.38417e-07 stx: 0., fx: 0.117884, dgx: -1.377e+08 sty: 9.53673e-07, fy: 2.11445e+25, dgy: 1.33029e+32 13 LS Function value: 1.26118e+18, Step length: 5.96032e-08 stx: 0., fx: 0.117884, dgx: -1.377e+08 sty: 2.38417e-07, fy: 5.16294e+21, dgy: 1.29926e+29 14 LS Function value: 3.08587e+14, Step length: 1.48997e-08 stx: 0., fx: 0.117884, dgx: -1.377e+08 sty: 5.96032e-08, fy: 1.26118e+18, dgy: 1.26939e+26 15 LS Function value: 7.60094e+10, Step length: 3.72382e-09 stx: 0., fx: 0.117884, dgx: -1.377e+08 sty: 1.48997e-08, fy: 3.08587e+14, dgy: 1.24193e+23 16 LS Function value: 1.923e+07, Step length: 9.29851e-10 stx: 0., fx: 0.117884, dgx: -1.377e+08 sty: 3.72382e-09, fy: 7.60094e+10, dgy: 1.2218e+20 17 LS Function value: 5418.89, Step length: 2.31341e-10 stx: 0., fx: 0.117884, dgx: -1.377e+08 sty: 9.29851e-10, fy: 1.923e+07, dgy: 1.22903e+17 18 LS Function value: 2.32235, Step length: 5.66692e-11 stx: 0., fx: 0.117884, dgx: -1.377e+08 sty: 2.31341e-10, fy: 5418.89, dgy: 1.35195e+14 19 LS Function value: 0.117578, Step length: 1.31782e-11 stx: 0., fx: 0.117884, dgx: -1.377e+08 sty: 5.66692e-11, fy: 2.32235, dgy: 2.05445e+11 20 LS Function value: 0.117439, Step length: 3.24647e-12 stx: 1.31782e-11, fx: 0.117578, dgx: 4.21259e+08 sty: 0., fy: 0.117884, dgy: -1.377e+08 21 LS Function value: 0.11689, Step length: 9.0376e-12 stx: 3.24647e-12, fx: 0.117439, dgx: -1.32157e+08 sty: 1.31782e-11, fy: 0.117578, dgy: 4.21259e+08 5 TAO, Function value: 0.11689, Residual: 0.0540029 Residual norms for tao_nls_ solve. 0 KSP Residual norm 5.400285447855e-02 1 KSP Residual norm 6.512918283257e-02 2 KSP Residual norm 1.148360046903e-01 3 KSP Residual norm 1.010928121441e-01 4 KSP Residual norm 6.485971836735e-02 5 KSP Residual norm 4.615521971362e-02 6 KSP Residual norm 2.902709674424e-02 7 KSP Residual norm 2.372772308974e-02 8 KSP Residual norm 1.904925641486e-02 9 KSP Residual norm 1.414125788519e-02 10 KSP Residual norm 1.320539010507e-02 11 KSP Residual norm 9.745386363006e-03 12 KSP Residual norm 7.401766942545e-03 13 KSP Residual norm 6.631061977841e-03 14 KSP Residual norm 5.966693954459e-03 15 KSP Residual norm 4.525072276611e-03 16 KSP Residual norm 3.408340207743e-03 17 KSP Residual norm 2.663265621840e-03 18 KSP Residual norm 2.072491553175e-03 19 KSP Residual norm 1.595991076595e-03 20 KSP Residual norm 1.229477613044e-03 21 KSP Residual norm 9.694958293863e-04 22 KSP Residual norm 8.520014521104e-04 23 KSP Residual norm 1.116349147635e-03 24 KSP Residual norm 6.915586648292e-04 25 KSP Residual norm 5.380917812438e-04 26 KSP Residual norm 4.217826822653e-04 27 KSP Residual norm 3.465721535647e-04 28 KSP Residual norm 2.848873812900e-04 29 KSP Residual norm 2.352040234792e-04 30 KSP Residual norm 2.054572560776e-04 31 KSP Residual norm 2.114102507293e-04 32 KSP Residual norm 2.175918611833e-04 33 KSP Residual norm 1.793335663726e-04 34 KSP Residual norm 1.753175349029e-04 35 KSP Residual norm 1.721057550088e-04 36 KSP Residual norm 1.575663535449e-04 37 KSP Residual norm 1.459973233198e-04 38 KSP Residual norm 1.370447384535e-04 39 KSP Residual norm 1.293915582096e-04 40 KSP Residual norm 1.234029163407e-04 41 KSP Residual norm 1.206463477220e-04 42 KSP Residual norm 1.280405476277e-04 43 KSP Residual norm 1.293435255610e-04 44 KSP Residual norm 1.077166952980e-04 45 KSP Residual norm 8.606815396809e-05 46 KSP Residual norm 6.877935596819e-05 47 KSP Residual norm 5.800254104764e-05 48 KSP Residual norm 5.059424319126e-05 49 KSP Residual norm 4.620743007342e-05 50 KSP Residual norm 4.453906713597e-05 51 KSP Residual norm 4.579022713750e-05 52 KSP Residual norm 4.747062577063e-05 53 KSP Residual norm 4.430176773708e-05 54 KSP Residual norm 4.174496341629e-05 55 KSP Residual norm 4.053423032099e-05 56 KSP Residual norm 4.074769522920e-05 57 KSP Residual norm 4.097540211911e-05 58 KSP Residual norm 4.050813715814e-05 59 KSP Residual norm 4.491113841391e-05 60 KSP Residual norm 5.044406896924e-05 61 KSP Residual norm 4.057321626045e-05 62 KSP Residual norm 3.807937013197e-05 63 KSP Residual norm 4.063220626318e-05 64 KSP Residual norm 4.680126880066e-05 65 KSP Residual norm 3.156728925723e-05 66 KSP Residual norm 2.928284108412e-05 67 KSP Residual norm 2.767191407805e-05 68 KSP Residual norm 2.547955496178e-05 69 KSP Residual norm 2.346003946471e-05 70 KSP Residual norm 2.158563973916e-05 71 KSP Residual norm 1.930828049949e-05 72 KSP Residual norm 1.707079237949e-05 73 KSP Residual norm 1.530105531502e-05 74 KSP Residual norm 1.389897362252e-05 75 KSP Residual norm 1.259208011285e-05 76 KSP Residual norm 1.143495913579e-05 77 KSP Residual norm 1.103221589248e-05 78 KSP Residual norm 1.137422072151e-05 79 KSP Residual norm 9.272576209889e-06 80 KSP Residual norm 8.154830290087e-06 81 KSP Residual norm 7.968131968545e-06 82 KSP Residual norm 1.137940805856e-05 83 KSP Residual norm 7.072458465525e-06 84 KSP Residual norm 6.414641508599e-06 85 KSP Residual norm 6.022871671536e-06 86 KSP Residual norm 5.916531108532e-06 87 KSP Residual norm 7.285288130158e-06 88 KSP Residual norm 5.670946085138e-06 89 KSP Residual norm 5.007495022152e-06 90 KSP Residual norm 4.735375353673e-06 91 KSP Residual norm 4.228998466375e-06 92 KSP Residual norm 3.712713923183e-06 93 KSP Residual norm 3.470802625326e-06 94 KSP Residual norm 3.521635718081e-06 95 KSP Residual norm 2.976592260743e-06 96 KSP Residual norm 2.504143899376e-06 97 KSP Residual norm 2.210826215502e-06 98 KSP Residual norm 1.976863674880e-06 99 KSP Residual norm 1.771698640393e-06 100 KSP Residual norm 1.646734584212e-06 101 KSP Residual norm 2.113572316569e-06 102 KSP Residual norm 1.731406712704e-06 103 KSP Residual norm 1.415110112789e-06 104 KSP Residual norm 1.298382060413e-06 105 KSP Residual norm 1.155812947537e-06 106 KSP Residual norm 1.029787970406e-06 107 KSP Residual norm 9.024337186209e-07 108 KSP Residual norm 7.847415699911e-07 109 KSP Residual norm 6.907844476017e-07 110 KSP Residual norm 6.297089428124e-07 111 KSP Residual norm 5.851637885097e-07 112 KSP Residual norm 5.131306607803e-07 Linear tao_nls_ solve converged due to CONVERGED_RTOL iterations 112 0 LS Function value: 0.11689, Step length: 0. 1 LS Function value: 0.101184, Step length: 1. stx: 0., fx: 0.11689, dgx: -0.0187944 sty: 0., fy: 0.11689, dgy: -0.0187944 6 TAO, Function value: 0.101184, Residual: 0.0298832 Residual norms for tao_nls_ solve. 0 KSP Residual norm 2.988318825066e-02 1 KSP Residual norm 6.999336979013e-02 2 KSP Residual norm 9.782542410829e-02 3 KSP Residual norm 1.121643670992e-01 4 KSP Residual norm 9.184918893023e-02 5 KSP Residual norm 6.503899777002e-02 6 KSP Residual norm 4.915758222601e-02 7 KSP Residual norm 3.667363763398e-02 8 KSP Residual norm 3.333497370714e-02 9 KSP Residual norm 3.047173591261e-02 10 KSP Residual norm 2.929231917984e-02 11 KSP Residual norm 3.070821465077e-02 12 KSP Residual norm 3.156649323208e-02 13 KSP Residual norm 2.953987347450e-02 14 KSP Residual norm 2.888310053485e-02 15 KSP Residual norm 2.716126528623e-02 16 KSP Residual norm 2.393731488846e-02 17 KSP Residual norm 2.173049264768e-02 18 KSP Residual norm 1.999994344085e-02 19 KSP Residual norm 1.774628300224e-02 20 KSP Residual norm 1.867219240131e-02 21 KSP Residual norm 2.109193608629e-02 22 KSP Residual norm 2.462350370207e-02 23 KSP Residual norm 2.862405077714e-02 24 KSP Residual norm 2.535297734084e-02 25 KSP Residual norm 3.147570076541e-02 26 KSP Residual norm 4.454257840211e-02 27 KSP Residual norm 9.103909837716e-02 28 KSP Residual norm 5.047306183102e+01 Linear tao_nls_ solve converged due to CONVERGED_CG_NEG_CURVE iterations 29 0 LS Function value: 0.101184, Step length: 0. 1 LS Function value: -nan., Step length: 1. stx: 0., fx: 0.101184, dgx: -5.53499e+07 sty: 0., fy: 0.101184, dgy: -5.53499e+07 2 LS Function value: 0.101184, Step length: 0. stx: 0., fx: 0.101184, dgx: -5.53499e+07 sty: 1., fy: 4.49423e+307, dgy: 4.49423e+307 0 LS Function value: 0.101184, Step length: 0. 1 LS Function value: 0.100563, Step length: 1. stx: 0., fx: 0.101184, dgx: -0.000893005 sty: 0., fy: 0.101184, dgy: -0.000893005 7 TAO, Function value: 0.100563, Residual: 0.0498602 Residual norms for tao_nls_ solve. 0 KSP Residual norm 4.986015799684e-02 1 KSP Residual norm 3.086218961281e-02 2 KSP Residual norm 3.447024083903e-02 3 KSP Residual norm 1.220840565488e-02 4 KSP Residual norm 5.252141517883e-03 5 KSP Residual norm 3.255344170709e-03 6 KSP Residual norm 3.371106805338e-03 7 KSP Residual norm 3.626200961960e-03 8 KSP Residual norm 2.820818787242e-03 9 KSP Residual norm 1.855528281325e-03 10 KSP Residual norm 1.410605305433e-03 11 KSP Residual norm 1.252300544357e-03 12 KSP Residual norm 1.092178376566e-03 13 KSP Residual norm 8.281995642347e-04 14 KSP Residual norm 5.395084724746e-04 15 KSP Residual norm 3.276036643104e-04 16 KSP Residual norm 2.208978354504e-04 17 KSP Residual norm 1.642429790532e-04 18 KSP Residual norm 1.441760086504e-04 19 KSP Residual norm 1.367381865261e-04 20 KSP Residual norm 1.073593110454e-04 21 KSP Residual norm 6.851701645549e-05 22 KSP Residual norm 4.657324024620e-05 23 KSP Residual norm 3.509348206388e-05 24 KSP Residual norm 1.678485973215e-05 25 KSP Residual norm 8.287113909059e-06 26 KSP Residual norm 4.213140670789e-06 27 KSP Residual norm 2.119353055704e-06 28 KSP Residual norm 1.074857544340e-06 29 KSP Residual norm 5.791909151134e-07 30 KSP Residual norm 3.106280949278e-07 Linear tao_nls_ solve converged due to CONVERGED_RTOL iterations 30 0 LS Function value: 0.100563, Step length: 0. 1 LS Function value: 0.0997883, Step length: 1. stx: 0., fx: 0.100563, dgx: -0.00153491 sty: 0., fy: 0.100563, dgy: -0.00153491 2 LS Function value: 0.0998972, Step length: 0.495946 stx: 1., fx: 0.0997883, dgx: 0.00156001 sty: 0., fy: 0.100563, dgy: -0.00153491 8 TAO, Function value: 0.0998972, Residual: 0.0294338 Residual norms for tao_nls_ solve. 0 KSP Residual norm 2.943381578075e-02 1 KSP Residual norm 6.256352440814e-02 2 KSP Residual norm 8.858013950551e-02 3 KSP Residual norm 9.354100880380e-02 4 KSP Residual norm 8.739903622566e-02 5 KSP Residual norm 6.154815724480e-02 6 KSP Residual norm 4.335485929078e-02 7 KSP Residual norm 3.208570894750e-02 8 KSP Residual norm 2.865550852665e-02 9 KSP Residual norm 3.218692592997e-02 10 KSP Residual norm 3.154406510415e-02 11 KSP Residual norm 3.218715751652e-02 12 KSP Residual norm 3.250679140589e-02 13 KSP Residual norm 3.095023214293e-02 14 KSP Residual norm 2.889394392237e-02 15 KSP Residual norm 2.681813581066e-02 16 KSP Residual norm 2.451645518319e-02 17 KSP Residual norm 2.254912274637e-02 18 KSP Residual norm 2.107067070883e-02 19 KSP Residual norm 2.759210067581e-02 20 KSP Residual norm 2.562709878509e-02 21 KSP Residual norm 2.276311654216e-02 22 KSP Residual norm 2.624260706220e-02 23 KSP Residual norm 3.138003136915e-02 24 KSP Residual norm 4.097052452638e-02 25 KSP Residual norm 6.752867136647e-02 26 KSP Residual norm 2.774250915877e-01 Linear tao_nls_ solve converged due to CONVERGED_CG_NEG_CURVE iterations 27 0 LS Function value: 0.0998972, Step length: 0. 1 LS Function value: -nan., Step length: 1. stx: 0., fx: 0.0998972, dgx: -6.17495e+07 sty: 0., fy: 0.0998972, dgy: -6.17495e+07 2 LS Function value: 0.0998972, Step length: 0. stx: 0., fx: 0.0998972, dgx: -6.17495e+07 sty: 1., fy: 4.49423e+307, dgy: 4.49423e+307 0 LS Function value: 0.0998972, Step length: 0. 1 LS Function value: 0.0994601, Step length: 1. stx: 0., fx: 0.0998972, dgx: -0.00086635 sty: 0., fy: 0.0998972, dgy: -0.00086635 9 TAO, Function value: 0.0994601, Residual: 0.0627203 Residual norms for tao_nls_ solve. 0 KSP Residual norm 6.272030900916e-02 1 KSP Residual norm 3.112371813746e-02 2 KSP Residual norm 3.486112477484e-02 3 KSP Residual norm 1.478285537888e-02 4 KSP Residual norm 1.432448204034e-02 5 KSP Residual norm 4.472439203545e-02 Linear tao_nls_ solve converged due to CONVERGED_CG_NEG_CURVE iterations 6 0 LS Function value: 0.0994601, Step length: 0. 1 LS Function value: 1.96952e+64, Step length: 1. stx: 0., fx: 0.0994601, dgx: -9.10643e+07 sty: 0., fy: 0.0994601, dgy: -9.10643e+07 2 LS Function value: 4.80839e+60, Step length: 0.25 stx: 0., fx: 0.0994601, dgx: -9.10643e+07 sty: 1., fy: 1.96952e+64, dgy: 1.18171e+65 3 LS Function value: 1.17392e+57, Step length: 0.0625 stx: 0., fx: 0.0994601, dgx: -9.10643e+07 sty: 0.25, fy: 4.80839e+60, dgy: 1.15401e+62 4 LS Function value: 2.86602e+53, Step length: 0.015625 stx: 0., fx: 0.0994601, dgx: -9.10643e+07 sty: 0.0625, fy: 1.17392e+57, dgy: 1.12697e+59 5 LS Function value: 6.99713e+49, Step length: 0.00390625 stx: 0., fx: 0.0994601, dgx: -9.10643e+07 sty: 0.015625, fy: 2.86602e+53, dgy: 1.10055e+56 6 LS Function value: 1.70828e+46, Step length: 0.000976562 stx: 0., fx: 0.0994601, dgx: -9.10643e+07 sty: 0.00390625, fy: 6.99713e+49, dgy: 1.07476e+53 7 LS Function value: 4.17061e+42, Step length: 0.000244141 stx: 0., fx: 0.0994601, dgx: -9.10643e+07 sty: 0.000976562, fy: 1.70828e+46, dgy: 1.04957e+50 8 LS Function value: 1.01822e+39, Step length: 6.10352e-05 stx: 0., fx: 0.0994601, dgx: -9.10643e+07 sty: 0.000244141, fy: 4.17061e+42, dgy: 1.02497e+47 9 LS Function value: 2.48588e+35, Step length: 1.52588e-05 stx: 0., fx: 0.0994601, dgx: -9.10643e+07 sty: 6.10352e-05, fy: 1.01822e+39, dgy: 1.00095e+44 10 LS Function value: 0.0994601, Step length: 0. stx: 0., fx: 0.0994601, dgx: -9.10643e+07 sty: 1.52588e-05, fy: 4.49423e+307, dgy: 4.49423e+307 0 LS Function value: 0.0994601, Step length: 0. 1 LS Function value: 0.101263, Step length: 1. stx: 0., fx: 0.0994601, dgx: -0.00393384 sty: 0., fy: 0.0994601, dgy: -0.00393384 2 LS Function value: 0.0994353, Step length: 0.5 stx: 0., fx: 0.0994601, dgx: -0.00393384 sty: 1., fy: 0.101263, dgy: 0.00527413 10 TAO, Function value: 0.0994353, Residual: 0.0723933 Residual norms for tao_nls_ solve. 0 KSP Residual norm 7.239325125692e-02 1 KSP Residual norm 3.264842236815e-03 2 KSP Residual norm 1.358841152684e-04 3 KSP Residual norm 4.076006658523e-06 4 KSP Residual norm 1.374045688780e-07 Linear tao_nls_ solve converged due to CONVERGED_RTOL iterations 4 0 LS Function value: 0.0994353, Step length: 0. 1 LS Function value: 0.0993447, Step length: 1. stx: 0., fx: 0.0994353, dgx: -9.50218e-05 sty: 0., fy: 0.0994353, dgy: -9.50218e-05 2 LS Function value: 0.0990652, Step length: 5. stx: 1., fx: 0.0993447, dgx: -8.62629e-05 sty: 0., fy: 0.0994353, dgy: -9.50218e-05 11 TAO, Function value: 0.0990652, Residual: 0.0429398 Residual norms for tao_nls_ solve. 0 KSP Residual norm 4.293976414948e-02 1 KSP Residual norm 3.482999076388e-02 2 KSP Residual norm 7.550011514518e-02 3 KSP Residual norm 9.172956814513e-02 4 KSP Residual norm 9.806624446728e-02 5 KSP Residual norm 8.165837299441e-02 6 KSP Residual norm 7.792230767957e-02 7 KSP Residual norm 8.338487762871e-02 8 KSP Residual norm 1.085872886084e-01 9 KSP Residual norm 3.208725716617e-01 Linear tao_nls_ solve converged due to CONVERGED_CG_NEG_CURVE iterations 10 0 LS Function value: 0.0990652, Step length: 0. 1 LS Function value: 5.64843e+61, Step length: 1. stx: 0., fx: 0.0990652, dgx: -2.07706e+08 sty: 0., fy: 0.0990652, dgy: -2.07706e+08 2 LS Function value: 1.37901e+58, Step length: 0.25 stx: 0., fx: 0.0990652, dgx: -2.07706e+08 sty: 1., fy: 5.64843e+61, dgy: 3.38906e+62 3 LS Function value: 3.36673e+54, Step length: 0.0625 stx: 0., fx: 0.0990652, dgx: -2.07706e+08 sty: 0.25, fy: 1.37901e+58, dgy: 3.30963e+59 4 LS Function value: 8.21955e+50, Step length: 0.015625 stx: 0., fx: 0.0990652, dgx: -2.07706e+08 sty: 0.0625, fy: 3.36673e+54, dgy: 3.23206e+56 5 LS Function value: 2.00673e+47, Step length: 0.00390625 stx: 0., fx: 0.0990652, dgx: -2.07706e+08 sty: 0.015625, fy: 8.21955e+50, dgy: 3.15631e+53 6 LS Function value: 4.89924e+43, Step length: 0.000976562 stx: 0., fx: 0.0990652, dgx: -2.07706e+08 sty: 0.00390625, fy: 2.00673e+47, dgy: 3.08233e+50 7 LS Function value: 1.1961e+40, Step length: 0.000244141 stx: 0., fx: 0.0990652, dgx: -2.07706e+08 sty: 0.000976562, fy: 4.89924e+43, dgy: 3.01009e+47 8 LS Function value: 2.92017e+36, Step length: 6.10352e-05 stx: 0., fx: 0.0990652, dgx: -2.07706e+08 sty: 0.000244141, fy: 1.1961e+40, dgy: 2.93954e+44 9 LS Function value: 7.12935e+32, Step length: 1.52588e-05 stx: 0., fx: 0.0990652, dgx: -2.07706e+08 sty: 6.10352e-05, fy: 2.92017e+36, dgy: 2.87065e+41 10 LS Function value: 1.74058e+29, Step length: 3.8147e-06 stx: 0., fx: 0.0990652, dgx: -2.07706e+08 sty: 1.52588e-05, fy: 7.12935e+32, dgy: 2.80337e+38 11 LS Function value: 4.24963e+25, Step length: 9.53673e-07 stx: 0., fx: 0.0990652, dgx: -2.07706e+08 sty: 3.8147e-06, fy: 1.74058e+29, dgy: 2.73769e+35 12 LS Function value: 1.03767e+22, Step length: 2.38417e-07 stx: 0., fx: 0.0990652, dgx: -2.07706e+08 sty: 9.53673e-07, fy: 4.24963e+25, dgy: 2.67361e+32 13 LS Function value: 2.53493e+18, Step length: 5.9603e-08 stx: 0., fx: 0.0990652, dgx: -2.07706e+08 sty: 2.38417e-07, fy: 1.03767e+22, dgy: 2.61128e+29 14 LS Function value: 6.2041e+14, Step length: 1.48995e-08 stx: 0., fx: 0.0990652, dgx: -2.07706e+08 sty: 5.9603e-08, fy: 2.53493e+18, dgy: 2.5514e+26 15 LS Function value: 1.5297e+11, Step length: 3.72365e-09 stx: 0., fx: 0.0990652, dgx: -2.07706e+08 sty: 1.48995e-08, fy: 6.2041e+14, dgy: 2.49673e+23 16 LS Function value: 3.88459e+07, Step length: 9.29684e-10 stx: 0., fx: 0.0990652, dgx: -2.07706e+08 sty: 3.72365e-09, fy: 1.5297e+11, dgy: 2.45835e+20 17 LS Function value: 11067.8, Step length: 2.31185e-10 stx: 0., fx: 0.0990652, dgx: -2.07706e+08 sty: 9.29684e-10, fy: 3.88459e+07, dgy: 2.48079e+17 18 LS Function value: 4.67708, Step length: 5.65509e-11 stx: 0., fx: 0.0990652, dgx: -2.07706e+08 sty: 2.31185e-10, fy: 11067.8, dgy: 2.75615e+14 19 LS Function value: 0.0991776, Step length: 1.31052e-11 stx: 0., fx: 0.0990652, dgx: -2.07706e+08 sty: 5.65509e-11, fy: 4.67708, dgy: 4.26014e+11 20 LS Function value: 0.0978498, Step length: 7.17299e-12 stx: 0., fx: 0.0990652, dgx: -2.07706e+08 sty: 1.31052e-11, fy: 0.0991776, dgy: 8.29029e+08 12 TAO, Function value: 0.0978498, Residual: 0.0383805 Residual norms for tao_nls_ solve. 0 KSP Residual norm 3.838049529329e-02 1 KSP Residual norm 4.104799670032e-02 2 KSP Residual norm 4.938791921895e-02 3 KSP Residual norm 5.469604838478e-02 4 KSP Residual norm 6.006372259852e-02 5 KSP Residual norm 4.944395630246e-02 6 KSP Residual norm 2.655210689877e-02 7 KSP Residual norm 1.815644470817e-02 8 KSP Residual norm 1.302187120108e-02 9 KSP Residual norm 9.107329952745e-03 10 KSP Residual norm 7.042457288764e-03 11 KSP Residual norm 5.444815362551e-03 12 KSP Residual norm 4.503256172252e-03 13 KSP Residual norm 3.840247386152e-03 14 KSP Residual norm 3.012453902742e-03 15 KSP Residual norm 2.556044756652e-03 16 KSP Residual norm 2.108321742002e-03 17 KSP Residual norm 1.601965217851e-03 18 KSP Residual norm 1.152626232746e-03 19 KSP Residual norm 8.965195982165e-04 20 KSP Residual norm 6.819217178252e-04 21 KSP Residual norm 5.054344452136e-04 22 KSP Residual norm 4.296893654391e-04 23 KSP Residual norm 3.678364140098e-04 24 KSP Residual norm 4.887788399733e-04 25 KSP Residual norm 3.064128031492e-04 26 KSP Residual norm 2.550193981708e-04 27 KSP Residual norm 2.552788692690e-04 28 KSP Residual norm 3.035479838299e-04 29 KSP Residual norm 1.927949597304e-04 30 KSP Residual norm 1.740836866957e-04 31 KSP Residual norm 1.690106661339e-04 32 KSP Residual norm 1.567480922634e-04 33 KSP Residual norm 1.370412550096e-04 34 KSP Residual norm 1.235082436513e-04 35 KSP Residual norm 1.094531819117e-04 36 KSP Residual norm 9.229464050676e-05 37 KSP Residual norm 8.711900575953e-05 38 KSP Residual norm 7.913782298169e-05 39 KSP Residual norm 5.759442548114e-05 40 KSP Residual norm 4.550193282518e-05 41 KSP Residual norm 3.594824027024e-05 42 KSP Residual norm 2.981668814485e-05 43 KSP Residual norm 2.471134378587e-05 44 KSP Residual norm 2.394802748084e-05 45 KSP Residual norm 2.653979472207e-05 46 KSP Residual norm 1.724795601158e-05 47 KSP Residual norm 1.474589242356e-05 48 KSP Residual norm 1.345737942134e-05 49 KSP Residual norm 1.207348932897e-05 50 KSP Residual norm 1.009591326964e-05 51 KSP Residual norm 1.220917080617e-05 52 KSP Residual norm 8.362545258160e-06 53 KSP Residual norm 6.649828398468e-06 54 KSP Residual norm 5.876803447075e-06 55 KSP Residual norm 5.087077865435e-06 56 KSP Residual norm 4.165868771843e-06 57 KSP Residual norm 3.462007153184e-06 58 KSP Residual norm 2.957111557610e-06 59 KSP Residual norm 2.494373559056e-06 60 KSP Residual norm 2.096593840383e-06 61 KSP Residual norm 1.829672825798e-06 62 KSP Residual norm 1.643121524068e-06 63 KSP Residual norm 1.513975481200e-06 64 KSP Residual norm 1.459217350465e-06 65 KSP Residual norm 2.010835774866e-06 66 KSP Residual norm 1.215710323297e-06 67 KSP Residual norm 1.048962607587e-06 68 KSP Residual norm 9.997094747731e-07 69 KSP Residual norm 1.155572428586e-06 70 KSP Residual norm 8.485710636954e-07 71 KSP Residual norm 7.061030275445e-07 72 KSP Residual norm 9.327292762046e-07 73 KSP Residual norm 6.334550315318e-07 74 KSP Residual norm 5.501693071445e-07 75 KSP Residual norm 4.545405963395e-07 76 KSP Residual norm 3.516669724468e-07 Linear tao_nls_ solve converged due to CONVERGED_RTOL iterations 76 0 LS Function value: 0.0978498, Step length: 0. 1 LS Function value: 0.0922924, Step length: 1. stx: 0., fx: 0.0978498, dgx: -0.00630624 sty: 0., fy: 0.0978498, dgy: -0.00630624 13 TAO, Function value: 0.0922924, Residual: 0.0210473 Residual norms for tao_nls_ solve. 0 KSP Residual norm 2.104731401826e-02 1 KSP Residual norm 4.762116742667e-02 2 KSP Residual norm 7.805419541658e-02 3 KSP Residual norm 9.341873721756e-02 4 KSP Residual norm 7.753680063642e-02 5 KSP Residual norm 6.607207685675e-02 6 KSP Residual norm 4.509556138225e-02 7 KSP Residual norm 3.426881469515e-02 8 KSP Residual norm 2.950794056811e-02 9 KSP Residual norm 2.629097306494e-02 10 KSP Residual norm 2.484558360009e-02 11 KSP Residual norm 2.582967158109e-02 12 KSP Residual norm 2.764499090287e-02 13 KSP Residual norm 2.802341904277e-02 14 KSP Residual norm 2.852485024089e-02 15 KSP Residual norm 2.794501907160e-02 16 KSP Residual norm 2.470291940190e-02 17 KSP Residual norm 2.292080716974e-02 18 KSP Residual norm 2.032680124976e-02 19 KSP Residual norm 1.857917699486e-02 20 KSP Residual norm 1.755293279061e-02 21 KSP Residual norm 1.677629420647e-02 22 KSP Residual norm 1.792530288920e-02 23 KSP Residual norm 1.899963434507e-02 24 KSP Residual norm 1.951185380153e-02 25 KSP Residual norm 2.156698731021e-02 26 KSP Residual norm 2.499180352098e-02 27 KSP Residual norm 2.915410223437e-02 28 KSP Residual norm 3.532178860640e-02 29 KSP Residual norm 5.045948033097e-02 30 KSP Residual norm 7.568651642299e-02 31 KSP Residual norm 1.205434866391e-01 Linear tao_nls_ solve converged due to CONVERGED_CG_NEG_CURVE iterations 32 0 LS Function value: 0.0922924, Step length: 0. 1 LS Function value: -nan., Step length: 1. stx: 0., fx: 0.0922924, dgx: -4.65993e+07 sty: 0., fy: 0.0922924, dgy: -4.65993e+07 2 LS Function value: 0.0922924, Step length: 0. stx: 0., fx: 0.0922924, dgx: -4.65993e+07 sty: 1., fy: 4.49423e+307, dgy: 4.49423e+307 0 LS Function value: 0.0922924, Step length: 0. 1 LS Function value: 0.0919899, Step length: 1. stx: 0., fx: 0.0922924, dgx: -0.000442989 sty: 0., fy: 0.0922924, dgy: -0.000442989 14 TAO, Function value: 0.0919899, Residual: 0.034581 Residual norms for tao_nls_ solve. 0 KSP Residual norm 3.458101184870e-02 1 KSP Residual norm 2.193480344616e-02 2 KSP Residual norm 2.708342555381e-02 3 KSP Residual norm 1.407264194222e-02 4 KSP Residual norm 5.477097475015e-03 5 KSP Residual norm 2.465345008426e-03 6 KSP Residual norm 1.441189435782e-03 7 KSP Residual norm 8.076358589676e-04 8 KSP Residual norm 5.726730931957e-04 9 KSP Residual norm 3.435424175489e-04 10 KSP Residual norm 2.361476797000e-04 11 KSP Residual norm 1.712113103646e-04 12 KSP Residual norm 1.089823076167e-04 13 KSP Residual norm 7.523641079404e-05 14 KSP Residual norm 5.717598558250e-05 15 KSP Residual norm 3.923841622241e-05 16 KSP Residual norm 2.757399897918e-05 17 KSP Residual norm 1.759168756333e-05 18 KSP Residual norm 1.030948097443e-05 19 KSP Residual norm 5.769736169778e-06 20 KSP Residual norm 3.166108974466e-06 21 KSP Residual norm 1.849066826681e-06 22 KSP Residual norm 1.030282190249e-06 23 KSP Residual norm 5.935145110659e-07 24 KSP Residual norm 3.583391281969e-07 25 KSP Residual norm 2.103120478293e-07 Linear tao_nls_ solve converged due to CONVERGED_RTOL iterations 25 0 LS Function value: 0.0919899, Step length: 0. 1 LS Function value: 0.0910633, Step length: 1. stx: 0., fx: 0.0919899, dgx: -0.00103175 sty: 0., fy: 0.0919899, dgy: -0.00103175 15 TAO, Function value: 0.0910633, Residual: 0.0180023 Residual norms for tao_nls_ solve. 0 KSP Residual norm 1.800228508702e-02 1 KSP Residual norm 9.068321095304e-02 2 KSP Residual norm 9.283808068772e-02 3 KSP Residual norm 6.512346166228e-02 4 KSP Residual norm 4.370296977616e-02 5 KSP Residual norm 3.747074618107e-02 6 KSP Residual norm 3.000225701036e-02 7 KSP Residual norm 2.820743299845e-02 8 KSP Residual norm 2.685179051907e-02 9 KSP Residual norm 2.560451608593e-02 10 KSP Residual norm 2.643649399607e-02 11 KSP Residual norm 2.843854700871e-02 12 KSP Residual norm 2.890744781333e-02 13 KSP Residual norm 2.863583040828e-02 14 KSP Residual norm 2.756901280202e-02 15 KSP Residual norm 2.424478703160e-02 16 KSP Residual norm 2.225795689885e-02 17 KSP Residual norm 2.075177866603e-02 18 KSP Residual norm 1.874495853208e-02 19 KSP Residual norm 1.833563213764e-02 20 KSP Residual norm 1.882702118296e-02 21 KSP Residual norm 2.036901127505e-02 22 KSP Residual norm 2.197692132504e-02 23 KSP Residual norm 2.478672937404e-02 24 KSP Residual norm 2.819390292106e-02 25 KSP Residual norm 3.627960119683e-02 26 KSP Residual norm 5.228600493564e-02 27 KSP Residual norm 8.941018771950e-02 28 KSP Residual norm 2.724033285836e-01 Linear tao_nls_ solve converged due to CONVERGED_CG_NEG_CURVE iterations 29 0 LS Function value: 0.0910633, Step length: 0. 1 LS Function value: -nan., Step length: 1. stx: 0., fx: 0.0910633, dgx: -5.0133e+07 sty: 0., fy: 0.0910633, dgy: -5.0133e+07 2 LS Function value: 0.0910633, Step length: 0. stx: 0., fx: 0.0910633, dgx: -5.0133e+07 sty: 1., fy: 4.49423e+307, dgy: 4.49423e+307 0 LS Function value: 0.0910633, Step length: 0. 1 LS Function value: 0.0907574, Step length: 1. stx: 0., fx: 0.0910633, dgx: -0.000324082 sty: 0., fy: 0.0910633, dgy: -0.000324082 16 TAO, Function value: 0.0907574, Residual: 0.0190785 Residual norms for tao_nls_ solve. 0 KSP Residual norm 1.907846986553e-02 1 KSP Residual norm 2.993864919749e-02 2 KSP Residual norm 1.834030942810e-02 3 KSP Residual norm 3.974075504154e-03 4 KSP Residual norm 1.525670251714e-03 5 KSP Residual norm 9.302168102960e-04 6 KSP Residual norm 7.980673364711e-04 7 KSP Residual norm 4.911592111207e-04 8 KSP Residual norm 2.359070053256e-04 9 KSP Residual norm 1.676705578307e-04 10 KSP Residual norm 9.591291567021e-05 11 KSP Residual norm 5.157180839109e-05 12 KSP Residual norm 3.088532939585e-05 13 KSP Residual norm 1.863863172393e-05 14 KSP Residual norm 1.002927489864e-05 15 KSP Residual norm 5.258525450171e-06 16 KSP Residual norm 3.277185574003e-06 17 KSP Residual norm 1.941023710434e-06 18 KSP Residual norm 1.153030802774e-06 19 KSP Residual norm 7.344864704914e-07 20 KSP Residual norm 4.656051782025e-07 21 KSP Residual norm 2.823114387057e-07 22 KSP Residual norm 1.591569325673e-07 Linear tao_nls_ solve converged due to CONVERGED_RTOL iterations 22 0 LS Function value: 0.0907574, Step length: 0. 1 LS Function value: 0.0899364, Step length: 1. stx: 0., fx: 0.0907574, dgx: -0.000844974 sty: 0., fy: 0.0907574, dgy: -0.000844974 2 LS Function value: 0.0871243, Step length: 5. stx: 1., fx: 0.0899364, dgx: -0.000797183 sty: 0., fy: 0.0907574, dgy: -0.000844974 17 TAO, Function value: 0.0871243, Residual: 0.0288583 Residual norms for tao_nls_ solve. 0 KSP Residual norm 2.885825081697e-02 1 KSP Residual norm 1.787381016414e-02 2 KSP Residual norm 3.823406540207e-02 3 KSP Residual norm 7.251101825671e-02 4 KSP Residual norm 6.968475840244e-02 5 KSP Residual norm 5.545284692751e-02 6 KSP Residual norm 4.011986642084e-02 7 KSP Residual norm 3.359538088814e-02 8 KSP Residual norm 2.865276564604e-02 9 KSP Residual norm 2.292818092292e-02 10 KSP Residual norm 2.300983089426e-02 11 KSP Residual norm 2.826726407450e-02 12 KSP Residual norm 2.987926720878e-02 13 KSP Residual norm 3.530692256318e-02 14 KSP Residual norm 5.118816964047e-02 15 KSP Residual norm 9.997102642267e-02 16 KSP Residual norm 1.159128427850e+00 Linear tao_nls_ solve converged due to CONVERGED_CG_NEG_CURVE iterations 17 0 LS Function value: 0.0871243, Step length: 0. 1 LS Function value: 3.5207e+60, Step length: 1. stx: 0., fx: 0.0871243, dgx: -7.9283e+07 sty: 0., fy: 0.0871243, dgy: -7.9283e+07 2 LS Function value: 8.59546e+56, Step length: 0.25 stx: 0., fx: 0.0871243, dgx: -7.9283e+07 sty: 1., fy: 3.5207e+60, dgy: 2.11242e+61 3 LS Function value: 2.0985e+53, Step length: 0.0625 stx: 0., fx: 0.0871243, dgx: -7.9283e+07 sty: 0.25, fy: 8.59546e+56, dgy: 2.06291e+58 4 LS Function value: 5.12329e+49, Step length: 0.015625 stx: 0., fx: 0.0871243, dgx: -7.9283e+07 sty: 0.0625, fy: 2.0985e+53, dgy: 2.01456e+55 5 LS Function value: 1.2508e+46, Step length: 0.00390625 stx: 0., fx: 0.0871243, dgx: -7.9283e+07 sty: 0.015625, fy: 5.12329e+49, dgy: 1.96734e+52 6 LS Function value: 3.05372e+42, Step length: 0.000976562 stx: 0., fx: 0.0871243, dgx: -7.9283e+07 sty: 0.00390625, fy: 1.2508e+46, dgy: 1.92124e+49 7 LS Function value: 7.45538e+38, Step length: 0.000244141 stx: 0., fx: 0.0871243, dgx: -7.9283e+07 sty: 0.000976562, fy: 3.05372e+42, dgy: 1.87621e+46 8 LS Function value: 1.82016e+35, Step length: 6.10352e-05 stx: 0., fx: 0.0871243, dgx: -7.9283e+07 sty: 0.000244141, fy: 7.45538e+38, dgy: 1.83223e+43 9 LS Function value: 4.44379e+31, Step length: 1.52588e-05 stx: 0., fx: 0.0871243, dgx: -7.9283e+07 sty: 6.10352e-05, fy: 1.82016e+35, dgy: 1.78929e+40 10 LS Function value: 1.08493e+28, Step length: 3.81469e-06 stx: 0., fx: 0.0871243, dgx: -7.9283e+07 sty: 1.52588e-05, fy: 4.44379e+31, dgy: 1.74736e+37 11 LS Function value: 2.64903e+24, Step length: 9.5367e-07 stx: 0., fx: 0.0871243, dgx: -7.9283e+07 sty: 3.81469e-06, fy: 1.08493e+28, dgy: 1.70645e+34 12 LS Function value: 6.46988e+20, Step length: 2.38414e-07 stx: 0., fx: 0.0871243, dgx: -7.9283e+07 sty: 9.5367e-07, fy: 2.64903e+24, dgy: 1.66659e+31 13 LS Function value: 1.58203e+17, Step length: 5.96005e-08 stx: 0., fx: 0.0871243, dgx: -7.9283e+07 sty: 2.38414e-07, fy: 6.46988e+20, dgy: 1.62806e+28 14 LS Function value: 3.88665e+13, Step length: 1.4897e-08 stx: 0., fx: 0.0871243, dgx: -7.9283e+07 sty: 5.96005e-08, fy: 1.58203e+17, dgy: 1.59197e+25 15 LS Function value: 9.72997e+09, Step length: 3.72114e-09 stx: 0., fx: 0.0871243, dgx: -7.9283e+07 sty: 1.4897e-08, fy: 3.88665e+13, dgy: 1.56279e+22 16 LS Function value: 2.62659e+06, Step length: 9.27139e-10 stx: 0., fx: 0.0871243, dgx: -7.9283e+07 sty: 3.72114e-09, fy: 9.72997e+09, dgy: 1.55837e+19 17 LS Function value: 941.405, Step length: 2.2856e-10 stx: 0., fx: 0.0871243, dgx: -7.9283e+07 sty: 9.27139e-10, fy: 2.62659e+06, dgy: 1.6544e+16 18 LS Function value: 0.725101, Step length: 5.38881e-11 stx: 0., fx: 0.0871243, dgx: -7.9283e+07 sty: 2.2856e-10, fy: 941.405, dgy: 2.23077e+13 19 LS Function value: 0.0867106, Step length: 1.14839e-11 stx: 0., fx: 0.0871243, dgx: -7.9283e+07 sty: 5.38881e-11, fy: 0.725101, dgy: 5.60936e+10 20 LS Function value: 0.0867388, Step length: 5.18834e-12 stx: 1.14839e-11, fx: 0.0867106, dgx: 9.62026e+07 sty: 0., fy: 0.0871243, dgy: -7.9283e+07 18 TAO, Function value: 0.0867388, Residual: 0.0244321 Residual norms for tao_nls_ solve. 0 KSP Residual norm 2.443205168743e-02 1 KSP Residual norm 2.098412667069e-02 2 KSP Residual norm 3.550481833459e-02 3 KSP Residual norm 4.449498111753e-02 4 KSP Residual norm 4.744984132717e-02 5 KSP Residual norm 3.919343896916e-02 6 KSP Residual norm 2.955187338457e-02 7 KSP Residual norm 2.241122590159e-02 8 KSP Residual norm 1.622604540401e-02 9 KSP Residual norm 1.247074792844e-02 10 KSP Residual norm 1.075859597631e-02 11 KSP Residual norm 9.123144854965e-03 12 KSP Residual norm 7.985132614603e-03 13 KSP Residual norm 7.380880572793e-03 14 KSP Residual norm 6.994755809795e-03 15 KSP Residual norm 6.719142800910e-03 16 KSP Residual norm 6.834025097084e-03 17 KSP Residual norm 5.963962145241e-03 18 KSP Residual norm 5.238014004998e-03 19 KSP Residual norm 4.389508841164e-03 20 KSP Residual norm 3.601994569453e-03 21 KSP Residual norm 2.943461835063e-03 22 KSP Residual norm 2.505326372737e-03 23 KSP Residual norm 2.109494808777e-03 24 KSP Residual norm 1.873094496220e-03 25 KSP Residual norm 1.581570491324e-03 26 KSP Residual norm 1.468213054471e-03 27 KSP Residual norm 1.388993584103e-03 28 KSP Residual norm 1.199780321863e-03 29 KSP Residual norm 1.094491663719e-03 30 KSP Residual norm 9.507674076599e-04 31 KSP Residual norm 8.363266381063e-04 32 KSP Residual norm 7.234390599953e-04 33 KSP Residual norm 6.173853729052e-04 34 KSP Residual norm 5.289363732769e-04 35 KSP Residual norm 4.626176220146e-04 36 KSP Residual norm 4.151986725873e-04 37 KSP Residual norm 3.767594534228e-04 38 KSP Residual norm 3.694026843925e-04 39 KSP Residual norm 3.446645632772e-04 40 KSP Residual norm 3.024162828939e-04 41 KSP Residual norm 2.552520881171e-04 42 KSP Residual norm 2.236216926729e-04 43 KSP Residual norm 2.102056460919e-04 44 KSP Residual norm 1.886099491633e-04 45 KSP Residual norm 1.638601602017e-04 46 KSP Residual norm 1.457164558366e-04 47 KSP Residual norm 1.409569431264e-04 48 KSP Residual norm 1.291515114299e-04 49 KSP Residual norm 1.132431463990e-04 50 KSP Residual norm 1.076177919634e-04 51 KSP Residual norm 1.058367189894e-04 52 KSP Residual norm 1.002891393287e-04 53 KSP Residual norm 9.438385355118e-05 54 KSP Residual norm 9.013769617815e-05 55 KSP Residual norm 8.518449850926e-05 56 KSP Residual norm 8.183160209919e-05 57 KSP Residual norm 8.129743551771e-05 58 KSP Residual norm 7.852736007598e-05 59 KSP Residual norm 7.399090170848e-05 60 KSP Residual norm 7.083405084731e-05 61 KSP Residual norm 6.855686587362e-05 62 KSP Residual norm 6.628854223829e-05 63 KSP Residual norm 6.373858125434e-05 64 KSP Residual norm 5.955252962102e-05 65 KSP Residual norm 5.595502959016e-05 66 KSP Residual norm 5.273029818308e-05 67 KSP Residual norm 5.152508564920e-05 68 KSP Residual norm 5.164750423179e-05 69 KSP Residual norm 4.616860130697e-05 70 KSP Residual norm 4.336740952452e-05 71 KSP Residual norm 3.713874840003e-05 72 KSP Residual norm 3.231183700152e-05 73 KSP Residual norm 2.891791412701e-05 74 KSP Residual norm 2.484688885620e-05 75 KSP Residual norm 2.162171350374e-05 76 KSP Residual norm 1.960899815821e-05 77 KSP Residual norm 1.716428745834e-05 78 KSP Residual norm 1.484021101696e-05 79 KSP Residual norm 1.401693245409e-05 80 KSP Residual norm 1.350966223895e-05 81 KSP Residual norm 1.143428717858e-05 82 KSP Residual norm 9.777630210565e-06 83 KSP Residual norm 8.442757391392e-06 84 KSP Residual norm 7.373747795081e-06 85 KSP Residual norm 6.422221523504e-06 86 KSP Residual norm 5.649564526103e-06 87 KSP Residual norm 5.178762104662e-06 88 KSP Residual norm 4.980684797390e-06 89 KSP Residual norm 4.271601807602e-06 90 KSP Residual norm 3.587863603931e-06 91 KSP Residual norm 3.206925582557e-06 92 KSP Residual norm 2.897865760379e-06 93 KSP Residual norm 2.583803834558e-06 94 KSP Residual norm 2.314927961747e-06 95 KSP Residual norm 2.156994036331e-06 96 KSP Residual norm 2.093936038564e-06 97 KSP Residual norm 1.859433216891e-06 98 KSP Residual norm 1.647119655346e-06 99 KSP Residual norm 1.510774691190e-06 100 KSP Residual norm 1.487121116955e-06 101 KSP Residual norm 1.627605267029e-06 102 KSP Residual norm 1.330081441829e-06 103 KSP Residual norm 1.208810482093e-06 104 KSP Residual norm 1.118213568637e-06 105 KSP Residual norm 1.057846555641e-06 106 KSP Residual norm 1.022428137110e-06 107 KSP Residual norm 9.795011556394e-07 108 KSP Residual norm 9.420356749493e-07 109 KSP Residual norm 9.304714955541e-07 110 KSP Residual norm 9.074147785036e-07 111 KSP Residual norm 8.800538164074e-07 112 KSP Residual norm 9.056694274851e-07 113 KSP Residual norm 8.573809759637e-07 114 KSP Residual norm 7.866113308931e-07 115 KSP Residual norm 7.534842260477e-07 116 KSP Residual norm 7.280994039316e-07 117 KSP Residual norm 6.991017285709e-07 118 KSP Residual norm 6.773801519886e-07 119 KSP Residual norm 6.668570487323e-07 120 KSP Residual norm 6.375984528746e-07 121 KSP Residual norm 5.991947631821e-07 122 KSP Residual norm 5.803340196580e-07 123 KSP Residual norm 5.712932887997e-07 124 KSP Residual norm 5.202887279150e-07 125 KSP Residual norm 4.749171587413e-07 126 KSP Residual norm 4.454554160414e-07 127 KSP Residual norm 4.212596024903e-07 128 KSP Residual norm 4.048308537421e-07 129 KSP Residual norm 3.995840666130e-07 130 KSP Residual norm 3.648104838909e-07 131 KSP Residual norm 3.410422305249e-07 132 KSP Residual norm 3.693867293102e-07 133 KSP Residual norm 3.370045614642e-07 134 KSP Residual norm 2.967687471120e-07 135 KSP Residual norm 2.987846241803e-07 136 KSP Residual norm 2.611519136962e-07 137 KSP Residual norm 2.239724548782e-07 Linear tao_nls_ solve converged due to CONVERGED_RTOL iterations 137 0 LS Function value: 0.0867388, Step length: 0. 1 LS Function value: 0.0809674, Step length: 1. stx: 0., fx: 0.0867388, dgx: -0.00689219 sty: 0., fy: 0.0867388, dgy: -0.00689219 19 TAO, Function value: 0.0809674, Residual: 0.0162138 Residual norms for tao_nls_ solve. 0 KSP Residual norm 1.621375913830e-02 1 KSP Residual norm 1.472926869900e-02 2 KSP Residual norm 2.489620045278e-02 3 KSP Residual norm 3.264210913469e-02 4 KSP Residual norm 4.691607813644e-02 5 KSP Residual norm 4.864488156873e-02 6 KSP Residual norm 4.279412830955e-02 7 KSP Residual norm 3.753865445913e-02 8 KSP Residual norm 2.956096746190e-02 9 KSP Residual norm 2.227057559151e-02 10 KSP Residual norm 1.979117558582e-02 11 KSP Residual norm 1.727546980567e-02 12 KSP Residual norm 1.549831736040e-02 13 KSP Residual norm 1.509841309612e-02 14 KSP Residual norm 1.522273445772e-02 15 KSP Residual norm 1.609439823743e-02 16 KSP Residual norm 1.580719639807e-02 17 KSP Residual norm 1.610435696503e-02 18 KSP Residual norm 1.616507450818e-02 19 KSP Residual norm 1.658166234839e-02 20 KSP Residual norm 1.518837353600e-02 21 KSP Residual norm 1.388042820769e-02 22 KSP Residual norm 1.254229422319e-02 23 KSP Residual norm 1.093224683690e-02 24 KSP Residual norm 1.001314559300e-02 25 KSP Residual norm 9.365647830833e-03 26 KSP Residual norm 8.655475085946e-03 27 KSP Residual norm 8.273465941441e-03 28 KSP Residual norm 8.120959899553e-03 29 KSP Residual norm 7.824217015145e-03 30 KSP Residual norm 7.516097275515e-03 31 KSP Residual norm 7.384123911063e-03 32 KSP Residual norm 7.106892484422e-03 33 KSP Residual norm 6.902145823070e-03 34 KSP Residual norm 6.791030051755e-03 35 KSP Residual norm 6.472555108770e-03 36 KSP Residual norm 6.117263190847e-03 37 KSP Residual norm 5.589326131006e-03 38 KSP Residual norm 5.222860092445e-03 39 KSP Residual norm 5.024456109631e-03 40 KSP Residual norm 4.733854064865e-03 41 KSP Residual norm 4.568710485238e-03 42 KSP Residual norm 4.299548685102e-03 43 KSP Residual norm 4.057366882444e-03 44 KSP Residual norm 3.986768357381e-03 45 KSP Residual norm 3.809519131726e-03 46 KSP Residual norm 3.714237552070e-03 47 KSP Residual norm 3.539107717747e-03 48 KSP Residual norm 3.370390100506e-03 49 KSP Residual norm 3.358157889590e-03 50 KSP Residual norm 3.171176009816e-03 51 KSP Residual norm 3.173946444129e-03 52 KSP Residual norm 2.990008909875e-03 53 KSP Residual norm 2.677285780687e-03 54 KSP Residual norm 2.587009069929e-03 55 KSP Residual norm 2.372212085356e-03 56 KSP Residual norm 2.158974131965e-03 57 KSP Residual norm 2.047930447151e-03 58 KSP Residual norm 2.002166037376e-03 59 KSP Residual norm 1.967913538637e-03 60 KSP Residual norm 1.941164063521e-03 61 KSP Residual norm 1.976908718776e-03 62 KSP Residual norm 1.728658165032e-03 63 KSP Residual norm 1.596515506104e-03 64 KSP Residual norm 1.548359619684e-03 65 KSP Residual norm 1.453004223063e-03 66 KSP Residual norm 1.343609526689e-03 67 KSP Residual norm 1.271767580760e-03 68 KSP Residual norm 1.226754255879e-03 69 KSP Residual norm 1.147873032723e-03 70 KSP Residual norm 1.086101376524e-03 71 KSP Residual norm 1.050795883128e-03 72 KSP Residual norm 9.868412701650e-04 73 KSP Residual norm 9.353973449520e-04 74 KSP Residual norm 9.029708391749e-04 75 KSP Residual norm 8.604371341705e-04 76 KSP Residual norm 8.222866963896e-04 77 KSP Residual norm 7.796725295591e-04 78 KSP Residual norm 7.527980690896e-04 79 KSP Residual norm 7.719531411983e-04 80 KSP Residual norm 6.953712525787e-04 81 KSP Residual norm 6.333942032963e-04 82 KSP Residual norm 5.875830197250e-04 83 KSP Residual norm 5.537186923068e-04 84 KSP Residual norm 5.389118931914e-04 85 KSP Residual norm 5.076374696123e-04 86 KSP Residual norm 4.698451982996e-04 87 KSP Residual norm 4.448618929755e-04 88 KSP Residual norm 4.192194161709e-04 89 KSP Residual norm 3.985375954084e-04 90 KSP Residual norm 4.023854615544e-04 91 KSP Residual norm 4.139017126040e-04 92 KSP Residual norm 3.730307257292e-04 93 KSP Residual norm 3.326518657653e-04 94 KSP Residual norm 3.058378344981e-04 95 KSP Residual norm 2.903482238100e-04 96 KSP Residual norm 2.770146026247e-04 97 KSP Residual norm 2.671140038795e-04 98 KSP Residual norm 2.702191464576e-04 99 KSP Residual norm 2.439903758672e-04 100 KSP Residual norm 2.213485349506e-04 101 KSP Residual norm 2.083127539612e-04 102 KSP Residual norm 1.975866139856e-04 103 KSP Residual norm 1.995921856788e-04 104 KSP Residual norm 1.954061655249e-04 105 KSP Residual norm 1.706098220960e-04 106 KSP Residual norm 1.582846558607e-04 107 KSP Residual norm 1.601373565066e-04 108 KSP Residual norm 1.666442147218e-04 109 KSP Residual norm 1.471148032905e-04 110 KSP Residual norm 1.333989842713e-04 111 KSP Residual norm 1.235317840275e-04 112 KSP Residual norm 1.162957588071e-04 113 KSP Residual norm 1.092803452853e-04 114 KSP Residual norm 1.023563749302e-04 115 KSP Residual norm 9.844909355496e-05 116 KSP Residual norm 9.471395468647e-05 117 KSP Residual norm 9.123338612223e-05 118 KSP Residual norm 9.028822612359e-05 119 KSP Residual norm 8.301074912340e-05 120 KSP Residual norm 7.565221584516e-05 121 KSP Residual norm 6.980563885475e-05 122 KSP Residual norm 6.626502831360e-05 123 KSP Residual norm 6.897822827714e-05 124 KSP Residual norm 6.387776490919e-05 125 KSP Residual norm 5.649795181269e-05 126 KSP Residual norm 5.318611215633e-05 127 KSP Residual norm 5.361540672157e-05 128 KSP Residual norm 5.365505299439e-05 129 KSP Residual norm 4.715370202651e-05 130 KSP Residual norm 4.719727078023e-05 131 KSP Residual norm 4.657869315154e-05 132 KSP Residual norm 4.148490129667e-05 133 KSP Residual norm 3.919785225931e-05 134 KSP Residual norm 3.828008915271e-05 135 KSP Residual norm 3.913180022224e-05 136 KSP Residual norm 3.374370065994e-05 137 KSP Residual norm 3.103461956946e-05 138 KSP Residual norm 3.085751347389e-05 139 KSP Residual norm 2.933385030587e-05 140 KSP Residual norm 2.620221958126e-05 141 KSP Residual norm 2.421424356758e-05 142 KSP Residual norm 2.294184704873e-05 143 KSP Residual norm 2.272085213447e-05 144 KSP Residual norm 2.244823342353e-05 145 KSP Residual norm 1.989328225117e-05 146 KSP Residual norm 1.812209589374e-05 147 KSP Residual norm 1.710250816242e-05 148 KSP Residual norm 1.720545748453e-05 149 KSP Residual norm 1.723564809425e-05 150 KSP Residual norm 1.479998243349e-05 151 KSP Residual norm 1.393470739727e-05 152 KSP Residual norm 1.371702490494e-05 153 KSP Residual norm 1.334117340177e-05 154 KSP Residual norm 1.399083820869e-05 155 KSP Residual norm 1.237232625378e-05 156 KSP Residual norm 1.113608002719e-05 157 KSP Residual norm 1.036025857388e-05 158 KSP Residual norm 9.750507097203e-06 159 KSP Residual norm 9.291078212706e-06 160 KSP Residual norm 8.801083907076e-06 161 KSP Residual norm 8.299569804609e-06 162 KSP Residual norm 7.798953504282e-06 163 KSP Residual norm 7.399777170972e-06 164 KSP Residual norm 7.071340562071e-06 165 KSP Residual norm 6.836414567777e-06 166 KSP Residual norm 6.754703176097e-06 167 KSP Residual norm 6.463366170149e-06 168 KSP Residual norm 5.795644087926e-06 169 KSP Residual norm 5.474363554643e-06 170 KSP Residual norm 5.764454465144e-06 171 KSP Residual norm 5.285383091661e-06 172 KSP Residual norm 4.763196195142e-06 173 KSP Residual norm 5.152095848793e-06 174 KSP Residual norm 4.653082901866e-06 175 KSP Residual norm 4.086186712483e-06 176 KSP Residual norm 3.794861291773e-06 177 KSP Residual norm 3.569657018458e-06 178 KSP Residual norm 3.582976629071e-06 179 KSP Residual norm 3.408688148470e-06 180 KSP Residual norm 3.011549356551e-06 181 KSP Residual norm 2.859123050179e-06 182 KSP Residual norm 3.007483302729e-06 183 KSP Residual norm 2.766281841933e-06 184 KSP Residual norm 2.439884517668e-06 185 KSP Residual norm 2.438783551956e-06 186 KSP Residual norm 2.510643031105e-06 187 KSP Residual norm 2.135424046604e-06 188 KSP Residual norm 1.988150747647e-06 189 KSP Residual norm 1.949511580488e-06 190 KSP Residual norm 1.831192427254e-06 191 KSP Residual norm 1.674347480226e-06 192 KSP Residual norm 1.539890417214e-06 193 KSP Residual norm 1.424107016294e-06 194 KSP Residual norm 1.356193309816e-06 195 KSP Residual norm 1.402363423941e-06 196 KSP Residual norm 1.419456141246e-06 197 KSP Residual norm 1.244023340500e-06 198 KSP Residual norm 1.184867097718e-06 199 KSP Residual norm 1.077187213685e-06 200 KSP Residual norm 9.952581708702e-07 201 KSP Residual norm 9.871085487694e-07 202 KSP Residual norm 9.777032479241e-07 203 KSP Residual norm 8.776088744173e-07 204 KSP Residual norm 7.933278828459e-07 205 KSP Residual norm 7.328680803172e-07 206 KSP Residual norm 6.933439225574e-07 207 KSP Residual norm 6.644044235998e-07 208 KSP Residual norm 6.476611724203e-07 209 KSP Residual norm 6.253775817192e-07 210 KSP Residual norm 5.623548537876e-07 211 KSP Residual norm 5.181117822086e-07 212 KSP Residual norm 5.103747420090e-07 213 KSP Residual norm 5.684942834223e-07 214 KSP Residual norm 4.874166085899e-07 215 KSP Residual norm 4.275815604314e-07 216 KSP Residual norm 4.057050022012e-07 217 KSP Residual norm 4.141514548087e-07 218 KSP Residual norm 3.949205742070e-07 219 KSP Residual norm 3.502690905396e-07 220 KSP Residual norm 3.360994277785e-07 221 KSP Residual norm 3.280813436068e-07 222 KSP Residual norm 3.099397905134e-07 223 KSP Residual norm 2.945209646346e-07 224 KSP Residual norm 2.823546594372e-07 225 KSP Residual norm 2.676996609678e-07 226 KSP Residual norm 2.478964564277e-07 227 KSP Residual norm 2.290270722763e-07 228 KSP Residual norm 2.189266702670e-07 229 KSP Residual norm 2.271525741752e-07 230 KSP Residual norm 2.084069338265e-07 231 KSP Residual norm 1.843658554370e-07 232 KSP Residual norm 1.782107398697e-07 233 KSP Residual norm 1.825744843132e-07 234 KSP Residual norm 1.774420902334e-07 235 KSP Residual norm 1.673978126777e-07 236 KSP Residual norm 1.547080608974e-07 Linear tao_nls_ solve converged due to CONVERGED_RTOL iterations 236 0 LS Function value: 0.0809674, Step length: 0. 1 LS Function value: 0.0723695, Step length: 1. stx: 0., fx: 0.0809674, dgx: -0.0121733 sty: 0., fy: 0.0809674, dgy: -0.0121733 20 TAO, Function value: 0.0723695, Residual: 0.00674339 Residual norms for tao_nls_ solve. 0 KSP Residual norm 6.743394279342e-03 1 KSP Residual norm 8.031145698634e-03 2 KSP Residual norm 7.983449565965e-03 3 KSP Residual norm 8.586847748130e-03 4 KSP Residual norm 1.210574973609e-02 5 KSP Residual norm 1.479735793403e-02 6 KSP Residual norm 1.576476274775e-02 7 KSP Residual norm 1.747409404639e-02 8 KSP Residual norm 1.934169325233e-02 9 KSP Residual norm 1.914380847899e-02 10 KSP Residual norm 1.869892808965e-02 11 KSP Residual norm 1.683617800576e-02 12 KSP Residual norm 1.469448498502e-02 13 KSP Residual norm 1.309639986035e-02 14 KSP Residual norm 1.210868315374e-02 15 KSP Residual norm 1.043678457354e-02 16 KSP Residual norm 9.340625236657e-03 17 KSP Residual norm 8.606388328567e-03 18 KSP Residual norm 7.765564892629e-03 19 KSP Residual norm 7.086746729371e-03 20 KSP Residual norm 6.882955380600e-03 21 KSP Residual norm 6.823590172361e-03 22 KSP Residual norm 6.290691190763e-03 23 KSP Residual norm 5.965723789865e-03 24 KSP Residual norm 5.954806607501e-03 25 KSP Residual norm 6.044083397239e-03 26 KSP Residual norm 6.121487702575e-03 27 KSP Residual norm 6.144924766705e-03 28 KSP Residual norm 6.073574557921e-03 29 KSP Residual norm 5.963437567910e-03 30 KSP Residual norm 5.952578468834e-03 31 KSP Residual norm 5.886331268600e-03 32 KSP Residual norm 5.567724193485e-03 33 KSP Residual norm 5.323209456259e-03 34 KSP Residual norm 5.228731426837e-03 35 KSP Residual norm 5.014736599273e-03 36 KSP Residual norm 4.762867018253e-03 37 KSP Residual norm 4.661529321769e-03 38 KSP Residual norm 4.362096998226e-03 39 KSP Residual norm 4.122504685031e-03 40 KSP Residual norm 4.024272378339e-03 41 KSP Residual norm 3.784283469237e-03 42 KSP Residual norm 3.531702600465e-03 43 KSP Residual norm 3.397738623354e-03 44 KSP Residual norm 3.306046103667e-03 45 KSP Residual norm 3.184113484056e-03 46 KSP Residual norm 3.085057906081e-03 47 KSP Residual norm 2.979186382690e-03 48 KSP Residual norm 2.859437829483e-03 49 KSP Residual norm 2.777432249445e-03 50 KSP Residual norm 2.712585249570e-03 51 KSP Residual norm 2.622956254302e-03 52 KSP Residual norm 2.520537477490e-03 53 KSP Residual norm 2.432300652996e-03 54 KSP Residual norm 2.361450534138e-03 55 KSP Residual norm 2.264029509550e-03 56 KSP Residual norm 2.144052358262e-03 57 KSP Residual norm 2.049580643376e-03 58 KSP Residual norm 1.969660754077e-03 59 KSP Residual norm 1.872241458173e-03 60 KSP Residual norm 1.805836018502e-03 61 KSP Residual norm 1.796343475662e-03 62 KSP Residual norm 1.776001741024e-03 63 KSP Residual norm 1.636474258060e-03 64 KSP Residual norm 1.512868017769e-03 65 KSP Residual norm 1.454327119061e-03 66 KSP Residual norm 1.406999938375e-03 67 KSP Residual norm 1.354488502642e-03 68 KSP Residual norm 1.299011119939e-03 69 KSP Residual norm 1.232421762563e-03 70 KSP Residual norm 1.181421037271e-03 71 KSP Residual norm 1.150255939321e-03 72 KSP Residual norm 1.102362793786e-03 73 KSP Residual norm 1.053444848859e-03 74 KSP Residual norm 1.043029802494e-03 75 KSP Residual norm 1.063076503340e-03 76 KSP Residual norm 1.004503391012e-03 77 KSP Residual norm 9.228127007466e-04 78 KSP Residual norm 8.741763725491e-04 79 KSP Residual norm 8.317937680257e-04 80 KSP Residual norm 7.887856118968e-04 81 KSP Residual norm 7.882960865872e-04 82 KSP Residual norm 8.072417565850e-04 83 KSP Residual norm 6.995780662433e-04 84 KSP Residual norm 6.429824502743e-04 85 KSP Residual norm 6.194619770444e-04 86 KSP Residual norm 5.948847158061e-04 87 KSP Residual norm 5.644215581513e-04 88 KSP Residual norm 5.399983099766e-04 89 KSP Residual norm 5.188432350401e-04 90 KSP Residual norm 4.913849985804e-04 91 KSP Residual norm 4.655727631133e-04 92 KSP Residual norm 4.472478727005e-04 93 KSP Residual norm 4.302777420459e-04 94 KSP Residual norm 4.115793544969e-04 95 KSP Residual norm 3.956674260245e-04 96 KSP Residual norm 3.926342106890e-04 97 KSP Residual norm 3.923518231611e-04 98 KSP Residual norm 3.673398891163e-04 99 KSP Residual norm 3.418645602961e-04 100 KSP Residual norm 3.247022943826e-04 101 KSP Residual norm 3.213601533994e-04 102 KSP Residual norm 3.184059584018e-04 103 KSP Residual norm 2.906787945433e-04 104 KSP Residual norm 2.712693595875e-04 105 KSP Residual norm 2.607260191332e-04 106 KSP Residual norm 2.476225072733e-04 107 KSP Residual norm 2.339049870962e-04 108 KSP Residual norm 2.236964606894e-04 109 KSP Residual norm 2.152200401165e-04 110 KSP Residual norm 2.101230772531e-04 111 KSP Residual norm 2.127334789627e-04 112 KSP Residual norm 2.139460713316e-04 113 KSP Residual norm 1.931286100659e-04 114 KSP Residual norm 1.820229510244e-04 115 KSP Residual norm 1.793569250944e-04 116 KSP Residual norm 1.777603197878e-04 117 KSP Residual norm 1.617706948238e-04 118 KSP Residual norm 1.491980908578e-04 119 KSP Residual norm 1.434612952726e-04 120 KSP Residual norm 1.390423693985e-04 121 KSP Residual norm 1.378483227422e-04 122 KSP Residual norm 1.337917548449e-04 123 KSP Residual norm 1.220805779403e-04 124 KSP Residual norm 1.148902573223e-04 125 KSP Residual norm 1.093049189600e-04 126 KSP Residual norm 1.036022862897e-04 127 KSP Residual norm 9.861497671300e-05 128 KSP Residual norm 9.489895976409e-05 129 KSP Residual norm 9.153729956604e-05 130 KSP Residual norm 9.081859697137e-05 131 KSP Residual norm 9.462248445483e-05 132 KSP Residual norm 8.741433436627e-05 133 KSP Residual norm 7.888641159381e-05 134 KSP Residual norm 7.503751248122e-05 135 KSP Residual norm 7.210586816065e-05 136 KSP Residual norm 7.324493830842e-05 137 KSP Residual norm 7.516823679117e-05 138 KSP Residual norm 6.729595461943e-05 139 KSP Residual norm 6.218970105571e-05 140 KSP Residual norm 5.879279469449e-05 141 KSP Residual norm 5.667469886889e-05 142 KSP Residual norm 5.678535469907e-05 143 KSP Residual norm 5.556662742287e-05 144 KSP Residual norm 5.151942805708e-05 145 KSP Residual norm 4.830383661594e-05 146 KSP Residual norm 4.582688345541e-05 147 KSP Residual norm 4.374368804256e-05 148 KSP Residual norm 4.210470225323e-05 149 KSP Residual norm 4.066131320298e-05 150 KSP Residual norm 3.924860487420e-05 151 KSP Residual norm 3.876598939558e-05 152 KSP Residual norm 3.854885190670e-05 153 KSP Residual norm 3.590797857002e-05 154 KSP Residual norm 3.332267191238e-05 155 KSP Residual norm 3.187504639207e-05 156 KSP Residual norm 3.081467054979e-05 157 KSP Residual norm 2.970947802200e-05 158 KSP Residual norm 2.930399038627e-05 159 KSP Residual norm 2.933284417756e-05 160 KSP Residual norm 2.756651221102e-05 161 KSP Residual norm 2.622954799233e-05 162 KSP Residual norm 2.460371783641e-05 163 KSP Residual norm 2.267985263447e-05 164 KSP Residual norm 2.124169720859e-05 165 KSP Residual norm 1.999238254372e-05 166 KSP Residual norm 1.889970728685e-05 167 KSP Residual norm 1.796897582151e-05 168 KSP Residual norm 1.729612913348e-05 169 KSP Residual norm 1.720418429129e-05 170 KSP Residual norm 1.714096457901e-05 171 KSP Residual norm 1.566721737272e-05 172 KSP Residual norm 1.474948084057e-05 173 KSP Residual norm 1.456994564373e-05 174 KSP Residual norm 1.371593056314e-05 175 KSP Residual norm 1.261296686042e-05 176 KSP Residual norm 1.188927281891e-05 177 KSP Residual norm 1.151700045632e-05 178 KSP Residual norm 1.165979442427e-05 179 KSP Residual norm 1.116138720596e-05 180 KSP Residual norm 1.005029991702e-05 181 KSP Residual norm 9.442679074519e-06 182 KSP Residual norm 9.112891149997e-06 183 KSP Residual norm 9.114293063327e-06 184 KSP Residual norm 8.763517277679e-06 185 KSP Residual norm 8.139047182698e-06 186 KSP Residual norm 8.235570688381e-06 187 KSP Residual norm 8.088393203465e-06 188 KSP Residual norm 7.193272245938e-06 189 KSP Residual norm 6.741658342438e-06 190 KSP Residual norm 6.385947521060e-06 191 KSP Residual norm 6.120283594843e-06 192 KSP Residual norm 6.026144095286e-06 193 KSP Residual norm 5.847436301869e-06 194 KSP Residual norm 5.509246004853e-06 195 KSP Residual norm 5.397319201405e-06 196 KSP Residual norm 5.415467042262e-06 197 KSP Residual norm 4.916233581957e-06 198 KSP Residual norm 4.553008027912e-06 199 KSP Residual norm 4.622757710411e-06 200 KSP Residual norm 4.575393866155e-06 201 KSP Residual norm 4.054496225413e-06 202 KSP Residual norm 3.819954184747e-06 203 KSP Residual norm 3.663703122318e-06 204 KSP Residual norm 3.614040671877e-06 205 KSP Residual norm 3.678484138432e-06 206 KSP Residual norm 3.436520031050e-06 207 KSP Residual norm 3.232906020169e-06 208 KSP Residual norm 3.186990314730e-06 209 KSP Residual norm 3.019549432250e-06 210 KSP Residual norm 2.807953256822e-06 211 KSP Residual norm 2.661741219978e-06 212 KSP Residual norm 2.607439367431e-06 213 KSP Residual norm 2.796766662012e-06 214 KSP Residual norm 2.602948383258e-06 215 KSP Residual norm 2.294318266822e-06 216 KSP Residual norm 2.195752646988e-06 217 KSP Residual norm 2.166784652379e-06 218 KSP Residual norm 2.131657566073e-06 219 KSP Residual norm 1.986868574991e-06 220 KSP Residual norm 1.858742561348e-06 221 KSP Residual norm 1.766276837486e-06 222 KSP Residual norm 1.689460026430e-06 223 KSP Residual norm 1.618395852216e-06 224 KSP Residual norm 1.549152248016e-06 225 KSP Residual norm 1.488900459638e-06 226 KSP Residual norm 1.438284910326e-06 227 KSP Residual norm 1.399548482046e-06 228 KSP Residual norm 1.363399743650e-06 229 KSP Residual norm 1.311717814854e-06 230 KSP Residual norm 1.257742314190e-06 231 KSP Residual norm 1.210495423244e-06 232 KSP Residual norm 1.156015816699e-06 233 KSP Residual norm 1.096931764102e-06 234 KSP Residual norm 1.045362171880e-06 235 KSP Residual norm 1.025134845773e-06 236 KSP Residual norm 1.054411975494e-06 237 KSP Residual norm 9.951012480695e-07 238 KSP Residual norm 9.206785831745e-07 239 KSP Residual norm 9.558793857415e-07 240 KSP Residual norm 9.295044397865e-07 241 KSP Residual norm 8.193772574699e-07 242 KSP Residual norm 7.703137959994e-07 243 KSP Residual norm 7.360025890614e-07 244 KSP Residual norm 7.302860866358e-07 245 KSP Residual norm 7.604620072339e-07 246 KSP Residual norm 6.931917322160e-07 247 KSP Residual norm 6.349419434789e-07 248 KSP Residual norm 6.107603780081e-07 249 KSP Residual norm 5.984352457458e-07 250 KSP Residual norm 5.841249285691e-07 251 KSP Residual norm 5.657444531139e-07 252 KSP Residual norm 5.719309207316e-07 253 KSP Residual norm 5.368403027109e-07 254 KSP Residual norm 4.821347822961e-07 255 KSP Residual norm 4.531564297366e-07 256 KSP Residual norm 4.501039182977e-07 257 KSP Residual norm 4.693999294911e-07 258 KSP Residual norm 4.207409385831e-07 259 KSP Residual norm 4.073236802238e-07 260 KSP Residual norm 4.209814859879e-07 261 KSP Residual norm 3.661280664564e-07 262 KSP Residual norm 3.368350962680e-07 263 KSP Residual norm 3.204013032555e-07 264 KSP Residual norm 3.057380602382e-07 265 KSP Residual norm 2.943021798433e-07 266 KSP Residual norm 2.888085065229e-07 267 KSP Residual norm 2.795562119489e-07 268 KSP Residual norm 2.629557599736e-07 269 KSP Residual norm 2.534810103990e-07 270 KSP Residual norm 2.490064293238e-07 271 KSP Residual norm 2.340193810202e-07 272 KSP Residual norm 2.225086558635e-07 273 KSP Residual norm 2.230308425354e-07 274 KSP Residual norm 2.076852427961e-07 275 KSP Residual norm 1.882430651327e-07 276 KSP Residual norm 1.767109494853e-07 277 KSP Residual norm 1.685016951822e-07 278 KSP Residual norm 1.617795477543e-07 279 KSP Residual norm 1.554946765149e-07 280 KSP Residual norm 1.505124143665e-07 281 KSP Residual norm 1.483868670402e-07 282 KSP Residual norm 1.446201519808e-07 283 KSP Residual norm 1.367235843478e-07 284 KSP Residual norm 1.288936171593e-07 285 KSP Residual norm 1.211128951802e-07 286 KSP Residual norm 1.137707538781e-07 287 KSP Residual norm 1.079306486577e-07 288 KSP Residual norm 1.051686945912e-07 289 KSP Residual norm 1.042525366196e-07 290 KSP Residual norm 9.747757681541e-08 291 KSP Residual norm 8.913320504073e-08 292 KSP Residual norm 8.339996027427e-08 293 KSP Residual norm 7.910616362748e-08 294 KSP Residual norm 7.656095858332e-08 295 KSP Residual norm 7.693552218389e-08 296 KSP Residual norm 7.834978452835e-08 297 KSP Residual norm 7.286960401494e-08 298 KSP Residual norm 6.681568561362e-08 Linear tao_nls_ solve converged due to CONVERGED_RTOL iterations 298 0 LS Function value: 0.0723695, Step length: 0. 1 LS Function value: 0.0697349, Step length: 1. stx: 0., fx: 0.0723695, dgx: -0.00384056 sty: 0., fy: 0.0723695, dgy: -0.00384056 21 TAO, Function value: 0.0697349, Residual: 0.00271122 Residual norms for tao_nls_ solve. 0 KSP Residual norm 2.711220953755e-03 1 KSP Residual norm 3.049701903253e-03 2 KSP Residual norm 3.022714582982e-03 3 KSP Residual norm 2.932529423873e-03 4 KSP Residual norm 3.615549086062e-03 5 KSP Residual norm 4.429897817331e-03 6 KSP Residual norm 4.761722261097e-03 7 KSP Residual norm 5.430482887586e-03 8 KSP Residual norm 6.320203899740e-03 9 KSP Residual norm 6.776923127009e-03 10 KSP Residual norm 6.832632140750e-03 11 KSP Residual norm 7.081077481151e-03 12 KSP Residual norm 7.225346985001e-03 13 KSP Residual norm 7.058074356103e-03 14 KSP Residual norm 6.831146408132e-03 15 KSP Residual norm 6.585524752240e-03 16 KSP Residual norm 6.219532195304e-03 17 KSP Residual norm 5.866658284103e-03 18 KSP Residual norm 5.562162190362e-03 19 KSP Residual norm 5.154085426863e-03 20 KSP Residual norm 4.698716113021e-03 21 KSP Residual norm 4.386186829856e-03 22 KSP Residual norm 4.091102957789e-03 23 KSP Residual norm 3.624009997332e-03 24 KSP Residual norm 3.260834350442e-03 25 KSP Residual norm 3.031051802395e-03 26 KSP Residual norm 2.793899854669e-03 27 KSP Residual norm 2.575090539065e-03 28 KSP Residual norm 2.436092602972e-03 29 KSP Residual norm 2.321460327090e-03 30 KSP Residual norm 2.201836945773e-03 31 KSP Residual norm 2.131447441233e-03 32 KSP Residual norm 2.091343618668e-03 33 KSP Residual norm 2.027795664847e-03 34 KSP Residual norm 1.983886831308e-03 35 KSP Residual norm 1.983577962205e-03 36 KSP Residual norm 2.005260226306e-03 37 KSP Residual norm 2.032252739335e-03 38 KSP Residual norm 2.064683717256e-03 39 KSP Residual norm 2.075109953200e-03 40 KSP Residual norm 2.076097065587e-03 41 KSP Residual norm 2.062939376733e-03 42 KSP Residual norm 2.029471304626e-03 43 KSP Residual norm 1.996599148315e-03 44 KSP Residual norm 1.962277650658e-03 45 KSP Residual norm 1.893277170253e-03 46 KSP Residual norm 1.790858582482e-03 47 KSP Residual norm 1.689233634131e-03 48 KSP Residual norm 1.594358476437e-03 49 KSP Residual norm 1.498341927630e-03 50 KSP Residual norm 1.411600548750e-03 51 KSP Residual norm 1.333474605969e-03 52 KSP Residual norm 1.251466590326e-03 53 KSP Residual norm 1.171388029678e-03 54 KSP Residual norm 1.100059417561e-03 55 KSP Residual norm 1.037596084485e-03 56 KSP Residual norm 9.882297955365e-04 57 KSP Residual norm 9.519751539257e-04 58 KSP Residual norm 9.188654865038e-04 59 KSP Residual norm 8.867861024770e-04 60 KSP Residual norm 8.622266059658e-04 61 KSP Residual norm 8.423953356070e-04 62 KSP Residual norm 8.126365743310e-04 63 KSP Residual norm 7.738062448970e-04 64 KSP Residual norm 7.364424893598e-04 65 KSP Residual norm 7.062047632888e-04 66 KSP Residual norm 6.808611796963e-04 67 KSP Residual norm 6.552732559312e-04 68 KSP Residual norm 6.336535982654e-04 69 KSP Residual norm 6.233342398694e-04 70 KSP Residual norm 6.206665722732e-04 71 KSP Residual norm 6.090819509585e-04 72 KSP Residual norm 5.905243927409e-04 73 KSP Residual norm 5.651576123519e-04 74 KSP Residual norm 5.309910374825e-04 75 KSP Residual norm 5.009457610529e-04 76 KSP Residual norm 4.770798073087e-04 77 KSP Residual norm 4.589698704397e-04 78 KSP Residual norm 4.418410349698e-04 79 KSP Residual norm 4.283887246924e-04 80 KSP Residual norm 4.174268411430e-04 81 KSP Residual norm 4.027336471857e-04 82 KSP Residual norm 3.857744261790e-04 83 KSP Residual norm 3.683276630646e-04 84 KSP Residual norm 3.561216556292e-04 85 KSP Residual norm 3.575513140207e-04 86 KSP Residual norm 3.508595833463e-04 87 KSP Residual norm 3.222858050153e-04 88 KSP Residual norm 3.013597212443e-04 89 KSP Residual norm 2.858261002419e-04 90 KSP Residual norm 2.726739456669e-04 91 KSP Residual norm 2.621336351215e-04 92 KSP Residual norm 2.529983528093e-04 93 KSP Residual norm 2.464029261409e-04 94 KSP Residual norm 2.449576384298e-04 95 KSP Residual norm 2.402554209331e-04 96 KSP Residual norm 2.257827007342e-04 97 KSP Residual norm 2.142920585489e-04 98 KSP Residual norm 2.083654461114e-04 99 KSP Residual norm 2.011169284610e-04 100 KSP Residual norm 1.939608990920e-04 101 KSP Residual norm 1.873222720616e-04 102 KSP Residual norm 1.802149435310e-04 103 KSP Residual norm 1.738028815321e-04 104 KSP Residual norm 1.670260101423e-04 105 KSP Residual norm 1.619740298359e-04 106 KSP Residual norm 1.601636523093e-04 107 KSP Residual norm 1.581328199611e-04 108 KSP Residual norm 1.484864786253e-04 109 KSP Residual norm 1.393731899628e-04 110 KSP Residual norm 1.322329369872e-04 111 KSP Residual norm 1.263988518362e-04 112 KSP Residual norm 1.237164819792e-04 113 KSP Residual norm 1.221684101105e-04 114 KSP Residual norm 1.174847643903e-04 115 KSP Residual norm 1.114185757825e-04 116 KSP Residual norm 1.068083620601e-04 117 KSP Residual norm 1.027625791258e-04 118 KSP Residual norm 9.896586316958e-05 119 KSP Residual norm 9.626721264018e-05 120 KSP Residual norm 9.325657455335e-05 121 KSP Residual norm 9.018532791735e-05 122 KSP Residual norm 8.707310118770e-05 123 KSP Residual norm 8.431086475402e-05 124 KSP Residual norm 8.134197934835e-05 125 KSP Residual norm 7.814122074628e-05 126 KSP Residual norm 7.513754111701e-05 127 KSP Residual norm 7.317807457951e-05 128 KSP Residual norm 7.353995382314e-05 129 KSP Residual norm 7.206615794258e-05 130 KSP Residual norm 6.656543163865e-05 131 KSP Residual norm 6.252009956883e-05 132 KSP Residual norm 5.974683047187e-05 133 KSP Residual norm 5.721230325431e-05 134 KSP Residual norm 5.455876089606e-05 135 KSP Residual norm 5.228793706401e-05 136 KSP Residual norm 5.036040103101e-05 137 KSP Residual norm 4.866527339969e-05 138 KSP Residual norm 4.707102932359e-05 139 KSP Residual norm 4.554542543846e-05 140 KSP Residual norm 4.412420751468e-05 141 KSP Residual norm 4.246816612843e-05 142 KSP Residual norm 4.040354267271e-05 143 KSP Residual norm 3.836004267277e-05 144 KSP Residual norm 3.675076639621e-05 145 KSP Residual norm 3.560259583002e-05 146 KSP Residual norm 3.476188560116e-05 147 KSP Residual norm 3.493434663017e-05 148 KSP Residual norm 3.507701096809e-05 149 KSP Residual norm 3.226844943894e-05 150 KSP Residual norm 3.005415485832e-05 151 KSP Residual norm 2.867586846175e-05 152 KSP Residual norm 2.775973983339e-05 153 KSP Residual norm 2.716029527881e-05 154 KSP Residual norm 2.609260564245e-05 155 KSP Residual norm 2.450499366128e-05 156 KSP Residual norm 2.308684453274e-05 157 KSP Residual norm 2.192504266144e-05 158 KSP Residual norm 2.086462131853e-05 159 KSP Residual norm 1.975343238780e-05 160 KSP Residual norm 1.871714514423e-05 161 KSP Residual norm 1.784763747205e-05 162 KSP Residual norm 1.705039452001e-05 163 KSP Residual norm 1.627624548681e-05 164 KSP Residual norm 1.567906926435e-05 165 KSP Residual norm 1.542940895800e-05 166 KSP Residual norm 1.526438047146e-05 167 KSP Residual norm 1.435055254331e-05 168 KSP Residual norm 1.350937685407e-05 169 KSP Residual norm 1.306391713752e-05 170 KSP Residual norm 1.283221177588e-05 171 KSP Residual norm 1.237592603396e-05 172 KSP Residual norm 1.165273057250e-05 173 KSP Residual norm 1.105680977858e-05 174 KSP Residual norm 1.066704808042e-05 175 KSP Residual norm 1.031901419906e-05 176 KSP Residual norm 1.002264529321e-05 177 KSP Residual norm 9.736316245307e-06 178 KSP Residual norm 9.381833579080e-06 179 KSP Residual norm 9.049342739191e-06 180 KSP Residual norm 8.818732926936e-06 181 KSP Residual norm 8.687402312559e-06 182 KSP Residual norm 8.482078780819e-06 183 KSP Residual norm 8.187094696654e-06 184 KSP Residual norm 8.067189773906e-06 185 KSP Residual norm 8.009723876594e-06 186 KSP Residual norm 7.576497086030e-06 187 KSP Residual norm 7.091452542809e-06 188 KSP Residual norm 6.771688641550e-06 189 KSP Residual norm 6.538077879932e-06 190 KSP Residual norm 6.295444518104e-06 191 KSP Residual norm 6.058529430414e-06 192 KSP Residual norm 5.859149955871e-06 193 KSP Residual norm 5.782432351528e-06 194 KSP Residual norm 5.863577514644e-06 195 KSP Residual norm 5.601686352869e-06 196 KSP Residual norm 5.216198371559e-06 197 KSP Residual norm 4.976589127803e-06 198 KSP Residual norm 4.860973465420e-06 199 KSP Residual norm 4.916906044828e-06 200 KSP Residual norm 4.893316214953e-06 201 KSP Residual norm 4.531477017341e-06 202 KSP Residual norm 4.264879001154e-06 203 KSP Residual norm 4.104412687727e-06 204 KSP Residual norm 3.966779930254e-06 205 KSP Residual norm 3.843178895894e-06 206 KSP Residual norm 3.746502336187e-06 207 KSP Residual norm 3.663485625892e-06 208 KSP Residual norm 3.538136838118e-06 209 KSP Residual norm 3.422386154770e-06 210 KSP Residual norm 3.410926682938e-06 211 KSP Residual norm 3.358137819423e-06 212 KSP Residual norm 3.128637523195e-06 213 KSP Residual norm 2.965365610579e-06 214 KSP Residual norm 2.857047338640e-06 215 KSP Residual norm 2.771639308896e-06 216 KSP Residual norm 2.682621667110e-06 217 KSP Residual norm 2.597695206471e-06 218 KSP Residual norm 2.517668659239e-06 219 KSP Residual norm 2.430453570354e-06 220 KSP Residual norm 2.332370142133e-06 221 KSP Residual norm 2.236190808567e-06 222 KSP Residual norm 2.146870695741e-06 223 KSP Residual norm 2.083248588078e-06 224 KSP Residual norm 2.036206057010e-06 225 KSP Residual norm 1.961576078747e-06 226 KSP Residual norm 1.856919360268e-06 227 KSP Residual norm 1.763191040837e-06 228 KSP Residual norm 1.683770866877e-06 229 KSP Residual norm 1.614232486918e-06 230 KSP Residual norm 1.549831205495e-06 231 KSP Residual norm 1.485135167096e-06 232 KSP Residual norm 1.420294131346e-06 233 KSP Residual norm 1.369552096273e-06 234 KSP Residual norm 1.357815208665e-06 235 KSP Residual norm 1.353598166503e-06 236 KSP Residual norm 1.262543759927e-06 237 KSP Residual norm 1.175836461031e-06 238 KSP Residual norm 1.122230740697e-06 239 KSP Residual norm 1.086992874333e-06 240 KSP Residual norm 1.085472851117e-06 241 KSP Residual norm 1.090887829764e-06 242 KSP Residual norm 1.006834060754e-06 243 KSP Residual norm 9.353450418344e-07 244 KSP Residual norm 8.922840719175e-07 245 KSP Residual norm 8.623452593007e-07 246 KSP Residual norm 8.461945728156e-07 247 KSP Residual norm 8.341951393874e-07 248 KSP Residual norm 7.868256919884e-07 249 KSP Residual norm 7.331866462150e-07 250 KSP Residual norm 6.940817129334e-07 251 KSP Residual norm 6.615239205691e-07 252 KSP Residual norm 6.317702325966e-07 253 KSP Residual norm 6.074458538365e-07 254 KSP Residual norm 5.904786899379e-07 255 KSP Residual norm 5.735936749565e-07 256 KSP Residual norm 5.463638643872e-07 257 KSP Residual norm 5.163245762462e-07 258 KSP Residual norm 4.891580215970e-07 259 KSP Residual norm 4.666415535858e-07 260 KSP Residual norm 4.485992444051e-07 261 KSP Residual norm 4.366474494780e-07 262 KSP Residual norm 4.269232955564e-07 263 KSP Residual norm 4.053138014304e-07 264 KSP Residual norm 3.790780421708e-07 265 KSP Residual norm 3.603288982613e-07 266 KSP Residual norm 3.564617342081e-07 267 KSP Residual norm 3.613098029096e-07 268 KSP Residual norm 3.362314778545e-07 269 KSP Residual norm 3.122195484158e-07 270 KSP Residual norm 2.975960948127e-07 271 KSP Residual norm 2.863771404707e-07 272 KSP Residual norm 2.805332409068e-07 273 KSP Residual norm 2.815610783548e-07 274 KSP Residual norm 2.704557296903e-07 275 KSP Residual norm 2.493591761312e-07 276 KSP Residual norm 2.350082848530e-07 277 KSP Residual norm 2.249728891422e-07 278 KSP Residual norm 2.151874203210e-07 279 KSP Residual norm 2.067979033564e-07 280 KSP Residual norm 2.002364460188e-07 281 KSP Residual norm 1.948304270470e-07 282 KSP Residual norm 1.894674720582e-07 283 KSP Residual norm 1.840916562484e-07 284 KSP Residual norm 1.760129581090e-07 285 KSP Residual norm 1.665419729479e-07 286 KSP Residual norm 1.595833796158e-07 287 KSP Residual norm 1.549859277004e-07 288 KSP Residual norm 1.505003845495e-07 289 KSP Residual norm 1.429309593162e-07 290 KSP Residual norm 1.344865483385e-07 291 KSP Residual norm 1.269014833396e-07 292 KSP Residual norm 1.204046528904e-07 293 KSP Residual norm 1.154247646695e-07 294 KSP Residual norm 1.110229656436e-07 295 KSP Residual norm 1.083053014645e-07 296 KSP Residual norm 1.104292819527e-07 297 KSP Residual norm 1.084827903907e-07 298 KSP Residual norm 9.851512872627e-08 299 KSP Residual norm 9.330494811219e-08 300 KSP Residual norm 9.060740240526e-08 301 KSP Residual norm 8.773709955189e-08 302 KSP Residual norm 8.406741497170e-08 303 KSP Residual norm 8.011187331253e-08 304 KSP Residual norm 7.641056943212e-08 305 KSP Residual norm 7.303959406110e-08 306 KSP Residual norm 7.042814533214e-08 307 KSP Residual norm 6.910497437356e-08 308 KSP Residual norm 6.882095619949e-08 309 KSP Residual norm 6.725131513775e-08 310 KSP Residual norm 6.360488460247e-08 311 KSP Residual norm 6.017501805187e-08 312 KSP Residual norm 5.779361372693e-08 313 KSP Residual norm 5.637097166355e-08 314 KSP Residual norm 5.465248241555e-08 315 KSP Residual norm 5.201972506923e-08 316 KSP Residual norm 4.929524198499e-08 317 KSP Residual norm 4.708807438711e-08 318 KSP Residual norm 4.522001488417e-08 319 KSP Residual norm 4.363897753140e-08 320 KSP Residual norm 4.219334645621e-08 321 KSP Residual norm 4.066851230036e-08 322 KSP Residual norm 3.910202876469e-08 323 KSP Residual norm 3.769486459140e-08 324 KSP Residual norm 3.630842882055e-08 325 KSP Residual norm 3.536729825769e-08 326 KSP Residual norm 3.537222779583e-08 327 KSP Residual norm 3.542096595284e-08 328 KSP Residual norm 3.327878333530e-08 329 KSP Residual norm 3.132869299472e-08 330 KSP Residual norm 3.008750835122e-08 331 KSP Residual norm 2.907620598227e-08 332 KSP Residual norm 2.808411337649e-08 333 KSP Residual norm 2.704123185372e-08 Linear tao_nls_ solve converged due to CONVERGED_RTOL iterations 333 0 LS Function value: 0.0697349, Step length: 0. 1 LS Function value: 0.06919, Step length: 1. stx: 0., fx: 0.0697349, dgx: -0.000859431 sty: 0., fy: 0.0697349, dgy: -0.000859431 22 TAO, Function value: 0.06919, Residual: 0.00111639 Residual norms for tao_nls_ solve. 0 KSP Residual norm 1.116387257518e-03 1 KSP Residual norm 1.194086088885e-03 2 KSP Residual norm 1.059449685138e-03 3 KSP Residual norm 9.242433754978e-04 4 KSP Residual norm 1.036851464279e-03 5 KSP Residual norm 1.102989897690e-03 6 KSP Residual norm 1.099252260241e-03 7 KSP Residual norm 1.170365626322e-03 8 KSP Residual norm 1.222527343640e-03 9 KSP Residual norm 1.259168572966e-03 10 KSP Residual norm 1.293506205183e-03 11 KSP Residual norm 1.332072119021e-03 12 KSP Residual norm 1.388582057913e-03 13 KSP Residual norm 1.414993748413e-03 14 KSP Residual norm 1.435896647380e-03 15 KSP Residual norm 1.484701868594e-03 16 KSP Residual norm 1.549387625751e-03 17 KSP Residual norm 1.623912850011e-03 18 KSP Residual norm 1.703249338921e-03 19 KSP Residual norm 1.781008560774e-03 20 KSP Residual norm 1.849525048129e-03 21 KSP Residual norm 1.910348148392e-03 22 KSP Residual norm 1.933472586985e-03 23 KSP Residual norm 1.916683644719e-03 24 KSP Residual norm 1.864958159057e-03 25 KSP Residual norm 1.797695945840e-03 26 KSP Residual norm 1.730093656675e-03 27 KSP Residual norm 1.618712690456e-03 28 KSP Residual norm 1.466041208827e-03 29 KSP Residual norm 1.331619801519e-03 30 KSP Residual norm 1.204908642498e-03 31 KSP Residual norm 1.085437849844e-03 32 KSP Residual norm 9.776815458175e-04 33 KSP Residual norm 8.766187690949e-04 34 KSP Residual norm 7.897946737062e-04 35 KSP Residual norm 7.121602080694e-04 36 KSP Residual norm 6.441948150201e-04 37 KSP Residual norm 5.909330520115e-04 38 KSP Residual norm 5.435062809009e-04 39 KSP Residual norm 4.991948957123e-04 40 KSP Residual norm 4.648009079054e-04 41 KSP Residual norm 4.376714973301e-04 42 KSP Residual norm 4.131444813453e-04 43 KSP Residual norm 3.939801078980e-04 44 KSP Residual norm 3.798112680926e-04 45 KSP Residual norm 3.649931519999e-04 46 KSP Residual norm 3.518919579721e-04 47 KSP Residual norm 3.438241418189e-04 48 KSP Residual norm 3.388628974511e-04 49 KSP Residual norm 3.353178131187e-04 50 KSP Residual norm 3.320930063355e-04 51 KSP Residual norm 3.290939585646e-04 52 KSP Residual norm 3.274862535170e-04 53 KSP Residual norm 3.273255536589e-04 54 KSP Residual norm 3.278950345413e-04 55 KSP Residual norm 3.291566936181e-04 56 KSP Residual norm 3.289060020496e-04 57 KSP Residual norm 3.242685895550e-04 58 KSP Residual norm 3.180148809198e-04 59 KSP Residual norm 3.120064480283e-04 60 KSP Residual norm 3.046809279250e-04 61 KSP Residual norm 2.953374145128e-04 62 KSP Residual norm 2.845457454549e-04 63 KSP Residual norm 2.733319026654e-04 64 KSP Residual norm 2.620362304411e-04 65 KSP Residual norm 2.496821407234e-04 66 KSP Residual norm 2.370743858430e-04 67 KSP Residual norm 2.262797835044e-04 68 KSP Residual norm 2.174050211906e-04 69 KSP Residual norm 2.075565394912e-04 70 KSP Residual norm 1.964070664783e-04 71 KSP Residual norm 1.860137990325e-04 72 KSP Residual norm 1.765260597745e-04 73 KSP Residual norm 1.685330791588e-04 74 KSP Residual norm 1.638487761469e-04 75 KSP Residual norm 1.588554979052e-04 76 KSP Residual norm 1.483081907948e-04 77 KSP Residual norm 1.381518402960e-04 78 KSP Residual norm 1.307432609361e-04 79 KSP Residual norm 1.245914415698e-04 80 KSP Residual norm 1.194771535366e-04 81 KSP Residual norm 1.153166791442e-04 82 KSP Residual norm 1.116892557771e-04 83 KSP Residual norm 1.086009881755e-04 84 KSP Residual norm 1.054167306580e-04 85 KSP Residual norm 1.024609148982e-04 86 KSP Residual norm 1.004495558034e-04 87 KSP Residual norm 9.893608729761e-05 88 KSP Residual norm 9.781020497241e-05 89 KSP Residual norm 9.402249432945e-05 90 KSP Residual norm 8.889103456283e-05 91 KSP Residual norm 8.501110988110e-05 92 KSP Residual norm 8.150743677026e-05 93 KSP Residual norm 7.826338260250e-05 94 KSP Residual norm 7.528579939174e-05 95 KSP Residual norm 7.286614674274e-05 96 KSP Residual norm 7.123822182833e-05 97 KSP Residual norm 6.989189187998e-05 98 KSP Residual norm 6.789373180380e-05 99 KSP Residual norm 6.626262908167e-05 100 KSP Residual norm 6.480824466085e-05 101 KSP Residual norm 6.384408823298e-05 102 KSP Residual norm 6.252183772992e-05 103 KSP Residual norm 6.075219143751e-05 104 KSP Residual norm 5.892599371400e-05 105 KSP Residual norm 5.734965095673e-05 106 KSP Residual norm 5.597653487077e-05 107 KSP Residual norm 5.438988083907e-05 108 KSP Residual norm 5.273218605638e-05 109 KSP Residual norm 5.055102756809e-05 110 KSP Residual norm 4.878731410607e-05 111 KSP Residual norm 4.689291180525e-05 112 KSP Residual norm 4.512055641753e-05 113 KSP Residual norm 4.297823337199e-05 114 KSP Residual norm 4.086411175761e-05 115 KSP Residual norm 3.868903566002e-05 116 KSP Residual norm 3.670589278595e-05 117 KSP Residual norm 3.475668885214e-05 118 KSP Residual norm 3.313782051568e-05 119 KSP Residual norm 3.168560275206e-05 120 KSP Residual norm 3.038284436781e-05 121 KSP Residual norm 2.899811075348e-05 122 KSP Residual norm 2.779408489797e-05 123 KSP Residual norm 2.690356274165e-05 124 KSP Residual norm 2.611521278182e-05 125 KSP Residual norm 2.504770027361e-05 126 KSP Residual norm 2.404762007288e-05 127 KSP Residual norm 2.344060253160e-05 128 KSP Residual norm 2.299781860259e-05 129 KSP Residual norm 2.240173696253e-05 130 KSP Residual norm 2.160970386827e-05 131 KSP Residual norm 2.094402084710e-05 132 KSP Residual norm 2.056954245947e-05 133 KSP Residual norm 2.042114617721e-05 134 KSP Residual norm 2.043456415201e-05 135 KSP Residual norm 2.035312724638e-05 136 KSP Residual norm 1.951040337322e-05 137 KSP Residual norm 1.867975557764e-05 138 KSP Residual norm 1.831250320129e-05 139 KSP Residual norm 1.808878877361e-05 140 KSP Residual norm 1.710333217119e-05 141 KSP Residual norm 1.592883108989e-05 142 KSP Residual norm 1.506259383073e-05 143 KSP Residual norm 1.430973819428e-05 144 KSP Residual norm 1.355097830975e-05 145 KSP Residual norm 1.287732852243e-05 146 KSP Residual norm 1.235943338421e-05 147 KSP Residual norm 1.200325469618e-05 148 KSP Residual norm 1.162246284313e-05 149 KSP Residual norm 1.098766869336e-05 150 KSP Residual norm 1.041999242485e-05 151 KSP Residual norm 9.986231767194e-06 152 KSP Residual norm 9.554949603235e-06 153 KSP Residual norm 9.163517720638e-06 154 KSP Residual norm 8.838502203224e-06 155 KSP Residual norm 8.597005961357e-06 156 KSP Residual norm 8.455211194112e-06 157 KSP Residual norm 8.300338355467e-06 158 KSP Residual norm 8.018174482477e-06 159 KSP Residual norm 7.707893270524e-06 160 KSP Residual norm 7.394035697193e-06 161 KSP Residual norm 7.079700625511e-06 162 KSP Residual norm 6.799364847234e-06 163 KSP Residual norm 6.551965119700e-06 164 KSP Residual norm 6.339693709462e-06 165 KSP Residual norm 6.156870434583e-06 166 KSP Residual norm 6.013660447151e-06 167 KSP Residual norm 5.882477897524e-06 168 KSP Residual norm 5.728483899458e-06 169 KSP Residual norm 5.565216027045e-06 170 KSP Residual norm 5.393466306367e-06 171 KSP Residual norm 5.214011816804e-06 172 KSP Residual norm 5.037589396271e-06 173 KSP Residual norm 4.857902920669e-06 174 KSP Residual norm 4.690444702997e-06 175 KSP Residual norm 4.554804508490e-06 176 KSP Residual norm 4.431719059477e-06 177 KSP Residual norm 4.284487299290e-06 178 KSP Residual norm 4.115987777777e-06 179 KSP Residual norm 3.932298906707e-06 180 KSP Residual norm 3.772645835779e-06 181 KSP Residual norm 3.638376620279e-06 182 KSP Residual norm 3.544444695431e-06 183 KSP Residual norm 3.476148677357e-06 184 KSP Residual norm 3.407648959778e-06 185 KSP Residual norm 3.327370520132e-06 186 KSP Residual norm 3.243418460772e-06 187 KSP Residual norm 3.165941748150e-06 188 KSP Residual norm 3.095734020015e-06 189 KSP Residual norm 3.031397235423e-06 190 KSP Residual norm 2.966094956592e-06 191 KSP Residual norm 2.893390522851e-06 192 KSP Residual norm 2.816627394701e-06 193 KSP Residual norm 2.751798647792e-06 194 KSP Residual norm 2.714207691051e-06 195 KSP Residual norm 2.694782346256e-06 196 KSP Residual norm 2.672918559305e-06 197 KSP Residual norm 2.593160084445e-06 198 KSP Residual norm 2.466598980809e-06 199 KSP Residual norm 2.351900004127e-06 200 KSP Residual norm 2.248131441724e-06 201 KSP Residual norm 2.150061915385e-06 202 KSP Residual norm 2.067183475158e-06 203 KSP Residual norm 1.996654770213e-06 204 KSP Residual norm 1.926296999874e-06 205 KSP Residual norm 1.848332799572e-06 206 KSP Residual norm 1.773459598944e-06 207 KSP Residual norm 1.706395789223e-06 208 KSP Residual norm 1.646358482358e-06 209 KSP Residual norm 1.592065182528e-06 210 KSP Residual norm 1.550012234987e-06 211 KSP Residual norm 1.518002388500e-06 212 KSP Residual norm 1.466768695124e-06 213 KSP Residual norm 1.389627501598e-06 214 KSP Residual norm 1.319967825606e-06 215 KSP Residual norm 1.258732599328e-06 216 KSP Residual norm 1.208226237206e-06 217 KSP Residual norm 1.180848857703e-06 218 KSP Residual norm 1.185133073712e-06 219 KSP Residual norm 1.133658732082e-06 220 KSP Residual norm 1.046568662754e-06 221 KSP Residual norm 9.936278717360e-07 222 KSP Residual norm 9.634412406361e-07 223 KSP Residual norm 9.524074745290e-07 224 KSP Residual norm 9.345488902744e-07 225 KSP Residual norm 8.833615524336e-07 226 KSP Residual norm 8.330075935761e-07 227 KSP Residual norm 7.965111975570e-07 228 KSP Residual norm 7.682579007776e-07 229 KSP Residual norm 7.461318867221e-07 230 KSP Residual norm 7.317204257821e-07 231 KSP Residual norm 7.218875505228e-07 232 KSP Residual norm 7.053200253465e-07 233 KSP Residual norm 6.749619371725e-07 234 KSP Residual norm 6.441685200577e-07 235 KSP Residual norm 6.167053716769e-07 236 KSP Residual norm 5.920872320683e-07 237 KSP Residual norm 5.686793194121e-07 238 KSP Residual norm 5.462107461517e-07 239 KSP Residual norm 5.226925592644e-07 240 KSP Residual norm 4.982797621843e-07 241 KSP Residual norm 4.743678684407e-07 242 KSP Residual norm 4.544781353180e-07 243 KSP Residual norm 4.363622527373e-07 244 KSP Residual norm 4.186383153861e-07 245 KSP Residual norm 4.022766314226e-07 246 KSP Residual norm 3.898933634039e-07 247 KSP Residual norm 3.792600454230e-07 248 KSP Residual norm 3.684900291566e-07 249 KSP Residual norm 3.565022461888e-07 250 KSP Residual norm 3.431770666818e-07 251 KSP Residual norm 3.302406582832e-07 252 KSP Residual norm 3.197149781660e-07 253 KSP Residual norm 3.117421741438e-07 254 KSP Residual norm 3.006225990591e-07 255 KSP Residual norm 2.865389872395e-07 256 KSP Residual norm 2.747782067735e-07 257 KSP Residual norm 2.665469379328e-07 258 KSP Residual norm 2.617538762481e-07 259 KSP Residual norm 2.583563189275e-07 260 KSP Residual norm 2.447894421918e-07 261 KSP Residual norm 2.297027848512e-07 262 KSP Residual norm 2.200448801799e-07 263 KSP Residual norm 2.131922078483e-07 264 KSP Residual norm 2.067933715855e-07 265 KSP Residual norm 2.000156041564e-07 266 KSP Residual norm 1.938505600216e-07 267 KSP Residual norm 1.879074457881e-07 268 KSP Residual norm 1.823521932855e-07 269 KSP Residual norm 1.781856414303e-07 270 KSP Residual norm 1.748280158607e-07 271 KSP Residual norm 1.691927616841e-07 272 KSP Residual norm 1.614469891728e-07 273 KSP Residual norm 1.547742809330e-07 274 KSP Residual norm 1.496700363940e-07 275 KSP Residual norm 1.452954047765e-07 276 KSP Residual norm 1.418567892160e-07 277 KSP Residual norm 1.381629446351e-07 278 KSP Residual norm 1.338180116960e-07 279 KSP Residual norm 1.294911570960e-07 280 KSP Residual norm 1.252845407348e-07 281 KSP Residual norm 1.221451573536e-07 282 KSP Residual norm 1.200008830870e-07 283 KSP Residual norm 1.181353127193e-07 284 KSP Residual norm 1.157260629549e-07 285 KSP Residual norm 1.136460188017e-07 286 KSP Residual norm 1.122147871517e-07 287 KSP Residual norm 1.100027941554e-07 288 KSP Residual norm 1.060585633240e-07 289 KSP Residual norm 1.017820421500e-07 290 KSP Residual norm 9.792097513992e-08 291 KSP Residual norm 9.409077619432e-08 292 KSP Residual norm 9.083025782826e-08 293 KSP Residual norm 8.828399818491e-08 294 KSP Residual norm 8.584654576791e-08 295 KSP Residual norm 8.391247328767e-08 296 KSP Residual norm 8.252848185233e-08 297 KSP Residual norm 8.066160372293e-08 298 KSP Residual norm 7.790988802915e-08 299 KSP Residual norm 7.505064481543e-08 300 KSP Residual norm 7.260965311959e-08 301 KSP Residual norm 7.078410350745e-08 302 KSP Residual norm 6.966834300873e-08 303 KSP Residual norm 6.866569305804e-08 304 KSP Residual norm 6.694877688774e-08 305 KSP Residual norm 6.519839292297e-08 306 KSP Residual norm 6.419812071243e-08 307 KSP Residual norm 6.298188074213e-08 308 KSP Residual norm 6.108083386866e-08 309 KSP Residual norm 5.959025865587e-08 310 KSP Residual norm 5.908859885073e-08 311 KSP Residual norm 5.858069554233e-08 312 KSP Residual norm 5.675183947550e-08 313 KSP Residual norm 5.490389084370e-08 314 KSP Residual norm 5.408579956588e-08 315 KSP Residual norm 5.336934580319e-08 316 KSP Residual norm 5.170876571564e-08 317 KSP Residual norm 4.929015477465e-08 318 KSP Residual norm 4.720748314983e-08 319 KSP Residual norm 4.566280884465e-08 320 KSP Residual norm 4.455941308326e-08 321 KSP Residual norm 4.441806406993e-08 322 KSP Residual norm 4.553834017569e-08 323 KSP Residual norm 4.375224764630e-08 324 KSP Residual norm 4.078587826173e-08 325 KSP Residual norm 3.915291307415e-08 326 KSP Residual norm 3.823960595944e-08 327 KSP Residual norm 3.774570497566e-08 328 KSP Residual norm 3.753993322570e-08 329 KSP Residual norm 3.662896670038e-08 330 KSP Residual norm 3.483343324942e-08 331 KSP Residual norm 3.325115758550e-08 332 KSP Residual norm 3.205598865855e-08 333 KSP Residual norm 3.105696040803e-08 334 KSP Residual norm 3.022988852886e-08 335 KSP Residual norm 2.955466023448e-08 336 KSP Residual norm 2.883209474985e-08 337 KSP Residual norm 2.801721753877e-08 338 KSP Residual norm 2.703840301284e-08 339 KSP Residual norm 2.609078699815e-08 340 KSP Residual norm 2.534121923176e-08 341 KSP Residual norm 2.469264746508e-08 342 KSP Residual norm 2.402636451202e-08 343 KSP Residual norm 2.329960424155e-08 344 KSP Residual norm 2.252352276917e-08 345 KSP Residual norm 2.179797884938e-08 346 KSP Residual norm 2.119799024058e-08 347 KSP Residual norm 2.053209505370e-08 348 KSP Residual norm 1.981148839838e-08 349 KSP Residual norm 1.904946966285e-08 350 KSP Residual norm 1.827975936672e-08 351 KSP Residual norm 1.755139153092e-08 352 KSP Residual norm 1.689446857824e-08 353 KSP Residual norm 1.649812625172e-08 354 KSP Residual norm 1.637305353833e-08 355 KSP Residual norm 1.587094147753e-08 356 KSP Residual norm 1.493062207489e-08 357 KSP Residual norm 1.419019253611e-08 358 KSP Residual norm 1.355068918627e-08 359 KSP Residual norm 1.303517734133e-08 360 KSP Residual norm 1.265061510376e-08 361 KSP Residual norm 1.233810916111e-08 362 KSP Residual norm 1.214518910553e-08 363 KSP Residual norm 1.193644280481e-08 364 KSP Residual norm 1.152553701870e-08 365 KSP Residual norm 1.100617261515e-08 Linear tao_nls_ solve converged due to CONVERGED_RTOL iterations 365 0 LS Function value: 0.06919, Step length: 0. 1 LS Function value: 0.0691413, Step length: 1. stx: 0., fx: 0.06919, dgx: -8.45693e-05 sty: 0., fy: 0.06919, dgy: -8.45693e-05 23 TAO, Function value: 0.0691413, Residual: 0.000389828 Residual norms for tao_nls_ solve. 0 KSP Residual norm 3.898281966765e-04 1 KSP Residual norm 4.260782235264e-04 2 KSP Residual norm 3.525588995984e-04 3 KSP Residual norm 3.154245120592e-04 4 KSP Residual norm 3.348602211243e-04 5 KSP Residual norm 3.227233898165e-04 6 KSP Residual norm 3.200878544641e-04 7 KSP Residual norm 3.255916931231e-04 8 KSP Residual norm 3.108545045604e-04 9 KSP Residual norm 2.926995326680e-04 10 KSP Residual norm 2.789969235013e-04 11 KSP Residual norm 2.597566072560e-04 12 KSP Residual norm 2.404183669778e-04 13 KSP Residual norm 2.236032695600e-04 14 KSP Residual norm 2.100254026242e-04 15 KSP Residual norm 1.936777805942e-04 16 KSP Residual norm 1.805249509742e-04 17 KSP Residual norm 1.718161610678e-04 18 KSP Residual norm 1.671335811434e-04 19 KSP Residual norm 1.640579575433e-04 20 KSP Residual norm 1.614395385096e-04 21 KSP Residual norm 1.622065218921e-04 22 KSP Residual norm 1.666757832164e-04 23 KSP Residual norm 1.725642244254e-04 24 KSP Residual norm 1.792082422207e-04 25 KSP Residual norm 1.864647756656e-04 26 KSP Residual norm 1.954865815390e-04 27 KSP Residual norm 2.068298103354e-04 28 KSP Residual norm 2.180784542754e-04 29 KSP Residual norm 2.278267330698e-04 30 KSP Residual norm 2.370889229720e-04 31 KSP Residual norm 2.447598206004e-04 32 KSP Residual norm 2.510916656310e-04 33 KSP Residual norm 2.565793685585e-04 34 KSP Residual norm 2.607051706211e-04 35 KSP Residual norm 2.606507439807e-04 36 KSP Residual norm 2.550953401501e-04 37 KSP Residual norm 2.465528545191e-04 38 KSP Residual norm 2.368096609144e-04 39 KSP Residual norm 2.256212163176e-04 40 KSP Residual norm 2.126665766128e-04 41 KSP Residual norm 1.992393499635e-04 42 KSP Residual norm 1.858617087795e-04 43 KSP Residual norm 1.732149543162e-04 44 KSP Residual norm 1.611977791226e-04 45 KSP Residual norm 1.495436218975e-04 46 KSP Residual norm 1.386381420233e-04 47 KSP Residual norm 1.283138469425e-04 48 KSP Residual norm 1.187114505932e-04 49 KSP Residual norm 1.100520115297e-04 50 KSP Residual norm 1.031216424229e-04 51 KSP Residual norm 9.753354909725e-05 52 KSP Residual norm 9.179483012318e-05 53 KSP Residual norm 8.628216475328e-05 54 KSP Residual norm 8.187093724669e-05 55 KSP Residual norm 7.853382161867e-05 56 KSP Residual norm 7.538950805364e-05 57 KSP Residual norm 7.263967771645e-05 58 KSP Residual norm 7.053600795314e-05 59 KSP Residual norm 6.858608656995e-05 60 KSP Residual norm 6.631437008542e-05 61 KSP Residual norm 6.378023166633e-05 62 KSP Residual norm 6.134910011810e-05 63 KSP Residual norm 5.842907476675e-05 64 KSP Residual norm 5.491638038659e-05 65 KSP Residual norm 5.137123921628e-05 66 KSP Residual norm 4.815466755534e-05 67 KSP Residual norm 4.546605646306e-05 68 KSP Residual norm 4.308577446980e-05 69 KSP Residual norm 4.084583525172e-05 70 KSP Residual norm 3.872304273041e-05 71 KSP Residual norm 3.684538623351e-05 72 KSP Residual norm 3.514826984255e-05 73 KSP Residual norm 3.352160920459e-05 74 KSP Residual norm 3.210195828274e-05 75 KSP Residual norm 3.083564039882e-05 76 KSP Residual norm 2.975873642305e-05 77 KSP Residual norm 2.888535385715e-05 78 KSP Residual norm 2.807651340353e-05 79 KSP Residual norm 2.726578610252e-05 80 KSP Residual norm 2.645566758288e-05 81 KSP Residual norm 2.561561208184e-05 82 KSP Residual norm 2.476008839651e-05 83 KSP Residual norm 2.396151562022e-05 84 KSP Residual norm 2.322556689482e-05 85 KSP Residual norm 2.265247424064e-05 86 KSP Residual norm 2.215470308987e-05 87 KSP Residual norm 2.156464554636e-05 88 KSP Residual norm 2.072003665699e-05 89 KSP Residual norm 1.990783313888e-05 90 KSP Residual norm 1.919556311154e-05 91 KSP Residual norm 1.864385885849e-05 92 KSP Residual norm 1.799757777651e-05 93 KSP Residual norm 1.735140238653e-05 94 KSP Residual norm 1.680659717844e-05 95 KSP Residual norm 1.644279486971e-05 96 KSP Residual norm 1.617619291910e-05 97 KSP Residual norm 1.586339077462e-05 98 KSP Residual norm 1.543088717396e-05 99 KSP Residual norm 1.498142787676e-05 100 KSP Residual norm 1.453451197974e-05 101 KSP Residual norm 1.407944389213e-05 102 KSP Residual norm 1.350361293292e-05 103 KSP Residual norm 1.306104539411e-05 104 KSP Residual norm 1.243281729264e-05 105 KSP Residual norm 1.198844345745e-05 106 KSP Residual norm 1.151042422710e-05 107 KSP Residual norm 1.139511530876e-05 108 KSP Residual norm 1.106260456727e-05 109 KSP Residual norm 1.050540682962e-05 110 KSP Residual norm 9.923802882570e-06 111 KSP Residual norm 9.569793032572e-06 112 KSP Residual norm 9.249356035945e-06 113 KSP Residual norm 8.969651535602e-06 114 KSP Residual norm 8.731548066878e-06 115 KSP Residual norm 8.552245962072e-06 116 KSP Residual norm 8.336880831734e-06 117 KSP Residual norm 8.152589114403e-06 118 KSP Residual norm 7.939725698757e-06 119 KSP Residual norm 7.800281890942e-06 120 KSP Residual norm 7.709171954693e-06 121 KSP Residual norm 7.660853613246e-06 122 KSP Residual norm 7.562876725213e-06 123 KSP Residual norm 7.451984865938e-06 124 KSP Residual norm 7.315004153865e-06 125 KSP Residual norm 7.172432069242e-06 126 KSP Residual norm 7.050531417930e-06 127 KSP Residual norm 6.902285946252e-06 128 KSP Residual norm 6.706944077121e-06 129 KSP Residual norm 6.489880201453e-06 130 KSP Residual norm 6.297649961735e-06 131 KSP Residual norm 6.100989515109e-06 132 KSP Residual norm 5.918015549613e-06 133 KSP Residual norm 5.700954025688e-06 134 KSP Residual norm 5.484355051094e-06 135 KSP Residual norm 5.305302823136e-06 136 KSP Residual norm 5.148878576909e-06 137 KSP Residual norm 4.921064859024e-06 138 KSP Residual norm 4.640792625906e-06 139 KSP Residual norm 4.349486542279e-06 140 KSP Residual norm 4.074142506049e-06 141 KSP Residual norm 3.826975016777e-06 142 KSP Residual norm 3.590147019382e-06 143 KSP Residual norm 3.369973412422e-06 144 KSP Residual norm 3.162163421457e-06 145 KSP Residual norm 2.967952535137e-06 146 KSP Residual norm 2.791123153509e-06 147 KSP Residual norm 2.622503909049e-06 148 KSP Residual norm 2.481300518722e-06 149 KSP Residual norm 2.351042003146e-06 150 KSP Residual norm 2.237423253898e-06 151 KSP Residual norm 2.149792731039e-06 152 KSP Residual norm 2.075050583349e-06 153 KSP Residual norm 1.989922921554e-06 154 KSP Residual norm 1.905512245285e-06 155 KSP Residual norm 1.838704455775e-06 156 KSP Residual norm 1.772550603921e-06 157 KSP Residual norm 1.711020509169e-06 158 KSP Residual norm 1.662121957024e-06 159 KSP Residual norm 1.618959868894e-06 160 KSP Residual norm 1.598333531334e-06 161 KSP Residual norm 1.604891182144e-06 162 KSP Residual norm 1.602036588382e-06 163 KSP Residual norm 1.551043995876e-06 164 KSP Residual norm 1.502018951845e-06 165 KSP Residual norm 1.482260261774e-06 166 KSP Residual norm 1.476456080491e-06 167 KSP Residual norm 1.466922054531e-06 168 KSP Residual norm 1.451757384949e-06 169 KSP Residual norm 1.436994994184e-06 170 KSP Residual norm 1.424614901288e-06 171 KSP Residual norm 1.407985049212e-06 172 KSP Residual norm 1.378746013242e-06 173 KSP Residual norm 1.344118671978e-06 174 KSP Residual norm 1.305902837320e-06 175 KSP Residual norm 1.263646207007e-06 176 KSP Residual norm 1.220779074086e-06 177 KSP Residual norm 1.175826071484e-06 178 KSP Residual norm 1.135488241104e-06 179 KSP Residual norm 1.093659311449e-06 180 KSP Residual norm 1.049479417276e-06 181 KSP Residual norm 1.003554697133e-06 182 KSP Residual norm 9.548575416652e-07 183 KSP Residual norm 9.117839137442e-07 184 KSP Residual norm 8.763848087774e-07 185 KSP Residual norm 8.453236181879e-07 186 KSP Residual norm 8.064902440746e-07 187 KSP Residual norm 7.634786146138e-07 188 KSP Residual norm 7.284431911976e-07 189 KSP Residual norm 7.000993347216e-07 190 KSP Residual norm 6.765452640047e-07 191 KSP Residual norm 6.476785794558e-07 192 KSP Residual norm 6.147940610392e-07 193 KSP Residual norm 5.873266361796e-07 194 KSP Residual norm 5.689309479644e-07 195 KSP Residual norm 5.567775268249e-07 196 KSP Residual norm 5.457200724331e-07 197 KSP Residual norm 5.332536335494e-07 198 KSP Residual norm 5.191111624975e-07 199 KSP Residual norm 5.040992868663e-07 200 KSP Residual norm 4.920247764236e-07 201 KSP Residual norm 4.819808459276e-07 202 KSP Residual norm 4.720950507737e-07 203 KSP Residual norm 4.610121064467e-07 204 KSP Residual norm 4.479361302596e-07 205 KSP Residual norm 4.351127506379e-07 206 KSP Residual norm 4.261698853089e-07 207 KSP Residual norm 4.224433066789e-07 208 KSP Residual norm 4.192224560813e-07 209 KSP Residual norm 4.098856214893e-07 210 KSP Residual norm 4.003532973939e-07 211 KSP Residual norm 3.899874878041e-07 212 KSP Residual norm 3.788836855235e-07 213 KSP Residual norm 3.653957322768e-07 214 KSP Residual norm 3.528609776264e-07 215 KSP Residual norm 3.410190291363e-07 216 KSP Residual norm 3.302573088720e-07 217 KSP Residual norm 3.188514372547e-07 218 KSP Residual norm 3.074368315911e-07 219 KSP Residual norm 2.966405355714e-07 220 KSP Residual norm 2.863627157990e-07 221 KSP Residual norm 2.788118201307e-07 222 KSP Residual norm 2.745207779498e-07 223 KSP Residual norm 2.716166115827e-07 224 KSP Residual norm 2.652918365757e-07 225 KSP Residual norm 2.550987849297e-07 226 KSP Residual norm 2.449051603692e-07 227 KSP Residual norm 2.362563659602e-07 228 KSP Residual norm 2.293139033846e-07 229 KSP Residual norm 2.251136645984e-07 230 KSP Residual norm 2.227155306074e-07 231 KSP Residual norm 2.151054167176e-07 232 KSP Residual norm 2.037200433523e-07 233 KSP Residual norm 1.948628440686e-07 234 KSP Residual norm 1.895395425916e-07 235 KSP Residual norm 1.862665697368e-07 236 KSP Residual norm 1.796951923480e-07 237 KSP Residual norm 1.707631959957e-07 238 KSP Residual norm 1.639026721660e-07 239 KSP Residual norm 1.573249465123e-07 240 KSP Residual norm 1.501999960264e-07 241 KSP Residual norm 1.433870769632e-07 242 KSP Residual norm 1.374091233882e-07 243 KSP Residual norm 1.314984527665e-07 244 KSP Residual norm 1.258866493624e-07 245 KSP Residual norm 1.205266011077e-07 246 KSP Residual norm 1.155113448174e-07 247 KSP Residual norm 1.108247675954e-07 248 KSP Residual norm 1.063171582120e-07 249 KSP Residual norm 1.019798342140e-07 250 KSP Residual norm 9.711612226865e-08 251 KSP Residual norm 9.225421774003e-08 252 KSP Residual norm 8.791335152430e-08 253 KSP Residual norm 8.445235801725e-08 254 KSP Residual norm 8.155677556189e-08 255 KSP Residual norm 7.862322167000e-08 256 KSP Residual norm 7.508628648836e-08 257 KSP Residual norm 7.161066431791e-08 258 KSP Residual norm 6.869815222681e-08 259 KSP Residual norm 6.614664193412e-08 260 KSP Residual norm 6.372245117887e-08 261 KSP Residual norm 6.115115919491e-08 262 KSP Residual norm 5.881546845897e-08 263 KSP Residual norm 5.678498179911e-08 264 KSP Residual norm 5.509657813628e-08 265 KSP Residual norm 5.366305504363e-08 266 KSP Residual norm 5.227108569537e-08 267 KSP Residual norm 5.039548345879e-08 268 KSP Residual norm 4.811139744390e-08 269 KSP Residual norm 4.608229410735e-08 270 KSP Residual norm 4.450402524000e-08 271 KSP Residual norm 4.329343218860e-08 272 KSP Residual norm 4.231645138082e-08 273 KSP Residual norm 4.122840486367e-08 274 KSP Residual norm 4.029622689483e-08 275 KSP Residual norm 3.989085580379e-08 276 KSP Residual norm 4.025099760073e-08 277 KSP Residual norm 4.057448469746e-08 278 KSP Residual norm 3.998868415826e-08 279 KSP Residual norm 3.924518412760e-08 280 KSP Residual norm 3.877631199775e-08 281 KSP Residual norm 3.828649159804e-08 282 KSP Residual norm 3.772177025126e-08 283 KSP Residual norm 3.723390658925e-08 284 KSP Residual norm 3.681662988980e-08 285 KSP Residual norm 3.642232192192e-08 286 KSP Residual norm 3.595460115934e-08 287 KSP Residual norm 3.532660592330e-08 288 KSP Residual norm 3.467229987434e-08 289 KSP Residual norm 3.425098861596e-08 290 KSP Residual norm 3.382859925495e-08 291 KSP Residual norm 3.272254071820e-08 292 KSP Residual norm 3.159897496630e-08 293 KSP Residual norm 3.099830572543e-08 294 KSP Residual norm 3.069458163886e-08 295 KSP Residual norm 2.999994770743e-08 296 KSP Residual norm 2.873730715789e-08 297 KSP Residual norm 2.745438793752e-08 298 KSP Residual norm 2.628436769098e-08 299 KSP Residual norm 2.522290033571e-08 300 KSP Residual norm 2.434466522422e-08 301 KSP Residual norm 2.357979630080e-08 302 KSP Residual norm 2.298724743201e-08 303 KSP Residual norm 2.258309215111e-08 304 KSP Residual norm 2.216246314543e-08 305 KSP Residual norm 2.123228971323e-08 306 KSP Residual norm 2.004822457030e-08 307 KSP Residual norm 1.910282311443e-08 308 KSP Residual norm 1.842412305820e-08 309 KSP Residual norm 1.793021931929e-08 310 KSP Residual norm 1.745501114677e-08 311 KSP Residual norm 1.701660894876e-08 312 KSP Residual norm 1.671936926765e-08 313 KSP Residual norm 1.646864912197e-08 314 KSP Residual norm 1.621505486300e-08 315 KSP Residual norm 1.598750539343e-08 316 KSP Residual norm 1.574298393330e-08 317 KSP Residual norm 1.546453840427e-08 318 KSP Residual norm 1.507022226293e-08 319 KSP Residual norm 1.467955120116e-08 320 KSP Residual norm 1.430865106359e-08 321 KSP Residual norm 1.403474714095e-08 322 KSP Residual norm 1.375648423156e-08 323 KSP Residual norm 1.352226519817e-08 324 KSP Residual norm 1.332856735609e-08 325 KSP Residual norm 1.299779043530e-08 326 KSP Residual norm 1.245372080308e-08 327 KSP Residual norm 1.190284042934e-08 328 KSP Residual norm 1.146089554134e-08 329 KSP Residual norm 1.110265416383e-08 330 KSP Residual norm 1.076819154460e-08 331 KSP Residual norm 1.050441720500e-08 332 KSP Residual norm 1.029079116154e-08 333 KSP Residual norm 1.013276118066e-08 334 KSP Residual norm 9.907342265802e-09 335 KSP Residual norm 9.526639683167e-09 336 KSP Residual norm 9.143902802023e-09 337 KSP Residual norm 8.855927587539e-09 338 KSP Residual norm 8.609692475267e-09 339 KSP Residual norm 8.372049188436e-09 340 KSP Residual norm 8.205268848155e-09 341 KSP Residual norm 8.111926692189e-09 342 KSP Residual norm 8.017477344214e-09 343 KSP Residual norm 7.791491464387e-09 344 KSP Residual norm 7.515900475468e-09 345 KSP Residual norm 7.273701758829e-09 346 KSP Residual norm 7.041856145086e-09 347 KSP Residual norm 6.782344645633e-09 348 KSP Residual norm 6.534436936293e-09 349 KSP Residual norm 6.331824728641e-09 350 KSP Residual norm 6.122563319983e-09 351 KSP Residual norm 5.906364443527e-09 352 KSP Residual norm 5.670605930012e-09 353 KSP Residual norm 5.441953958170e-09 354 KSP Residual norm 5.238521929389e-09 355 KSP Residual norm 5.075404447809e-09 356 KSP Residual norm 4.919978357104e-09 357 KSP Residual norm 4.714768946614e-09 358 KSP Residual norm 4.509305200417e-09 359 KSP Residual norm 4.340767241167e-09 360 KSP Residual norm 4.182717828382e-09 361 KSP Residual norm 4.038445446197e-09 362 KSP Residual norm 3.875618562620e-09 Linear tao_nls_ solve converged due to CONVERGED_RTOL iterations 362 0 LS Function value: 0.0691413, Step length: 0. 1 LS Function value: 0.0691394, Step length: 1. stx: 0., fx: 0.0691413, dgx: -3.36905e-06 sty: 0., fy: 0.0691413, dgy: -3.36905e-06 24 TAO, Function value: 0.0691394, Residual: 0.000136972 Residual norms for tao_nls_ solve. 0 KSP Residual norm 1.369715054972e-04 1 KSP Residual norm 1.427374650777e-04 2 KSP Residual norm 1.170547361542e-04 3 KSP Residual norm 1.200696079248e-04 4 KSP Residual norm 1.216593491442e-04 5 KSP Residual norm 1.213490903396e-04 6 KSP Residual norm 1.234258500040e-04 7 KSP Residual norm 1.258755093439e-04 8 KSP Residual norm 1.249877912703e-04 9 KSP Residual norm 1.234353435612e-04 10 KSP Residual norm 1.208435951779e-04 11 KSP Residual norm 1.161632999145e-04 12 KSP Residual norm 1.085833045757e-04 13 KSP Residual norm 1.004411572208e-04 14 KSP Residual norm 9.194947798856e-05 15 KSP Residual norm 8.256638946919e-05 16 KSP Residual norm 7.399752031762e-05 17 KSP Residual norm 6.661084091562e-05 18 KSP Residual norm 5.987395651737e-05 19 KSP Residual norm 5.334615633229e-05 20 KSP Residual norm 4.743016698481e-05 21 KSP Residual norm 4.236207237447e-05 22 KSP Residual norm 3.858375615320e-05 23 KSP Residual norm 3.563311414780e-05 24 KSP Residual norm 3.329759493202e-05 25 KSP Residual norm 3.170121767725e-05 26 KSP Residual norm 3.028575265084e-05 27 KSP Residual norm 2.912009196816e-05 28 KSP Residual norm 2.852652255743e-05 29 KSP Residual norm 2.844776382644e-05 30 KSP Residual norm 2.890731289856e-05 31 KSP Residual norm 3.010120133699e-05 32 KSP Residual norm 3.145161758921e-05 33 KSP Residual norm 3.224962621763e-05 34 KSP Residual norm 3.327879854930e-05 35 KSP Residual norm 3.446215286211e-05 36 KSP Residual norm 3.556123407045e-05 37 KSP Residual norm 3.670162266487e-05 38 KSP Residual norm 3.764180198464e-05 39 KSP Residual norm 3.836715835986e-05 40 KSP Residual norm 3.870554194276e-05 41 KSP Residual norm 3.853372448602e-05 42 KSP Residual norm 3.805620207279e-05 43 KSP Residual norm 3.733503549287e-05 44 KSP Residual norm 3.647963431672e-05 45 KSP Residual norm 3.532962795120e-05 46 KSP Residual norm 3.412269434590e-05 47 KSP Residual norm 3.299626226972e-05 48 KSP Residual norm 3.175489960804e-05 49 KSP Residual norm 3.042748439495e-05 50 KSP Residual norm 2.894164191152e-05 51 KSP Residual norm 2.744148870372e-05 52 KSP Residual norm 2.601078560028e-05 53 KSP Residual norm 2.476287622655e-05 54 KSP Residual norm 2.367278429871e-05 55 KSP Residual norm 2.256253180915e-05 56 KSP Residual norm 2.148821515202e-05 57 KSP Residual norm 2.052703577106e-05 58 KSP Residual norm 1.980182711842e-05 59 KSP Residual norm 1.917813966260e-05 60 KSP Residual norm 1.864817658151e-05 61 KSP Residual norm 1.821740092310e-05 62 KSP Residual norm 1.781240131109e-05 63 KSP Residual norm 1.734484106761e-05 64 KSP Residual norm 1.677873731902e-05 65 KSP Residual norm 1.614966960403e-05 66 KSP Residual norm 1.547215927428e-05 67 KSP Residual norm 1.479582004441e-05 68 KSP Residual norm 1.407017320473e-05 69 KSP Residual norm 1.329802211564e-05 70 KSP Residual norm 1.249580137316e-05 71 KSP Residual norm 1.168051340910e-05 72 KSP Residual norm 1.087744395060e-05 73 KSP Residual norm 1.011388512913e-05 74 KSP Residual norm 9.418978654494e-06 75 KSP Residual norm 8.819479022617e-06 76 KSP Residual norm 8.265837847848e-06 77 KSP Residual norm 7.674730265840e-06 78 KSP Residual norm 7.100341241784e-06 79 KSP Residual norm 6.619840099996e-06 80 KSP Residual norm 6.208795360303e-06 81 KSP Residual norm 5.841758912721e-06 82 KSP Residual norm 5.567604588460e-06 83 KSP Residual norm 5.403394121212e-06 84 KSP Residual norm 5.234240584307e-06 85 KSP Residual norm 4.962485687155e-06 86 KSP Residual norm 4.696253851608e-06 87 KSP Residual norm 4.432146974696e-06 88 KSP Residual norm 4.177281676427e-06 89 KSP Residual norm 3.942005452610e-06 90 KSP Residual norm 3.726896469123e-06 91 KSP Residual norm 3.550138793012e-06 92 KSP Residual norm 3.344827364759e-06 93 KSP Residual norm 3.131137013093e-06 94 KSP Residual norm 2.926565783011e-06 95 KSP Residual norm 2.759533734007e-06 96 KSP Residual norm 2.621350351432e-06 97 KSP Residual norm 2.504961062890e-06 98 KSP Residual norm 2.379175814744e-06 99 KSP Residual norm 2.270895827931e-06 100 KSP Residual norm 2.172200130831e-06 101 KSP Residual norm 2.091883825186e-06 102 KSP Residual norm 2.012143260581e-06 103 KSP Residual norm 1.943592993541e-06 104 KSP Residual norm 1.880339568256e-06 105 KSP Residual norm 1.832101695176e-06 106 KSP Residual norm 1.799196174418e-06 107 KSP Residual norm 1.770990096764e-06 108 KSP Residual norm 1.748270917788e-06 109 KSP Residual norm 1.756664809371e-06 110 KSP Residual norm 1.796687651648e-06 111 KSP Residual norm 1.797453718646e-06 112 KSP Residual norm 1.749611522865e-06 113 KSP Residual norm 1.723873964103e-06 114 KSP Residual norm 1.712630285295e-06 115 KSP Residual norm 1.714044749972e-06 116 KSP Residual norm 1.725129317033e-06 117 KSP Residual norm 1.746559653468e-06 118 KSP Residual norm 1.776898180772e-06 119 KSP Residual norm 1.819831664322e-06 120 KSP Residual norm 1.851519336954e-06 121 KSP Residual norm 1.856390623101e-06 122 KSP Residual norm 1.854949458719e-06 123 KSP Residual norm 1.863318793463e-06 124 KSP Residual norm 1.871196736244e-06 125 KSP Residual norm 1.878953713288e-06 126 KSP Residual norm 1.892344936557e-06 127 KSP Residual norm 1.900312148413e-06 128 KSP Residual norm 1.897429036721e-06 129 KSP Residual norm 1.874346070186e-06 130 KSP Residual norm 1.825339281808e-06 131 KSP Residual norm 1.772200400485e-06 132 KSP Residual norm 1.711224945342e-06 133 KSP Residual norm 1.635747750196e-06 134 KSP Residual norm 1.534131234239e-06 135 KSP Residual norm 1.430590558691e-06 136 KSP Residual norm 1.339901856433e-06 137 KSP Residual norm 1.260318178102e-06 138 KSP Residual norm 1.178697473840e-06 139 KSP Residual norm 1.102563482531e-06 140 KSP Residual norm 1.039365202406e-06 141 KSP Residual norm 9.842153889473e-07 142 KSP Residual norm 9.359309421927e-07 143 KSP Residual norm 8.910723185465e-07 144 KSP Residual norm 8.533137468572e-07 145 KSP Residual norm 8.099814580532e-07 146 KSP Residual norm 7.580002657785e-07 147 KSP Residual norm 7.119211173463e-07 148 KSP Residual norm 6.774617214522e-07 149 KSP Residual norm 6.477832209936e-07 150 KSP Residual norm 6.143074168231e-07 151 KSP Residual norm 5.815192028173e-07 152 KSP Residual norm 5.549971160383e-07 153 KSP Residual norm 5.367318003887e-07 154 KSP Residual norm 5.180966839589e-07 155 KSP Residual norm 4.865277850619e-07 156 KSP Residual norm 4.564395085916e-07 157 KSP Residual norm 4.343469869975e-07 158 KSP Residual norm 4.177791047205e-07 159 KSP Residual norm 4.039132797722e-07 160 KSP Residual norm 3.922271187426e-07 161 KSP Residual norm 3.825407597701e-07 162 KSP Residual norm 3.717440509350e-07 163 KSP Residual norm 3.611649068876e-07 164 KSP Residual norm 3.523367109002e-07 165 KSP Residual norm 3.453198978467e-07 166 KSP Residual norm 3.393506213586e-07 167 KSP Residual norm 3.324701010951e-07 168 KSP Residual norm 3.252392195117e-07 169 KSP Residual norm 3.171999817490e-07 170 KSP Residual norm 3.093676385616e-07 171 KSP Residual norm 3.011488965549e-07 172 KSP Residual norm 2.915123959489e-07 173 KSP Residual norm 2.810514880365e-07 174 KSP Residual norm 2.684156279695e-07 175 KSP Residual norm 2.547119462362e-07 176 KSP Residual norm 2.407159360858e-07 177 KSP Residual norm 2.283107761335e-07 178 KSP Residual norm 2.170842300700e-07 179 KSP Residual norm 2.075437520876e-07 180 KSP Residual norm 1.983573279902e-07 181 KSP Residual norm 1.888033501108e-07 182 KSP Residual norm 1.787232851984e-07 183 KSP Residual norm 1.698457405803e-07 184 KSP Residual norm 1.614272552660e-07 185 KSP Residual norm 1.537096831077e-07 186 KSP Residual norm 1.456389957360e-07 187 KSP Residual norm 1.398050748801e-07 188 KSP Residual norm 1.343965533620e-07 189 KSP Residual norm 1.280792062064e-07 190 KSP Residual norm 1.200110375020e-07 191 KSP Residual norm 1.119292466282e-07 192 KSP Residual norm 1.052047795990e-07 193 KSP Residual norm 9.908226087476e-08 194 KSP Residual norm 9.388059474020e-08 195 KSP Residual norm 8.963675334158e-08 196 KSP Residual norm 8.747340117899e-08 197 KSP Residual norm 8.581797288820e-08 198 KSP Residual norm 8.046986696708e-08 199 KSP Residual norm 7.522396600576e-08 200 KSP Residual norm 7.174712079691e-08 201 KSP Residual norm 6.901102011616e-08 202 KSP Residual norm 6.690569048555e-08 203 KSP Residual norm 6.510902908886e-08 204 KSP Residual norm 6.340414883560e-08 205 KSP Residual norm 6.116468862149e-08 206 KSP Residual norm 5.862959724903e-08 207 KSP Residual norm 5.665736759325e-08 208 KSP Residual norm 5.549209175987e-08 209 KSP Residual norm 5.429877181257e-08 210 KSP Residual norm 5.230105398757e-08 211 KSP Residual norm 5.036885645456e-08 212 KSP Residual norm 4.904966365541e-08 213 KSP Residual norm 4.795070243980e-08 214 KSP Residual norm 4.653891431113e-08 215 KSP Residual norm 4.514313341263e-08 216 KSP Residual norm 4.385579403783e-08 217 KSP Residual norm 4.275436735279e-08 218 KSP Residual norm 4.165737279682e-08 219 KSP Residual norm 4.043296916036e-08 220 KSP Residual norm 3.910118893599e-08 221 KSP Residual norm 3.783103170977e-08 222 KSP Residual norm 3.638741512560e-08 223 KSP Residual norm 3.471307015352e-08 224 KSP Residual norm 3.339447669209e-08 225 KSP Residual norm 3.236869920598e-08 226 KSP Residual norm 3.111706620247e-08 227 KSP Residual norm 2.986271936933e-08 228 KSP Residual norm 2.891977629545e-08 229 KSP Residual norm 2.815809908620e-08 230 KSP Residual norm 2.742180957835e-08 231 KSP Residual norm 2.667857103060e-08 232 KSP Residual norm 2.585244712323e-08 233 KSP Residual norm 2.496464673978e-08 234 KSP Residual norm 2.405981619653e-08 235 KSP Residual norm 2.314965502315e-08 236 KSP Residual norm 2.231458434452e-08 237 KSP Residual norm 2.156337873227e-08 238 KSP Residual norm 2.077769039153e-08 239 KSP Residual norm 1.998905260073e-08 240 KSP Residual norm 1.923918600966e-08 241 KSP Residual norm 1.856551607786e-08 242 KSP Residual norm 1.798385039494e-08 243 KSP Residual norm 1.749972110971e-08 244 KSP Residual norm 1.717243983139e-08 245 KSP Residual norm 1.700011898060e-08 246 KSP Residual norm 1.675575289178e-08 247 KSP Residual norm 1.633336366485e-08 248 KSP Residual norm 1.584218901373e-08 249 KSP Residual norm 1.534001951699e-08 250 KSP Residual norm 1.487462095113e-08 251 KSP Residual norm 1.443960975088e-08 252 KSP Residual norm 1.399211952487e-08 253 KSP Residual norm 1.354991892573e-08 254 KSP Residual norm 1.319014863969e-08 255 KSP Residual norm 1.284389230347e-08 256 KSP Residual norm 1.245531765808e-08 257 KSP Residual norm 1.197002734106e-08 258 KSP Residual norm 1.140409133049e-08 259 KSP Residual norm 1.087632592303e-08 260 KSP Residual norm 1.044331131911e-08 261 KSP Residual norm 1.005955273988e-08 262 KSP Residual norm 9.713849434221e-09 263 KSP Residual norm 9.441777434239e-09 264 KSP Residual norm 9.207091712794e-09 265 KSP Residual norm 8.987238595528e-09 266 KSP Residual norm 8.823177985750e-09 267 KSP Residual norm 8.701079227787e-09 268 KSP Residual norm 8.561792923106e-09 269 KSP Residual norm 8.326168190578e-09 270 KSP Residual norm 8.041000911848e-09 271 KSP Residual norm 7.801761220241e-09 272 KSP Residual norm 7.554139092007e-09 273 KSP Residual norm 7.306705618033e-09 274 KSP Residual norm 7.077583867408e-09 275 KSP Residual norm 6.891770099074e-09 276 KSP Residual norm 6.725514985579e-09 277 KSP Residual norm 6.525647478946e-09 278 KSP Residual norm 6.305604446856e-09 279 KSP Residual norm 6.081479823646e-09 280 KSP Residual norm 5.908683437780e-09 281 KSP Residual norm 5.829534365497e-09 282 KSP Residual norm 5.706402979339e-09 283 KSP Residual norm 5.402828754684e-09 284 KSP Residual norm 5.113906133755e-09 285 KSP Residual norm 4.911560713902e-09 286 KSP Residual norm 4.747777909774e-09 287 KSP Residual norm 4.672365664044e-09 288 KSP Residual norm 4.644285196856e-09 289 KSP Residual norm 4.470148031321e-09 290 KSP Residual norm 4.225547128275e-09 291 KSP Residual norm 4.052939774057e-09 292 KSP Residual norm 3.926865184193e-09 293 KSP Residual norm 3.797240602052e-09 294 KSP Residual norm 3.656875849037e-09 295 KSP Residual norm 3.542142691057e-09 296 KSP Residual norm 3.507322324056e-09 297 KSP Residual norm 3.473651609106e-09 298 KSP Residual norm 3.322624905585e-09 299 KSP Residual norm 3.179110792497e-09 300 KSP Residual norm 3.099290766449e-09 301 KSP Residual norm 3.060485499823e-09 302 KSP Residual norm 3.020595050904e-09 303 KSP Residual norm 2.988394548842e-09 304 KSP Residual norm 2.974449448775e-09 305 KSP Residual norm 2.978925959763e-09 306 KSP Residual norm 2.992613716416e-09 307 KSP Residual norm 2.987599619696e-09 308 KSP Residual norm 2.970667244988e-09 309 KSP Residual norm 2.935863764739e-09 310 KSP Residual norm 2.881801932798e-09 311 KSP Residual norm 2.838520654725e-09 312 KSP Residual norm 2.809798082630e-09 313 KSP Residual norm 2.762244607602e-09 314 KSP Residual norm 2.694172965472e-09 315 KSP Residual norm 2.621566217797e-09 316 KSP Residual norm 2.551643274637e-09 317 KSP Residual norm 2.482435577544e-09 318 KSP Residual norm 2.414032796846e-09 319 KSP Residual norm 2.338608269151e-09 320 KSP Residual norm 2.266964329768e-09 321 KSP Residual norm 2.194450850780e-09 322 KSP Residual norm 2.113320585691e-09 323 KSP Residual norm 2.022782099591e-09 324 KSP Residual norm 1.930095114779e-09 325 KSP Residual norm 1.845233250560e-09 326 KSP Residual norm 1.765756632761e-09 327 KSP Residual norm 1.689449704473e-09 328 KSP Residual norm 1.626756325544e-09 329 KSP Residual norm 1.575852491489e-09 330 KSP Residual norm 1.527842750982e-09 331 KSP Residual norm 1.478463972930e-09 332 KSP Residual norm 1.432472014086e-09 333 KSP Residual norm 1.387151879043e-09 334 KSP Residual norm 1.344838260832e-09 Linear tao_nls_ solve converged due to CONVERGED_RTOL iterations 334 0 LS Function value: 0.0691394, Step length: 0. 1 LS Function value: 0.0691393, Step length: 1. stx: 0., fx: 0.0691394, dgx: -3.27902e-07 sty: 0., fy: 0.0691394, dgy: -3.27902e-07 25 TAO, Function value: 0.0691393, Residual: 0.000151866 Residual norms for tao_nls_ solve. 0 KSP Residual norm 1.518663160494e-04 1 KSP Residual norm 1.257176766664e-04 2 KSP Residual norm 8.072597050786e-05 3 KSP Residual norm 6.862727736888e-05 4 KSP Residual norm 5.564968381826e-05 5 KSP Residual norm 4.979503604791e-05 6 KSP Residual norm 4.567948444705e-05 7 KSP Residual norm 4.313861699824e-05 8 KSP Residual norm 4.156150778980e-05 9 KSP Residual norm 4.067760456724e-05 10 KSP Residual norm 4.121755141201e-05 11 KSP Residual norm 4.145275501840e-05 12 KSP Residual norm 4.162140332027e-05 13 KSP Residual norm 4.221078775255e-05 14 KSP Residual norm 4.238555358157e-05 15 KSP Residual norm 4.139566775838e-05 16 KSP Residual norm 4.065896118090e-05 17 KSP Residual norm 3.994341628002e-05 18 KSP Residual norm 3.875636148099e-05 19 KSP Residual norm 3.724346096428e-05 20 KSP Residual norm 3.558911985522e-05 21 KSP Residual norm 3.380019527237e-05 22 KSP Residual norm 3.187138259663e-05 23 KSP Residual norm 2.998110867883e-05 24 KSP Residual norm 2.812216423764e-05 25 KSP Residual norm 2.629917758665e-05 26 KSP Residual norm 2.474610460256e-05 27 KSP Residual norm 2.327338424764e-05 28 KSP Residual norm 2.159322337955e-05 29 KSP Residual norm 1.996655210135e-05 30 KSP Residual norm 1.867144700422e-05 31 KSP Residual norm 1.768035572293e-05 32 KSP Residual norm 1.695727293947e-05 33 KSP Residual norm 1.652356819271e-05 34 KSP Residual norm 1.611650412478e-05 35 KSP Residual norm 1.539299816670e-05 36 KSP Residual norm 1.475812030829e-05 37 KSP Residual norm 1.434865785976e-05 38 KSP Residual norm 1.410075772185e-05 39 KSP Residual norm 1.397808392457e-05 40 KSP Residual norm 1.388096895389e-05 41 KSP Residual norm 1.379078697979e-05 42 KSP Residual norm 1.368991065430e-05 43 KSP Residual norm 1.353444284376e-05 44 KSP Residual norm 1.327660397716e-05 45 KSP Residual norm 1.290849333899e-05 46 KSP Residual norm 1.249421269418e-05 47 KSP Residual norm 1.197622039676e-05 48 KSP Residual norm 1.135201964481e-05 49 KSP Residual norm 1.067871177097e-05 50 KSP Residual norm 9.981673791191e-06 51 KSP Residual norm 9.301208979769e-06 52 KSP Residual norm 8.617850192041e-06 53 KSP Residual norm 7.919790795683e-06 54 KSP Residual norm 7.227218417630e-06 55 KSP Residual norm 6.578921223359e-06 56 KSP Residual norm 6.002507779656e-06 57 KSP Residual norm 5.481717367013e-06 58 KSP Residual norm 4.980151046066e-06 59 KSP Residual norm 4.503461598380e-06 60 KSP Residual norm 4.074297869203e-06 61 KSP Residual norm 3.701529625722e-06 62 KSP Residual norm 3.374196999599e-06 63 KSP Residual norm 3.085859797437e-06 64 KSP Residual norm 2.838347413430e-06 65 KSP Residual norm 2.621286129237e-06 66 KSP Residual norm 2.406695167726e-06 67 KSP Residual norm 2.198327546366e-06 68 KSP Residual norm 2.022585480849e-06 69 KSP Residual norm 1.881782116683e-06 70 KSP Residual norm 1.759698310430e-06 71 KSP Residual norm 1.652584839564e-06 72 KSP Residual norm 1.553744752141e-06 73 KSP Residual norm 1.450799731351e-06 74 KSP Residual norm 1.354776155561e-06 75 KSP Residual norm 1.269644159919e-06 76 KSP Residual norm 1.183606950227e-06 77 KSP Residual norm 1.104531038468e-06 78 KSP Residual norm 1.040518865213e-06 79 KSP Residual norm 9.788023876926e-07 80 KSP Residual norm 9.271451965247e-07 81 KSP Residual norm 8.803949117943e-07 82 KSP Residual norm 8.165719167672e-07 83 KSP Residual norm 7.549757661209e-07 84 KSP Residual norm 7.062071702356e-07 85 KSP Residual norm 6.633569453089e-07 86 KSP Residual norm 6.164141745231e-07 87 KSP Residual norm 5.639568609742e-07 88 KSP Residual norm 5.189243717299e-07 89 KSP Residual norm 4.809697004119e-07 90 KSP Residual norm 4.470766623229e-07 91 KSP Residual norm 4.153858239437e-07 92 KSP Residual norm 3.852828410284e-07 93 KSP Residual norm 3.586485572035e-07 94 KSP Residual norm 3.316244555044e-07 95 KSP Residual norm 3.091803468564e-07 96 KSP Residual norm 2.881365242636e-07 97 KSP Residual norm 2.691922099966e-07 98 KSP Residual norm 2.523977115787e-07 99 KSP Residual norm 2.364344849726e-07 100 KSP Residual norm 2.243812959695e-07 101 KSP Residual norm 2.132753826803e-07 102 KSP Residual norm 2.033293257368e-07 103 KSP Residual norm 1.941827918120e-07 104 KSP Residual norm 1.837021926346e-07 105 KSP Residual norm 1.756651186327e-07 106 KSP Residual norm 1.695196920651e-07 107 KSP Residual norm 1.644904104582e-07 108 KSP Residual norm 1.568273622659e-07 109 KSP Residual norm 1.482819459034e-07 110 KSP Residual norm 1.408150634750e-07 111 KSP Residual norm 1.350494470586e-07 112 KSP Residual norm 1.303787629537e-07 113 KSP Residual norm 1.270852325294e-07 114 KSP Residual norm 1.243034312264e-07 115 KSP Residual norm 1.215097892541e-07 116 KSP Residual norm 1.168602930639e-07 117 KSP Residual norm 1.115795266794e-07 118 KSP Residual norm 1.071099374969e-07 119 KSP Residual norm 1.033777386130e-07 120 KSP Residual norm 9.971322689897e-08 121 KSP Residual norm 9.617446869692e-08 122 KSP Residual norm 9.275811813570e-08 123 KSP Residual norm 8.931599895240e-08 124 KSP Residual norm 8.654556274781e-08 125 KSP Residual norm 8.490086986812e-08 126 KSP Residual norm 8.489166014653e-08 127 KSP Residual norm 8.542511549649e-08 128 KSP Residual norm 8.315200698335e-08 129 KSP Residual norm 8.053814393295e-08 130 KSP Residual norm 7.929050202597e-08 131 KSP Residual norm 7.903061203078e-08 132 KSP Residual norm 7.943564117043e-08 133 KSP Residual norm 8.040794829039e-08 134 KSP Residual norm 8.191038362071e-08 135 KSP Residual norm 8.376296472029e-08 136 KSP Residual norm 8.596852950200e-08 137 KSP Residual norm 8.856344274769e-08 138 KSP Residual norm 9.120930734000e-08 139 KSP Residual norm 9.303734165109e-08 140 KSP Residual norm 9.409058164394e-08 141 KSP Residual norm 9.520217099851e-08 142 KSP Residual norm 9.622722621641e-08 143 KSP Residual norm 9.630537509729e-08 144 KSP Residual norm 9.553324233703e-08 145 KSP Residual norm 9.432593707351e-08 146 KSP Residual norm 9.259100868785e-08 147 KSP Residual norm 9.050980036052e-08 148 KSP Residual norm 8.792415277157e-08 149 KSP Residual norm 8.454343627720e-08 150 KSP Residual norm 8.116735341365e-08 151 KSP Residual norm 7.893449548859e-08 152 KSP Residual norm 7.735190787774e-08 153 KSP Residual norm 7.268249654355e-08 154 KSP Residual norm 6.668120676646e-08 155 KSP Residual norm 6.168893583280e-08 156 KSP Residual norm 5.730715165927e-08 157 KSP Residual norm 5.312733744096e-08 158 KSP Residual norm 4.904802544406e-08 159 KSP Residual norm 4.533045823781e-08 160 KSP Residual norm 4.209697730387e-08 161 KSP Residual norm 3.915853481595e-08 162 KSP Residual norm 3.639564265851e-08 163 KSP Residual norm 3.388298793163e-08 164 KSP Residual norm 3.162497623246e-08 165 KSP Residual norm 2.979132649825e-08 166 KSP Residual norm 2.828543334965e-08 167 KSP Residual norm 2.688372352317e-08 168 KSP Residual norm 2.552678520687e-08 169 KSP Residual norm 2.424584545742e-08 170 KSP Residual norm 2.306533830979e-08 171 KSP Residual norm 2.206330608085e-08 172 KSP Residual norm 2.119082765691e-08 173 KSP Residual norm 2.033801455521e-08 174 KSP Residual norm 1.955148784843e-08 175 KSP Residual norm 1.882762766655e-08 176 KSP Residual norm 1.822939026475e-08 177 KSP Residual norm 1.770886509597e-08 178 KSP Residual norm 1.723257278681e-08 179 KSP Residual norm 1.686681814853e-08 180 KSP Residual norm 1.659000024132e-08 181 KSP Residual norm 1.643889492751e-08 182 KSP Residual norm 1.633798470070e-08 183 KSP Residual norm 1.633309522344e-08 184 KSP Residual norm 1.644801705021e-08 185 KSP Residual norm 1.681975762906e-08 186 KSP Residual norm 1.723192526814e-08 187 KSP Residual norm 1.726226819104e-08 188 KSP Residual norm 1.713973006717e-08 189 KSP Residual norm 1.733481613893e-08 190 KSP Residual norm 1.763781550270e-08 191 KSP Residual norm 1.768486099333e-08 192 KSP Residual norm 1.737561993665e-08 193 KSP Residual norm 1.713072845091e-08 194 KSP Residual norm 1.692988619969e-08 195 KSP Residual norm 1.675386139450e-08 196 KSP Residual norm 1.644551094814e-08 197 KSP Residual norm 1.599931337612e-08 198 KSP Residual norm 1.554561100704e-08 199 KSP Residual norm 1.511422681765e-08 200 KSP Residual norm 1.475415440985e-08 201 KSP Residual norm 1.446062633827e-08 202 KSP Residual norm 1.411477439835e-08 203 KSP Residual norm 1.359539611548e-08 204 KSP Residual norm 1.302056655368e-08 205 KSP Residual norm 1.249585767552e-08 206 KSP Residual norm 1.199231347094e-08 207 KSP Residual norm 1.149725260810e-08 208 KSP Residual norm 1.102124340278e-08 209 KSP Residual norm 1.057783295845e-08 210 KSP Residual norm 1.021307208178e-08 211 KSP Residual norm 9.918257022328e-09 212 KSP Residual norm 9.489973048604e-09 213 KSP Residual norm 8.871057734813e-09 214 KSP Residual norm 8.302994599919e-09 215 KSP Residual norm 7.850359553459e-09 216 KSP Residual norm 7.475761977642e-09 217 KSP Residual norm 7.146130248860e-09 218 KSP Residual norm 6.826366641328e-09 219 KSP Residual norm 6.574754411755e-09 220 KSP Residual norm 6.419821202184e-09 221 KSP Residual norm 6.175687968731e-09 222 KSP Residual norm 5.757823354335e-09 223 KSP Residual norm 5.417388630891e-09 224 KSP Residual norm 5.197943416699e-09 225 KSP Residual norm 5.025603499363e-09 226 KSP Residual norm 4.833952906136e-09 227 KSP Residual norm 4.616661074854e-09 228 KSP Residual norm 4.417122380989e-09 229 KSP Residual norm 4.247098180643e-09 230 KSP Residual norm 4.101682019156e-09 231 KSP Residual norm 3.979669220686e-09 232 KSP Residual norm 3.880518874417e-09 233 KSP Residual norm 3.792322278728e-09 234 KSP Residual norm 3.700469470823e-09 235 KSP Residual norm 3.601149656941e-09 236 KSP Residual norm 3.527017493906e-09 237 KSP Residual norm 3.483594811297e-09 238 KSP Residual norm 3.444050282132e-09 239 KSP Residual norm 3.386951995403e-09 240 KSP Residual norm 3.325044158192e-09 241 KSP Residual norm 3.277020039010e-09 242 KSP Residual norm 3.241260093023e-09 243 KSP Residual norm 3.195338451722e-09 244 KSP Residual norm 3.114945004260e-09 245 KSP Residual norm 3.030021894052e-09 246 KSP Residual norm 2.940336953854e-09 247 KSP Residual norm 2.823311136558e-09 248 KSP Residual norm 2.699824479443e-09 249 KSP Residual norm 2.598001807103e-09 250 KSP Residual norm 2.519552673811e-09 251 KSP Residual norm 2.452540848932e-09 252 KSP Residual norm 2.378385866540e-09 253 KSP Residual norm 2.308262603198e-09 254 KSP Residual norm 2.244274452072e-09 255 KSP Residual norm 2.172065817421e-09 256 KSP Residual norm 2.099863621657e-09 257 KSP Residual norm 2.023581847076e-09 258 KSP Residual norm 1.943215909075e-09 259 KSP Residual norm 1.865543115257e-09 260 KSP Residual norm 1.794965834621e-09 261 KSP Residual norm 1.727856794270e-09 262 KSP Residual norm 1.660305436157e-09 263 KSP Residual norm 1.599887827583e-09 264 KSP Residual norm 1.544036202843e-09 265 KSP Residual norm 1.495637008995e-09 Linear tao_nls_ solve converged due to CONVERGED_RTOL iterations 265 0 LS Function value: 0.0691393, Step length: 0. 1 LS Function value: 0.0691392, Step length: 1. stx: 0., fx: 0.0691393, dgx: -1.04908e-07 sty: 0., fy: 0.0691393, dgy: -1.04908e-07 26 TAO, Function value: 0.0691392, Residual: 3.23442e-05 Residual norms for tao_nls_ solve. 0 KSP Residual norm 3.234417226828e-05 1 KSP Residual norm 2.918994650781e-05 2 KSP Residual norm 2.259532352509e-05 3 KSP Residual norm 2.301417502224e-05 4 KSP Residual norm 2.147131420613e-05 5 KSP Residual norm 2.386687310913e-05 6 KSP Residual norm 2.648996072720e-05 7 KSP Residual norm 3.042104386720e-05 8 KSP Residual norm 3.367931056471e-05 9 KSP Residual norm 3.746402969607e-05 10 KSP Residual norm 4.142694856989e-05 11 KSP Residual norm 4.456640674414e-05 12 KSP Residual norm 4.826726292119e-05 13 KSP Residual norm 5.241781563194e-05 14 KSP Residual norm 5.538477682788e-05 15 KSP Residual norm 5.804026621047e-05 16 KSP Residual norm 6.055458107341e-05 17 KSP Residual norm 6.233777865002e-05 18 KSP Residual norm 6.304163855096e-05 19 KSP Residual norm 6.270837678169e-05 20 KSP Residual norm 6.179003362611e-05 21 KSP Residual norm 6.033926713373e-05 22 KSP Residual norm 5.886921074048e-05 23 KSP Residual norm 5.743907445763e-05 24 KSP Residual norm 5.555550448992e-05 25 KSP Residual norm 5.335876586759e-05 26 KSP Residual norm 5.130619904069e-05 27 KSP Residual norm 4.888716126521e-05 28 KSP Residual norm 4.537749816851e-05 29 KSP Residual norm 4.171026919495e-05 30 KSP Residual norm 3.855241110865e-05 31 KSP Residual norm 3.606423321692e-05 32 KSP Residual norm 3.443030672126e-05 33 KSP Residual norm 3.278289180883e-05 34 KSP Residual norm 3.021894499997e-05 35 KSP Residual norm 2.794820325576e-05 36 KSP Residual norm 2.634828964616e-05 37 KSP Residual norm 2.510776860350e-05 38 KSP Residual norm 2.409810449662e-05 39 KSP Residual norm 2.321885845712e-05 40 KSP Residual norm 2.234368882697e-05 41 KSP Residual norm 2.154351939133e-05 42 KSP Residual norm 2.083280278913e-05 43 KSP Residual norm 2.012185997158e-05 44 KSP Residual norm 1.928939483974e-05 45 KSP Residual norm 1.832150757344e-05 46 KSP Residual norm 1.732795831619e-05 47 KSP Residual norm 1.629985536599e-05 48 KSP Residual norm 1.524222711918e-05 49 KSP Residual norm 1.419755769992e-05 50 KSP Residual norm 1.319486099811e-05 51 KSP Residual norm 1.228334543135e-05 52 KSP Residual norm 1.145286432541e-05 53 KSP Residual norm 1.066878318114e-05 54 KSP Residual norm 9.940896656791e-06 55 KSP Residual norm 9.280518796691e-06 56 KSP Residual norm 8.720338162884e-06 57 KSP Residual norm 8.231405037130e-06 58 KSP Residual norm 7.761037809547e-06 59 KSP Residual norm 7.284551690565e-06 60 KSP Residual norm 6.809172256842e-06 61 KSP Residual norm 6.370188936310e-06 62 KSP Residual norm 5.967762816358e-06 63 KSP Residual norm 5.603111159668e-06 64 KSP Residual norm 5.278689462027e-06 65 KSP Residual norm 4.962460175841e-06 66 KSP Residual norm 4.610168661348e-06 67 KSP Residual norm 4.257924533821e-06 68 KSP Residual norm 3.961445567936e-06 69 KSP Residual norm 3.713923503842e-06 70 KSP Residual norm 3.474849419166e-06 71 KSP Residual norm 3.253701791033e-06 72 KSP Residual norm 3.070914555621e-06 73 KSP Residual norm 2.908181557571e-06 74 KSP Residual norm 2.777237128810e-06 75 KSP Residual norm 2.681521233617e-06 76 KSP Residual norm 2.611062319726e-06 77 KSP Residual norm 2.549924690558e-06 78 KSP Residual norm 2.443434760250e-06 79 KSP Residual norm 2.320108972969e-06 80 KSP Residual norm 2.236879844439e-06 81 KSP Residual norm 2.182163099243e-06 82 KSP Residual norm 2.120478278753e-06 83 KSP Residual norm 2.054859393046e-06 84 KSP Residual norm 1.967867559383e-06 85 KSP Residual norm 1.884423323994e-06 86 KSP Residual norm 1.803010567545e-06 87 KSP Residual norm 1.701896947862e-06 88 KSP Residual norm 1.605102310635e-06 89 KSP Residual norm 1.506396160197e-06 90 KSP Residual norm 1.397304238887e-06 91 KSP Residual norm 1.283146666108e-06 92 KSP Residual norm 1.180127787740e-06 93 KSP Residual norm 1.087306597268e-06 94 KSP Residual norm 9.935667029765e-07 95 KSP Residual norm 9.172731367786e-07 96 KSP Residual norm 8.438239445458e-07 97 KSP Residual norm 7.945464960405e-07 98 KSP Residual norm 7.333042825560e-07 99 KSP Residual norm 6.551966119811e-07 100 KSP Residual norm 5.891382716400e-07 101 KSP Residual norm 5.335106695577e-07 102 KSP Residual norm 4.851673924105e-07 103 KSP Residual norm 4.419868817609e-07 104 KSP Residual norm 3.987826823679e-07 105 KSP Residual norm 3.636775108034e-07 106 KSP Residual norm 3.323095349781e-07 107 KSP Residual norm 3.080316830228e-07 108 KSP Residual norm 2.854676922991e-07 109 KSP Residual norm 2.669936830166e-07 110 KSP Residual norm 2.494783794641e-07 111 KSP Residual norm 2.372930781943e-07 112 KSP Residual norm 2.271917553892e-07 113 KSP Residual norm 2.192712509252e-07 114 KSP Residual norm 2.059236462154e-07 115 KSP Residual norm 1.917008727185e-07 116 KSP Residual norm 1.795890425176e-07 117 KSP Residual norm 1.708185806701e-07 118 KSP Residual norm 1.638257076724e-07 119 KSP Residual norm 1.568628586611e-07 120 KSP Residual norm 1.464966025735e-07 121 KSP Residual norm 1.368662654393e-07 122 KSP Residual norm 1.276192716375e-07 123 KSP Residual norm 1.190750440253e-07 124 KSP Residual norm 1.118193449490e-07 125 KSP Residual norm 1.053500510954e-07 126 KSP Residual norm 9.932557816386e-08 127 KSP Residual norm 9.420823451359e-08 128 KSP Residual norm 9.014853320220e-08 129 KSP Residual norm 8.602765667747e-08 130 KSP Residual norm 8.083205053560e-08 131 KSP Residual norm 7.600634328557e-08 132 KSP Residual norm 7.195384811402e-08 133 KSP Residual norm 6.871002252907e-08 134 KSP Residual norm 6.606847594138e-08 135 KSP Residual norm 6.394707006973e-08 136 KSP Residual norm 6.217054736572e-08 137 KSP Residual norm 6.039434695269e-08 138 KSP Residual norm 5.893520000884e-08 139 KSP Residual norm 5.815027865021e-08 140 KSP Residual norm 5.752086279393e-08 141 KSP Residual norm 5.630288105456e-08 142 KSP Residual norm 5.529375115900e-08 143 KSP Residual norm 5.512445662376e-08 144 KSP Residual norm 5.627528719412e-08 145 KSP Residual norm 5.700756731141e-08 146 KSP Residual norm 5.589827750137e-08 147 KSP Residual norm 5.500356407398e-08 148 KSP Residual norm 5.498961246691e-08 149 KSP Residual norm 5.534656964091e-08 150 KSP Residual norm 5.591487117889e-08 151 KSP Residual norm 5.639728892705e-08 152 KSP Residual norm 5.632124681628e-08 153 KSP Residual norm 5.604343875967e-08 154 KSP Residual norm 5.591054649292e-08 155 KSP Residual norm 5.576906761131e-08 156 KSP Residual norm 5.573044479931e-08 157 KSP Residual norm 5.563177028378e-08 158 KSP Residual norm 5.519476573246e-08 159 KSP Residual norm 5.466499590450e-08 160 KSP Residual norm 5.405390877974e-08 161 KSP Residual norm 5.298509555970e-08 162 KSP Residual norm 5.145790951151e-08 163 KSP Residual norm 4.970981041670e-08 164 KSP Residual norm 4.784927131604e-08 165 KSP Residual norm 4.601314009372e-08 166 KSP Residual norm 4.414832151482e-08 167 KSP Residual norm 4.225730662747e-08 168 KSP Residual norm 4.030500514610e-08 169 KSP Residual norm 3.836504715964e-08 170 KSP Residual norm 3.640952284337e-08 171 KSP Residual norm 3.439871641430e-08 172 KSP Residual norm 3.241663582161e-08 173 KSP Residual norm 3.054744049674e-08 174 KSP Residual norm 2.877083856956e-08 175 KSP Residual norm 2.699944969158e-08 176 KSP Residual norm 2.538730458048e-08 177 KSP Residual norm 2.394872559939e-08 178 KSP Residual norm 2.268802024779e-08 179 KSP Residual norm 2.166987350062e-08 180 KSP Residual norm 2.097832015688e-08 181 KSP Residual norm 2.045402624354e-08 182 KSP Residual norm 1.940568070244e-08 183 KSP Residual norm 1.824046089049e-08 184 KSP Residual norm 1.736026559868e-08 185 KSP Residual norm 1.675887619907e-08 186 KSP Residual norm 1.631066991176e-08 187 KSP Residual norm 1.601409787504e-08 188 KSP Residual norm 1.574346566547e-08 189 KSP Residual norm 1.530569901823e-08 190 KSP Residual norm 1.475304009854e-08 191 KSP Residual norm 1.429060484940e-08 192 KSP Residual norm 1.388398866780e-08 193 KSP Residual norm 1.358340965390e-08 194 KSP Residual norm 1.328704850591e-08 195 KSP Residual norm 1.303649095265e-08 196 KSP Residual norm 1.276682592854e-08 197 KSP Residual norm 1.248965812632e-08 198 KSP Residual norm 1.220042370723e-08 199 KSP Residual norm 1.194634250308e-08 200 KSP Residual norm 1.180040319487e-08 201 KSP Residual norm 1.166848082902e-08 202 KSP Residual norm 1.128796801386e-08 203 KSP Residual norm 1.081004773740e-08 204 KSP Residual norm 1.042271102196e-08 205 KSP Residual norm 1.013285312119e-08 206 KSP Residual norm 9.953787942604e-09 207 KSP Residual norm 9.861622181758e-09 208 KSP Residual norm 9.602451912754e-09 209 KSP Residual norm 9.201857881093e-09 210 KSP Residual norm 8.902825009398e-09 211 KSP Residual norm 8.710451931656e-09 212 KSP Residual norm 8.566131589591e-09 213 KSP Residual norm 8.452107429963e-09 214 KSP Residual norm 8.333390827437e-09 215 KSP Residual norm 8.199415205024e-09 216 KSP Residual norm 8.058233285292e-09 217 KSP Residual norm 7.928543076457e-09 218 KSP Residual norm 7.798675276806e-09 219 KSP Residual norm 7.671398923819e-09 220 KSP Residual norm 7.514166050670e-09 221 KSP Residual norm 7.335159969842e-09 222 KSP Residual norm 7.155890113780e-09 223 KSP Residual norm 7.001182726713e-09 224 KSP Residual norm 6.904000293277e-09 225 KSP Residual norm 6.818762500132e-09 226 KSP Residual norm 6.648182413459e-09 227 KSP Residual norm 6.398669076853e-09 228 KSP Residual norm 6.191851303004e-09 229 KSP Residual norm 6.060248146442e-09 230 KSP Residual norm 5.973686959956e-09 231 KSP Residual norm 5.886366555588e-09 232 KSP Residual norm 5.800879704655e-09 233 KSP Residual norm 5.723057053278e-09 234 KSP Residual norm 5.627888413990e-09 235 KSP Residual norm 5.497264967251e-09 236 KSP Residual norm 5.368372951629e-09 237 KSP Residual norm 5.285001582888e-09 238 KSP Residual norm 5.242359406602e-09 239 KSP Residual norm 5.158337908053e-09 240 KSP Residual norm 4.962758163036e-09 241 KSP Residual norm 4.740963240648e-09 242 KSP Residual norm 4.545108253459e-09 243 KSP Residual norm 4.381188462489e-09 244 KSP Residual norm 4.238508696906e-09 245 KSP Residual norm 4.091840227666e-09 246 KSP Residual norm 3.916187003545e-09 247 KSP Residual norm 3.731492124165e-09 248 KSP Residual norm 3.564117678022e-09 249 KSP Residual norm 3.418683986120e-09 250 KSP Residual norm 3.287760844397e-09 251 KSP Residual norm 3.172030314199e-09 252 KSP Residual norm 3.067052959210e-09 253 KSP Residual norm 2.985643529680e-09 254 KSP Residual norm 2.924455685915e-09 255 KSP Residual norm 2.873394131188e-09 256 KSP Residual norm 2.834745685762e-09 257 KSP Residual norm 2.779897220610e-09 258 KSP Residual norm 2.696038906715e-09 259 KSP Residual norm 2.605018760067e-09 260 KSP Residual norm 2.524854300055e-09 261 KSP Residual norm 2.461113438603e-09 262 KSP Residual norm 2.406891618302e-09 263 KSP Residual norm 2.369556217385e-09 264 KSP Residual norm 2.334784017128e-09 265 KSP Residual norm 2.274139622418e-09 266 KSP Residual norm 2.183413506944e-09 267 KSP Residual norm 2.094987825309e-09 268 KSP Residual norm 2.041409508731e-09 269 KSP Residual norm 2.043871037199e-09 270 KSP Residual norm 2.062665625010e-09 271 KSP Residual norm 1.977638989467e-09 272 KSP Residual norm 1.877276391992e-09 273 KSP Residual norm 1.819486768819e-09 274 KSP Residual norm 1.783106658322e-09 275 KSP Residual norm 1.765840151958e-09 276 KSP Residual norm 1.757571823384e-09 277 KSP Residual norm 1.724976627390e-09 278 KSP Residual norm 1.667428059744e-09 279 KSP Residual norm 1.598993243835e-09 280 KSP Residual norm 1.535350247961e-09 281 KSP Residual norm 1.478598570154e-09 282 KSP Residual norm 1.424559000158e-09 283 KSP Residual norm 1.374815694882e-09 284 KSP Residual norm 1.322827012165e-09 285 KSP Residual norm 1.265608312230e-09 286 KSP Residual norm 1.213790533376e-09 287 KSP Residual norm 1.161439900851e-09 288 KSP Residual norm 1.106154903457e-09 289 KSP Residual norm 1.060453755118e-09 290 KSP Residual norm 1.019821719823e-09 291 KSP Residual norm 9.811102184207e-10 292 KSP Residual norm 9.484264130400e-10 293 KSP Residual norm 9.254138152574e-10 294 KSP Residual norm 9.015417584504e-10 295 KSP Residual norm 8.614927273138e-10 296 KSP Residual norm 8.172216315811e-10 297 KSP Residual norm 7.796814374659e-10 298 KSP Residual norm 7.462967403941e-10 299 KSP Residual norm 7.167222418579e-10 300 KSP Residual norm 6.929697257079e-10 301 KSP Residual norm 6.718875094201e-10 302 KSP Residual norm 6.502387154312e-10 303 KSP Residual norm 6.299438678745e-10 304 KSP Residual norm 6.117312680271e-10 305 KSP Residual norm 5.975589506092e-10 306 KSP Residual norm 5.827814548922e-10 307 KSP Residual norm 5.704315002691e-10 308 KSP Residual norm 5.614752920995e-10 309 KSP Residual norm 5.504226507620e-10 310 KSP Residual norm 5.321686495071e-10 311 KSP Residual norm 5.141532105511e-10 312 KSP Residual norm 4.997864353031e-10 313 KSP Residual norm 4.860964241738e-10 314 KSP Residual norm 4.735978895455e-10 315 KSP Residual norm 4.629649920207e-10 316 KSP Residual norm 4.510893120587e-10 317 KSP Residual norm 4.396044822847e-10 318 KSP Residual norm 4.304281314002e-10 319 KSP Residual norm 4.217353369170e-10 320 KSP Residual norm 4.124668817033e-10 321 KSP Residual norm 4.006105222831e-10 322 KSP Residual norm 3.890824495994e-10 323 KSP Residual norm 3.790440569389e-10 324 KSP Residual norm 3.714480912210e-10 325 KSP Residual norm 3.617346328128e-10 326 KSP Residual norm 3.502095377452e-10 327 KSP Residual norm 3.386452851654e-10 328 KSP Residual norm 3.278824692995e-10 329 KSP Residual norm 3.178187791152e-10 Linear tao_nls_ solve converged due to CONVERGED_RTOL iterations 329 0 LS Function value: 0.0691392, Step length: 0. 1 LS Function value: 0.0691394, Step length: 1. stx: 0., fx: 0.0691392, dgx: -9.49012e-08 sty: 0., fy: 0.0691392, dgy: -9.49012e-08 2 LS Function value: 0.0691392, Step length: 0.297235 stx: 0., fx: 0.0691392, dgx: -9.49012e-08 sty: 1., fy: 0.0691394, dgy: 1.03991e-06 27 TAO, Function value: 0.0691392, Residual: 6.20716e-05 Residual norms for tao_nls_ solve. 0 KSP Residual norm 6.207164296950e-05 1 KSP Residual norm 4.870239638133e-05 2 KSP Residual norm 3.408790823502e-05 3 KSP Residual norm 2.888095130127e-05 4 KSP Residual norm 2.504861751816e-05 5 KSP Residual norm 2.371230123118e-05 6 KSP Residual norm 2.327858207838e-05 7 KSP Residual norm 2.404569681595e-05 8 KSP Residual norm 2.527311314442e-05 9 KSP Residual norm 2.672851015847e-05 10 KSP Residual norm 2.905004949113e-05 11 KSP Residual norm 3.135219409509e-05 12 KSP Residual norm 3.340176597783e-05 13 KSP Residual norm 3.616214749369e-05 14 KSP Residual norm 3.927666358172e-05 15 KSP Residual norm 4.140403059493e-05 16 KSP Residual norm 4.269280282607e-05 17 KSP Residual norm 4.426956891087e-05 18 KSP Residual norm 4.542548201455e-05 19 KSP Residual norm 4.581149546611e-05 20 KSP Residual norm 4.572046307848e-05 21 KSP Residual norm 4.506624516950e-05 22 KSP Residual norm 4.424557025631e-05 23 KSP Residual norm 4.319687024479e-05 24 KSP Residual norm 4.163225481726e-05 25 KSP Residual norm 3.978619139255e-05 26 KSP Residual norm 3.792383086576e-05 27 KSP Residual norm 3.620470764580e-05 28 KSP Residual norm 3.436503219167e-05 29 KSP Residual norm 3.197310945540e-05 30 KSP Residual norm 2.922887533121e-05 31 KSP Residual norm 2.661464866145e-05 32 KSP Residual norm 2.424886768191e-05 33 KSP Residual norm 2.211120353980e-05 34 KSP Residual norm 2.023621802568e-05 35 KSP Residual norm 1.870972900428e-05 36 KSP Residual norm 1.730312447089e-05 37 KSP Residual norm 1.567845749079e-05 38 KSP Residual norm 1.414301097879e-05 39 KSP Residual norm 1.289661155197e-05 40 KSP Residual norm 1.185666622519e-05 41 KSP Residual norm 1.100120537221e-05 42 KSP Residual norm 1.033586282756e-05 43 KSP Residual norm 9.807776688677e-06 44 KSP Residual norm 9.303513451259e-06 45 KSP Residual norm 8.747925799597e-06 46 KSP Residual norm 8.197646580369e-06 47 KSP Residual norm 7.681593170030e-06 48 KSP Residual norm 7.190140687018e-06 49 KSP Residual norm 6.716638046893e-06 50 KSP Residual norm 6.267117037821e-06 51 KSP Residual norm 5.859759109712e-06 52 KSP Residual norm 5.487542072348e-06 53 KSP Residual norm 5.125079417454e-06 54 KSP Residual norm 4.768354547449e-06 55 KSP Residual norm 4.434849319674e-06 56 KSP Residual norm 4.156329118415e-06 57 KSP Residual norm 3.941777399996e-06 58 KSP Residual norm 3.777200501326e-06 59 KSP Residual norm 3.614416956937e-06 60 KSP Residual norm 3.409555241189e-06 61 KSP Residual norm 3.190479621051e-06 62 KSP Residual norm 2.988445455288e-06 63 KSP Residual norm 2.799370171558e-06 64 KSP Residual norm 2.623368903113e-06 65 KSP Residual norm 2.466703109619e-06 66 KSP Residual norm 2.318303195276e-06 67 KSP Residual norm 2.156628765537e-06 68 KSP Residual norm 1.984601978884e-06 69 KSP Residual norm 1.826865942078e-06 70 KSP Residual norm 1.677672947235e-06 71 KSP Residual norm 1.538936158198e-06 72 KSP Residual norm 1.414079153527e-06 73 KSP Residual norm 1.295625010980e-06 74 KSP Residual norm 1.194681881885e-06 75 KSP Residual norm 1.113049682749e-06 76 KSP Residual norm 1.029396561984e-06 77 KSP Residual norm 9.513755868598e-07 78 KSP Residual norm 8.917364302084e-07 79 KSP Residual norm 8.458154273030e-07 80 KSP Residual norm 8.117903441299e-07 81 KSP Residual norm 7.778704911959e-07 82 KSP Residual norm 7.379316196975e-07 83 KSP Residual norm 6.977347877653e-07 84 KSP Residual norm 6.608485013444e-07 85 KSP Residual norm 6.356219776965e-07 86 KSP Residual norm 6.205766570630e-07 87 KSP Residual norm 6.039228304723e-07 88 KSP Residual norm 5.897765764182e-07 89 KSP Residual norm 5.764193213947e-07 90 KSP Residual norm 5.609440618890e-07 91 KSP Residual norm 5.393376761912e-07 92 KSP Residual norm 5.185519458782e-07 93 KSP Residual norm 4.998965899280e-07 94 KSP Residual norm 4.791150851465e-07 95 KSP Residual norm 4.659227977942e-07 96 KSP Residual norm 4.452140935136e-07 97 KSP Residual norm 4.127311262180e-07 98 KSP Residual norm 3.805046140486e-07 99 KSP Residual norm 3.511823343617e-07 100 KSP Residual norm 3.260414946589e-07 101 KSP Residual norm 3.018501950985e-07 102 KSP Residual norm 2.763977826597e-07 103 KSP Residual norm 2.540731583568e-07 104 KSP Residual norm 2.308146010740e-07 105 KSP Residual norm 2.115927840755e-07 106 KSP Residual norm 1.930030802699e-07 107 KSP Residual norm 1.775512858830e-07 108 KSP Residual norm 1.631631230232e-07 109 KSP Residual norm 1.505392959062e-07 110 KSP Residual norm 1.389301332645e-07 111 KSP Residual norm 1.297668045645e-07 112 KSP Residual norm 1.218944885633e-07 113 KSP Residual norm 1.144432711363e-07 114 KSP Residual norm 1.060990556104e-07 115 KSP Residual norm 9.805534854088e-08 116 KSP Residual norm 9.134600734870e-08 117 KSP Residual norm 8.605227093850e-08 118 KSP Residual norm 8.245531546837e-08 119 KSP Residual norm 7.976618583618e-08 120 KSP Residual norm 7.479536962394e-08 121 KSP Residual norm 6.935675564141e-08 122 KSP Residual norm 6.508423679094e-08 123 KSP Residual norm 6.161655111675e-08 124 KSP Residual norm 5.879132258772e-08 125 KSP Residual norm 5.594030037441e-08 126 KSP Residual norm 5.291875508651e-08 127 KSP Residual norm 5.016496793775e-08 128 KSP Residual norm 4.817664281285e-08 129 KSP Residual norm 4.678675442000e-08 130 KSP Residual norm 4.554148314276e-08 131 KSP Residual norm 4.387891755553e-08 132 KSP Residual norm 4.207333810840e-08 133 KSP Residual norm 4.062670889060e-08 134 KSP Residual norm 3.964314800835e-08 135 KSP Residual norm 3.904261306892e-08 136 KSP Residual norm 3.866570690170e-08 137 KSP Residual norm 3.821786323010e-08 138 KSP Residual norm 3.771739251798e-08 139 KSP Residual norm 3.732295468132e-08 140 KSP Residual norm 3.691465928174e-08 141 KSP Residual norm 3.627260680639e-08 142 KSP Residual norm 3.558358601075e-08 143 KSP Residual norm 3.493957342658e-08 144 KSP Residual norm 3.455904290293e-08 145 KSP Residual norm 3.447305175083e-08 146 KSP Residual norm 3.467078792051e-08 147 KSP Residual norm 3.562905308650e-08 148 KSP Residual norm 3.664870524350e-08 149 KSP Residual norm 3.566840748997e-08 150 KSP Residual norm 3.465561137009e-08 151 KSP Residual norm 3.441155428207e-08 152 KSP Residual norm 3.425780038954e-08 153 KSP Residual norm 3.406098029758e-08 154 KSP Residual norm 3.396713010610e-08 155 KSP Residual norm 3.380073119785e-08 156 KSP Residual norm 3.338710250210e-08 157 KSP Residual norm 3.283503128624e-08 158 KSP Residual norm 3.221267872074e-08 159 KSP Residual norm 3.174881841429e-08 160 KSP Residual norm 3.143162046303e-08 161 KSP Residual norm 3.093769972950e-08 162 KSP Residual norm 3.011467802782e-08 163 KSP Residual norm 2.915222434886e-08 164 KSP Residual norm 2.816828567334e-08 165 KSP Residual norm 2.719124889222e-08 166 KSP Residual norm 2.617179553766e-08 167 KSP Residual norm 2.509448288201e-08 168 KSP Residual norm 2.397462775994e-08 169 KSP Residual norm 2.287942093970e-08 170 KSP Residual norm 2.183443152856e-08 171 KSP Residual norm 2.076669720150e-08 172 KSP Residual norm 1.963105399320e-08 173 KSP Residual norm 1.854074199769e-08 174 KSP Residual norm 1.759173077288e-08 175 KSP Residual norm 1.674496127909e-08 176 KSP Residual norm 1.569996221092e-08 177 KSP Residual norm 1.443822705329e-08 178 KSP Residual norm 1.330273856810e-08 179 KSP Residual norm 1.238765818192e-08 180 KSP Residual norm 1.160231817637e-08 181 KSP Residual norm 1.089171039840e-08 182 KSP Residual norm 1.023737562404e-08 183 KSP Residual norm 9.654459542317e-09 184 KSP Residual norm 9.137335587102e-09 185 KSP Residual norm 8.707154589221e-09 186 KSP Residual norm 8.334245327074e-09 187 KSP Residual norm 8.008928731795e-09 188 KSP Residual norm 7.734956749035e-09 189 KSP Residual norm 7.525206026453e-09 190 KSP Residual norm 7.352026778813e-09 191 KSP Residual norm 7.200463341764e-09 192 KSP Residual norm 7.062291663419e-09 193 KSP Residual norm 6.985942365039e-09 194 KSP Residual norm 6.949840631298e-09 195 KSP Residual norm 6.924733938522e-09 196 KSP Residual norm 6.885696905386e-09 197 KSP Residual norm 6.841713691455e-09 198 KSP Residual norm 6.809434560100e-09 199 KSP Residual norm 6.797926098661e-09 200 KSP Residual norm 6.808014026198e-09 201 KSP Residual norm 6.805498807547e-09 202 KSP Residual norm 6.734113838742e-09 203 KSP Residual norm 6.621984198175e-09 204 KSP Residual norm 6.512392102271e-09 205 KSP Residual norm 6.411649399707e-09 206 KSP Residual norm 6.316453457627e-09 207 KSP Residual norm 6.224702917395e-09 208 KSP Residual norm 6.138529686539e-09 209 KSP Residual norm 6.090033637993e-09 210 KSP Residual norm 6.095132474708e-09 211 KSP Residual norm 5.967278625457e-09 212 KSP Residual norm 5.704950510233e-09 213 KSP Residual norm 5.522197272185e-09 214 KSP Residual norm 5.423210939912e-09 215 KSP Residual norm 5.333076216162e-09 216 KSP Residual norm 5.207320994319e-09 217 KSP Residual norm 5.055438887727e-09 218 KSP Residual norm 4.906164730175e-09 219 KSP Residual norm 4.754905259667e-09 220 KSP Residual norm 4.591610542013e-09 221 KSP Residual norm 4.423656769426e-09 222 KSP Residual norm 4.272766051320e-09 223 KSP Residual norm 4.142341667612e-09 224 KSP Residual norm 4.034033989128e-09 225 KSP Residual norm 3.895902435101e-09 226 KSP Residual norm 3.717396613889e-09 227 KSP Residual norm 3.528610437721e-09 228 KSP Residual norm 3.370389091382e-09 229 KSP Residual norm 3.247595732305e-09 230 KSP Residual norm 3.152415091021e-09 231 KSP Residual norm 3.066910848832e-09 232 KSP Residual norm 3.002100350363e-09 233 KSP Residual norm 2.967713103362e-09 234 KSP Residual norm 2.915145929649e-09 235 KSP Residual norm 2.810499247724e-09 236 KSP Residual norm 2.715996703257e-09 237 KSP Residual norm 2.656986413384e-09 238 KSP Residual norm 2.600840669702e-09 239 KSP Residual norm 2.533916204908e-09 240 KSP Residual norm 2.471276640707e-09 241 KSP Residual norm 2.412717720400e-09 242 KSP Residual norm 2.346946360162e-09 243 KSP Residual norm 2.280489473057e-09 244 KSP Residual norm 2.221675448025e-09 245 KSP Residual norm 2.163405326037e-09 246 KSP Residual norm 2.089185761563e-09 247 KSP Residual norm 2.002500217781e-09 248 KSP Residual norm 1.920252602197e-09 249 KSP Residual norm 1.848293324560e-09 250 KSP Residual norm 1.784115043508e-09 251 KSP Residual norm 1.728656291432e-09 252 KSP Residual norm 1.681441494499e-09 253 KSP Residual norm 1.644681990002e-09 254 KSP Residual norm 1.609217211191e-09 255 KSP Residual norm 1.567379451601e-09 256 KSP Residual norm 1.527305416776e-09 257 KSP Residual norm 1.492459894704e-09 258 KSP Residual norm 1.462251168785e-09 259 KSP Residual norm 1.438430039087e-09 260 KSP Residual norm 1.427062606959e-09 261 KSP Residual norm 1.420556029865e-09 262 KSP Residual norm 1.378551627494e-09 263 KSP Residual norm 1.317566463420e-09 264 KSP Residual norm 1.267927458856e-09 265 KSP Residual norm 1.228735481103e-09 266 KSP Residual norm 1.201300740767e-09 267 KSP Residual norm 1.184360394015e-09 268 KSP Residual norm 1.162465234804e-09 269 KSP Residual norm 1.121917251682e-09 270 KSP Residual norm 1.080238933766e-09 271 KSP Residual norm 1.050440712043e-09 272 KSP Residual norm 1.028350471237e-09 273 KSP Residual norm 1.008601382077e-09 274 KSP Residual norm 9.928967308286e-10 275 KSP Residual norm 9.952421863052e-10 276 KSP Residual norm 1.002288537055e-09 277 KSP Residual norm 9.630484048005e-10 278 KSP Residual norm 9.162946023882e-10 279 KSP Residual norm 8.820228522479e-10 280 KSP Residual norm 8.563495213439e-10 281 KSP Residual norm 8.338396929663e-10 282 KSP Residual norm 8.134956273297e-10 283 KSP Residual norm 7.947003222506e-10 284 KSP Residual norm 7.724727225023e-10 285 KSP Residual norm 7.442358429480e-10 286 KSP Residual norm 7.179519930681e-10 287 KSP Residual norm 6.916101043054e-10 288 KSP Residual norm 6.637642439787e-10 289 KSP Residual norm 6.392690829817e-10 290 KSP Residual norm 6.181388860815e-10 Linear tao_nls_ solve converged due to CONVERGED_RTOL iterations 290 0 LS Function value: 0.0691392, Step length: 0. 1 LS Function value: 0.0691392, Step length: 1. stx: 0., fx: 0.0691392, dgx: -5.61929e-08 sty: 0., fy: 0.0691392, dgy: -5.61929e-08 28 TAO, Function value: 0.0691392, Residual: 0.000124392 Residual norms for tao_nls_ solve. 0 KSP Residual norm 1.243919138083e-04 1 KSP Residual norm 9.598853562990e-05 2 KSP Residual norm 6.444930874270e-05 3 KSP Residual norm 5.151675703978e-05 4 KSP Residual norm 4.031819883017e-05 5 KSP Residual norm 3.472562716378e-05 6 KSP Residual norm 3.033644845851e-05 7 KSP Residual norm 2.769552993406e-05 8 KSP Residual norm 2.601940872694e-05 9 KSP Residual norm 2.489167103989e-05 10 KSP Residual norm 2.444179412041e-05 11 KSP Residual norm 2.413650437320e-05 12 KSP Residual norm 2.401898775311e-05 13 KSP Residual norm 2.388688378050e-05 14 KSP Residual norm 2.368928819505e-05 15 KSP Residual norm 2.409143414917e-05 16 KSP Residual norm 2.489973698451e-05 17 KSP Residual norm 2.561907145707e-05 18 KSP Residual norm 2.590209789357e-05 19 KSP Residual norm 2.629022150414e-05 20 KSP Residual norm 2.672894643316e-05 21 KSP Residual norm 2.696483432870e-05 22 KSP Residual norm 2.710774887867e-05 23 KSP Residual norm 2.712790373984e-05 24 KSP Residual norm 2.682052421971e-05 25 KSP Residual norm 2.632547895187e-05 26 KSP Residual norm 2.575188871224e-05 27 KSP Residual norm 2.494530840321e-05 28 KSP Residual norm 2.382306708439e-05 29 KSP Residual norm 2.250613037147e-05 30 KSP Residual norm 2.115270136566e-05 31 KSP Residual norm 1.983480058096e-05 32 KSP Residual norm 1.849818196230e-05 33 KSP Residual norm 1.706151323717e-05 34 KSP Residual norm 1.555232444078e-05 35 KSP Residual norm 1.408289902271e-05 36 KSP Residual norm 1.270432400189e-05 37 KSP Residual norm 1.143528323914e-05 38 KSP Residual norm 1.031770420205e-05 39 KSP Residual norm 9.413118021926e-06 40 KSP Residual norm 8.688246009280e-06 41 KSP Residual norm 7.800006000882e-06 42 KSP Residual norm 6.871887257434e-06 43 KSP Residual norm 6.132374442378e-06 44 KSP Residual norm 5.545078554973e-06 45 KSP Residual norm 5.062276412426e-06 46 KSP Residual norm 4.652665477993e-06 47 KSP Residual norm 4.236813801367e-06 48 KSP Residual norm 3.783945420875e-06 49 KSP Residual norm 3.366032167710e-06 50 KSP Residual norm 3.006770151892e-06 51 KSP Residual norm 2.696005062384e-06 52 KSP Residual norm 2.417723004150e-06 53 KSP Residual norm 2.160208903565e-06 54 KSP Residual norm 1.921473333392e-06 55 KSP Residual norm 1.704574633539e-06 56 KSP Residual norm 1.518971358683e-06 57 KSP Residual norm 1.365067173323e-06 58 KSP Residual norm 1.237738384978e-06 59 KSP Residual norm 1.131072305903e-06 60 KSP Residual norm 1.042922420881e-06 61 KSP Residual norm 9.641314406829e-07 62 KSP Residual norm 8.864910112534e-07 63 KSP Residual norm 8.149507563951e-07 64 KSP Residual norm 7.590775119629e-07 65 KSP Residual norm 7.254614138290e-07 66 KSP Residual norm 6.941886923984e-07 67 KSP Residual norm 6.366811211910e-07 68 KSP Residual norm 5.830219462423e-07 69 KSP Residual norm 5.462150757116e-07 70 KSP Residual norm 5.154522959850e-07 71 KSP Residual norm 4.876808461793e-07 72 KSP Residual norm 4.637042865688e-07 73 KSP Residual norm 4.395012800685e-07 74 KSP Residual norm 4.139484784246e-07 75 KSP Residual norm 3.880291216830e-07 76 KSP Residual norm 3.598504081423e-07 77 KSP Residual norm 3.323842765900e-07 78 KSP Residual norm 3.088491112842e-07 79 KSP Residual norm 2.868458466842e-07 80 KSP Residual norm 2.669668780592e-07 81 KSP Residual norm 2.465619100889e-07 82 KSP Residual norm 2.272961194614e-07 83 KSP Residual norm 2.111519473206e-07 84 KSP Residual norm 1.946953612697e-07 85 KSP Residual norm 1.788025493229e-07 86 KSP Residual norm 1.649444827079e-07 87 KSP Residual norm 1.527625378575e-07 88 KSP Residual norm 1.425342373182e-07 89 KSP Residual norm 1.334205935953e-07 90 KSP Residual norm 1.252927848215e-07 91 KSP Residual norm 1.179792119386e-07 92 KSP Residual norm 1.121973081878e-07 93 KSP Residual norm 1.050654555067e-07 94 KSP Residual norm 9.632586990219e-08 95 KSP Residual norm 8.993660204601e-08 96 KSP Residual norm 8.480314753476e-08 97 KSP Residual norm 8.051763984462e-08 98 KSP Residual norm 7.735730482369e-08 99 KSP Residual norm 7.429669254072e-08 100 KSP Residual norm 7.184227305285e-08 101 KSP Residual norm 6.923205251665e-08 102 KSP Residual norm 6.598064032149e-08 103 KSP Residual norm 6.366219742396e-08 104 KSP Residual norm 6.131490440446e-08 105 KSP Residual norm 5.948624465811e-08 106 KSP Residual norm 5.729158459072e-08 107 KSP Residual norm 5.498255644601e-08 108 KSP Residual norm 5.268534252790e-08 109 KSP Residual norm 5.047397461135e-08 110 KSP Residual norm 4.837685335912e-08 111 KSP Residual norm 4.671237966029e-08 112 KSP Residual norm 4.526147468824e-08 113 KSP Residual norm 4.407392671549e-08 114 KSP Residual norm 4.311128027127e-08 115 KSP Residual norm 4.201028578940e-08 116 KSP Residual norm 4.072572866082e-08 117 KSP Residual norm 3.921935688079e-08 118 KSP Residual norm 3.776903357828e-08 119 KSP Residual norm 3.655971094717e-08 120 KSP Residual norm 3.561727107948e-08 121 KSP Residual norm 3.507198004135e-08 122 KSP Residual norm 3.447603239222e-08 123 KSP Residual norm 3.220921730480e-08 124 KSP Residual norm 3.008323957181e-08 125 KSP Residual norm 2.864731522184e-08 126 KSP Residual norm 2.737033676857e-08 127 KSP Residual norm 2.594017648802e-08 128 KSP Residual norm 2.458931708649e-08 129 KSP Residual norm 2.341243352765e-08 130 KSP Residual norm 2.250881820846e-08 131 KSP Residual norm 2.178189006707e-08 132 KSP Residual norm 2.100699787623e-08 133 KSP Residual norm 2.015372416577e-08 134 KSP Residual norm 1.950061248967e-08 135 KSP Residual norm 1.893837357493e-08 136 KSP Residual norm 1.822403233945e-08 137 KSP Residual norm 1.760280328619e-08 138 KSP Residual norm 1.723225781423e-08 139 KSP Residual norm 1.699024250960e-08 140 KSP Residual norm 1.678906434889e-08 141 KSP Residual norm 1.658921760292e-08 142 KSP Residual norm 1.638932495379e-08 143 KSP Residual norm 1.615127678532e-08 144 KSP Residual norm 1.598479679593e-08 145 KSP Residual norm 1.594442399283e-08 146 KSP Residual norm 1.591806579538e-08 147 KSP Residual norm 1.588564227413e-08 148 KSP Residual norm 1.585208776898e-08 149 KSP Residual norm 1.573202997142e-08 150 KSP Residual norm 1.551482804766e-08 151 KSP Residual norm 1.531241149845e-08 152 KSP Residual norm 1.517203512514e-08 153 KSP Residual norm 1.512405117799e-08 154 KSP Residual norm 1.494709857196e-08 155 KSP Residual norm 1.443987053051e-08 156 KSP Residual norm 1.388065771425e-08 157 KSP Residual norm 1.338444995704e-08 158 KSP Residual norm 1.281943564419e-08 159 KSP Residual norm 1.221554058763e-08 160 KSP Residual norm 1.167221242525e-08 161 KSP Residual norm 1.121032267440e-08 162 KSP Residual norm 1.077591035965e-08 163 KSP Residual norm 1.028039469768e-08 164 KSP Residual norm 9.710014443432e-09 165 KSP Residual norm 9.120216108941e-09 166 KSP Residual norm 8.562926309810e-09 167 KSP Residual norm 8.073213145679e-09 168 KSP Residual norm 7.653401100859e-09 169 KSP Residual norm 7.287994030094e-09 170 KSP Residual norm 6.967247478244e-09 171 KSP Residual norm 6.714202862140e-09 172 KSP Residual norm 6.541129987340e-09 173 KSP Residual norm 6.316750402093e-09 174 KSP Residual norm 6.005451415985e-09 175 KSP Residual norm 5.725683734009e-09 176 KSP Residual norm 5.489185013805e-09 177 KSP Residual norm 5.267057849304e-09 178 KSP Residual norm 5.061530591600e-09 179 KSP Residual norm 4.892440920253e-09 180 KSP Residual norm 4.739469260982e-09 181 KSP Residual norm 4.621610988363e-09 182 KSP Residual norm 4.580449056421e-09 183 KSP Residual norm 4.574093172689e-09 184 KSP Residual norm 4.419068407344e-09 185 KSP Residual norm 4.215219891333e-09 186 KSP Residual norm 4.095353742483e-09 187 KSP Residual norm 4.028919234503e-09 188 KSP Residual norm 3.982730273276e-09 189 KSP Residual norm 3.955186655898e-09 190 KSP Residual norm 3.939559497361e-09 191 KSP Residual norm 3.914144506646e-09 192 KSP Residual norm 3.858421168214e-09 193 KSP Residual norm 3.784319036247e-09 194 KSP Residual norm 3.717169024542e-09 195 KSP Residual norm 3.661304277963e-09 196 KSP Residual norm 3.599925777336e-09 197 KSP Residual norm 3.533837304288e-09 198 KSP Residual norm 3.466972405681e-09 199 KSP Residual norm 3.380036062328e-09 200 KSP Residual norm 3.276915036967e-09 201 KSP Residual norm 3.172338849418e-09 202 KSP Residual norm 3.083930723439e-09 203 KSP Residual norm 2.987739777403e-09 204 KSP Residual norm 2.877471647472e-09 205 KSP Residual norm 2.764507689262e-09 206 KSP Residual norm 2.655507309830e-09 207 KSP Residual norm 2.541255400924e-09 208 KSP Residual norm 2.420631533963e-09 209 KSP Residual norm 2.304229683933e-09 210 KSP Residual norm 2.198885017399e-09 211 KSP Residual norm 2.099285276249e-09 212 KSP Residual norm 2.006650438511e-09 213 KSP Residual norm 1.925924752138e-09 214 KSP Residual norm 1.873323061695e-09 215 KSP Residual norm 1.848001579692e-09 216 KSP Residual norm 1.789274059132e-09 217 KSP Residual norm 1.687983742960e-09 218 KSP Residual norm 1.611541277691e-09 219 KSP Residual norm 1.555819838189e-09 220 KSP Residual norm 1.500444983759e-09 221 KSP Residual norm 1.441128069787e-09 222 KSP Residual norm 1.387157217671e-09 223 KSP Residual norm 1.337777263415e-09 224 KSP Residual norm 1.293648135042e-09 225 KSP Residual norm 1.253706328712e-09 226 KSP Residual norm 1.222271580010e-09 Linear tao_nls_ solve converged due to CONVERGED_RTOL iterations 226 0 LS Function value: 0.0691392, Step length: 0. 1 LS Function value: 0.0691391, Step length: 1. stx: 0., fx: 0.0691392, dgx: -5.95363e-08 sty: 0., fy: 0.0691392, dgy: -5.95363e-08 29 TAO, Function value: 0.0691391, Residual: 2.98657e-05 Residual norms for tao_nls_ solve. 0 KSP Residual norm 2.986569367135e-05 1 KSP Residual norm 2.595526526105e-05 2 KSP Residual norm 1.990068950020e-05 3 KSP Residual norm 1.874588602695e-05 4 KSP Residual norm 1.721191778239e-05 5 KSP Residual norm 1.856204638573e-05 6 KSP Residual norm 2.041009965410e-05 7 KSP Residual norm 2.313933354203e-05 8 KSP Residual norm 2.596268625346e-05 9 KSP Residual norm 2.878721277410e-05 10 KSP Residual norm 3.196694097186e-05 11 KSP Residual norm 3.528885389747e-05 12 KSP Residual norm 3.858754455075e-05 13 KSP Residual norm 4.207841763633e-05 14 KSP Residual norm 4.392571904962e-05 15 KSP Residual norm 4.613097451407e-05 16 KSP Residual norm 4.887023147561e-05 17 KSP Residual norm 5.093432967146e-05 18 KSP Residual norm 5.181839806152e-05 19 KSP Residual norm 5.228128440104e-05 20 KSP Residual norm 5.226881147043e-05 21 KSP Residual norm 5.172899960549e-05 22 KSP Residual norm 5.101465835856e-05 23 KSP Residual norm 5.017119318362e-05 24 KSP Residual norm 4.865286014500e-05 25 KSP Residual norm 4.672749949957e-05 26 KSP Residual norm 4.464851697067e-05 27 KSP Residual norm 4.216106130358e-05 28 KSP Residual norm 3.933065082793e-05 29 KSP Residual norm 3.633402608835e-05 30 KSP Residual norm 3.327314896415e-05 31 KSP Residual norm 3.029655560928e-05 32 KSP Residual norm 2.744137629453e-05 33 KSP Residual norm 2.466312676581e-05 34 KSP Residual norm 2.204423231726e-05 35 KSP Residual norm 1.972573227358e-05 36 KSP Residual norm 1.771977947740e-05 37 KSP Residual norm 1.595528034477e-05 38 KSP Residual norm 1.441939445351e-05 39 KSP Residual norm 1.311700250434e-05 40 KSP Residual norm 1.203427041387e-05 41 KSP Residual norm 1.096531962900e-05 42 KSP Residual norm 9.858553292231e-06 43 KSP Residual norm 8.875923013598e-06 44 KSP Residual norm 8.082877884596e-06 45 KSP Residual norm 7.489168356530e-06 46 KSP Residual norm 6.915617638760e-06 47 KSP Residual norm 6.150924265951e-06 48 KSP Residual norm 5.400454749171e-06 49 KSP Residual norm 4.774769735735e-06 50 KSP Residual norm 4.241643830077e-06 51 KSP Residual norm 3.785354132251e-06 52 KSP Residual norm 3.396153705452e-06 53 KSP Residual norm 3.050270479849e-06 54 KSP Residual norm 2.732789041681e-06 55 KSP Residual norm 2.451279191412e-06 56 KSP Residual norm 2.227060534039e-06 57 KSP Residual norm 2.053081212963e-06 58 KSP Residual norm 1.900719442117e-06 59 KSP Residual norm 1.757193851978e-06 60 KSP Residual norm 1.632969909671e-06 61 KSP Residual norm 1.532639254660e-06 62 KSP Residual norm 1.450780455529e-06 63 KSP Residual norm 1.381420057580e-06 64 KSP Residual norm 1.325202434840e-06 65 KSP Residual norm 1.279950191216e-06 66 KSP Residual norm 1.220981472834e-06 67 KSP Residual norm 1.145566375755e-06 68 KSP Residual norm 1.077126201454e-06 69 KSP Residual norm 1.021288578843e-06 70 KSP Residual norm 9.680038641038e-07 71 KSP Residual norm 9.195698131525e-07 72 KSP Residual norm 8.745251715495e-07 73 KSP Residual norm 8.221506334905e-07 74 KSP Residual norm 7.701444031592e-07 75 KSP Residual norm 7.277826627396e-07 76 KSP Residual norm 6.952837218429e-07 77 KSP Residual norm 6.762212052826e-07 78 KSP Residual norm 6.313711147809e-07 79 KSP Residual norm 5.700883161495e-07 80 KSP Residual norm 5.252858782972e-07 81 KSP Residual norm 4.885795969737e-07 82 KSP Residual norm 4.582697298344e-07 83 KSP Residual norm 4.317310030577e-07 84 KSP Residual norm 4.003068193432e-07 85 KSP Residual norm 3.712293512717e-07 86 KSP Residual norm 3.480378792972e-07 87 KSP Residual norm 3.278238408495e-07 88 KSP Residual norm 3.119164852519e-07 89 KSP Residual norm 2.998585949321e-07 90 KSP Residual norm 2.895443452697e-07 91 KSP Residual norm 2.774283459752e-07 92 KSP Residual norm 2.680175233485e-07 93 KSP Residual norm 2.610465654805e-07 94 KSP Residual norm 2.498096808308e-07 95 KSP Residual norm 2.396452293554e-07 96 KSP Residual norm 2.304902008050e-07 97 KSP Residual norm 2.232826073294e-07 98 KSP Residual norm 2.190382256332e-07 99 KSP Residual norm 2.154912750879e-07 100 KSP Residual norm 2.123263422899e-07 101 KSP Residual norm 2.076268249067e-07 102 KSP Residual norm 2.003943329163e-07 103 KSP Residual norm 1.964959750018e-07 104 KSP Residual norm 1.917095137952e-07 105 KSP Residual norm 1.880945966304e-07 106 KSP Residual norm 1.840693915192e-07 107 KSP Residual norm 1.810660055430e-07 108 KSP Residual norm 1.786527568866e-07 109 KSP Residual norm 1.759239306357e-07 110 KSP Residual norm 1.729473647031e-07 111 KSP Residual norm 1.713213186822e-07 112 KSP Residual norm 1.702112582347e-07 113 KSP Residual norm 1.694723524965e-07 114 KSP Residual norm 1.683932378830e-07 115 KSP Residual norm 1.667220723653e-07 116 KSP Residual norm 1.644707737681e-07 117 KSP Residual norm 1.616536573179e-07 118 KSP Residual norm 1.582452410905e-07 119 KSP Residual norm 1.544461668646e-07 120 KSP Residual norm 1.500187836895e-07 121 KSP Residual norm 1.441641841465e-07 122 KSP Residual norm 1.386245513213e-07 123 KSP Residual norm 1.330853685464e-07 124 KSP Residual norm 1.271020366397e-07 125 KSP Residual norm 1.205702225422e-07 126 KSP Residual norm 1.144672198735e-07 127 KSP Residual norm 1.087155086061e-07 128 KSP Residual norm 1.042898177552e-07 129 KSP Residual norm 1.009533167426e-07 130 KSP Residual norm 9.830246871261e-08 131 KSP Residual norm 9.551790449286e-08 132 KSP Residual norm 9.365976519244e-08 133 KSP Residual norm 9.261581906854e-08 134 KSP Residual norm 9.014690908567e-08 135 KSP Residual norm 8.697202340979e-08 136 KSP Residual norm 8.510809933245e-08 137 KSP Residual norm 8.410946928279e-08 138 KSP Residual norm 8.339834420332e-08 139 KSP Residual norm 8.324669437299e-08 140 KSP Residual norm 8.379439549927e-08 141 KSP Residual norm 8.430018231781e-08 142 KSP Residual norm 8.390361831469e-08 143 KSP Residual norm 8.418641967091e-08 144 KSP Residual norm 8.449398110672e-08 145 KSP Residual norm 8.261830807843e-08 146 KSP Residual norm 8.028342117978e-08 147 KSP Residual norm 7.881333339787e-08 148 KSP Residual norm 7.783369413151e-08 149 KSP Residual norm 7.662920572555e-08 150 KSP Residual norm 7.520803988870e-08 151 KSP Residual norm 7.391208723800e-08 152 KSP Residual norm 7.242696329530e-08 153 KSP Residual norm 7.023156066593e-08 154 KSP Residual norm 6.777891482865e-08 155 KSP Residual norm 6.573731774839e-08 156 KSP Residual norm 6.420407819341e-08 157 KSP Residual norm 6.260445632359e-08 158 KSP Residual norm 6.044145630012e-08 159 KSP Residual norm 5.816805766145e-08 160 KSP Residual norm 5.604426179792e-08 161 KSP Residual norm 5.410105517559e-08 162 KSP Residual norm 5.219341253097e-08 163 KSP Residual norm 5.007147620614e-08 164 KSP Residual norm 4.783188671859e-08 165 KSP Residual norm 4.562051376200e-08 166 KSP Residual norm 4.352133298894e-08 167 KSP Residual norm 4.157569373668e-08 168 KSP Residual norm 3.943696913260e-08 169 KSP Residual norm 3.681342907875e-08 170 KSP Residual norm 3.425445247595e-08 171 KSP Residual norm 3.220667876445e-08 172 KSP Residual norm 3.069754014517e-08 173 KSP Residual norm 2.940182568977e-08 174 KSP Residual norm 2.783098887072e-08 175 KSP Residual norm 2.619581779821e-08 176 KSP Residual norm 2.489473944409e-08 177 KSP Residual norm 2.394671472918e-08 178 KSP Residual norm 2.327904915577e-08 179 KSP Residual norm 2.270217432290e-08 180 KSP Residual norm 2.178458683430e-08 181 KSP Residual norm 2.087803652101e-08 182 KSP Residual norm 2.026180482200e-08 183 KSP Residual norm 1.977983093770e-08 184 KSP Residual norm 1.922769654381e-08 185 KSP Residual norm 1.867132337033e-08 186 KSP Residual norm 1.826951175991e-08 187 KSP Residual norm 1.805114646616e-08 188 KSP Residual norm 1.791914094213e-08 189 KSP Residual norm 1.764199464129e-08 190 KSP Residual norm 1.714590820310e-08 191 KSP Residual norm 1.665003009872e-08 192 KSP Residual norm 1.622764964727e-08 193 KSP Residual norm 1.586720645106e-08 194 KSP Residual norm 1.555385887773e-08 195 KSP Residual norm 1.527792442082e-08 196 KSP Residual norm 1.499684275532e-08 197 KSP Residual norm 1.477251846762e-08 198 KSP Residual norm 1.463456548823e-08 199 KSP Residual norm 1.453936811347e-08 200 KSP Residual norm 1.449489136060e-08 201 KSP Residual norm 1.446815667230e-08 202 KSP Residual norm 1.434558038069e-08 203 KSP Residual norm 1.399866328430e-08 204 KSP Residual norm 1.355854840704e-08 205 KSP Residual norm 1.317003485476e-08 206 KSP Residual norm 1.287240046173e-08 207 KSP Residual norm 1.268241632281e-08 208 KSP Residual norm 1.259953842790e-08 209 KSP Residual norm 1.228784446212e-08 210 KSP Residual norm 1.156585550912e-08 211 KSP Residual norm 1.092126363767e-08 212 KSP Residual norm 1.047998985786e-08 213 KSP Residual norm 1.019915131065e-08 214 KSP Residual norm 9.982091040398e-09 215 KSP Residual norm 9.730771722168e-09 216 KSP Residual norm 9.445692835213e-09 217 KSP Residual norm 9.165463792831e-09 218 KSP Residual norm 8.905274114144e-09 219 KSP Residual norm 8.640191536690e-09 220 KSP Residual norm 8.364642414008e-09 221 KSP Residual norm 8.088454623572e-09 222 KSP Residual norm 7.835831379912e-09 223 KSP Residual norm 7.566801856292e-09 224 KSP Residual norm 7.294968786541e-09 225 KSP Residual norm 7.071422420671e-09 226 KSP Residual norm 6.933246978305e-09 227 KSP Residual norm 6.715103811148e-09 228 KSP Residual norm 6.388348903709e-09 229 KSP Residual norm 6.111242314098e-09 230 KSP Residual norm 5.887534530818e-09 231 KSP Residual norm 5.691939737065e-09 232 KSP Residual norm 5.568152171336e-09 233 KSP Residual norm 5.474695505636e-09 234 KSP Residual norm 5.271362372050e-09 235 KSP Residual norm 5.002737714046e-09 236 KSP Residual norm 4.781106943287e-09 237 KSP Residual norm 4.607040792510e-09 238 KSP Residual norm 4.438309554628e-09 239 KSP Residual norm 4.264097828137e-09 240 KSP Residual norm 4.107958255725e-09 241 KSP Residual norm 3.966681191453e-09 242 KSP Residual norm 3.814521718489e-09 243 KSP Residual norm 3.654407858209e-09 244 KSP Residual norm 3.507088586940e-09 245 KSP Residual norm 3.379796151754e-09 246 KSP Residual norm 3.276269027736e-09 247 KSP Residual norm 3.190837377562e-09 248 KSP Residual norm 3.105746714786e-09 249 KSP Residual norm 3.012500121939e-09 250 KSP Residual norm 2.916756095652e-09 251 KSP Residual norm 2.832036786955e-09 252 KSP Residual norm 2.779856707851e-09 253 KSP Residual norm 2.784675929406e-09 254 KSP Residual norm 2.803107521693e-09 255 KSP Residual norm 2.702503322505e-09 256 KSP Residual norm 2.577072439641e-09 257 KSP Residual norm 2.500748184691e-09 258 KSP Residual norm 2.444995817996e-09 259 KSP Residual norm 2.393672183276e-09 260 KSP Residual norm 2.338120619107e-09 261 KSP Residual norm 2.287150221291e-09 262 KSP Residual norm 2.242368007517e-09 263 KSP Residual norm 2.204282214026e-09 264 KSP Residual norm 2.162024549872e-09 265 KSP Residual norm 2.117089297046e-09 266 KSP Residual norm 2.086102914920e-09 267 KSP Residual norm 2.074850496906e-09 268 KSP Residual norm 2.097930710423e-09 269 KSP Residual norm 2.089287032548e-09 270 KSP Residual norm 2.007883910590e-09 271 KSP Residual norm 1.938638339347e-09 272 KSP Residual norm 1.911740221027e-09 273 KSP Residual norm 1.903592394501e-09 274 KSP Residual norm 1.853292437960e-09 275 KSP Residual norm 1.769624153321e-09 276 KSP Residual norm 1.703351861929e-09 277 KSP Residual norm 1.649031355116e-09 278 KSP Residual norm 1.601390282650e-09 279 KSP Residual norm 1.551942696196e-09 280 KSP Residual norm 1.503100477804e-09 281 KSP Residual norm 1.451310442736e-09 282 KSP Residual norm 1.389474574350e-09 283 KSP Residual norm 1.328901152718e-09 284 KSP Residual norm 1.278186679527e-09 285 KSP Residual norm 1.234145419156e-09 286 KSP Residual norm 1.198449700223e-09 287 KSP Residual norm 1.159635672120e-09 288 KSP Residual norm 1.114897573311e-09 289 KSP Residual norm 1.073669005337e-09 290 KSP Residual norm 1.038859658520e-09 291 KSP Residual norm 1.001978607708e-09 292 KSP Residual norm 9.642672387752e-10 293 KSP Residual norm 9.289908584018e-10 294 KSP Residual norm 8.946279454036e-10 295 KSP Residual norm 8.546947047084e-10 296 KSP Residual norm 8.151445130967e-10 297 KSP Residual norm 7.810102802125e-10 298 KSP Residual norm 7.546219922100e-10 299 KSP Residual norm 7.315576456805e-10 300 KSP Residual norm 7.118772377037e-10 301 KSP Residual norm 6.924469334784e-10 302 KSP Residual norm 6.770690065034e-10 303 KSP Residual norm 6.661251767766e-10 304 KSP Residual norm 6.501752931318e-10 305 KSP Residual norm 6.335959697335e-10 306 KSP Residual norm 6.167662292759e-10 307 KSP Residual norm 6.024933555753e-10 308 KSP Residual norm 5.902712175221e-10 309 KSP Residual norm 5.757304673271e-10 310 KSP Residual norm 5.571403387221e-10 311 KSP Residual norm 5.402982333906e-10 312 KSP Residual norm 5.273451916868e-10 313 KSP Residual norm 5.159299588792e-10 314 KSP Residual norm 5.079527784419e-10 315 KSP Residual norm 5.050731438446e-10 316 KSP Residual norm 4.949300070390e-10 317 KSP Residual norm 4.752313100445e-10 318 KSP Residual norm 4.563485814745e-10 319 KSP Residual norm 4.418845071846e-10 320 KSP Residual norm 4.315686484517e-10 321 KSP Residual norm 4.263463768433e-10 322 KSP Residual norm 4.238399671162e-10 323 KSP Residual norm 4.142929083200e-10 324 KSP Residual norm 3.989843941547e-10 325 KSP Residual norm 3.871260316737e-10 326 KSP Residual norm 3.772347312826e-10 327 KSP Residual norm 3.689217616731e-10 328 KSP Residual norm 3.620958258016e-10 329 KSP Residual norm 3.554100967543e-10 330 KSP Residual norm 3.505949276508e-10 331 KSP Residual norm 3.464002702944e-10 332 KSP Residual norm 3.388286100272e-10 333 KSP Residual norm 3.296250178380e-10 334 KSP Residual norm 3.251306215011e-10 335 KSP Residual norm 3.229235205434e-10 336 KSP Residual norm 3.081933899545e-10 337 KSP Residual norm 2.912778263336e-10 Linear tao_nls_ solve converged due to CONVERGED_RTOL iterations 337 0 LS Function value: 0.0691391, Step length: 0. 1 LS Function value: 0.0691392, Step length: 1. stx: 0., fx: 0.0691391, dgx: -6.2315e-08 sty: 0., fy: 0.0691391, dgy: -6.2315e-08 2 LS Function value: 0.0691391, Step length: 0.358118 stx: 0., fx: 0.0691391, dgx: -6.2315e-08 sty: 1., fy: 0.0691392, dgy: 4.16778e-07 30 TAO, Function value: 0.0691391, Residual: 5.74416e-05 Residual norms for tao_nls_ solve. 0 KSP Residual norm 5.744155984375e-05 1 KSP Residual norm 4.574098054832e-05 2 KSP Residual norm 3.206194573338e-05 3 KSP Residual norm 2.656580543878e-05 4 KSP Residual norm 2.140903603801e-05 5 KSP Residual norm 1.960244116657e-05 6 KSP Residual norm 1.867782465859e-05 7 KSP Residual norm 1.885411926172e-05 8 KSP Residual norm 1.969381764365e-05 9 KSP Residual norm 2.082184272116e-05 10 KSP Residual norm 2.210843349162e-05 11 KSP Residual norm 2.339867857942e-05 12 KSP Residual norm 2.459903809059e-05 13 KSP Residual norm 2.607222898859e-05 14 KSP Residual norm 2.749511237411e-05 15 KSP Residual norm 2.913446580297e-05 16 KSP Residual norm 3.097363109462e-05 17 KSP Residual norm 3.277562012921e-05 18 KSP Residual norm 3.425992704348e-05 19 KSP Residual norm 3.543584204851e-05 20 KSP Residual norm 3.637621663188e-05 21 KSP Residual norm 3.691508333103e-05 22 KSP Residual norm 3.666594199581e-05 23 KSP Residual norm 3.606294942902e-05 24 KSP Residual norm 3.520530428025e-05 25 KSP Residual norm 3.422462460364e-05 26 KSP Residual norm 3.332597698370e-05 27 KSP Residual norm 3.197041379491e-05 28 KSP Residual norm 3.008226342066e-05 29 KSP Residual norm 2.805170955198e-05 30 KSP Residual norm 2.608152166511e-05 31 KSP Residual norm 2.417722747892e-05 32 KSP Residual norm 2.222248064700e-05 33 KSP Residual norm 2.022787237623e-05 34 KSP Residual norm 1.829279738636e-05 35 KSP Residual norm 1.649251365359e-05 36 KSP Residual norm 1.482568854653e-05 37 KSP Residual norm 1.327889620173e-05 38 KSP Residual norm 1.186152723967e-05 39 KSP Residual norm 1.055728026849e-05 40 KSP Residual norm 9.422551042173e-06 41 KSP Residual norm 8.494568664693e-06 42 KSP Residual norm 7.784718389064e-06 43 KSP Residual norm 7.195336259481e-06 44 KSP Residual norm 6.504915794955e-06 45 KSP Residual norm 5.806226972156e-06 46 KSP Residual norm 5.244671692335e-06 47 KSP Residual norm 4.816913847470e-06 48 KSP Residual norm 4.418089555029e-06 49 KSP Residual norm 3.945002586828e-06 50 KSP Residual norm 3.474967390403e-06 51 KSP Residual norm 3.075923135629e-06 52 KSP Residual norm 2.735705472168e-06 53 KSP Residual norm 2.432306591227e-06 54 KSP Residual norm 2.157491510760e-06 55 KSP Residual norm 1.911858328936e-06 56 KSP Residual norm 1.708502772479e-06 57 KSP Residual norm 1.551736876527e-06 58 KSP Residual norm 1.428518565556e-06 59 KSP Residual norm 1.312430213751e-06 60 KSP Residual norm 1.198688274316e-06 61 KSP Residual norm 1.100508845001e-06 62 KSP Residual norm 1.019226534034e-06 63 KSP Residual norm 9.496717308195e-07 64 KSP Residual norm 8.905235972921e-07 65 KSP Residual norm 8.448546233914e-07 66 KSP Residual norm 8.122700090586e-07 67 KSP Residual norm 7.782408180783e-07 68 KSP Residual norm 7.280777632273e-07 69 KSP Residual norm 6.817993063039e-07 70 KSP Residual norm 6.479469577686e-07 71 KSP Residual norm 6.344092216777e-07 72 KSP Residual norm 6.319896253981e-07 73 KSP Residual norm 5.901444552501e-07 74 KSP Residual norm 5.428189987212e-07 75 KSP Residual norm 5.137565867272e-07 76 KSP Residual norm 4.877908202744e-07 77 KSP Residual norm 4.605077853257e-07 78 KSP Residual norm 4.359335718522e-07 79 KSP Residual norm 4.109873184155e-07 80 KSP Residual norm 3.852490403556e-07 81 KSP Residual norm 3.584022610545e-07 82 KSP Residual norm 3.354435415452e-07 83 KSP Residual norm 3.181294942547e-07 84 KSP Residual norm 2.984161662391e-07 85 KSP Residual norm 2.779916905412e-07 86 KSP Residual norm 2.607791509605e-07 87 KSP Residual norm 2.454302753683e-07 88 KSP Residual norm 2.329794315202e-07 89 KSP Residual norm 2.245576931865e-07 90 KSP Residual norm 2.189544798975e-07 91 KSP Residual norm 2.087174249510e-07 92 KSP Residual norm 1.972844830618e-07 93 KSP Residual norm 1.893134954677e-07 94 KSP Residual norm 1.808102735000e-07 95 KSP Residual norm 1.736588566413e-07 96 KSP Residual norm 1.674684018692e-07 97 KSP Residual norm 1.620287681897e-07 98 KSP Residual norm 1.582495224102e-07 99 KSP Residual norm 1.541432861232e-07 100 KSP Residual norm 1.507444440536e-07 101 KSP Residual norm 1.482524728395e-07 102 KSP Residual norm 1.441404555932e-07 103 KSP Residual norm 1.416284343134e-07 104 KSP Residual norm 1.381390813085e-07 105 KSP Residual norm 1.352716410763e-07 106 KSP Residual norm 1.325099638602e-07 107 KSP Residual norm 1.295061847603e-07 108 KSP Residual norm 1.268018504791e-07 109 KSP Residual norm 1.240575514187e-07 110 KSP Residual norm 1.221250645329e-07 111 KSP Residual norm 1.213615365086e-07 112 KSP Residual norm 1.213121758092e-07 113 KSP Residual norm 1.213128688400e-07 114 KSP Residual norm 1.211002242509e-07 115 KSP Residual norm 1.202297437737e-07 116 KSP Residual norm 1.190223855011e-07 117 KSP Residual norm 1.175740900837e-07 118 KSP Residual norm 1.157731780475e-07 119 KSP Residual norm 1.137474746189e-07 120 KSP Residual norm 1.114121922966e-07 121 KSP Residual norm 1.083203760602e-07 122 KSP Residual norm 1.054187466088e-07 123 KSP Residual norm 1.016952512521e-07 124 KSP Residual norm 9.682314410664e-08 125 KSP Residual norm 9.178472598859e-08 126 KSP Residual norm 8.759406333441e-08 127 KSP Residual norm 8.390831929696e-08 128 KSP Residual norm 8.084207260405e-08 129 KSP Residual norm 7.716476302551e-08 130 KSP Residual norm 7.320798978617e-08 131 KSP Residual norm 7.052230911474e-08 132 KSP Residual norm 7.011395257791e-08 133 KSP Residual norm 6.809807968324e-08 134 KSP Residual norm 6.357622225647e-08 135 KSP Residual norm 6.082218978235e-08 136 KSP Residual norm 5.978644611610e-08 137 KSP Residual norm 5.974199731633e-08 138 KSP Residual norm 5.951285059376e-08 139 KSP Residual norm 5.776645009279e-08 140 KSP Residual norm 5.593737049461e-08 141 KSP Residual norm 5.490739668261e-08 142 KSP Residual norm 5.421053508634e-08 143 KSP Residual norm 5.339097663976e-08 144 KSP Residual norm 5.266170206649e-08 145 KSP Residual norm 5.232576403276e-08 146 KSP Residual norm 5.209111246457e-08 147 KSP Residual norm 5.188991411813e-08 148 KSP Residual norm 5.174812886464e-08 149 KSP Residual norm 5.138581205273e-08 150 KSP Residual norm 5.074769003999e-08 151 KSP Residual norm 5.003625239760e-08 152 KSP Residual norm 4.916530987957e-08 153 KSP Residual norm 4.809424562204e-08 154 KSP Residual norm 4.704475558698e-08 155 KSP Residual norm 4.611683251267e-08 156 KSP Residual norm 4.538319166943e-08 157 KSP Residual norm 4.458327929260e-08 158 KSP Residual norm 4.336045528906e-08 159 KSP Residual norm 4.200345233499e-08 160 KSP Residual norm 4.073931929008e-08 161 KSP Residual norm 3.955478411957e-08 162 KSP Residual norm 3.829745197289e-08 163 KSP Residual norm 3.676090999631e-08 164 KSP Residual norm 3.511647400340e-08 165 KSP Residual norm 3.363362032187e-08 166 KSP Residual norm 3.249672830671e-08 167 KSP Residual norm 3.157543809402e-08 168 KSP Residual norm 2.996400829485e-08 169 KSP Residual norm 2.773609381039e-08 170 KSP Residual norm 2.587133026747e-08 171 KSP Residual norm 2.451519655352e-08 172 KSP Residual norm 2.350708863645e-08 173 KSP Residual norm 2.236461307953e-08 174 KSP Residual norm 2.094116113038e-08 175 KSP Residual norm 1.964960156116e-08 176 KSP Residual norm 1.854985015498e-08 177 KSP Residual norm 1.753766836295e-08 178 KSP Residual norm 1.664870145853e-08 179 KSP Residual norm 1.601835176039e-08 180 KSP Residual norm 1.550101674273e-08 181 KSP Residual norm 1.486243373265e-08 182 KSP Residual norm 1.421398061537e-08 183 KSP Residual norm 1.371033361676e-08 184 KSP Residual norm 1.328653580136e-08 185 KSP Residual norm 1.289907413287e-08 186 KSP Residual norm 1.256169016211e-08 187 KSP Residual norm 1.231361494535e-08 188 KSP Residual norm 1.217907478802e-08 189 KSP Residual norm 1.215209003511e-08 190 KSP Residual norm 1.202751277096e-08 191 KSP Residual norm 1.172360429191e-08 192 KSP Residual norm 1.142844067359e-08 193 KSP Residual norm 1.123626444998e-08 194 KSP Residual norm 1.115165170747e-08 195 KSP Residual norm 1.115047884510e-08 196 KSP Residual norm 1.115201543370e-08 197 KSP Residual norm 1.122388855012e-08 198 KSP Residual norm 1.138273156811e-08 199 KSP Residual norm 1.122263772400e-08 200 KSP Residual norm 1.085383138149e-08 201 KSP Residual norm 1.059878112315e-08 202 KSP Residual norm 1.045277449795e-08 203 KSP Residual norm 1.032184925205e-08 204 KSP Residual norm 1.019746331553e-08 205 KSP Residual norm 1.008810268431e-08 206 KSP Residual norm 9.909818948038e-09 207 KSP Residual norm 9.595552881197e-09 208 KSP Residual norm 9.232353327862e-09 209 KSP Residual norm 8.895403860390e-09 210 KSP Residual norm 8.554195869078e-09 211 KSP Residual norm 8.202891575064e-09 212 KSP Residual norm 7.865518280805e-09 213 KSP Residual norm 7.573644457227e-09 214 KSP Residual norm 7.293157622757e-09 215 KSP Residual norm 6.986475249545e-09 216 KSP Residual norm 6.676673363763e-09 217 KSP Residual norm 6.388765857655e-09 218 KSP Residual norm 6.125847091186e-09 219 KSP Residual norm 5.879963259178e-09 220 KSP Residual norm 5.668413371402e-09 221 KSP Residual norm 5.529159293639e-09 222 KSP Residual norm 5.390502026258e-09 223 KSP Residual norm 5.074991777512e-09 224 KSP Residual norm 4.753214067017e-09 225 KSP Residual norm 4.513181492270e-09 226 KSP Residual norm 4.287368043541e-09 227 KSP Residual norm 4.051155284028e-09 228 KSP Residual norm 3.855660541241e-09 229 KSP Residual norm 3.724774652460e-09 230 KSP Residual norm 3.644445635704e-09 231 KSP Residual norm 3.544401406353e-09 232 KSP Residual norm 3.404797102851e-09 233 KSP Residual norm 3.274344612976e-09 234 KSP Residual norm 3.165132269954e-09 235 KSP Residual norm 3.070508141450e-09 236 KSP Residual norm 2.994168141319e-09 237 KSP Residual norm 2.924508843837e-09 238 KSP Residual norm 2.839149759760e-09 239 KSP Residual norm 2.750949384651e-09 240 KSP Residual norm 2.679818707225e-09 241 KSP Residual norm 2.619889291261e-09 242 KSP Residual norm 2.555861548388e-09 243 KSP Residual norm 2.489122750488e-09 244 KSP Residual norm 2.426532724246e-09 245 KSP Residual norm 2.369913586346e-09 246 KSP Residual norm 2.326491456538e-09 247 KSP Residual norm 2.310714273344e-09 248 KSP Residual norm 2.314616812779e-09 249 KSP Residual norm 2.258060528222e-09 250 KSP Residual norm 2.153327883758e-09 251 KSP Residual norm 2.080426978381e-09 252 KSP Residual norm 2.038124355448e-09 253 KSP Residual norm 2.008059514892e-09 254 KSP Residual norm 1.981549511548e-09 255 KSP Residual norm 1.954857165103e-09 256 KSP Residual norm 1.949064522269e-09 257 KSP Residual norm 1.969841923837e-09 258 KSP Residual norm 1.926670180083e-09 259 KSP Residual norm 1.824233743638e-09 260 KSP Residual norm 1.743447674580e-09 261 KSP Residual norm 1.684234722445e-09 262 KSP Residual norm 1.629196798185e-09 263 KSP Residual norm 1.579141417325e-09 264 KSP Residual norm 1.535224543745e-09 265 KSP Residual norm 1.495644732997e-09 266 KSP Residual norm 1.455539187599e-09 267 KSP Residual norm 1.406236763883e-09 268 KSP Residual norm 1.357808233186e-09 269 KSP Residual norm 1.319667568851e-09 270 KSP Residual norm 1.284004210004e-09 271 KSP Residual norm 1.248368795368e-09 272 KSP Residual norm 1.217657448409e-09 273 KSP Residual norm 1.196184006824e-09 274 KSP Residual norm 1.171725551255e-09 275 KSP Residual norm 1.128998118072e-09 276 KSP Residual norm 1.080855302977e-09 277 KSP Residual norm 1.038247238921e-09 278 KSP Residual norm 1.002809510737e-09 279 KSP Residual norm 9.705831156651e-10 280 KSP Residual norm 9.431486490489e-10 281 KSP Residual norm 9.139969737260e-10 282 KSP Residual norm 8.751992860932e-10 283 KSP Residual norm 8.372185543618e-10 284 KSP Residual norm 8.068298463442e-10 285 KSP Residual norm 7.822848536122e-10 286 KSP Residual norm 7.651599444411e-10 287 KSP Residual norm 7.475689899568e-10 288 KSP Residual norm 7.265921274797e-10 289 KSP Residual norm 7.080985819801e-10 290 KSP Residual norm 6.932701623952e-10 291 KSP Residual norm 6.732121112410e-10 292 KSP Residual norm 6.489056705575e-10 293 KSP Residual norm 6.270663820638e-10 294 KSP Residual norm 6.094739714518e-10 295 KSP Residual norm 5.905272853908e-10 296 KSP Residual norm 5.704985658116e-10 Linear tao_nls_ solve converged due to CONVERGED_RTOL iterations 296 0 LS Function value: 0.0691391, Step length: 0. 1 LS Function value: 0.0691391, Step length: 1. stx: 0., fx: 0.0691391, dgx: -3.84523e-08 sty: 0., fy: 0.0691391, dgy: -3.84523e-08 31 TAO, Function value: 0.0691391, Residual: 8.90497e-05 Residual norms for tao_nls_ solve. 0 KSP Residual norm 8.904967389355e-05 1 KSP Residual norm 6.991115065216e-05 2 KSP Residual norm 4.796477991784e-05 3 KSP Residual norm 3.784777696713e-05 4 KSP Residual norm 2.923807179371e-05 5 KSP Residual norm 2.500636728028e-05 6 KSP Residual norm 2.199976528390e-05 7 KSP Residual norm 2.042220599032e-05 8 KSP Residual norm 1.962510383411e-05 9 KSP Residual norm 1.935486458449e-05 10 KSP Residual norm 1.909052895357e-05 11 KSP Residual norm 1.909389939278e-05 12 KSP Residual norm 1.908111684673e-05 13 KSP Residual norm 1.937618542334e-05 14 KSP Residual norm 1.987076122523e-05 15 KSP Residual norm 2.052798987946e-05 16 KSP Residual norm 2.147624764543e-05 17 KSP Residual norm 2.255799011230e-05 18 KSP Residual norm 2.361373685756e-05 19 KSP Residual norm 2.463951799808e-05 20 KSP Residual norm 2.550536688794e-05 21 KSP Residual norm 2.613807251235e-05 22 KSP Residual norm 2.624691078117e-05 23 KSP Residual norm 2.605305955930e-05 24 KSP Residual norm 2.576960256875e-05 25 KSP Residual norm 2.531529540367e-05 26 KSP Residual norm 2.480479683525e-05 27 KSP Residual norm 2.421868140016e-05 28 KSP Residual norm 2.360339360631e-05 29 KSP Residual norm 2.290949453394e-05 30 KSP Residual norm 2.191411605860e-05 31 KSP Residual norm 2.071203269823e-05 32 KSP Residual norm 1.934172824531e-05 33 KSP Residual norm 1.788439322110e-05 34 KSP Residual norm 1.644059557460e-05 35 KSP Residual norm 1.502753803967e-05 36 KSP Residual norm 1.365007429632e-05 37 KSP Residual norm 1.233283709881e-05 38 KSP Residual norm 1.108183661708e-05 39 KSP Residual norm 9.875293813679e-06 40 KSP Residual norm 8.793291184720e-06 41 KSP Residual norm 7.915548666941e-06 42 KSP Residual norm 7.262029034572e-06 43 KSP Residual norm 6.691969642512e-06 44 KSP Residual norm 5.986219233487e-06 45 KSP Residual norm 5.303996112414e-06 46 KSP Residual norm 4.750639537306e-06 47 KSP Residual norm 4.293919582394e-06 48 KSP Residual norm 3.906520946141e-06 49 KSP Residual norm 3.580876699766e-06 50 KSP Residual norm 3.259993208888e-06 51 KSP Residual norm 2.912294103934e-06 52 KSP Residual norm 2.591406227142e-06 53 KSP Residual norm 2.314916418275e-06 54 KSP Residual norm 2.071030099661e-06 55 KSP Residual norm 1.851855131975e-06 56 KSP Residual norm 1.664840369076e-06 57 KSP Residual norm 1.512089092083e-06 58 KSP Residual norm 1.386642474012e-06 59 KSP Residual norm 1.283958565871e-06 60 KSP Residual norm 1.203201865106e-06 61 KSP Residual norm 1.128227259988e-06 62 KSP Residual norm 1.047569506633e-06 63 KSP Residual norm 9.769025646569e-07 64 KSP Residual norm 9.288175520017e-07 65 KSP Residual norm 9.055133230745e-07 66 KSP Residual norm 8.734699839705e-07 67 KSP Residual norm 8.115673110535e-07 68 KSP Residual norm 7.603854582009e-07 69 KSP Residual norm 7.286891956798e-07 70 KSP Residual norm 7.030034178948e-07 71 KSP Residual norm 6.737462251809e-07 72 KSP Residual norm 6.418019786366e-07 73 KSP Residual norm 6.077561495719e-07 74 KSP Residual norm 5.740041217359e-07 75 KSP Residual norm 5.456198174292e-07 76 KSP Residual norm 5.167285027910e-07 77 KSP Residual norm 4.854934952920e-07 78 KSP Residual norm 4.551625429949e-07 79 KSP Residual norm 4.234287260301e-07 80 KSP Residual norm 3.939250224295e-07 81 KSP Residual norm 3.677218338331e-07 82 KSP Residual norm 3.466756535237e-07 83 KSP Residual norm 3.267621505719e-07 84 KSP Residual norm 2.994295816822e-07 85 KSP Residual norm 2.753891273782e-07 86 KSP Residual norm 2.591217305893e-07 87 KSP Residual norm 2.462906204585e-07 88 KSP Residual norm 2.323997991789e-07 89 KSP Residual norm 2.144669082103e-07 90 KSP Residual norm 1.985641721017e-07 91 KSP Residual norm 1.856475905835e-07 92 KSP Residual norm 1.747689047363e-07 93 KSP Residual norm 1.648456096825e-07 94 KSP Residual norm 1.529238042123e-07 95 KSP Residual norm 1.428908733508e-07 96 KSP Residual norm 1.352043527838e-07 97 KSP Residual norm 1.282459155209e-07 98 KSP Residual norm 1.230820153045e-07 99 KSP Residual norm 1.184928536065e-07 100 KSP Residual norm 1.141177712469e-07 101 KSP Residual norm 1.097971956229e-07 102 KSP Residual norm 1.044394192249e-07 103 KSP Residual norm 1.013099349941e-07 104 KSP Residual norm 9.849751684216e-08 105 KSP Residual norm 9.588879787562e-08 106 KSP Residual norm 9.309586508336e-08 107 KSP Residual norm 9.006502615363e-08 108 KSP Residual norm 8.797565776476e-08 109 KSP Residual norm 8.623541829735e-08 110 KSP Residual norm 8.539276717102e-08 111 KSP Residual norm 8.561233763859e-08 112 KSP Residual norm 8.660893793228e-08 113 KSP Residual norm 8.742716762915e-08 114 KSP Residual norm 8.788886317039e-08 115 KSP Residual norm 8.819591390026e-08 116 KSP Residual norm 8.873451127726e-08 117 KSP Residual norm 8.950862444699e-08 118 KSP Residual norm 9.046135054613e-08 119 KSP Residual norm 9.189190930412e-08 120 KSP Residual norm 9.326314718846e-08 121 KSP Residual norm 9.283830670999e-08 122 KSP Residual norm 9.148965813275e-08 123 KSP Residual norm 8.982653213712e-08 124 KSP Residual norm 8.712828256880e-08 125 KSP Residual norm 8.345125294866e-08 126 KSP Residual norm 7.986001445267e-08 127 KSP Residual norm 7.665660914764e-08 128 KSP Residual norm 7.366315776975e-08 129 KSP Residual norm 7.056940766570e-08 130 KSP Residual norm 6.744403449145e-08 131 KSP Residual norm 6.434192984250e-08 132 KSP Residual norm 6.100605208402e-08 133 KSP Residual norm 5.772852537700e-08 134 KSP Residual norm 5.505457514860e-08 135 KSP Residual norm 5.341816834616e-08 136 KSP Residual norm 5.223731762035e-08 137 KSP Residual norm 4.989312905033e-08 138 KSP Residual norm 4.711965245452e-08 139 KSP Residual norm 4.472270319220e-08 140 KSP Residual norm 4.274964295550e-08 141 KSP Residual norm 4.105561386737e-08 142 KSP Residual norm 3.950454623997e-08 143 KSP Residual norm 3.799472610157e-08 144 KSP Residual norm 3.671563758101e-08 145 KSP Residual norm 3.579963833178e-08 146 KSP Residual norm 3.498386803512e-08 147 KSP Residual norm 3.416402964726e-08 148 KSP Residual norm 3.345920321041e-08 149 KSP Residual norm 3.282871892988e-08 150 KSP Residual norm 3.225799089680e-08 151 KSP Residual norm 3.177616883721e-08 152 KSP Residual norm 3.125441486292e-08 153 KSP Residual norm 3.070317462126e-08 154 KSP Residual norm 3.022743283437e-08 155 KSP Residual norm 2.982085654865e-08 156 KSP Residual norm 2.965332987998e-08 157 KSP Residual norm 2.995102882768e-08 158 KSP Residual norm 3.046192074890e-08 159 KSP Residual norm 3.011452161474e-08 160 KSP Residual norm 2.928575791440e-08 161 KSP Residual norm 2.888884135850e-08 162 KSP Residual norm 2.867568716743e-08 163 KSP Residual norm 2.821678138356e-08 164 KSP Residual norm 2.755405748002e-08 165 KSP Residual norm 2.691680013508e-08 166 KSP Residual norm 2.628777928782e-08 167 KSP Residual norm 2.559485446530e-08 168 KSP Residual norm 2.486045018643e-08 169 KSP Residual norm 2.402866781981e-08 170 KSP Residual norm 2.318073933315e-08 171 KSP Residual norm 2.248988625853e-08 172 KSP Residual norm 2.187341204241e-08 173 KSP Residual norm 2.086786253659e-08 174 KSP Residual norm 1.957659309488e-08 175 KSP Residual norm 1.844876720389e-08 176 KSP Residual norm 1.757817356183e-08 177 KSP Residual norm 1.697020763092e-08 178 KSP Residual norm 1.647409926394e-08 179 KSP Residual norm 1.573944855377e-08 180 KSP Residual norm 1.480412942849e-08 181 KSP Residual norm 1.408967955454e-08 182 KSP Residual norm 1.355400161996e-08 183 KSP Residual norm 1.286373865670e-08 184 KSP Residual norm 1.207975010593e-08 185 KSP Residual norm 1.143016154422e-08 186 KSP Residual norm 1.090689006760e-08 187 KSP Residual norm 1.046694919752e-08 188 KSP Residual norm 1.007773018783e-08 189 KSP Residual norm 9.742817970361e-09 190 KSP Residual norm 9.459115252285e-09 191 KSP Residual norm 9.208108966194e-09 192 KSP Residual norm 8.939663888775e-09 193 KSP Residual norm 8.603377431173e-09 194 KSP Residual norm 8.275490197779e-09 195 KSP Residual norm 8.018042774333e-09 196 KSP Residual norm 7.801983637333e-09 197 KSP Residual norm 7.610690229124e-09 198 KSP Residual norm 7.453580839401e-09 199 KSP Residual norm 7.304255493710e-09 200 KSP Residual norm 7.171465714479e-09 201 KSP Residual norm 7.067472548467e-09 202 KSP Residual norm 7.026363083868e-09 203 KSP Residual norm 6.988580172106e-09 204 KSP Residual norm 6.895878472742e-09 205 KSP Residual norm 6.769235925673e-09 206 KSP Residual norm 6.647045856934e-09 207 KSP Residual norm 6.519048850429e-09 208 KSP Residual norm 6.369661844371e-09 209 KSP Residual norm 6.225399122405e-09 210 KSP Residual norm 6.099838428904e-09 211 KSP Residual norm 5.972339234575e-09 212 KSP Residual norm 5.842875553667e-09 213 KSP Residual norm 5.720379645265e-09 214 KSP Residual norm 5.622308850644e-09 215 KSP Residual norm 5.585444436903e-09 216 KSP Residual norm 5.619503758249e-09 217 KSP Residual norm 5.499483661617e-09 218 KSP Residual norm 5.226743917144e-09 219 KSP Residual norm 5.004601581323e-09 220 KSP Residual norm 4.819017499289e-09 221 KSP Residual norm 4.635205009847e-09 222 KSP Residual norm 4.459600044313e-09 223 KSP Residual norm 4.295097019204e-09 224 KSP Residual norm 4.134292657563e-09 225 KSP Residual norm 3.965968501769e-09 226 KSP Residual norm 3.802104595962e-09 227 KSP Residual norm 3.636486948940e-09 228 KSP Residual norm 3.472374879672e-09 229 KSP Residual norm 3.322499402619e-09 230 KSP Residual norm 3.197009161377e-09 231 KSP Residual norm 3.086331095287e-09 232 KSP Residual norm 3.005610892799e-09 233 KSP Residual norm 2.973385781913e-09 234 KSP Residual norm 2.869979701256e-09 235 KSP Residual norm 2.679502738122e-09 236 KSP Residual norm 2.534822519741e-09 237 KSP Residual norm 2.445357838965e-09 238 KSP Residual norm 2.378772733485e-09 239 KSP Residual norm 2.322024940518e-09 240 KSP Residual norm 2.270492329422e-09 241 KSP Residual norm 2.183679791203e-09 242 KSP Residual norm 2.066437373735e-09 243 KSP Residual norm 1.964863991004e-09 244 KSP Residual norm 1.887313192978e-09 245 KSP Residual norm 1.820339942756e-09 246 KSP Residual norm 1.764019491547e-09 247 KSP Residual norm 1.718219402900e-09 248 KSP Residual norm 1.661908001738e-09 249 KSP Residual norm 1.598936940025e-09 250 KSP Residual norm 1.548068393013e-09 251 KSP Residual norm 1.508157434803e-09 252 KSP Residual norm 1.462471326623e-09 253 KSP Residual norm 1.416318536571e-09 254 KSP Residual norm 1.383104681694e-09 255 KSP Residual norm 1.356277788152e-09 256 KSP Residual norm 1.331403406703e-09 257 KSP Residual norm 1.307187216888e-09 258 KSP Residual norm 1.286023922319e-09 259 KSP Residual norm 1.268788816842e-09 260 KSP Residual norm 1.246919970432e-09 261 KSP Residual norm 1.215363860117e-09 262 KSP Residual norm 1.177603764392e-09 263 KSP Residual norm 1.143567136594e-09 264 KSP Residual norm 1.114810324380e-09 265 KSP Residual norm 1.088935197884e-09 266 KSP Residual norm 1.067057981592e-09 267 KSP Residual norm 1.043356500387e-09 268 KSP Residual norm 1.017737055783e-09 269 KSP Residual norm 9.963177061097e-10 270 KSP Residual norm 9.767408259706e-10 271 KSP Residual norm 9.601045948419e-10 272 KSP Residual norm 9.442520284393e-10 273 KSP Residual norm 9.199912192552e-10 274 KSP Residual norm 8.834934185330e-10 Linear tao_nls_ solve converged due to CONVERGED_RTOL iterations 274 0 LS Function value: 0.0691391, Step length: 0. 1 LS Function value: 0.069139, Step length: 1. stx: 0., fx: 0.0691391, dgx: -3.71917e-08 sty: 0., fy: 0.0691391, dgy: -3.71917e-08 32 TAO, Function value: 0.069139, Residual: 3.70651e-05 Residual norms for tao_nls_ solve. 0 KSP Residual norm 3.706507016073e-05 1 KSP Residual norm 3.018132415771e-05 2 KSP Residual norm 2.193482439073e-05 3 KSP Residual norm 1.843803366627e-05 4 KSP Residual norm 1.576332781105e-05 5 KSP Residual norm 1.524657299390e-05 6 KSP Residual norm 1.549406379596e-05 7 KSP Residual norm 1.674570969655e-05 8 KSP Residual norm 1.823597388362e-05 9 KSP Residual norm 2.025507446787e-05 10 KSP Residual norm 2.176229385004e-05 11 KSP Residual norm 2.372737560158e-05 12 KSP Residual norm 2.598002392289e-05 13 KSP Residual norm 2.826089396819e-05 14 KSP Residual norm 3.057106447366e-05 15 KSP Residual norm 3.298701339707e-05 16 KSP Residual norm 3.584420684022e-05 17 KSP Residual norm 3.827605336779e-05 18 KSP Residual norm 3.980529513724e-05 19 KSP Residual norm 4.111335015034e-05 20 KSP Residual norm 4.186448987638e-05 21 KSP Residual norm 4.211359671311e-05 22 KSP Residual norm 4.180732548398e-05 23 KSP Residual norm 4.126601761932e-05 24 KSP Residual norm 4.059135309836e-05 25 KSP Residual norm 3.956267555464e-05 26 KSP Residual norm 3.834059548963e-05 27 KSP Residual norm 3.682778893723e-05 28 KSP Residual norm 3.502868465173e-05 29 KSP Residual norm 3.307965214901e-05 30 KSP Residual norm 3.105575531123e-05 31 KSP Residual norm 2.908818260490e-05 32 KSP Residual norm 2.702832439465e-05 33 KSP Residual norm 2.478792715537e-05 34 KSP Residual norm 2.249398104842e-05 35 KSP Residual norm 2.028116992643e-05 36 KSP Residual norm 1.824115518433e-05 37 KSP Residual norm 1.639661473106e-05 38 KSP Residual norm 1.469839362573e-05 39 KSP Residual norm 1.312740221672e-05 40 KSP Residual norm 1.176501363521e-05 41 KSP Residual norm 1.067675859419e-05 42 KSP Residual norm 9.799232275552e-06 43 KSP Residual norm 8.918551492490e-06 44 KSP Residual norm 7.982966365971e-06 45 KSP Residual norm 7.144819412713e-06 46 KSP Residual norm 6.429340277259e-06 47 KSP Residual norm 5.800772546016e-06 48 KSP Residual norm 5.231787496247e-06 49 KSP Residual norm 4.738229414937e-06 50 KSP Residual norm 4.329248292647e-06 51 KSP Residual norm 3.964903074545e-06 52 KSP Residual norm 3.583234939193e-06 53 KSP Residual norm 3.211222718318e-06 54 KSP Residual norm 2.890541182387e-06 55 KSP Residual norm 2.619390295717e-06 56 KSP Residual norm 2.402052235048e-06 57 KSP Residual norm 2.239878861015e-06 58 KSP Residual norm 2.125383897279e-06 59 KSP Residual norm 2.039211886750e-06 60 KSP Residual norm 1.955028665384e-06 61 KSP Residual norm 1.872186521993e-06 62 KSP Residual norm 1.805367342845e-06 63 KSP Residual norm 1.752073683876e-06 64 KSP Residual norm 1.704806720116e-06 65 KSP Residual norm 1.661586359103e-06 66 KSP Residual norm 1.618729427128e-06 67 KSP Residual norm 1.578299744442e-06 68 KSP Residual norm 1.552439455130e-06 69 KSP Residual norm 1.534224741360e-06 70 KSP Residual norm 1.487338213153e-06 71 KSP Residual norm 1.419644595681e-06 72 KSP Residual norm 1.364551880141e-06 73 KSP Residual norm 1.306887923393e-06 74 KSP Residual norm 1.230338623496e-06 75 KSP Residual norm 1.152544231708e-06 76 KSP Residual norm 1.077714116735e-06 77 KSP Residual norm 1.006033112673e-06 78 KSP Residual norm 9.453956923528e-07 79 KSP Residual norm 8.918457864818e-07 80 KSP Residual norm 8.412936562137e-07 81 KSP Residual norm 7.831253306374e-07 82 KSP Residual norm 7.199254284898e-07 83 KSP Residual norm 6.665975246992e-07 84 KSP Residual norm 6.190948460449e-07 85 KSP Residual norm 5.805750613128e-07 86 KSP Residual norm 5.503067411311e-07 87 KSP Residual norm 5.197813842476e-07 88 KSP Residual norm 4.937089391854e-07 89 KSP Residual norm 4.744394671488e-07 90 KSP Residual norm 4.568288916900e-07 91 KSP Residual norm 4.310999831981e-07 92 KSP Residual norm 4.079648647800e-07 93 KSP Residual norm 3.962045476143e-07 94 KSP Residual norm 3.838939804358e-07 95 KSP Residual norm 3.660659958739e-07 96 KSP Residual norm 3.492513077613e-07 97 KSP Residual norm 3.350921963329e-07 98 KSP Residual norm 3.268427001883e-07 99 KSP Residual norm 3.210166532452e-07 100 KSP Residual norm 3.134480176419e-07 101 KSP Residual norm 3.059478083532e-07 102 KSP Residual norm 2.965704526634e-07 103 KSP Residual norm 2.927879823041e-07 104 KSP Residual norm 2.899988709214e-07 105 KSP Residual norm 2.878306926420e-07 106 KSP Residual norm 2.874199941633e-07 107 KSP Residual norm 2.848471007363e-07 108 KSP Residual norm 2.826492907451e-07 109 KSP Residual norm 2.790643317284e-07 110 KSP Residual norm 2.777101614280e-07 111 KSP Residual norm 2.792806886770e-07 112 KSP Residual norm 2.790838577551e-07 113 KSP Residual norm 2.747696001200e-07 114 KSP Residual norm 2.705202176768e-07 115 KSP Residual norm 2.675343653623e-07 116 KSP Residual norm 2.649000027392e-07 117 KSP Residual norm 2.614113801882e-07 118 KSP Residual norm 2.551178388743e-07 119 KSP Residual norm 2.472563476668e-07 120 KSP Residual norm 2.380659525325e-07 121 KSP Residual norm 2.278585536568e-07 122 KSP Residual norm 2.179093462711e-07 123 KSP Residual norm 2.079636443446e-07 124 KSP Residual norm 1.984343656099e-07 125 KSP Residual norm 1.876881657603e-07 126 KSP Residual norm 1.759934210465e-07 127 KSP Residual norm 1.647268076198e-07 128 KSP Residual norm 1.554528429614e-07 129 KSP Residual norm 1.476633705073e-07 130 KSP Residual norm 1.407123991097e-07 131 KSP Residual norm 1.353727277467e-07 132 KSP Residual norm 1.318108919269e-07 133 KSP Residual norm 1.273832271916e-07 134 KSP Residual norm 1.216343553735e-07 135 KSP Residual norm 1.160776546383e-07 136 KSP Residual norm 1.107189105035e-07 137 KSP Residual norm 1.061420212208e-07 138 KSP Residual norm 1.029215000808e-07 139 KSP Residual norm 1.010642408495e-07 140 KSP Residual norm 1.006359801052e-07 141 KSP Residual norm 1.002258873115e-07 142 KSP Residual norm 9.769969742260e-08 143 KSP Residual norm 9.458051686101e-08 144 KSP Residual norm 9.253624774034e-08 145 KSP Residual norm 9.116977465874e-08 146 KSP Residual norm 8.958253415354e-08 147 KSP Residual norm 8.801862139799e-08 148 KSP Residual norm 8.720870747767e-08 149 KSP Residual norm 8.678468846184e-08 150 KSP Residual norm 8.622789911051e-08 151 KSP Residual norm 8.566520588480e-08 152 KSP Residual norm 8.518326373420e-08 153 KSP Residual norm 8.475338060610e-08 154 KSP Residual norm 8.451944904624e-08 155 KSP Residual norm 8.438581055852e-08 156 KSP Residual norm 8.480951459373e-08 157 KSP Residual norm 8.623867580472e-08 158 KSP Residual norm 8.659254174084e-08 159 KSP Residual norm 8.482992041392e-08 160 KSP Residual norm 8.307777816155e-08 161 KSP Residual norm 8.198326368958e-08 162 KSP Residual norm 8.114319604324e-08 163 KSP Residual norm 8.041336257949e-08 164 KSP Residual norm 7.918796975642e-08 165 KSP Residual norm 7.732318152922e-08 166 KSP Residual norm 7.481431186656e-08 167 KSP Residual norm 7.194349817735e-08 168 KSP Residual norm 6.915484065894e-08 169 KSP Residual norm 6.648992270556e-08 170 KSP Residual norm 6.385434993983e-08 171 KSP Residual norm 6.112044666247e-08 172 KSP Residual norm 5.819225134818e-08 173 KSP Residual norm 5.501344303051e-08 174 KSP Residual norm 5.197686853158e-08 175 KSP Residual norm 4.958627761065e-08 176 KSP Residual norm 4.780290020285e-08 177 KSP Residual norm 4.585859044743e-08 178 KSP Residual norm 4.335647195517e-08 179 KSP Residual norm 4.097584367301e-08 180 KSP Residual norm 3.896520441367e-08 181 KSP Residual norm 3.714877455121e-08 182 KSP Residual norm 3.532204903372e-08 183 KSP Residual norm 3.349588212041e-08 184 KSP Residual norm 3.167527218195e-08 185 KSP Residual norm 2.997948193943e-08 186 KSP Residual norm 2.841654605491e-08 187 KSP Residual norm 2.692103264146e-08 188 KSP Residual norm 2.554998317863e-08 189 KSP Residual norm 2.435258704894e-08 190 KSP Residual norm 2.326221345241e-08 191 KSP Residual norm 2.218457523260e-08 192 KSP Residual norm 2.109972323377e-08 193 KSP Residual norm 2.018719428843e-08 194 KSP Residual norm 1.960305672038e-08 195 KSP Residual norm 1.939429061998e-08 196 KSP Residual norm 1.866602721681e-08 197 KSP Residual norm 1.752577036338e-08 198 KSP Residual norm 1.675923470911e-08 199 KSP Residual norm 1.618898509523e-08 200 KSP Residual norm 1.566073331691e-08 201 KSP Residual norm 1.514897966239e-08 202 KSP Residual norm 1.471816500784e-08 203 KSP Residual norm 1.428379529295e-08 204 KSP Residual norm 1.384201680524e-08 205 KSP Residual norm 1.348860556227e-08 206 KSP Residual norm 1.323707150379e-08 207 KSP Residual norm 1.300329686097e-08 208 KSP Residual norm 1.268913571244e-08 209 KSP Residual norm 1.238469323448e-08 210 KSP Residual norm 1.216897481289e-08 211 KSP Residual norm 1.202216052568e-08 212 KSP Residual norm 1.186470116351e-08 213 KSP Residual norm 1.166755356330e-08 214 KSP Residual norm 1.145198153086e-08 215 KSP Residual norm 1.123521216810e-08 216 KSP Residual norm 1.104683830204e-08 217 KSP Residual norm 1.091097909747e-08 218 KSP Residual norm 1.081020226600e-08 219 KSP Residual norm 1.069140951785e-08 220 KSP Residual norm 1.050232768743e-08 221 KSP Residual norm 1.027159659464e-08 222 KSP Residual norm 1.005364680799e-08 223 KSP Residual norm 9.885795489949e-09 224 KSP Residual norm 9.797732646376e-09 225 KSP Residual norm 9.746002980382e-09 226 KSP Residual norm 9.490748288909e-09 227 KSP Residual norm 9.046324068153e-09 228 KSP Residual norm 8.696432617565e-09 229 KSP Residual norm 8.461512493669e-09 230 KSP Residual norm 8.261674589785e-09 231 KSP Residual norm 8.060746676739e-09 232 KSP Residual norm 7.878657100240e-09 233 KSP Residual norm 7.703566396782e-09 234 KSP Residual norm 7.490152656024e-09 235 KSP Residual norm 7.221637520494e-09 236 KSP Residual norm 6.947483776752e-09 237 KSP Residual norm 6.720080814147e-09 238 KSP Residual norm 6.517353365686e-09 239 KSP Residual norm 6.270377695633e-09 240 KSP Residual norm 5.971803192986e-09 241 KSP Residual norm 5.664573067927e-09 242 KSP Residual norm 5.376796682092e-09 243 KSP Residual norm 5.130015473102e-09 244 KSP Residual norm 4.922893111778e-09 245 KSP Residual norm 4.746507349872e-09 246 KSP Residual norm 4.585095300798e-09 247 KSP Residual norm 4.423651965948e-09 248 KSP Residual norm 4.265997153439e-09 249 KSP Residual norm 4.111120099713e-09 250 KSP Residual norm 3.976952730839e-09 251 KSP Residual norm 3.896126697923e-09 252 KSP Residual norm 3.864960020402e-09 253 KSP Residual norm 3.742781248776e-09 254 KSP Residual norm 3.563322361475e-09 255 KSP Residual norm 3.447455316114e-09 256 KSP Residual norm 3.420173583632e-09 257 KSP Residual norm 3.359838264916e-09 258 KSP Residual norm 3.151119572527e-09 259 KSP Residual norm 2.957579548305e-09 260 KSP Residual norm 2.816365205584e-09 261 KSP Residual norm 2.720832199379e-09 262 KSP Residual norm 2.660294936716e-09 263 KSP Residual norm 2.590608027705e-09 264 KSP Residual norm 2.481254032968e-09 265 KSP Residual norm 2.375410080711e-09 266 KSP Residual norm 2.307240493937e-09 267 KSP Residual norm 2.266906998223e-09 268 KSP Residual norm 2.235126230200e-09 269 KSP Residual norm 2.201388879436e-09 270 KSP Residual norm 2.157864627928e-09 271 KSP Residual norm 2.115684403453e-09 272 KSP Residual norm 2.079252722319e-09 273 KSP Residual norm 2.042074774181e-09 274 KSP Residual norm 1.995646905808e-09 275 KSP Residual norm 1.943824909933e-09 276 KSP Residual norm 1.896125654079e-09 277 KSP Residual norm 1.845233116752e-09 278 KSP Residual norm 1.799067979359e-09 279 KSP Residual norm 1.749599764321e-09 280 KSP Residual norm 1.695520471604e-09 281 KSP Residual norm 1.643521832721e-09 282 KSP Residual norm 1.593316225950e-09 283 KSP Residual norm 1.550060628034e-09 284 KSP Residual norm 1.508572382076e-09 285 KSP Residual norm 1.469800333302e-09 286 KSP Residual norm 1.442219293231e-09 287 KSP Residual norm 1.421939614520e-09 288 KSP Residual norm 1.399021240910e-09 289 KSP Residual norm 1.372967909863e-09 290 KSP Residual norm 1.341337593941e-09 291 KSP Residual norm 1.307817650448e-09 292 KSP Residual norm 1.274210704029e-09 293 KSP Residual norm 1.234854952145e-09 294 KSP Residual norm 1.187350255486e-09 295 KSP Residual norm 1.140498145070e-09 296 KSP Residual norm 1.096227694363e-09 297 KSP Residual norm 1.058372272906e-09 298 KSP Residual norm 1.023975754175e-09 299 KSP Residual norm 9.917909611842e-10 300 KSP Residual norm 9.663391880518e-10 301 KSP Residual norm 9.469955080594e-10 302 KSP Residual norm 9.288302378341e-10 303 KSP Residual norm 9.070544239190e-10 304 KSP Residual norm 8.876132606355e-10 305 KSP Residual norm 8.756071018758e-10 306 KSP Residual norm 8.598335857724e-10 307 KSP Residual norm 8.344696694358e-10 308 KSP Residual norm 8.092861752037e-10 309 KSP Residual norm 7.878230298555e-10 310 KSP Residual norm 7.727749022005e-10 311 KSP Residual norm 7.680826868340e-10 312 KSP Residual norm 7.526447911957e-10 313 KSP Residual norm 7.150094616393e-10 314 KSP Residual norm 6.818357053992e-10 315 KSP Residual norm 6.608317272867e-10 316 KSP Residual norm 6.486484079500e-10 317 KSP Residual norm 6.485407446676e-10 318 KSP Residual norm 6.578741290036e-10 319 KSP Residual norm 6.349408900029e-10 320 KSP Residual norm 6.059472571106e-10 321 KSP Residual norm 5.934105678680e-10 322 KSP Residual norm 5.870926478878e-10 323 KSP Residual norm 5.750710946844e-10 324 KSP Residual norm 5.577367211107e-10 325 KSP Residual norm 5.432063214893e-10 326 KSP Residual norm 5.332009997093e-10 327 KSP Residual norm 5.227830162211e-10 328 KSP Residual norm 5.101858309408e-10 329 KSP Residual norm 4.949120819033e-10 330 KSP Residual norm 4.799648134145e-10 331 KSP Residual norm 4.661441465596e-10 332 KSP Residual norm 4.505184798837e-10 333 KSP Residual norm 4.337615091354e-10 334 KSP Residual norm 4.193804357379e-10 335 KSP Residual norm 4.056211300346e-10 336 KSP Residual norm 3.924753975529e-10 337 KSP Residual norm 3.822565596022e-10 338 KSP Residual norm 3.774481993084e-10 339 KSP Residual norm 3.747022216884e-10 340 KSP Residual norm 3.660344199416e-10 Linear tao_nls_ solve converged due to CONVERGED_RTOL iterations 340 0 LS Function value: 0.069139, Step length: 0. 1 LS Function value: 0.0691391, Step length: 1. stx: 0., fx: 0.069139, dgx: -4.30641e-08 sty: 0., fy: 0.069139, dgy: -4.30641e-08 2 LS Function value: 0.069139, Step length: 0.468265 stx: 0., fx: 0.069139, dgx: -4.30641e-08 sty: 1., fy: 0.0691391, dgy: 1.5642e-07 33 TAO, Function value: 0.069139, Residual: 6.54532e-05 Residual norms for tao_nls_ solve. 0 KSP Residual norm 6.545323099273e-05 1 KSP Residual norm 5.108563743407e-05 2 KSP Residual norm 3.557973148592e-05 3 KSP Residual norm 2.807580170778e-05 4 KSP Residual norm 2.233345574421e-05 5 KSP Residual norm 1.959850575468e-05 6 KSP Residual norm 1.779377036734e-05 7 KSP Residual norm 1.728219184812e-05 8 KSP Residual norm 1.723954547954e-05 9 KSP Residual norm 1.804744887780e-05 10 KSP Residual norm 1.847335014006e-05 11 KSP Residual norm 1.924162221138e-05 12 KSP Residual norm 2.032005754828e-05 13 KSP Residual norm 2.143472926309e-05 14 KSP Residual norm 2.284022526392e-05 15 KSP Residual norm 2.438449554568e-05 16 KSP Residual norm 2.626974596937e-05 17 KSP Residual norm 2.846383506361e-05 18 KSP Residual norm 3.029676563208e-05 19 KSP Residual norm 3.166733231698e-05 20 KSP Residual norm 3.246978751627e-05 21 KSP Residual norm 3.289036858225e-05 22 KSP Residual norm 3.290668443992e-05 23 KSP Residual norm 3.264754492501e-05 24 KSP Residual norm 3.226767242790e-05 25 KSP Residual norm 3.164684921998e-05 26 KSP Residual norm 3.089126715641e-05 27 KSP Residual norm 3.008183243202e-05 28 KSP Residual norm 2.908864624217e-05 29 KSP Residual norm 2.776267286730e-05 30 KSP Residual norm 2.622837655048e-05 31 KSP Residual norm 2.468023545795e-05 32 KSP Residual norm 2.304577687114e-05 33 KSP Residual norm 2.134559217597e-05 34 KSP Residual norm 1.961792915101e-05 35 KSP Residual norm 1.785525149729e-05 36 KSP Residual norm 1.613233218039e-05 37 KSP Residual norm 1.454147651892e-05 38 KSP Residual norm 1.306727209706e-05 39 KSP Residual norm 1.165014851726e-05 40 KSP Residual norm 1.033477805186e-05 41 KSP Residual norm 9.198651529410e-06 42 KSP Residual norm 8.246687054674e-06 43 KSP Residual norm 7.467984407129e-06 44 KSP Residual norm 6.826280786238e-06 45 KSP Residual norm 6.221430122024e-06 46 KSP Residual norm 5.594413925935e-06 47 KSP Residual norm 5.003941163238e-06 48 KSP Residual norm 4.478044107279e-06 49 KSP Residual norm 4.025407504588e-06 50 KSP Residual norm 3.660642959263e-06 51 KSP Residual norm 3.388127661895e-06 52 KSP Residual norm 3.109543823460e-06 53 KSP Residual norm 2.768097172582e-06 54 KSP Residual norm 2.465565812466e-06 55 KSP Residual norm 2.220370151694e-06 56 KSP Residual norm 2.019132876584e-06 57 KSP Residual norm 1.855959201308e-06 58 KSP Residual norm 1.724303114917e-06 59 KSP Residual norm 1.618660780651e-06 60 KSP Residual norm 1.536360376438e-06 61 KSP Residual norm 1.472713069044e-06 62 KSP Residual norm 1.426111797173e-06 63 KSP Residual norm 1.392793556754e-06 64 KSP Residual norm 1.371516787104e-06 65 KSP Residual norm 1.363566994036e-06 66 KSP Residual norm 1.346069629633e-06 67 KSP Residual norm 1.298247054153e-06 68 KSP Residual norm 1.251343872722e-06 69 KSP Residual norm 1.221561725597e-06 70 KSP Residual norm 1.198402997796e-06 71 KSP Residual norm 1.176805870415e-06 72 KSP Residual norm 1.159444254142e-06 73 KSP Residual norm 1.135604368187e-06 74 KSP Residual norm 1.106292950461e-06 75 KSP Residual norm 1.077856266471e-06 76 KSP Residual norm 1.034043940103e-06 77 KSP Residual norm 9.778695849516e-07 78 KSP Residual norm 9.254389032041e-07 79 KSP Residual norm 8.765780947783e-07 80 KSP Residual norm 8.352365693030e-07 81 KSP Residual norm 7.997130999398e-07 82 KSP Residual norm 7.631401349689e-07 83 KSP Residual norm 7.221867988768e-07 84 KSP Residual norm 6.759381733536e-07 85 KSP Residual norm 6.369813477348e-07 86 KSP Residual norm 6.079418308567e-07 87 KSP Residual norm 5.813779090523e-07 88 KSP Residual norm 5.620292837414e-07 89 KSP Residual norm 5.404147165094e-07 90 KSP Residual norm 5.094813398631e-07 91 KSP Residual norm 4.801554621256e-07 92 KSP Residual norm 4.584200726177e-07 93 KSP Residual norm 4.417666607489e-07 94 KSP Residual norm 4.175343200259e-07 95 KSP Residual norm 3.936983280923e-07 96 KSP Residual norm 3.772521067928e-07 97 KSP Residual norm 3.624122764433e-07 98 KSP Residual norm 3.522416060051e-07 99 KSP Residual norm 3.440700574938e-07 100 KSP Residual norm 3.327176434595e-07 101 KSP Residual norm 3.206256703547e-07 102 KSP Residual norm 3.062418979016e-07 103 KSP Residual norm 2.965246725670e-07 104 KSP Residual norm 2.882098637630e-07 105 KSP Residual norm 2.807841537085e-07 106 KSP Residual norm 2.768714130842e-07 107 KSP Residual norm 2.711182984878e-07 108 KSP Residual norm 2.649310340716e-07 109 KSP Residual norm 2.558376029356e-07 110 KSP Residual norm 2.482901755800e-07 111 KSP Residual norm 2.435886851064e-07 112 KSP Residual norm 2.407263488350e-07 113 KSP Residual norm 2.376657413678e-07 114 KSP Residual norm 2.331070541121e-07 115 KSP Residual norm 2.269565122531e-07 116 KSP Residual norm 2.198828060798e-07 117 KSP Residual norm 2.122484254129e-07 118 KSP Residual norm 2.045085813179e-07 119 KSP Residual norm 1.974101968851e-07 120 KSP Residual norm 1.898622049265e-07 121 KSP Residual norm 1.816863863703e-07 122 KSP Residual norm 1.753790022986e-07 123 KSP Residual norm 1.698416881280e-07 124 KSP Residual norm 1.599194715301e-07 125 KSP Residual norm 1.475195275640e-07 126 KSP Residual norm 1.371461165121e-07 127 KSP Residual norm 1.282450111502e-07 128 KSP Residual norm 1.200530262083e-07 129 KSP Residual norm 1.126040243085e-07 130 KSP Residual norm 1.058497913451e-07 131 KSP Residual norm 9.965303838430e-08 132 KSP Residual norm 9.349708130102e-08 133 KSP Residual norm 8.794490277039e-08 134 KSP Residual norm 8.358802931544e-08 135 KSP Residual norm 8.037736194917e-08 136 KSP Residual norm 7.734988642395e-08 137 KSP Residual norm 7.389681490502e-08 138 KSP Residual norm 7.075310522040e-08 139 KSP Residual norm 6.817931102662e-08 140 KSP Residual norm 6.617028278567e-08 141 KSP Residual norm 6.498679236554e-08 142 KSP Residual norm 6.477825607946e-08 143 KSP Residual norm 6.406433951622e-08 144 KSP Residual norm 6.147173854709e-08 145 KSP Residual norm 5.920845245655e-08 146 KSP Residual norm 5.788541349489e-08 147 KSP Residual norm 5.704594639048e-08 148 KSP Residual norm 5.665261842113e-08 149 KSP Residual norm 5.631543288231e-08 150 KSP Residual norm 5.570516327691e-08 151 KSP Residual norm 5.511845781953e-08 152 KSP Residual norm 5.479616149330e-08 153 KSP Residual norm 5.470895445354e-08 154 KSP Residual norm 5.490270005814e-08 155 KSP Residual norm 5.515905938400e-08 156 KSP Residual norm 5.544961401828e-08 157 KSP Residual norm 5.604710424897e-08 158 KSP Residual norm 5.688264526759e-08 159 KSP Residual norm 5.793027288212e-08 160 KSP Residual norm 5.821046214822e-08 161 KSP Residual norm 5.753290816764e-08 162 KSP Residual norm 5.700129079013e-08 163 KSP Residual norm 5.698832136923e-08 164 KSP Residual norm 5.701648598165e-08 165 KSP Residual norm 5.716996375489e-08 166 KSP Residual norm 5.748191853052e-08 167 KSP Residual norm 5.681772892807e-08 168 KSP Residual norm 5.466418045106e-08 169 KSP Residual norm 5.249934624712e-08 170 KSP Residual norm 5.103645209893e-08 171 KSP Residual norm 4.998072144867e-08 172 KSP Residual norm 4.840458036348e-08 173 KSP Residual norm 4.589555856739e-08 174 KSP Residual norm 4.349145402754e-08 175 KSP Residual norm 4.149349314129e-08 176 KSP Residual norm 3.961201284061e-08 177 KSP Residual norm 3.781128264068e-08 178 KSP Residual norm 3.607121790350e-08 179 KSP Residual norm 3.450371661629e-08 180 KSP Residual norm 3.341178989266e-08 181 KSP Residual norm 3.299759583557e-08 182 KSP Residual norm 3.205358944415e-08 183 KSP Residual norm 2.990109108349e-08 184 KSP Residual norm 2.793686764613e-08 185 KSP Residual norm 2.638346995762e-08 186 KSP Residual norm 2.501006432729e-08 187 KSP Residual norm 2.379531151216e-08 188 KSP Residual norm 2.258588213461e-08 189 KSP Residual norm 2.131054511304e-08 190 KSP Residual norm 2.009451938227e-08 191 KSP Residual norm 1.899066143004e-08 192 KSP Residual norm 1.798354266493e-08 193 KSP Residual norm 1.703974031884e-08 194 KSP Residual norm 1.621306513278e-08 195 KSP Residual norm 1.545758404611e-08 196 KSP Residual norm 1.466544063222e-08 197 KSP Residual norm 1.389379285018e-08 198 KSP Residual norm 1.325912478066e-08 199 KSP Residual norm 1.268929129894e-08 200 KSP Residual norm 1.215558850704e-08 201 KSP Residual norm 1.165843474888e-08 202 KSP Residual norm 1.126473262954e-08 203 KSP Residual norm 1.087805740056e-08 204 KSP Residual norm 1.041810564157e-08 205 KSP Residual norm 9.978340522007e-09 206 KSP Residual norm 9.606701317442e-09 207 KSP Residual norm 9.278541591514e-09 208 KSP Residual norm 8.959968093517e-09 209 KSP Residual norm 8.735746421532e-09 210 KSP Residual norm 8.615120377191e-09 211 KSP Residual norm 8.386585269495e-09 212 KSP Residual norm 8.018659512833e-09 213 KSP Residual norm 7.746856926309e-09 214 KSP Residual norm 7.602229340772e-09 215 KSP Residual norm 7.510271860189e-09 216 KSP Residual norm 7.403739106992e-09 217 KSP Residual norm 7.274283387564e-09 218 KSP Residual norm 7.148656579715e-09 219 KSP Residual norm 7.028065114257e-09 220 KSP Residual norm 6.886976273954e-09 221 KSP Residual norm 6.744177237265e-09 222 KSP Residual norm 6.641156891922e-09 223 KSP Residual norm 6.625903093193e-09 224 KSP Residual norm 6.644370124907e-09 225 KSP Residual norm 6.505602346407e-09 226 KSP Residual norm 6.268175017000e-09 227 KSP Residual norm 6.061656205441e-09 228 KSP Residual norm 5.908242233017e-09 229 KSP Residual norm 5.795124462539e-09 230 KSP Residual norm 5.700960611964e-09 231 KSP Residual norm 5.615833797872e-09 232 KSP Residual norm 5.557044591441e-09 233 KSP Residual norm 5.533417672850e-09 234 KSP Residual norm 5.558848995779e-09 235 KSP Residual norm 5.579009286320e-09 236 KSP Residual norm 5.373951831561e-09 237 KSP Residual norm 5.077526882390e-09 238 KSP Residual norm 4.863708033872e-09 239 KSP Residual norm 4.699634910964e-09 240 KSP Residual norm 4.554832115050e-09 241 KSP Residual norm 4.429349677881e-09 242 KSP Residual norm 4.318069795012e-09 243 KSP Residual norm 4.186491010894e-09 244 KSP Residual norm 3.997694852128e-09 245 KSP Residual norm 3.804466013566e-09 246 KSP Residual norm 3.654116258926e-09 247 KSP Residual norm 3.529365426518e-09 248 KSP Residual norm 3.386455996101e-09 249 KSP Residual norm 3.234907998425e-09 250 KSP Residual norm 3.096135956393e-09 251 KSP Residual norm 2.981112927014e-09 252 KSP Residual norm 2.884522405383e-09 253 KSP Residual norm 2.811416270108e-09 254 KSP Residual norm 2.760277623349e-09 255 KSP Residual norm 2.690414565576e-09 256 KSP Residual norm 2.589684334264e-09 257 KSP Residual norm 2.482526703927e-09 258 KSP Residual norm 2.381848361417e-09 259 KSP Residual norm 2.277563489223e-09 260 KSP Residual norm 2.170836708053e-09 261 KSP Residual norm 2.075313328816e-09 262 KSP Residual norm 1.985691437443e-09 263 KSP Residual norm 1.899535590768e-09 264 KSP Residual norm 1.826266817726e-09 265 KSP Residual norm 1.766301535172e-09 266 KSP Residual norm 1.730433056117e-09 267 KSP Residual norm 1.701555450859e-09 268 KSP Residual norm 1.646459097787e-09 269 KSP Residual norm 1.579618233340e-09 270 KSP Residual norm 1.524474189157e-09 271 KSP Residual norm 1.482027698288e-09 272 KSP Residual norm 1.446724732870e-09 273 KSP Residual norm 1.413146963835e-09 274 KSP Residual norm 1.376675948184e-09 275 KSP Residual norm 1.340585286878e-09 276 KSP Residual norm 1.313852353937e-09 277 KSP Residual norm 1.287386721785e-09 278 KSP Residual norm 1.251471332362e-09 279 KSP Residual norm 1.203608370848e-09 280 KSP Residual norm 1.156999142643e-09 281 KSP Residual norm 1.119828498677e-09 282 KSP Residual norm 1.086070555298e-09 283 KSP Residual norm 1.055843928238e-09 284 KSP Residual norm 1.028307019476e-09 285 KSP Residual norm 1.003073696979e-09 286 KSP Residual norm 9.834254423902e-10 287 KSP Residual norm 9.658710971841e-10 288 KSP Residual norm 9.447276221663e-10 289 KSP Residual norm 9.257799859922e-10 290 KSP Residual norm 9.162568221901e-10 291 KSP Residual norm 9.184633374013e-10 292 KSP Residual norm 9.028525973792e-10 293 KSP Residual norm 8.520192986494e-10 294 KSP Residual norm 8.080477225248e-10 295 KSP Residual norm 7.798703301499e-10 296 KSP Residual norm 7.582280654354e-10 297 KSP Residual norm 7.375043595383e-10 298 KSP Residual norm 7.132149702594e-10 299 KSP Residual norm 6.889261230075e-10 300 KSP Residual norm 6.706709183879e-10 301 KSP Residual norm 6.562104538454e-10 302 KSP Residual norm 6.385077640185e-10 Linear tao_nls_ solve converged due to CONVERGED_RTOL iterations 302 0 LS Function value: 0.069139, Step length: 0. 1 LS Function value: 0.069139, Step length: 1. stx: 0., fx: 0.069139, dgx: -3.55959e-08 sty: 0., fy: 0.069139, dgy: -3.55959e-08 34 TAO, Function value: 0.069139, Residual: 8.64191e-05 Residual norms for tao_nls_ solve. 0 KSP Residual norm 8.641910585250e-05 1 KSP Residual norm 6.520242620587e-05 2 KSP Residual norm 4.504259706394e-05 3 KSP Residual norm 3.506634256376e-05 4 KSP Residual norm 2.773117708332e-05 5 KSP Residual norm 2.406366953943e-05 6 KSP Residual norm 2.126195379200e-05 7 KSP Residual norm 1.999261160013e-05 8 KSP Residual norm 1.926823924367e-05 9 KSP Residual norm 1.964981074706e-05 10 KSP Residual norm 1.989834263734e-05 11 KSP Residual norm 2.039221356633e-05 12 KSP Residual norm 2.143672245581e-05 13 KSP Residual norm 2.243857885994e-05 14 KSP Residual norm 2.369572009777e-05 15 KSP Residual norm 2.517230348130e-05 16 KSP Residual norm 2.685507511290e-05 17 KSP Residual norm 2.892305993181e-05 18 KSP Residual norm 3.090789672705e-05 19 KSP Residual norm 3.255782743436e-05 20 KSP Residual norm 3.356284061103e-05 21 KSP Residual norm 3.382950565341e-05 22 KSP Residual norm 3.358566087061e-05 23 KSP Residual norm 3.310092338340e-05 24 KSP Residual norm 3.257708702645e-05 25 KSP Residual norm 3.195177201161e-05 26 KSP Residual norm 3.122382529514e-05 27 KSP Residual norm 3.026064076028e-05 28 KSP Residual norm 2.890630912296e-05 29 KSP Residual norm 2.741124186710e-05 30 KSP Residual norm 2.599715625423e-05 31 KSP Residual norm 2.461458297641e-05 32 KSP Residual norm 2.313554601115e-05 33 KSP Residual norm 2.160621622066e-05 34 KSP Residual norm 2.003203612240e-05 35 KSP Residual norm 1.834037283437e-05 36 KSP Residual norm 1.660311366182e-05 37 KSP Residual norm 1.497725252247e-05 38 KSP Residual norm 1.349106247777e-05 39 KSP Residual norm 1.210632775473e-05 40 KSP Residual norm 1.086480791135e-05 41 KSP Residual norm 9.852075499423e-06 42 KSP Residual norm 8.902609377236e-06 43 KSP Residual norm 7.856854436163e-06 44 KSP Residual norm 6.914370298346e-06 45 KSP Residual norm 6.162679219993e-06 46 KSP Residual norm 5.545677551841e-06 47 KSP Residual norm 5.002064489691e-06 48 KSP Residual norm 4.495749459528e-06 49 KSP Residual norm 4.032341209592e-06 50 KSP Residual norm 3.622127671597e-06 51 KSP Residual norm 3.270244028886e-06 52 KSP Residual norm 2.969146581727e-06 53 KSP Residual norm 2.707565167476e-06 54 KSP Residual norm 2.482134939788e-06 55 KSP Residual norm 2.299179056928e-06 56 KSP Residual norm 2.141944607526e-06 57 KSP Residual norm 1.985623640729e-06 58 KSP Residual norm 1.851890368597e-06 59 KSP Residual norm 1.762883701672e-06 60 KSP Residual norm 1.723820287790e-06 61 KSP Residual norm 1.705666505731e-06 62 KSP Residual norm 1.665160195877e-06 63 KSP Residual norm 1.620393610766e-06 64 KSP Residual norm 1.597672351213e-06 65 KSP Residual norm 1.593255867781e-06 66 KSP Residual norm 1.593737397862e-06 67 KSP Residual norm 1.588796000833e-06 68 KSP Residual norm 1.580273850513e-06 69 KSP Residual norm 1.572958198791e-06 70 KSP Residual norm 1.564205255710e-06 71 KSP Residual norm 1.553890479121e-06 72 KSP Residual norm 1.545312009743e-06 73 KSP Residual norm 1.521414278667e-06 74 KSP Residual norm 1.482160313952e-06 75 KSP Residual norm 1.449467784650e-06 76 KSP Residual norm 1.416428450710e-06 77 KSP Residual norm 1.372078765877e-06 78 KSP Residual norm 1.316322900443e-06 79 KSP Residual norm 1.249889595608e-06 80 KSP Residual norm 1.185171765994e-06 81 KSP Residual norm 1.123944862829e-06 82 KSP Residual norm 1.068263494114e-06 83 KSP Residual norm 1.023886763602e-06 84 KSP Residual norm 9.765713977603e-07 85 KSP Residual norm 9.299688558774e-07 86 KSP Residual norm 8.851668347282e-07 87 KSP Residual norm 8.366254201082e-07 88 KSP Residual norm 7.936034507506e-07 89 KSP Residual norm 7.628330108758e-07 90 KSP Residual norm 7.417745661240e-07 91 KSP Residual norm 6.969486240803e-07 92 KSP Residual norm 6.403671114040e-07 93 KSP Residual norm 5.981302816087e-07 94 KSP Residual norm 5.616937489758e-07 95 KSP Residual norm 5.308246687300e-07 96 KSP Residual norm 5.076253121610e-07 97 KSP Residual norm 4.844226297949e-07 98 KSP Residual norm 4.628469225486e-07 99 KSP Residual norm 4.369428325311e-07 100 KSP Residual norm 4.128944256493e-07 101 KSP Residual norm 3.975602276712e-07 102 KSP Residual norm 3.835547507823e-07 103 KSP Residual norm 3.693426569922e-07 104 KSP Residual norm 3.524993481522e-07 105 KSP Residual norm 3.345509933884e-07 106 KSP Residual norm 3.234950109432e-07 107 KSP Residual norm 3.130670989686e-07 108 KSP Residual norm 3.072825989077e-07 109 KSP Residual norm 2.969561417374e-07 110 KSP Residual norm 2.844374023106e-07 111 KSP Residual norm 2.738105389838e-07 112 KSP Residual norm 2.648563955494e-07 113 KSP Residual norm 2.564101334261e-07 114 KSP Residual norm 2.488333799060e-07 115 KSP Residual norm 2.433535647864e-07 116 KSP Residual norm 2.410694675240e-07 117 KSP Residual norm 2.349670962365e-07 118 KSP Residual norm 2.229631636471e-07 119 KSP Residual norm 2.117781663251e-07 120 KSP Residual norm 2.010750279291e-07 121 KSP Residual norm 1.899895998981e-07 122 KSP Residual norm 1.792716995232e-07 123 KSP Residual norm 1.693750788040e-07 124 KSP Residual norm 1.611187098969e-07 125 KSP Residual norm 1.536863215419e-07 126 KSP Residual norm 1.455792157025e-07 127 KSP Residual norm 1.359358936953e-07 128 KSP Residual norm 1.263370930886e-07 129 KSP Residual norm 1.184751594103e-07 130 KSP Residual norm 1.128400609943e-07 131 KSP Residual norm 1.084047630010e-07 132 KSP Residual norm 1.014540468844e-07 133 KSP Residual norm 9.376450913364e-08 134 KSP Residual norm 8.778270934937e-08 135 KSP Residual norm 8.266696028069e-08 136 KSP Residual norm 7.786951853363e-08 137 KSP Residual norm 7.353821641967e-08 138 KSP Residual norm 6.970541537329e-08 139 KSP Residual norm 6.593230016676e-08 140 KSP Residual norm 6.242606968271e-08 141 KSP Residual norm 5.937227782579e-08 142 KSP Residual norm 5.660426245225e-08 143 KSP Residual norm 5.379215016121e-08 144 KSP Residual norm 5.097203894240e-08 145 KSP Residual norm 4.833132666376e-08 146 KSP Residual norm 4.582534019311e-08 147 KSP Residual norm 4.342427966517e-08 148 KSP Residual norm 4.131947407690e-08 149 KSP Residual norm 3.946483956425e-08 150 KSP Residual norm 3.779867226225e-08 151 KSP Residual norm 3.639124099830e-08 152 KSP Residual norm 3.511450007473e-08 153 KSP Residual norm 3.392319083264e-08 154 KSP Residual norm 3.289231235858e-08 155 KSP Residual norm 3.193378117654e-08 156 KSP Residual norm 3.113460003472e-08 157 KSP Residual norm 3.077744922672e-08 158 KSP Residual norm 3.060629579141e-08 159 KSP Residual norm 3.030009584154e-08 160 KSP Residual norm 2.987867290001e-08 161 KSP Residual norm 2.956018600794e-08 162 KSP Residual norm 2.940924018988e-08 163 KSP Residual norm 2.946949328323e-08 164 KSP Residual norm 2.957773982642e-08 165 KSP Residual norm 2.954800761444e-08 166 KSP Residual norm 2.940530178524e-08 167 KSP Residual norm 2.927185203382e-08 168 KSP Residual norm 2.941139729868e-08 169 KSP Residual norm 2.972954929837e-08 170 KSP Residual norm 2.939028504954e-08 171 KSP Residual norm 2.862109122660e-08 172 KSP Residual norm 2.839895158399e-08 173 KSP Residual norm 2.870432486222e-08 174 KSP Residual norm 2.857128429150e-08 175 KSP Residual norm 2.773825513266e-08 176 KSP Residual norm 2.707282254269e-08 177 KSP Residual norm 2.672221390118e-08 178 KSP Residual norm 2.652442004370e-08 179 KSP Residual norm 2.636860484932e-08 180 KSP Residual norm 2.627936779792e-08 181 KSP Residual norm 2.628151324187e-08 182 KSP Residual norm 2.622466611194e-08 183 KSP Residual norm 2.588501023315e-08 184 KSP Residual norm 2.518318969697e-08 185 KSP Residual norm 2.439077881791e-08 186 KSP Residual norm 2.356152665369e-08 187 KSP Residual norm 2.272659742059e-08 188 KSP Residual norm 2.188218924253e-08 189 KSP Residual norm 2.106158410903e-08 190 KSP Residual norm 2.033512489145e-08 191 KSP Residual norm 1.966914137009e-08 192 KSP Residual norm 1.898167625392e-08 193 KSP Residual norm 1.820576819510e-08 194 KSP Residual norm 1.740489663237e-08 195 KSP Residual norm 1.671800469233e-08 196 KSP Residual norm 1.607880653208e-08 197 KSP Residual norm 1.536041001115e-08 198 KSP Residual norm 1.466910575082e-08 199 KSP Residual norm 1.405697039820e-08 200 KSP Residual norm 1.354549077870e-08 201 KSP Residual norm 1.298489213423e-08 202 KSP Residual norm 1.236209668722e-08 203 KSP Residual norm 1.174534370619e-08 204 KSP Residual norm 1.111341352899e-08 205 KSP Residual norm 1.053875581106e-08 206 KSP Residual norm 1.012377311646e-08 207 KSP Residual norm 9.852416763476e-09 208 KSP Residual norm 9.366662708414e-09 209 KSP Residual norm 8.758252807536e-09 210 KSP Residual norm 8.282742016298e-09 211 KSP Residual norm 7.857633880688e-09 212 KSP Residual norm 7.403628465354e-09 213 KSP Residual norm 6.997905908687e-09 214 KSP Residual norm 6.670244639939e-09 215 KSP Residual norm 6.383099639548e-09 216 KSP Residual norm 6.100158266986e-09 217 KSP Residual norm 5.824562371657e-09 218 KSP Residual norm 5.570706008570e-09 219 KSP Residual norm 5.340819293524e-09 220 KSP Residual norm 5.119199784355e-09 221 KSP Residual norm 4.928511756599e-09 222 KSP Residual norm 4.821880404612e-09 223 KSP Residual norm 4.794984980069e-09 224 KSP Residual norm 4.589980292040e-09 225 KSP Residual norm 4.308599217434e-09 226 KSP Residual norm 4.129547216637e-09 227 KSP Residual norm 3.989930038037e-09 228 KSP Residual norm 3.846414447552e-09 229 KSP Residual norm 3.707520074542e-09 230 KSP Residual norm 3.591648410578e-09 231 KSP Residual norm 3.501437886578e-09 232 KSP Residual norm 3.432914449087e-09 233 KSP Residual norm 3.370222122513e-09 234 KSP Residual norm 3.308840315268e-09 235 KSP Residual norm 3.251898679187e-09 236 KSP Residual norm 3.189368452523e-09 237 KSP Residual norm 3.112994038028e-09 238 KSP Residual norm 3.032596245836e-09 239 KSP Residual norm 2.953182862426e-09 240 KSP Residual norm 2.870806068180e-09 241 KSP Residual norm 2.790253331135e-09 242 KSP Residual norm 2.714083495113e-09 243 KSP Residual norm 2.648397578486e-09 244 KSP Residual norm 2.596793567610e-09 245 KSP Residual norm 2.544633316550e-09 246 KSP Residual norm 2.487423749513e-09 247 KSP Residual norm 2.434030269326e-09 248 KSP Residual norm 2.384547143663e-09 249 KSP Residual norm 2.355852489076e-09 250 KSP Residual norm 2.352535862386e-09 251 KSP Residual norm 2.325024820612e-09 252 KSP Residual norm 2.250504826584e-09 253 KSP Residual norm 2.179890442626e-09 254 KSP Residual norm 2.132476469227e-09 255 KSP Residual norm 2.096804251010e-09 256 KSP Residual norm 2.068901652224e-09 257 KSP Residual norm 2.025955847693e-09 258 KSP Residual norm 1.950674714977e-09 259 KSP Residual norm 1.863374828887e-09 260 KSP Residual norm 1.780887686460e-09 261 KSP Residual norm 1.709644507061e-09 262 KSP Residual norm 1.644325661833e-09 263 KSP Residual norm 1.583994922734e-09 264 KSP Residual norm 1.533830168704e-09 265 KSP Residual norm 1.480403618566e-09 266 KSP Residual norm 1.424589568004e-09 267 KSP Residual norm 1.367254047960e-09 268 KSP Residual norm 1.313602476721e-09 269 KSP Residual norm 1.273066561299e-09 270 KSP Residual norm 1.236044130973e-09 271 KSP Residual norm 1.201855480812e-09 272 KSP Residual norm 1.174742593071e-09 273 KSP Residual norm 1.150579463304e-09 274 KSP Residual norm 1.109310384785e-09 275 KSP Residual norm 1.057711539906e-09 276 KSP Residual norm 1.021176684580e-09 277 KSP Residual norm 1.011522303063e-09 278 KSP Residual norm 9.951334487975e-10 279 KSP Residual norm 9.285969238819e-10 280 KSP Residual norm 8.702228768154e-10 281 KSP Residual norm 8.326289197611e-10 Linear tao_nls_ solve converged due to CONVERGED_RTOL iterations 281 0 LS Function value: 0.069139, Step length: 0. 1 LS Function value: 0.069139, Step length: 1. stx: 0., fx: 0.069139, dgx: -4.31256e-08 sty: 0., fy: 0.069139, dgy: -4.31256e-08 35 TAO, Function value: 0.069139, Residual: 7.92529e-05 Residual norms for tao_nls_ solve. 0 KSP Residual norm 7.925285017473e-05 1 KSP Residual norm 5.743642484722e-05 2 KSP Residual norm 4.087227133774e-05 3 KSP Residual norm 3.214100426392e-05 4 KSP Residual norm 2.645739345814e-05 5 KSP Residual norm 2.390306274815e-05 6 KSP Residual norm 2.163139498758e-05 7 KSP Residual norm 2.082286524600e-05 8 KSP Residual norm 2.086013666654e-05 9 KSP Residual norm 2.204779591937e-05 10 KSP Residual norm 2.391694837937e-05 11 KSP Residual norm 2.603891991087e-05 12 KSP Residual norm 2.867828541353e-05 13 KSP Residual norm 3.166349367146e-05 14 KSP Residual norm 3.401565566081e-05 15 KSP Residual norm 3.599023161282e-05 16 KSP Residual norm 3.812050076652e-05 17 KSP Residual norm 4.078147602428e-05 18 KSP Residual norm 4.333126112604e-05 19 KSP Residual norm 4.504038087712e-05 20 KSP Residual norm 4.558255868571e-05 21 KSP Residual norm 4.515128517152e-05 22 KSP Residual norm 4.414287427691e-05 23 KSP Residual norm 4.287949703605e-05 24 KSP Residual norm 4.168185655995e-05 25 KSP Residual norm 4.054093543476e-05 26 KSP Residual norm 3.930412543404e-05 27 KSP Residual norm 3.764459539073e-05 28 KSP Residual norm 3.565182674339e-05 29 KSP Residual norm 3.376416612379e-05 30 KSP Residual norm 3.201933333880e-05 31 KSP Residual norm 3.025986096023e-05 32 KSP Residual norm 2.839474143876e-05 33 KSP Residual norm 2.650701388145e-05 34 KSP Residual norm 2.457305250547e-05 35 KSP Residual norm 2.247485928993e-05 36 KSP Residual norm 2.032047342938e-05 37 KSP Residual norm 1.837768852976e-05 38 KSP Residual norm 1.671787766288e-05 39 KSP Residual norm 1.516967448624e-05 40 KSP Residual norm 1.353190318482e-05 41 KSP Residual norm 1.188690113212e-05 42 KSP Residual norm 1.039299711631e-05 43 KSP Residual norm 9.116516940898e-06 44 KSP Residual norm 8.041287205404e-06 45 KSP Residual norm 7.153605127092e-06 46 KSP Residual norm 6.421398709018e-06 47 KSP Residual norm 5.803811192191e-06 48 KSP Residual norm 5.255883537948e-06 49 KSP Residual norm 4.766055513534e-06 50 KSP Residual norm 4.358821953432e-06 51 KSP Residual norm 4.050546063436e-06 52 KSP Residual norm 3.752430482240e-06 53 KSP Residual norm 3.421193729811e-06 54 KSP Residual norm 3.142599858729e-06 55 KSP Residual norm 2.940458300481e-06 56 KSP Residual norm 2.802227207172e-06 57 KSP Residual norm 2.725502008400e-06 58 KSP Residual norm 2.724929166128e-06 59 KSP Residual norm 2.772689286938e-06 60 KSP Residual norm 2.757588489101e-06 61 KSP Residual norm 2.717674287184e-06 62 KSP Residual norm 2.735366296608e-06 63 KSP Residual norm 2.789482756255e-06 64 KSP Residual norm 2.853547951106e-06 65 KSP Residual norm 2.918268933930e-06 66 KSP Residual norm 2.976244206497e-06 67 KSP Residual norm 3.014910985436e-06 68 KSP Residual norm 3.030587178558e-06 69 KSP Residual norm 3.030438429452e-06 70 KSP Residual norm 3.017065922244e-06 71 KSP Residual norm 2.989780493331e-06 72 KSP Residual norm 2.959173454314e-06 73 KSP Residual norm 2.904141257422e-06 74 KSP Residual norm 2.824378072090e-06 75 KSP Residual norm 2.755586854531e-06 76 KSP Residual norm 2.686581241694e-06 77 KSP Residual norm 2.594497124367e-06 78 KSP Residual norm 2.476389326210e-06 79 KSP Residual norm 2.339067058470e-06 80 KSP Residual norm 2.205837451514e-06 81 KSP Residual norm 2.081973751090e-06 82 KSP Residual norm 1.971593291511e-06 83 KSP Residual norm 1.881600082642e-06 84 KSP Residual norm 1.781689789438e-06 85 KSP Residual norm 1.679174058614e-06 86 KSP Residual norm 1.582167496144e-06 87 KSP Residual norm 1.479544760892e-06 88 KSP Residual norm 1.379073332866e-06 89 KSP Residual norm 1.280936932474e-06 90 KSP Residual norm 1.194502235946e-06 91 KSP Residual norm 1.107792508781e-06 92 KSP Residual norm 1.027677361902e-06 93 KSP Residual norm 9.636478515759e-07 94 KSP Residual norm 8.981406050240e-07 95 KSP Residual norm 8.322100410782e-07 96 KSP Residual norm 7.688580766482e-07 97 KSP Residual norm 7.050005728328e-07 98 KSP Residual norm 6.565441350430e-07 99 KSP Residual norm 6.158516534444e-07 100 KSP Residual norm 5.778402276723e-07 101 KSP Residual norm 5.413872803269e-07 102 KSP Residual norm 5.066143302212e-07 103 KSP Residual norm 4.821773519616e-07 104 KSP Residual norm 4.670024378167e-07 105 KSP Residual norm 4.415098135658e-07 106 KSP Residual norm 4.141004592998e-07 107 KSP Residual norm 3.872263022423e-07 108 KSP Residual norm 3.699818123499e-07 109 KSP Residual norm 3.534062909536e-07 110 KSP Residual norm 3.411745302943e-07 111 KSP Residual norm 3.337405059691e-07 112 KSP Residual norm 3.197159533206e-07 113 KSP Residual norm 2.979457931726e-07 114 KSP Residual norm 2.814928022642e-07 115 KSP Residual norm 2.666243752639e-07 116 KSP Residual norm 2.494022921584e-07 117 KSP Residual norm 2.335950438739e-07 118 KSP Residual norm 2.217365127740e-07 119 KSP Residual norm 2.125183785340e-07 120 KSP Residual norm 2.047845481727e-07 121 KSP Residual norm 1.960076838724e-07 122 KSP Residual norm 1.865370342578e-07 123 KSP Residual norm 1.772211837149e-07 124 KSP Residual norm 1.700933033381e-07 125 KSP Residual norm 1.659388331644e-07 126 KSP Residual norm 1.639257570987e-07 127 KSP Residual norm 1.559262854313e-07 128 KSP Residual norm 1.441151486424e-07 129 KSP Residual norm 1.351173729002e-07 130 KSP Residual norm 1.280867407464e-07 131 KSP Residual norm 1.218354898975e-07 132 KSP Residual norm 1.159283053654e-07 133 KSP Residual norm 1.104502782780e-07 134 KSP Residual norm 1.052534706177e-07 135 KSP Residual norm 1.002513991924e-07 136 KSP Residual norm 9.560336391718e-08 137 KSP Residual norm 9.167334639759e-08 138 KSP Residual norm 8.842484622405e-08 139 KSP Residual norm 8.493709841413e-08 140 KSP Residual norm 8.090116624768e-08 141 KSP Residual norm 7.700944624303e-08 142 KSP Residual norm 7.342064321052e-08 143 KSP Residual norm 6.983167024515e-08 144 KSP Residual norm 6.626586949019e-08 145 KSP Residual norm 6.302769939494e-08 146 KSP Residual norm 6.007956798874e-08 147 KSP Residual norm 5.727741469665e-08 148 KSP Residual norm 5.463269252065e-08 149 KSP Residual norm 5.197600865948e-08 150 KSP Residual norm 4.931292629884e-08 151 KSP Residual norm 4.687729675480e-08 152 KSP Residual norm 4.464213047589e-08 153 KSP Residual norm 4.253599054541e-08 154 KSP Residual norm 4.062940030594e-08 155 KSP Residual norm 3.883304178990e-08 156 KSP Residual norm 3.713643030433e-08 157 KSP Residual norm 3.582139648539e-08 158 KSP Residual norm 3.459353100053e-08 159 KSP Residual norm 3.330506183714e-08 160 KSP Residual norm 3.223229457142e-08 161 KSP Residual norm 3.164178693698e-08 162 KSP Residual norm 3.129579963894e-08 163 KSP Residual norm 3.025331645394e-08 164 KSP Residual norm 2.864190066800e-08 165 KSP Residual norm 2.734952826236e-08 166 KSP Residual norm 2.649342347956e-08 167 KSP Residual norm 2.573787638342e-08 168 KSP Residual norm 2.481270490518e-08 169 KSP Residual norm 2.385461787244e-08 170 KSP Residual norm 2.296920915906e-08 171 KSP Residual norm 2.212748337415e-08 172 KSP Residual norm 2.141971323147e-08 173 KSP Residual norm 2.090667790760e-08 174 KSP Residual norm 2.062652011209e-08 175 KSP Residual norm 2.047510506992e-08 176 KSP Residual norm 2.017321830577e-08 177 KSP Residual norm 1.969224880418e-08 178 KSP Residual norm 1.930667469778e-08 179 KSP Residual norm 1.906847489887e-08 180 KSP Residual norm 1.889291746169e-08 181 KSP Residual norm 1.875463751422e-08 182 KSP Residual norm 1.856888599406e-08 183 KSP Residual norm 1.831531212533e-08 184 KSP Residual norm 1.796876154289e-08 185 KSP Residual norm 1.763293116928e-08 186 KSP Residual norm 1.741600973331e-08 187 KSP Residual norm 1.728755180620e-08 188 KSP Residual norm 1.710648350276e-08 189 KSP Residual norm 1.681857510747e-08 190 KSP Residual norm 1.658537363392e-08 191 KSP Residual norm 1.648433044783e-08 192 KSP Residual norm 1.645957416923e-08 193 KSP Residual norm 1.646076134727e-08 194 KSP Residual norm 1.653566384360e-08 195 KSP Residual norm 1.664353444188e-08 196 KSP Residual norm 1.640215061791e-08 197 KSP Residual norm 1.595746728640e-08 198 KSP Residual norm 1.570050479778e-08 199 KSP Residual norm 1.551393000689e-08 200 KSP Residual norm 1.524290676870e-08 201 KSP Residual norm 1.486073517388e-08 202 KSP Residual norm 1.455992163312e-08 203 KSP Residual norm 1.440712277142e-08 204 KSP Residual norm 1.433971502519e-08 205 KSP Residual norm 1.411719399819e-08 206 KSP Residual norm 1.347957704930e-08 207 KSP Residual norm 1.286978113157e-08 208 KSP Residual norm 1.243111899302e-08 209 KSP Residual norm 1.211387922082e-08 210 KSP Residual norm 1.177150206150e-08 211 KSP Residual norm 1.135223970478e-08 212 KSP Residual norm 1.091686256577e-08 213 KSP Residual norm 1.055265326448e-08 214 KSP Residual norm 1.025765327461e-08 215 KSP Residual norm 9.984888397235e-09 216 KSP Residual norm 9.774137076125e-09 217 KSP Residual norm 9.731390567033e-09 218 KSP Residual norm 9.494183985112e-09 219 KSP Residual norm 8.862225679973e-09 220 KSP Residual norm 8.339654193592e-09 221 KSP Residual norm 7.951524415470e-09 222 KSP Residual norm 7.621645258547e-09 223 KSP Residual norm 7.324209179124e-09 224 KSP Residual norm 7.039018174987e-09 225 KSP Residual norm 6.764941567527e-09 226 KSP Residual norm 6.488544970221e-09 227 KSP Residual norm 6.206243913118e-09 228 KSP Residual norm 5.945629650120e-09 229 KSP Residual norm 5.723038262232e-09 230 KSP Residual norm 5.520961834318e-09 231 KSP Residual norm 5.330906266336e-09 232 KSP Residual norm 5.150712177587e-09 233 KSP Residual norm 4.950742614194e-09 234 KSP Residual norm 4.732346994431e-09 235 KSP Residual norm 4.526338866045e-09 236 KSP Residual norm 4.353111835704e-09 237 KSP Residual norm 4.234629085609e-09 238 KSP Residual norm 4.161634506970e-09 239 KSP Residual norm 4.048431639830e-09 240 KSP Residual norm 3.868910891934e-09 241 KSP Residual norm 3.699216062807e-09 242 KSP Residual norm 3.560172587232e-09 243 KSP Residual norm 3.440425432462e-09 244 KSP Residual norm 3.345564442094e-09 245 KSP Residual norm 3.266112737545e-09 246 KSP Residual norm 3.191799255054e-09 247 KSP Residual norm 3.122444569020e-09 248 KSP Residual norm 3.049274659308e-09 249 KSP Residual norm 2.986714524481e-09 250 KSP Residual norm 2.929211448361e-09 251 KSP Residual norm 2.857339664513e-09 252 KSP Residual norm 2.776030160767e-09 253 KSP Residual norm 2.701431998494e-09 254 KSP Residual norm 2.637678474666e-09 255 KSP Residual norm 2.577189779196e-09 256 KSP Residual norm 2.524977324702e-09 257 KSP Residual norm 2.478394928740e-09 258 KSP Residual norm 2.429548446185e-09 259 KSP Residual norm 2.377557155066e-09 260 KSP Residual norm 2.321739104580e-09 261 KSP Residual norm 2.270993769113e-09 262 KSP Residual norm 2.215820761266e-09 263 KSP Residual norm 2.156923661556e-09 264 KSP Residual norm 2.123614058054e-09 265 KSP Residual norm 2.143427184012e-09 266 KSP Residual norm 2.147519926139e-09 267 KSP Residual norm 2.026105126640e-09 268 KSP Residual norm 1.933392312768e-09 269 KSP Residual norm 1.897239816468e-09 270 KSP Residual norm 1.871028148874e-09 271 KSP Residual norm 1.830888894595e-09 272 KSP Residual norm 1.776070822675e-09 273 KSP Residual norm 1.718553369389e-09 274 KSP Residual norm 1.666464191789e-09 275 KSP Residual norm 1.616608455457e-09 276 KSP Residual norm 1.583774452691e-09 277 KSP Residual norm 1.566915034219e-09 278 KSP Residual norm 1.547495039470e-09 279 KSP Residual norm 1.488061887043e-09 280 KSP Residual norm 1.415136055833e-09 281 KSP Residual norm 1.362365995817e-09 282 KSP Residual norm 1.320205691620e-09 283 KSP Residual norm 1.288435409286e-09 284 KSP Residual norm 1.267832625723e-09 285 KSP Residual norm 1.247617450737e-09 286 KSP Residual norm 1.205460138694e-09 287 KSP Residual norm 1.160053285188e-09 288 KSP Residual norm 1.122782106917e-09 289 KSP Residual norm 1.094564960441e-09 290 KSP Residual norm 1.076309668308e-09 291 KSP Residual norm 1.067519863413e-09 292 KSP Residual norm 1.046145640488e-09 293 KSP Residual norm 1.004853319640e-09 294 KSP Residual norm 9.647338552797e-10 295 KSP Residual norm 9.323972702758e-10 296 KSP Residual norm 9.025756419788e-10 297 KSP Residual norm 8.703704403962e-10 298 KSP Residual norm 8.405152264648e-10 299 KSP Residual norm 8.134611217781e-10 300 KSP Residual norm 7.911109639259e-10 Linear tao_nls_ solve converged due to CONVERGED_RTOL iterations 300 0 LS Function value: 0.069139, Step length: 0. 1 LS Function value: 0.069139, Step length: 1. stx: 0., fx: 0.069139, dgx: -5.76226e-08 sty: 0., fy: 0.069139, dgy: -5.76226e-08 36 TAO, Function value: 0.069139, Residual: 0.000170766 Residual norms for tao_nls_ solve. 0 KSP Residual norm 1.707657585701e-04 1 KSP Residual norm 1.167873028639e-04 2 KSP Residual norm 8.415828438964e-05 3 KSP Residual norm 6.420990086943e-05 4 KSP Residual norm 4.929556126222e-05 5 KSP Residual norm 4.122760027054e-05 6 KSP Residual norm 3.426375647212e-05 7 KSP Residual norm 2.968254762197e-05 8 KSP Residual norm 2.746305896855e-05 9 KSP Residual norm 2.616378100156e-05 10 KSP Residual norm 2.574383968986e-05 11 KSP Residual norm 2.571226031871e-05 12 KSP Residual norm 2.610485773584e-05 13 KSP Residual norm 2.698481645302e-05 14 KSP Residual norm 2.791341612805e-05 15 KSP Residual norm 2.902216914128e-05 16 KSP Residual norm 3.022476117873e-05 17 KSP Residual norm 3.156757670508e-05 18 KSP Residual norm 3.223843944385e-05 19 KSP Residual norm 3.213806269218e-05 20 KSP Residual norm 3.179135923942e-05 21 KSP Residual norm 3.110031162668e-05 22 KSP Residual norm 3.004310945655e-05 23 KSP Residual norm 2.876145935711e-05 24 KSP Residual norm 2.756091322005e-05 25 KSP Residual norm 2.655862790693e-05 26 KSP Residual norm 2.563174313708e-05 27 KSP Residual norm 2.461257897371e-05 28 KSP Residual norm 2.342276899174e-05 29 KSP Residual norm 2.227966580818e-05 30 KSP Residual norm 2.125331754566e-05 31 KSP Residual norm 2.026992240444e-05 32 KSP Residual norm 1.926475329308e-05 33 KSP Residual norm 1.822947393600e-05 34 KSP Residual norm 1.711133931038e-05 35 KSP Residual norm 1.581258813141e-05 36 KSP Residual norm 1.437412302975e-05 37 KSP Residual norm 1.297183725925e-05 38 KSP Residual norm 1.163618503388e-05 39 KSP Residual norm 1.030748531183e-05 40 KSP Residual norm 9.037582426467e-06 41 KSP Residual norm 7.906845081692e-06 42 KSP Residual norm 6.941571092584e-06 43 KSP Residual norm 6.164503543172e-06 44 KSP Residual norm 5.591736260497e-06 45 KSP Residual norm 5.134164461140e-06 46 KSP Residual norm 4.623271554924e-06 47 KSP Residual norm 4.133588216217e-06 48 KSP Residual norm 3.741917347771e-06 49 KSP Residual norm 3.416791042368e-06 50 KSP Residual norm 3.142629321298e-06 51 KSP Residual norm 2.930136834211e-06 52 KSP Residual norm 2.787442840592e-06 53 KSP Residual norm 2.705690665068e-06 54 KSP Residual norm 2.580349972411e-06 55 KSP Residual norm 2.407321435787e-06 56 KSP Residual norm 2.293024292513e-06 57 KSP Residual norm 2.231853980662e-06 58 KSP Residual norm 2.179432659986e-06 59 KSP Residual norm 2.127252223652e-06 60 KSP Residual norm 2.095446446442e-06 61 KSP Residual norm 2.080311266014e-06 62 KSP Residual norm 2.079410239300e-06 63 KSP Residual norm 2.091797054534e-06 64 KSP Residual norm 2.108369511585e-06 65 KSP Residual norm 2.114785292623e-06 66 KSP Residual norm 2.105026965491e-06 67 KSP Residual norm 2.081677237537e-06 68 KSP Residual norm 2.053053444407e-06 69 KSP Residual norm 2.024031789971e-06 70 KSP Residual norm 1.982170278186e-06 71 KSP Residual norm 1.919060671423e-06 72 KSP Residual norm 1.831348880795e-06 73 KSP Residual norm 1.723519808074e-06 74 KSP Residual norm 1.610621110992e-06 75 KSP Residual norm 1.511153288409e-06 76 KSP Residual norm 1.420818637955e-06 77 KSP Residual norm 1.336166900773e-06 78 KSP Residual norm 1.258594852395e-06 79 KSP Residual norm 1.176505006853e-06 80 KSP Residual norm 1.093257250053e-06 81 KSP Residual norm 1.017611873143e-06 82 KSP Residual norm 9.556077304892e-07 83 KSP Residual norm 9.078453488418e-07 84 KSP Residual norm 8.614518122797e-07 85 KSP Residual norm 8.258836490512e-07 86 KSP Residual norm 7.973275706333e-07 87 KSP Residual norm 7.627615805304e-07 88 KSP Residual norm 7.231925347081e-07 89 KSP Residual norm 6.850275136454e-07 90 KSP Residual norm 6.575755442377e-07 91 KSP Residual norm 6.333200762457e-07 92 KSP Residual norm 6.127038625838e-07 93 KSP Residual norm 5.997202646703e-07 94 KSP Residual norm 5.844127292855e-07 95 KSP Residual norm 5.725243789829e-07 96 KSP Residual norm 5.664068429427e-07 97 KSP Residual norm 5.545515732311e-07 98 KSP Residual norm 5.418172629095e-07 99 KSP Residual norm 5.301366468431e-07 100 KSP Residual norm 5.186301661820e-07 101 KSP Residual norm 5.061366382481e-07 102 KSP Residual norm 4.939821747572e-07 103 KSP Residual norm 4.906025809768e-07 104 KSP Residual norm 4.986268040898e-07 105 KSP Residual norm 4.791509945207e-07 106 KSP Residual norm 4.499551523455e-07 107 KSP Residual norm 4.283227077413e-07 108 KSP Residual norm 4.143797452870e-07 109 KSP Residual norm 3.996659170469e-07 110 KSP Residual norm 3.844585144529e-07 111 KSP Residual norm 3.688688863523e-07 112 KSP Residual norm 3.510366442037e-07 113 KSP Residual norm 3.313899240908e-07 114 KSP Residual norm 3.119807344110e-07 115 KSP Residual norm 2.946106228152e-07 116 KSP Residual norm 2.797857640627e-07 117 KSP Residual norm 2.651171084253e-07 118 KSP Residual norm 2.502218726204e-07 119 KSP Residual norm 2.352022922858e-07 120 KSP Residual norm 2.225572664963e-07 121 KSP Residual norm 2.115521164502e-07 122 KSP Residual norm 2.031880226274e-07 123 KSP Residual norm 1.921992368889e-07 124 KSP Residual norm 1.788205638542e-07 125 KSP Residual norm 1.668927116885e-07 126 KSP Residual norm 1.568732072441e-07 127 KSP Residual norm 1.478269808254e-07 128 KSP Residual norm 1.396461251644e-07 129 KSP Residual norm 1.326271760198e-07 130 KSP Residual norm 1.268032211284e-07 131 KSP Residual norm 1.222771941767e-07 132 KSP Residual norm 1.172479380978e-07 133 KSP Residual norm 1.112524241170e-07 134 KSP Residual norm 1.060370386411e-07 135 KSP Residual norm 1.027091640210e-07 136 KSP Residual norm 1.011284262197e-07 137 KSP Residual norm 1.009302934881e-07 138 KSP Residual norm 1.008634018783e-07 139 KSP Residual norm 9.981373376464e-08 140 KSP Residual norm 9.852203848790e-08 141 KSP Residual norm 9.792679176115e-08 142 KSP Residual norm 9.791052940320e-08 143 KSP Residual norm 9.803615279412e-08 144 KSP Residual norm 9.817002455532e-08 145 KSP Residual norm 9.871905471904e-08 146 KSP Residual norm 9.959021399558e-08 147 KSP Residual norm 1.007005037125e-07 148 KSP Residual norm 1.024647536368e-07 149 KSP Residual norm 1.046866273783e-07 150 KSP Residual norm 1.067347576037e-07 151 KSP Residual norm 1.092095960517e-07 152 KSP Residual norm 1.123323527602e-07 153 KSP Residual norm 1.141506822591e-07 154 KSP Residual norm 1.136828803471e-07 155 KSP Residual norm 1.130298573719e-07 156 KSP Residual norm 1.120837810302e-07 157 KSP Residual norm 1.102593902006e-07 158 KSP Residual norm 1.083626465945e-07 159 KSP Residual norm 1.061634449570e-07 160 KSP Residual norm 1.036390982937e-07 161 KSP Residual norm 1.014162604764e-07 162 KSP Residual norm 9.885672105835e-08 163 KSP Residual norm 9.618613328330e-08 164 KSP Residual norm 9.339305581197e-08 165 KSP Residual norm 9.049776379532e-08 166 KSP Residual norm 8.733090940797e-08 167 KSP Residual norm 8.312464577028e-08 168 KSP Residual norm 7.823524911565e-08 169 KSP Residual norm 7.386390012231e-08 170 KSP Residual norm 7.007590863583e-08 171 KSP Residual norm 6.636362325774e-08 172 KSP Residual norm 6.258866535258e-08 173 KSP Residual norm 5.899783334304e-08 174 KSP Residual norm 5.587473328144e-08 175 KSP Residual norm 5.321021239028e-08 176 KSP Residual norm 5.075035833881e-08 177 KSP Residual norm 4.771431832438e-08 178 KSP Residual norm 4.430614171717e-08 179 KSP Residual norm 4.123638630487e-08 180 KSP Residual norm 3.848735199687e-08 181 KSP Residual norm 3.602847834289e-08 182 KSP Residual norm 3.369353533599e-08 183 KSP Residual norm 3.150448768313e-08 184 KSP Residual norm 2.944102963422e-08 185 KSP Residual norm 2.747594884921e-08 186 KSP Residual norm 2.576834205748e-08 187 KSP Residual norm 2.439486921143e-08 188 KSP Residual norm 2.333954790482e-08 189 KSP Residual norm 2.250313431017e-08 190 KSP Residual norm 2.173706205248e-08 191 KSP Residual norm 2.097181116780e-08 192 KSP Residual norm 2.011251314287e-08 193 KSP Residual norm 1.924273420177e-08 194 KSP Residual norm 1.851705545873e-08 195 KSP Residual norm 1.794883380969e-08 196 KSP Residual norm 1.742672225104e-08 197 KSP Residual norm 1.685353704929e-08 198 KSP Residual norm 1.632820667072e-08 199 KSP Residual norm 1.587600304949e-08 200 KSP Residual norm 1.565288665860e-08 201 KSP Residual norm 1.575284307828e-08 202 KSP Residual norm 1.549216005367e-08 203 KSP Residual norm 1.461822740573e-08 204 KSP Residual norm 1.397639896695e-08 205 KSP Residual norm 1.366276669589e-08 206 KSP Residual norm 1.350108054272e-08 207 KSP Residual norm 1.337683241572e-08 208 KSP Residual norm 1.319557296010e-08 209 KSP Residual norm 1.313421360883e-08 210 KSP Residual norm 1.325449593565e-08 211 KSP Residual norm 1.329066154748e-08 212 KSP Residual norm 1.303957624654e-08 213 KSP Residual norm 1.279946632806e-08 214 KSP Residual norm 1.268723145893e-08 215 KSP Residual norm 1.256941224314e-08 216 KSP Residual norm 1.238617189525e-08 217 KSP Residual norm 1.215440787816e-08 218 KSP Residual norm 1.192379465294e-08 219 KSP Residual norm 1.171066328695e-08 220 KSP Residual norm 1.152015703672e-08 221 KSP Residual norm 1.134379097132e-08 222 KSP Residual norm 1.112719359812e-08 223 KSP Residual norm 1.084715975125e-08 224 KSP Residual norm 1.052547454562e-08 225 KSP Residual norm 1.021485345315e-08 226 KSP Residual norm 9.900372549234e-09 227 KSP Residual norm 9.575039866368e-09 228 KSP Residual norm 9.266923174078e-09 229 KSP Residual norm 8.987713571030e-09 230 KSP Residual norm 8.692757849371e-09 231 KSP Residual norm 8.368465123732e-09 232 KSP Residual norm 8.071281300885e-09 233 KSP Residual norm 7.823599497009e-09 234 KSP Residual norm 7.585596253786e-09 235 KSP Residual norm 7.332240183808e-09 236 KSP Residual norm 7.084600215520e-09 237 KSP Residual norm 6.891430674061e-09 238 KSP Residual norm 6.786074861194e-09 239 KSP Residual norm 6.692372652028e-09 240 KSP Residual norm 6.467608487430e-09 241 KSP Residual norm 6.168619694736e-09 242 KSP Residual norm 5.901912253007e-09 243 KSP Residual norm 5.670374875985e-09 244 KSP Residual norm 5.469686818086e-09 245 KSP Residual norm 5.274299140643e-09 246 KSP Residual norm 5.090256167267e-09 247 KSP Residual norm 4.958934652715e-09 248 KSP Residual norm 4.889503892318e-09 249 KSP Residual norm 4.861603010998e-09 250 KSP Residual norm 4.699060059681e-09 251 KSP Residual norm 4.416342977392e-09 252 KSP Residual norm 4.181683813108e-09 253 KSP Residual norm 3.992818153591e-09 254 KSP Residual norm 3.827854777131e-09 255 KSP Residual norm 3.676228561571e-09 256 KSP Residual norm 3.558026713967e-09 257 KSP Residual norm 3.475895419347e-09 258 KSP Residual norm 3.383161584288e-09 259 KSP Residual norm 3.247542242359e-09 260 KSP Residual norm 3.104678467003e-09 261 KSP Residual norm 2.983731689526e-09 262 KSP Residual norm 2.868973895929e-09 263 KSP Residual norm 2.756312536227e-09 264 KSP Residual norm 2.660342188698e-09 265 KSP Residual norm 2.562210414582e-09 266 KSP Residual norm 2.472810823933e-09 267 KSP Residual norm 2.387780625518e-09 268 KSP Residual norm 2.299690912190e-09 269 KSP Residual norm 2.227358091571e-09 270 KSP Residual norm 2.168218680359e-09 271 KSP Residual norm 2.121285817997e-09 272 KSP Residual norm 2.062126960885e-09 273 KSP Residual norm 1.982045578001e-09 274 KSP Residual norm 1.903488492532e-09 275 KSP Residual norm 1.836928547672e-09 276 KSP Residual norm 1.781312157776e-09 277 KSP Residual norm 1.732294003257e-09 278 KSP Residual norm 1.692950447277e-09 Linear tao_nls_ solve converged due to CONVERGED_RTOL iterations 278 0 LS Function value: 0.069139, Step length: 0. 1 LS Function value: 0.0691389, Step length: 1. stx: 0., fx: 0.069139, dgx: -8.10552e-08 sty: 0., fy: 0.069139, dgy: -8.10552e-08 37 TAO, Function value: 0.0691389, Residual: 3.92409e-05 Residual norms for tao_nls_ solve. 0 KSP Residual norm 3.924091132336e-05 1 KSP Residual norm 3.159588903333e-05 2 KSP Residual norm 2.680847184859e-05 3 KSP Residual norm 2.169658368750e-05 4 KSP Residual norm 2.082560831984e-05 5 KSP Residual norm 2.246740364175e-05 6 KSP Residual norm 2.379841655164e-05 7 KSP Residual norm 2.669543774025e-05 8 KSP Residual norm 3.031867134564e-05 9 KSP Residual norm 3.343934345392e-05 10 KSP Residual norm 3.713504156680e-05 11 KSP Residual norm 4.082063419284e-05 12 KSP Residual norm 4.431079877962e-05 13 KSP Residual norm 4.755775646106e-05 14 KSP Residual norm 4.946044573727e-05 15 KSP Residual norm 5.114368081534e-05 16 KSP Residual norm 5.305034709701e-05 17 KSP Residual norm 5.433506419230e-05 18 KSP Residual norm 5.463993333681e-05 19 KSP Residual norm 5.458019430764e-05 20 KSP Residual norm 5.466572640803e-05 21 KSP Residual norm 5.401698543639e-05 22 KSP Residual norm 5.272065175588e-05 23 KSP Residual norm 5.177301633917e-05 24 KSP Residual norm 5.095228398117e-05 25 KSP Residual norm 4.949164230359e-05 26 KSP Residual norm 4.754164892346e-05 27 KSP Residual norm 4.555997114297e-05 28 KSP Residual norm 4.344177005248e-05 29 KSP Residual norm 4.109273104438e-05 30 KSP Residual norm 3.839098195784e-05 31 KSP Residual norm 3.549908867075e-05 32 KSP Residual norm 3.261548113105e-05 33 KSP Residual norm 2.996916379797e-05 34 KSP Residual norm 2.775013293640e-05 35 KSP Residual norm 2.578824176074e-05 36 KSP Residual norm 2.346942322672e-05 37 KSP Residual norm 2.090695184492e-05 38 KSP Residual norm 1.865024270052e-05 39 KSP Residual norm 1.680958711451e-05 40 KSP Residual norm 1.528786310090e-05 41 KSP Residual norm 1.396209603227e-05 42 KSP Residual norm 1.278937897456e-05 43 KSP Residual norm 1.178390478297e-05 44 KSP Residual norm 1.094397721581e-05 45 KSP Residual norm 1.026245877782e-05 46 KSP Residual norm 9.691103801495e-06 47 KSP Residual norm 9.204094895478e-06 48 KSP Residual norm 8.823290122583e-06 49 KSP Residual norm 8.555903837513e-06 50 KSP Residual norm 8.257066124473e-06 51 KSP Residual norm 7.857181241400e-06 52 KSP Residual norm 7.560375148993e-06 53 KSP Residual norm 7.433738214878e-06 54 KSP Residual norm 7.494258161375e-06 55 KSP Residual norm 7.818562225430e-06 56 KSP Residual norm 8.009692432934e-06 57 KSP Residual norm 7.769541607854e-06 58 KSP Residual norm 7.689620507611e-06 59 KSP Residual norm 7.736725490907e-06 60 KSP Residual norm 7.781579669300e-06 61 KSP Residual norm 7.777328036654e-06 62 KSP Residual norm 7.695967819753e-06 63 KSP Residual norm 7.520616918367e-06 64 KSP Residual norm 7.284395410821e-06 65 KSP Residual norm 7.018726530927e-06 66 KSP Residual norm 6.720388647905e-06 67 KSP Residual norm 6.373396909536e-06 68 KSP Residual norm 6.029727902108e-06 69 KSP Residual norm 5.715214271558e-06 70 KSP Residual norm 5.383237001522e-06 71 KSP Residual norm 5.021627696359e-06 72 KSP Residual norm 4.683618936244e-06 73 KSP Residual norm 4.376836951880e-06 74 KSP Residual norm 4.117707364556e-06 75 KSP Residual norm 3.939010737751e-06 76 KSP Residual norm 3.771161230191e-06 77 KSP Residual norm 3.545421932318e-06 78 KSP Residual norm 3.323560457881e-06 79 KSP Residual norm 3.155045757958e-06 80 KSP Residual norm 3.033636461703e-06 81 KSP Residual norm 2.938750765557e-06 82 KSP Residual norm 2.854798826881e-06 83 KSP Residual norm 2.783628526563e-06 84 KSP Residual norm 2.705309347462e-06 85 KSP Residual norm 2.629201237899e-06 86 KSP Residual norm 2.546951205553e-06 87 KSP Residual norm 2.469169356127e-06 88 KSP Residual norm 2.434070801217e-06 89 KSP Residual norm 2.414313901551e-06 90 KSP Residual norm 2.408115915811e-06 91 KSP Residual norm 2.388705408173e-06 92 KSP Residual norm 2.370774598300e-06 93 KSP Residual norm 2.381054865115e-06 94 KSP Residual norm 2.386727201073e-06 95 KSP Residual norm 2.402458656873e-06 96 KSP Residual norm 2.415020716690e-06 97 KSP Residual norm 2.416186210172e-06 98 KSP Residual norm 2.434605258561e-06 99 KSP Residual norm 2.435970501531e-06 100 KSP Residual norm 2.424472937930e-06 101 KSP Residual norm 2.404426773496e-06 102 KSP Residual norm 2.370208359348e-06 103 KSP Residual norm 2.333591624146e-06 104 KSP Residual norm 2.309240291533e-06 105 KSP Residual norm 2.320458759365e-06 106 KSP Residual norm 2.233089782691e-06 107 KSP Residual norm 2.098807113975e-06 108 KSP Residual norm 2.011209469131e-06 109 KSP Residual norm 1.912028448982e-06 110 KSP Residual norm 1.824265866449e-06 111 KSP Residual norm 1.755756916616e-06 112 KSP Residual norm 1.700342170367e-06 113 KSP Residual norm 1.642705335816e-06 114 KSP Residual norm 1.593420953281e-06 115 KSP Residual norm 1.550524748981e-06 116 KSP Residual norm 1.498768796001e-06 117 KSP Residual norm 1.441101287538e-06 118 KSP Residual norm 1.395997737659e-06 119 KSP Residual norm 1.362624946537e-06 120 KSP Residual norm 1.303944800517e-06 121 KSP Residual norm 1.214649891239e-06 122 KSP Residual norm 1.128279068078e-06 123 KSP Residual norm 1.052921869537e-06 124 KSP Residual norm 9.907817180777e-07 125 KSP Residual norm 9.328307101049e-07 126 KSP Residual norm 8.688736068153e-07 127 KSP Residual norm 7.958917512809e-07 128 KSP Residual norm 7.361225474413e-07 129 KSP Residual norm 6.935773828936e-07 130 KSP Residual norm 6.506458802861e-07 131 KSP Residual norm 5.988776170164e-07 132 KSP Residual norm 5.555169038983e-07 133 KSP Residual norm 5.254157594628e-07 134 KSP Residual norm 5.016168693054e-07 135 KSP Residual norm 4.782635180747e-07 136 KSP Residual norm 4.535505009199e-07 137 KSP Residual norm 4.297857244934e-07 138 KSP Residual norm 4.079458619665e-07 139 KSP Residual norm 3.884332533839e-07 140 KSP Residual norm 3.696153176142e-07 141 KSP Residual norm 3.509459119069e-07 142 KSP Residual norm 3.325811798637e-07 143 KSP Residual norm 3.140172261874e-07 144 KSP Residual norm 2.941599764029e-07 145 KSP Residual norm 2.750910456826e-07 146 KSP Residual norm 2.571693525666e-07 147 KSP Residual norm 2.400655130442e-07 148 KSP Residual norm 2.237792969833e-07 149 KSP Residual norm 2.091956167525e-07 150 KSP Residual norm 1.959476057154e-07 151 KSP Residual norm 1.843236104748e-07 152 KSP Residual norm 1.743213178428e-07 153 KSP Residual norm 1.649789780075e-07 154 KSP Residual norm 1.576138362677e-07 155 KSP Residual norm 1.555204673073e-07 156 KSP Residual norm 1.545015706350e-07 157 KSP Residual norm 1.441146107235e-07 158 KSP Residual norm 1.351436698122e-07 159 KSP Residual norm 1.296631785749e-07 160 KSP Residual norm 1.256031523036e-07 161 KSP Residual norm 1.225362450471e-07 162 KSP Residual norm 1.196248949689e-07 163 KSP Residual norm 1.166104826641e-07 164 KSP Residual norm 1.127187694211e-07 165 KSP Residual norm 1.079954463166e-07 166 KSP Residual norm 1.033283789691e-07 167 KSP Residual norm 9.932930337503e-08 168 KSP Residual norm 9.611995528521e-08 169 KSP Residual norm 9.325532035022e-08 170 KSP Residual norm 9.000501548197e-08 171 KSP Residual norm 8.705215024878e-08 172 KSP Residual norm 8.525544429438e-08 173 KSP Residual norm 8.434403016601e-08 174 KSP Residual norm 8.381499947888e-08 175 KSP Residual norm 8.291905513150e-08 176 KSP Residual norm 8.155393708863e-08 177 KSP Residual norm 8.023300041365e-08 178 KSP Residual norm 7.945713047814e-08 179 KSP Residual norm 7.874557890331e-08 180 KSP Residual norm 7.762857535268e-08 181 KSP Residual norm 7.641624084555e-08 182 KSP Residual norm 7.528501454651e-08 183 KSP Residual norm 7.455823174954e-08 184 KSP Residual norm 7.397461161219e-08 185 KSP Residual norm 7.332166173297e-08 186 KSP Residual norm 7.268423402239e-08 187 KSP Residual norm 7.221215375001e-08 188 KSP Residual norm 7.160517182541e-08 189 KSP Residual norm 7.101520457912e-08 190 KSP Residual norm 7.061957179468e-08 191 KSP Residual norm 7.057063823456e-08 192 KSP Residual norm 7.029530685468e-08 193 KSP Residual norm 6.955782192685e-08 194 KSP Residual norm 6.884194034901e-08 195 KSP Residual norm 6.826716999262e-08 196 KSP Residual norm 6.763824423487e-08 197 KSP Residual norm 6.684813321254e-08 198 KSP Residual norm 6.602813632744e-08 199 KSP Residual norm 6.503123599846e-08 200 KSP Residual norm 6.375320499297e-08 201 KSP Residual norm 6.237274003757e-08 202 KSP Residual norm 6.138235732902e-08 203 KSP Residual norm 6.138912996254e-08 204 KSP Residual norm 6.111911518239e-08 205 KSP Residual norm 5.810025173867e-08 206 KSP Residual norm 5.539015720286e-08 207 KSP Residual norm 5.494012987503e-08 208 KSP Residual norm 5.500784645239e-08 209 KSP Residual norm 5.227843587816e-08 210 KSP Residual norm 4.952589901694e-08 211 KSP Residual norm 4.775986064511e-08 212 KSP Residual norm 4.623609878863e-08 213 KSP Residual norm 4.474327207154e-08 214 KSP Residual norm 4.336242675191e-08 215 KSP Residual norm 4.192925345877e-08 216 KSP Residual norm 4.042981431518e-08 217 KSP Residual norm 3.892737608809e-08 218 KSP Residual norm 3.745419450203e-08 219 KSP Residual norm 3.602869996758e-08 220 KSP Residual norm 3.463819517987e-08 221 KSP Residual norm 3.333540287651e-08 222 KSP Residual norm 3.214432404228e-08 223 KSP Residual norm 3.123741213725e-08 224 KSP Residual norm 3.028735590016e-08 225 KSP Residual norm 2.888963824032e-08 226 KSP Residual norm 2.737075227880e-08 227 KSP Residual norm 2.614742539006e-08 228 KSP Residual norm 2.512374086339e-08 229 KSP Residual norm 2.408363704756e-08 230 KSP Residual norm 2.296350728133e-08 231 KSP Residual norm 2.187712163427e-08 232 KSP Residual norm 2.087336551888e-08 233 KSP Residual norm 1.993000837977e-08 234 KSP Residual norm 1.905487307549e-08 235 KSP Residual norm 1.832596190520e-08 236 KSP Residual norm 1.769826238990e-08 237 KSP Residual norm 1.711076397427e-08 238 KSP Residual norm 1.663040637874e-08 239 KSP Residual norm 1.629559898657e-08 240 KSP Residual norm 1.592639442892e-08 241 KSP Residual norm 1.539808347298e-08 242 KSP Residual norm 1.485220469809e-08 243 KSP Residual norm 1.434138245903e-08 244 KSP Residual norm 1.387754076609e-08 245 KSP Residual norm 1.341715955749e-08 246 KSP Residual norm 1.295564714711e-08 247 KSP Residual norm 1.256599594403e-08 248 KSP Residual norm 1.218553437189e-08 249 KSP Residual norm 1.180014279099e-08 250 KSP Residual norm 1.137395503650e-08 251 KSP Residual norm 1.082869413642e-08 252 KSP Residual norm 1.024892688793e-08 253 KSP Residual norm 9.748278359024e-09 254 KSP Residual norm 9.371411413738e-09 255 KSP Residual norm 9.129541073063e-09 256 KSP Residual norm 9.160409129689e-09 257 KSP Residual norm 9.130467154404e-09 258 KSP Residual norm 8.515979373687e-09 259 KSP Residual norm 8.048653142042e-09 260 KSP Residual norm 7.768372617688e-09 261 KSP Residual norm 7.571948228546e-09 262 KSP Residual norm 7.391276808031e-09 263 KSP Residual norm 7.191581490295e-09 264 KSP Residual norm 6.993752409868e-09 265 KSP Residual norm 6.760738452993e-09 266 KSP Residual norm 6.558613465550e-09 267 KSP Residual norm 6.380845951088e-09 268 KSP Residual norm 6.213650010628e-09 269 KSP Residual norm 6.071694331454e-09 270 KSP Residual norm 5.913169048931e-09 271 KSP Residual norm 5.707594304179e-09 272 KSP Residual norm 5.471543018045e-09 273 KSP Residual norm 5.261698584148e-09 274 KSP Residual norm 5.074476274979e-09 275 KSP Residual norm 4.893666184240e-09 276 KSP Residual norm 4.711910444419e-09 277 KSP Residual norm 4.556080449975e-09 278 KSP Residual norm 4.426410404099e-09 279 KSP Residual norm 4.297557408507e-09 280 KSP Residual norm 4.119929142459e-09 281 KSP Residual norm 3.964677214119e-09 282 KSP Residual norm 3.856581166798e-09 283 KSP Residual norm 3.773958603397e-09 284 KSP Residual norm 3.697448281057e-09 285 KSP Residual norm 3.629454897386e-09 286 KSP Residual norm 3.579227360844e-09 287 KSP Residual norm 3.541511691688e-09 288 KSP Residual norm 3.497738539088e-09 289 KSP Residual norm 3.462377239542e-09 290 KSP Residual norm 3.441835612413e-09 291 KSP Residual norm 3.368004788789e-09 292 KSP Residual norm 3.213811188403e-09 293 KSP Residual norm 3.061379890361e-09 294 KSP Residual norm 2.946288666505e-09 295 KSP Residual norm 2.851493539112e-09 296 KSP Residual norm 2.765293075161e-09 297 KSP Residual norm 2.681158942503e-09 298 KSP Residual norm 2.607752428072e-09 299 KSP Residual norm 2.540005837295e-09 300 KSP Residual norm 2.468474451432e-09 301 KSP Residual norm 2.387610749450e-09 302 KSP Residual norm 2.309297325862e-09 303 KSP Residual norm 2.245751974908e-09 304 KSP Residual norm 2.217709022616e-09 305 KSP Residual norm 2.238177233036e-09 306 KSP Residual norm 2.258352291550e-09 307 KSP Residual norm 2.156855615727e-09 308 KSP Residual norm 2.048373899686e-09 309 KSP Residual norm 1.988633994813e-09 310 KSP Residual norm 1.942299397991e-09 311 KSP Residual norm 1.895815057275e-09 312 KSP Residual norm 1.863919348005e-09 313 KSP Residual norm 1.846018260376e-09 314 KSP Residual norm 1.822965506684e-09 315 KSP Residual norm 1.775214655158e-09 316 KSP Residual norm 1.705872509080e-09 317 KSP Residual norm 1.639004678464e-09 318 KSP Residual norm 1.585096041812e-09 319 KSP Residual norm 1.548189837892e-09 320 KSP Residual norm 1.516849654853e-09 321 KSP Residual norm 1.476455845943e-09 322 KSP Residual norm 1.418660698980e-09 323 KSP Residual norm 1.361728969855e-09 324 KSP Residual norm 1.312707416274e-09 325 KSP Residual norm 1.273496005044e-09 326 KSP Residual norm 1.240315949318e-09 327 KSP Residual norm 1.215563821705e-09 328 KSP Residual norm 1.202356572134e-09 329 KSP Residual norm 1.190112894978e-09 330 KSP Residual norm 1.170033035545e-09 331 KSP Residual norm 1.153182683134e-09 332 KSP Residual norm 1.143607958664e-09 333 KSP Residual norm 1.124139497114e-09 334 KSP Residual norm 1.084888815782e-09 335 KSP Residual norm 1.045874646565e-09 336 KSP Residual norm 1.011632382101e-09 337 KSP Residual norm 9.832435043945e-10 338 KSP Residual norm 9.673057380653e-10 339 KSP Residual norm 9.530652023295e-10 340 KSP Residual norm 9.177831672136e-10 341 KSP Residual norm 8.720385050482e-10 342 KSP Residual norm 8.328865727355e-10 343 KSP Residual norm 7.997482635430e-10 344 KSP Residual norm 7.682856547373e-10 345 KSP Residual norm 7.392079011127e-10 346 KSP Residual norm 7.100396271068e-10 347 KSP Residual norm 6.835039802313e-10 348 KSP Residual norm 6.616949398721e-10 349 KSP Residual norm 6.445447460209e-10 350 KSP Residual norm 6.321444816311e-10 351 KSP Residual norm 6.242579007434e-10 352 KSP Residual norm 6.197896704245e-10 353 KSP Residual norm 6.218783067693e-10 354 KSP Residual norm 6.252127172904e-10 355 KSP Residual norm 6.108735991856e-10 356 KSP Residual norm 5.911201668078e-10 357 KSP Residual norm 5.706482326950e-10 358 KSP Residual norm 5.497385106675e-10 359 KSP Residual norm 5.297110405976e-10 360 KSP Residual norm 5.100372958560e-10 361 KSP Residual norm 4.909924840123e-10 362 KSP Residual norm 4.717046645839e-10 363 KSP Residual norm 4.540728943079e-10 364 KSP Residual norm 4.396411156695e-10 365 KSP Residual norm 4.290067951976e-10 366 KSP Residual norm 4.224291447189e-10 367 KSP Residual norm 4.130623988227e-10 368 KSP Residual norm 3.960244713361e-10 369 KSP Residual norm 3.782239567988e-10 Linear tao_nls_ solve converged due to CONVERGED_RTOL iterations 369 0 LS Function value: 0.0691389, Step length: 0. 1 LS Function value: 0.069139, Step length: 1. stx: 0., fx: 0.0691389, dgx: -7.7208e-08 sty: 0., fy: 0.0691389, dgy: -7.7208e-08 2 LS Function value: 0.0691389, Step length: 0.336196 stx: 0., fx: 0.0691389, dgx: -7.7208e-08 sty: 1., fy: 0.069139, dgy: 5.83052e-07 38 TAO, Function value: 0.0691389, Residual: 6.8474e-05 Residual norms for tao_nls_ solve. 0 KSP Residual norm 6.847403875890e-05 1 KSP Residual norm 5.021763929289e-05 2 KSP Residual norm 3.924616826055e-05 3 KSP Residual norm 2.896535855276e-05 4 KSP Residual norm 2.312203529870e-05 5 KSP Residual norm 2.127788423731e-05 6 KSP Residual norm 1.985724703851e-05 7 KSP Residual norm 1.949569563816e-05 8 KSP Residual norm 2.054135290421e-05 9 KSP Residual norm 2.178312649540e-05 10 KSP Residual norm 2.347188872404e-05 11 KSP Residual norm 2.556183629351e-05 12 KSP Residual norm 2.791464834356e-05 13 KSP Residual norm 3.025893275227e-05 14 KSP Residual norm 3.190417270595e-05 15 KSP Residual norm 3.298122277692e-05 16 KSP Residual norm 3.380453656987e-05 17 KSP Residual norm 3.431204653659e-05 18 KSP Residual norm 3.441747698275e-05 19 KSP Residual norm 3.386490693860e-05 20 KSP Residual norm 3.328729702156e-05 21 KSP Residual norm 3.292427672769e-05 22 KSP Residual norm 3.214823633524e-05 23 KSP Residual norm 3.092519763310e-05 24 KSP Residual norm 2.984696332855e-05 25 KSP Residual norm 2.884522960782e-05 26 KSP Residual norm 2.761633126675e-05 27 KSP Residual norm 2.640992530306e-05 28 KSP Residual norm 2.531669884811e-05 29 KSP Residual norm 2.426423683229e-05 30 KSP Residual norm 2.307097034442e-05 31 KSP Residual norm 2.166796847988e-05 32 KSP Residual norm 2.007777710218e-05 33 KSP Residual norm 1.840910922068e-05 34 KSP Residual norm 1.678889342028e-05 35 KSP Residual norm 1.523753008677e-05 36 KSP Residual norm 1.374305624848e-05 37 KSP Residual norm 1.234806639108e-05 38 KSP Residual norm 1.109764350319e-05 39 KSP Residual norm 1.000491263823e-05 40 KSP Residual norm 9.085713371047e-06 41 KSP Residual norm 8.355197148921e-06 42 KSP Residual norm 7.754631257663e-06 43 KSP Residual norm 7.081258676780e-06 44 KSP Residual norm 6.380678511422e-06 45 KSP Residual norm 5.821489888154e-06 46 KSP Residual norm 5.382633559063e-06 47 KSP Residual norm 5.014695476155e-06 48 KSP Residual norm 4.689829374242e-06 49 KSP Residual norm 4.403399321393e-06 50 KSP Residual norm 4.169910525861e-06 51 KSP Residual norm 3.998527117966e-06 52 KSP Residual norm 3.885805471665e-06 53 KSP Residual norm 3.749345209268e-06 54 KSP Residual norm 3.536823027246e-06 55 KSP Residual norm 3.358186680917e-06 56 KSP Residual norm 3.270001428473e-06 57 KSP Residual norm 3.245674966461e-06 58 KSP Residual norm 3.214285498573e-06 59 KSP Residual norm 3.153587773201e-06 60 KSP Residual norm 3.101055787871e-06 61 KSP Residual norm 3.063255522704e-06 62 KSP Residual norm 3.023901127842e-06 63 KSP Residual norm 2.970569554467e-06 64 KSP Residual norm 2.906528221827e-06 65 KSP Residual norm 2.832610623287e-06 66 KSP Residual norm 2.738492907139e-06 67 KSP Residual norm 2.610571164216e-06 68 KSP Residual norm 2.466444485140e-06 69 KSP Residual norm 2.312894496658e-06 70 KSP Residual norm 2.133985737217e-06 71 KSP Residual norm 1.946066652990e-06 72 KSP Residual norm 1.781883577445e-06 73 KSP Residual norm 1.643965371726e-06 74 KSP Residual norm 1.526309097601e-06 75 KSP Residual norm 1.415592057182e-06 76 KSP Residual norm 1.287593852413e-06 77 KSP Residual norm 1.158759628410e-06 78 KSP Residual norm 1.052241057831e-06 79 KSP Residual norm 9.667577222206e-07 80 KSP Residual norm 8.980232507478e-07 81 KSP Residual norm 8.402624264469e-07 82 KSP Residual norm 7.858473739181e-07 83 KSP Residual norm 7.365458184581e-07 84 KSP Residual norm 6.888593239699e-07 85 KSP Residual norm 6.446843142207e-07 86 KSP Residual norm 6.021244104973e-07 87 KSP Residual norm 5.613233264290e-07 88 KSP Residual norm 5.309440274525e-07 89 KSP Residual norm 5.063267839357e-07 90 KSP Residual norm 4.868391929899e-07 91 KSP Residual norm 4.669081535272e-07 92 KSP Residual norm 4.493303695223e-07 93 KSP Residual norm 4.391618630402e-07 94 KSP Residual norm 4.303375786786e-07 95 KSP Residual norm 4.255945809855e-07 96 KSP Residual norm 4.235472760231e-07 97 KSP Residual norm 4.227982254709e-07 98 KSP Residual norm 4.276633841079e-07 99 KSP Residual norm 4.330254119372e-07 100 KSP Residual norm 4.430035892362e-07 101 KSP Residual norm 4.553946478420e-07 102 KSP Residual norm 4.553718399805e-07 103 KSP Residual norm 4.467522534715e-07 104 KSP Residual norm 4.458258368338e-07 105 KSP Residual norm 4.458594476470e-07 106 KSP Residual norm 4.455633095627e-07 107 KSP Residual norm 4.434666899370e-07 108 KSP Residual norm 4.428317303796e-07 109 KSP Residual norm 4.360813519172e-07 110 KSP Residual norm 4.270788286966e-07 111 KSP Residual norm 4.190389554313e-07 112 KSP Residual norm 4.126859103236e-07 113 KSP Residual norm 4.060720858341e-07 114 KSP Residual norm 4.000883900815e-07 115 KSP Residual norm 3.941936969889e-07 116 KSP Residual norm 3.874097861027e-07 117 KSP Residual norm 3.803766977583e-07 118 KSP Residual norm 3.746444101652e-07 119 KSP Residual norm 3.682825811053e-07 120 KSP Residual norm 3.570248272659e-07 121 KSP Residual norm 3.428589289954e-07 122 KSP Residual norm 3.290421244734e-07 123 KSP Residual norm 3.154838137227e-07 124 KSP Residual norm 3.030182830592e-07 125 KSP Residual norm 2.902808576458e-07 126 KSP Residual norm 2.768790515801e-07 127 KSP Residual norm 2.616358500822e-07 128 KSP Residual norm 2.483944165009e-07 129 KSP Residual norm 2.362192967292e-07 130 KSP Residual norm 2.206757478847e-07 131 KSP Residual norm 2.038643579038e-07 132 KSP Residual norm 1.897759416835e-07 133 KSP Residual norm 1.791754599223e-07 134 KSP Residual norm 1.713066706719e-07 135 KSP Residual norm 1.651270610306e-07 136 KSP Residual norm 1.576630415962e-07 137 KSP Residual norm 1.486300450659e-07 138 KSP Residual norm 1.404121025882e-07 139 KSP Residual norm 1.336280448783e-07 140 KSP Residual norm 1.275254824379e-07 141 KSP Residual norm 1.220632069320e-07 142 KSP Residual norm 1.175587393405e-07 143 KSP Residual norm 1.137054880268e-07 144 KSP Residual norm 1.093600671891e-07 145 KSP Residual norm 1.047305744770e-07 146 KSP Residual norm 1.002750032568e-07 147 KSP Residual norm 9.600143452037e-08 148 KSP Residual norm 9.183028843934e-08 149 KSP Residual norm 8.858986442100e-08 150 KSP Residual norm 8.649833971284e-08 151 KSP Residual norm 8.477457151089e-08 152 KSP Residual norm 8.109826032381e-08 153 KSP Residual norm 7.625414288590e-08 154 KSP Residual norm 7.176270391295e-08 155 KSP Residual norm 6.784432050322e-08 156 KSP Residual norm 6.477991979503e-08 157 KSP Residual norm 6.267008371611e-08 158 KSP Residual norm 6.133111570852e-08 159 KSP Residual norm 6.016533448872e-08 160 KSP Residual norm 5.902334823957e-08 161 KSP Residual norm 5.820885325501e-08 162 KSP Residual norm 5.756584637355e-08 163 KSP Residual norm 5.689480678052e-08 164 KSP Residual norm 5.592294627938e-08 165 KSP Residual norm 5.461883381447e-08 166 KSP Residual norm 5.329800068245e-08 167 KSP Residual norm 5.233678518400e-08 168 KSP Residual norm 5.149258871489e-08 169 KSP Residual norm 5.067183259594e-08 170 KSP Residual norm 4.978986455981e-08 171 KSP Residual norm 4.899792314774e-08 172 KSP Residual norm 4.851332392273e-08 173 KSP Residual norm 4.821593099921e-08 174 KSP Residual norm 4.808058493370e-08 175 KSP Residual norm 4.806605715639e-08 176 KSP Residual norm 4.816635163078e-08 177 KSP Residual norm 4.829495377863e-08 178 KSP Residual norm 4.851102463479e-08 179 KSP Residual norm 4.825644474065e-08 180 KSP Residual norm 4.724669691763e-08 181 KSP Residual norm 4.617975704887e-08 182 KSP Residual norm 4.516648197689e-08 183 KSP Residual norm 4.423735285380e-08 184 KSP Residual norm 4.337709459869e-08 185 KSP Residual norm 4.267161642143e-08 186 KSP Residual norm 4.211127720064e-08 187 KSP Residual norm 4.158622126329e-08 188 KSP Residual norm 4.092680895589e-08 189 KSP Residual norm 4.021911623332e-08 190 KSP Residual norm 3.949026991404e-08 191 KSP Residual norm 3.888275118584e-08 192 KSP Residual norm 3.829800557585e-08 193 KSP Residual norm 3.754051906744e-08 194 KSP Residual norm 3.661423049773e-08 195 KSP Residual norm 3.563518153799e-08 196 KSP Residual norm 3.484683971190e-08 197 KSP Residual norm 3.440260216994e-08 198 KSP Residual norm 3.416622663973e-08 199 KSP Residual norm 3.301081634649e-08 200 KSP Residual norm 3.110537368994e-08 201 KSP Residual norm 2.943704779523e-08 202 KSP Residual norm 2.806886459980e-08 203 KSP Residual norm 2.682090919785e-08 204 KSP Residual norm 2.556216575498e-08 205 KSP Residual norm 2.439170097125e-08 206 KSP Residual norm 2.349554327666e-08 207 KSP Residual norm 2.307984276617e-08 208 KSP Residual norm 2.250225463693e-08 209 KSP Residual norm 2.122124118238e-08 210 KSP Residual norm 2.008300581216e-08 211 KSP Residual norm 1.923668300998e-08 212 KSP Residual norm 1.846323161411e-08 213 KSP Residual norm 1.774807662713e-08 214 KSP Residual norm 1.722086751046e-08 215 KSP Residual norm 1.681675870722e-08 216 KSP Residual norm 1.636595204089e-08 217 KSP Residual norm 1.571778165726e-08 218 KSP Residual norm 1.501310254541e-08 219 KSP Residual norm 1.431676534467e-08 220 KSP Residual norm 1.358328609450e-08 221 KSP Residual norm 1.285983804475e-08 222 KSP Residual norm 1.223116523262e-08 223 KSP Residual norm 1.172042547046e-08 224 KSP Residual norm 1.130563917929e-08 225 KSP Residual norm 1.098003242395e-08 226 KSP Residual norm 1.069473597322e-08 227 KSP Residual norm 1.031335833920e-08 228 KSP Residual norm 9.781456943909e-09 229 KSP Residual norm 9.315099553742e-09 230 KSP Residual norm 8.980174711642e-09 231 KSP Residual norm 8.700470908839e-09 232 KSP Residual norm 8.442378832754e-09 233 KSP Residual norm 8.215749503310e-09 234 KSP Residual norm 8.018209483035e-09 235 KSP Residual norm 7.812311263712e-09 236 KSP Residual norm 7.488882064026e-09 237 KSP Residual norm 7.124051740244e-09 238 KSP Residual norm 6.834096095869e-09 239 KSP Residual norm 6.612332338243e-09 240 KSP Residual norm 6.432109394428e-09 241 KSP Residual norm 6.305950000006e-09 242 KSP Residual norm 6.188832890869e-09 243 KSP Residual norm 6.034460933372e-09 244 KSP Residual norm 5.890603592499e-09 245 KSP Residual norm 5.783851151035e-09 246 KSP Residual norm 5.727388501109e-09 247 KSP Residual norm 5.799642832629e-09 248 KSP Residual norm 5.842210471521e-09 249 KSP Residual norm 5.540649537456e-09 250 KSP Residual norm 5.222386437002e-09 251 KSP Residual norm 5.008960504590e-09 252 KSP Residual norm 4.849463576338e-09 253 KSP Residual norm 4.728814871973e-09 254 KSP Residual norm 4.638958631652e-09 255 KSP Residual norm 4.472716107424e-09 256 KSP Residual norm 4.236565396185e-09 257 KSP Residual norm 4.042579382144e-09 258 KSP Residual norm 3.891613439325e-09 259 KSP Residual norm 3.768078236297e-09 260 KSP Residual norm 3.644830969796e-09 261 KSP Residual norm 3.538956926029e-09 262 KSP Residual norm 3.456663376362e-09 263 KSP Residual norm 3.385726682570e-09 264 KSP Residual norm 3.315791253213e-09 265 KSP Residual norm 3.212712106336e-09 266 KSP Residual norm 3.117279824406e-09 267 KSP Residual norm 3.039644108186e-09 268 KSP Residual norm 2.964090265734e-09 269 KSP Residual norm 2.887081131431e-09 270 KSP Residual norm 2.805420637610e-09 271 KSP Residual norm 2.728688332515e-09 272 KSP Residual norm 2.653519940514e-09 273 KSP Residual norm 2.562018472590e-09 274 KSP Residual norm 2.456065469302e-09 275 KSP Residual norm 2.360468124088e-09 276 KSP Residual norm 2.276279692991e-09 277 KSP Residual norm 2.192908138368e-09 278 KSP Residual norm 2.102158201219e-09 279 KSP Residual norm 2.032585645265e-09 280 KSP Residual norm 1.985289206565e-09 281 KSP Residual norm 1.950534119224e-09 282 KSP Residual norm 1.903983772456e-09 283 KSP Residual norm 1.848287745952e-09 284 KSP Residual norm 1.800071842203e-09 285 KSP Residual norm 1.762026532426e-09 286 KSP Residual norm 1.745525631902e-09 287 KSP Residual norm 1.748164891136e-09 288 KSP Residual norm 1.727765529272e-09 289 KSP Residual norm 1.668052344934e-09 290 KSP Residual norm 1.612924763495e-09 291 KSP Residual norm 1.568428263610e-09 292 KSP Residual norm 1.521234236351e-09 293 KSP Residual norm 1.471595877461e-09 294 KSP Residual norm 1.433285143613e-09 295 KSP Residual norm 1.423636922863e-09 296 KSP Residual norm 1.439034138743e-09 297 KSP Residual norm 1.393033151364e-09 298 KSP Residual norm 1.315713507670e-09 299 KSP Residual norm 1.279217917229e-09 300 KSP Residual norm 1.260383069787e-09 301 KSP Residual norm 1.231084825935e-09 302 KSP Residual norm 1.181598946651e-09 303 KSP Residual norm 1.132381860431e-09 304 KSP Residual norm 1.094007958339e-09 305 KSP Residual norm 1.067500208291e-09 306 KSP Residual norm 1.050263028795e-09 307 KSP Residual norm 1.028340167771e-09 308 KSP Residual norm 9.989588258776e-10 309 KSP Residual norm 9.734467533932e-10 310 KSP Residual norm 9.514201754635e-10 311 KSP Residual norm 9.316925388041e-10 312 KSP Residual norm 9.137992220181e-10 313 KSP Residual norm 9.001541872600e-10 314 KSP Residual norm 8.864292796355e-10 315 KSP Residual norm 8.749058428626e-10 316 KSP Residual norm 8.613232589002e-10 317 KSP Residual norm 8.380683817788e-10 318 KSP Residual norm 8.111033071476e-10 319 KSP Residual norm 7.892946595877e-10 320 KSP Residual norm 7.700411159942e-10 321 KSP Residual norm 7.479859490327e-10 322 KSP Residual norm 7.195150134743e-10 323 KSP Residual norm 6.906801460987e-10 324 KSP Residual norm 6.645517925225e-10 Linear tao_nls_ solve converged due to CONVERGED_RTOL iterations 324 0 LS Function value: 0.0691389, Step length: 0. 1 LS Function value: 0.0691389, Step length: 1. stx: 0., fx: 0.0691389, dgx: -3.69502e-08 sty: 0., fy: 0.0691389, dgy: -3.69502e-08 39 TAO, Function value: 0.0691389, Residual: 7.66474e-05 Residual norms for tao_nls_ solve. 0 KSP Residual norm 7.664742837647e-05 1 KSP Residual norm 5.540964281783e-05 2 KSP Residual norm 4.100402539521e-05 3 KSP Residual norm 3.006777836498e-05 4 KSP Residual norm 2.225322814242e-05 5 KSP Residual norm 1.854353508582e-05 6 KSP Residual norm 1.574078676271e-05 7 KSP Residual norm 1.393275069080e-05 8 KSP Residual norm 1.297113401654e-05 9 KSP Residual norm 1.257482576112e-05 10 KSP Residual norm 1.256890080138e-05 11 KSP Residual norm 1.279005839083e-05 12 KSP Residual norm 1.328276040551e-05 13 KSP Residual norm 1.395982495077e-05 14 KSP Residual norm 1.468780534643e-05 15 KSP Residual norm 1.533423349143e-05 16 KSP Residual norm 1.581409399326e-05 17 KSP Residual norm 1.618752319992e-05 18 KSP Residual norm 1.655381648524e-05 19 KSP Residual norm 1.676993942619e-05 20 KSP Residual norm 1.698468134561e-05 21 KSP Residual norm 1.712836640892e-05 22 KSP Residual norm 1.690727322590e-05 23 KSP Residual norm 1.646015462131e-05 24 KSP Residual norm 1.600270067184e-05 25 KSP Residual norm 1.559652523159e-05 26 KSP Residual norm 1.515256604057e-05 27 KSP Residual norm 1.469304847731e-05 28 KSP Residual norm 1.426660877495e-05 29 KSP Residual norm 1.385723946697e-05 30 KSP Residual norm 1.339087852916e-05 31 KSP Residual norm 1.287251607836e-05 32 KSP Residual norm 1.225875250205e-05 33 KSP Residual norm 1.152813248389e-05 34 KSP Residual norm 1.075093023008e-05 35 KSP Residual norm 9.979399544656e-06 36 KSP Residual norm 9.212899950447e-06 37 KSP Residual norm 8.431607708837e-06 38 KSP Residual norm 7.651143634384e-06 39 KSP Residual norm 6.902121689407e-06 40 KSP Residual norm 6.203052492080e-06 41 KSP Residual norm 5.560391450180e-06 42 KSP Residual norm 4.983916242287e-06 43 KSP Residual norm 4.468007689912e-06 44 KSP Residual norm 4.009094639564e-06 45 KSP Residual norm 3.618902207730e-06 46 KSP Residual norm 3.308049388784e-06 47 KSP Residual norm 3.071402318923e-06 48 KSP Residual norm 2.867215167130e-06 49 KSP Residual norm 2.646487936562e-06 50 KSP Residual norm 2.425803301151e-06 51 KSP Residual norm 2.233917117427e-06 52 KSP Residual norm 2.104410459649e-06 53 KSP Residual norm 2.011855513443e-06 54 KSP Residual norm 1.847579911835e-06 55 KSP Residual norm 1.685769478506e-06 56 KSP Residual norm 1.582234796692e-06 57 KSP Residual norm 1.518399520887e-06 58 KSP Residual norm 1.481704762295e-06 59 KSP Residual norm 1.468890089098e-06 60 KSP Residual norm 1.484979492834e-06 61 KSP Residual norm 1.540720574205e-06 62 KSP Residual norm 1.612922008321e-06 63 KSP Residual norm 1.640445426407e-06 64 KSP Residual norm 1.666220282999e-06 65 KSP Residual norm 1.723679208945e-06 66 KSP Residual norm 1.789760319671e-06 67 KSP Residual norm 1.843263598831e-06 68 KSP Residual norm 1.900084466968e-06 69 KSP Residual norm 1.968877572970e-06 70 KSP Residual norm 1.999739816120e-06 71 KSP Residual norm 1.977034465772e-06 72 KSP Residual norm 1.947851345707e-06 73 KSP Residual norm 1.914930132476e-06 74 KSP Residual norm 1.860844795588e-06 75 KSP Residual norm 1.807851033707e-06 76 KSP Residual norm 1.760902066444e-06 77 KSP Residual norm 1.691195937830e-06 78 KSP Residual norm 1.582223319742e-06 79 KSP Residual norm 1.466885345064e-06 80 KSP Residual norm 1.368954318426e-06 81 KSP Residual norm 1.283965458565e-06 82 KSP Residual norm 1.198447779067e-06 83 KSP Residual norm 1.112744529300e-06 84 KSP Residual norm 1.031303130055e-06 85 KSP Residual norm 9.703628899075e-07 86 KSP Residual norm 9.142856877745e-07 87 KSP Residual norm 8.436478231528e-07 88 KSP Residual norm 7.790600310091e-07 89 KSP Residual norm 7.233822265174e-07 90 KSP Residual norm 6.729907949532e-07 91 KSP Residual norm 6.215236058236e-07 92 KSP Residual norm 5.757273325734e-07 93 KSP Residual norm 5.404156036079e-07 94 KSP Residual norm 5.056581641429e-07 95 KSP Residual norm 4.723694856602e-07 96 KSP Residual norm 4.381141755548e-07 97 KSP Residual norm 4.055694336905e-07 98 KSP Residual norm 3.823865417906e-07 99 KSP Residual norm 3.664837625481e-07 100 KSP Residual norm 3.462096412261e-07 101 KSP Residual norm 3.142614626370e-07 102 KSP Residual norm 2.876007202933e-07 103 KSP Residual norm 2.677686506777e-07 104 KSP Residual norm 2.509871650855e-07 105 KSP Residual norm 2.353293710370e-07 106 KSP Residual norm 2.227233035577e-07 107 KSP Residual norm 2.114171681049e-07 108 KSP Residual norm 2.013242940690e-07 109 KSP Residual norm 1.886658588038e-07 110 KSP Residual norm 1.760851197982e-07 111 KSP Residual norm 1.646389389624e-07 112 KSP Residual norm 1.539991544168e-07 113 KSP Residual norm 1.443539234716e-07 114 KSP Residual norm 1.355713241776e-07 115 KSP Residual norm 1.264486241769e-07 116 KSP Residual norm 1.174824887376e-07 117 KSP Residual norm 1.095061588697e-07 118 KSP Residual norm 1.036888163160e-07 119 KSP Residual norm 1.005938754044e-07 120 KSP Residual norm 9.772457039829e-08 121 KSP Residual norm 9.267864099332e-08 122 KSP Residual norm 8.775226838332e-08 123 KSP Residual norm 8.433318379820e-08 124 KSP Residual norm 8.199199210531e-08 125 KSP Residual norm 7.993169162142e-08 126 KSP Residual norm 7.767141316919e-08 127 KSP Residual norm 7.445504085490e-08 128 KSP Residual norm 7.072823362563e-08 129 KSP Residual norm 6.688185769042e-08 130 KSP Residual norm 6.330033790479e-08 131 KSP Residual norm 6.023063884778e-08 132 KSP Residual norm 5.754040195068e-08 133 KSP Residual norm 5.491125146055e-08 134 KSP Residual norm 5.206075128118e-08 135 KSP Residual norm 4.966994880227e-08 136 KSP Residual norm 4.808022268602e-08 137 KSP Residual norm 4.614231603839e-08 138 KSP Residual norm 4.344630870709e-08 139 KSP Residual norm 4.110131533668e-08 140 KSP Residual norm 3.931704291842e-08 141 KSP Residual norm 3.800076028785e-08 142 KSP Residual norm 3.710146895851e-08 143 KSP Residual norm 3.643824038658e-08 144 KSP Residual norm 3.576795101313e-08 145 KSP Residual norm 3.510848709616e-08 146 KSP Residual norm 3.426121473919e-08 147 KSP Residual norm 3.325979579913e-08 148 KSP Residual norm 3.243532238984e-08 149 KSP Residual norm 3.209760120822e-08 150 KSP Residual norm 3.178076173045e-08 151 KSP Residual norm 3.087720317110e-08 152 KSP Residual norm 2.939241770650e-08 153 KSP Residual norm 2.777220821048e-08 154 KSP Residual norm 2.623842035039e-08 155 KSP Residual norm 2.496971375419e-08 156 KSP Residual norm 2.399634049025e-08 157 KSP Residual norm 2.316173072728e-08 158 KSP Residual norm 2.232899878675e-08 159 KSP Residual norm 2.156280456737e-08 160 KSP Residual norm 2.087400506251e-08 161 KSP Residual norm 2.029202051622e-08 162 KSP Residual norm 1.977854510906e-08 163 KSP Residual norm 1.935389934449e-08 164 KSP Residual norm 1.909704877943e-08 165 KSP Residual norm 1.891889313560e-08 166 KSP Residual norm 1.852948701810e-08 167 KSP Residual norm 1.791053101463e-08 168 KSP Residual norm 1.726362714259e-08 169 KSP Residual norm 1.670135919320e-08 170 KSP Residual norm 1.619539388794e-08 171 KSP Residual norm 1.568416823684e-08 172 KSP Residual norm 1.520400598536e-08 173 KSP Residual norm 1.476892103989e-08 174 KSP Residual norm 1.437234374171e-08 175 KSP Residual norm 1.398717697765e-08 176 KSP Residual norm 1.359731316270e-08 177 KSP Residual norm 1.322896833939e-08 178 KSP Residual norm 1.291249475108e-08 179 KSP Residual norm 1.258212305717e-08 180 KSP Residual norm 1.219516345602e-08 181 KSP Residual norm 1.181616126615e-08 182 KSP Residual norm 1.147418921282e-08 183 KSP Residual norm 1.114085569859e-08 184 KSP Residual norm 1.079494986915e-08 185 KSP Residual norm 1.048886522922e-08 186 KSP Residual norm 1.021164526834e-08 187 KSP Residual norm 9.895959321567e-09 188 KSP Residual norm 9.567928379198e-09 189 KSP Residual norm 9.333636205600e-09 190 KSP Residual norm 9.152473171211e-09 191 KSP Residual norm 8.985835618806e-09 192 KSP Residual norm 8.824195538666e-09 193 KSP Residual norm 8.670473583171e-09 194 KSP Residual norm 8.624591672922e-09 195 KSP Residual norm 8.772852203705e-09 196 KSP Residual norm 8.652090777173e-09 197 KSP Residual norm 8.133155585774e-09 198 KSP Residual norm 7.774055983198e-09 199 KSP Residual norm 7.544863538086e-09 200 KSP Residual norm 7.321050109608e-09 201 KSP Residual norm 7.079254106777e-09 202 KSP Residual norm 6.862561635678e-09 203 KSP Residual norm 6.695421858607e-09 204 KSP Residual norm 6.550179435846e-09 205 KSP Residual norm 6.411015598388e-09 206 KSP Residual norm 6.266683795965e-09 207 KSP Residual norm 6.140182828505e-09 208 KSP Residual norm 6.026918815329e-09 209 KSP Residual norm 5.921862398065e-09 210 KSP Residual norm 5.829475811402e-09 211 KSP Residual norm 5.772753711891e-09 212 KSP Residual norm 5.759366132391e-09 213 KSP Residual norm 5.675590342096e-09 214 KSP Residual norm 5.490125490872e-09 215 KSP Residual norm 5.315880538576e-09 216 KSP Residual norm 5.175151362497e-09 217 KSP Residual norm 5.043247311963e-09 218 KSP Residual norm 4.918941566285e-09 219 KSP Residual norm 4.810749751098e-09 220 KSP Residual norm 4.714896912079e-09 221 KSP Residual norm 4.640666867153e-09 222 KSP Residual norm 4.560161829436e-09 223 KSP Residual norm 4.437676947637e-09 224 KSP Residual norm 4.290707382321e-09 225 KSP Residual norm 4.160512245481e-09 226 KSP Residual norm 4.055186565096e-09 227 KSP Residual norm 3.969948314491e-09 228 KSP Residual norm 3.888256522098e-09 229 KSP Residual norm 3.772576049831e-09 230 KSP Residual norm 3.632234479376e-09 231 KSP Residual norm 3.500347559378e-09 232 KSP Residual norm 3.384274369799e-09 233 KSP Residual norm 3.287655055150e-09 234 KSP Residual norm 3.212664120588e-09 235 KSP Residual norm 3.169439017413e-09 236 KSP Residual norm 3.135706476384e-09 237 KSP Residual norm 3.052442247016e-09 238 KSP Residual norm 2.929538648670e-09 239 KSP Residual norm 2.820023719295e-09 240 KSP Residual norm 2.720958235612e-09 241 KSP Residual norm 2.645964123999e-09 242 KSP Residual norm 2.613074150908e-09 243 KSP Residual norm 2.652247058322e-09 244 KSP Residual norm 2.647571408765e-09 245 KSP Residual norm 2.491066810750e-09 246 KSP Residual norm 2.381626477577e-09 247 KSP Residual norm 2.333555309973e-09 248 KSP Residual norm 2.315690707213e-09 249 KSP Residual norm 2.268023949036e-09 250 KSP Residual norm 2.182077353478e-09 251 KSP Residual norm 2.108708629506e-09 252 KSP Residual norm 2.060215544826e-09 253 KSP Residual norm 2.018526089550e-09 254 KSP Residual norm 1.983262331630e-09 255 KSP Residual norm 1.951387595117e-09 256 KSP Residual norm 1.909308911204e-09 257 KSP Residual norm 1.857123526752e-09 258 KSP Residual norm 1.797572865818e-09 259 KSP Residual norm 1.746754527345e-09 260 KSP Residual norm 1.704171893916e-09 261 KSP Residual norm 1.674631355179e-09 262 KSP Residual norm 1.630793672296e-09 263 KSP Residual norm 1.558572498721e-09 264 KSP Residual norm 1.490438026373e-09 265 KSP Residual norm 1.431210303235e-09 266 KSP Residual norm 1.386505771979e-09 267 KSP Residual norm 1.348155903532e-09 268 KSP Residual norm 1.309551785357e-09 269 KSP Residual norm 1.269927018543e-09 270 KSP Residual norm 1.232500846325e-09 271 KSP Residual norm 1.200903394702e-09 272 KSP Residual norm 1.167927173841e-09 273 KSP Residual norm 1.136553935976e-09 274 KSP Residual norm 1.102357382903e-09 275 KSP Residual norm 1.064927523189e-09 276 KSP Residual norm 1.026751053839e-09 277 KSP Residual norm 9.873400187997e-10 278 KSP Residual norm 9.430177466700e-10 279 KSP Residual norm 9.036835948275e-10 280 KSP Residual norm 8.700061915444e-10 281 KSP Residual norm 8.403728838219e-10 282 KSP Residual norm 8.061263336824e-10 283 KSP Residual norm 7.676044477637e-10 284 KSP Residual norm 7.319984710454e-10 Linear tao_nls_ solve converged due to CONVERGED_RTOL iterations 284 0 LS Function value: 0.0691389, Step length: 0. 1 LS Function value: 0.0691389, Step length: 1. stx: 0., fx: 0.0691389, dgx: -1.85747e-08 sty: 0., fy: 0.0691389, dgy: -1.85747e-08 40 TAO, Function value: 0.0691389, Residual: 1.74531e-05 Residual norms for tao_nls_ solve. 0 KSP Residual norm 1.745305240262e-05 1 KSP Residual norm 1.407567229897e-05 2 KSP Residual norm 1.068983293646e-05 3 KSP Residual norm 8.389464807009e-06 4 KSP Residual norm 7.083170062341e-06 5 KSP Residual norm 6.822282734774e-06 6 KSP Residual norm 6.923179802178e-06 7 KSP Residual norm 7.372093320828e-06 8 KSP Residual norm 7.971773335637e-06 9 KSP Residual norm 8.779404910937e-06 10 KSP Residual norm 9.716311412018e-06 11 KSP Residual norm 1.057528788943e-05 12 KSP Residual norm 1.142087987054e-05 13 KSP Residual norm 1.217905136270e-05 14 KSP Residual norm 1.290032438267e-05 15 KSP Residual norm 1.362142208415e-05 16 KSP Residual norm 1.427453218204e-05 17 KSP Residual norm 1.474461901126e-05 18 KSP Residual norm 1.500523037380e-05 19 KSP Residual norm 1.513301514979e-05 20 KSP Residual norm 1.513459250831e-05 21 KSP Residual norm 1.503369224325e-05 22 KSP Residual norm 1.478600125666e-05 23 KSP Residual norm 1.439197794882e-05 24 KSP Residual norm 1.388975454721e-05 25 KSP Residual norm 1.338784100774e-05 26 KSP Residual norm 1.293547182921e-05 27 KSP Residual norm 1.244071222702e-05 28 KSP Residual norm 1.179846182434e-05 29 KSP Residual norm 1.100512778358e-05 30 KSP Residual norm 1.013823825619e-05 31 KSP Residual norm 9.292793856575e-06 32 KSP Residual norm 8.463109111623e-06 33 KSP Residual norm 7.645877519732e-06 34 KSP Residual norm 6.890993528288e-06 35 KSP Residual norm 6.224897552149e-06 36 KSP Residual norm 5.628268036255e-06 37 KSP Residual norm 5.085431830567e-06 38 KSP Residual norm 4.624505918779e-06 39 KSP Residual norm 4.282808698463e-06 40 KSP Residual norm 4.016600656538e-06 41 KSP Residual norm 3.675344007858e-06 42 KSP Residual norm 3.331921349867e-06 43 KSP Residual norm 3.079781080738e-06 44 KSP Residual norm 2.901199110841e-06 45 KSP Residual norm 2.764592070939e-06 46 KSP Residual norm 2.651878987110e-06 47 KSP Residual norm 2.553994852948e-06 48 KSP Residual norm 2.467293755017e-06 49 KSP Residual norm 2.392250679557e-06 50 KSP Residual norm 2.326293919373e-06 51 KSP Residual norm 2.265610587219e-06 52 KSP Residual norm 2.216094165634e-06 53 KSP Residual norm 2.208232192598e-06 54 KSP Residual norm 2.279045128367e-06 55 KSP Residual norm 2.306114053387e-06 56 KSP Residual norm 2.222171830666e-06 57 KSP Residual norm 2.214437787649e-06 58 KSP Residual norm 2.278198382843e-06 59 KSP Residual norm 2.386011187633e-06 60 KSP Residual norm 2.521311177146e-06 61 KSP Residual norm 2.637767913718e-06 62 KSP Residual norm 2.692324070372e-06 63 KSP Residual norm 2.724684780872e-06 64 KSP Residual norm 2.766262227173e-06 65 KSP Residual norm 2.816430163434e-06 66 KSP Residual norm 2.853311857826e-06 67 KSP Residual norm 2.829125444629e-06 68 KSP Residual norm 2.752789448150e-06 69 KSP Residual norm 2.658799456502e-06 70 KSP Residual norm 2.541008279504e-06 71 KSP Residual norm 2.407847632672e-06 72 KSP Residual norm 2.287905120213e-06 73 KSP Residual norm 2.175507614623e-06 74 KSP Residual norm 2.059293086629e-06 75 KSP Residual norm 1.964242794145e-06 76 KSP Residual norm 1.885041057850e-06 77 KSP Residual norm 1.785968037746e-06 78 KSP Residual norm 1.666716717604e-06 79 KSP Residual norm 1.556472259595e-06 80 KSP Residual norm 1.462848347963e-06 81 KSP Residual norm 1.384556871729e-06 82 KSP Residual norm 1.304451436153e-06 83 KSP Residual norm 1.221546514130e-06 84 KSP Residual norm 1.144741422282e-06 85 KSP Residual norm 1.085260431400e-06 86 KSP Residual norm 1.026501503312e-06 87 KSP Residual norm 9.508245173403e-07 88 KSP Residual norm 8.801636632567e-07 89 KSP Residual norm 8.247167029907e-07 90 KSP Residual norm 7.747102059990e-07 91 KSP Residual norm 7.197430118305e-07 92 KSP Residual norm 6.723142821000e-07 93 KSP Residual norm 6.356018657477e-07 94 KSP Residual norm 6.052924202341e-07 95 KSP Residual norm 5.824208192384e-07 96 KSP Residual norm 5.592860436578e-07 97 KSP Residual norm 5.360643776474e-07 98 KSP Residual norm 5.148378548891e-07 99 KSP Residual norm 4.945104725316e-07 100 KSP Residual norm 4.770023431310e-07 101 KSP Residual norm 4.531288290036e-07 102 KSP Residual norm 4.299983281194e-07 103 KSP Residual norm 4.101913169868e-07 104 KSP Residual norm 3.972676568088e-07 105 KSP Residual norm 3.893397658064e-07 106 KSP Residual norm 3.643557149259e-07 107 KSP Residual norm 3.344134512934e-07 108 KSP Residual norm 3.103628376273e-07 109 KSP Residual norm 2.866486422159e-07 110 KSP Residual norm 2.652276921273e-07 111 KSP Residual norm 2.451140846784e-07 112 KSP Residual norm 2.271850665918e-07 113 KSP Residual norm 2.121948842363e-07 114 KSP Residual norm 1.998846929641e-07 115 KSP Residual norm 1.891436167580e-07 116 KSP Residual norm 1.793631691659e-07 117 KSP Residual norm 1.706541100437e-07 118 KSP Residual norm 1.651156859417e-07 119 KSP Residual norm 1.630082060730e-07 120 KSP Residual norm 1.608764303774e-07 121 KSP Residual norm 1.546882799654e-07 122 KSP Residual norm 1.470018165687e-07 123 KSP Residual norm 1.407121232204e-07 124 KSP Residual norm 1.355543631204e-07 125 KSP Residual norm 1.306641752361e-07 126 KSP Residual norm 1.252291103668e-07 127 KSP Residual norm 1.180346899208e-07 128 KSP Residual norm 1.101312523213e-07 129 KSP Residual norm 1.031221119601e-07 130 KSP Residual norm 9.796194204799e-08 131 KSP Residual norm 9.361775587046e-08 132 KSP Residual norm 8.682912170912e-08 133 KSP Residual norm 7.962277224322e-08 134 KSP Residual norm 7.423226416993e-08 135 KSP Residual norm 7.024166913207e-08 136 KSP Residual norm 6.714803043371e-08 137 KSP Residual norm 6.444456845147e-08 138 KSP Residual norm 6.180272674425e-08 139 KSP Residual norm 5.920258693764e-08 140 KSP Residual norm 5.695970970615e-08 141 KSP Residual norm 5.534309246596e-08 142 KSP Residual norm 5.418658191336e-08 143 KSP Residual norm 5.317290252995e-08 144 KSP Residual norm 5.204101032048e-08 145 KSP Residual norm 5.111276911716e-08 146 KSP Residual norm 5.027374488891e-08 147 KSP Residual norm 4.945157279532e-08 148 KSP Residual norm 4.849772439882e-08 149 KSP Residual norm 4.727559517530e-08 150 KSP Residual norm 4.569027414422e-08 151 KSP Residual norm 4.414549176550e-08 152 KSP Residual norm 4.263605212776e-08 153 KSP Residual norm 4.068531941302e-08 154 KSP Residual norm 3.873746063425e-08 155 KSP Residual norm 3.764236171710e-08 156 KSP Residual norm 3.757557014909e-08 157 KSP Residual norm 3.706676579542e-08 158 KSP Residual norm 3.538974010737e-08 159 KSP Residual norm 3.438343389988e-08 160 KSP Residual norm 3.408152039831e-08 161 KSP Residual norm 3.339309328571e-08 162 KSP Residual norm 3.235448636124e-08 163 KSP Residual norm 3.158493378498e-08 164 KSP Residual norm 3.108000240650e-08 165 KSP Residual norm 3.061524926275e-08 166 KSP Residual norm 3.004716995933e-08 167 KSP Residual norm 2.933476659719e-08 168 KSP Residual norm 2.849120617342e-08 169 KSP Residual norm 2.762412079669e-08 170 KSP Residual norm 2.678558571853e-08 171 KSP Residual norm 2.589958143544e-08 172 KSP Residual norm 2.497004056709e-08 173 KSP Residual norm 2.404746767155e-08 174 KSP Residual norm 2.320630218765e-08 175 KSP Residual norm 2.249284883623e-08 176 KSP Residual norm 2.184043575595e-08 177 KSP Residual norm 2.119190407955e-08 178 KSP Residual norm 2.055517078468e-08 179 KSP Residual norm 1.992006145709e-08 180 KSP Residual norm 1.923462416777e-08 181 KSP Residual norm 1.852645403623e-08 182 KSP Residual norm 1.788013141684e-08 183 KSP Residual norm 1.725653909203e-08 184 KSP Residual norm 1.662600253560e-08 185 KSP Residual norm 1.605941777735e-08 186 KSP Residual norm 1.560222225050e-08 187 KSP Residual norm 1.523942568643e-08 188 KSP Residual norm 1.479966629716e-08 189 KSP Residual norm 1.427559659611e-08 190 KSP Residual norm 1.379226135110e-08 191 KSP Residual norm 1.335657538119e-08 192 KSP Residual norm 1.295187400451e-08 193 KSP Residual norm 1.253744847110e-08 194 KSP Residual norm 1.214435032867e-08 195 KSP Residual norm 1.177018141158e-08 196 KSP Residual norm 1.143459508620e-08 197 KSP Residual norm 1.116562059571e-08 198 KSP Residual norm 1.099846699623e-08 199 KSP Residual norm 1.091204359814e-08 200 KSP Residual norm 1.067502789520e-08 201 KSP Residual norm 1.022851611882e-08 202 KSP Residual norm 9.812636304276e-09 203 KSP Residual norm 9.509356603734e-09 204 KSP Residual norm 9.259334297431e-09 205 KSP Residual norm 9.092126305345e-09 206 KSP Residual norm 9.069541747029e-09 207 KSP Residual norm 9.212925647429e-09 208 KSP Residual norm 9.008917291812e-09 209 KSP Residual norm 8.512139967697e-09 210 KSP Residual norm 8.181328361649e-09 211 KSP Residual norm 7.949642143434e-09 212 KSP Residual norm 7.745118852506e-09 213 KSP Residual norm 7.562272544586e-09 214 KSP Residual norm 7.396894256577e-09 215 KSP Residual norm 7.243439831181e-09 216 KSP Residual norm 7.092526173503e-09 217 KSP Residual norm 6.932362666085e-09 218 KSP Residual norm 6.723989449384e-09 219 KSP Residual norm 6.487208125166e-09 220 KSP Residual norm 6.252580855487e-09 221 KSP Residual norm 6.062277793552e-09 222 KSP Residual norm 5.904812647222e-09 223 KSP Residual norm 5.762720828346e-09 224 KSP Residual norm 5.615969823888e-09 225 KSP Residual norm 5.476217125993e-09 226 KSP Residual norm 5.325792212368e-09 227 KSP Residual norm 5.162661473744e-09 228 KSP Residual norm 5.005359395157e-09 229 KSP Residual norm 4.864597817327e-09 230 KSP Residual norm 4.716379220038e-09 231 KSP Residual norm 4.564842549359e-09 232 KSP Residual norm 4.414884442206e-09 233 KSP Residual norm 4.271342906674e-09 234 KSP Residual norm 4.125028372629e-09 235 KSP Residual norm 3.998623943879e-09 236 KSP Residual norm 3.904037868696e-09 237 KSP Residual norm 3.778562160882e-09 238 KSP Residual norm 3.596267464887e-09 239 KSP Residual norm 3.432848920564e-09 240 KSP Residual norm 3.296390693941e-09 241 KSP Residual norm 3.187711266877e-09 242 KSP Residual norm 3.097701989111e-09 243 KSP Residual norm 3.029556210330e-09 244 KSP Residual norm 2.993441962887e-09 245 KSP Residual norm 2.979879159956e-09 246 KSP Residual norm 2.946012464681e-09 247 KSP Residual norm 2.861029561118e-09 248 KSP Residual norm 2.766982281895e-09 249 KSP Residual norm 2.686121188939e-09 250 KSP Residual norm 2.619084118293e-09 251 KSP Residual norm 2.562539581169e-09 252 KSP Residual norm 2.511119495458e-09 253 KSP Residual norm 2.471822039507e-09 254 KSP Residual norm 2.442996764035e-09 255 KSP Residual norm 2.393280928022e-09 256 KSP Residual norm 2.326872525797e-09 257 KSP Residual norm 2.273042747707e-09 258 KSP Residual norm 2.184678545879e-09 259 KSP Residual norm 2.082901888379e-09 260 KSP Residual norm 2.009446366927e-09 261 KSP Residual norm 1.951113196366e-09 262 KSP Residual norm 1.886176120047e-09 263 KSP Residual norm 1.824563973928e-09 264 KSP Residual norm 1.769822568229e-09 265 KSP Residual norm 1.715830450317e-09 266 KSP Residual norm 1.671807726711e-09 267 KSP Residual norm 1.633322466893e-09 268 KSP Residual norm 1.601328534357e-09 269 KSP Residual norm 1.570587531019e-09 270 KSP Residual norm 1.543819927354e-09 271 KSP Residual norm 1.513272913020e-09 272 KSP Residual norm 1.475351960148e-09 273 KSP Residual norm 1.434438114529e-09 274 KSP Residual norm 1.395323378473e-09 275 KSP Residual norm 1.352004534646e-09 276 KSP Residual norm 1.299139387865e-09 277 KSP Residual norm 1.243119257394e-09 278 KSP Residual norm 1.193470911306e-09 279 KSP Residual norm 1.159120849637e-09 280 KSP Residual norm 1.124202871916e-09 281 KSP Residual norm 1.071359566398e-09 282 KSP Residual norm 1.021789740606e-09 283 KSP Residual norm 9.885345177026e-10 284 KSP Residual norm 9.626930827252e-10 285 KSP Residual norm 9.260619383610e-10 286 KSP Residual norm 8.840156833035e-10 287 KSP Residual norm 8.483531514083e-10 288 KSP Residual norm 8.175730947290e-10 289 KSP Residual norm 7.865670370059e-10 290 KSP Residual norm 7.613913348062e-10 291 KSP Residual norm 7.379176037873e-10 292 KSP Residual norm 7.118292194805e-10 293 KSP Residual norm 6.818621052428e-10 294 KSP Residual norm 6.534669243713e-10 295 KSP Residual norm 6.306185776897e-10 296 KSP Residual norm 6.084483154673e-10 297 KSP Residual norm 5.870520122425e-10 298 KSP Residual norm 5.678470295369e-10 299 KSP Residual norm 5.539645222731e-10 300 KSP Residual norm 5.434771704068e-10 301 KSP Residual norm 5.278161954278e-10 302 KSP Residual norm 5.044729569925e-10 303 KSP Residual norm 4.816170616598e-10 304 KSP Residual norm 4.644944205772e-10 305 KSP Residual norm 4.498665003092e-10 306 KSP Residual norm 4.380983274797e-10 307 KSP Residual norm 4.345677066269e-10 308 KSP Residual norm 4.303572236437e-10 309 KSP Residual norm 4.063944493103e-10 310 KSP Residual norm 3.822632656023e-10 311 KSP Residual norm 3.675943124794e-10 312 KSP Residual norm 3.580323001068e-10 313 KSP Residual norm 3.527457095350e-10 314 KSP Residual norm 3.463933439564e-10 315 KSP Residual norm 3.351618292447e-10 316 KSP Residual norm 3.256988977902e-10 317 KSP Residual norm 3.187664249554e-10 318 KSP Residual norm 3.115074856224e-10 319 KSP Residual norm 3.031471996680e-10 320 KSP Residual norm 2.954043092552e-10 321 KSP Residual norm 2.884532656632e-10 322 KSP Residual norm 2.813306651267e-10 323 KSP Residual norm 2.743551347016e-10 324 KSP Residual norm 2.664802522962e-10 325 KSP Residual norm 2.584787056522e-10 326 KSP Residual norm 2.507561770121e-10 327 KSP Residual norm 2.431765930374e-10 328 KSP Residual norm 2.364397365038e-10 329 KSP Residual norm 2.308914789943e-10 330 KSP Residual norm 2.252407101438e-10 331 KSP Residual norm 2.196557789848e-10 332 KSP Residual norm 2.133097768070e-10 333 KSP Residual norm 2.064590219585e-10 334 KSP Residual norm 2.009944761397e-10 335 KSP Residual norm 1.962064999765e-10 336 KSP Residual norm 1.908767568835e-10 337 KSP Residual norm 1.848495369168e-10 338 KSP Residual norm 1.795289441155e-10 339 KSP Residual norm 1.745835846470e-10 340 KSP Residual norm 1.706578015852e-10 Linear tao_nls_ solve converged due to CONVERGED_RTOL iterations 340 0 LS Function value: 0.0691389, Step length: 0. 1 LS Function value: 0.0691388, Step length: 1. stx: 0., fx: 0.0691389, dgx: -6.05376e-09 sty: 0., fy: 0.0691389, dgy: -6.05376e-09 41 TAO, Function value: 0.0691388, Residual: 2.22711e-05 Residual norms for tao_nls_ solve. 0 KSP Residual norm 2.227112186456e-05 1 KSP Residual norm 1.758285725655e-05 2 KSP Residual norm 1.206416220112e-05 3 KSP Residual norm 9.158817976392e-06 4 KSP Residual norm 6.775236331092e-06 5 KSP Residual norm 5.385351604914e-06 6 KSP Residual norm 4.491881687527e-06 7 KSP Residual norm 3.901386673346e-06 8 KSP Residual norm 3.441404442652e-06 9 KSP Residual norm 3.156136115249e-06 10 KSP Residual norm 3.002620664766e-06 11 KSP Residual norm 2.864358860847e-06 12 KSP Residual norm 2.765321930276e-06 13 KSP Residual norm 2.697360576616e-06 14 KSP Residual norm 2.668106607983e-06 15 KSP Residual norm 2.673256198371e-06 16 KSP Residual norm 2.724828401905e-06 17 KSP Residual norm 2.820141246499e-06 18 KSP Residual norm 2.936153861277e-06 19 KSP Residual norm 3.065756634611e-06 20 KSP Residual norm 3.201673782661e-06 21 KSP Residual norm 3.343963084798e-06 22 KSP Residual norm 3.469443922102e-06 23 KSP Residual norm 3.576208535562e-06 24 KSP Residual norm 3.680960569832e-06 25 KSP Residual norm 3.794964927086e-06 26 KSP Residual norm 3.907398163554e-06 27 KSP Residual norm 3.998782167180e-06 28 KSP Residual norm 4.072728948740e-06 29 KSP Residual norm 4.118169005655e-06 30 KSP Residual norm 4.143891075252e-06 31 KSP Residual norm 4.172919434706e-06 32 KSP Residual norm 4.188931477255e-06 33 KSP Residual norm 4.168614253022e-06 34 KSP Residual norm 4.115343569030e-06 35 KSP Residual norm 4.030355233212e-06 36 KSP Residual norm 3.891297790759e-06 37 KSP Residual norm 3.698476287399e-06 38 KSP Residual norm 3.480060033634e-06 39 KSP Residual norm 3.256045778207e-06 40 KSP Residual norm 3.051932174571e-06 41 KSP Residual norm 2.895205768171e-06 42 KSP Residual norm 2.716449528320e-06 43 KSP Residual norm 2.455258418863e-06 44 KSP Residual norm 2.225060165351e-06 45 KSP Residual norm 2.046004604515e-06 46 KSP Residual norm 1.893097634198e-06 47 KSP Residual norm 1.755039846044e-06 48 KSP Residual norm 1.631727801695e-06 49 KSP Residual norm 1.528590505802e-06 50 KSP Residual norm 1.456741901942e-06 51 KSP Residual norm 1.428226254866e-06 52 KSP Residual norm 1.377536954862e-06 53 KSP Residual norm 1.266343181189e-06 54 KSP Residual norm 1.174364795256e-06 55 KSP Residual norm 1.106625810571e-06 56 KSP Residual norm 1.052219818064e-06 57 KSP Residual norm 1.009907142779e-06 58 KSP Residual norm 9.794246617419e-07 59 KSP Residual norm 9.598621638180e-07 60 KSP Residual norm 9.483513890699e-07 61 KSP Residual norm 9.462703525756e-07 62 KSP Residual norm 9.551218102116e-07 63 KSP Residual norm 9.627803834554e-07 64 KSP Residual norm 9.536416413329e-07 65 KSP Residual norm 9.360722154760e-07 66 KSP Residual norm 9.173231576286e-07 67 KSP Residual norm 8.944364645710e-07 68 KSP Residual norm 8.731090027701e-07 69 KSP Residual norm 8.585085187799e-07 70 KSP Residual norm 8.453841719710e-07 71 KSP Residual norm 8.260580974979e-07 72 KSP Residual norm 7.971534869201e-07 73 KSP Residual norm 7.581949188063e-07 74 KSP Residual norm 7.140024435922e-07 75 KSP Residual norm 6.731207625809e-07 76 KSP Residual norm 6.336888289063e-07 77 KSP Residual norm 5.952590794603e-07 78 KSP Residual norm 5.645593333436e-07 79 KSP Residual norm 5.347433880319e-07 80 KSP Residual norm 5.060225392952e-07 81 KSP Residual norm 4.832020564289e-07 82 KSP Residual norm 4.602365060866e-07 83 KSP Residual norm 4.328835298711e-07 84 KSP Residual norm 4.052441716010e-07 85 KSP Residual norm 3.857472006944e-07 86 KSP Residual norm 3.720463152550e-07 87 KSP Residual norm 3.574289958291e-07 88 KSP Residual norm 3.410642516556e-07 89 KSP Residual norm 3.263734857899e-07 90 KSP Residual norm 3.158516338311e-07 91 KSP Residual norm 3.061575583085e-07 92 KSP Residual norm 2.988783653188e-07 93 KSP Residual norm 2.936078960902e-07 94 KSP Residual norm 2.913901137543e-07 95 KSP Residual norm 2.939114374982e-07 96 KSP Residual norm 2.985724685512e-07 97 KSP Residual norm 2.987296876518e-07 98 KSP Residual norm 2.929923803065e-07 99 KSP Residual norm 2.875424629412e-07 100 KSP Residual norm 2.869324751220e-07 101 KSP Residual norm 2.859236778319e-07 102 KSP Residual norm 2.821100542975e-07 103 KSP Residual norm 2.768560752284e-07 104 KSP Residual norm 2.705891842016e-07 105 KSP Residual norm 2.651995373355e-07 106 KSP Residual norm 2.593649698582e-07 107 KSP Residual norm 2.532854523247e-07 108 KSP Residual norm 2.465075331779e-07 109 KSP Residual norm 2.357188881127e-07 110 KSP Residual norm 2.239304674120e-07 111 KSP Residual norm 2.116456515500e-07 112 KSP Residual norm 1.997819606952e-07 113 KSP Residual norm 1.884601545443e-07 114 KSP Residual norm 1.768165782634e-07 115 KSP Residual norm 1.641549674704e-07 116 KSP Residual norm 1.518308288484e-07 117 KSP Residual norm 1.407860457492e-07 118 KSP Residual norm 1.323946510351e-07 119 KSP Residual norm 1.263340452014e-07 120 KSP Residual norm 1.203548952917e-07 121 KSP Residual norm 1.128944478936e-07 122 KSP Residual norm 1.059616198783e-07 123 KSP Residual norm 9.988628201079e-08 124 KSP Residual norm 9.330131630668e-08 125 KSP Residual norm 8.637261379132e-08 126 KSP Residual norm 8.017138893190e-08 127 KSP Residual norm 7.456508271334e-08 128 KSP Residual norm 6.967271313953e-08 129 KSP Residual norm 6.534666444331e-08 130 KSP Residual norm 6.130524753878e-08 131 KSP Residual norm 5.724645621671e-08 132 KSP Residual norm 5.283716141276e-08 133 KSP Residual norm 4.862557367604e-08 134 KSP Residual norm 4.509046864444e-08 135 KSP Residual norm 4.219171181052e-08 136 KSP Residual norm 3.957295969656e-08 137 KSP Residual norm 3.704783342357e-08 138 KSP Residual norm 3.462107402644e-08 139 KSP Residual norm 3.217363621724e-08 140 KSP Residual norm 2.988679707982e-08 141 KSP Residual norm 2.803857766192e-08 142 KSP Residual norm 2.654088651210e-08 143 KSP Residual norm 2.517907087893e-08 144 KSP Residual norm 2.394167972165e-08 145 KSP Residual norm 2.304546452217e-08 146 KSP Residual norm 2.237719706295e-08 147 KSP Residual norm 2.174004368337e-08 148 KSP Residual norm 2.098648802175e-08 149 KSP Residual norm 2.016745839698e-08 150 KSP Residual norm 1.939937780576e-08 151 KSP Residual norm 1.865459064501e-08 152 KSP Residual norm 1.782316456990e-08 153 KSP Residual norm 1.699177193544e-08 154 KSP Residual norm 1.621772419844e-08 155 KSP Residual norm 1.557314357257e-08 156 KSP Residual norm 1.502805571544e-08 157 KSP Residual norm 1.453688210040e-08 158 KSP Residual norm 1.408304736061e-08 159 KSP Residual norm 1.365695603548e-08 160 KSP Residual norm 1.324514913528e-08 161 KSP Residual norm 1.283389751093e-08 162 KSP Residual norm 1.243332025133e-08 163 KSP Residual norm 1.209321964806e-08 164 KSP Residual norm 1.183088729459e-08 165 KSP Residual norm 1.162581102259e-08 166 KSP Residual norm 1.142807420178e-08 167 KSP Residual norm 1.114066839292e-08 168 KSP Residual norm 1.075230862796e-08 169 KSP Residual norm 1.040422359770e-08 170 KSP Residual norm 1.014942531086e-08 171 KSP Residual norm 9.954022209454e-09 172 KSP Residual norm 9.832995034909e-09 173 KSP Residual norm 9.761504167180e-09 174 KSP Residual norm 9.671345151208e-09 175 KSP Residual norm 9.540448369473e-09 176 KSP Residual norm 9.398955708564e-09 177 KSP Residual norm 9.319129743529e-09 178 KSP Residual norm 9.289530173190e-09 179 KSP Residual norm 9.243409882063e-09 180 KSP Residual norm 9.151797698558e-09 181 KSP Residual norm 9.044434229181e-09 182 KSP Residual norm 8.952852806960e-09 183 KSP Residual norm 8.851952077376e-09 184 KSP Residual norm 8.710015458635e-09 185 KSP Residual norm 8.521434799882e-09 186 KSP Residual norm 8.305599363801e-09 187 KSP Residual norm 8.124979353143e-09 188 KSP Residual norm 8.019864579837e-09 189 KSP Residual norm 8.056557270691e-09 190 KSP Residual norm 8.097009591275e-09 191 KSP Residual norm 7.858009591646e-09 192 KSP Residual norm 7.592106842000e-09 193 KSP Residual norm 7.405235935704e-09 194 KSP Residual norm 7.307736389905e-09 195 KSP Residual norm 7.283074659402e-09 196 KSP Residual norm 7.209530002302e-09 197 KSP Residual norm 6.971034133241e-09 198 KSP Residual norm 6.715402258452e-09 199 KSP Residual norm 6.534883218229e-09 200 KSP Residual norm 6.407025856999e-09 201 KSP Residual norm 6.336625675947e-09 202 KSP Residual norm 6.261463732648e-09 203 KSP Residual norm 6.060117743311e-09 204 KSP Residual norm 5.765853401760e-09 205 KSP Residual norm 5.485814744119e-09 206 KSP Residual norm 5.239748429286e-09 207 KSP Residual norm 5.017658759868e-09 208 KSP Residual norm 4.796757356203e-09 209 KSP Residual norm 4.587011979211e-09 210 KSP Residual norm 4.388293986108e-09 211 KSP Residual norm 4.194275837300e-09 212 KSP Residual norm 4.006737045820e-09 213 KSP Residual norm 3.834626812321e-09 214 KSP Residual norm 3.681263516305e-09 215 KSP Residual norm 3.547937252931e-09 216 KSP Residual norm 3.433171093683e-09 217 KSP Residual norm 3.338443565329e-09 218 KSP Residual norm 3.241687508869e-09 219 KSP Residual norm 3.128946689059e-09 220 KSP Residual norm 3.004247904944e-09 221 KSP Residual norm 2.901970244821e-09 222 KSP Residual norm 2.825527487783e-09 223 KSP Residual norm 2.751972163110e-09 224 KSP Residual norm 2.651247349629e-09 225 KSP Residual norm 2.543774178581e-09 226 KSP Residual norm 2.444120301395e-09 227 KSP Residual norm 2.347868635544e-09 228 KSP Residual norm 2.254901453641e-09 229 KSP Residual norm 2.172517605661e-09 230 KSP Residual norm 2.093764129305e-09 231 KSP Residual norm 2.020899129224e-09 232 KSP Residual norm 1.964135445661e-09 233 KSP Residual norm 1.939493373728e-09 234 KSP Residual norm 1.907533099767e-09 235 KSP Residual norm 1.830700441668e-09 236 KSP Residual norm 1.765577231800e-09 237 KSP Residual norm 1.757064677992e-09 238 KSP Residual norm 1.741901834539e-09 239 KSP Residual norm 1.650455310602e-09 240 KSP Residual norm 1.563503768047e-09 241 KSP Residual norm 1.504762112591e-09 242 KSP Residual norm 1.456702215498e-09 243 KSP Residual norm 1.416030475428e-09 244 KSP Residual norm 1.392932771294e-09 245 KSP Residual norm 1.384562032682e-09 246 KSP Residual norm 1.347760088223e-09 247 KSP Residual norm 1.279857006362e-09 248 KSP Residual norm 1.223883507927e-09 249 KSP Residual norm 1.183169785901e-09 250 KSP Residual norm 1.152559791915e-09 251 KSP Residual norm 1.131591714517e-09 252 KSP Residual norm 1.112194821964e-09 253 KSP Residual norm 1.081982411879e-09 254 KSP Residual norm 1.047662566657e-09 255 KSP Residual norm 1.020240576623e-09 256 KSP Residual norm 9.922479363765e-10 257 KSP Residual norm 9.602961074472e-10 258 KSP Residual norm 9.305308638136e-10 259 KSP Residual norm 9.102172403116e-10 260 KSP Residual norm 8.936459327534e-10 261 KSP Residual norm 8.723421388674e-10 262 KSP Residual norm 8.422074543886e-10 263 KSP Residual norm 8.116763248415e-10 264 KSP Residual norm 7.828270939363e-10 265 KSP Residual norm 7.539742094911e-10 266 KSP Residual norm 7.281771338702e-10 267 KSP Residual norm 7.037199500323e-10 268 KSP Residual norm 6.823830025041e-10 269 KSP Residual norm 6.627405663277e-10 270 KSP Residual norm 6.466151726455e-10 271 KSP Residual norm 6.310209520136e-10 272 KSP Residual norm 6.148871571668e-10 273 KSP Residual norm 5.999376638652e-10 274 KSP Residual norm 5.861318259061e-10 275 KSP Residual norm 5.741834737114e-10 276 KSP Residual norm 5.686112901196e-10 277 KSP Residual norm 5.654995492580e-10 278 KSP Residual norm 5.485709078557e-10 279 KSP Residual norm 5.245761917163e-10 280 KSP Residual norm 5.048322264938e-10 281 KSP Residual norm 4.890067342030e-10 282 KSP Residual norm 4.754532053101e-10 283 KSP Residual norm 4.640634645588e-10 284 KSP Residual norm 4.575900554931e-10 285 KSP Residual norm 4.595866205505e-10 286 KSP Residual norm 4.495965892199e-10 287 KSP Residual norm 4.232545675397e-10 288 KSP Residual norm 4.036609509847e-10 289 KSP Residual norm 3.884588893249e-10 290 KSP Residual norm 3.755771171024e-10 291 KSP Residual norm 3.640046883608e-10 292 KSP Residual norm 3.522419817713e-10 293 KSP Residual norm 3.394474320330e-10 294 KSP Residual norm 3.277961212513e-10 295 KSP Residual norm 3.200973030211e-10 296 KSP Residual norm 3.138018220043e-10 297 KSP Residual norm 3.041114054105e-10 298 KSP Residual norm 2.909329194691e-10 299 KSP Residual norm 2.804080160059e-10 300 KSP Residual norm 2.711850046286e-10 301 KSP Residual norm 2.633592935991e-10 302 KSP Residual norm 2.557809179371e-10 303 KSP Residual norm 2.485603866968e-10 304 KSP Residual norm 2.421804884608e-10 305 KSP Residual norm 2.354996014027e-10 306 KSP Residual norm 2.287449294454e-10 307 KSP Residual norm 2.221477743293e-10 Linear tao_nls_ solve converged due to CONVERGED_RTOL iterations 307 0 LS Function value: 0.0691388, Step length: 0. 1 LS Function value: 0.0691388, Step length: 1. stx: 0., fx: 0.0691388, dgx: -1.60069e-09 sty: 0., fy: 0.0691388, dgy: -1.60069e-09 42 TAO, Function value: 0.0691388, Residual: 2.12559e-06 Residual norms for tao_nls_ solve. 0 KSP Residual norm 2.125587305991e-06 1 KSP Residual norm 1.842111664435e-06 2 KSP Residual norm 1.297688459386e-06 3 KSP Residual norm 1.077501023608e-06 4 KSP Residual norm 9.129975292545e-07 5 KSP Residual norm 8.557585626815e-07 6 KSP Residual norm 8.745965528233e-07 7 KSP Residual norm 9.098670227820e-07 8 KSP Residual norm 9.604530404050e-07 9 KSP Residual norm 1.052176172959e-06 10 KSP Residual norm 1.155994618727e-06 11 KSP Residual norm 1.245378109797e-06 12 KSP Residual norm 1.357774719157e-06 13 KSP Residual norm 1.478302923318e-06 14 KSP Residual norm 1.596772116071e-06 15 KSP Residual norm 1.729424132883e-06 16 KSP Residual norm 1.870649495246e-06 17 KSP Residual norm 2.011296639429e-06 18 KSP Residual norm 2.111264727884e-06 19 KSP Residual norm 2.200473098952e-06 20 KSP Residual norm 2.290647930688e-06 21 KSP Residual norm 2.360293143481e-06 22 KSP Residual norm 2.397501314590e-06 23 KSP Residual norm 2.405247215932e-06 24 KSP Residual norm 2.410341154694e-06 25 KSP Residual norm 2.380241944090e-06 26 KSP Residual norm 2.318530843227e-06 27 KSP Residual norm 2.242418842763e-06 28 KSP Residual norm 2.152468900671e-06 29 KSP Residual norm 2.043630313282e-06 30 KSP Residual norm 1.916402817980e-06 31 KSP Residual norm 1.784626074908e-06 32 KSP Residual norm 1.647406455193e-06 33 KSP Residual norm 1.508849247270e-06 34 KSP Residual norm 1.374792800288e-06 35 KSP Residual norm 1.250758364929e-06 36 KSP Residual norm 1.134548855577e-06 37 KSP Residual norm 1.023677125758e-06 38 KSP Residual norm 9.208774659484e-07 39 KSP Residual norm 8.298940385483e-07 40 KSP Residual norm 7.527684397370e-07 41 KSP Residual norm 6.887408016624e-07 42 KSP Residual norm 6.347003613865e-07 43 KSP Residual norm 5.891348404372e-07 44 KSP Residual norm 5.541072794032e-07 45 KSP Residual norm 5.323917222434e-07 46 KSP Residual norm 5.198164249344e-07 47 KSP Residual norm 5.007454339332e-07 48 KSP Residual norm 4.761615103744e-07 49 KSP Residual norm 4.587073260808e-07 50 KSP Residual norm 4.475244883284e-07 51 KSP Residual norm 4.403223887021e-07 52 KSP Residual norm 4.377649708311e-07 53 KSP Residual norm 4.371351194506e-07 54 KSP Residual norm 4.337504223436e-07 55 KSP Residual norm 4.323952298884e-07 56 KSP Residual norm 4.384355657753e-07 57 KSP Residual norm 4.473594869194e-07 58 KSP Residual norm 4.520344930793e-07 59 KSP Residual norm 4.545719480154e-07 60 KSP Residual norm 4.601672697984e-07 61 KSP Residual norm 4.704593041749e-07 62 KSP Residual norm 4.840479942604e-07 63 KSP Residual norm 4.955780317755e-07 64 KSP Residual norm 4.995043799936e-07 65 KSP Residual norm 4.942387104003e-07 66 KSP Residual norm 4.836009268230e-07 67 KSP Residual norm 4.681966566003e-07 68 KSP Residual norm 4.502657260737e-07 69 KSP Residual norm 4.320073326978e-07 70 KSP Residual norm 4.113636342500e-07 71 KSP Residual norm 3.897768434647e-07 72 KSP Residual norm 3.724704391061e-07 73 KSP Residual norm 3.574795594835e-07 74 KSP Residual norm 3.380486309426e-07 75 KSP Residual norm 3.169463312477e-07 76 KSP Residual norm 2.980205760026e-07 77 KSP Residual norm 2.809430420484e-07 78 KSP Residual norm 2.669937295205e-07 79 KSP Residual norm 2.534000483777e-07 80 KSP Residual norm 2.416276188062e-07 81 KSP Residual norm 2.330694854274e-07 82 KSP Residual norm 2.247906908886e-07 83 KSP Residual norm 2.150122562449e-07 84 KSP Residual norm 2.027227644729e-07 85 KSP Residual norm 1.897266454865e-07 86 KSP Residual norm 1.773872023671e-07 87 KSP Residual norm 1.649339718417e-07 88 KSP Residual norm 1.542258090852e-07 89 KSP Residual norm 1.452160177262e-07 90 KSP Residual norm 1.364223645248e-07 91 KSP Residual norm 1.277032969845e-07 92 KSP Residual norm 1.203632801110e-07 93 KSP Residual norm 1.140753253416e-07 94 KSP Residual norm 1.082818582844e-07 95 KSP Residual norm 1.035247517527e-07 96 KSP Residual norm 9.961086220192e-08 97 KSP Residual norm 9.685945818358e-08 98 KSP Residual norm 9.496918417083e-08 99 KSP Residual norm 9.344714482031e-08 100 KSP Residual norm 9.202382550188e-08 101 KSP Residual norm 9.021493407018e-08 102 KSP Residual norm 8.892541946558e-08 103 KSP Residual norm 8.743609441247e-08 104 KSP Residual norm 8.673510434491e-08 105 KSP Residual norm 8.514556544762e-08 106 KSP Residual norm 8.168124791866e-08 107 KSP Residual norm 7.673082384211e-08 108 KSP Residual norm 7.154644083946e-08 109 KSP Residual norm 6.701041671476e-08 110 KSP Residual norm 6.308430489266e-08 111 KSP Residual norm 5.914220113278e-08 112 KSP Residual norm 5.497581666691e-08 113 KSP Residual norm 5.090126478915e-08 114 KSP Residual norm 4.746756998492e-08 115 KSP Residual norm 4.456810747146e-08 116 KSP Residual norm 4.165738717289e-08 117 KSP Residual norm 3.872274961513e-08 118 KSP Residual norm 3.638319681651e-08 119 KSP Residual norm 3.455538362131e-08 120 KSP Residual norm 3.295275182282e-08 121 KSP Residual norm 3.132761115844e-08 122 KSP Residual norm 2.974680242531e-08 123 KSP Residual norm 2.823799216800e-08 124 KSP Residual norm 2.690616928403e-08 125 KSP Residual norm 2.565205961960e-08 126 KSP Residual norm 2.437710865538e-08 127 KSP Residual norm 2.305117300420e-08 128 KSP Residual norm 2.166603592572e-08 129 KSP Residual norm 2.030625967689e-08 130 KSP Residual norm 1.901234933316e-08 131 KSP Residual norm 1.786641636673e-08 132 KSP Residual norm 1.668575532497e-08 133 KSP Residual norm 1.540164147541e-08 134 KSP Residual norm 1.421020782945e-08 135 KSP Residual norm 1.316691396032e-08 136 KSP Residual norm 1.222149503344e-08 137 KSP Residual norm 1.140998630942e-08 138 KSP Residual norm 1.076673264046e-08 139 KSP Residual norm 1.033065364265e-08 140 KSP Residual norm 1.003308708260e-08 141 KSP Residual norm 9.672840139025e-09 142 KSP Residual norm 9.315371038461e-09 143 KSP Residual norm 9.085424378714e-09 144 KSP Residual norm 8.799778443663e-09 145 KSP Residual norm 8.480826938230e-09 146 KSP Residual norm 8.242771715419e-09 147 KSP Residual norm 8.101639936352e-09 148 KSP Residual norm 8.085245895629e-09 149 KSP Residual norm 8.150788068337e-09 150 KSP Residual norm 8.008697945689e-09 151 KSP Residual norm 7.754429202503e-09 152 KSP Residual norm 7.597518212556e-09 153 KSP Residual norm 7.530957842044e-09 154 KSP Residual norm 7.505785100761e-09 155 KSP Residual norm 7.418387657863e-09 156 KSP Residual norm 7.261769074920e-09 157 KSP Residual norm 7.197304365609e-09 158 KSP Residual norm 7.325376685595e-09 159 KSP Residual norm 7.566114784186e-09 160 KSP Residual norm 7.413089755747e-09 161 KSP Residual norm 7.158149776501e-09 162 KSP Residual norm 7.078182129015e-09 163 KSP Residual norm 7.062740978358e-09 164 KSP Residual norm 7.021590829924e-09 165 KSP Residual norm 6.957313851361e-09 166 KSP Residual norm 6.890685025062e-09 167 KSP Residual norm 6.806306843048e-09 168 KSP Residual norm 6.691806127634e-09 169 KSP Residual norm 6.561840778678e-09 170 KSP Residual norm 6.427119314448e-09 171 KSP Residual norm 6.284091223749e-09 172 KSP Residual norm 6.144562330195e-09 173 KSP Residual norm 5.992724846217e-09 174 KSP Residual norm 5.807892057405e-09 175 KSP Residual norm 5.607091422529e-09 176 KSP Residual norm 5.422439545331e-09 177 KSP Residual norm 5.284810157636e-09 178 KSP Residual norm 5.181682253831e-09 179 KSP Residual norm 5.079062113803e-09 180 KSP Residual norm 4.907732809675e-09 181 KSP Residual norm 4.689165836644e-09 182 KSP Residual norm 4.477432934657e-09 183 KSP Residual norm 4.274579304073e-09 184 KSP Residual norm 4.080754560317e-09 185 KSP Residual norm 3.909231279140e-09 186 KSP Residual norm 3.763564973943e-09 187 KSP Residual norm 3.641962564658e-09 188 KSP Residual norm 3.516920751152e-09 189 KSP Residual norm 3.389269728129e-09 190 KSP Residual norm 3.277260135540e-09 191 KSP Residual norm 3.165775394956e-09 192 KSP Residual norm 3.053176229190e-09 193 KSP Residual norm 2.938518224745e-09 194 KSP Residual norm 2.832842208693e-09 195 KSP Residual norm 2.739856640695e-09 196 KSP Residual norm 2.653290329877e-09 197 KSP Residual norm 2.570737110048e-09 198 KSP Residual norm 2.493264869458e-09 199 KSP Residual norm 2.424384579319e-09 200 KSP Residual norm 2.358265778793e-09 201 KSP Residual norm 2.294403466020e-09 202 KSP Residual norm 2.240316088106e-09 203 KSP Residual norm 2.201795318095e-09 204 KSP Residual norm 2.203948520194e-09 205 KSP Residual norm 2.249519089973e-09 206 KSP Residual norm 2.161749808175e-09 207 KSP Residual norm 2.031432417332e-09 208 KSP Residual norm 1.957316101660e-09 209 KSP Residual norm 1.911772322815e-09 210 KSP Residual norm 1.869332190629e-09 211 KSP Residual norm 1.821536274290e-09 212 KSP Residual norm 1.772929460354e-09 213 KSP Residual norm 1.726733323635e-09 214 KSP Residual norm 1.677458089120e-09 215 KSP Residual norm 1.623752684266e-09 216 KSP Residual norm 1.567353275456e-09 217 KSP Residual norm 1.515810016644e-09 218 KSP Residual norm 1.469844738185e-09 219 KSP Residual norm 1.428798138430e-09 220 KSP Residual norm 1.384599195625e-09 221 KSP Residual norm 1.345135997535e-09 222 KSP Residual norm 1.317350812667e-09 223 KSP Residual norm 1.280179631069e-09 224 KSP Residual norm 1.219817992763e-09 225 KSP Residual norm 1.168741349260e-09 226 KSP Residual norm 1.129319505228e-09 227 KSP Residual norm 1.091894141922e-09 228 KSP Residual norm 1.054404007978e-09 229 KSP Residual norm 1.019691822696e-09 230 KSP Residual norm 9.830919581199e-10 231 KSP Residual norm 9.475584941591e-10 232 KSP Residual norm 9.135918075261e-10 233 KSP Residual norm 8.851703554255e-10 234 KSP Residual norm 8.559978989743e-10 235 KSP Residual norm 8.244451504296e-10 236 KSP Residual norm 7.964205229658e-10 237 KSP Residual norm 7.757937917120e-10 238 KSP Residual norm 7.604996727901e-10 239 KSP Residual norm 7.418948986654e-10 240 KSP Residual norm 7.139731311976e-10 241 KSP Residual norm 6.838042297533e-10 242 KSP Residual norm 6.579583153158e-10 243 KSP Residual norm 6.376413895988e-10 244 KSP Residual norm 6.232148733423e-10 245 KSP Residual norm 6.148191486676e-10 246 KSP Residual norm 6.062911779447e-10 247 KSP Residual norm 5.889366289316e-10 248 KSP Residual norm 5.671570779055e-10 249 KSP Residual norm 5.492384315320e-10 250 KSP Residual norm 5.366210969971e-10 251 KSP Residual norm 5.297906387601e-10 252 KSP Residual norm 5.265147828732e-10 253 KSP Residual norm 5.189619927955e-10 254 KSP Residual norm 5.040048270784e-10 255 KSP Residual norm 4.914648180642e-10 256 KSP Residual norm 4.866970545132e-10 257 KSP Residual norm 4.887664623481e-10 258 KSP Residual norm 4.792827075614e-10 259 KSP Residual norm 4.557298378587e-10 260 KSP Residual norm 4.349800706061e-10 261 KSP Residual norm 4.190247647471e-10 262 KSP Residual norm 4.055901060724e-10 263 KSP Residual norm 3.928969788696e-10 264 KSP Residual norm 3.785956588835e-10 265 KSP Residual norm 3.643146911449e-10 266 KSP Residual norm 3.513474693823e-10 267 KSP Residual norm 3.412157387791e-10 268 KSP Residual norm 3.326667035225e-10 269 KSP Residual norm 3.232125579129e-10 270 KSP Residual norm 3.133102795927e-10 271 KSP Residual norm 3.026632262759e-10 272 KSP Residual norm 2.909828915848e-10 273 KSP Residual norm 2.796229289184e-10 274 KSP Residual norm 2.691012993276e-10 275 KSP Residual norm 2.593834711805e-10 276 KSP Residual norm 2.517516589658e-10 277 KSP Residual norm 2.444875470619e-10 278 KSP Residual norm 2.363485887332e-10 279 KSP Residual norm 2.280970317344e-10 280 KSP Residual norm 2.203444111335e-10 281 KSP Residual norm 2.132475594425e-10 282 KSP Residual norm 2.068717873433e-10 283 KSP Residual norm 2.011425044935e-10 284 KSP Residual norm 1.959536251398e-10 285 KSP Residual norm 1.902712153566e-10 286 KSP Residual norm 1.834351329789e-10 287 KSP Residual norm 1.759854644892e-10 288 KSP Residual norm 1.683808022058e-10 289 KSP Residual norm 1.609261691001e-10 290 KSP Residual norm 1.551687339918e-10 291 KSP Residual norm 1.500126123031e-10 292 KSP Residual norm 1.458020450038e-10 293 KSP Residual norm 1.430334599943e-10 294 KSP Residual norm 1.406822426822e-10 295 KSP Residual norm 1.365080545823e-10 296 KSP Residual norm 1.316655716550e-10 297 KSP Residual norm 1.276763158409e-10 298 KSP Residual norm 1.241488596867e-10 299 KSP Residual norm 1.210267782396e-10 300 KSP Residual norm 1.182724105866e-10 301 KSP Residual norm 1.161034850476e-10 302 KSP Residual norm 1.140866621614e-10 303 KSP Residual norm 1.122511781736e-10 304 KSP Residual norm 1.102567560492e-10 305 KSP Residual norm 1.079962206357e-10 306 KSP Residual norm 1.072641690569e-10 307 KSP Residual norm 1.061070894777e-10 308 KSP Residual norm 1.003227388291e-10 309 KSP Residual norm 9.484310216962e-11 310 KSP Residual norm 9.106526260323e-11 311 KSP Residual norm 8.828962336522e-11 312 KSP Residual norm 8.623673235114e-11 313 KSP Residual norm 8.493242053450e-11 314 KSP Residual norm 8.335770449570e-11 315 KSP Residual norm 8.088450294360e-11 316 KSP Residual norm 7.831600277588e-11 317 KSP Residual norm 7.573062541830e-11 318 KSP Residual norm 7.318374440856e-11 319 KSP Residual norm 7.125665708661e-11 320 KSP Residual norm 6.985674106500e-11 321 KSP Residual norm 6.866150612056e-11 322 KSP Residual norm 6.751415322244e-11 323 KSP Residual norm 6.615757162111e-11 324 KSP Residual norm 6.463248708028e-11 325 KSP Residual norm 6.286711804082e-11 326 KSP Residual norm 6.096111963032e-11 327 KSP Residual norm 5.890372557945e-11 328 KSP Residual norm 5.718678066119e-11 329 KSP Residual norm 5.582420401208e-11 330 KSP Residual norm 5.468074876868e-11 331 KSP Residual norm 5.345415382175e-11 332 KSP Residual norm 5.214487087053e-11 333 KSP Residual norm 5.058930718248e-11 334 KSP Residual norm 4.921643376406e-11 335 KSP Residual norm 4.810793595634e-11 336 KSP Residual norm 4.697180054668e-11 337 KSP Residual norm 4.566266022384e-11 338 KSP Residual norm 4.454469025936e-11 339 KSP Residual norm 4.342256170290e-11 340 KSP Residual norm 4.213570214152e-11 341 KSP Residual norm 4.096798779532e-11 342 KSP Residual norm 3.984521359720e-11 343 KSP Residual norm 3.866358144610e-11 344 KSP Residual norm 3.738477641292e-11 345 KSP Residual norm 3.594287109627e-11 346 KSP Residual norm 3.471194312667e-11 347 KSP Residual norm 3.363443745520e-11 348 KSP Residual norm 3.272447171236e-11 349 KSP Residual norm 3.192681887246e-11 350 KSP Residual norm 3.131890479593e-11 351 KSP Residual norm 3.089643377485e-11 352 KSP Residual norm 3.069599301476e-11 353 KSP Residual norm 3.052179997762e-11 354 KSP Residual norm 2.961012724752e-11 355 KSP Residual norm 2.855330076094e-11 356 KSP Residual norm 2.754778539422e-11 357 KSP Residual norm 2.644171068235e-11 358 KSP Residual norm 2.540813750713e-11 359 KSP Residual norm 2.471737921463e-11 360 KSP Residual norm 2.447355622305e-11 361 KSP Residual norm 2.418398556535e-11 362 KSP Residual norm 2.334092777508e-11 363 KSP Residual norm 2.225970624139e-11 364 KSP Residual norm 2.141099151863e-11 365 KSP Residual norm 2.065738632012e-11 Linear tao_nls_ solve converged due to CONVERGED_RTOL iterations 365 0 LS Function value: 0.0691388, Step length: 0. 1 LS Function value: 0.0691388, Step length: 1. stx: 0., fx: 0.0691388, dgx: -1.46624e-10 sty: 0., fy: 0.0691388, dgy: -1.46624e-10 43 TAO, Function value: 0.0691388, Residual: 8.31462e-07 Residual norms for tao_nls_ solve. 0 KSP Residual norm 8.314618436927e-07 1 KSP Residual norm 6.864412890377e-07 2 KSP Residual norm 4.529453262313e-07 3 KSP Residual norm 3.598243165972e-07 4 KSP Residual norm 2.715637599571e-07 5 KSP Residual norm 2.160694323527e-07 6 KSP Residual norm 1.850562496988e-07 7 KSP Residual norm 1.608460601183e-07 8 KSP Residual norm 1.404216416894e-07 9 KSP Residual norm 1.279934418718e-07 10 KSP Residual norm 1.197673404693e-07 11 KSP Residual norm 1.106770827802e-07 12 KSP Residual norm 1.040435478899e-07 13 KSP Residual norm 9.943878394315e-08 14 KSP Residual norm 9.542265989249e-08 15 KSP Residual norm 9.239718581728e-08 16 KSP Residual norm 9.055108767144e-08 17 KSP Residual norm 9.006283167976e-08 18 KSP Residual norm 8.976300438292e-08 19 KSP Residual norm 9.043763342857e-08 20 KSP Residual norm 9.242369072720e-08 21 KSP Residual norm 9.513828068986e-08 22 KSP Residual norm 9.862985257694e-08 23 KSP Residual norm 1.022139860486e-07 24 KSP Residual norm 1.067610067994e-07 25 KSP Residual norm 1.122866563334e-07 26 KSP Residual norm 1.184979456499e-07 27 KSP Residual norm 1.255843858334e-07 28 KSP Residual norm 1.342922098564e-07 29 KSP Residual norm 1.451276587399e-07 30 KSP Residual norm 1.533220665690e-07 31 KSP Residual norm 1.586516993027e-07 32 KSP Residual norm 1.647278611956e-07 33 KSP Residual norm 1.715067725467e-07 34 KSP Residual norm 1.778769539885e-07 35 KSP Residual norm 1.831720298791e-07 36 KSP Residual norm 1.861254772787e-07 37 KSP Residual norm 1.857384589017e-07 38 KSP Residual norm 1.822797884411e-07 39 KSP Residual norm 1.766443421437e-07 40 KSP Residual norm 1.696910795236e-07 41 KSP Residual norm 1.619660962824e-07 42 KSP Residual norm 1.532674978996e-07 43 KSP Residual norm 1.439040714749e-07 44 KSP Residual norm 1.355436811157e-07 45 KSP Residual norm 1.299153492459e-07 46 KSP Residual norm 1.252919186986e-07 47 KSP Residual norm 1.164460489248e-07 48 KSP Residual norm 1.068070255892e-07 49 KSP Residual norm 9.931802054881e-08 50 KSP Residual norm 9.296822779447e-08 51 KSP Residual norm 8.730865506666e-08 52 KSP Residual norm 8.294901170109e-08 53 KSP Residual norm 8.069688382649e-08 54 KSP Residual norm 7.811274681984e-08 55 KSP Residual norm 7.193003350944e-08 56 KSP Residual norm 6.612041233088e-08 57 KSP Residual norm 6.190408273901e-08 58 KSP Residual norm 5.881565392739e-08 59 KSP Residual norm 5.658394346136e-08 60 KSP Residual norm 5.497996637779e-08 61 KSP Residual norm 5.380321272862e-08 62 KSP Residual norm 5.294497100604e-08 63 KSP Residual norm 5.181337015768e-08 64 KSP Residual norm 5.018598002733e-08 65 KSP Residual norm 4.859640846535e-08 66 KSP Residual norm 4.729066220478e-08 67 KSP Residual norm 4.619660514026e-08 68 KSP Residual norm 4.547733675710e-08 69 KSP Residual norm 4.496428838958e-08 70 KSP Residual norm 4.366786930914e-08 71 KSP Residual norm 4.161198256544e-08 72 KSP Residual norm 3.979827129423e-08 73 KSP Residual norm 3.823124005822e-08 74 KSP Residual norm 3.666231571214e-08 75 KSP Residual norm 3.509780684297e-08 76 KSP Residual norm 3.331877359622e-08 77 KSP Residual norm 3.150739200868e-08 78 KSP Residual norm 2.983341112510e-08 79 KSP Residual norm 2.810138115279e-08 80 KSP Residual norm 2.668129348603e-08 81 KSP Residual norm 2.557013062441e-08 82 KSP Residual norm 2.409670134194e-08 83 KSP Residual norm 2.238020959298e-08 84 KSP Residual norm 2.076706503625e-08 85 KSP Residual norm 1.928262771561e-08 86 KSP Residual norm 1.793353946620e-08 87 KSP Residual norm 1.654064265195e-08 88 KSP Residual norm 1.527350393351e-08 89 KSP Residual norm 1.419475313040e-08 90 KSP Residual norm 1.323238207208e-08 91 KSP Residual norm 1.227264212020e-08 92 KSP Residual norm 1.135116581233e-08 93 KSP Residual norm 1.047003865130e-08 94 KSP Residual norm 9.664251235207e-09 95 KSP Residual norm 9.004748453242e-09 96 KSP Residual norm 8.463870058529e-09 97 KSP Residual norm 8.046871889327e-09 98 KSP Residual norm 7.663643093877e-09 99 KSP Residual norm 7.204741504539e-09 100 KSP Residual norm 6.799043623059e-09 101 KSP Residual norm 6.461365495858e-09 102 KSP Residual norm 6.258538500919e-09 103 KSP Residual norm 6.184222737384e-09 104 KSP Residual norm 6.207945214311e-09 105 KSP Residual norm 5.937208571458e-09 106 KSP Residual norm 5.629382494595e-09 107 KSP Residual norm 5.468097194723e-09 108 KSP Residual norm 5.363802075736e-09 109 KSP Residual norm 5.284907294762e-09 110 KSP Residual norm 5.246387246630e-09 111 KSP Residual norm 5.206146089638e-09 112 KSP Residual norm 5.165936322159e-09 113 KSP Residual norm 5.124972664082e-09 114 KSP Residual norm 5.087227173476e-09 115 KSP Residual norm 5.030038290020e-09 116 KSP Residual norm 4.938277181757e-09 117 KSP Residual norm 4.826054035040e-09 118 KSP Residual norm 4.745095337552e-09 119 KSP Residual norm 4.687787645792e-09 120 KSP Residual norm 4.636642002831e-09 121 KSP Residual norm 4.560978146292e-09 122 KSP Residual norm 4.471135343900e-09 123 KSP Residual norm 4.376703382670e-09 124 KSP Residual norm 4.293856995579e-09 125 KSP Residual norm 4.216269411820e-09 126 KSP Residual norm 4.136519138383e-09 127 KSP Residual norm 4.079991920989e-09 128 KSP Residual norm 4.049099145713e-09 129 KSP Residual norm 3.960361883269e-09 130 KSP Residual norm 3.775287662028e-09 131 KSP Residual norm 3.597849478955e-09 132 KSP Residual norm 3.435809334619e-09 133 KSP Residual norm 3.274423617108e-09 134 KSP Residual norm 3.115056491378e-09 135 KSP Residual norm 2.957125695873e-09 136 KSP Residual norm 2.795406497332e-09 137 KSP Residual norm 2.646654030058e-09 138 KSP Residual norm 2.528257264914e-09 139 KSP Residual norm 2.419625824199e-09 140 KSP Residual norm 2.257883152503e-09 141 KSP Residual norm 2.084417519227e-09 142 KSP Residual norm 1.941698130138e-09 143 KSP Residual norm 1.825299382286e-09 144 KSP Residual norm 1.724007064945e-09 145 KSP Residual norm 1.630693857630e-09 146 KSP Residual norm 1.532382321832e-09 147 KSP Residual norm 1.436726125736e-09 148 KSP Residual norm 1.358530745156e-09 149 KSP Residual norm 1.308552447132e-09 150 KSP Residual norm 1.264347298618e-09 151 KSP Residual norm 1.190696763772e-09 152 KSP Residual norm 1.109763002756e-09 153 KSP Residual norm 1.045305434600e-09 154 KSP Residual norm 1.003597261101e-09 155 KSP Residual norm 9.972858114212e-10 156 KSP Residual norm 9.871097339157e-10 157 KSP Residual norm 9.134164870245e-10 158 KSP Residual norm 8.541887575655e-10 159 KSP Residual norm 8.172422442037e-10 160 KSP Residual norm 7.879250394455e-10 161 KSP Residual norm 7.569486375147e-10 162 KSP Residual norm 7.257168725861e-10 163 KSP Residual norm 7.001958086670e-10 164 KSP Residual norm 6.795286207251e-10 165 KSP Residual norm 6.610592891362e-10 166 KSP Residual norm 6.384205532868e-10 167 KSP Residual norm 6.083051498339e-10 168 KSP Residual norm 5.774833363890e-10 169 KSP Residual norm 5.515927614815e-10 170 KSP Residual norm 5.305354287614e-10 171 KSP Residual norm 5.122480536377e-10 172 KSP Residual norm 4.969708054914e-10 173 KSP Residual norm 4.835143799388e-10 174 KSP Residual norm 4.698356360220e-10 175 KSP Residual norm 4.561172649206e-10 176 KSP Residual norm 4.435998760575e-10 177 KSP Residual norm 4.332595677192e-10 178 KSP Residual norm 4.237385420024e-10 179 KSP Residual norm 4.151567397209e-10 180 KSP Residual norm 4.070923301760e-10 181 KSP Residual norm 3.997656616008e-10 182 KSP Residual norm 3.925268825279e-10 183 KSP Residual norm 3.843927476700e-10 184 KSP Residual norm 3.752364591899e-10 185 KSP Residual norm 3.655397078531e-10 186 KSP Residual norm 3.562150975095e-10 187 KSP Residual norm 3.476264757424e-10 188 KSP Residual norm 3.390173764025e-10 189 KSP Residual norm 3.301332966245e-10 190 KSP Residual norm 3.222883391542e-10 191 KSP Residual norm 3.154336074632e-10 192 KSP Residual norm 3.105707199731e-10 193 KSP Residual norm 3.100649697513e-10 194 KSP Residual norm 3.113809852407e-10 195 KSP Residual norm 3.068875529889e-10 196 KSP Residual norm 2.988243271258e-10 197 KSP Residual norm 2.934700163031e-10 198 KSP Residual norm 2.901077138294e-10 199 KSP Residual norm 2.877331174443e-10 200 KSP Residual norm 2.879076486224e-10 201 KSP Residual norm 2.952206626099e-10 202 KSP Residual norm 3.011046729465e-10 203 KSP Residual norm 2.879042090239e-10 204 KSP Residual norm 2.767072031479e-10 205 KSP Residual norm 2.722352183143e-10 206 KSP Residual norm 2.701443973573e-10 207 KSP Residual norm 2.682666210594e-10 208 KSP Residual norm 2.655536862858e-10 209 KSP Residual norm 2.615749240364e-10 210 KSP Residual norm 2.542148041308e-10 211 KSP Residual norm 2.460903653249e-10 212 KSP Residual norm 2.401903572646e-10 213 KSP Residual norm 2.367695877160e-10 214 KSP Residual norm 2.329092694799e-10 215 KSP Residual norm 2.258048929823e-10 216 KSP Residual norm 2.172756784844e-10 217 KSP Residual norm 2.098276379823e-10 218 KSP Residual norm 2.036036433968e-10 219 KSP Residual norm 1.985286768002e-10 220 KSP Residual norm 1.942208420029e-10 221 KSP Residual norm 1.895912383595e-10 222 KSP Residual norm 1.830624523451e-10 223 KSP Residual norm 1.758883323166e-10 224 KSP Residual norm 1.694098343395e-10 225 KSP Residual norm 1.631403608913e-10 226 KSP Residual norm 1.565437160993e-10 227 KSP Residual norm 1.501643026227e-10 228 KSP Residual norm 1.442149468657e-10 229 KSP Residual norm 1.386554689218e-10 230 KSP Residual norm 1.332227209875e-10 231 KSP Residual norm 1.281287222543e-10 232 KSP Residual norm 1.230410218486e-10 233 KSP Residual norm 1.183387170528e-10 234 KSP Residual norm 1.134126813880e-10 235 KSP Residual norm 1.082066363028e-10 236 KSP Residual norm 1.030718179079e-10 237 KSP Residual norm 9.862905833962e-11 238 KSP Residual norm 9.472022365292e-11 239 KSP Residual norm 9.100300801669e-11 240 KSP Residual norm 8.741917436944e-11 241 KSP Residual norm 8.397741340016e-11 242 KSP Residual norm 8.066346387157e-11 243 KSP Residual norm 7.758082374933e-11 244 KSP Residual norm 7.480967093304e-11 245 KSP Residual norm 7.254548361159e-11 246 KSP Residual norm 7.091442653920e-11 247 KSP Residual norm 6.907401734025e-11 248 KSP Residual norm 6.585244834758e-11 249 KSP Residual norm 6.258933799150e-11 250 KSP Residual norm 6.081826826289e-11 251 KSP Residual norm 6.066816348515e-11 252 KSP Residual norm 5.861081250884e-11 253 KSP Residual norm 5.463747721766e-11 254 KSP Residual norm 5.194828349101e-11 255 KSP Residual norm 5.045609492854e-11 256 KSP Residual norm 4.928263359758e-11 257 KSP Residual norm 4.740277863796e-11 258 KSP Residual norm 4.500936041627e-11 259 KSP Residual norm 4.292836905264e-11 260 KSP Residual norm 4.141470583909e-11 261 KSP Residual norm 4.026013240501e-11 262 KSP Residual norm 3.935784889184e-11 263 KSP Residual norm 3.838478647806e-11 264 KSP Residual norm 3.673417475689e-11 265 KSP Residual norm 3.485643899688e-11 266 KSP Residual norm 3.331517712794e-11 267 KSP Residual norm 3.216374850494e-11 268 KSP Residual norm 3.119376003012e-11 269 KSP Residual norm 3.017214667816e-11 270 KSP Residual norm 2.903664916035e-11 271 KSP Residual norm 2.780112142402e-11 272 KSP Residual norm 2.671007385360e-11 273 KSP Residual norm 2.579426932725e-11 274 KSP Residual norm 2.499123994272e-11 275 KSP Residual norm 2.417813674097e-11 276 KSP Residual norm 2.339386701821e-11 277 KSP Residual norm 2.266842341447e-11 278 KSP Residual norm 2.195635182278e-11 279 KSP Residual norm 2.134858429123e-11 280 KSP Residual norm 2.081519901917e-11 281 KSP Residual norm 2.031213518348e-11 282 KSP Residual norm 1.983821973670e-11 283 KSP Residual norm 1.958690761899e-11 284 KSP Residual norm 1.953193528809e-11 285 KSP Residual norm 1.907141191467e-11 286 KSP Residual norm 1.835461166454e-11 287 KSP Residual norm 1.790294126744e-11 288 KSP Residual norm 1.764335555866e-11 289 KSP Residual norm 1.736241555712e-11 290 KSP Residual norm 1.711090045573e-11 291 KSP Residual norm 1.674715879719e-11 292 KSP Residual norm 1.636576391299e-11 293 KSP Residual norm 1.598267988283e-11 294 KSP Residual norm 1.557158881270e-11 295 KSP Residual norm 1.514871524125e-11 296 KSP Residual norm 1.470575765366e-11 297 KSP Residual norm 1.431475129077e-11 298 KSP Residual norm 1.396369174567e-11 299 KSP Residual norm 1.365830701302e-11 300 KSP Residual norm 1.363716771840e-11 301 KSP Residual norm 1.401729868322e-11 302 KSP Residual norm 1.367828540634e-11 303 KSP Residual norm 1.286371167325e-11 304 KSP Residual norm 1.245609928631e-11 305 KSP Residual norm 1.224491267818e-11 306 KSP Residual norm 1.210067967374e-11 307 KSP Residual norm 1.194263081399e-11 308 KSP Residual norm 1.168490747703e-11 309 KSP Residual norm 1.137991380064e-11 310 KSP Residual norm 1.107556185417e-11 311 KSP Residual norm 1.087577301355e-11 312 KSP Residual norm 1.072064209421e-11 313 KSP Residual norm 1.044809099089e-11 314 KSP Residual norm 1.004301895662e-11 315 KSP Residual norm 9.717907274149e-12 316 KSP Residual norm 9.504928783962e-12 317 KSP Residual norm 9.332772961888e-12 318 KSP Residual norm 9.001598862699e-12 319 KSP Residual norm 8.549624333577e-12 320 KSP Residual norm 8.145697219656e-12 Linear tao_nls_ solve converged due to CONVERGED_RTOL iterations 320 0 LS Function value: 0.0691388, Step length: 0. 1 LS Function value: 0.0691388, Step length: 1. stx: 0., fx: 0.0691388, dgx: -2.67456e-12 sty: 0., fy: 0.0691388, dgy: -2.67456e-12 44 TAO, Function value: 0.0691388, Residual: 6.20283e-09 TAO solve converged due to CONVERGED_GATOL iterations 44 Tao solve required 44 iterations From jed at jedbrown.org Tue Jun 9 23:06:16 2020 From: jed at jedbrown.org (Jed Brown) Date: Tue, 09 Jun 2020 22:06:16 -0600 Subject: [petsc-users] Make stream In-Reply-To: References: <874krkdjjg.fsf@jedbrown.org> Message-ID: <87pna7d0p3.fsf@jedbrown.org> Fande Kong writes: >> There's a lot more to AMG setup than memory bandwidth (architecture >> matters a lot, even between different generation CPUs). > > > Could you elaborate a bit more on this? From my understanding, one big part > of AMG SetUp is RAP that should be pretty much bandwidth. The RAP isn't "pretty much bandwidth". See below for some Skylake/POWER9/EPYC results and analysis (copied from an off-list thread). I'll leave in some other bandwidth comments that may or may not be relevant to you. The short story is that Skylake and EPYC are both much better than POWER9 at MatPtAP despite POWER9 having similar bandwidth as EPYC and thus being significantly faster than Skylake for MatMult/smoothing. Jed Brown writes: > I'm attaching a log from my machine (Noether), which is 2-socket EPYC > 7452 (32 cores each). Each socket has 8xDDR4-3200 and 128 MB of L3 > cache. This is the same node architecture as the new BER/E3SM machine > being installed at Argonne (though that one will probably have > higher-clocked and/or more cores per socket). Note that these CPUs are > about $2k each while Skylake 8180 are about $10k. > > Some excerpts/comments below. > [...] In addition to the notes below, I'd like to call out how important streaming stores are on EPYC. With vanilla code or _mm256_store_pd, we get the following performance $ mpiexec -n 64 --bind-to core --map-by core:1 src/benchmarks/streams/MPIVersion Copy 162609.2392 Scale 159119.8259 Add 174687.6250 Triad 175840.1587 but replacing _mm256_store_pd with _mm256_stream_pd gives this $ mpiexec -n 64 --bind-to core --map-by core:1 src/benchmarks/streams/MPIVersion Copy 259951.9936 Scale 259381.0589 Add 250216.3389 Triad 249292.9701 This is just preposterously huge, but very repeatable using gcc and clang, and inspecting the assembly. This suggests that it would be useful for vector kernels to have streaming and non-streaming variants. That is, if I drop the vector length by 20 (so the working set is 2.3 MB/core instead of 46 MB in the default version), then we get 2.4 TB/s Triad with _mm256_store_pd: $ mpiexec -n 64 --bind-to core --map-by core:1 src/benchmarks/streams/MPIVersion Copy 2159915.7058 Scale 2212671.7087 Add 2414758.2757 Triad 2402671.1178 and a thoroughly embarrassing 353 GB/s with _mm256_stream_pd: $ mpiexec -n 64 --bind-to core --map-by core:1 src/benchmarks/streams/MPIVersion Copy 235934.6653 Scale 237446.8507 Add 352805.7288 Triad 352992.9692 I don't know a good way to automatically determine whether to expect the memory to be in cache, but we could make it a global (or per-object) run-time selection. > Jed Brown writes: > >> "Smith, Barry F." writes: >> >>> Thanks. The PowerPC is pretty crappy compared to Skylake. >> >> Compare the MGSmooth times. The POWER9 is faster than the Skylake >> because it has more memory bandwidth. >> >> $ rg 'MGInterp Level 4|MGSmooth Level 4' ex56* >> ex56-JLSE-skylake-56ranks-converged.txt >> 254:MGSmooth Level 4 68 1.0 1.8808e+00 1.2 7.93e+08 1.3 3.6e+04 1.9e+04 3.4e+01 8 29 10 16 3 62 60 18 54 25 22391 >> 256:MGInterp Level 4 68 1.0 4.0043e-01 1.8 1.45e+08 1.3 2.2e+04 2.5e+03 0.0e+00 1 5 6 1 0 9 11 11 4 0 19109 >> >> ex56-summit-cpu-36ranks-converged.txt >> 265:MGSmooth Level 4 68 1.0 1.1531e+00 1.1 1.22e+09 1.2 2.3e+04 2.6e+04 3.4e+01 3 29 7 13 3 61 60 12 54 25 36519 0 0 0.00e+00 0 0.00e+00 0 >> 267:MGInterp Level 4 68 1.0 2.0749e-01 1.1 2.23e+08 1.2 1.4e+04 3.4e+03 0.0e+00 0 5 4 1 0 11 11 7 4 0 36925 0 0 0.00e+00 0 0.00e+00 0 >> >> ex56-summit-gpu-24ranks-converged.txt >> 275:MGSmooth Level 4 68 1.0 1.4499e-01 1.2 1.85e+09 1.2 1.0e+04 5.3e+04 3.4e+01 0 29 7 13 3 26 60 12 55 25 299156 940881 115 2.46e+01 116 8.64e+01 100 >> 277:MGInterp Level 4 68 1.0 1.7674e-01 1.0 3.23e+08 1.2 6.1e+03 6.7e+03 0.0e+00 0 5 4 1 0 33 11 7 4 0 42715 621223 36 2.98e+01 136 3.95e+00 100 >> >> ex56-summit-gpu-36ranks-converged.txt >> 275:MGSmooth Level 4 68 1.0 1.4877e-01 1.2 1.25e+09 1.2 2.3e+04 2.6e+04 3.4e+01 0 29 7 13 3 19 60 12 54 25 291548 719522 115 1.83e+01 116 5.80e+01 100 >> 277:MGInterp Level 4 68 1.0 2.4317e-01 1.0 2.20e+08 1.2 1.4e+04 3.4e+03 0.0e+00 0 5 4 1 0 33 11 7 4 0 31062 586044 36 1.99e+01 136 2.82e+00 100 > > 258:MGSmooth Level 4 68 1.0 9.6950e-01 1.3 6.15e+08 1.3 4.0e+04 1.4e+04 2.0e+00 6 28 10 15 0 59 59 18 54 25 39423 > 260:MGInterp Level 4 68 1.0 2.5707e-01 1.5 1.23e+08 1.2 2.7e+04 1.9e+03 0.0e+00 1 5 7 1 0 13 12 12 5 0 29294 > > Epyc is faster than Power9 is faster than Sklake. > >> >> The Skylake is a lot faster at PtAP. It'd be interesting to better >> understand why. Perhaps it has to do with caching or aggressiveness of >> out-of-order execution. >> >> $ rg 'PtAP' ex56* >> ex56-JLSE-skylake-56ranks-converged.txt >> 164:MatPtAP 4 1.0 1.4214e+00 1.0 3.94e+08 1.5 1.1e+04 7.4e+04 4.4e+01 6 13 3 20 4 8 28 8 39 5 13754 >> 165:MatPtAPSymbolic 4 1.0 8.3981e-01 1.0 0.00e+00 0.0 6.5e+03 7.3e+04 2.8e+01 4 0 2 12 2 5 0 5 23 3 0 >> 166:MatPtAPNumeric 4 1.0 5.8402e-01 1.0 3.94e+08 1.5 4.5e+03 7.5e+04 1.6e+01 2 13 1 8 1 3 28 3 16 2 33474 >> >> ex56-summit-cpu-36ranks-converged.txt >> 164:MatPtAP 4 1.0 3.9077e+00 1.0 5.89e+08 1.4 1.6e+04 7.4e+04 4.4e+01 9 13 5 26 4 11 28 12 46 5 4991 0 0 0.00e+00 0 0.00e+00 0 >> 165:MatPtAPSymbolic 4 1.0 1.9525e+00 1.0 0.00e+00 0.0 1.2e+04 7.3e+04 2.8e+01 5 0 4 19 3 5 0 9 34 3 0 0 0 0.00e+00 0 0.00e+00 0 >> 166:MatPtAPNumeric 4 1.0 1.9621e+00 1.0 5.89e+08 1.4 4.0e+03 7.5e+04 1.6e+01 5 13 1 7 1 5 28 3 12 2 9940 0 0 0.00e+00 0 0.00e+00 0 >> >> ex56-summit-gpu-24ranks-converged.txt >> 167:MatPtAP 4 1.0 5.7210e+00 1.0 8.48e+08 1.3 7.5e+03 1.3e+05 4.4e+01 8 13 5 25 4 11 28 12 46 5 3415 0 16 3.36e+01 4 6.30e-02 0 >> 168:MatPtAPSymbolic 4 1.0 2.8717e+00 1.0 0.00e+00 0.0 5.5e+03 1.3e+05 2.8e+01 4 0 4 19 3 5 0 9 34 3 0 0 0 0.00e+00 0 0.00e+00 0 >> 169:MatPtAPNumeric 4 1.0 2.8537e+00 1.0 8.48e+08 1.3 2.0e+03 1.3e+05 1.6e+01 4 13 1 7 1 5 28 3 12 2 6846 0 16 3.36e+01 4 6.30e-02 0 >> >> ex56-summit-gpu-36ranks-converged.txt >> 167:MatPtAP 4 1.0 4.0340e+00 1.0 5.89e+08 1.4 1.6e+04 7.4e+04 4.4e+01 8 13 5 26 4 11 28 12 46 5 4835 0 16 2.30e+01 4 5.18e-02 0 >> 168:MatPtAPSymbolic 4 1.0 2.0355e+00 1.0 0.00e+00 0.0 1.2e+04 7.3e+04 2.8e+01 4 0 4 19 3 5 0 9 34 3 0 0 0 0.00e+00 0 0.00e+00 0 >> 169:MatPtAPNumeric 4 1.0 2.0050e+00 1.0 5.89e+08 1.4 4.0e+03 7.5e+04 1.6e+01 4 13 1 7 1 5 28 3 12 2 9728 0 16 2.30e+01 4 5.18e-02 0 > > 153:MatPtAPSymbolic 4 1.0 7.6053e-01 1.0 0.00e+00 0.0 7.6e+03 5.8e+04 2.8e+01 5 0 2 12 2 6 0 5 22 3 0 > 154:MatPtAPNumeric 4 1.0 6.5172e-01 1.0 3.21e+08 1.4 6.4e+03 4.8e+04 2.4e+01 4 14 2 8 2 5 27 4 16 2 28861 > > Epyc similar to Skylake here. > >> I'd really like to compare an EPYC for these operations. I bet it's >> pretty good. (More bandwidth than Skylake, bigger caches, but no >> AVX512.) >> >>> So the biggest consumer is MatPtAP I guess that should be done first. >>> >>> It would be good to have these results exclude the Jacobian and Function evaluation which really dominate the time and add clutter making it difficult to see the problems with the rest of SNESSolve. >>> >>> >>> Did you notice: >>> >>> MGInterp Level 4 68 1.0 1.7674e-01 1.0 3.23e+08 1.2 6.1e+03 6.7e+03 0.0e+00 0 5 4 1 0 33 11 7 4 0 42715 621223 36 2.98e+01 136 3.95e+00 100 >>> >>> it is terrible! Well over half of the KSPSolve time is in this one relatively minor routine. All of the interps are terribly slow. Is it related to the transpose multiple or something? >> >> Yes, it's definitely the MatMultTranspose, which must be about 3x more >> expensive than restriction even on the CPU. PCMG/PCGAMG should >> explicitly transpose (unless the user sets an option to aggressively >> minimize memory usage). >> >> $ rg 'MGInterp|MultTrans' ex56* >> ex56-JLSE-skylake-56ranks-converged.txt >> 222:MatMultTranspose 136 1.0 3.5105e-01 3.7 7.91e+07 1.3 2.5e+04 1.3e+03 0.0e+00 1 3 7 1 0 5 6 13 3 0 11755 >> 247:MGInterp Level 1 68 1.0 3.3894e-04 2.2 2.35e+05 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 693 >> 250:MGInterp Level 2 68 1.0 1.1212e-0278.0 1.17e+06 0.0 1.8e+03 7.7e+02 0.0e+00 0 0 1 0 0 0 0 1 0 0 2172 >> 253:MGInterp Level 3 68 1.0 6.7105e-02 5.3 1.23e+07 1.8 2.7e+04 4.2e+02 0.0e+00 0 0 8 0 0 1 1 14 1 0 8594 >> 256:MGInterp Level 4 68 1.0 4.0043e-01 1.8 1.45e+08 1.3 2.2e+04 2.5e+03 0.0e+00 1 5 6 1 0 9 11 11 4 0 19109 >> >> ex56-summit-cpu-36ranks-converged.txt >> 229:MatMultTranspose 136 1.0 1.4832e-01 1.4 1.21e+08 1.2 1.9e+04 1.5e+03 0.0e+00 0 3 6 1 0 6 6 10 3 0 27842 0 0 0.00e+00 0 0.00e+00 0 >> 258:MGInterp Level 1 68 1.0 2.9145e-04 1.5 1.08e+05 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 370 0 0 0.00e+00 0 0.00e+00 0 >> 261:MGInterp Level 2 68 1.0 5.7095e-03 1.5 9.16e+05 2.5 2.4e+03 7.1e+02 0.0e+00 0 0 1 0 0 0 0 1 0 0 4093 0 0 0.00e+00 0 0.00e+00 0 >> 264:MGInterp Level 3 68 1.0 3.5654e-02 2.8 1.77e+07 1.5 2.3e+04 3.9e+02 0.0e+00 0 0 7 0 0 1 1 12 1 0 16095 0 0 0.00e+00 0 0.00e+00 0 >> 267:MGInterp Level 4 68 1.0 2.0749e-01 1.1 2.23e+08 1.2 1.4e+04 3.4e+03 0.0e+00 0 5 4 1 0 11 11 7 4 0 36925 0 0 0.00e+00 0 0.00e+00 0 >> >> ex56-summit-gpu-24ranks-converged.txt >> 236:MatMultTranspose 136 1.0 2.1445e-01 1.0 1.72e+08 1.2 9.5e+03 2.6e+03 0.0e+00 0 3 6 1 0 39 6 11 3 0 18719 451131 8 3.11e+01 272 2.19e+00 100 >> 268:MGInterp Level 1 68 1.0 4.0388e-03 2.8 1.08e+05 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 27 79 37 5.84e-04 68 6.80e-05 100 >> 271:MGInterp Level 2 68 1.0 2.9033e-02 2.9 1.25e+06 1.9 1.6e+03 7.8e+02 0.0e+00 0 0 1 0 0 5 0 2 0 0 812 11539 36 1.14e-01 136 5.41e-02 100 >> 274:MGInterp Level 3 68 1.0 4.9503e-02 1.1 2.50e+07 1.4 1.1e+04 6.3e+02 0.0e+00 0 0 7 0 0 9 1 13 1 0 11476 100889 36 2.29e+00 136 3.74e-01 100 >> 277:MGInterp Level 4 68 1.0 1.7674e-01 1.0 3.23e+08 1.2 6.1e+03 6.7e+03 0.0e+00 0 5 4 1 0 33 11 7 4 0 42715 621223 36 2.98e+01 136 3.95e+00 100 >> >> ex56-summit-gpu-36ranks-converged.txt >> 236:MatMultTranspose 136 1.0 2.9692e-01 1.0 1.17e+08 1.2 1.9e+04 1.5e+03 0.0e+00 1 3 6 1 0 40 6 10 3 0 13521 336701 8 2.08e+01 272 1.59e+00 100 >> 268:MGInterp Level 1 68 1.0 3.8752e-03 2.5 1.03e+05 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 27 79 37 3.95e-04 68 4.53e-05 100 >> 271:MGInterp Level 2 68 1.0 3.5465e-02 2.2 9.12e+05 2.5 2.4e+03 7.1e+02 0.0e+00 0 0 1 0 0 4 0 1 0 0 655 5989 36 8.16e-02 136 4.89e-02 100 >> 274:MGInterp Level 3 68 1.0 6.7101e-02 1.1 1.75e+07 1.5 2.3e+04 3.9e+02 0.0e+00 0 0 7 0 0 9 1 12 1 0 8455 56175 36 1.55e+00 136 3.03e-01 100 >> 277:MGInterp Level 4 68 1.0 2.4317e-01 1.0 2.20e+08 1.2 1.4e+04 3.4e+03 0.0e+00 0 5 4 1 0 33 11 7 4 0 31062 586044 36 1.99e+01 136 2.82e+00 100 > > 223:MatMultTranspose 136 1.0 2.0702e-01 2.9 6.59e+07 1.2 2.7e+04 1.1e+03 0.0e+00 1 3 7 1 0 7 6 12 3 0 19553 > 251:MGInterp Level 1 68 1.0 2.8062e-04 1.5 9.79e+04 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 349 > 254:MGInterp Level 2 68 1.0 6.2506e-0331.9 9.69e+05 0.0 2.1e+03 6.3e+02 0.0e+00 0 0 1 0 0 0 0 1 0 0 3458 > 257:MGInterp Level 3 68 1.0 4.8159e-02 6.5 9.62e+06 1.5 2.5e+04 4.2e+02 0.0e+00 0 0 6 0 0 1 1 11 1 0 11199 > 260:MGInterp Level 4 68 1.0 2.5707e-01 1.5 1.23e+08 1.2 2.7e+04 1.9e+03 0.0e+00 1 5 7 1 0 13 12 12 5 0 29294 > > Power9 still has an edge here. From jed at jedbrown.org Tue Jun 9 23:11:30 2020 From: jed at jedbrown.org (Jed Brown) Date: Tue, 09 Jun 2020 22:11:30 -0600 Subject: [petsc-users] Make stream In-Reply-To: References: <5364BA02-DF4C-446D-844A-86EC019EB7B2@petsc.dev> Message-ID: <87mu5bd0gd.fsf@jedbrown.org> Fande Kong writes: > Thanks so much, Barry, > > On Tue, Jun 9, 2020 at 3:08 PM Barry Smith wrote: > >> >> You might look at the notes about MPI binding. It might give you a bit >> better performance. >> https://www.mcs.anl.gov/petsc/documentation/faq.html#computers >> > > I am using mvapich2, and still trying to look for which binding command > lines I can use. http://mvapich.cse.ohio-state.edu/static/media/mvapich/mvapich2-2.3.4-userguide.html#x1-650006.5.1 https://wiki.mpich.org/mpich/index.php/Using_the_Hydra_Process_Manager#Process-core_Binding From eda.oktay at metu.edu.tr Wed Jun 10 07:56:06 2020 From: eda.oktay at metu.edu.tr (Eda Oktay) Date: Wed, 10 Jun 2020 15:56:06 +0300 Subject: [petsc-users] MatGetRow for global rows of a parallel matrix Message-ID: Hi all, I am trying to get all the rows of a parallel matrix as individual vectors. For instance, if I have 72*4 matrix, I want to get 72 different vectors having size 4. As far as I understood, MatGetRow is only for local rows, so MatGetOwnershipRange is used, however, when I tried this one, I couldn't get the whole and desired row vectors. In MatGetRow explanation, it is written that I should use MatCreateSubMatrices first, then use MatGetRow. But I couldn't understand to which extent I should create submatrices. I just need to have all 72 rows as 72 different vectors each having 4 elements. Thanks! Eda From mlohry at gmail.com Wed Jun 10 08:00:30 2020 From: mlohry at gmail.com (Mark Lohry) Date: Wed, 10 Jun 2020 09:00:30 -0400 Subject: [petsc-users] L infinity norm convergence tests Message-ID: Hi all, is there a built-in way to use L-infinity norms for the SNES and KSP convergence tests, or do I need to write a manual KSPSetConvergenceTest function? Thanks, Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Jun 10 08:03:40 2020 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 10 Jun 2020 09:03:40 -0400 Subject: [petsc-users] MatGetRow for global rows of a parallel matrix In-Reply-To: References: Message-ID: On Wed, Jun 10, 2020 at 8:56 AM Eda Oktay wrote: > Hi all, > > I am trying to get all the rows of a parallel matrix as individual > vectors. For instance, if I have 72*4 matrix, I want to get 72 > different vectors having size 4. > > As far as I understood, MatGetRow is only for local rows, so > MatGetOwnershipRange is used, however, when I tried this one, I > couldn't get the whole and desired row vectors. > > In MatGetRow explanation, it is written that I should use > MatCreateSubMatrices first, then use MatGetRow. But I couldn't > understand to which extent I should create submatrices. I just need to > have all 72 rows as 72 different vectors each having 4 elements. > 1) For sparse matrices, the storage is always divided by row, so that values can only be retrieved for local rows with MatGetRow() 2) Is this matrix sparse? It sounds like it is dense. 3) Are you asking to get all matrix values on all processes? If so, I think the easiest thing to do is first wrap a Vec around the values, then use VecScatterToAll(), then wrap each one in a MatDense again. Thanks, Matt > Thanks! > > Eda > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Jun 10 08:04:11 2020 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 10 Jun 2020 09:04:11 -0400 Subject: [petsc-users] L infinity norm convergence tests In-Reply-To: References: Message-ID: On Wed, Jun 10, 2020 at 9:02 AM Mark Lohry wrote: > Hi all, is there a built-in way to use L-infinity norms for the SNES and > KSP convergence tests, or do I need to write a manual KSPSetConvergenceTest > function? > You need to write a custom test. Thanks, Matt > Thanks, > Mark > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From eda.oktay at metu.edu.tr Wed Jun 10 08:08:22 2020 From: eda.oktay at metu.edu.tr (Eda Oktay) Date: Wed, 10 Jun 2020 16:08:22 +0300 Subject: [petsc-users] MatGetRow for global rows of a parallel matrix In-Reply-To: References: Message-ID: Dear Matt, Matthew Knepley , 10 Haz 2020 ?ar, 16:03 tarihinde ?unu yazd?: > > On Wed, Jun 10, 2020 at 8:56 AM Eda Oktay wrote: >> >> Hi all, >> >> I am trying to get all the rows of a parallel matrix as individual >> vectors. For instance, if I have 72*4 matrix, I want to get 72 >> different vectors having size 4. >> >> As far as I understood, MatGetRow is only for local rows, so >> MatGetOwnershipRange is used, however, when I tried this one, I >> couldn't get the whole and desired row vectors. >> >> In MatGetRow explanation, it is written that I should use >> MatCreateSubMatrices first, then use MatGetRow. But I couldn't >> understand to which extent I should create submatrices. I just need to >> have all 72 rows as 72 different vectors each having 4 elements. > > > 1) For sparse matrices, the storage is always divided by row, so that values can only be retrieved for local rows with MatGetRow() > > 2) Is this matrix sparse? It sounds like it is dense. Matrix is dense. > > 3) Are you asking to get all matrix values on all processes? If so, I think the easiest thing to do is first wrap a Vec around the > values, then use VecScatterToAll(), then wrap each one in a MatDense again. Yes, I want all row vectors on all processes. In a dense matrix, should I still wrap a Vec around the values? I know I should use scatter but I couldn't even wrap a Vec around them. Thanks so much! Eda > > Thanks, > > Matt > >> >> Thanks! >> >> Eda > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ From knepley at gmail.com Wed Jun 10 08:16:36 2020 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 10 Jun 2020 09:16:36 -0400 Subject: [petsc-users] MatGetRow for global rows of a parallel matrix In-Reply-To: References: Message-ID: On Wed, Jun 10, 2020 at 9:08 AM Eda Oktay wrote: > Dear Matt, > > Matthew Knepley , 10 Haz 2020 ?ar, 16:03 tarihinde > ?unu yazd?: > > > > On Wed, Jun 10, 2020 at 8:56 AM Eda Oktay wrote: > >> > >> Hi all, > >> > >> I am trying to get all the rows of a parallel matrix as individual > >> vectors. For instance, if I have 72*4 matrix, I want to get 72 > >> different vectors having size 4. > >> > >> As far as I understood, MatGetRow is only for local rows, so > >> MatGetOwnershipRange is used, however, when I tried this one, I > >> couldn't get the whole and desired row vectors. > >> > >> In MatGetRow explanation, it is written that I should use > >> MatCreateSubMatrices first, then use MatGetRow. But I couldn't > >> understand to which extent I should create submatrices. I just need to > >> have all 72 rows as 72 different vectors each having 4 elements. > > > > > > 1) For sparse matrices, the storage is always divided by row, so that > values can only be retrieved for local rows with MatGetRow() > > > > 2) Is this matrix sparse? It sounds like it is dense. > > Matrix is dense. > > > > > 3) Are you asking to get all matrix values on all processes? If so, I > think the easiest thing to do is first wrap a Vec around the > > values, then use VecScatterToAll(), then wrap each one in a MatDense > again. > > Yes, I want all row vectors on all processes. In a dense matrix, > should I still wrap a Vec around the values? I know I should use > scatter but I couldn't even wrap a Vec around them. > I would do MatGetSize(&N); MatGetLocalSize(&m); https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatDenseGetArray.html https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecPlaceArray.html https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecScatterCreateToAll.html https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecResetArray.html#VecResetArray https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatCreateDense.html Thanks, Matt > Thanks so much! > > Eda > > > > > Thanks, > > > > Matt > > > >> > >> Thanks! > >> > >> Eda > > > > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > > -- Norbert Wiener > > > > https://www.cse.buffalo.edu/~knepley/ > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From eda.oktay at metu.edu.tr Wed Jun 10 09:09:43 2020 From: eda.oktay at metu.edu.tr (Eda Oktay) Date: Wed, 10 Jun 2020 17:09:43 +0300 Subject: [petsc-users] MatGetRow for global rows of a parallel matrix In-Reply-To: References: Message-ID: Dear Matt, I have one last question I believe. Up to creating a dense matrix I did what you've suggested. Thank you so much for that. I created a new dense matrix. Now, how should I wrap each vector in a MatDense again? I mean, what is wrapping vectors in a matrix? To put each of them again as rows? Thanks! Eda Matthew Knepley , 10 Haz 2020 ?ar, 16:16 tarihinde ?unu yazd?: > > On Wed, Jun 10, 2020 at 9:08 AM Eda Oktay wrote: >> >> Dear Matt, >> >> Matthew Knepley , 10 Haz 2020 ?ar, 16:03 tarihinde >> ?unu yazd?: >> > >> > On Wed, Jun 10, 2020 at 8:56 AM Eda Oktay wrote: >> >> >> >> Hi all, >> >> >> >> I am trying to get all the rows of a parallel matrix as individual >> >> vectors. For instance, if I have 72*4 matrix, I want to get 72 >> >> different vectors having size 4. >> >> >> >> As far as I understood, MatGetRow is only for local rows, so >> >> MatGetOwnershipRange is used, however, when I tried this one, I >> >> couldn't get the whole and desired row vectors. >> >> >> >> In MatGetRow explanation, it is written that I should use >> >> MatCreateSubMatrices first, then use MatGetRow. But I couldn't >> >> understand to which extent I should create submatrices. I just need to >> >> have all 72 rows as 72 different vectors each having 4 elements. >> > >> > >> > 1) For sparse matrices, the storage is always divided by row, so that values can only be retrieved for local rows with MatGetRow() >> > >> > 2) Is this matrix sparse? It sounds like it is dense. >> >> Matrix is dense. >> >> > >> > 3) Are you asking to get all matrix values on all processes? If so, I think the easiest thing to do is first wrap a Vec around the >> > values, then use VecScatterToAll(), then wrap each one in a MatDense again. >> >> Yes, I want all row vectors on all processes. In a dense matrix, >> should I still wrap a Vec around the values? I know I should use >> scatter but I couldn't even wrap a Vec around them. > > > I would do > > MatGetSize(&N); > MatGetLocalSize(&m); > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatDenseGetArray.html > > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecPlaceArray.html > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecScatterCreateToAll.html > > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecResetArray.html#VecResetArray > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatCreateDense.html > > > > > Thanks, > > Matt > >> >> Thanks so much! >> >> Eda >> >> > >> > Thanks, >> > >> > Matt >> > >> >> >> >> Thanks! >> >> >> >> Eda >> > >> > >> > >> > -- >> > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >> > -- Norbert Wiener >> > >> > https://www.cse.buffalo.edu/~knepley/ > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ From knepley at gmail.com Wed Jun 10 10:11:01 2020 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 10 Jun 2020 11:11:01 -0400 Subject: [petsc-users] MatGetRow for global rows of a parallel matrix In-Reply-To: References: Message-ID: On Wed, Jun 10, 2020 at 10:09 AM Eda Oktay wrote: > Dear Matt, > > I have one last question I believe. Up to creating a dense matrix I > did what you've suggested. Thank you so much for that. > > I created a new dense matrix. Now, how should I wrap each vector in a > MatDense again? I mean, what is wrapping vectors in a matrix? To put > each of them again as rows? > I thought you need a dense matrix for something, since you started with one. If you do not, just do VecGetArray() on the vector from CreateToAll and use the values. Thanks, Matt > Thanks! > > Eda > > Matthew Knepley , 10 Haz 2020 ?ar, 16:16 tarihinde > ?unu yazd?: > > > > On Wed, Jun 10, 2020 at 9:08 AM Eda Oktay wrote: > >> > >> Dear Matt, > >> > >> Matthew Knepley , 10 Haz 2020 ?ar, 16:03 tarihinde > >> ?unu yazd?: > >> > > >> > On Wed, Jun 10, 2020 at 8:56 AM Eda Oktay > wrote: > >> >> > >> >> Hi all, > >> >> > >> >> I am trying to get all the rows of a parallel matrix as individual > >> >> vectors. For instance, if I have 72*4 matrix, I want to get 72 > >> >> different vectors having size 4. > >> >> > >> >> As far as I understood, MatGetRow is only for local rows, so > >> >> MatGetOwnershipRange is used, however, when I tried this one, I > >> >> couldn't get the whole and desired row vectors. > >> >> > >> >> In MatGetRow explanation, it is written that I should use > >> >> MatCreateSubMatrices first, then use MatGetRow. But I couldn't > >> >> understand to which extent I should create submatrices. I just need > to > >> >> have all 72 rows as 72 different vectors each having 4 elements. > >> > > >> > > >> > 1) For sparse matrices, the storage is always divided by row, so that > values can only be retrieved for local rows with MatGetRow() > >> > > >> > 2) Is this matrix sparse? It sounds like it is dense. > >> > >> Matrix is dense. > >> > >> > > >> > 3) Are you asking to get all matrix values on all processes? If so, I > think the easiest thing to do is first wrap a Vec around the > >> > values, then use VecScatterToAll(), then wrap each one in a > MatDense again. > >> > >> Yes, I want all row vectors on all processes. In a dense matrix, > >> should I still wrap a Vec around the values? I know I should use > >> scatter but I couldn't even wrap a Vec around them. > > > > > > I would do > > > > MatGetSize(&N); > > MatGetLocalSize(&m); > > > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatDenseGetArray.html > > > > > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecPlaceArray.html > > > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecScatterCreateToAll.html > > > > > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecResetArray.html#VecResetArray > > > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatCreateDense.html > > > > > > > > > > Thanks, > > > > Matt > > > >> > >> Thanks so much! > >> > >> Eda > >> > >> > > >> > Thanks, > >> > > >> > Matt > >> > > >> >> > >> >> Thanks! > >> >> > >> >> Eda > >> > > >> > > >> > > >> > -- > >> > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > >> > -- Norbert Wiener > >> > > >> > https://www.cse.buffalo.edu/~knepley/ > > > > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > > -- Norbert Wiener > > > > https://www.cse.buffalo.edu/~knepley/ > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From adener at anl.gov Wed Jun 10 10:29:26 2020 From: adener at anl.gov (Dener, Alp) Date: Wed, 10 Jun 2020 15:29:26 +0000 Subject: [petsc-users] TAO STCG initial perturbation In-Reply-To: References: Message-ID: Hello, STCG is being used to compute a search direction by inverting the Hessian of the objective onto the gradient. The Hessian has to be positive definitive for this search direction to be a valid descent direction. To enforce this, STCG terminates the KSP solution when it encounters negative curvature (i.e. ?uphill? directions). The resulting search direction from this early termination is still a descent direction but it?s a pretty bad one, meaning that the line search is forced to do extra work to find a valid (and sometimes pretty small) step length. This is normal. Once a negative curvature is detected, the NLS algorithm imposes a shift to the diagonal of the Hessian in order to guarantee that it will be positive-definite on the next optimization iteration. This diagonal shift is increased in magnitude if you keep hitting the negative curvature KSP termination, and it?s gradually decreased on iterations where the KSP solution succeeds. However this does not prevent the line search from doing all that extra work on iterations where the search direction solution never converged fully. It simply helps generate better search directions in subsequent iterations and helps the overall optimization solution eventually converge to a valid minimum. As a side note, if your function, gradient and/or Hessian evaluations are expensive, it might actually make more sense to use the quasi-Newton method (BQNLS) instead of Newton line search. The optimization solution is likely to take more overall iterations with BQNLS(-tao_type bqnls), but the BFGS approximation used by the algorithm is inherently guaranteed to be positive-definite and tends to generate very well scaled search directions so it might help the line search do less work. Combined with the elimination of Hessian evaluations, it may very well solve your problem faster in CPU time even if it takes more nonlinear iterations overall. This is technically a bound constrained method but it solves unconstrained problems too when you don?t define any bounds. About Levenberg-Marquardt: a user started the branch to eventually contribute an LM solver, but I have not heard any updates on it since end of April. For least-squares type problems, you can try using the regularized Gauss-Newton solver (-tao_type brgn). The problem definition interface is a bit different. BRGN requires the problem to be defined as a residual and its Jacobian, and it will assemble the gradient and the Hessian on its own and feed it into the standard Newton line search solver underneath. There are also a few different regularization options available, like proximal point Tikhonov (l2prox) and a sparsity term (l1dict) that can be set with the (-tao_brgn_regularization_type) option. These get automatically cooked into the gradient and Hessian. Combined with the automatic diagonal shifts for the Hessian, BRGN really is almost equivalent to an LM solver so it might do what you need. Hope this helps! ? Alp On Jun 9, 2020, at 10:55 PM, zakaryah . > wrote: I am using TAO to minimize the elastic strain energy for somewhat complicated applied forces. With Newton linesearch (NLS), the STCG KSP, and several preconditioners (none, Jacobi, lmvm), the solver finds a minimum within an acceptable number of iterations (<50). Still, in the interest of performance, I am wondering about what happens after the KSP encounters an indefinite matrix (KSP_CONVERGED_CG_NEG_CURVE). After this happens, the line search performs extremely poorly, with function values at step length 1 reaching absurdly large values. As the line search tries smaller step lengths, the function values fall, then typically reach values representing a decline from the previous iteration, but only with tiny step lengths (~1e-11). The line search takes many iterations to arrive at such an improvement. Here is an example, run with -tao_type nls -tao_nls_ksp_type stcg -tao_nls_pc_type none -tao_nls_ksp_norm_type unpreconditioned: 0 TAO, Function value: 0.160612, Residual: 0.0736074 0 TAO, Function value: 0.121568, Residual: 0.0424117 Residual norms for tao_nls_ solve. 0 KSP Residual norm 4.241174778983e-02 1 KSP Residual norm 5.806891169509e-02 2 KSP Residual norm 6.492953448014e-02 3 KSP Residual norm 5.856559430984e-02 4 KSP Residual norm 5.262548559710e-02 5 KSP Residual norm 4.863633473400e-02 6 KSP Residual norm 4.725156347026e-02 7 KSP Residual norm 4.748458009319e-02 8 KSP Residual norm 4.885339641711e-02 9 KSP Residual norm 5.065071226354e-02 10 KSP Residual norm 5.085544070851e-02 11 KSP Residual norm 5.127093547976e-02 12 KSP Residual norm 5.155225884843e-02 13 KSP Residual norm 5.219895021408e-02 14 KSP Residual norm 6.480520610077e-02 15 KSP Residual norm 1.433515456621e-01 Linear tao_nls_ solve converged due to CONVERGED_CG_NEG_CURVE iterations 16 0 LS Function value: 0.121568, Step length: 0. 1 LS Function value: 5.99839e+59, Step length: 1. stx: 0., fx: 0.121568, dgx: -1.32893e+08 sty: 0., fy: 0.121568, dgy: -1.32893e+08 2 LS Function value: 1.46445e+56, Step length: 0.25 stx: 0., fx: 0.121568, dgx: -1.32893e+08 sty: 1., fy: 5.99839e+59, dgy: 3.59903e+60 3 LS Function value: 3.57532e+52, Step length: 0.0625 stx: 0., fx: 0.121568, dgx: -1.32893e+08 sty: 0.25, fy: 1.46445e+56, dgy: 3.51468e+57 4 LS Function value: 8.7288e+48, Step length: 0.015625 stx: 0., fx: 0.121568, dgx: -1.32893e+08 sty: 0.0625, fy: 3.57532e+52, dgy: 3.4323e+54 5 LS Function value: 2.13105e+45, Step length: 0.00390625 stx: 0., fx: 0.121568, dgx: -1.32893e+08 sty: 0.015625, fy: 8.7288e+48, dgy: 3.35186e+51 6 LS Function value: 5.20277e+41, Step length: 0.000976562 stx: 0., fx: 0.121568, dgx: -1.32893e+08 sty: 0.00390625, fy: 2.13105e+45, dgy: 3.2733e+48 7 LS Function value: 1.27021e+38, Step length: 0.000244141 stx: 0., fx: 0.121568, dgx: -1.32893e+08 sty: 0.000976562, fy: 5.20277e+41, dgy: 3.19658e+45 8 LS Function value: 3.1011e+34, Step length: 6.10352e-05 stx: 0., fx: 0.121568, dgx: -1.32893e+08 sty: 0.000244141, fy: 1.27021e+38, dgy: 3.12166e+42 9 LS Function value: 7.57106e+30, Step length: 1.52588e-05 stx: 0., fx: 0.121568, dgx: -1.32893e+08 sty: 6.10352e-05, fy: 3.1011e+34, dgy: 3.0485e+39 10 LS Function value: 1.84842e+27, Step length: 3.8147e-06 stx: 0., fx: 0.121568, dgx: -1.32893e+08 sty: 1.52588e-05, fy: 7.57106e+30, dgy: 2.97706e+36 11 LS Function value: 4.51295e+23, Step length: 9.53672e-07 stx: 0., fx: 0.121568, dgx: -1.32893e+08 sty: 3.8147e-06, fy: 1.84842e+27, dgy: 2.90731e+33 12 LS Function value: 1.10199e+20, Step length: 2.38417e-07 stx: 0., fx: 0.121568, dgx: -1.32893e+08 sty: 9.53672e-07, fy: 4.51295e+23, dgy: 2.83927e+30 13 LS Function value: 2.69231e+16, Step length: 5.96028e-08 stx: 0., fx: 0.121568, dgx: -1.32893e+08 sty: 2.38417e-07, fy: 1.10199e+20, dgy: 2.77314e+27 14 LS Function value: 6.59192e+12, Step length: 1.48993e-08 stx: 0., fx: 0.121568, dgx: -1.32893e+08 sty: 5.96028e-08, fy: 2.69231e+16, dgy: 2.70974e+24 15 LS Function value: 1.62897e+09, Step length: 3.72338e-09 stx: 0., fx: 0.121568, dgx: -1.32893e+08 sty: 1.48993e-08, fy: 6.59192e+12, dgy: 2.65254e+21 16 LS Function value: 421305., Step length: 9.29291e-10 stx: 0., fx: 0.121568, dgx: -1.32893e+08 sty: 3.72338e-09, fy: 1.62897e+09, dgy: 2.61626e+18 17 LS Function value: 141.939, Step length: 2.30343e-10 stx: 0., fx: 0.121568, dgx: -1.32893e+08 sty: 9.29291e-10, fy: 421305., dgy: 2.67496e+15 18 LS Function value: 0.213621, Step length: 5.45712e-11 stx: 0., fx: 0.121568, dgx: -1.32893e+08 sty: 2.30343e-10, fy: 141.939, dgy: 3.35874e+12 19 LS Function value: 0.120058, Step length: 1.32286e-11 stx: 0., fx: 0.121568, dgx: -1.32893e+08 sty: 5.45712e-11, fy: 0.213621, dgy: 8.60977e+09 1 TAO, Function value: 0.120058, Residual: 0.0484495 When the KSP does not encounter negative curvature, the linesearch performs well: 3 TAO, Function value: 0.118376, Residual: 0.0545446 Residual norms for tao_nls_ solve. 0 KSP Residual norm 5.454463134947e-02 1 KSP Residual norm 2.394277461960e-02 2 KSP Residual norm 6.674182971627e-03 3 KSP Residual norm 1.235116278289e-03 4 KSP Residual norm 1.714792759324e-04 5 KSP Residual norm 3.928769927518e-05 6 KSP Residual norm 8.464174666739e-06 7 KSP Residual norm 1.583763581407e-06 8 KSP Residual norm 3.251685842746e-07 Linear tao_nls_ solve converged due to CONVERGED_RTOL iterations 8 0 LS Function value: 0.118376, Step length: 0. 1 LS Function value: 0.117884, Step length: 1. stx: 0., fx: 0.118376, dgx: -0.000530697 sty: 0., fy: 0.118376, dgy: -0.000530697 I have attached the full log for this particular solve. The points of negative curvature do not surprise me - this is a difficult problem, with many singularities in the Jacobian/Hessian. In fact, I am very happy with how well STCG is performing, globally. My question is what to make of the poor performance of the linesearch after encountering these points. As I understand it, most of the flags for adjusting the solver after encountering an indefinite matrix involve altering the magnitude of the perturbation by which the Hessian is offset for calculating the update direction. I am not clear on how to adjust the routine for determining the initial step size. More generally, should I expect these points of negative curvature to cause difficulties for the solver, and be satisfied with a large number of iterations before finding an improvement at tiny step size? I have a loosely related question about the status of the Levenberg Marquardt solver within Tao? I remember hearing on the list that this was being worked on, but I am not sure from the branch description whether it is working or intended for general use. I would love to hear any updates on that, because I think it could be useful for my problem. Thanks! -------------- next part -------------- An HTML attachment was scrubbed... URL: From eda.oktay at metu.edu.tr Wed Jun 10 11:07:40 2020 From: eda.oktay at metu.edu.tr (Eda Oktay) Date: Wed, 10 Jun 2020 19:07:40 +0300 Subject: [petsc-users] MatGetRow for global rows of a parallel matrix In-Reply-To: References: Message-ID: Der Matt, When I looked at the results, I found that there are some problems I couldn't understand. First of all, I am working on a 72*4 matrix and as I said before, I want to have 72 different vectors having size 4 each, whose elements consist of the elements in the same row. And of course, all vectors should be in all processors (currently I am using 4 processors). When I use your scatter code, the output vector is divided into 4 parts for 4 processors and each vector consists of 18 row vectors whose elements are arranged in a way that if I want to find zeroth row vector, its elements are located in 0th,18th,36th,54th elements. So, isn't scatter's goal is to scatter all values to all processors? Furthermore, I am trying to use my vectors in that way but isn't there any possible way that I can reach my goal entirely? Thanks so much for your help, Eda Matthew Knepley , 10 Haz 2020 ?ar, 18:11 tarihinde ?unu yazd?: > > On Wed, Jun 10, 2020 at 10:09 AM Eda Oktay wrote: >> >> Dear Matt, >> >> I have one last question I believe. Up to creating a dense matrix I >> did what you've suggested. Thank you so much for that. >> >> I created a new dense matrix. Now, how should I wrap each vector in a >> MatDense again? I mean, what is wrapping vectors in a matrix? To put >> each of them again as rows? > > > I thought you need a dense matrix for something, since you started with one. If you > do not, just do VecGetArray() on the vector from CreateToAll and use the values. > > Thanks, > > Matt > >> >> Thanks! >> >> Eda >> >> Matthew Knepley , 10 Haz 2020 ?ar, 16:16 tarihinde >> ?unu yazd?: >> > >> > On Wed, Jun 10, 2020 at 9:08 AM Eda Oktay wrote: >> >> >> >> Dear Matt, >> >> >> >> Matthew Knepley , 10 Haz 2020 ?ar, 16:03 tarihinde >> >> ?unu yazd?: >> >> > >> >> > On Wed, Jun 10, 2020 at 8:56 AM Eda Oktay wrote: >> >> >> >> >> >> Hi all, >> >> >> >> >> >> I am trying to get all the rows of a parallel matrix as individual >> >> >> vectors. For instance, if I have 72*4 matrix, I want to get 72 >> >> >> different vectors having size 4. >> >> >> >> >> >> As far as I understood, MatGetRow is only for local rows, so >> >> >> MatGetOwnershipRange is used, however, when I tried this one, I >> >> >> couldn't get the whole and desired row vectors. >> >> >> >> >> >> In MatGetRow explanation, it is written that I should use >> >> >> MatCreateSubMatrices first, then use MatGetRow. But I couldn't >> >> >> understand to which extent I should create submatrices. I just need to >> >> >> have all 72 rows as 72 different vectors each having 4 elements. >> >> > >> >> > >> >> > 1) For sparse matrices, the storage is always divided by row, so that values can only be retrieved for local rows with MatGetRow() >> >> > >> >> > 2) Is this matrix sparse? It sounds like it is dense. >> >> >> >> Matrix is dense. >> >> >> >> > >> >> > 3) Are you asking to get all matrix values on all processes? If so, I think the easiest thing to do is first wrap a Vec around the >> >> > values, then use VecScatterToAll(), then wrap each one in a MatDense again. >> >> >> >> Yes, I want all row vectors on all processes. In a dense matrix, >> >> should I still wrap a Vec around the values? I know I should use >> >> scatter but I couldn't even wrap a Vec around them. >> > >> > >> > I would do >> > >> > MatGetSize(&N); >> > MatGetLocalSize(&m); >> > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatDenseGetArray.html >> > >> > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecPlaceArray.html >> > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecScatterCreateToAll.html >> > >> > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecResetArray.html#VecResetArray >> > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatCreateDense.html >> > >> > >> > >> > >> > Thanks, >> > >> > Matt >> > >> >> >> >> Thanks so much! >> >> >> >> Eda >> >> >> >> > >> >> > Thanks, >> >> > >> >> > Matt >> >> > >> >> >> >> >> >> Thanks! >> >> >> >> >> >> Eda >> >> > >> >> > >> >> > >> >> > -- >> >> > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >> >> > -- Norbert Wiener >> >> > >> >> > https://www.cse.buffalo.edu/~knepley/ >> > >> > >> > >> > -- >> > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >> > -- Norbert Wiener >> > >> > https://www.cse.buffalo.edu/~knepley/ > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ From knepley at gmail.com Wed Jun 10 11:12:46 2020 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 10 Jun 2020 12:12:46 -0400 Subject: [petsc-users] MatGetRow for global rows of a parallel matrix In-Reply-To: References: Message-ID: On Wed, Jun 10, 2020 at 12:07 PM Eda Oktay wrote: > Der Matt, > > When I looked at the results, I found that there are some problems I > couldn't understand. > > First of all, I am working on a 72*4 matrix and as I said before, I > want to have 72 different vectors having size 4 each, whose elements > consist of the elements in the same row. And of course, all vectors > should be in all processors (currently I am using 4 processors). > > When I use your scatter code, the output vector is divided into 4 > parts for 4 processors and each vector consists of 18 row vectors > whose elements are arranged in a way that if I want to find zeroth row > vector, its elements are located in 0th,18th,36th,54th elements. > Was the global size of the vector you wrapped around the dense matrix 72*4? If you use CreateToAll(), it will make a vector on each process which has the global size of the original vector. Thanks, Matt > So, isn't scatter's goal is to scatter all values to all processors? > > Furthermore, I am trying to use my vectors in that way but isn't there > any possible way that I can reach my goal entirely? > > Thanks so much for your help, > > Eda > > Matthew Knepley , 10 Haz 2020 ?ar, 18:11 tarihinde > ?unu yazd?: > > > > On Wed, Jun 10, 2020 at 10:09 AM Eda Oktay > wrote: > >> > >> Dear Matt, > >> > >> I have one last question I believe. Up to creating a dense matrix I > >> did what you've suggested. Thank you so much for that. > >> > >> I created a new dense matrix. Now, how should I wrap each vector in a > >> MatDense again? I mean, what is wrapping vectors in a matrix? To put > >> each of them again as rows? > > > > > > I thought you need a dense matrix for something, since you started with > one. If you > > do not, just do VecGetArray() on the vector from CreateToAll and use the > values. > > > > Thanks, > > > > Matt > > > >> > >> Thanks! > >> > >> Eda > >> > >> Matthew Knepley , 10 Haz 2020 ?ar, 16:16 tarihinde > >> ?unu yazd?: > >> > > >> > On Wed, Jun 10, 2020 at 9:08 AM Eda Oktay > wrote: > >> >> > >> >> Dear Matt, > >> >> > >> >> Matthew Knepley , 10 Haz 2020 ?ar, 16:03 > tarihinde > >> >> ?unu yazd?: > >> >> > > >> >> > On Wed, Jun 10, 2020 at 8:56 AM Eda Oktay > wrote: > >> >> >> > >> >> >> Hi all, > >> >> >> > >> >> >> I am trying to get all the rows of a parallel matrix as individual > >> >> >> vectors. For instance, if I have 72*4 matrix, I want to get 72 > >> >> >> different vectors having size 4. > >> >> >> > >> >> >> As far as I understood, MatGetRow is only for local rows, so > >> >> >> MatGetOwnershipRange is used, however, when I tried this one, I > >> >> >> couldn't get the whole and desired row vectors. > >> >> >> > >> >> >> In MatGetRow explanation, it is written that I should use > >> >> >> MatCreateSubMatrices first, then use MatGetRow. But I couldn't > >> >> >> understand to which extent I should create submatrices. I just > need to > >> >> >> have all 72 rows as 72 different vectors each having 4 elements. > >> >> > > >> >> > > >> >> > 1) For sparse matrices, the storage is always divided by row, so > that values can only be retrieved for local rows with MatGetRow() > >> >> > > >> >> > 2) Is this matrix sparse? It sounds like it is dense. > >> >> > >> >> Matrix is dense. > >> >> > >> >> > > >> >> > 3) Are you asking to get all matrix values on all processes? If > so, I think the easiest thing to do is first wrap a Vec around the > >> >> > values, then use VecScatterToAll(), then wrap each one in a > MatDense again. > >> >> > >> >> Yes, I want all row vectors on all processes. In a dense matrix, > >> >> should I still wrap a Vec around the values? I know I should use > >> >> scatter but I couldn't even wrap a Vec around them. > >> > > >> > > >> > I would do > >> > > >> > MatGetSize(&N); > >> > MatGetLocalSize(&m); > >> > > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatDenseGetArray.html > >> > > >> > > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecPlaceArray.html > >> > > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecScatterCreateToAll.html > >> > > >> > > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecResetArray.html#VecResetArray > >> > > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatCreateDense.html > >> > > >> > > >> > > >> > > >> > Thanks, > >> > > >> > Matt > >> > > >> >> > >> >> Thanks so much! > >> >> > >> >> Eda > >> >> > >> >> > > >> >> > Thanks, > >> >> > > >> >> > Matt > >> >> > > >> >> >> > >> >> >> Thanks! > >> >> >> > >> >> >> Eda > >> >> > > >> >> > > >> >> > > >> >> > -- > >> >> > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > >> >> > -- Norbert Wiener > >> >> > > >> >> > https://www.cse.buffalo.edu/~knepley/ > >> > > >> > > >> > > >> > -- > >> > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > >> > -- Norbert Wiener > >> > > >> > https://www.cse.buffalo.edu/~knepley/ > > > > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > > -- Norbert Wiener > > > > https://www.cse.buffalo.edu/~knepley/ > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From eda.oktay at metu.edu.tr Wed Jun 10 11:26:17 2020 From: eda.oktay at metu.edu.tr (Eda Oktay) Date: Wed, 10 Jun 2020 19:26:17 +0300 Subject: [petsc-users] MatGetRow for global rows of a parallel matrix In-Reply-To: References: Message-ID: Matthew Knepley , 10 Haz 2020 ?ar, 19:13 tarihinde ?unu yazd?: > > On Wed, Jun 10, 2020 at 12:07 PM Eda Oktay wrote: >> >> Der Matt, >> >> When I looked at the results, I found that there are some problems I >> couldn't understand. >> >> First of all, I am working on a 72*4 matrix and as I said before, I >> want to have 72 different vectors having size 4 each, whose elements >> consist of the elements in the same row. And of course, all vectors >> should be in all processors (currently I am using 4 processors). >> >> When I use your scatter code, the output vector is divided into 4 >> parts for 4 processors and each vector consists of 18 row vectors >> whose elements are arranged in a way that if I want to find zeroth row >> vector, its elements are located in 0th,18th,36th,54th elements. > > > Was the global size of the vector you wrapped around the dense matrix 72*4? Yes it is. I set up its global size to 72*4. > > If you use CreateToAll(), it will make a vector on each process which has the global size of the original vector. Although I set 72*4, the size of the vectors in each process is 72. Thanks, Eda > > Thanks, > > Matt > >> >> So, isn't scatter's goal is to scatter all values to all processors? >> >> Furthermore, I am trying to use my vectors in that way but isn't there >> any possible way that I can reach my goal entirely? >> >> Thanks so much for your help, >> >> Eda >> >> Matthew Knepley , 10 Haz 2020 ?ar, 18:11 tarihinde >> ?unu yazd?: >> > >> > On Wed, Jun 10, 2020 at 10:09 AM Eda Oktay wrote: >> >> >> >> Dear Matt, >> >> >> >> I have one last question I believe. Up to creating a dense matrix I >> >> did what you've suggested. Thank you so much for that. >> >> >> >> I created a new dense matrix. Now, how should I wrap each vector in a >> >> MatDense again? I mean, what is wrapping vectors in a matrix? To put >> >> each of them again as rows? >> > >> > >> > I thought you need a dense matrix for something, since you started with one. If you >> > do not, just do VecGetArray() on the vector from CreateToAll and use the values. >> > >> > Thanks, >> > >> > Matt >> > >> >> >> >> Thanks! >> >> >> >> Eda >> >> >> >> Matthew Knepley , 10 Haz 2020 ?ar, 16:16 tarihinde >> >> ?unu yazd?: >> >> > >> >> > On Wed, Jun 10, 2020 at 9:08 AM Eda Oktay wrote: >> >> >> >> >> >> Dear Matt, >> >> >> >> >> >> Matthew Knepley , 10 Haz 2020 ?ar, 16:03 tarihinde >> >> >> ?unu yazd?: >> >> >> > >> >> >> > On Wed, Jun 10, 2020 at 8:56 AM Eda Oktay wrote: >> >> >> >> >> >> >> >> Hi all, >> >> >> >> >> >> >> >> I am trying to get all the rows of a parallel matrix as individual >> >> >> >> vectors. For instance, if I have 72*4 matrix, I want to get 72 >> >> >> >> different vectors having size 4. >> >> >> >> >> >> >> >> As far as I understood, MatGetRow is only for local rows, so >> >> >> >> MatGetOwnershipRange is used, however, when I tried this one, I >> >> >> >> couldn't get the whole and desired row vectors. >> >> >> >> >> >> >> >> In MatGetRow explanation, it is written that I should use >> >> >> >> MatCreateSubMatrices first, then use MatGetRow. But I couldn't >> >> >> >> understand to which extent I should create submatrices. I just need to >> >> >> >> have all 72 rows as 72 different vectors each having 4 elements. >> >> >> > >> >> >> > >> >> >> > 1) For sparse matrices, the storage is always divided by row, so that values can only be retrieved for local rows with MatGetRow() >> >> >> > >> >> >> > 2) Is this matrix sparse? It sounds like it is dense. >> >> >> >> >> >> Matrix is dense. >> >> >> >> >> >> > >> >> >> > 3) Are you asking to get all matrix values on all processes? If so, I think the easiest thing to do is first wrap a Vec around the >> >> >> > values, then use VecScatterToAll(), then wrap each one in a MatDense again. >> >> >> >> >> >> Yes, I want all row vectors on all processes. In a dense matrix, >> >> >> should I still wrap a Vec around the values? I know I should use >> >> >> scatter but I couldn't even wrap a Vec around them. >> >> > >> >> > >> >> > I would do >> >> > >> >> > MatGetSize(&N); >> >> > MatGetLocalSize(&m); >> >> > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatDenseGetArray.html >> >> > >> >> > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecPlaceArray.html >> >> > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecScatterCreateToAll.html >> >> > >> >> > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecResetArray.html#VecResetArray >> >> > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatCreateDense.html >> >> > >> >> > >> >> > >> >> > >> >> > Thanks, >> >> > >> >> > Matt >> >> > >> >> >> >> >> >> Thanks so much! >> >> >> >> >> >> Eda >> >> >> >> >> >> > >> >> >> > Thanks, >> >> >> > >> >> >> > Matt >> >> >> > >> >> >> >> >> >> >> >> Thanks! >> >> >> >> >> >> >> >> Eda >> >> >> > >> >> >> > >> >> >> > >> >> >> > -- >> >> >> > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >> >> >> > -- Norbert Wiener >> >> >> > >> >> >> > https://www.cse.buffalo.edu/~knepley/ >> >> > >> >> > >> >> > >> >> > -- >> >> > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >> >> > -- Norbert Wiener >> >> > >> >> > https://www.cse.buffalo.edu/~knepley/ >> > >> > >> > >> > -- >> > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >> > -- Norbert Wiener >> > >> > https://www.cse.buffalo.edu/~knepley/ > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ From knepley at gmail.com Wed Jun 10 11:36:01 2020 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 10 Jun 2020 12:36:01 -0400 Subject: [petsc-users] MatGetRow for global rows of a parallel matrix In-Reply-To: References: Message-ID: On Wed, Jun 10, 2020 at 12:26 PM Eda Oktay wrote: > Matthew Knepley , 10 Haz 2020 ?ar, 19:13 tarihinde > ?unu yazd?: > > > > On Wed, Jun 10, 2020 at 12:07 PM Eda Oktay > wrote: > >> > >> Der Matt, > >> > >> When I looked at the results, I found that there are some problems I > >> couldn't understand. > >> > >> First of all, I am working on a 72*4 matrix and as I said before, I > >> want to have 72 different vectors having size 4 each, whose elements > >> consist of the elements in the same row. And of course, all vectors > >> should be in all processors (currently I am using 4 processors). > >> > >> When I use your scatter code, the output vector is divided into 4 > >> parts for 4 processors and each vector consists of 18 row vectors > >> whose elements are arranged in a way that if I want to find zeroth row > >> vector, its elements are located in 0th,18th,36th,54th elements. > > > > > > Was the global size of the vector you wrapped around the dense matrix > 72*4? > > Yes it is. I set up its global size to 72*4. > > > > > If you use CreateToAll(), it will make a vector on each process which > has the global size of the original vector. > > Although I set 72*4, the size of the vectors in each process is 72. > You can understand how it is hard to accept, as this code is tested every night. Can you VecView() the input vector to CreateToAll and the output vector, and send that output? Thanks, Matt > Thanks, > > Eda > > > > > Thanks, > > > > Matt > > > >> > >> So, isn't scatter's goal is to scatter all values to all processors? > >> > >> Furthermore, I am trying to use my vectors in that way but isn't there > >> any possible way that I can reach my goal entirely? > >> > >> Thanks so much for your help, > >> > >> Eda > >> > >> Matthew Knepley , 10 Haz 2020 ?ar, 18:11 tarihinde > >> ?unu yazd?: > >> > > >> > On Wed, Jun 10, 2020 at 10:09 AM Eda Oktay > wrote: > >> >> > >> >> Dear Matt, > >> >> > >> >> I have one last question I believe. Up to creating a dense matrix I > >> >> did what you've suggested. Thank you so much for that. > >> >> > >> >> I created a new dense matrix. Now, how should I wrap each vector in a > >> >> MatDense again? I mean, what is wrapping vectors in a matrix? To put > >> >> each of them again as rows? > >> > > >> > > >> > I thought you need a dense matrix for something, since you started > with one. If you > >> > do not, just do VecGetArray() on the vector from CreateToAll and use > the values. > >> > > >> > Thanks, > >> > > >> > Matt > >> > > >> >> > >> >> Thanks! > >> >> > >> >> Eda > >> >> > >> >> Matthew Knepley , 10 Haz 2020 ?ar, 16:16 > tarihinde > >> >> ?unu yazd?: > >> >> > > >> >> > On Wed, Jun 10, 2020 at 9:08 AM Eda Oktay > wrote: > >> >> >> > >> >> >> Dear Matt, > >> >> >> > >> >> >> Matthew Knepley , 10 Haz 2020 ?ar, 16:03 > tarihinde > >> >> >> ?unu yazd?: > >> >> >> > > >> >> >> > On Wed, Jun 10, 2020 at 8:56 AM Eda Oktay < > eda.oktay at metu.edu.tr> wrote: > >> >> >> >> > >> >> >> >> Hi all, > >> >> >> >> > >> >> >> >> I am trying to get all the rows of a parallel matrix as > individual > >> >> >> >> vectors. For instance, if I have 72*4 matrix, I want to get 72 > >> >> >> >> different vectors having size 4. > >> >> >> >> > >> >> >> >> As far as I understood, MatGetRow is only for local rows, so > >> >> >> >> MatGetOwnershipRange is used, however, when I tried this one, I > >> >> >> >> couldn't get the whole and desired row vectors. > >> >> >> >> > >> >> >> >> In MatGetRow explanation, it is written that I should use > >> >> >> >> MatCreateSubMatrices first, then use MatGetRow. But I couldn't > >> >> >> >> understand to which extent I should create submatrices. I just > need to > >> >> >> >> have all 72 rows as 72 different vectors each having 4 > elements. > >> >> >> > > >> >> >> > > >> >> >> > 1) For sparse matrices, the storage is always divided by row, > so that values can only be retrieved for local rows with MatGetRow() > >> >> >> > > >> >> >> > 2) Is this matrix sparse? It sounds like it is dense. > >> >> >> > >> >> >> Matrix is dense. > >> >> >> > >> >> >> > > >> >> >> > 3) Are you asking to get all matrix values on all processes? If > so, I think the easiest thing to do is first wrap a Vec around the > >> >> >> > values, then use VecScatterToAll(), then wrap each one in a > MatDense again. > >> >> >> > >> >> >> Yes, I want all row vectors on all processes. In a dense matrix, > >> >> >> should I still wrap a Vec around the values? I know I should use > >> >> >> scatter but I couldn't even wrap a Vec around them. > >> >> > > >> >> > > >> >> > I would do > >> >> > > >> >> > MatGetSize(&N); > >> >> > MatGetLocalSize(&m); > >> >> > > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatDenseGetArray.html > >> >> > > >> >> > > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecPlaceArray.html > >> >> > > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecScatterCreateToAll.html > >> >> > > >> >> > > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecResetArray.html#VecResetArray > >> >> > > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatCreateDense.html > >> >> > > >> >> > > >> >> > > >> >> > > >> >> > Thanks, > >> >> > > >> >> > Matt > >> >> > > >> >> >> > >> >> >> Thanks so much! > >> >> >> > >> >> >> Eda > >> >> >> > >> >> >> > > >> >> >> > Thanks, > >> >> >> > > >> >> >> > Matt > >> >> >> > > >> >> >> >> > >> >> >> >> Thanks! > >> >> >> >> > >> >> >> >> Eda > >> >> >> > > >> >> >> > > >> >> >> > > >> >> >> > -- > >> >> >> > What most experimenters take for granted before they begin > their experiments is infinitely more interesting than any results to which > their experiments lead. > >> >> >> > -- Norbert Wiener > >> >> >> > > >> >> >> > https://www.cse.buffalo.edu/~knepley/ > >> >> > > >> >> > > >> >> > > >> >> > -- > >> >> > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > >> >> > -- Norbert Wiener > >> >> > > >> >> > https://www.cse.buffalo.edu/~knepley/ > >> > > >> > > >> > > >> > -- > >> > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > >> > -- Norbert Wiener > >> > > >> > https://www.cse.buffalo.edu/~knepley/ > > > > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > > -- Norbert Wiener > > > > https://www.cse.buffalo.edu/~knepley/ > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Wed Jun 10 11:59:12 2020 From: bsmith at petsc.dev (Barry Smith) Date: Wed, 10 Jun 2020 11:59:12 -0500 Subject: [petsc-users] MatGetRow for global rows of a parallel matrix In-Reply-To: References: Message-ID: You can use MatCreateSubMatrices() with each process getting a single sequential dense sub matrix that consists of the entire matrix. Use VecDuplicateVecs() to create an array of 72 vectors (create a single seq vector of size 4 as the input to this routine) Then use MatDenseGetArrayRead() to access the upper left corner of the new sequential dense matrix Loop over the vectors calling VecGetArray() Then loop over the row of the dense array filling up the vector Because dense matrices are stored by column, you have to do this looping to fill up the vectors, they can't share the space with the matrix. Barry > On Jun 10, 2020, at 11:36 AM, Matthew Knepley wrote: > > On Wed, Jun 10, 2020 at 12:26 PM Eda Oktay > wrote: > Matthew Knepley >, 10 Haz 2020 ?ar, 19:13 tarihinde > ?unu yazd?: > > > > On Wed, Jun 10, 2020 at 12:07 PM Eda Oktay > wrote: > >> > >> Der Matt, > >> > >> When I looked at the results, I found that there are some problems I > >> couldn't understand. > >> > >> First of all, I am working on a 72*4 matrix and as I said before, I > >> want to have 72 different vectors having size 4 each, whose elements > >> consist of the elements in the same row. And of course, all vectors > >> should be in all processors (currently I am using 4 processors). > >> > >> When I use your scatter code, the output vector is divided into 4 > >> parts for 4 processors and each vector consists of 18 row vectors > >> whose elements are arranged in a way that if I want to find zeroth row > >> vector, its elements are located in 0th,18th,36th,54th elements. > > > > > > Was the global size of the vector you wrapped around the dense matrix 72*4? > > Yes it is. I set up its global size to 72*4. > > > > > If you use CreateToAll(), it will make a vector on each process which has the global size of the original vector. > > Although I set 72*4, the size of the vectors in each process is 72. > > You can understand how it is hard to accept, as this code is tested every night. Can you VecView() the input vector > to CreateToAll and the output vector, and send that output? > > Thanks, > > Matt > > Thanks, > > Eda > > > > > Thanks, > > > > Matt > > > >> > >> So, isn't scatter's goal is to scatter all values to all processors? > >> > >> Furthermore, I am trying to use my vectors in that way but isn't there > >> any possible way that I can reach my goal entirely? > >> > >> Thanks so much for your help, > >> > >> Eda > >> > >> Matthew Knepley >, 10 Haz 2020 ?ar, 18:11 tarihinde > >> ?unu yazd?: > >> > > >> > On Wed, Jun 10, 2020 at 10:09 AM Eda Oktay > wrote: > >> >> > >> >> Dear Matt, > >> >> > >> >> I have one last question I believe. Up to creating a dense matrix I > >> >> did what you've suggested. Thank you so much for that. > >> >> > >> >> I created a new dense matrix. Now, how should I wrap each vector in a > >> >> MatDense again? I mean, what is wrapping vectors in a matrix? To put > >> >> each of them again as rows? > >> > > >> > > >> > I thought you need a dense matrix for something, since you started with one. If you > >> > do not, just do VecGetArray() on the vector from CreateToAll and use the values. > >> > > >> > Thanks, > >> > > >> > Matt > >> > > >> >> > >> >> Thanks! > >> >> > >> >> Eda > >> >> > >> >> Matthew Knepley >, 10 Haz 2020 ?ar, 16:16 tarihinde > >> >> ?unu yazd?: > >> >> > > >> >> > On Wed, Jun 10, 2020 at 9:08 AM Eda Oktay > wrote: > >> >> >> > >> >> >> Dear Matt, > >> >> >> > >> >> >> Matthew Knepley >, 10 Haz 2020 ?ar, 16:03 tarihinde > >> >> >> ?unu yazd?: > >> >> >> > > >> >> >> > On Wed, Jun 10, 2020 at 8:56 AM Eda Oktay > wrote: > >> >> >> >> > >> >> >> >> Hi all, > >> >> >> >> > >> >> >> >> I am trying to get all the rows of a parallel matrix as individual > >> >> >> >> vectors. For instance, if I have 72*4 matrix, I want to get 72 > >> >> >> >> different vectors having size 4. > >> >> >> >> > >> >> >> >> As far as I understood, MatGetRow is only for local rows, so > >> >> >> >> MatGetOwnershipRange is used, however, when I tried this one, I > >> >> >> >> couldn't get the whole and desired row vectors. > >> >> >> >> > >> >> >> >> In MatGetRow explanation, it is written that I should use > >> >> >> >> MatCreateSubMatrices first, then use MatGetRow. But I couldn't > >> >> >> >> understand to which extent I should create submatrices. I just need to > >> >> >> >> have all 72 rows as 72 different vectors each having 4 elements. > >> >> >> > > >> >> >> > > >> >> >> > 1) For sparse matrices, the storage is always divided by row, so that values can only be retrieved for local rows with MatGetRow() > >> >> >> > > >> >> >> > 2) Is this matrix sparse? It sounds like it is dense. > >> >> >> > >> >> >> Matrix is dense. > >> >> >> > >> >> >> > > >> >> >> > 3) Are you asking to get all matrix values on all processes? If so, I think the easiest thing to do is first wrap a Vec around the > >> >> >> > values, then use VecScatterToAll(), then wrap each one in a MatDense again. > >> >> >> > >> >> >> Yes, I want all row vectors on all processes. In a dense matrix, > >> >> >> should I still wrap a Vec around the values? I know I should use > >> >> >> scatter but I couldn't even wrap a Vec around them. > >> >> > > >> >> > > >> >> > I would do > >> >> > > >> >> > MatGetSize(&N); > >> >> > MatGetLocalSize(&m); > >> >> > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatDenseGetArray.html > >> >> > > >> >> > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecPlaceArray.html > >> >> > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecScatterCreateToAll.html > >> >> > > >> >> > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecResetArray.html#VecResetArray > >> >> > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatCreateDense.html > >> >> > > >> >> > > >> >> > > >> >> > > >> >> > Thanks, > >> >> > > >> >> > Matt > >> >> > > >> >> >> > >> >> >> Thanks so much! > >> >> >> > >> >> >> Eda > >> >> >> > >> >> >> > > >> >> >> > Thanks, > >> >> >> > > >> >> >> > Matt > >> >> >> > > >> >> >> >> > >> >> >> >> Thanks! > >> >> >> >> > >> >> >> >> Eda > >> >> >> > > >> >> >> > > >> >> >> > > >> >> >> > -- > >> >> >> > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > >> >> >> > -- Norbert Wiener > >> >> >> > > >> >> >> > https://www.cse.buffalo.edu/~knepley/ > >> >> > > >> >> > > >> >> > > >> >> > -- > >> >> > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > >> >> > -- Norbert Wiener > >> >> > > >> >> > https://www.cse.buffalo.edu/~knepley/ > >> > > >> > > >> > > >> > -- > >> > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > >> > -- Norbert Wiener > >> > > >> > https://www.cse.buffalo.edu/~knepley/ > > > > > > > > -- > > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > > -- Norbert Wiener > > > > https://www.cse.buffalo.edu/~knepley/ > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From adener at anl.gov Wed Jun 10 14:10:23 2020 From: adener at anl.gov (Dener, Alp) Date: Wed, 10 Jun 2020 19:10:23 +0000 Subject: [petsc-users] TAO STCG initial perturbation In-Reply-To: References: Message-ID: <2A5F02F9-9218-40A0-9A1F-3004E5A5E279@anl.gov> Hi Zak, You got it right with the TaoBRGNGetSubsolver -> TaoGetKSP workflow. This will get you the KSP object correctly. BRGN is not a stand-alone solver. It?s a wrapper that combines the user-provided residual and Jacobian callbacks to assemble the gradient and Hessian under the Gauss-Newton formulation, and feed that information into a standard TAO solver like NLS. It also incorporates some regularizations to the problem too. Technically any user could also do this themselves and directly use NLS to solve a least squares problem, but BRGN reduces the workload a little bit. To use it for your least-squares problem, you would need to set TaoType() to TAOBRGN and then provide the Tao solver two callbacks: one for the residual using TaoSetResidualRoutine() and another for the Jacobian using TaoSetJacobianResidualRoutine(). The regularization can be changed using -tao_brgn_regularization_type command line options. Note that the BRGN sub solver can also be changed. It?s set to BNLS by default ? this is bound constrained Newton line search but it also solves unconstrained problems when no bounds are defined. You could always try to use BNTR for a trust-region method instead of line search, or switch to BNQLS to do a quasi-Gauss-Newton solution. The sub solver?s settings can be accessed via the -tao_brgn_subsolver_ prefix. So for instance if you wanted to change type, you could do "-tao_brgn_subsolver_tao_type bntr" for Newton trust-region. Alp On Jun 10, 2020, at 1:55 PM, zakaryah . > wrote: Oh, maybe I can answer my own question - for BRGN, is it the subsolver that has a KSP? Then I would call TaoBRGNGetSubsolver(tao, &subsolver); TaoGetKSP(subsolver,&ksp); ? On Wed, Jun 10, 2020 at 2:52 PM zakaryah . > wrote: Hi Alp, Thanks very much for this thorough answer. I understand the situation much better now. I will experiment with quasi-Newton methods - this should be straightforward to test and evaluate. I agree that the regularized Gauss-Newton may be useful in place of the Levenberg Marquardt per se. I am having some trouble understanding how it is set up and how to use it for my purposes. Using BRGN, does the Tao solver itself not have a KSP? After setting up the solver, running TaoGetKSP returns a NULL KSP. It seems I am doing something wrong, but with other Tao types, like nls, everything seems to work fine. Thanks for your help! Cheers, Zak On Wed, Jun 10, 2020 at 11:29 AM Dener, Alp > wrote: Hello, STCG is being used to compute a search direction by inverting the Hessian of the objective onto the gradient. The Hessian has to be positive definitive for this search direction to be a valid descent direction. To enforce this, STCG terminates the KSP solution when it encounters negative curvature (i.e. ?uphill? directions). The resulting search direction from this early termination is still a descent direction but it?s a pretty bad one, meaning that the line search is forced to do extra work to find a valid (and sometimes pretty small) step length. This is normal. Once a negative curvature is detected, the NLS algorithm imposes a shift to the diagonal of the Hessian in order to guarantee that it will be positive-definite on the next optimization iteration. This diagonal shift is increased in magnitude if you keep hitting the negative curvature KSP termination, and it?s gradually decreased on iterations where the KSP solution succeeds. However this does not prevent the line search from doing all that extra work on iterations where the search direction solution never converged fully. It simply helps generate better search directions in subsequent iterations and helps the overall optimization solution eventually converge to a valid minimum. As a side note, if your function, gradient and/or Hessian evaluations are expensive, it might actually make more sense to use the quasi-Newton method (BQNLS) instead of Newton line search. The optimization solution is likely to take more overall iterations with BQNLS(-tao_type bqnls), but the BFGS approximation used by the algorithm is inherently guaranteed to be positive-definite and tends to generate very well scaled search directions so it might help the line search do less work. Combined with the elimination of Hessian evaluations, it may very well solve your problem faster in CPU time even if it takes more nonlinear iterations overall. This is technically a bound constrained method but it solves unconstrained problems too when you don?t define any bounds. About Levenberg-Marquardt: a user started the branch to eventually contribute an LM solver, but I have not heard any updates on it since end of April. For least-squares type problems, you can try using the regularized Gauss-Newton solver (-tao_type brgn). The problem definition interface is a bit different. BRGN requires the problem to be defined as a residual and its Jacobian, and it will assemble the gradient and the Hessian on its own and feed it into the standard Newton line search solver underneath. There are also a few different regularization options available, like proximal point Tikhonov (l2prox) and a sparsity term (l1dict) that can be set with the (-tao_brgn_regularization_type) option. These get automatically cooked into the gradient and Hessian. Combined with the automatic diagonal shifts for the Hessian, BRGN really is almost equivalent to an LM solver so it might do what you need. Hope this helps! ? Alp On Jun 9, 2020, at 10:55 PM, zakaryah . > wrote: I am using TAO to minimize the elastic strain energy for somewhat complicated applied forces. With Newton linesearch (NLS), the STCG KSP, and several preconditioners (none, Jacobi, lmvm), the solver finds a minimum within an acceptable number of iterations (<50). Still, in the interest of performance, I am wondering about what happens after the KSP encounters an indefinite matrix (KSP_CONVERGED_CG_NEG_CURVE). After this happens, the line search performs extremely poorly, with function values at step length 1 reaching absurdly large values. As the line search tries smaller step lengths, the function values fall, then typically reach values representing a decline from the previous iteration, but only with tiny step lengths (~1e-11). The line search takes many iterations to arrive at such an improvement. Here is an example, run with -tao_type nls -tao_nls_ksp_type stcg -tao_nls_pc_type none -tao_nls_ksp_norm_type unpreconditioned: 0 TAO, Function value: 0.160612, Residual: 0.0736074 0 TAO, Function value: 0.121568, Residual: 0.0424117 Residual norms for tao_nls_ solve. 0 KSP Residual norm 4.241174778983e-02 1 KSP Residual norm 5.806891169509e-02 2 KSP Residual norm 6.492953448014e-02 3 KSP Residual norm 5.856559430984e-02 4 KSP Residual norm 5.262548559710e-02 5 KSP Residual norm 4.863633473400e-02 6 KSP Residual norm 4.725156347026e-02 7 KSP Residual norm 4.748458009319e-02 8 KSP Residual norm 4.885339641711e-02 9 KSP Residual norm 5.065071226354e-02 10 KSP Residual norm 5.085544070851e-02 11 KSP Residual norm 5.127093547976e-02 12 KSP Residual norm 5.155225884843e-02 13 KSP Residual norm 5.219895021408e-02 14 KSP Residual norm 6.480520610077e-02 15 KSP Residual norm 1.433515456621e-01 Linear tao_nls_ solve converged due to CONVERGED_CG_NEG_CURVE iterations 16 0 LS Function value: 0.121568, Step length: 0. 1 LS Function value: 5.99839e+59, Step length: 1. stx: 0., fx: 0.121568, dgx: -1.32893e+08 sty: 0., fy: 0.121568, dgy: -1.32893e+08 2 LS Function value: 1.46445e+56, Step length: 0.25 stx: 0., fx: 0.121568, dgx: -1.32893e+08 sty: 1., fy: 5.99839e+59, dgy: 3.59903e+60 3 LS Function value: 3.57532e+52, Step length: 0.0625 stx: 0., fx: 0.121568, dgx: -1.32893e+08 sty: 0.25, fy: 1.46445e+56, dgy: 3.51468e+57 4 LS Function value: 8.7288e+48, Step length: 0.015625 stx: 0., fx: 0.121568, dgx: -1.32893e+08 sty: 0.0625, fy: 3.57532e+52, dgy: 3.4323e+54 5 LS Function value: 2.13105e+45, Step length: 0.00390625 stx: 0., fx: 0.121568, dgx: -1.32893e+08 sty: 0.015625, fy: 8.7288e+48, dgy: 3.35186e+51 6 LS Function value: 5.20277e+41, Step length: 0.000976562 stx: 0., fx: 0.121568, dgx: -1.32893e+08 sty: 0.00390625, fy: 2.13105e+45, dgy: 3.2733e+48 7 LS Function value: 1.27021e+38, Step length: 0.000244141 stx: 0., fx: 0.121568, dgx: -1.32893e+08 sty: 0.000976562, fy: 5.20277e+41, dgy: 3.19658e+45 8 LS Function value: 3.1011e+34, Step length: 6.10352e-05 stx: 0., fx: 0.121568, dgx: -1.32893e+08 sty: 0.000244141, fy: 1.27021e+38, dgy: 3.12166e+42 9 LS Function value: 7.57106e+30, Step length: 1.52588e-05 stx: 0., fx: 0.121568, dgx: -1.32893e+08 sty: 6.10352e-05, fy: 3.1011e+34, dgy: 3.0485e+39 10 LS Function value: 1.84842e+27, Step length: 3.8147e-06 stx: 0., fx: 0.121568, dgx: -1.32893e+08 sty: 1.52588e-05, fy: 7.57106e+30, dgy: 2.97706e+36 11 LS Function value: 4.51295e+23, Step length: 9.53672e-07 stx: 0., fx: 0.121568, dgx: -1.32893e+08 sty: 3.8147e-06, fy: 1.84842e+27, dgy: 2.90731e+33 12 LS Function value: 1.10199e+20, Step length: 2.38417e-07 stx: 0., fx: 0.121568, dgx: -1.32893e+08 sty: 9.53672e-07, fy: 4.51295e+23, dgy: 2.83927e+30 13 LS Function value: 2.69231e+16, Step length: 5.96028e-08 stx: 0., fx: 0.121568, dgx: -1.32893e+08 sty: 2.38417e-07, fy: 1.10199e+20, dgy: 2.77314e+27 14 LS Function value: 6.59192e+12, Step length: 1.48993e-08 stx: 0., fx: 0.121568, dgx: -1.32893e+08 sty: 5.96028e-08, fy: 2.69231e+16, dgy: 2.70974e+24 15 LS Function value: 1.62897e+09, Step length: 3.72338e-09 stx: 0., fx: 0.121568, dgx: -1.32893e+08 sty: 1.48993e-08, fy: 6.59192e+12, dgy: 2.65254e+21 16 LS Function value: 421305., Step length: 9.29291e-10 stx: 0., fx: 0.121568, dgx: -1.32893e+08 sty: 3.72338e-09, fy: 1.62897e+09, dgy: 2.61626e+18 17 LS Function value: 141.939, Step length: 2.30343e-10 stx: 0., fx: 0.121568, dgx: -1.32893e+08 sty: 9.29291e-10, fy: 421305., dgy: 2.67496e+15 18 LS Function value: 0.213621, Step length: 5.45712e-11 stx: 0., fx: 0.121568, dgx: -1.32893e+08 sty: 2.30343e-10, fy: 141.939, dgy: 3.35874e+12 19 LS Function value: 0.120058, Step length: 1.32286e-11 stx: 0., fx: 0.121568, dgx: -1.32893e+08 sty: 5.45712e-11, fy: 0.213621, dgy: 8.60977e+09 1 TAO, Function value: 0.120058, Residual: 0.0484495 When the KSP does not encounter negative curvature, the linesearch performs well: 3 TAO, Function value: 0.118376, Residual: 0.0545446 Residual norms for tao_nls_ solve. 0 KSP Residual norm 5.454463134947e-02 1 KSP Residual norm 2.394277461960e-02 2 KSP Residual norm 6.674182971627e-03 3 KSP Residual norm 1.235116278289e-03 4 KSP Residual norm 1.714792759324e-04 5 KSP Residual norm 3.928769927518e-05 6 KSP Residual norm 8.464174666739e-06 7 KSP Residual norm 1.583763581407e-06 8 KSP Residual norm 3.251685842746e-07 Linear tao_nls_ solve converged due to CONVERGED_RTOL iterations 8 0 LS Function value: 0.118376, Step length: 0. 1 LS Function value: 0.117884, Step length: 1. stx: 0., fx: 0.118376, dgx: -0.000530697 sty: 0., fy: 0.118376, dgy: -0.000530697 I have attached the full log for this particular solve. The points of negative curvature do not surprise me - this is a difficult problem, with many singularities in the Jacobian/Hessian. In fact, I am very happy with how well STCG is performing, globally. My question is what to make of the poor performance of the linesearch after encountering these points. As I understand it, most of the flags for adjusting the solver after encountering an indefinite matrix involve altering the magnitude of the perturbation by which the Hessian is offset for calculating the update direction. I am not clear on how to adjust the routine for determining the initial step size. More generally, should I expect these points of negative curvature to cause difficulties for the solver, and be satisfied with a large number of iterations before finding an improvement at tiny step size? I have a loosely related question about the status of the Levenberg Marquardt solver within Tao? I remember hearing on the list that this was being worked on, but I am not sure from the branch description whether it is working or intended for general use. I would love to hear any updates on that, because I think it could be useful for my problem. Thanks! -------------- next part -------------- An HTML attachment was scrubbed... URL: From danyang.su at gmail.com Thu Jun 11 00:58:53 2020 From: danyang.su at gmail.com (Danyang Su) Date: Wed, 10 Jun 2020 22:58:53 -0700 Subject: [petsc-users] Parallel writing in HDF5-1.12.0 when some processors have no data to write Message-ID: Hi All, After updating to HDF5-1.12.0, I got some problem if some processors have no data to write or not necessary to write. Since parallel writing is collective, I cannot disable those processors from writing. For the old version, there seems no such problem. So far, the problem only occurs on Linux using GNU compiler. The same code has no problem using intel compiler or latest gnu compiler on MacOS. Looks like it is caused by zero memory space. However, as documented in -------------- next part -------------- An HTML attachment was scrubbed... URL: From danyang.su at gmail.com Thu Jun 11 01:05:35 2020 From: danyang.su at gmail.com (Danyang Su) Date: Wed, 10 Jun 2020 23:05:35 -0700 Subject: [petsc-users] FW: Parallel writing in HDF5-1.12.0 when some processors have no data to write In-Reply-To: References: Message-ID: Hi All, Sorry to send the previous incomplete email accidentally. After updating to HDF5-1.12.0, I got some problem if some processors have no data to write or not necessary to write. Since parallel writing is collective, I cannot disable those processors from writing. For the old version, there seems no such problem. So far, the problem only occurs on Linux using GNU compiler. The same code has no problem using intel compiler or latest gnu compiler on MacOS. I have already included h5sselect_none in the code for those processors without data. But it does not take effect. The problem is documented in the following link (How do you write data when one process doesn't have or need to write data ?). https://support.hdfgroup.org/HDF5/hdf5-quest.html#par-nodata Similar problem has also been reported on HDF Forum by others. https://forum.hdfgroup.org/t/bug-on-hdf5-1-12-0-fortran-parallel/6864 Any suggestion for that? Thanks, Danyang -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Thu Jun 11 07:24:14 2020 From: mfadams at lbl.gov (Mark Adams) Date: Thu, 11 Jun 2020 08:24:14 -0400 Subject: [petsc-users] A series of GPU questions In-Reply-To: References: <4AD81243-3693-455E-B6D7-708BCDBFFBE5@epcc.ed.ac.uk> Message-ID: > > >> >> Would we instead just have 40 (or perhaps slightly fewer) MPI processes >> all sharing the GPUs? Surely this would be inefficient, and would PETSc >> distribute the work across all 4 GPUs, or would every process end out using >> a single GPU? >> > See > https://docs.olcf.ornl.gov/systems/summit_user_guide.html#volta-multi-process-service. > > I'll jump in here but I would recommend not worrying about the number of GPUs and MPI processes (and don't bother with OMP). As the MPS link above shows MPS wants to allow for slicing/scheduling the GPU in space and/or time. That is, very flexible. One would hope that this will get better and adapt to new hardware and so that your code does not have to. I would focus on getting as much parallelism in your code as possible. GPUs need a lot of threads to run well and with DNS you might have a chance to feed it properly, but I'd just try to get as much as you can. > In some cases, we did see better performance with multiple mpi ranks/GPU > than 1 rank/GPU. The optimal configuration depends on the code. Think two > extremes: One code with work done all on GPU and the other all on CPU. > Probably you only need 1 mpi rank/node for the former, but full ranks for > the latter. > Another dimension is assuming all work is on the GPU, at least asymptotically, then it's a matter of how much parallelism you have. (OK, not that simple ...) At one extreme you have one giant GPU, in which case you probably want to use multiple ranks and hope MPS can slice the GPU up in space to make it look like multiple GPUs of the right size for me. Anecdotally, I have a kernel that is a solver in velocity space that sits in a phase space application (configuration space X and velocity space V) with a tensor decomposition (so the solves between X and V are not coupled). My V space solver is expensive (maybe like complex chemistry in DNS that is independent of the spacial solver) and on smallish problems (less parallelism available) I see an increase of throughput of 5x in going from 1 to 7 cores (MPI ranks) / GPU (IBM/Nvidia, 42 cores and 6 GPU, per node), just running the same problem, embarrassingly parallel. I increased the problem work by 16x and I still got 3x throughput speedup going from 1 to 7 cores. -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Thu Jun 11 07:58:28 2020 From: bsmith at petsc.dev (Barry Smith) Date: Thu, 11 Jun 2020 07:58:28 -0500 Subject: [petsc-users] Parallel writing in HDF5-1.12.0 when some processors have no data to write In-Reply-To: References: Message-ID: Are you making HDF5 calls that fail or is it PETSc routines calling HDF5 that fail? Regardless it sounds like the easiest fix is to switch back to the previous HDF5 and wait for HDF5 to fix what sounds to be a bug. Barry > On Jun 11, 2020, at 1:05 AM, Danyang Su wrote: > > Hi All, > > Sorry to send the previous incomplete email accidentally. > > After updating to HDF5-1.12.0, I got some problem if some processors have no data to write or not necessary to write. Since parallel writing is collective, I cannot disable those processors from writing. For the old version, there seems no such problem. So far, the problem only occurs on Linux using GNU compiler. The same code has no problem using intel compiler or latest gnu compiler on MacOS. > > I have already included h5sselect_none in the code for those processors without data. But it does not take effect. The problem is documented in the following link (How do you write data when one process doesn't have or need to write data ?). > https://support.hdfgroup.org/HDF5/hdf5-quest.html#par-nodata > > Similar problem has also been reported on HDF Forum by others. > https://forum.hdfgroup.org/t/bug-on-hdf5-1-12-0-fortran-parallel/6864 > > Any suggestion for that? > > Thanks, > > Danyang -------------- next part -------------- An HTML attachment was scrubbed... URL: From karl.linkui at gmail.com Thu Jun 11 10:49:54 2020 From: karl.linkui at gmail.com (Karl Lin) Date: Thu, 11 Jun 2020 10:49:54 -0500 Subject: [petsc-users] matcreate and assembly issue Message-ID: Hi, there We have written a program using Petsc to solve large sparse matrix system. It has been working fine for a while. Recently we encountered a problem when the size of the sparse matrix is larger than 10TB. We used several hundred nodes and 2200 processes. The program always crashes during MatAssemblyBegin.Upon a closer look, there seems to be something unusual. We have a little memory check during loading the matrix to keep track of rss. The printout of rss in the log shows normal increase up to rank 2160, i.e., if we load in a portion of matrix that is 1GB, after MatSetValues for that portion, rss will increase roughly about that number. From rank 2161 onwards, the rss in every rank doesn't increase after matrix loaded. Then comes MatAssemblyBegin, the program crashed on rank 2160. Is there a upper limit on the number of processes Petsc can handle? or is there a upper limit in terms of the size of the matrix petsc can handle? Thank you very much for any info. Regards, Karl -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Jun 11 10:56:14 2020 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 11 Jun 2020 11:56:14 -0400 Subject: [petsc-users] matcreate and assembly issue In-Reply-To: References: Message-ID: On Thu, Jun 11, 2020 at 11:51 AM Karl Lin wrote: > Hi, there > > We have written a program using Petsc to solve large sparse matrix system. > It has been working fine for a while. Recently we encountered a problem > when the size of the sparse matrix is larger than 10TB. We used several > hundred nodes and 2200 processes. The program always crashes during > MatAssemblyBegin.Upon a closer look, there seems to be something unusual. > We have a little memory check during loading the matrix to keep track of > rss. The printout of rss in the log shows normal increase up to rank 2160, > i.e., if we load in a portion of matrix that is 1GB, after MatSetValues for > that portion, rss will increase roughly about that number. From rank 2161 > onwards, the rss in every rank doesn't increase after matrix loaded. Then > comes MatAssemblyBegin, the program crashed on rank 2160. > > Is there a upper limit on the number of processes Petsc can handle? or is > there a upper limit in terms of the size of the matrix petsc can handle? > Thank you very much for any info. > It sounds like you overflowed int somewhere. We try and check for this, but catching every place is hard. Try reconfiguring with --with-64-bit-indices Thanks, Matt > Regards, > > Karl > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From karl.linkui at gmail.com Thu Jun 11 11:51:59 2020 From: karl.linkui at gmail.com (Karl Lin) Date: Thu, 11 Jun 2020 11:51:59 -0500 Subject: [petsc-users] matcreate and assembly issue In-Reply-To: References: Message-ID: Hi, Matthew Thanks for the suggestion, just did another run and here are some detailed stack traces, maybe will provide some more insight: *** Process received signal *** Signal: Aborted (6) Signal code: (-6) /lib64/libpthread.so.0(+0xf5f0)[0x2b56c46dc5f0] [ 1] /lib64/libc.so.6(gsignal+0x37)[0x2b56c5486337] [ 2] /lib64/libc.so.6(abort+0x148)[0x2b56c5487a28] [ 3] /libpetsc.so.3.10(PetscTraceBackErrorHandler+0xc4)[0x2b56c1e6a2d4] [ 4] /libpetsc.so.3.10(PetscError+0x1b5)[0x2b56c1e69f65] [ 5] /libpetsc.so.3.10(PetscCommBuildTwoSidedFReq+0x19f0)[0x2b56c1e03cf0] [ 6] /libpetsc.so.3.10(+0x77db17)[0x2b56c2425b17] [ 7] /libpetsc.so.3.10(+0x77a164)[0x2b56c2422164] [ 8] /libpetsc.so.3.10(MatAssemblyBegin_MPIAIJ+0x36)[0x2b56c23912b6] [ 9] /libpetsc.so.3.10(MatAssemblyBegin+0xca)[0x2b56c1feccda] By reconfiguring, you mean recompiling petsc with that option, correct? Thank you. Karl On Thu, Jun 11, 2020 at 10:56 AM Matthew Knepley wrote: > On Thu, Jun 11, 2020 at 11:51 AM Karl Lin wrote: > >> Hi, there >> >> We have written a program using Petsc to solve large sparse matrix >> system. It has been working fine for a while. Recently we encountered a >> problem when the size of the sparse matrix is larger than 10TB. We used >> several hundred nodes and 2200 processes. The program always crashes during >> MatAssemblyBegin.Upon a closer look, there seems to be something unusual. >> We have a little memory check during loading the matrix to keep track of >> rss. The printout of rss in the log shows normal increase up to rank 2160, >> i.e., if we load in a portion of matrix that is 1GB, after MatSetValues for >> that portion, rss will increase roughly about that number. From rank 2161 >> onwards, the rss in every rank doesn't increase after matrix loaded. Then >> comes MatAssemblyBegin, the program crashed on rank 2160. >> >> Is there a upper limit on the number of processes Petsc can handle? or is >> there a upper limit in terms of the size of the matrix petsc can handle? >> Thank you very much for any info. >> > > It sounds like you overflowed int somewhere. We try and check for this, > but catching every place is hard. Try reconfiguring with > > --with-64-bit-indices > > Thanks, > > Matt > > >> Regards, >> >> Karl >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Thu Jun 11 12:00:08 2020 From: balay at mcs.anl.gov (Satish Balay) Date: Thu, 11 Jun 2020 12:00:08 -0500 (CDT) Subject: [petsc-users] matcreate and assembly issue In-Reply-To: References: Message-ID: On Thu, 11 Jun 2020, Karl Lin wrote: > Hi, Matthew > > Thanks for the suggestion, just did another run and here are some detailed > stack traces, maybe will provide some more insight: > *** Process received signal *** > Signal: Aborted (6) > Signal code: (-6) > /lib64/libpthread.so.0(+0xf5f0)[0x2b56c46dc5f0] > [ 1] /lib64/libc.so.6(gsignal+0x37)[0x2b56c5486337] > [ 2] /lib64/libc.so.6(abort+0x148)[0x2b56c5487a28] > [ 3] /libpetsc.so.3.10(PetscTraceBackErrorHandler+0xc4)[0x2b56c1e6a2d4] > [ 4] /libpetsc.so.3.10(PetscError+0x1b5)[0x2b56c1e69f65] > [ 5] /libpetsc.so.3.10(PetscCommBuildTwoSidedFReq+0x19f0)[0x2b56c1e03cf0] > [ 6] /libpetsc.so.3.10(+0x77db17)[0x2b56c2425b17] > [ 7] /libpetsc.so.3.10(+0x77a164)[0x2b56c2422164] > [ 8] /libpetsc.so.3.10(MatAssemblyBegin_MPIAIJ+0x36)[0x2b56c23912b6] > [ 9] /libpetsc.so.3.10(MatAssemblyBegin+0xca)[0x2b56c1feccda] > > By reconfiguring, you mean recompiling petsc with that option, correct? yes. you can use a different PETSC_ARCH for this build - so that both builds are usable [by just switching PETSC_ARCH from your appliation makefile] Satish > > Thank you. > > Karl > > On Thu, Jun 11, 2020 at 10:56 AM Matthew Knepley wrote: > > > On Thu, Jun 11, 2020 at 11:51 AM Karl Lin wrote: > > > >> Hi, there > >> > >> We have written a program using Petsc to solve large sparse matrix > >> system. It has been working fine for a while. Recently we encountered a > >> problem when the size of the sparse matrix is larger than 10TB. We used > >> several hundred nodes and 2200 processes. The program always crashes during > >> MatAssemblyBegin.Upon a closer look, there seems to be something unusual. > >> We have a little memory check during loading the matrix to keep track of > >> rss. The printout of rss in the log shows normal increase up to rank 2160, > >> i.e., if we load in a portion of matrix that is 1GB, after MatSetValues for > >> that portion, rss will increase roughly about that number. From rank 2161 > >> onwards, the rss in every rank doesn't increase after matrix loaded. Then > >> comes MatAssemblyBegin, the program crashed on rank 2160. > >> > >> Is there a upper limit on the number of processes Petsc can handle? or is > >> there a upper limit in terms of the size of the matrix petsc can handle? > >> Thank you very much for any info. > >> > > > > It sounds like you overflowed int somewhere. We try and check for this, > > but catching every place is hard. Try reconfiguring with > > > > --with-64-bit-indices > > > > Thanks, > > > > Matt > > > > > >> Regards, > >> > >> Karl > >> > > > > > > -- > > What most experimenters take for granted before they begin their > > experiments is infinitely more interesting than any results to which their > > experiments lead. > > -- Norbert Wiener > > > > https://www.cse.buffalo.edu/~knepley/ > > > > > From knepley at gmail.com Thu Jun 11 12:00:09 2020 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 11 Jun 2020 13:00:09 -0400 Subject: [petsc-users] matcreate and assembly issue In-Reply-To: References: Message-ID: On Thu, Jun 11, 2020 at 12:52 PM Karl Lin wrote: > Hi, Matthew > > Thanks for the suggestion, just did another run and here are some detailed > stack traces, maybe will provide some more insight: > *** Process received signal *** > Signal: Aborted (6) > Signal code: (-6) > /lib64/libpthread.so.0(+0xf5f0)[0x2b56c46dc5f0] > [ 1] /lib64/libc.so.6(gsignal+0x37)[0x2b56c5486337] > [ 2] /lib64/libc.so.6(abort+0x148)[0x2b56c5487a28] > [ 3] /libpetsc.so.3.10(PetscTraceBackErrorHandler+0xc4)[0x2b56c1e6a2d4] > [ 4] /libpetsc.so.3.10(PetscError+0x1b5)[0x2b56c1e69f65] > [ 5] /libpetsc.so.3.10(PetscCommBuildTwoSidedFReq+0x19f0)[0x2b56c1e03cf0] > [ 6] /libpetsc.so.3.10(+0x77db17)[0x2b56c2425b17] > [ 7] /libpetsc.so.3.10(+0x77a164)[0x2b56c2422164] > [ 8] /libpetsc.so.3.10(MatAssemblyBegin_MPIAIJ+0x36)[0x2b56c23912b6] > [ 9] /libpetsc.so.3.10(MatAssemblyBegin+0xca)[0x2b56c1feccda] > > By reconfiguring, you mean recompiling petsc with that option, correct? > Reconfiguring. Thanks, Matt > Thank you. > > Karl > > On Thu, Jun 11, 2020 at 10:56 AM Matthew Knepley > wrote: > >> On Thu, Jun 11, 2020 at 11:51 AM Karl Lin wrote: >> >>> Hi, there >>> >>> We have written a program using Petsc to solve large sparse matrix >>> system. It has been working fine for a while. Recently we encountered a >>> problem when the size of the sparse matrix is larger than 10TB. We used >>> several hundred nodes and 2200 processes. The program always crashes during >>> MatAssemblyBegin.Upon a closer look, there seems to be something unusual. >>> We have a little memory check during loading the matrix to keep track of >>> rss. The printout of rss in the log shows normal increase up to rank 2160, >>> i.e., if we load in a portion of matrix that is 1GB, after MatSetValues for >>> that portion, rss will increase roughly about that number. From rank 2161 >>> onwards, the rss in every rank doesn't increase after matrix loaded. Then >>> comes MatAssemblyBegin, the program crashed on rank 2160. >>> >>> Is there a upper limit on the number of processes Petsc can handle? >>> or is there a upper limit in terms of the size of the matrix petsc can >>> handle? Thank you very much for any info. >>> >> >> It sounds like you overflowed int somewhere. We try and check for this, >> but catching every place is hard. Try reconfiguring with >> >> --with-64-bit-indices >> >> Thanks, >> >> Matt >> >> >>> Regards, >>> >>> Karl >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From zakaryah at gmail.com Thu Jun 11 12:10:07 2020 From: zakaryah at gmail.com (zakaryah .) Date: Thu, 11 Jun 2020 13:10:07 -0400 Subject: [petsc-users] TAO STCG initial perturbation In-Reply-To: <2A5F02F9-9218-40A0-9A1F-3004E5A5E279@anl.gov> References: <2A5F02F9-9218-40A0-9A1F-3004E5A5E279@anl.gov> Message-ID: Hi Alp, Thanks for the help. Quasi-Newton seems promising - the Tao solver eventually converges, sometimes after hundreds or even thousands of iterations, with each iterate proceeding very quickly thanks to not evaluating the Hessian. I have only tried this with the problem set up as a general optimization, i.e., not a least-squares problem per se. Your help got me started with testing BRGN, which I'd like to explore more. Can you give me some advice on preconditioners for the subsolver? I guess that the subsolver uses a shell matrix to calculate elements of J^T J from the Jacobian J. I have good preconditioners for J, but I'm not sure what to do in order to apply them to the shell matrix. Thanks, Zak On Wed, Jun 10, 2020 at 3:10 PM Dener, Alp wrote: > Hi Zak, > > You got it right with the TaoBRGNGetSubsolver -> TaoGetKSP workflow. This > will get you the KSP object correctly. > > BRGN is not a stand-alone solver. It?s a wrapper that combines the > user-provided residual and Jacobian callbacks to assemble the gradient and > Hessian under the Gauss-Newton formulation, and feed that information into > a standard TAO solver like NLS. It also incorporates some regularizations > to the problem too. Technically any user could also do this themselves and > directly use NLS to solve a least squares problem, but BRGN reduces the > workload a little bit. > > To use it for your least-squares problem, you would need to set TaoType() > to TAOBRGN and then provide the Tao solver two callbacks: one for the > residual using TaoSetResidualRoutine() and another for the Jacobian using > TaoSetJacobianResidualRoutine(). The regularization can be changed using > -tao_brgn_regularization_type command line > options. > > Note that the BRGN sub solver can also be changed. It?s set to BNLS by > default ? this is bound constrained Newton line search but it also solves > unconstrained problems when no bounds are defined. You could always try to > use BNTR for a trust-region method instead of line search, or switch to > BNQLS to do a quasi-Gauss-Newton solution. The sub solver?s settings can be > accessed via the -tao_brgn_subsolver_ prefix. So for instance if you wanted > to change type, you could do "-tao_brgn_subsolver_tao_type bntr" for Newton > trust-region. > > Alp > > On Jun 10, 2020, at 1:55 PM, zakaryah . wrote: > > Oh, maybe I can answer my own question - for BRGN, is it the subsolver > that has a KSP? Then I would call > > TaoBRGNGetSubsolver(tao, &subsolver); > TaoGetKSP(subsolver,&ksp); > > ? > > On Wed, Jun 10, 2020 at 2:52 PM zakaryah . wrote: > >> Hi Alp, >> >> Thanks very much for this thorough answer. I understand the situation >> much better now. >> >> I will experiment with quasi-Newton methods - this should be >> straightforward to test and evaluate. I agree that the regularized >> Gauss-Newton may be useful in place of the Levenberg Marquardt per se. I am >> having some trouble understanding how it is set up and how to use it for my >> purposes. >> >> Using BRGN, does the Tao solver itself not have a KSP? After setting up >> the solver, running TaoGetKSP returns a NULL KSP. It seems I am doing >> something wrong, but with other Tao types, like nls, everything seems to >> work fine. >> >> Thanks for your help! >> >> Cheers, Zak >> >> >> On Wed, Jun 10, 2020 at 11:29 AM Dener, Alp wrote: >> >>> Hello, >>> >>> STCG is being used to compute a search direction by inverting the >>> Hessian of the objective onto the gradient. The Hessian has to be positive >>> definitive for this search direction to be a valid descent direction. To >>> enforce this, STCG terminates the KSP solution when it encounters negative >>> curvature (i.e. ?uphill? directions). The resulting search direction from >>> this early termination is still a descent direction but it?s a pretty bad >>> one, meaning that the line search is forced to do extra work to find a >>> valid (and sometimes pretty small) step length. This is normal. >>> >>> Once a negative curvature is detected, the NLS algorithm imposes a shift >>> to the diagonal of the Hessian in order to guarantee that it will be >>> positive-definite on the next optimization iteration. This diagonal shift >>> is increased in magnitude if you keep hitting the negative curvature KSP >>> termination, and it?s gradually decreased on iterations where the KSP >>> solution succeeds. However this does not prevent the line search from doing >>> all that extra work on iterations where the search direction solution never >>> converged fully. It simply helps generate better search directions in >>> subsequent iterations and helps the overall optimization solution >>> eventually converge to a valid minimum. >>> >>> As a side note, if your function, gradient and/or Hessian evaluations >>> are expensive, it might actually make more sense to use the quasi-Newton >>> method (BQNLS) instead of Newton line search. The optimization solution is >>> likely to take more overall iterations with BQNLS(-tao_type bqnls), but the >>> BFGS approximation used by the algorithm is inherently guaranteed to be >>> positive-definite and tends to generate very well scaled search directions >>> so it might help the line search do less work. Combined with the >>> elimination of Hessian evaluations, it may very well solve your problem >>> faster in CPU time even if it takes more nonlinear iterations overall. This >>> is technically a bound constrained method but it solves unconstrained >>> problems too when you don?t define any bounds. >>> >>> About Levenberg-Marquardt: a user started the branch to eventually >>> contribute an LM solver, but I have not heard any updates on it since end >>> of April. For least-squares type problems, you can try using the >>> regularized Gauss-Newton solver (-tao_type brgn). The problem definition >>> interface is a bit different. BRGN requires the problem to be defined as a >>> residual and its Jacobian, and it will assemble the gradient and the >>> Hessian on its own and feed it into the standard Newton line search solver >>> underneath. There are also a few different regularization options >>> available, like proximal point Tikhonov (l2prox) and a sparsity term >>> (l1dict) that can be set with the (-tao_brgn_regularization_type) option. >>> These get automatically cooked into the gradient and Hessian. Combined with >>> the automatic diagonal shifts for the Hessian, BRGN really is almost >>> equivalent to an LM solver so it might do what you need. >>> >>> Hope this helps! >>> ? >>> Alp >>> >>> On Jun 9, 2020, at 10:55 PM, zakaryah . wrote: >>> >>> I am using TAO to minimize the elastic strain energy for somewhat >>> complicated applied forces. With Newton linesearch (NLS), the STCG KSP, and >>> several preconditioners (none, Jacobi, lmvm), the solver finds a minimum >>> within an acceptable number of iterations (<50). Still, in the interest of >>> performance, I am wondering about what happens after the KSP encounters an >>> indefinite matrix (KSP_CONVERGED_CG_NEG_CURVE). After this happens, the >>> line search performs extremely poorly, with function values at step length >>> 1 reaching absurdly large values. As the line search tries smaller step >>> lengths, the function values fall, then typically reach values representing >>> a decline from the previous iteration, but only with tiny step lengths >>> (~1e-11). The line search takes many iterations to arrive at such an >>> improvement. Here is an example, run with -tao_type nls -tao_nls_ksp_type >>> stcg -tao_nls_pc_type none -tao_nls_ksp_norm_type unpreconditioned: >>> >>> 0 TAO, Function value: 0.160612, Residual: 0.0736074 >>> >>> >>> >>> 0 TAO, Function value: 0.121568, Residual: 0.0424117 >>> >>> >>> >>> Residual norms for tao_nls_ solve. >>> >>> >>> >>> 0 KSP Residual norm 4.241174778983e-02 >>> >>> >>> >>> 1 KSP Residual norm 5.806891169509e-02 >>> >>> >>> >>> 2 KSP Residual norm 6.492953448014e-02 >>> >>> >>> >>> 3 KSP Residual norm 5.856559430984e-02 >>> >>> >>> >>> 4 KSP Residual norm 5.262548559710e-02 >>> >>> >>> >>> 5 KSP Residual norm 4.863633473400e-02 >>> >>> >>> >>> 6 KSP Residual norm 4.725156347026e-02 >>> >>> >>> >>> 7 KSP Residual norm 4.748458009319e-02 >>> >>> >>> >>> 8 KSP Residual norm 4.885339641711e-02 >>> >>> >>> >>> 9 KSP Residual norm 5.065071226354e-02 >>> >>> >>> >>> 10 KSP Residual norm 5.085544070851e-02 >>> >>> >>> >>> 11 KSP Residual norm 5.127093547976e-02 >>> >>> >>> >>> 12 KSP Residual norm 5.155225884843e-02 >>> >>> >>> >>> 13 KSP Residual norm 5.219895021408e-02 >>> >>> >>> >>> 14 KSP Residual norm 6.480520610077e-02 >>> >>> >>> >>> 15 KSP Residual norm 1.433515456621e-01 >>> >>> >>> >>> Linear tao_nls_ solve converged due to CONVERGED_CG_NEG_CURVE >>> iterations 16 >>> >>> >>> 0 LS Function value: 0.121568, Step length: 0. >>> >>> >>> >>> 1 LS Function value: 5.99839e+59, Step length: 1. >>> >>> >>> >>> stx: 0., fx: 0.121568, dgx: -1.32893e+08 >>> >>> >>> >>> sty: 0., fy: 0.121568, dgy: -1.32893e+08 >>> 2 LS Function value: 1.46445e+56, Step length: 0.25 >>> >>> >>> >>> stx: 0., fx: 0.121568, dgx: -1.32893e+08 >>> >>> >>> >>> sty: 1., fy: 5.99839e+59, dgy: 3.59903e+60 >>> >>> >>> >>> 3 LS Function value: 3.57532e+52, Step length: 0.0625 >>> >>> >>> >>> stx: 0., fx: 0.121568, dgx: -1.32893e+08 >>> >>> >>> >>> sty: 0.25, fy: 1.46445e+56, dgy: 3.51468e+57 >>> >>> >>> >>> 4 LS Function value: 8.7288e+48, Step length: 0.015625 >>> >>> >>> >>> stx: 0., fx: 0.121568, dgx: -1.32893e+08 >>> >>> >>> >>> sty: 0.0625, fy: 3.57532e+52, dgy: 3.4323e+54 >>> >>> >>> >>> 5 LS Function value: 2.13105e+45, Step length: 0.00390625 >>> >>> >>> >>> stx: 0., fx: 0.121568, dgx: -1.32893e+08 >>> >>> >>> >>> sty: 0.015625, fy: 8.7288e+48, dgy: 3.35186e+51 >>> >>> >>> >>> 6 LS Function value: 5.20277e+41, Step length: 0.000976562 >>> >>> >>> >>> stx: 0., fx: 0.121568, dgx: -1.32893e+08 >>> >>> >>> >>> sty: 0.00390625, fy: 2.13105e+45, dgy: 3.2733e+48 >>> >>> >>> >>> 7 LS Function value: 1.27021e+38, Step length: 0.000244141 >>> >>> >>> >>> stx: 0., fx: 0.121568, dgx: -1.32893e+08 >>> >>> >>> >>> sty: 0.000976562, fy: 5.20277e+41, dgy: 3.19658e+45 >>> >>> >>> >>> 8 LS Function value: 3.1011e+34, Step length: 6.10352e-05 >>> >>> >>> >>> stx: 0., fx: 0.121568, dgx: -1.32893e+08 >>> >>> >>> >>> sty: 0.000244141, fy: 1.27021e+38, dgy: 3.12166e+42 >>> >>> >>> >>> 9 LS Function value: 7.57106e+30, Step length: 1.52588e-05 >>> >>> >>> >>> stx: 0., fx: 0.121568, dgx: -1.32893e+08 >>> >>> >>> >>> sty: 6.10352e-05, fy: 3.1011e+34, dgy: 3.0485e+39 >>> >>> >>> >>> 10 LS Function value: 1.84842e+27, Step length: 3.8147e-06 >>> stx: 0., fx: 0.121568, dgx: -1.32893e+08 >>> >>> >>> >>> sty: 1.52588e-05, fy: 7.57106e+30, dgy: 2.97706e+36 >>> >>> >>> >>> 11 LS Function value: 4.51295e+23, Step length: 9.53672e-07 >>> >>> >>> >>> stx: 0., fx: 0.121568, dgx: -1.32893e+08 >>> >>> >>> >>> sty: 3.8147e-06, fy: 1.84842e+27, dgy: 2.90731e+33 >>> >>> >>> >>> 12 LS Function value: 1.10199e+20, Step length: 2.38417e-07 >>> >>> >>> >>> stx: 0., fx: 0.121568, dgx: -1.32893e+08 >>> >>> >>> >>> sty: 9.53672e-07, fy: 4.51295e+23, dgy: 2.83927e+30 >>> >>> >>> >>> 13 LS Function value: 2.69231e+16, Step length: 5.96028e-08 >>> >>> >>> >>> stx: 0., fx: 0.121568, dgx: -1.32893e+08 >>> >>> >>> >>> sty: 2.38417e-07, fy: 1.10199e+20, dgy: 2.77314e+27 >>> >>> >>> >>> 14 LS Function value: 6.59192e+12, Step length: 1.48993e-08 >>> >>> >>> >>> stx: 0., fx: 0.121568, dgx: -1.32893e+08 >>> >>> >>> >>> sty: 5.96028e-08, fy: 2.69231e+16, dgy: 2.70974e+24 >>> >>> >>> >>> 15 LS Function value: 1.62897e+09, Step length: 3.72338e-09 >>> >>> >>> >>> stx: 0., fx: 0.121568, dgx: -1.32893e+08 >>> >>> >>> >>> sty: 1.48993e-08, fy: 6.59192e+12, dgy: 2.65254e+21 >>> >>> >>> >>> 16 LS Function value: 421305., Step length: 9.29291e-10 >>> >>> >>> >>> stx: 0., fx: 0.121568, dgx: -1.32893e+08 >>> >>> >>> >>> sty: 3.72338e-09, fy: 1.62897e+09, dgy: 2.61626e+18 >>> >>> >>> >>> 17 LS Function value: 141.939, Step length: 2.30343e-10 >>> >>> >>> >>> stx: 0., fx: 0.121568, dgx: -1.32893e+08 >>> >>> >>> >>> sty: 9.29291e-10, fy: 421305., dgy: 2.67496e+15 >>> >>> >>> >>> 18 LS Function value: 0.213621, Step length: 5.45712e-11 >>> >>> >>> >>> stx: 0., fx: 0.121568, dgx: -1.32893e+08 >>> >>> >>> >>> sty: 2.30343e-10, fy: 141.939, dgy: 3.35874e+12 >>> >>> >>> >>> 19 LS Function value: 0.120058, Step length: 1.32286e-11 >>> >>> >>> >>> stx: 0., fx: 0.121568, dgx: -1.32893e+08 >>> sty: 5.45712e-11, fy: 0.213621, dgy: 8.60977e+09 >>> 1 TAO, Function value: 0.120058, Residual: 0.0484495 >>> >>> When the KSP does not encounter negative curvature, the linesearch >>> performs well: >>> 3 TAO, Function value: 0.118376, Residual: 0.0545446 >>> >>> >>> >>> Residual norms for tao_nls_ solve. >>> >>> >>> >>> 0 KSP Residual norm 5.454463134947e-02 >>> >>> >>> >>> 1 KSP Residual norm 2.394277461960e-02 >>> >>> >>> >>> 2 KSP Residual norm 6.674182971627e-03 >>> >>> >>> >>> 3 KSP Residual norm 1.235116278289e-03 >>> >>> >>> >>> 4 KSP Residual norm 1.714792759324e-04 >>> >>> >>> >>> 5 KSP Residual norm 3.928769927518e-05 >>> >>> >>> >>> 6 KSP Residual norm 8.464174666739e-06 >>> >>> >>> >>> 7 KSP Residual norm 1.583763581407e-06 >>> >>> >>> >>> 8 KSP Residual norm 3.251685842746e-07 >>> >>> >>> >>> Linear tao_nls_ solve converged due to CONVERGED_RTOL iterations 8 >>> >>> >>> >>> 0 LS Function value: 0.118376, Step length: 0. >>> >>> >>> >>> 1 LS Function value: 0.117884, Step length: 1. >>> >>> >>> >>> stx: 0., fx: 0.118376, dgx: -0.000530697 >>> >>> >>> >>> sty: 0., fy: 0.118376, dgy: -0.000530697 >>> >>> I have attached the full log for this particular solve. The points of >>> negative curvature do not surprise me - this is a difficult problem, with >>> many singularities in the Jacobian/Hessian. In fact, I am very happy with >>> how well STCG is performing, globally. My question is what to make of the >>> poor performance of the linesearch after encountering these points. As I >>> understand it, most of the flags for adjusting the solver after >>> encountering an indefinite matrix involve altering the magnitude of the >>> perturbation by which the Hessian is offset for calculating the update >>> direction. I am not clear on how to adjust the routine for determining the >>> initial step size. More generally, should I expect these points of negative >>> curvature to cause difficulties for the solver, and be satisfied with a >>> large number of iterations before finding an improvement at tiny step size? >>> >>> I have a loosely related question about the status of the Levenberg >>> Marquardt solver within Tao? I remember hearing on the list that this was >>> being worked on, but I am not sure from the branch description whether it >>> is working or intended for general use. I would love to hear any updates on >>> that, because I think it could be useful for my problem. >>> >>> Thanks! >>> >>> >>> >>> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From adener at anl.gov Thu Jun 11 14:01:42 2020 From: adener at anl.gov (Dener, Alp) Date: Thu, 11 Jun 2020 19:01:42 +0000 Subject: [petsc-users] TAO STCG initial perturbation In-Reply-To: References: <2A5F02F9-9218-40A0-9A1F-3004E5A5E279@anl.gov> Message-ID: <15B4EFA9-197C-4883-A2EB-954264695CC1@anl.gov> Hi Zak, Gauss-Newton finds the least-squares solution of overdetermined systems, e.g. nonlinear regression. It minimizes the squared L2-norm of a nonlinear residual ||r(x)||_2^2 where the Jacobian J = dr/dx is rectangular with full column rank. Since this J is not invertible, Gauss-Newton uses the left pseudo-inverse (J^T J)^{-1} J^T to compute an approximate Newton direction. However, when J is invertible, the pseudo-inverse exactly reduces to the classic inverse J^{-1} and you recover the full Newton method. If you have a problem with an invertible J, then you should directly use a Newton method for it (BNLS or BNTR) instead of using Gauss-Newton (BRGN). To utilize your preconditioner to J, you can get a pointer to the PC object via TaoGetKSP -> KSPGetPC, set its type to PCSHELL, and then define the operator for it via PCShellSetApply. If it?s a good preconditioner, it should significantly improve the KSP solution and, by extension, likely reduce the work done in the line search. Alp On Jun 11, 2020, at 12:10 PM, zakaryah . > wrote: Hi Alp, Thanks for the help. Quasi-Newton seems promising - the Tao solver eventually converges, sometimes after hundreds or even thousands of iterations, with each iterate proceeding very quickly thanks to not evaluating the Hessian. I have only tried this with the problem set up as a general optimization, i.e., not a least-squares problem per se. Your help got me started with testing BRGN, which I'd like to explore more. Can you give me some advice on preconditioners for the subsolver? I guess that the subsolver uses a shell matrix to calculate elements of J^T J from the Jacobian J. I have good preconditioners for J, but I'm not sure what to do in order to apply them to the shell matrix. Thanks, Zak On Wed, Jun 10, 2020 at 3:10 PM Dener, Alp > wrote: Hi Zak, You got it right with the TaoBRGNGetSubsolver -> TaoGetKSP workflow. This will get you the KSP object correctly. BRGN is not a stand-alone solver. It?s a wrapper that combines the user-provided residual and Jacobian callbacks to assemble the gradient and Hessian under the Gauss-Newton formulation, and feed that information into a standard TAO solver like NLS. It also incorporates some regularizations to the problem too. Technically any user could also do this themselves and directly use NLS to solve a least squares problem, but BRGN reduces the workload a little bit. To use it for your least-squares problem, you would need to set TaoType() to TAOBRGN and then provide the Tao solver two callbacks: one for the residual using TaoSetResidualRoutine() and another for the Jacobian using TaoSetJacobianResidualRoutine(). The regularization can be changed using -tao_brgn_regularization_type command line options. Note that the BRGN sub solver can also be changed. It?s set to BNLS by default ? this is bound constrained Newton line search but it also solves unconstrained problems when no bounds are defined. You could always try to use BNTR for a trust-region method instead of line search, or switch to BNQLS to do a quasi-Gauss-Newton solution. The sub solver?s settings can be accessed via the -tao_brgn_subsolver_ prefix. So for instance if you wanted to change type, you could do "-tao_brgn_subsolver_tao_type bntr" for Newton trust-region. Alp On Jun 10, 2020, at 1:55 PM, zakaryah . > wrote: Oh, maybe I can answer my own question - for BRGN, is it the subsolver that has a KSP? Then I would call TaoBRGNGetSubsolver(tao, &subsolver); TaoGetKSP(subsolver,&ksp); ? On Wed, Jun 10, 2020 at 2:52 PM zakaryah . > wrote: Hi Alp, Thanks very much for this thorough answer. I understand the situation much better now. I will experiment with quasi-Newton methods - this should be straightforward to test and evaluate. I agree that the regularized Gauss-Newton may be useful in place of the Levenberg Marquardt per se. I am having some trouble understanding how it is set up and how to use it for my purposes. Using BRGN, does the Tao solver itself not have a KSP? After setting up the solver, running TaoGetKSP returns a NULL KSP. It seems I am doing something wrong, but with other Tao types, like nls, everything seems to work fine. Thanks for your help! Cheers, Zak On Wed, Jun 10, 2020 at 11:29 AM Dener, Alp > wrote: Hello, STCG is being used to compute a search direction by inverting the Hessian of the objective onto the gradient. The Hessian has to be positive definitive for this search direction to be a valid descent direction. To enforce this, STCG terminates the KSP solution when it encounters negative curvature (i.e. ?uphill? directions). The resulting search direction from this early termination is still a descent direction but it?s a pretty bad one, meaning that the line search is forced to do extra work to find a valid (and sometimes pretty small) step length. This is normal. Once a negative curvature is detected, the NLS algorithm imposes a shift to the diagonal of the Hessian in order to guarantee that it will be positive-definite on the next optimization iteration. This diagonal shift is increased in magnitude if you keep hitting the negative curvature KSP termination, and it?s gradually decreased on iterations where the KSP solution succeeds. However this does not prevent the line search from doing all that extra work on iterations where the search direction solution never converged fully. It simply helps generate better search directions in subsequent iterations and helps the overall optimization solution eventually converge to a valid minimum. As a side note, if your function, gradient and/or Hessian evaluations are expensive, it might actually make more sense to use the quasi-Newton method (BQNLS) instead of Newton line search. The optimization solution is likely to take more overall iterations with BQNLS(-tao_type bqnls), but the BFGS approximation used by the algorithm is inherently guaranteed to be positive-definite and tends to generate very well scaled search directions so it might help the line search do less work. Combined with the elimination of Hessian evaluations, it may very well solve your problem faster in CPU time even if it takes more nonlinear iterations overall. This is technically a bound constrained method but it solves unconstrained problems too when you don?t define any bounds. About Levenberg-Marquardt: a user started the branch to eventually contribute an LM solver, but I have not heard any updates on it since end of April. For least-squares type problems, you can try using the regularized Gauss-Newton solver (-tao_type brgn). The problem definition interface is a bit different. BRGN requires the problem to be defined as a residual and its Jacobian, and it will assemble the gradient and the Hessian on its own and feed it into the standard Newton line search solver underneath. There are also a few different regularization options available, like proximal point Tikhonov (l2prox) and a sparsity term (l1dict) that can be set with the (-tao_brgn_regularization_type) option. These get automatically cooked into the gradient and Hessian. Combined with the automatic diagonal shifts for the Hessian, BRGN really is almost equivalent to an LM solver so it might do what you need. Hope this helps! ? Alp On Jun 9, 2020, at 10:55 PM, zakaryah . > wrote: I am using TAO to minimize the elastic strain energy for somewhat complicated applied forces. With Newton linesearch (NLS), the STCG KSP, and several preconditioners (none, Jacobi, lmvm), the solver finds a minimum within an acceptable number of iterations (<50). Still, in the interest of performance, I am wondering about what happens after the KSP encounters an indefinite matrix (KSP_CONVERGED_CG_NEG_CURVE). After this happens, the line search performs extremely poorly, with function values at step length 1 reaching absurdly large values. As the line search tries smaller step lengths, the function values fall, then typically reach values representing a decline from the previous iteration, but only with tiny step lengths (~1e-11). The line search takes many iterations to arrive at such an improvement. Here is an example, run with -tao_type nls -tao_nls_ksp_type stcg -tao_nls_pc_type none -tao_nls_ksp_norm_type unpreconditioned: 0 TAO, Function value: 0.160612, Residual: 0.0736074 0 TAO, Function value: 0.121568, Residual: 0.0424117 Residual norms for tao_nls_ solve. 0 KSP Residual norm 4.241174778983e-02 1 KSP Residual norm 5.806891169509e-02 2 KSP Residual norm 6.492953448014e-02 3 KSP Residual norm 5.856559430984e-02 4 KSP Residual norm 5.262548559710e-02 5 KSP Residual norm 4.863633473400e-02 6 KSP Residual norm 4.725156347026e-02 7 KSP Residual norm 4.748458009319e-02 8 KSP Residual norm 4.885339641711e-02 9 KSP Residual norm 5.065071226354e-02 10 KSP Residual norm 5.085544070851e-02 11 KSP Residual norm 5.127093547976e-02 12 KSP Residual norm 5.155225884843e-02 13 KSP Residual norm 5.219895021408e-02 14 KSP Residual norm 6.480520610077e-02 15 KSP Residual norm 1.433515456621e-01 Linear tao_nls_ solve converged due to CONVERGED_CG_NEG_CURVE iterations 16 0 LS Function value: 0.121568, Step length: 0. 1 LS Function value: 5.99839e+59, Step length: 1. stx: 0., fx: 0.121568, dgx: -1.32893e+08 sty: 0., fy: 0.121568, dgy: -1.32893e+08 2 LS Function value: 1.46445e+56, Step length: 0.25 stx: 0., fx: 0.121568, dgx: -1.32893e+08 sty: 1., fy: 5.99839e+59, dgy: 3.59903e+60 3 LS Function value: 3.57532e+52, Step length: 0.0625 stx: 0., fx: 0.121568, dgx: -1.32893e+08 sty: 0.25, fy: 1.46445e+56, dgy: 3.51468e+57 4 LS Function value: 8.7288e+48, Step length: 0.015625 stx: 0., fx: 0.121568, dgx: -1.32893e+08 sty: 0.0625, fy: 3.57532e+52, dgy: 3.4323e+54 5 LS Function value: 2.13105e+45, Step length: 0.00390625 stx: 0., fx: 0.121568, dgx: -1.32893e+08 sty: 0.015625, fy: 8.7288e+48, dgy: 3.35186e+51 6 LS Function value: 5.20277e+41, Step length: 0.000976562 stx: 0., fx: 0.121568, dgx: -1.32893e+08 sty: 0.00390625, fy: 2.13105e+45, dgy: 3.2733e+48 7 LS Function value: 1.27021e+38, Step length: 0.000244141 stx: 0., fx: 0.121568, dgx: -1.32893e+08 sty: 0.000976562, fy: 5.20277e+41, dgy: 3.19658e+45 8 LS Function value: 3.1011e+34, Step length: 6.10352e-05 stx: 0., fx: 0.121568, dgx: -1.32893e+08 sty: 0.000244141, fy: 1.27021e+38, dgy: 3.12166e+42 9 LS Function value: 7.57106e+30, Step length: 1.52588e-05 stx: 0., fx: 0.121568, dgx: -1.32893e+08 sty: 6.10352e-05, fy: 3.1011e+34, dgy: 3.0485e+39 10 LS Function value: 1.84842e+27, Step length: 3.8147e-06 stx: 0., fx: 0.121568, dgx: -1.32893e+08 sty: 1.52588e-05, fy: 7.57106e+30, dgy: 2.97706e+36 11 LS Function value: 4.51295e+23, Step length: 9.53672e-07 stx: 0., fx: 0.121568, dgx: -1.32893e+08 sty: 3.8147e-06, fy: 1.84842e+27, dgy: 2.90731e+33 12 LS Function value: 1.10199e+20, Step length: 2.38417e-07 stx: 0., fx: 0.121568, dgx: -1.32893e+08 sty: 9.53672e-07, fy: 4.51295e+23, dgy: 2.83927e+30 13 LS Function value: 2.69231e+16, Step length: 5.96028e-08 stx: 0., fx: 0.121568, dgx: -1.32893e+08 sty: 2.38417e-07, fy: 1.10199e+20, dgy: 2.77314e+27 14 LS Function value: 6.59192e+12, Step length: 1.48993e-08 stx: 0., fx: 0.121568, dgx: -1.32893e+08 sty: 5.96028e-08, fy: 2.69231e+16, dgy: 2.70974e+24 15 LS Function value: 1.62897e+09, Step length: 3.72338e-09 stx: 0., fx: 0.121568, dgx: -1.32893e+08 sty: 1.48993e-08, fy: 6.59192e+12, dgy: 2.65254e+21 16 LS Function value: 421305., Step length: 9.29291e-10 stx: 0., fx: 0.121568, dgx: -1.32893e+08 sty: 3.72338e-09, fy: 1.62897e+09, dgy: 2.61626e+18 17 LS Function value: 141.939, Step length: 2.30343e-10 stx: 0., fx: 0.121568, dgx: -1.32893e+08 sty: 9.29291e-10, fy: 421305., dgy: 2.67496e+15 18 LS Function value: 0.213621, Step length: 5.45712e-11 stx: 0., fx: 0.121568, dgx: -1.32893e+08 sty: 2.30343e-10, fy: 141.939, dgy: 3.35874e+12 19 LS Function value: 0.120058, Step length: 1.32286e-11 stx: 0., fx: 0.121568, dgx: -1.32893e+08 sty: 5.45712e-11, fy: 0.213621, dgy: 8.60977e+09 1 TAO, Function value: 0.120058, Residual: 0.0484495 When the KSP does not encounter negative curvature, the linesearch performs well: 3 TAO, Function value: 0.118376, Residual: 0.0545446 Residual norms for tao_nls_ solve. 0 KSP Residual norm 5.454463134947e-02 1 KSP Residual norm 2.394277461960e-02 2 KSP Residual norm 6.674182971627e-03 3 KSP Residual norm 1.235116278289e-03 4 KSP Residual norm 1.714792759324e-04 5 KSP Residual norm 3.928769927518e-05 6 KSP Residual norm 8.464174666739e-06 7 KSP Residual norm 1.583763581407e-06 8 KSP Residual norm 3.251685842746e-07 Linear tao_nls_ solve converged due to CONVERGED_RTOL iterations 8 0 LS Function value: 0.118376, Step length: 0. 1 LS Function value: 0.117884, Step length: 1. stx: 0., fx: 0.118376, dgx: -0.000530697 sty: 0., fy: 0.118376, dgy: -0.000530697 I have attached the full log for this particular solve. The points of negative curvature do not surprise me - this is a difficult problem, with many singularities in the Jacobian/Hessian. In fact, I am very happy with how well STCG is performing, globally. My question is what to make of the poor performance of the linesearch after encountering these points. As I understand it, most of the flags for adjusting the solver after encountering an indefinite matrix involve altering the magnitude of the perturbation by which the Hessian is offset for calculating the update direction. I am not clear on how to adjust the routine for determining the initial step size. More generally, should I expect these points of negative curvature to cause difficulties for the solver, and be satisfied with a large number of iterations before finding an improvement at tiny step size? I have a loosely related question about the status of the Levenberg Marquardt solver within Tao? I remember hearing on the list that this was being worked on, but I am not sure from the branch description whether it is working or intended for general use. I would love to hear any updates on that, because I think it could be useful for my problem. Thanks! -------------- next part -------------- An HTML attachment was scrubbed... URL: From mail2amneet at gmail.com Thu Jun 11 14:47:24 2020 From: mail2amneet at gmail.com (Amneet Bhalla) Date: Thu, 11 Jun 2020 12:47:24 -0700 Subject: [petsc-users] Calling MATLAB code within a parallel C/C++ code In-Reply-To: <5E3EB0AC-3E6E-41AB-8441-11B8941F8554@petsc.dev> References: <028B5BA9-4ECA-44B9-BE51-E24000E50185@petsc.dev> <9BE0A2F9-24DF-452B-AABB-1B8FAA748181@petsc.dev> <943847d1-7b6a-8841-f2f8-91652b2f64f2@berkeley.edu> <5E3EB0AC-3E6E-41AB-8441-11B8941F8554@petsc.dev> Message-ID: Ok, I'm able to call and use MATLAB functionality from PETSc. If I'm understanding correctly, when PetscMatlabEngineCreate() is called, a MATLAB workspace is created, that persists until PetscMatlabEngineDestroy() is called. PETSc can access/put/manipulate variables in this workspace, and can also call several MATLAB scripts and MATLAB functions, and subsequently access/manipulate new data created in the existing workspace. Is there something else that I should keep in mind to take advantage of Engine functionality or to avoid pitfalls. Also are there any limitations on MATLAB toolboxes that cannot be used via the Engine framework? I don't anticipate using any graphical interface functionality via MatlabEngine. On Tue, Jun 9, 2020 at 4:26 PM Barry Smith wrote: > > Great, yes you should not need the DYLIB paths > > The branch is off master but what you can do is copy the > config/BuildSystem/config/MatlabEngine.py to your PETSc 3.12.2 version and > the configure > should work (it won't automatically run the test case but you can test it > manually). > > Barry > > > On Jun 9, 2020, at 5:48 PM, Amneet Bhalla wrote: > > That worked! > > amneetb at Amneets-MBP:~/Softwares/PETSc-Gitlab/petsc-matlab$ make > PETSC_DIR=/Users/amneetb/Softwares/PETSc-Gitlab/petsc-matlab > PETSC_ARCH=darwin-dbg check > Running check examples to verify correct installation > Using PETSC_DIR=/Users/amneetb/Softwares/PETSc-Gitlab/petsc-matlab and > PETSC_ARCH=darwin-dbg > C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI process > C/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI processes > C/C++ example src/snes/tutorials/ex19 run successfully with hypre > Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI process > C/C++ example src/vec/vec/tutorials/ex31 run successfully with Matlab > engine > Completed test examples > > Should I remove DYLIB paths from .profile since these are now baked in the > linker? > > Also, Is this branch based-off of latest master? I'm currently using PETSc > v3.12.2 for IBAMR. Anything close to this version should work for me. > > > On Tue, Jun 9, 2020 at 3:17 PM Barry Smith wrote: > >> >> I tried including the directories into my DYLD environmental variables >> and it didn't work; I am not sure why. This might be why >> https://stackoverflow.com/questions/35568122/why-isnt-dyld-library-path-being-propagated-here >> >> Anyways try my branch and that should work. >> >> Barry >> >> >> >> On Jun 9, 2020, at 5:09 PM, Amneet Bhalla wrote: >> >> >> That's where I also found it. Thanks, Sanjay! >> >> (This path is included in my .profile file >> >> export DYLD_FALLBACK_LIBRARY_PATH= >> /Applications/MATLAB_R2019b.app/bin/maci64:/Applications/MATLAB_R2019b.app/sys/os/maci64: >> $DYLD_FALLBACK_LIBRARY_PATH >> >> export DYLD_LIBRARY_PATH= >> /Applications/MATLAB_R2019b.app/bin/maci64:/Applications/MATLAB_R2019b.app/sys/os/maci64: >> $DYLD_LIBRARY_PATH >> >> >> >> ) >> >> Do I need to tell PETSc about this path as well? Something like: >> >> >> --with-matlab-engine-lib=/Applications/MATLAB_R2019b.app/bin/maci64/libeng.dylib >> >> >> On Tue, Jun 9, 2020 at 2:56 PM Sanjay Govindjee wrote: >> >>> Try looking in: >>> >>> $(MLROOT)/bin/maci64 >>> >>> where MLROOT = /Applications/MATLAB_R2019a.app or the like >>> >>> >>> >>> On 6/9/20 2:51 PM, Junchao Zhang wrote: >>> >>> Find recursively where is libeng.dylib under >>> /Applications/MATLAB_R2019b.app/extern/engines/ >>> >>> --Junchao Zhang >>> >>> >>> On Tue, Jun 9, 2020 at 4:27 PM Amneet Bhalla >>> wrote: >>> >>>> Forgot to mention that configure found the following about MATLAB >>>> >>>> Matlab: >>>> Includes: -I/Applications/MATLAB_R2019b.app/extern/include >>>> /Applications/MATLAB_R2019b.app >>>> MatlabEngine: >>>> Library: -L/Applications/MATLAB_R2019b.app/bin/maci64 >>>> -L/Applications/MATLAB_R2019b.app/extern/lib/maci64 -leng -lmex -lmx -lmat >>>> -lut -licudata -licui18n -licuuc >>>> >>>> On Tue, Jun 9, 2020 at 2:20 PM Amneet Bhalla >>>> wrote: >>>> >>>>> Do these additional configure flags for MATLAB look OK? >>>>> >>>>> --with-matlab=1 --with-matlab-dir=/Applications/MATLAB_R2019b.app >>>>> --with-matlab-engine=1 >>>>> --with-matlab-engine-dir=/Applications/MATLAB_R2019b.app/extern/engines/ >>>>> >>>>> >>>>> >>>>> With this configuration make compiled files like: >>>>> >>>>> CC darwin-dbg/obj/sys/classes/viewer/impls/matlab/vmatlab.o >>>>> CC >>>>> darwin-dbg/obj/sys/classes/viewer/impls/matlab/ftn-custom/zvmatlabf.o >>>>> CC darwin-dbg/obj/sys/classes/matlabengine/matlab.o >>>>> CC darwin-dbg/obj/sys/classes/matlabengine/ftn-auto/matlabf.o >>>>> CC >>>>> darwin-dbg/obj/sys/classes/matlabengine/ftn-custom/zmatlabf.o >>>>> CC darwin-dbg/obj/vec/pf/impls/matlab/cmatlab.o >>>>> CC darwin-dbg/obj/vec/vec/utils/matlab/gcreatev.o >>>>> >>>>> make[2]: Leaving directory >>>>> `/Users/amneetb/Softwares/PETSc-Gitlab/PETSc' >>>>> BEGINNING TO COMPILE MATLAB INTERFACE >>>>> Building with 'Xcode with Clang'. >>>>> ld: warning: cannot export hidden symbol _mexfilerequiredapiversion >>>>> from c_mexapi_version.o >>>>> >>>>> ld: warning: cannot export hidden symbol _mexfilerequiredapiversion >>>>> from c_mexapi_version.o >>>>> >>>>> MEX completed successfully. >>>>> Building with 'Xcode with Clang'. >>>>> ld: warning: cannot export hidden symbol _mexfilerequiredapiversion >>>>> from c_mexapi_version.o >>>>> >>>>> ld: warning: cannot export hidden symbol _mexfilerequiredapiversion >>>>> from c_mexapi_version.o >>>>> >>>>> MEX completed successfully. >>>>> Building with 'Xcode with Clang'. >>>>> ld: warning: cannot export hidden symbol _mexfilerequiredapiversion >>>>> from c_mexapi_version.o >>>>> >>>>> MEX completed successfully. >>>>> Building with 'Xcode with Clang'. >>>>> ld: warning: cannot export hidden symbol _mexfilerequiredapiversion >>>>> from c_mexapi_version.o >>>>> >>>>> ld: warning: cannot export hidden symbol _mexfilerequiredapiversion >>>>> from c_mexapi_version.o >>>>> >>>>> MEX completed successfully. >>>>> >>>>> >>>>> However, make check gives error >>>>> >>>>> amneetb at Amneets-MBP:~/Softwares/PETSc-Gitlab/PETSc$ make >>>>> PETSC_DIR=/Users/amneetb/Softwares/PETSc-Gitlab/PETSc PETSC_ARCH=darwin-dbg >>>>> check >>>>> Running test examples to verify correct installation >>>>> Using PETSC_DIR=/Users/amneetb/Softwares/PETSc-Gitlab/PETSc and >>>>> PETSC_ARCH=darwin-dbg >>>>> Possible error running C/C++ src/snes/examples/tutorials/ex19 with 1 >>>>> MPI process >>>>> See http://www.mcs.anl.gov/petsc/documentation/faq.html >>>>> dyld: Library not loaded: @rpath/libeng.dylib >>>>> Referenced from: >>>>> /Users/amneetb/Softwares/PETSc-Gitlab/PETSc/src/snes/examples/tutorials/./ex19 >>>>> Reason: image not found >>>>> >>>>> >>>>> =================================================================================== >>>>> = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES >>>>> = PID 14057 RUNNING AT Amneets-MBP >>>>> = EXIT CODE: 6 >>>>> = CLEANING UP REMAINING PROCESSES >>>>> = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES >>>>> >>>>> =================================================================================== >>>>> YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Abort trap: 6 >>>>> (signal 6) >>>>> This typically refers to a problem with your application. >>>>> Please see the FAQ page for debugging suggestions >>>>> Possible error running C/C++ src/snes/examples/tutorials/ex19 with 2 >>>>> MPI processes >>>>> See http://www.mcs.anl.gov/petsc/documentation/faq.html >>>>> dyld: Library not loaded: @rpath/libeng.dylib >>>>> Referenced from: >>>>> /Users/amneetb/Softwares/PETSc-Gitlab/PETSc/src/snes/examples/tutorials/./ex19 >>>>> Reason: image not found >>>>> dyld: Library not loaded: @rpath/libeng.dylib >>>>> Referenced from: >>>>> /Users/amneetb/Softwares/PETSc-Gitlab/PETSc/src/snes/examples/tutorials/./ex19 >>>>> Reason: image not found >>>>> >>>>> >>>>> =================================================================================== >>>>> = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES >>>>> = PID 14062 RUNNING AT Amneets-MBP >>>>> = EXIT CODE: 6 >>>>> = CLEANING UP REMAINING PROCESSES >>>>> = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES >>>>> >>>>> =================================================================================== >>>>> YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Abort trap: 6 >>>>> (signal 6) >>>>> This typically refers to a problem with your application. >>>>> Please see the FAQ page for debugging suggestions >>>>> 1,5c1,17 >>>>> < lid velocity = 0.0016, prandtl # = 1., grashof # = 1. >>>>> < 0 SNES Function norm 0.0406612 >>>>> < 1 SNES Function norm 4.12227e-06 >>>>> < 2 SNES Function norm 6.098e-11 >>>>> < Number of SNES iterations = 2 >>>>> --- >>>>> > dyld: Library not loaded: @rpath/libeng.dylib >>>>> > Referenced from: >>>>> /Users/amneetb/Softwares/PETSc-Gitlab/PETSc/src/snes/examples/tutorials/./ex19 >>>>> > Reason: image not found >>>>> > dyld: Library not loaded: @rpath/libeng.dylib >>>>> > Referenced from: >>>>> /Users/amneetb/Softwares/PETSc-Gitlab/PETSc/src/snes/examples/tutorials/./ex19 >>>>> > Reason: image not found >>>>> > >>>>> > >>>>> =================================================================================== >>>>> > = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES >>>>> > = PID 14072 RUNNING AT Amneets-MBP >>>>> > = EXIT CODE: 6 >>>>> > = CLEANING UP REMAINING PROCESSES >>>>> > = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES >>>>> > >>>>> =================================================================================== >>>>> > YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Abort trap: 6 >>>>> (signal 6) >>>>> > This typically refers to a problem with your application. >>>>> > Please see the FAQ page for debugging suggestions >>>>> /Users/amneetb/Softwares/PETSc-Gitlab/PETSc/src/snes/examples/tutorials >>>>> Possible problem with ex19 running with hypre, diffs above >>>>> ========================================= >>>>> Possible error running Fortran example >>>>> src/snes/examples/tutorials/ex5f with 1 MPI process >>>>> See http://www.mcs.anl.gov/petsc/documentation/faq.html >>>>> dyld: Library not loaded: @rpath/libeng.dylib >>>>> Referenced from: >>>>> /Users/amneetb/Softwares/PETSc-Gitlab/PETSc/src/snes/examples/tutorials/./ex5f >>>>> Reason: image not found >>>>> >>>>> >>>>> =================================================================================== >>>>> = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES >>>>> = PID 14211 RUNNING AT Amneets-MBP >>>>> = EXIT CODE: 6 >>>>> = CLEANING UP REMAINING PROCESSES >>>>> = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES >>>>> >>>>> =================================================================================== >>>>> YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Abort trap: 6 >>>>> (signal 6) >>>>> This typically refers to a problem with your application. >>>>> Please see the FAQ page for debugging suggestions >>>>> Completed test examples >>>>> >>>>> >>>>> I have added the following paths in my .profile and sourced it before >>>>> make all and make check >>>>> >>>>> export DYLD_FALLBACK_LIBRARY_PATH= >>>>> /Applications/MATLAB_R2019b.app/bin/maci64:/Applications/MATLAB_R2019b.app/sys/os/maci64: >>>>> $DYLD_FALLBACK_LIBRARY_PATH >>>>> >>>>> export DYLD_LIBRARY_PATH= >>>>> /Applications/MATLAB_R2019b.app/bin/maci64:/Applications/MATLAB_R2019b.app/sys/os/maci64: >>>>> $DYLD_LIBRARY_PATH >>>>> >>>>> >>>>> >>>>> On Mon, Jun 8, 2020 at 6:39 PM Barry Smith wrote: >>>>> >>>>>> >>>>>> Looks like the tool to list the examples in the manual pages is >>>>>> broken since it didn't find these other examples, I'll take a look at it. >>>>>> >>>>>> There is also a discussion in the users manual >>>>>> >>>>>> $ git grep MatlabEngine | grep ex >>>>>> src/docs/tex/manual/part2.tex:PetscMatlabEngineCreate(MPI_Comm >>>>>> comm,char *machine,PetscMatlabEngine *e); >>>>>> src/docs/tex/manual/part2.tex:PetscMatlabEnginePut(PetscMatlabEngine >>>>>> e,PetscObject obj); >>>>>> src/docs/tex/manual/part2.tex:PetscMatlabEngineGet(PetscMatlabEngine >>>>>> e,PetscObject obj); >>>>>> src/docs/tex/manual/part2.tex:PetscMatlabEnginePutArray(PetscMatlabEngine >>>>>> e,int m,int n,PetscScalar *array,char *name); >>>>>> src/docs/tex/manual/part2.tex:PetscMatlabEngineGetArray(PetscMatlabEngine >>>>>> e,int m,int n,PetscScalar *array,char *name); >>>>>> >>>>>> src/docs/tex/manual/part2.tex:PetscMatlabEngineEvaluate(PetscMatlabEngine,"format",...); >>>>>> src/docs/tex/manual/part2.tex:PetscMatlabEngineEvaluate(PetscMatlabEngine,"x >>>>>> = \%g *y + z;",avalue); >>>>>> src/docs/tex/manual/part2.tex:PetscMatlabEngineGetOutput(PetscMatlabEngine,char >>>>>> **); >>>>>> >>>>>> src/docs/tex/manual/part2.tex:PetscMatlabEnginedPrintOutput(PetscMatlabEngine,FILE*). >>>>>> src/snes/tutorials/ex5.c: ierr = >>>>>> PetscMatlabEnginePut(PETSC_MATLAB_ENGINE_(comm),(PetscObject)localX);CHKERRQ(ierr); >>>>>> src/snes/tutorials/ex5.c: ierr = >>>>>> PetscMatlabEngineEvaluate(PETSC_MATLAB_ENGINE_(comm),"localF=ex5m(localX,%18.16e,%18.16e,%18.16e)",hx,hy,lambda);CHKERRQ(ierr); >>>>>> src/snes/tutorials/ex5.c: ierr = >>>>>> PetscMatlabEngineGet(PETSC_MATLAB_ENGINE_(comm),(PetscObject)localF);CHKERRQ(ierr); >>>>>> src/sys/tests/ex16.c:static char help[] = "Demonstrates >>>>>> PetscMatlabEngineXXX()\n"; >>>>>> src/sys/tests/ex16.c: ierr = >>>>>> PetscMatlabEngineGetOutput(PETSC_MATLAB_ENGINE_(PETSC_COMM_WORLD),&output);CHKERRQ(ierr); >>>>>> src/sys/tests/ex16.c: ierr = >>>>>> PetscMatlabEngineEvaluate(PETSC_MATLAB_ENGINE_(PETSC_COMM_WORLD),"MPI_Comm_rank");CHKERRQ(ierr); >>>>>> src/sys/tests/ex16.c: ierr = >>>>>> PetscMatlabEngineEvaluate(PETSC_MATLAB_ENGINE_(PETSC_COMM_WORLD),user);CHKERRQ(ierr); >>>>>> src/vec/vec/tests/ex31.c:static const char help[] = "Demonstrates >>>>>> PetscMatlabEngineXXX()\n"; >>>>>> src/vec/vec/tests/ex31.c: ierr = >>>>>> PetscMatlabEngineGetOutput(PETSC_MATLAB_ENGINE_WORLD,&output); >>>>>> src/vec/vec/tests/ex31.c: ierr = >>>>>> PetscMatlabEngineEvaluate(PETSC_MATLAB_ENGINE_WORLD,"MPI_Comm_rank"); >>>>>> src/vec/vec/tests/ex31.c: ierr = >>>>>> PetscMatlabEnginePut(PETSC_MATLAB_ENGINE_WORLD,(PetscObject)x);CHKERRQ(ierr); >>>>>> src/vec/vec/tests/ex31.c: ierr = >>>>>> PetscMatlabEngineEvaluate(PETSC_MATLAB_ENGINE_WORLD,"x = x + >>>>>> MPI_Comm_rank;\n"); >>>>>> src/vec/vec/tests/ex31.c: ierr = >>>>>> PetscMatlabEngineGet(PETSC_MATLAB_ENGINE_WORLD,(PetscObject)x);CHKERRQ(ierr); >>>>>> src/vec/vec/tests/ex31.c: ierr = >>>>>> PetscMatlabEngineEvaluate(PETSC_MATLAB_ENGINE_WORLD,"whos\n");CHKERRQ(ierr); >>>>>> >>>>>> On Jun 8, 2020, at 8:26 PM, Amneet Bhalla >>>>>> wrote: >>>>>> >>>>>> Great! >>>>>> >>>>>> Clicking on some of the functions pointed me to this example >>>>>> https://www.mcs.anl.gov/petsc/petsc-3.13/src/snes/tutorials/ex5.c.html >>>>>> >>>>>> Are there more examples, which are perhaps simpler than the above, >>>>>> which I can refer to? >>>>>> >>>>>> On Mon, Jun 8, 2020 at 5:51 PM Barry Smith wrote: >>>>>> >>>>>>> >>>>>>> You should use the PetscMatlabEngine >>>>>>> https://www.mcs.anl.gov/petsc/petsc-3.13/docs/manualpages/Sys/PetscMatlabEngine.html >>>>>>> >>>>>>> Barry >>>>>>> >>>>>>> >>>>>>> On Jun 8, 2020, at 7:09 PM, Amneet Bhalla >>>>>>> wrote: >>>>>>> >>>>>>> Hi Folks, >>>>>>> >>>>>>> My parallel application written in C++ (which also uses PETSc >>>>>>> library for linear solvers) requires calling control and optimization >>>>>>> toolbox of MATLAB at every time step to get a small number of double >>>>>>> values. Likewise the MATLAB function will need to know some values from the >>>>>>> parallel C++ application to return those doubles. I'm wondering if there is >>>>>>> a recommended way of calling MATLAB from parallel C/C++/Fortran codes, and >>>>>>> if PETSc has some interface that could be leveraged for this exchange. >>>>>>> >>>>>>> Thanks, >>>>>>> -- >>>>>>> --Amneet >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> -- >>>>>> --Amneet >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>> >>>>> -- >>>>> --Amneet >>>>> >>>>> >>>>> >>>>> >>>> >>>> -- >>>> --Amneet >>>> >>>> >>>> >>>> >>> >> >> -- >> --Amneet >> >> >> >> >> > > -- > --Amneet > > > > > -- --Amneet -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Thu Jun 11 14:50:32 2020 From: jed at jedbrown.org (Jed Brown) Date: Thu, 11 Jun 2020 13:50:32 -0600 Subject: [petsc-users] L infinity norm convergence tests In-Reply-To: References: Message-ID: <87wo4dxtyv.fsf@jedbrown.org> Matthew Knepley writes: > On Wed, Jun 10, 2020 at 9:02 AM Mark Lohry wrote: > >> Hi all, is there a built-in way to use L-infinity norms for the SNES and >> KSP convergence tests, or do I need to write a manual KSPSetConvergenceTest >> function? >> > > You need to write a custom test. Note that it may incur a large cost with some methods. In GMRES, for example, the solution and residual are not explicitly available on each iteration. They are in GCR, at the cost of storing two vectors per Krylov iteration instead of one for GMRES. From jed at jedbrown.org Thu Jun 11 16:07:24 2020 From: jed at jedbrown.org (Jed Brown) Date: Thu, 11 Jun 2020 15:07:24 -0600 Subject: [petsc-users] TAO STCG initial perturbation In-Reply-To: References: Message-ID: <87zh9971mb.fsf@jedbrown.org> "Dener, Alp via petsc-users" writes: > About Levenberg-Marquardt: a user started the branch to eventually contribute an LM solver, but I have not heard any updates on it since end of April. For least-squares type problems, you can try using the regularized Gauss-Newton solver (-tao_type brgn). The problem definition interface is a bit different. BRGN requires the problem to be defined as a residual and its Jacobian, and it will assemble the gradient and the Hessian on its own and feed it into the standard Newton line search solver underneath. There are also a few different regularization options available, like proximal point Tikhonov (l2prox) and a sparsity term (l1dict) that can be set with the (-tao_brgn_regularization_type) option. These get automatically cooked into the gradient and Hessian. Combined with the automatic diagonal shifts for the Hessian, BRGN really is almost equivalent to an LM solver so it might do what you need. That'd be here, based off your previous comments. https://gitlab.com/tkonolige/petsc/-/commits/brgn_lm Tristan graduated last month so I don't know if he'll be doing further legwork on this. I'll see if he can make a placeholder MR. From bsmith at petsc.dev Thu Jun 11 20:21:51 2020 From: bsmith at petsc.dev (Barry Smith) Date: Thu, 11 Jun 2020 20:21:51 -0500 Subject: [petsc-users] Calling MATLAB code within a parallel C/C++ code In-Reply-To: References: <028B5BA9-4ECA-44B9-BE51-E24000E50185@petsc.dev> <9BE0A2F9-24DF-452B-AABB-1B8FAA748181@petsc.dev> <943847d1-7b6a-8841-f2f8-91652b2f64f2@berkeley.edu> <5E3EB0AC-3E6E-41AB-8441-11B8941F8554@petsc.dev> Message-ID: <5F8AA414-DE52-4F0F-A607-17908C4E69F8@petsc.dev> > On Jun 11, 2020, at 2:47 PM, Amneet Bhalla wrote: > > Ok, I'm able to call and use MATLAB functionality from PETSc. If I'm understanding correctly, when PetscMatlabEngineCreate() is called, a MATLAB workspace is created, that persists until PetscMatlabEngineDestroy() is called. PETSc can access/put/manipulate variables in this workspace, and can also call several MATLAB scripts and MATLAB functions, and subsequently access/manipulate new data created in the existing workspace. Is there something else that I should keep in mind to take advantage of Engine functionality or to avoid pitfalls. Also are there any limitations on MATLAB toolboxes that cannot be used via the Engine framework? I don't anticipate using any graphical interface functionality via MatlabEngine. Yes this is all correct. It seems the Matlab engine is Matlab so it provides all the functionality of standalone Matlab. You can use Matlab graphics as usual if you run your program with the additional option "-matlab_engine_graphics We just had it off by default because sometimes Matlab Engine may have trouble access the users graphics/window system but hopefully it won't be a problem Barry > > > > On Tue, Jun 9, 2020 at 4:26 PM Barry Smith > wrote: > > Great, yes you should not need the DYLIB paths > > The branch is off master but what you can do is copy the config/BuildSystem/config/MatlabEngine.py to your PETSc 3.12.2 version and the configure > should work (it won't automatically run the test case but you can test it manually). > > Barry > > >> On Jun 9, 2020, at 5:48 PM, Amneet Bhalla > wrote: >> >> That worked! >> >> amneetb at Amneets-MBP:~/Softwares/PETSc-Gitlab/petsc-matlab$ make PETSC_DIR=/Users/amneetb/Softwares/PETSc-Gitlab/petsc-matlab PETSC_ARCH=darwin-dbg check >> Running check examples to verify correct installation >> Using PETSC_DIR=/Users/amneetb/Softwares/PETSc-Gitlab/petsc-matlab and PETSC_ARCH=darwin-dbg >> C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI process >> C/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI processes >> C/C++ example src/snes/tutorials/ex19 run successfully with hypre >> Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI process >> C/C++ example src/vec/vec/tutorials/ex31 run successfully with Matlab engine >> Completed test examples >> >> Should I remove DYLIB paths from .profile since these are now baked in the linker? >> >> Also, Is this branch based-off of latest master? I'm currently using PETSc v3.12.2 for IBAMR. Anything close to this version should work for me. >> >> >> On Tue, Jun 9, 2020 at 3:17 PM Barry Smith > wrote: >> >> I tried including the directories into my DYLD environmental variables and it didn't work; I am not sure why. This might be why https://stackoverflow.com/questions/35568122/why-isnt-dyld-library-path-being-propagated-here >> >> Anyways try my branch and that should work. >> >> Barry >> >> >> >>> On Jun 9, 2020, at 5:09 PM, Amneet Bhalla > wrote: >>> >>> >>> That's where I also found it. Thanks, Sanjay! >>> >>> (This path is included in my .profile file >>> >>> export DYLD_FALLBACK_LIBRARY_PATH=/Applications/MATLAB_R2019b.app/bin/maci64:/Applications/MATLAB_R2019b.app/sys/os/maci64:$DYLD_FALLBACK_LIBRARY_PATH >>> >>> export DYLD_LIBRARY_PATH=/Applications/MATLAB_R2019b.app/bin/maci64:/Applications/MATLAB_R2019b.app/sys/os/maci64:$DYLD_LIBRARY_PATH >>> >>> >>> >>> ) >>> >>> Do I need to tell PETSc about this path as well? Something like: >>> >>> --with-matlab-engine-lib=/Applications/MATLAB_R2019b.app/bin/maci64/libeng.dylib >>> >>> >>> On Tue, Jun 9, 2020 at 2:56 PM Sanjay Govindjee > wrote: >>> Try looking in: >>> >>> $(MLROOT)/bin/maci64 >>> >>> where MLROOT = /Applications/MATLAB_R2019a.app or the like >>> >>> >>> On 6/9/20 2:51 PM, Junchao Zhang wrote: >>>> Find recursively where is libeng.dylib under /Applications/MATLAB_R2019b.app/extern/engines/ >>>> >>>> --Junchao Zhang >>>> >>>> >>>> On Tue, Jun 9, 2020 at 4:27 PM Amneet Bhalla > wrote: >>>> Forgot to mention that configure found the following about MATLAB >>>> >>>> Matlab: >>>> Includes: -I/Applications/MATLAB_R2019b.app/extern/include >>>> /Applications/MATLAB_R2019b.app >>>> MatlabEngine: >>>> Library: -L/Applications/MATLAB_R2019b.app/bin/maci64 -L/Applications/MATLAB_R2019b.app/extern/lib/maci64 -leng -lmex -lmx -lmat -lut -licudata -licui18n -licuuc >>>> >>>> On Tue, Jun 9, 2020 at 2:20 PM Amneet Bhalla > wrote: >>>> Do these additional configure flags for MATLAB look OK? >>>> >>>> --with-matlab=1 --with-matlab-dir=/Applications/MATLAB_R2019b.app --with-matlab-engine=1 --with-matlab-engine-dir=/Applications/MATLAB_R2019b.app/extern/engines/ >>>> >>>> >>>> >>>> With this configuration make compiled files like: >>>> >>>> CC darwin-dbg/obj/sys/classes/viewer/impls/matlab/vmatlab.o >>>> CC darwin-dbg/obj/sys/classes/viewer/impls/matlab/ftn-custom/zvmatlabf.o >>>> CC darwin-dbg/obj/sys/classes/matlabengine/matlab.o >>>> CC darwin-dbg/obj/sys/classes/matlabengine/ftn-auto/matlabf.o >>>> CC darwin-dbg/obj/sys/classes/matlabengine/ftn-custom/zmatlabf.o >>>> CC darwin-dbg/obj/vec/pf/impls/matlab/cmatlab.o >>>> CC darwin-dbg/obj/vec/vec/utils/matlab/gcreatev.o >>>> >>>> make[2]: Leaving directory `/Users/amneetb/Softwares/PETSc-Gitlab/PETSc' >>>> BEGINNING TO COMPILE MATLAB INTERFACE >>>> Building with 'Xcode with Clang'. >>>> ld: warning: cannot export hidden symbol _mexfilerequiredapiversion from c_mexapi_version.o >>>> >>>> ld: warning: cannot export hidden symbol _mexfilerequiredapiversion from c_mexapi_version.o >>>> >>>> MEX completed successfully. >>>> Building with 'Xcode with Clang'. >>>> ld: warning: cannot export hidden symbol _mexfilerequiredapiversion from c_mexapi_version.o >>>> >>>> ld: warning: cannot export hidden symbol _mexfilerequiredapiversion from c_mexapi_version.o >>>> >>>> MEX completed successfully. >>>> Building with 'Xcode with Clang'. >>>> ld: warning: cannot export hidden symbol _mexfilerequiredapiversion from c_mexapi_version.o >>>> >>>> MEX completed successfully. >>>> Building with 'Xcode with Clang'. >>>> ld: warning: cannot export hidden symbol _mexfilerequiredapiversion from c_mexapi_version.o >>>> >>>> ld: warning: cannot export hidden symbol _mexfilerequiredapiversion from c_mexapi_version.o >>>> >>>> MEX completed successfully. >>>> >>>> >>>> However, make check gives error >>>> >>>> amneetb at Amneets-MBP:~/Softwares/PETSc-Gitlab/PETSc$ make PETSC_DIR=/Users/amneetb/Softwares/PETSc-Gitlab/PETSc PETSC_ARCH=darwin-dbg check >>>> Running test examples to verify correct installation >>>> Using PETSC_DIR=/Users/amneetb/Softwares/PETSc-Gitlab/PETSc and PETSC_ARCH=darwin-dbg >>>> Possible error running C/C++ src/snes/examples/tutorials/ex19 with 1 MPI process >>>> See http://www.mcs.anl.gov/petsc/documentation/faq.html >>>> dyld: Library not loaded: @rpath/libeng.dylib >>>> Referenced from: /Users/amneetb/Softwares/PETSc-Gitlab/PETSc/src/snes/examples/tutorials/./ex19 >>>> Reason: image not found >>>> >>>> =================================================================================== >>>> = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES >>>> = PID 14057 RUNNING AT Amneets-MBP >>>> = EXIT CODE: 6 >>>> = CLEANING UP REMAINING PROCESSES >>>> = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES >>>> =================================================================================== >>>> YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Abort trap: 6 (signal 6) >>>> This typically refers to a problem with your application. >>>> Please see the FAQ page for debugging suggestions >>>> Possible error running C/C++ src/snes/examples/tutorials/ex19 with 2 MPI processes >>>> See http://www.mcs.anl.gov/petsc/documentation/faq.html >>>> dyld: Library not loaded: @rpath/libeng.dylib >>>> Referenced from: /Users/amneetb/Softwares/PETSc-Gitlab/PETSc/src/snes/examples/tutorials/./ex19 >>>> Reason: image not found >>>> dyld: Library not loaded: @rpath/libeng.dylib >>>> Referenced from: /Users/amneetb/Softwares/PETSc-Gitlab/PETSc/src/snes/examples/tutorials/./ex19 >>>> Reason: image not found >>>> >>>> =================================================================================== >>>> = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES >>>> = PID 14062 RUNNING AT Amneets-MBP >>>> = EXIT CODE: 6 >>>> = CLEANING UP REMAINING PROCESSES >>>> = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES >>>> =================================================================================== >>>> YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Abort trap: 6 (signal 6) >>>> This typically refers to a problem with your application. >>>> Please see the FAQ page for debugging suggestions >>>> 1,5c1,17 >>>> < lid velocity = 0.0016, prandtl # = 1., grashof # = 1. >>>> < 0 SNES Function norm 0.0406612 >>>> < 1 SNES Function norm 4.12227e-06 >>>> < 2 SNES Function norm 6.098e-11 >>>> < Number of SNES iterations = 2 >>>> --- >>>> > dyld: Library not loaded: @rpath/libeng.dylib >>>> > Referenced from: /Users/amneetb/Softwares/PETSc-Gitlab/PETSc/src/snes/examples/tutorials/./ex19 >>>> > Reason: image not found >>>> > dyld: Library not loaded: @rpath/libeng.dylib >>>> > Referenced from: /Users/amneetb/Softwares/PETSc-Gitlab/PETSc/src/snes/examples/tutorials/./ex19 >>>> > Reason: image not found >>>> > >>>> > =================================================================================== >>>> > = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES >>>> > = PID 14072 RUNNING AT Amneets-MBP >>>> > = EXIT CODE: 6 >>>> > = CLEANING UP REMAINING PROCESSES >>>> > = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES >>>> > =================================================================================== >>>> > YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Abort trap: 6 (signal 6) >>>> > This typically refers to a problem with your application. >>>> > Please see the FAQ page for debugging suggestions >>>> /Users/amneetb/Softwares/PETSc-Gitlab/PETSc/src/snes/examples/tutorials >>>> Possible problem with ex19 running with hypre, diffs above >>>> ========================================= >>>> Possible error running Fortran example src/snes/examples/tutorials/ex5f with 1 MPI process >>>> See http://www.mcs.anl.gov/petsc/documentation/faq.html >>>> dyld: Library not loaded: @rpath/libeng.dylib >>>> Referenced from: /Users/amneetb/Softwares/PETSc-Gitlab/PETSc/src/snes/examples/tutorials/./ex5f >>>> Reason: image not found >>>> >>>> =================================================================================== >>>> = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES >>>> = PID 14211 RUNNING AT Amneets-MBP >>>> = EXIT CODE: 6 >>>> = CLEANING UP REMAINING PROCESSES >>>> = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES >>>> =================================================================================== >>>> YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Abort trap: 6 (signal 6) >>>> This typically refers to a problem with your application. >>>> Please see the FAQ page for debugging suggestions >>>> Completed test examples >>>> >>>> >>>> I have added the following paths in my .profile and sourced it before make all and make check >>>> >>>> export DYLD_FALLBACK_LIBRARY_PATH=/Applications/MATLAB_R2019b.app/bin/maci64:/Applications/MATLAB_R2019b.app/sys/os/maci64:$DYLD_FALLBACK_LIBRARY_PATH >>>> >>>> export DYLD_LIBRARY_PATH=/Applications/MATLAB_R2019b.app/bin/maci64:/Applications/MATLAB_R2019b.app/sys/os/maci64:$DYLD_LIBRARY_PATH >>>> >>>> >>>> >>>> On Mon, Jun 8, 2020 at 6:39 PM Barry Smith > wrote: >>>> >>>> Looks like the tool to list the examples in the manual pages is broken since it didn't find these other examples, I'll take a look at it. >>>> >>>> There is also a discussion in the users manual >>>> >>>> $ git grep MatlabEngine | grep ex >>>> src/docs/tex/manual/part2.tex:PetscMatlabEngineCreate(MPI_Comm comm,char *machine,PetscMatlabEngine *e); >>>> src/docs/tex/manual/part2.tex:PetscMatlabEnginePut(PetscMatlabEngine e,PetscObject obj); >>>> src/docs/tex/manual/part2.tex:PetscMatlabEngineGet(PetscMatlabEngine e,PetscObject obj); >>>> src/docs/tex/manual/part2.tex:PetscMatlabEnginePutArray(PetscMatlabEngine e,int m,int n,PetscScalar *array,char *name); >>>> src/docs/tex/manual/part2.tex:PetscMatlabEngineGetArray(PetscMatlabEngine e,int m,int n,PetscScalar *array,char *name); >>>> src/docs/tex/manual/part2.tex:PetscMatlabEngineEvaluate(PetscMatlabEngine,"format",...); >>>> src/docs/tex/manual/part2.tex:PetscMatlabEngineEvaluate(PetscMatlabEngine,"x = \%g *y + z;",avalue); >>>> src/docs/tex/manual/part2.tex:PetscMatlabEngineGetOutput(PetscMatlabEngine,char **); >>>> src/docs/tex/manual/part2.tex:PetscMatlabEnginedPrintOutput(PetscMatlabEngine,FILE*). >>>> src/snes/tutorials/ex5.c: ierr = PetscMatlabEnginePut(PETSC_MATLAB_ENGINE_(comm),(PetscObject)localX);CHKERRQ(ierr); >>>> src/snes/tutorials/ex5.c: ierr = PetscMatlabEngineEvaluate(PETSC_MATLAB_ENGINE_(comm),"localF=ex5m(localX,%18.16e,%18.16e,%18.16e)",hx,hy,lambda);CHKERRQ(ierr); >>>> src/snes/tutorials/ex5.c: ierr = PetscMatlabEngineGet(PETSC_MATLAB_ENGINE_(comm),(PetscObject)localF);CHKERRQ(ierr); >>>> src/sys/tests/ex16.c:static char help[] = "Demonstrates PetscMatlabEngineXXX()\n"; >>>> src/sys/tests/ex16.c: ierr = PetscMatlabEngineGetOutput(PETSC_MATLAB_ENGINE_(PETSC_COMM_WORLD),&output);CHKERRQ(ierr); >>>> src/sys/tests/ex16.c: ierr = PetscMatlabEngineEvaluate(PETSC_MATLAB_ENGINE_(PETSC_COMM_WORLD),"MPI_Comm_rank");CHKERRQ(ierr); >>>> src/sys/tests/ex16.c: ierr = PetscMatlabEngineEvaluate(PETSC_MATLAB_ENGINE_(PETSC_COMM_WORLD),user);CHKERRQ(ierr); >>>> src/vec/vec/tests/ex31.c:static const char help[] = "Demonstrates PetscMatlabEngineXXX()\n"; >>>> src/vec/vec/tests/ex31.c: ierr = PetscMatlabEngineGetOutput(PETSC_MATLAB_ENGINE_WORLD,&output); >>>> src/vec/vec/tests/ex31.c: ierr = PetscMatlabEngineEvaluate(PETSC_MATLAB_ENGINE_WORLD,"MPI_Comm_rank"); >>>> src/vec/vec/tests/ex31.c: ierr = PetscMatlabEnginePut(PETSC_MATLAB_ENGINE_WORLD,(PetscObject)x);CHKERRQ(ierr); >>>> src/vec/vec/tests/ex31.c: ierr = PetscMatlabEngineEvaluate(PETSC_MATLAB_ENGINE_WORLD,"x = x + MPI_Comm_rank;\n"); >>>> src/vec/vec/tests/ex31.c: ierr = PetscMatlabEngineGet(PETSC_MATLAB_ENGINE_WORLD,(PetscObject)x);CHKERRQ(ierr); >>>> src/vec/vec/tests/ex31.c: ierr = PetscMatlabEngineEvaluate(PETSC_MATLAB_ENGINE_WORLD,"whos\n");CHKERRQ(ierr); >>>> >>>>> On Jun 8, 2020, at 8:26 PM, Amneet Bhalla > wrote: >>>>> >>>>> Great! >>>>> >>>>> Clicking on some of the functions pointed me to this example >>>>> https://www.mcs.anl.gov/petsc/petsc-3.13/src/snes/tutorials/ex5.c.html >>>>> >>>>> Are there more examples, which are perhaps simpler than the above, which I can refer to? >>>>> >>>>> On Mon, Jun 8, 2020 at 5:51 PM Barry Smith > wrote: >>>>> >>>>> You should use the PetscMatlabEngine https://www.mcs.anl.gov/petsc/petsc-3.13/docs/manualpages/Sys/PetscMatlabEngine.html >>>>> >>>>> Barry >>>>> >>>>> >>>>>> On Jun 8, 2020, at 7:09 PM, Amneet Bhalla > wrote: >>>>>> >>>>>> Hi Folks, >>>>>> >>>>>> My parallel application written in C++ (which also uses PETSc library for linear solvers) requires calling control and optimization toolbox of MATLAB at every time step to get a small number of double values. Likewise the MATLAB function will need to know some values from the parallel C++ application to return those doubles. I'm wondering if there is a recommended way of calling MATLAB from parallel C/C++/Fortran codes, and if PETSc has some interface that could be leveraged for this exchange. >>>>>> >>>>>> Thanks, >>>>>> -- >>>>>> --Amneet >>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> --Amneet >>>>> >>>>> >>>>> >>>> >>>> >>>> >>>> -- >>>> --Amneet >>>> >>>> >>>> >>>> >>>> >>>> -- >>>> --Amneet >>>> >>>> >>>> >>> >>> >>> >>> -- >>> --Amneet >>> >>> >>> >> >> >> >> -- >> --Amneet >> >> >> > > > > -- > --Amneet > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From danyang.su at gmail.com Thu Jun 11 21:58:13 2020 From: danyang.su at gmail.com (Danyang Su) Date: Thu, 11 Jun 2020 19:58:13 -0700 Subject: [petsc-users] Parallel writing in HDF5-1.12.0 when some processors have no data to write In-Reply-To: References: Message-ID: <360F2E38-6D7C-496E-877A-9177B38D1F6C@gmail.com> Hi Barry, The HDF5 calls fail. I reconfigure PETSc with HDF 1.10.5 version and it works fine on different platforms. So, it is more likely there is a bug in the latest HDF version. Thanks. All the best, Danyang On June 11, 2020 5:58:28 a.m. PDT, Barry Smith wrote: > >Are you making HDF5 calls that fail or is it PETSc routines calling >HDF5 that fail? > >Regardless it sounds like the easiest fix is to switch back to the >previous HDF5 and wait for HDF5 to fix what sounds to be a bug. > > Barry > > >> On Jun 11, 2020, at 1:05 AM, Danyang Su wrote: >> >> Hi All, >> >> Sorry to send the previous incomplete email accidentally. >> >> After updating to HDF5-1.12.0, I got some problem if some processors >have no data to write or not necessary to write. Since parallel writing >is collective, I cannot disable those processors from writing. For the >old version, there seems no such problem. So far, the problem only >occurs on Linux using GNU compiler. The same code has no problem using >intel compiler or latest gnu compiler on MacOS. >> >> I have already included h5sselect_none in the code for those >processors without data. But it does not take effect. The problem is >documented in the following link (How do you write data when one >process doesn't have or need to write data ?). >> https://support.hdfgroup.org/HDF5/hdf5-quest.html#par-nodata > >> >> Similar problem has also been reported on HDF Forum by others. >> https://forum.hdfgroup.org/t/bug-on-hdf5-1-12-0-fortran-parallel/6864 > >> >> Any suggestion for that? >> >> Thanks, >> >> Danyang -- Sent from my Android device with K-9 Mail. Please excuse my brevity. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Thu Jun 11 22:32:29 2020 From: jed at jedbrown.org (Jed Brown) Date: Thu, 11 Jun 2020 21:32:29 -0600 Subject: [petsc-users] Parallel writing in HDF5-1.12.0 when some processors have no data to write In-Reply-To: <360F2E38-6D7C-496E-877A-9177B38D1F6C@gmail.com> References: <360F2E38-6D7C-496E-877A-9177B38D1F6C@gmail.com> Message-ID: <871rml6jsi.fsf@jedbrown.org> Danyang Su writes: > Hi Barry, > > The HDF5 calls fail. I reconfigure PETSc with HDF 1.10.5 version and it works fine on different platforms. So, it is more likely there is a bug in the latest HDF version. I would double-check that you have not subtly violated a collective requirement in the interface, then report to upstream. From karl.linkui at gmail.com Thu Jun 11 23:31:34 2020 From: karl.linkui at gmail.com (Karl Lin) Date: Thu, 11 Jun 2020 23:31:34 -0500 Subject: [petsc-users] matcreate and assembly issue In-Reply-To: References: Message-ID: After recompiling with 64bit option, the program ran successfully. Thank you very much for the insight. On Thu, Jun 11, 2020 at 12:00 PM Satish Balay wrote: > On Thu, 11 Jun 2020, Karl Lin wrote: > > > Hi, Matthew > > > > Thanks for the suggestion, just did another run and here are some > detailed > > stack traces, maybe will provide some more insight: > > *** Process received signal *** > > Signal: Aborted (6) > > Signal code: (-6) > > /lib64/libpthread.so.0(+0xf5f0)[0x2b56c46dc5f0] > > [ 1] /lib64/libc.so.6(gsignal+0x37)[0x2b56c5486337] > > [ 2] /lib64/libc.so.6(abort+0x148)[0x2b56c5487a28] > > [ 3] /libpetsc.so.3.10(PetscTraceBackErrorHandler+0xc4)[0x2b56c1e6a2d4] > > [ 4] /libpetsc.so.3.10(PetscError+0x1b5)[0x2b56c1e69f65] > > [ 5] > /libpetsc.so.3.10(PetscCommBuildTwoSidedFReq+0x19f0)[0x2b56c1e03cf0] > > [ 6] /libpetsc.so.3.10(+0x77db17)[0x2b56c2425b17] > > [ 7] /libpetsc.so.3.10(+0x77a164)[0x2b56c2422164] > > [ 8] /libpetsc.so.3.10(MatAssemblyBegin_MPIAIJ+0x36)[0x2b56c23912b6] > > [ 9] /libpetsc.so.3.10(MatAssemblyBegin+0xca)[0x2b56c1feccda] > > > > By reconfiguring, you mean recompiling petsc with that option, correct? > > yes. you can use a different PETSC_ARCH for this build - so that both > builds are usable [by just switching PETSC_ARCH from your appliation > makefile] > > Satish > > > > > Thank you. > > > > Karl > > > > On Thu, Jun 11, 2020 at 10:56 AM Matthew Knepley > wrote: > > > > > On Thu, Jun 11, 2020 at 11:51 AM Karl Lin > wrote: > > > > > >> Hi, there > > >> > > >> We have written a program using Petsc to solve large sparse matrix > > >> system. It has been working fine for a while. Recently we encountered > a > > >> problem when the size of the sparse matrix is larger than 10TB. We > used > > >> several hundred nodes and 2200 processes. The program always crashes > during > > >> MatAssemblyBegin.Upon a closer look, there seems to be something > unusual. > > >> We have a little memory check during loading the matrix to keep track > of > > >> rss. The printout of rss in the log shows normal increase up to rank > 2160, > > >> i.e., if we load in a portion of matrix that is 1GB, after > MatSetValues for > > >> that portion, rss will increase roughly about that number. From rank > 2161 > > >> onwards, the rss in every rank doesn't increase after matrix loaded. > Then > > >> comes MatAssemblyBegin, the program crashed on rank 2160. > > >> > > >> Is there a upper limit on the number of processes Petsc can handle? > or is > > >> there a upper limit in terms of the size of the matrix petsc can > handle? > > >> Thank you very much for any info. > > >> > > > > > > It sounds like you overflowed int somewhere. We try and check for this, > > > but catching every place is hard. Try reconfiguring with > > > > > > --with-64-bit-indices > > > > > > Thanks, > > > > > > Matt > > > > > > > > >> Regards, > > >> > > >> Karl > > >> > > > > > > > > > -- > > > What most experimenters take for granted before they begin their > > > experiments is infinitely more interesting than any results to which > their > > > experiments lead. > > > -- Norbert Wiener > > > > > > https://www.cse.buffalo.edu/~knepley/ > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mail2amneet at gmail.com Fri Jun 12 00:00:43 2020 From: mail2amneet at gmail.com (Amneet Bhalla) Date: Thu, 11 Jun 2020 22:00:43 -0700 Subject: [petsc-users] Calling MATLAB code within a parallel C/C++ code In-Reply-To: <5F8AA414-DE52-4F0F-A607-17908C4E69F8@petsc.dev> References: <028B5BA9-4ECA-44B9-BE51-E24000E50185@petsc.dev> <9BE0A2F9-24DF-452B-AABB-1B8FAA748181@petsc.dev> <943847d1-7b6a-8841-f2f8-91652b2f64f2@berkeley.edu> <5E3EB0AC-3E6E-41AB-8441-11B8941F8554@petsc.dev> <5F8AA414-DE52-4F0F-A607-17908C4E69F8@petsc.dev> Message-ID: This is so cool! Thank you for providing a gateway to MATLAB via PETSc. On Thu, Jun 11, 2020 at 6:21 PM Barry Smith wrote: > > > On Jun 11, 2020, at 2:47 PM, Amneet Bhalla wrote: > > Ok, I'm able to call and use MATLAB functionality from PETSc. If I'm > understanding correctly, when PetscMatlabEngineCreate() is called, a > MATLAB workspace is created, that persists until PetscMatlabEngineDestroy() > is called. PETSc can access/put/manipulate variables in this workspace, and > can also call several MATLAB scripts and MATLAB functions, and subsequently > access/manipulate new data created in the existing workspace. Is there > something else that I should keep in mind to take advantage of Engine > functionality or to avoid pitfalls. Also are there any limitations on > MATLAB toolboxes that cannot be used via the Engine framework? I don't > anticipate using any graphical interface functionality via MatlabEngine. > > > Yes this is all correct. It seems the Matlab engine is Matlab so it > provides all the functionality of standalone Matlab. > > You can use Matlab graphics as usual if you run your program with the > additional option "-matlab_engine_graphics We just had it off by default > because sometimes Matlab Engine may have trouble access the users > graphics/window system but hopefully it won't be a problem > > Barry > > > > > > On Tue, Jun 9, 2020 at 4:26 PM Barry Smith wrote: > >> >> Great, yes you should not need the DYLIB paths >> >> The branch is off master but what you can do is copy the >> config/BuildSystem/config/MatlabEngine.py to your PETSc 3.12.2 version and >> the configure >> should work (it won't automatically run the test case but you can test it >> manually). >> >> Barry >> >> >> On Jun 9, 2020, at 5:48 PM, Amneet Bhalla wrote: >> >> That worked! >> >> amneetb at Amneets-MBP:~/Softwares/PETSc-Gitlab/petsc-matlab$ make >> PETSC_DIR=/Users/amneetb/Softwares/PETSc-Gitlab/petsc-matlab >> PETSC_ARCH=darwin-dbg check >> Running check examples to verify correct installation >> Using PETSC_DIR=/Users/amneetb/Softwares/PETSc-Gitlab/petsc-matlab and >> PETSC_ARCH=darwin-dbg >> C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI process >> C/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI >> processes >> C/C++ example src/snes/tutorials/ex19 run successfully with hypre >> Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI >> process >> C/C++ example src/vec/vec/tutorials/ex31 run successfully with Matlab >> engine >> Completed test examples >> >> Should I remove DYLIB paths from .profile since these are now baked in >> the linker? >> >> Also, Is this branch based-off of latest master? I'm currently using >> PETSc v3.12.2 for IBAMR. Anything close to this version should work for >> me. >> >> >> On Tue, Jun 9, 2020 at 3:17 PM Barry Smith wrote: >> >>> >>> I tried including the directories into my DYLD environmental >>> variables and it didn't work; I am not sure why. This might be why >>> https://stackoverflow.com/questions/35568122/why-isnt-dyld-library-path-being-propagated-here >>> >>> Anyways try my branch and that should work. >>> >>> Barry >>> >>> >>> >>> On Jun 9, 2020, at 5:09 PM, Amneet Bhalla wrote: >>> >>> >>> That's where I also found it. Thanks, Sanjay! >>> >>> (This path is included in my .profile file >>> >>> export DYLD_FALLBACK_LIBRARY_PATH= >>> /Applications/MATLAB_R2019b.app/bin/maci64:/Applications/MATLAB_R2019b.app/sys/os/maci64: >>> $DYLD_FALLBACK_LIBRARY_PATH >>> >>> export DYLD_LIBRARY_PATH= >>> /Applications/MATLAB_R2019b.app/bin/maci64:/Applications/MATLAB_R2019b.app/sys/os/maci64: >>> $DYLD_LIBRARY_PATH >>> >>> >>> >>> ) >>> >>> Do I need to tell PETSc about this path as well? Something like: >>> >>> >>> --with-matlab-engine-lib=/Applications/MATLAB_R2019b.app/bin/maci64/libeng.dylib >>> >>> >>> On Tue, Jun 9, 2020 at 2:56 PM Sanjay Govindjee >>> wrote: >>> >>>> Try looking in: >>>> >>>> $(MLROOT)/bin/maci64 >>>> >>>> where MLROOT = /Applications/MATLAB_R2019a.app or the like >>>> >>>> >>>> >>>> On 6/9/20 2:51 PM, Junchao Zhang wrote: >>>> >>>> Find recursively where is libeng.dylib under >>>> /Applications/MATLAB_R2019b.app/extern/engines/ >>>> >>>> --Junchao Zhang >>>> >>>> >>>> On Tue, Jun 9, 2020 at 4:27 PM Amneet Bhalla >>>> wrote: >>>> >>>>> Forgot to mention that configure found the following about MATLAB >>>>> >>>>> Matlab: >>>>> Includes: -I/Applications/MATLAB_R2019b.app/extern/include >>>>> /Applications/MATLAB_R2019b.app >>>>> MatlabEngine: >>>>> Library: -L/Applications/MATLAB_R2019b.app/bin/maci64 >>>>> -L/Applications/MATLAB_R2019b.app/extern/lib/maci64 -leng -lmex -lmx -lmat >>>>> -lut -licudata -licui18n -licuuc >>>>> >>>>> On Tue, Jun 9, 2020 at 2:20 PM Amneet Bhalla >>>>> wrote: >>>>> >>>>>> Do these additional configure flags for MATLAB look OK? >>>>>> >>>>>> --with-matlab=1 --with-matlab-dir=/Applications/MATLAB_R2019b.app >>>>>> --with-matlab-engine=1 >>>>>> --with-matlab-engine-dir=/Applications/MATLAB_R2019b.app/extern/engines/ >>>>>> >>>>>> >>>>>> >>>>>> With this configuration make compiled files like: >>>>>> >>>>>> CC darwin-dbg/obj/sys/classes/viewer/impls/matlab/vmatlab.o >>>>>> CC >>>>>> darwin-dbg/obj/sys/classes/viewer/impls/matlab/ftn-custom/zvmatlabf.o >>>>>> CC darwin-dbg/obj/sys/classes/matlabengine/matlab.o >>>>>> CC >>>>>> darwin-dbg/obj/sys/classes/matlabengine/ftn-auto/matlabf.o >>>>>> CC >>>>>> darwin-dbg/obj/sys/classes/matlabengine/ftn-custom/zmatlabf.o >>>>>> CC darwin-dbg/obj/vec/pf/impls/matlab/cmatlab.o >>>>>> CC darwin-dbg/obj/vec/vec/utils/matlab/gcreatev.o >>>>>> >>>>>> make[2]: Leaving directory >>>>>> `/Users/amneetb/Softwares/PETSc-Gitlab/PETSc' >>>>>> BEGINNING TO COMPILE MATLAB INTERFACE >>>>>> Building with 'Xcode with Clang'. >>>>>> ld: warning: cannot export hidden symbol _mexfilerequiredapiversion >>>>>> from c_mexapi_version.o >>>>>> >>>>>> ld: warning: cannot export hidden symbol _mexfilerequiredapiversion >>>>>> from c_mexapi_version.o >>>>>> >>>>>> MEX completed successfully. >>>>>> Building with 'Xcode with Clang'. >>>>>> ld: warning: cannot export hidden symbol _mexfilerequiredapiversion >>>>>> from c_mexapi_version.o >>>>>> >>>>>> ld: warning: cannot export hidden symbol _mexfilerequiredapiversion >>>>>> from c_mexapi_version.o >>>>>> >>>>>> MEX completed successfully. >>>>>> Building with 'Xcode with Clang'. >>>>>> ld: warning: cannot export hidden symbol _mexfilerequiredapiversion >>>>>> from c_mexapi_version.o >>>>>> >>>>>> MEX completed successfully. >>>>>> Building with 'Xcode with Clang'. >>>>>> ld: warning: cannot export hidden symbol _mexfilerequiredapiversion >>>>>> from c_mexapi_version.o >>>>>> >>>>>> ld: warning: cannot export hidden symbol _mexfilerequiredapiversion >>>>>> from c_mexapi_version.o >>>>>> >>>>>> MEX completed successfully. >>>>>> >>>>>> >>>>>> However, make check gives error >>>>>> >>>>>> amneetb at Amneets-MBP:~/Softwares/PETSc-Gitlab/PETSc$ make >>>>>> PETSC_DIR=/Users/amneetb/Softwares/PETSc-Gitlab/PETSc PETSC_ARCH=darwin-dbg >>>>>> check >>>>>> Running test examples to verify correct installation >>>>>> Using PETSC_DIR=/Users/amneetb/Softwares/PETSc-Gitlab/PETSc and >>>>>> PETSC_ARCH=darwin-dbg >>>>>> Possible error running C/C++ src/snes/examples/tutorials/ex19 with 1 >>>>>> MPI process >>>>>> See http://www.mcs.anl.gov/petsc/documentation/faq.html >>>>>> dyld: Library not loaded: @rpath/libeng.dylib >>>>>> Referenced from: >>>>>> /Users/amneetb/Softwares/PETSc-Gitlab/PETSc/src/snes/examples/tutorials/./ex19 >>>>>> Reason: image not found >>>>>> >>>>>> >>>>>> =================================================================================== >>>>>> = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES >>>>>> = PID 14057 RUNNING AT Amneets-MBP >>>>>> = EXIT CODE: 6 >>>>>> = CLEANING UP REMAINING PROCESSES >>>>>> = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES >>>>>> >>>>>> =================================================================================== >>>>>> YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Abort trap: 6 >>>>>> (signal 6) >>>>>> This typically refers to a problem with your application. >>>>>> Please see the FAQ page for debugging suggestions >>>>>> Possible error running C/C++ src/snes/examples/tutorials/ex19 with 2 >>>>>> MPI processes >>>>>> See http://www.mcs.anl.gov/petsc/documentation/faq.html >>>>>> dyld: Library not loaded: @rpath/libeng.dylib >>>>>> Referenced from: >>>>>> /Users/amneetb/Softwares/PETSc-Gitlab/PETSc/src/snes/examples/tutorials/./ex19 >>>>>> Reason: image not found >>>>>> dyld: Library not loaded: @rpath/libeng.dylib >>>>>> Referenced from: >>>>>> /Users/amneetb/Softwares/PETSc-Gitlab/PETSc/src/snes/examples/tutorials/./ex19 >>>>>> Reason: image not found >>>>>> >>>>>> >>>>>> =================================================================================== >>>>>> = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES >>>>>> = PID 14062 RUNNING AT Amneets-MBP >>>>>> = EXIT CODE: 6 >>>>>> = CLEANING UP REMAINING PROCESSES >>>>>> = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES >>>>>> >>>>>> =================================================================================== >>>>>> YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Abort trap: 6 >>>>>> (signal 6) >>>>>> This typically refers to a problem with your application. >>>>>> Please see the FAQ page for debugging suggestions >>>>>> 1,5c1,17 >>>>>> < lid velocity = 0.0016, prandtl # = 1., grashof # = 1. >>>>>> < 0 SNES Function norm 0.0406612 >>>>>> < 1 SNES Function norm 4.12227e-06 >>>>>> < 2 SNES Function norm 6.098e-11 >>>>>> < Number of SNES iterations = 2 >>>>>> --- >>>>>> > dyld: Library not loaded: @rpath/libeng.dylib >>>>>> > Referenced from: >>>>>> /Users/amneetb/Softwares/PETSc-Gitlab/PETSc/src/snes/examples/tutorials/./ex19 >>>>>> > Reason: image not found >>>>>> > dyld: Library not loaded: @rpath/libeng.dylib >>>>>> > Referenced from: >>>>>> /Users/amneetb/Softwares/PETSc-Gitlab/PETSc/src/snes/examples/tutorials/./ex19 >>>>>> > Reason: image not found >>>>>> > >>>>>> > >>>>>> =================================================================================== >>>>>> > = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES >>>>>> > = PID 14072 RUNNING AT Amneets-MBP >>>>>> > = EXIT CODE: 6 >>>>>> > = CLEANING UP REMAINING PROCESSES >>>>>> > = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES >>>>>> > >>>>>> =================================================================================== >>>>>> > YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Abort trap: 6 >>>>>> (signal 6) >>>>>> > This typically refers to a problem with your application. >>>>>> > Please see the FAQ page for debugging suggestions >>>>>> >>>>>> /Users/amneetb/Softwares/PETSc-Gitlab/PETSc/src/snes/examples/tutorials >>>>>> Possible problem with ex19 running with hypre, diffs above >>>>>> ========================================= >>>>>> Possible error running Fortran example >>>>>> src/snes/examples/tutorials/ex5f with 1 MPI process >>>>>> See http://www.mcs.anl.gov/petsc/documentation/faq.html >>>>>> dyld: Library not loaded: @rpath/libeng.dylib >>>>>> Referenced from: >>>>>> /Users/amneetb/Softwares/PETSc-Gitlab/PETSc/src/snes/examples/tutorials/./ex5f >>>>>> Reason: image not found >>>>>> >>>>>> >>>>>> =================================================================================== >>>>>> = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES >>>>>> = PID 14211 RUNNING AT Amneets-MBP >>>>>> = EXIT CODE: 6 >>>>>> = CLEANING UP REMAINING PROCESSES >>>>>> = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES >>>>>> >>>>>> =================================================================================== >>>>>> YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Abort trap: 6 >>>>>> (signal 6) >>>>>> This typically refers to a problem with your application. >>>>>> Please see the FAQ page for debugging suggestions >>>>>> Completed test examples >>>>>> >>>>>> >>>>>> I have added the following paths in my .profile and sourced it before >>>>>> make all and make check >>>>>> >>>>>> export DYLD_FALLBACK_LIBRARY_PATH= >>>>>> /Applications/MATLAB_R2019b.app/bin/maci64:/Applications/MATLAB_R2019b.app/sys/os/maci64: >>>>>> $DYLD_FALLBACK_LIBRARY_PATH >>>>>> >>>>>> export DYLD_LIBRARY_PATH= >>>>>> /Applications/MATLAB_R2019b.app/bin/maci64:/Applications/MATLAB_R2019b.app/sys/os/maci64: >>>>>> $DYLD_LIBRARY_PATH >>>>>> >>>>>> >>>>>> >>>>>> On Mon, Jun 8, 2020 at 6:39 PM Barry Smith wrote: >>>>>> >>>>>>> >>>>>>> Looks like the tool to list the examples in the manual pages is >>>>>>> broken since it didn't find these other examples, I'll take a look at it. >>>>>>> >>>>>>> There is also a discussion in the users manual >>>>>>> >>>>>>> $ git grep MatlabEngine | grep ex >>>>>>> src/docs/tex/manual/part2.tex:PetscMatlabEngineCreate(MPI_Comm >>>>>>> comm,char *machine,PetscMatlabEngine *e); >>>>>>> src/docs/tex/manual/part2.tex:PetscMatlabEnginePut(PetscMatlabEngine >>>>>>> e,PetscObject obj); >>>>>>> src/docs/tex/manual/part2.tex:PetscMatlabEngineGet(PetscMatlabEngine >>>>>>> e,PetscObject obj); >>>>>>> src/docs/tex/manual/part2.tex:PetscMatlabEnginePutArray(PetscMatlabEngine >>>>>>> e,int m,int n,PetscScalar *array,char *name); >>>>>>> src/docs/tex/manual/part2.tex:PetscMatlabEngineGetArray(PetscMatlabEngine >>>>>>> e,int m,int n,PetscScalar *array,char *name); >>>>>>> >>>>>>> src/docs/tex/manual/part2.tex:PetscMatlabEngineEvaluate(PetscMatlabEngine,"format",...); >>>>>>> src/docs/tex/manual/part2.tex:PetscMatlabEngineEvaluate(PetscMatlabEngine,"x >>>>>>> = \%g *y + z;",avalue); >>>>>>> src/docs/tex/manual/part2.tex:PetscMatlabEngineGetOutput(PetscMatlabEngine,char >>>>>>> **); >>>>>>> >>>>>>> src/docs/tex/manual/part2.tex:PetscMatlabEnginedPrintOutput(PetscMatlabEngine,FILE*). >>>>>>> src/snes/tutorials/ex5.c: ierr = >>>>>>> PetscMatlabEnginePut(PETSC_MATLAB_ENGINE_(comm),(PetscObject)localX);CHKERRQ(ierr); >>>>>>> src/snes/tutorials/ex5.c: ierr = >>>>>>> PetscMatlabEngineEvaluate(PETSC_MATLAB_ENGINE_(comm),"localF=ex5m(localX,%18.16e,%18.16e,%18.16e)",hx,hy,lambda);CHKERRQ(ierr); >>>>>>> src/snes/tutorials/ex5.c: ierr = >>>>>>> PetscMatlabEngineGet(PETSC_MATLAB_ENGINE_(comm),(PetscObject)localF);CHKERRQ(ierr); >>>>>>> src/sys/tests/ex16.c:static char help[] = "Demonstrates >>>>>>> PetscMatlabEngineXXX()\n"; >>>>>>> src/sys/tests/ex16.c: ierr = >>>>>>> PetscMatlabEngineGetOutput(PETSC_MATLAB_ENGINE_(PETSC_COMM_WORLD),&output);CHKERRQ(ierr); >>>>>>> src/sys/tests/ex16.c: ierr = >>>>>>> PetscMatlabEngineEvaluate(PETSC_MATLAB_ENGINE_(PETSC_COMM_WORLD),"MPI_Comm_rank");CHKERRQ(ierr); >>>>>>> src/sys/tests/ex16.c: ierr = >>>>>>> PetscMatlabEngineEvaluate(PETSC_MATLAB_ENGINE_(PETSC_COMM_WORLD),user);CHKERRQ(ierr); >>>>>>> src/vec/vec/tests/ex31.c:static const char help[] = "Demonstrates >>>>>>> PetscMatlabEngineXXX()\n"; >>>>>>> src/vec/vec/tests/ex31.c: ierr = >>>>>>> PetscMatlabEngineGetOutput(PETSC_MATLAB_ENGINE_WORLD,&output); >>>>>>> src/vec/vec/tests/ex31.c: ierr = >>>>>>> PetscMatlabEngineEvaluate(PETSC_MATLAB_ENGINE_WORLD,"MPI_Comm_rank"); >>>>>>> src/vec/vec/tests/ex31.c: ierr = >>>>>>> PetscMatlabEnginePut(PETSC_MATLAB_ENGINE_WORLD,(PetscObject)x);CHKERRQ(ierr); >>>>>>> src/vec/vec/tests/ex31.c: ierr = >>>>>>> PetscMatlabEngineEvaluate(PETSC_MATLAB_ENGINE_WORLD,"x = x + >>>>>>> MPI_Comm_rank;\n"); >>>>>>> src/vec/vec/tests/ex31.c: ierr = >>>>>>> PetscMatlabEngineGet(PETSC_MATLAB_ENGINE_WORLD,(PetscObject)x);CHKERRQ(ierr); >>>>>>> src/vec/vec/tests/ex31.c: ierr = >>>>>>> PetscMatlabEngineEvaluate(PETSC_MATLAB_ENGINE_WORLD,"whos\n");CHKERRQ(ierr); >>>>>>> >>>>>>> On Jun 8, 2020, at 8:26 PM, Amneet Bhalla >>>>>>> wrote: >>>>>>> >>>>>>> Great! >>>>>>> >>>>>>> Clicking on some of the functions pointed me to this example >>>>>>> >>>>>>> https://www.mcs.anl.gov/petsc/petsc-3.13/src/snes/tutorials/ex5.c.html >>>>>>> >>>>>>> Are there more examples, which are perhaps simpler than the above, >>>>>>> which I can refer to? >>>>>>> >>>>>>> On Mon, Jun 8, 2020 at 5:51 PM Barry Smith wrote: >>>>>>> >>>>>>>> >>>>>>>> You should use the PetscMatlabEngine >>>>>>>> https://www.mcs.anl.gov/petsc/petsc-3.13/docs/manualpages/Sys/PetscMatlabEngine.html >>>>>>>> >>>>>>>> Barry >>>>>>>> >>>>>>>> >>>>>>>> On Jun 8, 2020, at 7:09 PM, Amneet Bhalla >>>>>>>> wrote: >>>>>>>> >>>>>>>> Hi Folks, >>>>>>>> >>>>>>>> My parallel application written in C++ (which also uses PETSc >>>>>>>> library for linear solvers) requires calling control and optimization >>>>>>>> toolbox of MATLAB at every time step to get a small number of double >>>>>>>> values. Likewise the MATLAB function will need to know some values from the >>>>>>>> parallel C++ application to return those doubles. I'm wondering if there is >>>>>>>> a recommended way of calling MATLAB from parallel C/C++/Fortran codes, and >>>>>>>> if PETSc has some interface that could be leveraged for this exchange. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> -- >>>>>>>> --Amneet >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> -- >>>>>>> --Amneet >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> -- >>>>>> --Amneet >>>>>> >>>>>> >>>>>> >>>>>> >>>>> >>>>> -- >>>>> --Amneet >>>>> >>>>> >>>>> >>>>> >>>> >>> >>> -- >>> --Amneet >>> >>> >>> >>> >>> >> >> -- >> --Amneet >> >> >> >> >> > > -- > --Amneet > > > > > -- --Amneet -------------- next part -------------- An HTML attachment was scrubbed... URL: From danyang.su at gmail.com Fri Jun 12 01:15:12 2020 From: danyang.su at gmail.com (Danyang Su) Date: Thu, 11 Jun 2020 23:15:12 -0700 Subject: [petsc-users] Parallel writing in HDF5-1.12.0 when some processors have no data to write In-Reply-To: <871rml6jsi.fsf@jedbrown.org> References: <360F2E38-6D7C-496E-877A-9177B38D1F6C@gmail.com> <871rml6jsi.fsf@jedbrown.org> Message-ID: <8A7B62D0-0861-455A-8660-C4ECB392B88F@gmail.com> Hi Jed, Thanks for your double check. The HDF 1.10.6 version also works. But versions from 1.12.x stop working. Attached is the code section where I have problem. !c write the dataset collectively !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!! CODE CRASHES HERE IF SOME PROCESSORS HAVE NO DATA TO WRITE!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! call h5dwrite_f(dset_id, H5T_NATIVE_DOUBLE, dataset, hdf5_dsize, & hdf5_ierr, file_space_id=filespace, & mem_space_id=memspace, xfer_prp = xlist_id) Please let me know if there is something wrong in the code that causes the problem. Thanks, Danyang ?On 2020-06-11, 8:32 PM, "Jed Brown" wrote: Danyang Su writes: > Hi Barry, > > The HDF5 calls fail. I reconfigure PETSc with HDF 1.10.5 version and it works fine on different platforms. So, it is more likely there is a bug in the latest HDF version. I would double-check that you have not subtly violated a collective requirement in the interface, then report to upstream. -------------- next part -------------- A non-text attachment was scrubbed... Name: example.F90 Type: application/octet-stream Size: 5321 bytes Desc: not available URL: From jed at jedbrown.org Fri Jun 12 08:05:15 2020 From: jed at jedbrown.org (Jed Brown) Date: Fri, 12 Jun 2020 07:05:15 -0600 Subject: [petsc-users] Parallel writing in HDF5-1.12.0 when some processors have no data to write In-Reply-To: <8A7B62D0-0861-455A-8660-C4ECB392B88F@gmail.com> References: <360F2E38-6D7C-496E-877A-9177B38D1F6C@gmail.com> <871rml6jsi.fsf@jedbrown.org> <8A7B62D0-0861-455A-8660-C4ECB392B88F@gmail.com> Message-ID: <87k10c5t9w.fsf@jedbrown.org> Danyang Su writes: > Hi Jed, > > Thanks for your double check. > > The HDF 1.10.6 version also works. But versions from 1.12.x stop working. I'd suggest making a reduced test case in order to submit a bug report. This was the relevant change in PETSc for hdf5-1.12. https://gitlab.com/petsc/petsc/commit/806daeb7de397195b5132278177f4d5553f9f612 > Attached is the code section where I have problem. > > !c write the dataset collectively > !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! > !!!! CODE CRASHES HERE IF SOME PROCESSORS HAVE NO DATA TO WRITE!!!! > !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! > call h5dwrite_f(dset_id, H5T_NATIVE_DOUBLE, dataset, hdf5_dsize, & > hdf5_ierr, file_space_id=filespace, & > mem_space_id=memspace, xfer_prp = xlist_id) > > Please let me know if there is something wrong in the code that causes the problem. From lu_qin_2000 at yahoo.com Fri Jun 12 11:48:43 2020 From: lu_qin_2000 at yahoo.com (Qin Lu) Date: Fri, 12 Jun 2020 16:48:43 +0000 (UTC) Subject: [petsc-users] Direct linear solver in Petsc References: <272365764.1519002.1591980523950.ref@mail.yahoo.com> Message-ID: <272365764.1519002.1591980523950@mail.yahoo.com> Hello, I plan to solve a small sparse linear equation system using the direct solver, since the number of unknowns is small (less than 1000). Here I got a few questions: 1. Is there a general guide line on the size of the system that direct solver is more efficient than iterative solver?2. Is the native Petsc direct solver (using KSPPREONLY and PCLU) designed to solve dense system or sparse system?3. Is the native Petsc direct solver comparable to SuperLU and MUMPS in performance? Do they use different algorithms? Thanks in advance for your help. Best Regards,Qin -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Jun 12 12:13:07 2020 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 12 Jun 2020 13:13:07 -0400 Subject: [petsc-users] Direct linear solver in Petsc In-Reply-To: <272365764.1519002.1591980523950@mail.yahoo.com> References: <272365764.1519002.1591980523950.ref@mail.yahoo.com> <272365764.1519002.1591980523950@mail.yahoo.com> Message-ID: On Fri, Jun 12, 2020 at 12:49 PM Qin Lu via petsc-users < petsc-users at mcs.anl.gov> wrote: > Hello, > > I plan to solve a small sparse linear equation system using the direct > solver, since the number of unknowns is small (less than 1000). Here I got > a few questions: > > 1. Is there a general guide line on the size of the system that direct > solver is more efficient than iterative solver? > I would say at least to 50K unknowns, maybe 100K, depending on sparsity. > 2. Is the native Petsc direct solver (using KSPPREONLY and PCLU) designed > to solve dense system or sparse system? > Sparse. We use LAPACK for dense. > 3. Is the native Petsc direct solver comparable to SuperLU and MUMPS in > performance? Do they use different algorithms? > It is likely slower than SuperLU and not as robust as MUMPS, but there is no substitute for just running and comparing. Thanks, Matt > Thanks in advance for your help. > > Best Regards, > Qin > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Fri Jun 12 12:13:38 2020 From: mfadams at lbl.gov (Mark Adams) Date: Fri, 12 Jun 2020 13:13:38 -0400 Subject: [petsc-users] Direct linear solver in Petsc In-Reply-To: <272365764.1519002.1591980523950@mail.yahoo.com> References: <272365764.1519002.1591980523950.ref@mail.yahoo.com> <272365764.1519002.1591980523950@mail.yahoo.com> Message-ID: On Fri, Jun 12, 2020 at 12:56 PM Qin Lu via petsc-users < petsc-users at mcs.anl.gov> wrote: > Hello, > > I plan to solve a small sparse linear equation system using the direct > solver, since the number of unknowns is small (less than 1000). Here I got > a few questions: > > 1. Is there a general guide line on the size of the system that direct > solver is more efficient than iterative solver? > No. Direct solvers are about O(N^2) complexity and iteratives solvers can be O(N) - O(N^inf) > 2. Is the native Petsc direct solver (using KSPPREONLY and PCLU) designed > to solve dense system or sparse system? > sparse. > 3. Is the native Petsc direct solver comparable to SuperLU and MUMPS in > performance? Do they use different algorithms? > PETSc's is only in serial and CPU. MUMPS and SuperLU are MPI parallel. SuperLU runs on GPUs. SuperLU and MUMPS have more optimizations, such that there is probably a significant difference on some problems but it's hard to say in general. > > Thanks in advance for your help. > > Best Regards, > Qin > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lu_qin_2000 at yahoo.com Fri Jun 12 12:33:39 2020 From: lu_qin_2000 at yahoo.com (Qin Lu) Date: Fri, 12 Jun 2020 17:33:39 +0000 (UTC) Subject: [petsc-users] Direct linear solver in Petsc In-Reply-To: References: <272365764.1519002.1591980523950.ref@mail.yahoo.com> <272365764.1519002.1591980523950@mail.yahoo.com> Message-ID: <2041822634.2387462.1591983219837@mail.yahoo.com> Thanks Matthew and Mark! Qin On Friday, June 12, 2020, 12:22:08 PM CDT, Mark Adams wrote: On Fri, Jun 12, 2020 at 12:56 PM Qin Lu via petsc-users wrote: Hello, I plan to solve a small sparse linear equation system using the direct solver, since the number of unknowns is small (less than 1000). Here I got a few questions: 1. Is there a general guide line on the size of the system that direct solver is more efficient than iterative solver? No. Direct solvers are about O(N^2) complexity and iteratives solvers can be O(N) - O(N^inf)? 2. Is the native Petsc direct solver (using KSPPREONLY and PCLU) designed to solve dense system or sparse system? sparse.? 3. Is the native Petsc direct solver comparable to SuperLU and MUMPS in performance? Do they use different algorithms? PETSc's is only in serial and CPU. MUMPS and SuperLU are MPI parallel. SuperLU runs on GPUs. SuperLU and MUMPS have more optimizations, such that there is probably a significant difference on some problems but it's hard to say in general.?? Thanks in advance for your help. Best Regards,Qin -------------- next part -------------- An HTML attachment was scrubbed... URL: From shaswat121994 at gmail.com Fri Jun 12 14:17:47 2020 From: shaswat121994 at gmail.com (Shashwat Tiwari) Date: Sat, 13 Jun 2020 00:47:47 +0530 Subject: [petsc-users] Implementing periodicity using DMPlex Message-ID: Hi, I am writing a first order 2D solver for unstructured grids with periodic boundaries using DMPlex. After generating the mesh, I use "DMSetPeriodicity" function to set periodicity in both directions. After which I partition the mesh (DMPlexDistribute), construct ghost cells (DMPlexConstructGhostCells), create a section, and set some initial values in the global vector. Then I use "VecGhostUpdateBegin" to start updating the boundary ghost cell values, but, I get the following error in case I use multiple processors: [0]PETSC ERROR: Invalid argument [0]PETSC ERROR: Vector is not ghosted [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. if I run with a single process, there is no error but the values remain empty (zero) and are not updated. Kindly let me know, if I am missing some crucial step before I can update the ghost values in order to implement the periodic bc, or if there is any other approach to achieve it. I am attaching a small code to demonstrate the issue for your reference. Regards, Shashwat -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: test.c Type: text/x-csrc Size: 5575 bytes Desc: not available URL: From jed at jedbrown.org Fri Jun 12 14:26:34 2020 From: jed at jedbrown.org (jed at jedbrown.org) Date: Fri, 12 Jun 2020 13:26:34 -0600 Subject: [petsc-users] Implementing periodicity using DMPlex In-Reply-To: Message-ID: <0cd9d36f-b897-4e7c-ac83-0908a93f0b8e@email.android.com> An HTML attachment was scrubbed... URL: From danyang.su at gmail.com Fri Jun 12 14:29:43 2020 From: danyang.su at gmail.com (Danyang Su) Date: Fri, 12 Jun 2020 12:29:43 -0700 Subject: [petsc-users] Parallel writing in HDF5-1.12.0 when some processors have no data to write In-Reply-To: <87k10c5t9w.fsf@jedbrown.org> References: <360F2E38-6D7C-496E-877A-9177B38D1F6C@gmail.com> <871rml6jsi.fsf@jedbrown.org> <8A7B62D0-0861-455A-8660-C4ECB392B88F@gmail.com> <87k10c5t9w.fsf@jedbrown.org> Message-ID: Hi Jed, Attached is the example for your test. This example uses H5Sset_none to tell H5Dwrite call that there will be no data. 4-th process HAS to participate we are in a collective mode. The code is ported and modified based on the C example from https://support.hdfgroup.org/ftp/HDF5/examples/parallel/coll_test.c The compiling flags in the makefile are same as those used in my own code. To compile the code, please run 'make all' To test the code, please run 'mpiexec -n 4 ./hdf5_zero_data'. Any number of processors larger than 4 should help to detect the problem. The code may crash on HDF5 1.12.0 but works fine on HDF5 1.10.x. The following platforms have been tested: Macos-Mojave + GNU-8.2 + HDF5-1.12.0 -> Works fine Ubuntu-16.04 + GNU-5.4 + HDF5-1.12.0 -> Crashes Ubuntu-16.04 + GNU-7.5 + HDF5-1.12.0 -> Crashes Ubuntu-16.04 + GNU-5.4 + HDF5-1.10.x -> Works fine Centos-7 + Intel2018 + HDF5-12.0 -> Works fine Possible error when code crashes At line 6686 of file H5_gen.F90 Fortran runtime error: Index '1' of dimension 1 of array 'buf' above upper bound of 0 Thanks, Danyang ?On 2020-06-12, 6:05 AM, "Jed Brown" wrote: Danyang Su writes: > Hi Jed, > > Thanks for your double check. > > The HDF 1.10.6 version also works. But versions from 1.12.x stop working. I'd suggest making a reduced test case in order to submit a bug report. This was the relevant change in PETSc for hdf5-1.12. https://gitlab.com/petsc/petsc/commit/806daeb7de397195b5132278177f4d5553f9f612 > Attached is the code section where I have problem. > > !c write the dataset collectively > !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! > !!!! CODE CRASHES HERE IF SOME PROCESSORS HAVE NO DATA TO WRITE!!!! > !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! > call h5dwrite_f(dset_id, H5T_NATIVE_DOUBLE, dataset, hdf5_dsize, & > hdf5_ierr, file_space_id=filespace, & > mem_space_id=memspace, xfer_prp = xlist_id) > > Please let me know if there is something wrong in the code that causes the problem. -------------- next part -------------- A non-text attachment was scrubbed... Name: hdf5_zero_data.F90 Type: application/octet-stream Size: 5487 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: makefile Type: application/octet-stream Size: 5065 bytes Desc: not available URL: From ys453 at cam.ac.uk Fri Jun 12 17:45:53 2020 From: ys453 at cam.ac.uk (Y. Shidi) Date: Fri, 12 Jun 2020 23:45:53 +0100 Subject: [petsc-users] Petsc options Message-ID: Dear developers, I want to use "-info" to view the preallocation of the matrix. I cannot add it directly when I run the code. I have tried: PetscOptionsInsertString(), PetscOptionsSetValue(). Neither of them works. So I am wondering if there is any other functions. Thank you for your time. Kind Regards, Shidi From balay at mcs.anl.gov Fri Jun 12 17:52:43 2020 From: balay at mcs.anl.gov (Satish Balay) Date: Fri, 12 Jun 2020 17:52:43 -0500 (CDT) Subject: [petsc-users] Petsc options In-Reply-To: References: Message-ID: You can: specify via env variable - before running the appliation: export PETSC_OPTIONS="-info" mpiexec ./binary Or add options to ~/.petscrc file [one option per line] - and then run the code. Satish On Fri, 12 Jun 2020, Y. Shidi wrote: > Dear developers, > > I want to use "-info" to view the preallocation of the matrix. > I cannot add it directly when I run the code. > I have tried: > PetscOptionsInsertString(), PetscOptionsSetValue(). > Neither of them works. > So I am wondering if there is any other functions. > > Thank you for your time. > > Kind Regards, > Shidi > From bsmith at petsc.dev Fri Jun 12 18:18:37 2020 From: bsmith at petsc.dev (Barry Smith) Date: Fri, 12 Jun 2020 18:18:37 -0500 Subject: [petsc-users] Direct linear solver in Petsc In-Reply-To: <2041822634.2387462.1591983219837@mail.yahoo.com> References: <272365764.1519002.1591980523950.ref@mail.yahoo.com> <272365764.1519002.1591980523950@mail.yahoo.com> <2041822634.2387462.1591983219837@mail.yahoo.com> Message-ID: <99489140-35B6-4AC2-A320-889CC7299DAE@petsc.dev> For the problem sizes you describe with sparse matrices the PETSc built in one is competitive with the external solvers, sometimes faster, it is not worth using the external solvers for such small problems. The external solvers have much more elaborate implements of sparse factorizations and solve but those elaborations are for getting higher performance on large problems. For dense matrices PETSc uses the LAPACK solver which does fine at those sizes. Barry > On Jun 12, 2020, at 12:33 PM, Qin Lu via petsc-users wrote: > > Thanks Matthew and Mark! > > Qin > > On Friday, June 12, 2020, 12:22:08 PM CDT, Mark Adams wrote: > > > > > On Fri, Jun 12, 2020 at 12:56 PM Qin Lu via petsc-users > wrote: > Hello, > > I plan to solve a small sparse linear equation system using the direct solver, since the number of unknowns is small (less than 1000). Here I got a few questions: > > 1. Is there a general guide line on the size of the system that direct solver is more efficient than iterative solver? > > No. Direct solvers are about O(N^2) complexity and iteratives solvers can be O(N) - O(N^inf) > > 2. Is the native Petsc direct solver (using KSPPREONLY and PCLU) designed to solve dense system or sparse system? > > sparse. > > 3. Is the native Petsc direct solver comparable to SuperLU and MUMPS in performance? Do they use different algorithms? > > PETSc's is only in serial and CPU. MUMPS and SuperLU are MPI parallel. SuperLU runs on GPUs. > > SuperLU and MUMPS have more optimizations, such that there is probably a significant difference on some problems but it's hard to say in general. > > > Thanks in advance for your help. > > Best Regards, > Qin -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Fri Jun 12 19:52:51 2020 From: mfadams at lbl.gov (Mark Adams) Date: Fri, 12 Jun 2020 20:52:51 -0400 Subject: [petsc-users] Direct linear solver in Petsc In-Reply-To: References: <272365764.1519002.1591980523950.ref@mail.yahoo.com> <272365764.1519002.1591980523950@mail.yahoo.com> Message-ID: On Fri, Jun 12, 2020 at 1:18 PM Matthew Knepley wrote: > On Fri, Jun 12, 2020 at 12:49 PM Qin Lu via petsc-users < > petsc-users at mcs.anl.gov> wrote: > >> Hello, >> >> I plan to solve a small sparse linear equation system using the direct >> solver, since the number of unknowns is small (less than 1000). Here I got >> a few questions: >> >> 1. Is there a general guide line on the size of the system that direct >> solver is more efficient than iterative solver? >> > > I would say at least to 50K unknowns, maybe 100K, depending on sparsity. > The dimension of your problem is important here, 1D is perfect in serial, 2D 100K at reachable, 100K in 3D is painful. And multiple dofs per vertex don't hurt much so you could think of these numbers as vertices. > > >> 2. Is the native Petsc direct solver (using KSPPREONLY and PCLU) designed >> to solve dense system or sparse system? >> > > Sparse. We use LAPACK for dense. > > >> 3. Is the native Petsc direct solver comparable to SuperLU and MUMPS in >> performance? Do they use different algorithms? >> > > It is likely slower than SuperLU and not as robust as MUMPS, but there is > no substitute for just running and comparing. > > Thanks, > > Matt > > >> Thanks in advance for your help. >> >> Best Regards, >> Qin >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lu_qin_2000 at yahoo.com Fri Jun 12 20:00:12 2020 From: lu_qin_2000 at yahoo.com (Qin Lu) Date: Sat, 13 Jun 2020 01:00:12 +0000 (UTC) Subject: [petsc-users] Direct linear solver in Petsc In-Reply-To: <99489140-35B6-4AC2-A320-889CC7299DAE@petsc.dev> References: <272365764.1519002.1591980523950.ref@mail.yahoo.com> <272365764.1519002.1591980523950@mail.yahoo.com> <2041822634.2387462.1591983219837@mail.yahoo.com> <99489140-35B6-4AC2-A320-889CC7299DAE@petsc.dev> Message-ID: <1347077101.132435.1592010012614@mail.yahoo.com> Thanks Barry! Qin On Friday, June 12, 2020, 6:18 PM, Barry Smith wrote: ? For the problem sizes you describe with sparse matrices the PETSc built in one is competitive with the external solvers, sometimes faster, it is not worth using the external solvers for such small problems. The external solvers have much more elaborate implements of sparse factorizations and solve but those elaborations are for getting higher performance on large problems.?? ?For dense matrices PETSc uses the LAPACK solver which does fine at those sizes. ? ?Barry On Jun 12, 2020, at 12:33 PM, Qin Lu via petsc-users wrote: Thanks Matthew and Mark! Qin On Friday, June 12, 2020, 12:22:08 PM CDT, Mark Adams wrote: On Fri, Jun 12, 2020 at 12:56 PM Qin Lu via petsc-users wrote: Hello, I plan to solve a small sparse linear equation system using the direct solver, since the number of unknowns is small (less than 1000). Here I got a few questions: 1. Is there a general guide line on the size of the system that direct solver is more efficient than iterative solver? No. Direct solvers are about O(N^2) complexity and iteratives solvers can be O(N) - O(N^inf)? 2. Is the native Petsc direct solver (using KSPPREONLY and PCLU) designed to solve dense system or sparse system? sparse.? 3. Is the native Petsc direct solver comparable to SuperLU and MUMPS in performance? Do they use different algorithms? PETSc's is only in serial and CPU. MUMPS and SuperLU are MPI parallel. SuperLU runs on GPUs. SuperLU and MUMPS have more optimizations, such that there is probably a significant difference on some problems but it's hard to say in general.?? Thanks in advance for your help. Best Regards,Qin -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Fri Jun 12 20:43:24 2020 From: jed at jedbrown.org (Jed Brown) Date: Fri, 12 Jun 2020 19:43:24 -0600 Subject: [petsc-users] Make stream In-Reply-To: <87pna7d0p3.fsf@jedbrown.org> References: <874krkdjjg.fsf@jedbrown.org> <87pna7d0p3.fsf@jedbrown.org> Message-ID: <87h7vf68qr.fsf@jedbrown.org> Jed Brown writes: > Fande Kong writes: > >>> There's a lot more to AMG setup than memory bandwidth (architecture >>> matters a lot, even between different generation CPUs). >> >> >> Could you elaborate a bit more on this? From my understanding, one big part >> of AMG SetUp is RAP that should be pretty much bandwidth. > > The RAP isn't "pretty much bandwidth". See below for some > Skylake/POWER9/EPYC results and analysis (copied from an off-list > thread). I'll leave in some other bandwidth comments that may or may > not be relevant to you. The short story is that Skylake and EPYC are > both much better than POWER9 at MatPtAP despite POWER9 having similar > bandwidth as EPYC and thus being significantly faster than Skylake for > MatMult/smoothing. > > > Jed Brown writes: > >> I'm attaching a log from my machine (Noether), which is 2-socket EPYC >> 7452 (32 cores each). Each socket has 8xDDR4-3200 and 128 MB of L3 >> cache. This is the same node architecture as the new BER/E3SM machine >> being installed at Argonne (though that one will probably have >> higher-clocked and/or more cores per socket). Note that these CPUs are >> about $2k each while Skylake 8180 are about $10k. >> >> Some excerpts/comments below. >> > > [...] > > In addition to the notes below, I'd like to call out how important > streaming stores are on EPYC. With vanilla code or _mm256_store_pd, we > get the following performance > > $ mpiexec -n 64 --bind-to core --map-by core:1 src/benchmarks/streams/MPIVersion > Copy 162609.2392 Scale 159119.8259 Add 174687.6250 Triad 175840.1587 > > but replacing _mm256_store_pd with _mm256_stream_pd gives this > > $ mpiexec -n 64 --bind-to core --map-by core:1 src/benchmarks/streams/MPIVersion > Copy 259951.9936 Scale 259381.0589 Add 250216.3389 Triad 249292.9701 I turned on NPS4 (a BIOS setting that creates a NUMA node for each pair of memory channels) and get a modest performance boost. $ mpiexec -n 64 --bind-to core --map-by core:1 src/benchmarks/streams/MPIVersion Copy 289645.3776 Scale 289186.2783 Add 273220.0133 Triad 272911.2263 On this architecture, best performance comes from one process per 4-core CCX (shared L3). $ mpiexec -n 16 --bind-to core --map-by core:4 src/benchmarks/streams/MPIVersion Copy 300704.8859 Scale 304556.3380 Add 295970.1132 Triad 298891.3821 > This is just preposterously huge, but very repeatable using gcc and > clang, and inspecting the assembly. This suggests that it would be > useful for vector kernels to have streaming and non-streaming variants. > That is, if I drop the vector length by 20 (so the working set is 2.3 > MB/core instead of 46 MB in the default version), then we get 2.4 TB/s > Triad with _mm256_store_pd: > > $ mpiexec -n 64 --bind-to core --map-by core:1 src/benchmarks/streams/MPIVersion > Copy 2159915.7058 Scale 2212671.7087 Add 2414758.2757 Triad 2402671.1178 > > and a thoroughly embarrassing 353 GB/s with _mm256_stream_pd: > > $ mpiexec -n 64 --bind-to core --map-by core:1 src/benchmarks/streams/MPIVersion > Copy 235934.6653 Scale 237446.8507 Add 352805.7288 Triad 352992.9692 > > > I don't know a good way to automatically determine whether to expect the > memory to be in cache, but we could make it a global (or per-object) > run-time selection. > >> Jed Brown writes: >> >>> "Smith, Barry F." writes: >>> >>>> Thanks. The PowerPC is pretty crappy compared to Skylake. >>> >>> Compare the MGSmooth times. The POWER9 is faster than the Skylake >>> because it has more memory bandwidth. >>> >>> $ rg 'MGInterp Level 4|MGSmooth Level 4' ex56* >>> ex56-JLSE-skylake-56ranks-converged.txt >>> 254:MGSmooth Level 4 68 1.0 1.8808e+00 1.2 7.93e+08 1.3 3.6e+04 1.9e+04 3.4e+01 8 29 10 16 3 62 60 18 54 25 22391 >>> 256:MGInterp Level 4 68 1.0 4.0043e-01 1.8 1.45e+08 1.3 2.2e+04 2.5e+03 0.0e+00 1 5 6 1 0 9 11 11 4 0 19109 >>> >>> ex56-summit-cpu-36ranks-converged.txt >>> 265:MGSmooth Level 4 68 1.0 1.1531e+00 1.1 1.22e+09 1.2 2.3e+04 2.6e+04 3.4e+01 3 29 7 13 3 61 60 12 54 25 36519 0 0 0.00e+00 0 0.00e+00 0 >>> 267:MGInterp Level 4 68 1.0 2.0749e-01 1.1 2.23e+08 1.2 1.4e+04 3.4e+03 0.0e+00 0 5 4 1 0 11 11 7 4 0 36925 0 0 0.00e+00 0 0.00e+00 0 >>> >>> ex56-summit-gpu-24ranks-converged.txt >>> 275:MGSmooth Level 4 68 1.0 1.4499e-01 1.2 1.85e+09 1.2 1.0e+04 5.3e+04 3.4e+01 0 29 7 13 3 26 60 12 55 25 299156 940881 115 2.46e+01 116 8.64e+01 100 >>> 277:MGInterp Level 4 68 1.0 1.7674e-01 1.0 3.23e+08 1.2 6.1e+03 6.7e+03 0.0e+00 0 5 4 1 0 33 11 7 4 0 42715 621223 36 2.98e+01 136 3.95e+00 100 >>> >>> ex56-summit-gpu-36ranks-converged.txt >>> 275:MGSmooth Level 4 68 1.0 1.4877e-01 1.2 1.25e+09 1.2 2.3e+04 2.6e+04 3.4e+01 0 29 7 13 3 19 60 12 54 25 291548 719522 115 1.83e+01 116 5.80e+01 100 >>> 277:MGInterp Level 4 68 1.0 2.4317e-01 1.0 2.20e+08 1.2 1.4e+04 3.4e+03 0.0e+00 0 5 4 1 0 33 11 7 4 0 31062 586044 36 1.99e+01 136 2.82e+00 100 >> >> 258:MGSmooth Level 4 68 1.0 9.6950e-01 1.3 6.15e+08 1.3 4.0e+04 1.4e+04 2.0e+00 6 28 10 15 0 59 59 18 54 25 39423 >> 260:MGInterp Level 4 68 1.0 2.5707e-01 1.5 1.23e+08 1.2 2.7e+04 1.9e+03 0.0e+00 1 5 7 1 0 13 12 12 5 0 29294 >> >> Epyc is faster than Power9 is faster than Sklake. >> >>> >>> The Skylake is a lot faster at PtAP. It'd be interesting to better >>> understand why. Perhaps it has to do with caching or aggressiveness of >>> out-of-order execution. >>> >>> $ rg 'PtAP' ex56* >>> ex56-JLSE-skylake-56ranks-converged.txt >>> 164:MatPtAP 4 1.0 1.4214e+00 1.0 3.94e+08 1.5 1.1e+04 7.4e+04 4.4e+01 6 13 3 20 4 8 28 8 39 5 13754 >>> 165:MatPtAPSymbolic 4 1.0 8.3981e-01 1.0 0.00e+00 0.0 6.5e+03 7.3e+04 2.8e+01 4 0 2 12 2 5 0 5 23 3 0 >>> 166:MatPtAPNumeric 4 1.0 5.8402e-01 1.0 3.94e+08 1.5 4.5e+03 7.5e+04 1.6e+01 2 13 1 8 1 3 28 3 16 2 33474 >>> >>> ex56-summit-cpu-36ranks-converged.txt >>> 164:MatPtAP 4 1.0 3.9077e+00 1.0 5.89e+08 1.4 1.6e+04 7.4e+04 4.4e+01 9 13 5 26 4 11 28 12 46 5 4991 0 0 0.00e+00 0 0.00e+00 0 >>> 165:MatPtAPSymbolic 4 1.0 1.9525e+00 1.0 0.00e+00 0.0 1.2e+04 7.3e+04 2.8e+01 5 0 4 19 3 5 0 9 34 3 0 0 0 0.00e+00 0 0.00e+00 0 >>> 166:MatPtAPNumeric 4 1.0 1.9621e+00 1.0 5.89e+08 1.4 4.0e+03 7.5e+04 1.6e+01 5 13 1 7 1 5 28 3 12 2 9940 0 0 0.00e+00 0 0.00e+00 0 >>> >>> ex56-summit-gpu-24ranks-converged.txt >>> 167:MatPtAP 4 1.0 5.7210e+00 1.0 8.48e+08 1.3 7.5e+03 1.3e+05 4.4e+01 8 13 5 25 4 11 28 12 46 5 3415 0 16 3.36e+01 4 6.30e-02 0 >>> 168:MatPtAPSymbolic 4 1.0 2.8717e+00 1.0 0.00e+00 0.0 5.5e+03 1.3e+05 2.8e+01 4 0 4 19 3 5 0 9 34 3 0 0 0 0.00e+00 0 0.00e+00 0 >>> 169:MatPtAPNumeric 4 1.0 2.8537e+00 1.0 8.48e+08 1.3 2.0e+03 1.3e+05 1.6e+01 4 13 1 7 1 5 28 3 12 2 6846 0 16 3.36e+01 4 6.30e-02 0 >>> >>> ex56-summit-gpu-36ranks-converged.txt >>> 167:MatPtAP 4 1.0 4.0340e+00 1.0 5.89e+08 1.4 1.6e+04 7.4e+04 4.4e+01 8 13 5 26 4 11 28 12 46 5 4835 0 16 2.30e+01 4 5.18e-02 0 >>> 168:MatPtAPSymbolic 4 1.0 2.0355e+00 1.0 0.00e+00 0.0 1.2e+04 7.3e+04 2.8e+01 4 0 4 19 3 5 0 9 34 3 0 0 0 0.00e+00 0 0.00e+00 0 >>> 169:MatPtAPNumeric 4 1.0 2.0050e+00 1.0 5.89e+08 1.4 4.0e+03 7.5e+04 1.6e+01 4 13 1 7 1 5 28 3 12 2 9728 0 16 2.30e+01 4 5.18e-02 0 >> >> 153:MatPtAPSymbolic 4 1.0 7.6053e-01 1.0 0.00e+00 0.0 7.6e+03 5.8e+04 2.8e+01 5 0 2 12 2 6 0 5 22 3 0 >> 154:MatPtAPNumeric 4 1.0 6.5172e-01 1.0 3.21e+08 1.4 6.4e+03 4.8e+04 2.4e+01 4 14 2 8 2 5 27 4 16 2 28861 >> >> Epyc similar to Skylake here. >> >>> I'd really like to compare an EPYC for these operations. I bet it's >>> pretty good. (More bandwidth than Skylake, bigger caches, but no >>> AVX512.) >>> >>>> So the biggest consumer is MatPtAP I guess that should be done first. >>>> >>>> It would be good to have these results exclude the Jacobian and Function evaluation which really dominate the time and add clutter making it difficult to see the problems with the rest of SNESSolve. >>>> >>>> >>>> Did you notice: >>>> >>>> MGInterp Level 4 68 1.0 1.7674e-01 1.0 3.23e+08 1.2 6.1e+03 6.7e+03 0.0e+00 0 5 4 1 0 33 11 7 4 0 42715 621223 36 2.98e+01 136 3.95e+00 100 >>>> >>>> it is terrible! Well over half of the KSPSolve time is in this one relatively minor routine. All of the interps are terribly slow. Is it related to the transpose multiple or something? >>> >>> Yes, it's definitely the MatMultTranspose, which must be about 3x more >>> expensive than restriction even on the CPU. PCMG/PCGAMG should >>> explicitly transpose (unless the user sets an option to aggressively >>> minimize memory usage). >>> >>> $ rg 'MGInterp|MultTrans' ex56* >>> ex56-JLSE-skylake-56ranks-converged.txt >>> 222:MatMultTranspose 136 1.0 3.5105e-01 3.7 7.91e+07 1.3 2.5e+04 1.3e+03 0.0e+00 1 3 7 1 0 5 6 13 3 0 11755 >>> 247:MGInterp Level 1 68 1.0 3.3894e-04 2.2 2.35e+05 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 693 >>> 250:MGInterp Level 2 68 1.0 1.1212e-0278.0 1.17e+06 0.0 1.8e+03 7.7e+02 0.0e+00 0 0 1 0 0 0 0 1 0 0 2172 >>> 253:MGInterp Level 3 68 1.0 6.7105e-02 5.3 1.23e+07 1.8 2.7e+04 4.2e+02 0.0e+00 0 0 8 0 0 1 1 14 1 0 8594 >>> 256:MGInterp Level 4 68 1.0 4.0043e-01 1.8 1.45e+08 1.3 2.2e+04 2.5e+03 0.0e+00 1 5 6 1 0 9 11 11 4 0 19109 >>> >>> ex56-summit-cpu-36ranks-converged.txt >>> 229:MatMultTranspose 136 1.0 1.4832e-01 1.4 1.21e+08 1.2 1.9e+04 1.5e+03 0.0e+00 0 3 6 1 0 6 6 10 3 0 27842 0 0 0.00e+00 0 0.00e+00 0 >>> 258:MGInterp Level 1 68 1.0 2.9145e-04 1.5 1.08e+05 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 370 0 0 0.00e+00 0 0.00e+00 0 >>> 261:MGInterp Level 2 68 1.0 5.7095e-03 1.5 9.16e+05 2.5 2.4e+03 7.1e+02 0.0e+00 0 0 1 0 0 0 0 1 0 0 4093 0 0 0.00e+00 0 0.00e+00 0 >>> 264:MGInterp Level 3 68 1.0 3.5654e-02 2.8 1.77e+07 1.5 2.3e+04 3.9e+02 0.0e+00 0 0 7 0 0 1 1 12 1 0 16095 0 0 0.00e+00 0 0.00e+00 0 >>> 267:MGInterp Level 4 68 1.0 2.0749e-01 1.1 2.23e+08 1.2 1.4e+04 3.4e+03 0.0e+00 0 5 4 1 0 11 11 7 4 0 36925 0 0 0.00e+00 0 0.00e+00 0 >>> >>> ex56-summit-gpu-24ranks-converged.txt >>> 236:MatMultTranspose 136 1.0 2.1445e-01 1.0 1.72e+08 1.2 9.5e+03 2.6e+03 0.0e+00 0 3 6 1 0 39 6 11 3 0 18719 451131 8 3.11e+01 272 2.19e+00 100 >>> 268:MGInterp Level 1 68 1.0 4.0388e-03 2.8 1.08e+05 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 27 79 37 5.84e-04 68 6.80e-05 100 >>> 271:MGInterp Level 2 68 1.0 2.9033e-02 2.9 1.25e+06 1.9 1.6e+03 7.8e+02 0.0e+00 0 0 1 0 0 5 0 2 0 0 812 11539 36 1.14e-01 136 5.41e-02 100 >>> 274:MGInterp Level 3 68 1.0 4.9503e-02 1.1 2.50e+07 1.4 1.1e+04 6.3e+02 0.0e+00 0 0 7 0 0 9 1 13 1 0 11476 100889 36 2.29e+00 136 3.74e-01 100 >>> 277:MGInterp Level 4 68 1.0 1.7674e-01 1.0 3.23e+08 1.2 6.1e+03 6.7e+03 0.0e+00 0 5 4 1 0 33 11 7 4 0 42715 621223 36 2.98e+01 136 3.95e+00 100 >>> >>> ex56-summit-gpu-36ranks-converged.txt >>> 236:MatMultTranspose 136 1.0 2.9692e-01 1.0 1.17e+08 1.2 1.9e+04 1.5e+03 0.0e+00 1 3 6 1 0 40 6 10 3 0 13521 336701 8 2.08e+01 272 1.59e+00 100 >>> 268:MGInterp Level 1 68 1.0 3.8752e-03 2.5 1.03e+05 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 27 79 37 3.95e-04 68 4.53e-05 100 >>> 271:MGInterp Level 2 68 1.0 3.5465e-02 2.2 9.12e+05 2.5 2.4e+03 7.1e+02 0.0e+00 0 0 1 0 0 4 0 1 0 0 655 5989 36 8.16e-02 136 4.89e-02 100 >>> 274:MGInterp Level 3 68 1.0 6.7101e-02 1.1 1.75e+07 1.5 2.3e+04 3.9e+02 0.0e+00 0 0 7 0 0 9 1 12 1 0 8455 56175 36 1.55e+00 136 3.03e-01 100 >>> 277:MGInterp Level 4 68 1.0 2.4317e-01 1.0 2.20e+08 1.2 1.4e+04 3.4e+03 0.0e+00 0 5 4 1 0 33 11 7 4 0 31062 586044 36 1.99e+01 136 2.82e+00 100 >> >> 223:MatMultTranspose 136 1.0 2.0702e-01 2.9 6.59e+07 1.2 2.7e+04 1.1e+03 0.0e+00 1 3 7 1 0 7 6 12 3 0 19553 >> 251:MGInterp Level 1 68 1.0 2.8062e-04 1.5 9.79e+04 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 349 >> 254:MGInterp Level 2 68 1.0 6.2506e-0331.9 9.69e+05 0.0 2.1e+03 6.3e+02 0.0e+00 0 0 1 0 0 0 0 1 0 0 3458 >> 257:MGInterp Level 3 68 1.0 4.8159e-02 6.5 9.62e+06 1.5 2.5e+04 4.2e+02 0.0e+00 0 0 6 0 0 1 1 11 1 0 11199 >> 260:MGInterp Level 4 68 1.0 2.5707e-01 1.5 1.23e+08 1.2 2.7e+04 1.9e+03 0.0e+00 1 5 7 1 0 13 12 12 5 0 29294 >> >> Power9 still has an edge here. From jed at jedbrown.org Sat Jun 13 00:55:28 2020 From: jed at jedbrown.org (Jed Brown) Date: Fri, 12 Jun 2020 23:55:28 -0600 Subject: [petsc-users] Implementing periodicity using DMPlex In-Reply-To: References: <0cd9d36f-b897-4e7c-ac83-0908a93f0b8e@email.android.com> Message-ID: <87k10b4ii7.fsf@jedbrown.org> Please always use "reply-all" so that your messages go to the list. This is standard mailing list etiquette. It is important to preserve threading for people who find this discussion later and so that we do not waste our time re-answering the same questions that have already been answered in private side-conversations. You'll likely get an answer faster that way too. Shashwat Tiwari writes: > Yeah I have use that too. I am getting zero values even in the Local Vector. Corresponding to the ghost cells you created for that purpose, or otherwise? > On Sat, 13 Jun 2020, 12:57 am , wrote: > >> You need DMGlobalToLocal. >> >> VecGhost is a different thing. >> >> On Jun 12, 2020 13:17, Shashwat Tiwari wrote: >> >> Hi, >> I am writing a first order 2D solver for unstructured grids with periodic >> boundaries using DMPlex. After generating the mesh, I use >> "DMSetPeriodicity" function to set periodicity in both directions. After >> which I partition the mesh (DMPlexDistribute), construct ghost cells >> (DMPlexConstructGhostCells), create a section, and set some initial values >> in the global vector. Then I use "VecGhostUpdateBegin" to start updating >> the boundary ghost cell values, but, I get the following error in case I >> use multiple processors: >> >> [0]PETSC ERROR: Invalid argument >> [0]PETSC ERROR: Vector is not ghosted >> [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html >> for trouble shooting. >> >> if I run with a single process, there is no error but the values remain >> empty (zero) and are not updated. Kindly let me know, if I am missing some >> crucial step before I can update the ghost values in order to implement the >> periodic bc, or if there is any other approach to achieve it. I am >> attaching a small code to demonstrate the issue for your reference. >> >> Regards, >> Shashwat >> >> >> From s_g at berkeley.edu Sat Jun 13 01:30:57 2020 From: s_g at berkeley.edu (Sanjay Govindjee) Date: Fri, 12 Jun 2020 23:30:57 -0700 Subject: [petsc-users] Debugging suggestions: GAMG Message-ID: I have a FEA problem that I am trying to solve with GAMG.? The problem solves just fine with direct solvers (mumps, superlu) and iterative solvers (gmres, ml, hypre-boomer) etc. However with GAMG I am getting a divide by zero that I am having trouble tracking down.? Below is the gdb stack trace and the source lines going up the stack. When I run in valgrind the problem runs fine (and gets the correct answer). Valgrind reports nothing of note (just lots of indirectly lost blocks? related to PMP_INIT). I'm only running on one processor. Any suggestions on where to start to trace the problem? -sanjay #0? 0x00007fb262dc5be1 in ieeeck_ () from /lib64/liblapack.so.3 #1? 0x00007fb262dc5332 in ilaenv_ () from /lib64/liblapack.so.3 #2? 0x00007fb262dbbcef in dlasq2_ () from /lib64/liblapack.so.3 #3? 0x00007fb262dbb78c in dlasq1_ () from /lib64/liblapack.so.3 #4? 0x00007fb262da1e2e in dbdsqr_ () from /lib64/liblapack.so.3 #5? 0x00007fb262960110 in dgesvd_ () from /lib64/liblapack.so.3 #6? 0x00007fb264e74b66 in KSPComputeExtremeSingularValues_GMRES (ksp=0x1816560, emax=0x7ffc5010e7c8, emin=0x7ffc5010e7d0) at /home/sg/petsc-3.13.2/src/ksp/ksp/impls/gmres/gmreig.c:32 #7? 0x00007fb264dfe69a in KSPComputeExtremeSingularValues (ksp=0x1816560, emax=0x7ffc5010e7c8, emin=0x7ffc5010e7d0) at /home/sg/petsc-3.13.2/src/ksp/ksp/interface/itfunc.c:64 #8? 0x00007fb264b44a1f in PCGAMGOptProlongator_AGG (pc=0x12f3d30, Amat=0x11a2630, a_P=0x7ffc5010ebe0) at /home/sg/petsc-3.13.2/src/ksp/pc/impls/gamg/agg.c:1145 #9? 0x00007fb264b248a1 in PCSetUp_GAMG (pc=0x12f3d30) at /home/sg/petsc-3.13.2/src/ksp/pc/impls/gamg/gamg.c:557 #10 0x00007fb264d8535b in PCSetUp (pc=0x12f3d30) at /home/sg/petsc-3.13.2/src/ksp/pc/interface/precon.c:898 #11 0x00007fb264e01a93 in KSPSetUp (ksp=0x128dd80) at /home/sg/petsc-3.13.2/src/ksp/ksp/interface/itfunc.c:376 #12 0x00007fb264e057af in KSPSolve_Private (ksp=0x128dd80, b=0x1259f30, x=0x125d910) at /home/sg/petsc-3.13.2/src/ksp/ksp/interface/itfunc.c:633 #13 0x00007fb264e086b9 in KSPSolve (ksp=0x128dd80, b=0x1259f30, x=0x125d910) at /home/sg/petsc-3.13.2/src/ksp/ksp/interface/itfunc.c:853 #14 0x00007fb264e46216 in kspsolve_ (ksp=0x832670 <__pfeapc_MOD_kspsol>, b=0x832698 <__pfeapc_MOD_rhs>, x=0x8326a0 <__pfeapc_MOD_sol>, __ierr=0x7ffc5010f358) ??? at /home/sg/petsc-3.13.2/src/ksp/ksp/interface/ftn-auto/itfuncf.c:266 #15 0x000000000043298d in usolve (flags=..., b=...) at usolve.F:313 #16 0x000000000044afba in psolve (stype=-3, b=..., fp=..., factor=.TRUE., solve=.TRUE., cfr=.FALSE., prnt=.TRUE.) at psolve.f:212 #17 0x00000000006b7393 in pmacr1 (lct=..., ct=..., j=3, _lct=_lct at entry=15) at pmacr1.f:578 #18 0x00000000005c247b in pmacr (initf=.FALSE.) at pmacr.f:578 #19 0x000000000044ff20 in pcontr () at pcontr.f:1307 #20 0x0000000000404d9b in feap () at feap86.f:162 #21 main (argc=, argv=) at feap86.f:168 #22 0x00007fb261aaef43 in __libc_start_main () from /lib64/libc.so.6 #23 0x0000000000404dde in _start () (gdb) list 1?????? : No such file or directory. (gdb) up #1? 0x00007fb262dc5332 in ilaenv_ () from /lib64/liblapack.so.3 (gdb) up #2? 0x00007fb262dbbcef in dlasq2_ () from /lib64/liblapack.so.3 (gdb) up #3? 0x00007fb262dbb78c in dlasq1_ () from /lib64/liblapack.so.3 (gdb) up #4? 0x00007fb262da1e2e in dbdsqr_ () from /lib64/liblapack.so.3 (gdb) up #5? 0x00007fb262960110 in dgesvd_ () from /lib64/liblapack.so.3 (gdb) up #6? 0x00007fb264e74b66 in KSPComputeExtremeSingularValues_GMRES (ksp=0x1816560, emax=0x7ffc5010e7c8, emin=0x7ffc5010e7d0) at /home/sg/petsc-3.13.2/src/ksp/ksp/impls/gmres/gmreig.c:32 32 PetscStackCallBLAS("LAPACKgesvd",LAPACKgesvd_("N","N",&bn,&bn,R,&bN,realpart,&sdummy,&idummy,&sdummy,&idummy,work,&lwork,&lierr)); (gdb) up #7? 0x00007fb264dfe69a in KSPComputeExtremeSingularValues (ksp=0x1816560, emax=0x7ffc5010e7c8, emin=0x7ffc5010e7d0) at /home/sg/petsc-3.13.2/src/ksp/ksp/interface/itfunc.c:64 64????????? ierr = (*ksp->ops->computeextremesingularvalues)(ksp,emax,emin);CHKERRQ(ierr); (gdb) up #8? 0x00007fb264b44a1f in PCGAMGOptProlongator_AGG (pc=0x12f3d30, Amat=0x11a2630, a_P=0x7ffc5010ebe0) at /home/sg/petsc-3.13.2/src/ksp/pc/impls/gamg/agg.c:1145 1145????????? ierr = KSPComputeExtremeSingularValues(eksp, &emax, &emin);CHKERRQ(ierr); (gdb) up #9? 0x00007fb264b248a1 in PCSetUp_GAMG (pc=0x12f3d30) at /home/sg/petsc-3.13.2/src/ksp/pc/impls/gamg/gamg.c:557 557?????????????? ierr = pc_gamg->ops->optprolongator(pc, Aarr[level], &Prol11);CHKERRQ(ierr); (gdb) up #10 0x00007fb264d8535b in PCSetUp (pc=0x12f3d30) at /home/sg/petsc-3.13.2/src/ksp/pc/interface/precon.c:898 898???????? ierr = (*pc->ops->setup)(pc);CHKERRQ(ierr); (gdb) up #11 0x00007fb264e01a93 in KSPSetUp (ksp=0x128dd80) at /home/sg/petsc-3.13.2/src/ksp/ksp/interface/itfunc.c:376 376?????? ierr = PCSetUp(ksp->pc);CHKERRQ(ierr); (gdb) up #12 0x00007fb264e057af in KSPSolve_Private (ksp=0x128dd80, b=0x1259f30, x=0x125d910) at /home/sg/petsc-3.13.2/src/ksp/ksp/interface/itfunc.c:633 633?????? ierr = KSPSetUp(ksp);CHKERRQ(ierr); (gdb) up #13 0x00007fb264e086b9 in KSPSolve (ksp=0x128dd80, b=0x1259f30, x=0x125d910) at /home/sg/petsc-3.13.2/src/ksp/ksp/interface/itfunc.c:853 853?????? ierr = KSPSolve_Private(ksp,b,x);CHKERRQ(ierr); (gdb) up #14 0x00007fb264e46216 in kspsolve_ (ksp=0x832670 <__pfeapc_MOD_kspsol>, b=0x832698 <__pfeapc_MOD_rhs>, x=0x8326a0 <__pfeapc_MOD_sol>, __ierr=0x7ffc5010f358) ??? at /home/sg/petsc-3.13.2/src/ksp/ksp/interface/ftn-auto/itfuncf.c:266 266???? *__ierr = KSPSolve( (gdb) up #15 0x000000000043298d in usolve (flags=..., b=...) at usolve.F:313 313?????????????? call KSPSolve???????? (kspsol, rhs, sol, ierr) -------------- next part -------------- An HTML attachment was scrubbed... URL: From prateekgupta1709 at gmail.com Sat Jun 13 03:41:35 2020 From: prateekgupta1709 at gmail.com (Prateek Gupta) Date: Sat, 13 Jun 2020 10:41:35 +0200 Subject: [petsc-users] Adaptive remeshing/refining using DMPlex Message-ID: Hi, I intend to implement adaptive remeshing and refining in my FE code. Currently, I am not using the DMPlex class of PETSc. As anyone would be able to guess, renumbering of mesh nodes and then , in some cases, load balancing via repartitioning requires a lot of book keeping. 1. I am wondering if DMPlex has built in functions which could be used to automatically renumber the mesh upon operations like edge-flipping in tetrahedra? 2. If not, then does DAG representation of the grid offer any ease/performance benefits, if I had to go down and implement? 3. Upon adaptive refining and coarsening, is there a way to dynamically update my global solution vectors (defined on nodes)? Or do I have to destroy old vectors and reallocate? Any help/pointers are greatly appreciated :) Thank you. Sincerely, Prateek Gupta, PhD D-MAVT, ETHZ -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Sat Jun 13 07:10:48 2020 From: mfadams at lbl.gov (Mark Adams) Date: Sat, 13 Jun 2020 08:10:48 -0400 Subject: [petsc-users] Debugging suggestions: GAMG In-Reply-To: References: Message-ID: That is odd. Are these problems symmetric positive definite? Eigen estimates are a pain in practice but I've never seen this. Hypre has (better) smoothers that don't need this and the AMG algorithm does not need them either. I think ML does pretty much the same thing as me. If SPD then you definitely want '-pc_gamg_esteig_ksp_type cg'. CG converges faster and is more robust. Even if not strictly SPD like with large deformation or plasticity CG is probably better. You can run with -info and grep on GAMG to see what eigen (high) estimates it came up with for each level. They should be >1 and < 4, or so. I wonder if your LAPACK is funky. You might try a fresh build (delete the "arch" directory) and download LAPACK, but this problem will go away with CG. Note, if you use Chebyshev smoothing then it needs eigen estimates also. THis is usually where eigen estimate problems come up. If you use jacobi PC in the smoother, GAMG will use the eigen estimate from the smoothed aggregation smoothing (what is failing for you here). -pc_gamg_use_sa_esteig will force it to use, or not use, it. For instance, if you sor/ilu/asm then the eigen estimates are probably fine to reuse but GAMG does not by default. If the cheby eigen estimates are low then the solver fails. THere is a safety factor to increase the eigen estimate in chychev to avoid this failure. And, -pc_gamg_esteig_ksp_max_it X sets the number of iterations in the estimator. You can increase this and you should see your eigen estimates go up and converge. You should see that CG converges much faster than GMRES (the default). On Sat, Jun 13, 2020 at 2:32 AM Sanjay Govindjee wrote: > I have a FEA problem that I am trying to solve with GAMG. The problem > solves > just fine with direct solvers (mumps, superlu) and iterative solvers > (gmres, ml, hypre-boomer) etc. > > However with GAMG I am getting a divide by zero that I am having trouble > tracking down. Below > is the gdb stack trace and the source lines going up the stack. > > When I run in valgrind the problem runs fine (and gets the correct answer). > Valgrind reports nothing of note (just lots of indirectly lost blocks > related to PMP_INIT). > > I'm only running on one processor. > > Any suggestions on where to start to trace the problem? > > -sanjay > > #0 0x00007fb262dc5be1 in ieeeck_ () from /lib64/liblapack.so.3 > #1 0x00007fb262dc5332 in ilaenv_ () from /lib64/liblapack.so.3 > #2 0x00007fb262dbbcef in dlasq2_ () from /lib64/liblapack.so.3 > #3 0x00007fb262dbb78c in dlasq1_ () from /lib64/liblapack.so.3 > #4 0x00007fb262da1e2e in dbdsqr_ () from /lib64/liblapack.so.3 > #5 0x00007fb262960110 in dgesvd_ () from /lib64/liblapack.so.3 > #6 0x00007fb264e74b66 in KSPComputeExtremeSingularValues_GMRES > (ksp=0x1816560, emax=0x7ffc5010e7c8, emin=0x7ffc5010e7d0) at > /home/sg/petsc-3.13.2/src/ksp/ksp/impls/gmres/gmreig.c:32 > #7 0x00007fb264dfe69a in KSPComputeExtremeSingularValues (ksp=0x1816560, > emax=0x7ffc5010e7c8, emin=0x7ffc5010e7d0) at > /home/sg/petsc-3.13.2/src/ksp/ksp/interface/itfunc.c:64 > #8 0x00007fb264b44a1f in PCGAMGOptProlongator_AGG (pc=0x12f3d30, > Amat=0x11a2630, a_P=0x7ffc5010ebe0) at > /home/sg/petsc-3.13.2/src/ksp/pc/impls/gamg/agg.c:1145 > #9 0x00007fb264b248a1 in PCSetUp_GAMG (pc=0x12f3d30) at > /home/sg/petsc-3.13.2/src/ksp/pc/impls/gamg/gamg.c:557 > #10 0x00007fb264d8535b in PCSetUp (pc=0x12f3d30) at > /home/sg/petsc-3.13.2/src/ksp/pc/interface/precon.c:898 > #11 0x00007fb264e01a93 in KSPSetUp (ksp=0x128dd80) at > /home/sg/petsc-3.13.2/src/ksp/ksp/interface/itfunc.c:376 > #12 0x00007fb264e057af in KSPSolve_Private (ksp=0x128dd80, b=0x1259f30, > x=0x125d910) at /home/sg/petsc-3.13.2/src/ksp/ksp/interface/itfunc.c:633 > #13 0x00007fb264e086b9 in KSPSolve (ksp=0x128dd80, b=0x1259f30, > x=0x125d910) at /home/sg/petsc-3.13.2/src/ksp/ksp/interface/itfunc.c:853 > #14 0x00007fb264e46216 in kspsolve_ (ksp=0x832670 <__pfeapc_MOD_kspsol>, > b=0x832698 <__pfeapc_MOD_rhs>, x=0x8326a0 <__pfeapc_MOD_sol>, > __ierr=0x7ffc5010f358) > at /home/sg/petsc-3.13.2/src/ksp/ksp/interface/ftn-auto/itfuncf.c:266 > #15 0x000000000043298d in usolve (flags=..., b=...) at usolve.F:313 > #16 0x000000000044afba in psolve (stype=-3, b=..., fp=..., factor=.TRUE., > solve=.TRUE., cfr=.FALSE., prnt=.TRUE.) at psolve.f:212 > #17 0x00000000006b7393 in pmacr1 (lct=..., ct=..., j=3, _lct=_lct at entry=15) > at pmacr1.f:578 > #18 0x00000000005c247b in pmacr (initf=.FALSE.) at pmacr.f:578 > #19 0x000000000044ff20 in pcontr () at pcontr.f:1307 > #20 0x0000000000404d9b in feap () at feap86.f:162 > #21 main (argc=, argv=) at feap86.f:168 > #22 0x00007fb261aaef43 in __libc_start_main () from /lib64/libc.so.6 > #23 0x0000000000404dde in _start () > > (gdb) list > 1 : No such file or directory. > (gdb) up > #1 0x00007fb262dc5332 in ilaenv_ () from /lib64/liblapack.so.3 > (gdb) up > #2 0x00007fb262dbbcef in dlasq2_ () from /lib64/liblapack.so.3 > (gdb) up > #3 0x00007fb262dbb78c in dlasq1_ () from /lib64/liblapack.so.3 > (gdb) up > #4 0x00007fb262da1e2e in dbdsqr_ () from /lib64/liblapack.so.3 > (gdb) up > #5 0x00007fb262960110 in dgesvd_ () from /lib64/liblapack.so.3 > (gdb) up > #6 0x00007fb264e74b66 in KSPComputeExtremeSingularValues_GMRES > (ksp=0x1816560, emax=0x7ffc5010e7c8, emin=0x7ffc5010e7d0) at > /home/sg/petsc-3.13.2/src/ksp/ksp/impls/gmres/gmreig.c:32 > 32 > PetscStackCallBLAS("LAPACKgesvd",LAPACKgesvd_("N","N",&bn,&bn,R,&bN,realpart,&sdummy,&idummy,&sdummy,&idummy,work,&lwork,&lierr)); > (gdb) up > #7 0x00007fb264dfe69a in KSPComputeExtremeSingularValues (ksp=0x1816560, > emax=0x7ffc5010e7c8, emin=0x7ffc5010e7d0) at > /home/sg/petsc-3.13.2/src/ksp/ksp/interface/itfunc.c:64 > 64 ierr = > (*ksp->ops->computeextremesingularvalues)(ksp,emax,emin);CHKERRQ(ierr); > (gdb) up > #8 0x00007fb264b44a1f in PCGAMGOptProlongator_AGG (pc=0x12f3d30, > Amat=0x11a2630, a_P=0x7ffc5010ebe0) at > /home/sg/petsc-3.13.2/src/ksp/pc/impls/gamg/agg.c:1145 > 1145 ierr = KSPComputeExtremeSingularValues(eksp, &emax, > &emin);CHKERRQ(ierr); > (gdb) up > #9 0x00007fb264b248a1 in PCSetUp_GAMG (pc=0x12f3d30) at > /home/sg/petsc-3.13.2/src/ksp/pc/impls/gamg/gamg.c:557 > 557 ierr = pc_gamg->ops->optprolongator(pc, Aarr[level], > &Prol11);CHKERRQ(ierr); > (gdb) up > #10 0x00007fb264d8535b in PCSetUp (pc=0x12f3d30) at > /home/sg/petsc-3.13.2/src/ksp/pc/interface/precon.c:898 > 898 ierr = (*pc->ops->setup)(pc);CHKERRQ(ierr); > (gdb) up > #11 0x00007fb264e01a93 in KSPSetUp (ksp=0x128dd80) at > /home/sg/petsc-3.13.2/src/ksp/ksp/interface/itfunc.c:376 > 376 ierr = PCSetUp(ksp->pc);CHKERRQ(ierr); > (gdb) up > #12 0x00007fb264e057af in KSPSolve_Private (ksp=0x128dd80, b=0x1259f30, > x=0x125d910) at /home/sg/petsc-3.13.2/src/ksp/ksp/interface/itfunc.c:633 > 633 ierr = KSPSetUp(ksp);CHKERRQ(ierr); > (gdb) up > #13 0x00007fb264e086b9 in KSPSolve (ksp=0x128dd80, b=0x1259f30, > x=0x125d910) at /home/sg/petsc-3.13.2/src/ksp/ksp/interface/itfunc.c:853 > 853 ierr = KSPSolve_Private(ksp,b,x);CHKERRQ(ierr); > (gdb) up > #14 0x00007fb264e46216 in kspsolve_ (ksp=0x832670 <__pfeapc_MOD_kspsol>, > b=0x832698 <__pfeapc_MOD_rhs>, x=0x8326a0 <__pfeapc_MOD_sol>, > __ierr=0x7ffc5010f358) > at /home/sg/petsc-3.13.2/src/ksp/ksp/interface/ftn-auto/itfuncf.c:266 > 266 *__ierr = KSPSolve( > (gdb) up > #15 0x000000000043298d in usolve (flags=..., b=...) at usolve.F:313 > 313 call KSPSolve (kspsol, rhs, sol, ierr) > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Sat Jun 13 07:58:47 2020 From: jed at jedbrown.org (Jed Brown) Date: Sat, 13 Jun 2020 06:58:47 -0600 Subject: [petsc-users] Debugging suggestions: GAMG In-Reply-To: References: Message-ID: <87h7vf3ywo.fsf@jedbrown.org> Mark Adams writes: > That is odd. Are these problems symmetric positive definite? > > Eigen estimates are a pain in practice but I've never seen this. Hypre has > (better) smoothers that don't need this and the AMG algorithm does not need > them either. I think ML does pretty much the same thing as me. We could certainly offer an "l1 Jacobi" for operator smoothing; it's effectively using the Gershgorin bound, though it can be very pessimistic. At a minimum, it would be useful diagnostic information and we could stop the Krylov estimate early if its maxbound (which includes safety and is non-increasing with iteration) exceeds the Gershgorin bound. > If SPD then you definitely want '-pc_gamg_esteig_ksp_type cg'. CG converges > faster and is more robust. Mark, this comes up a lot. Should we select it by default if MAT_SPD is set? > Even if not strictly SPD like with large > deformation or plasticity CG is probably better. You can run with -info and > grep on GAMG to see what eigen (high) estimates it came up with for each > level. They should be >1 and < 4, or so. > > I wonder if your LAPACK is funky. You might try a fresh build (delete the > "arch" directory) and download LAPACK, but this problem will go away with > CG. Sanjay, you could use the debugger to print out the small dense matrix that LAPACK is choking on. It isn't supposed to divide by zero ever. From bsmith at petsc.dev Sat Jun 13 11:04:33 2020 From: bsmith at petsc.dev (Barry Smith) Date: Sat, 13 Jun 2020 11:04:33 -0500 Subject: [petsc-users] Debugging suggestions: GAMG In-Reply-To: References: Message-ID: <24AC8444-94DD-4885-8E04-9107DD7DB673@petsc.dev> The LAPACK routine ieeeck_ intentionally does a divide by zero to check if the system can handle it without generating an exception. It doesn't have anything to do with the particular matrix data passed to LAPACK. In KSPComputeExtremeSingularValues_GMRES() we have the code structure ierr = PetscFPTrapPush(PETSC_FP_TRAP_OFF);CHKERRQ(ierr); #if !defined(PETSC_USE_COMPLEX) PetscStackCallBLAS("LAPACKgesvd",LAPACKgesvd_("N","N",&bn,&bn,R,&bN,realpart,&sdummy,&idummy,&sdummy,&idummy,work,&lwork,&lierr)); #else PetscStackCallBLAS("LAPACKgesvd",LAPACKgesvd_("N","N",&bn,&bn,R,&bN,realpart,&sdummy,&idummy,&sdummy,&idummy,work,&lwork,realpart+N,&lierr)); #endif if (lierr) SETERRQ1(PETSC_COMM_SELF,PETSC_ERR_LIB,"Error in SVD Lapack routine %d",(int)lierr); ierr = PetscFPTrapPop();CHKERRQ(ierr); So PETSc tries to turn off trapping of floating point exceptions before calling the LAPACK routines that eventually lead to the exception. PetscErrorCode PetscFPTrapPush(PetscFPTrap trap) { PetscErrorCode ierr; struct PetscFPTrapLink *link; PetscFunctionBegin; ierr = PetscNew(&link);CHKERRQ(ierr); link->trapmode = _trapmode; link->next = _trapstack; _trapstack = link; if (trap != _trapmode) {ierr = PetscSetFPTrap(trap);CHKERRQ(ierr);} PetscFunctionReturn(0); } PetscErrorCode PetscSetFPTrap(PetscFPTrap flag) { char *out; PetscFunctionBegin; /* Clear accumulated exceptions. Used to suppress meaningless messages from f77 programs */ (void) ieee_flags("clear","exception","all",&out); if (flag == PETSC_FP_TRAP_ON) { /* To trap more fp exceptions, including underflow, change the line below to if (ieee_handler("set","all",PetscDefaultFPTrap)) { */ if (ieee_handler("set","common",PetscDefaultFPTrap)) (*PetscErrorPrintf)("Can't set floatingpoint handler\n"); } else if (ieee_handler("clear","common",PetscDefaultFPTrap)) (*PetscErrorPrintf)("Can't clear floatingpoint handler\n"); _trapmode = flag; PetscFunctionReturn(0); } So either the ieee_handler clear is not working for your system or some other code, AFTER PETSc calls ieee_handler sets the ieee_handler to trap divide by zero. A git grep -i ieee_handler shows that the reference BLAS/LAPACK and OpenBLAS never seem to call the ieee_handler. We need to know what lapack/blas you are using and how they were compiled. Some Fortran compilers/linkers set nonstandard exception handlers, but since PETSc clears them I don't know how they could get set again You could try in gdb to put a break point in ieee_handler and find all the places it gets called, maybe this will lead to the location of the cause. Barry > On Jun 13, 2020, at 1:30 AM, Sanjay Govindjee wrote: > > I have a FEA problem that I am trying to solve with GAMG. The problem solves > just fine with direct solvers (mumps, superlu) and iterative solvers (gmres, ml, hypre-boomer) etc. > > However with GAMG I am getting a divide by zero that I am having trouble tracking down. Below > is the gdb stack trace and the source lines going up the stack. > > When I run in valgrind the problem runs fine (and gets the correct answer). > Valgrind reports nothing of note (just lots of indirectly lost blocks related to PMP_INIT). > > I'm only running on one processor. > > Any suggestions on where to start to trace the problem? > > -sanjay > > #0 0x00007fb262dc5be1 in ieeeck_ () from /lib64/liblapack.so.3 > #1 0x00007fb262dc5332 in ilaenv_ () from /lib64/liblapack.so.3 > #2 0x00007fb262dbbcef in dlasq2_ () from /lib64/liblapack.so.3 > #3 0x00007fb262dbb78c in dlasq1_ () from /lib64/liblapack.so.3 > #4 0x00007fb262da1e2e in dbdsqr_ () from /lib64/liblapack.so.3 > #5 0x00007fb262960110 in dgesvd_ () from /lib64/liblapack.so.3 > #6 0x00007fb264e74b66 in KSPComputeExtremeSingularValues_GMRES (ksp=0x1816560, emax=0x7ffc5010e7c8, emin=0x7ffc5010e7d0) at /home/sg/petsc-3.13.2/src/ksp/ksp/impls/gmres/gmreig.c:32 > #7 0x00007fb264dfe69a in KSPComputeExtremeSingularValues (ksp=0x1816560, emax=0x7ffc5010e7c8, emin=0x7ffc5010e7d0) at /home/sg/petsc-3.13.2/src/ksp/ksp/interface/itfunc.c:64 > #8 0x00007fb264b44a1f in PCGAMGOptProlongator_AGG (pc=0x12f3d30, Amat=0x11a2630, a_P=0x7ffc5010ebe0) at /home/sg/petsc-3.13.2/src/ksp/pc/impls/gamg/agg.c:1145 > #9 0x00007fb264b248a1 in PCSetUp_GAMG (pc=0x12f3d30) at /home/sg/petsc-3.13.2/src/ksp/pc/impls/gamg/gamg.c:557 > #10 0x00007fb264d8535b in PCSetUp (pc=0x12f3d30) at /home/sg/petsc-3.13.2/src/ksp/pc/interface/precon.c:898 > #11 0x00007fb264e01a93 in KSPSetUp (ksp=0x128dd80) at /home/sg/petsc-3.13.2/src/ksp/ksp/interface/itfunc.c:376 > #12 0x00007fb264e057af in KSPSolve_Private (ksp=0x128dd80, b=0x1259f30, x=0x125d910) at /home/sg/petsc-3.13.2/src/ksp/ksp/interface/itfunc.c:633 > #13 0x00007fb264e086b9 in KSPSolve (ksp=0x128dd80, b=0x1259f30, x=0x125d910) at /home/sg/petsc-3.13.2/src/ksp/ksp/interface/itfunc.c:853 > #14 0x00007fb264e46216 in kspsolve_ (ksp=0x832670 <__pfeapc_MOD_kspsol>, b=0x832698 <__pfeapc_MOD_rhs>, x=0x8326a0 <__pfeapc_MOD_sol>, __ierr=0x7ffc5010f358) > at /home/sg/petsc-3.13.2/src/ksp/ksp/interface/ftn-auto/itfuncf.c:266 > #15 0x000000000043298d in usolve (flags=..., b=...) at usolve.F:313 > #16 0x000000000044afba in psolve (stype=-3, b=..., fp=..., factor=.TRUE., solve=.TRUE., cfr=.FALSE., prnt=.TRUE.) at psolve.f:212 > #17 0x00000000006b7393 in pmacr1 (lct=..., ct=..., j=3, _lct=_lct at entry=15) at pmacr1.f:578 > #18 0x00000000005c247b in pmacr (initf=.FALSE.) at pmacr.f:578 > #19 0x000000000044ff20 in pcontr () at pcontr.f:1307 > #20 0x0000000000404d9b in feap () at feap86.f:162 > #21 main (argc=, argv=) at feap86.f:168 > #22 0x00007fb261aaef43 in __libc_start_main () from /lib64/libc.so.6 > #23 0x0000000000404dde in _start () > > (gdb) list > 1 : No such file or directory. > (gdb) up > #1 0x00007fb262dc5332 in ilaenv_ () from /lib64/liblapack.so.3 > (gdb) up > #2 0x00007fb262dbbcef in dlasq2_ () from /lib64/liblapack.so.3 > (gdb) up > #3 0x00007fb262dbb78c in dlasq1_ () from /lib64/liblapack.so.3 > (gdb) up > #4 0x00007fb262da1e2e in dbdsqr_ () from /lib64/liblapack.so.3 > (gdb) up > #5 0x00007fb262960110 in dgesvd_ () from /lib64/liblapack.so.3 > (gdb) up > #6 0x00007fb264e74b66 in KSPComputeExtremeSingularValues_GMRES (ksp=0x1816560, emax=0x7ffc5010e7c8, emin=0x7ffc5010e7d0) at /home/sg/petsc-3.13.2/src/ksp/ksp/impls/gmres/gmreig.c:32 > 32 PetscStackCallBLAS("LAPACKgesvd",LAPACKgesvd_("N","N",&bn,&bn,R,&bN,realpart,&sdummy,&idummy,&sdummy,&idummy,work,&lwork,&lierr)); > (gdb) up > #7 0x00007fb264dfe69a in KSPComputeExtremeSingularValues (ksp=0x1816560, emax=0x7ffc5010e7c8, emin=0x7ffc5010e7d0) at /home/sg/petsc-3.13.2/src/ksp/ksp/interface/itfunc.c:64 > 64 ierr = (*ksp->ops->computeextremesingularvalues)(ksp,emax,emin);CHKERRQ(ierr); > (gdb) up > #8 0x00007fb264b44a1f in PCGAMGOptProlongator_AGG (pc=0x12f3d30, Amat=0x11a2630, a_P=0x7ffc5010ebe0) at /home/sg/petsc-3.13.2/src/ksp/pc/impls/gamg/agg.c:1145 > 1145 ierr = KSPComputeExtremeSingularValues(eksp, &emax, &emin);CHKERRQ(ierr); > (gdb) up > #9 0x00007fb264b248a1 in PCSetUp_GAMG (pc=0x12f3d30) at /home/sg/petsc-3.13.2/src/ksp/pc/impls/gamg/gamg.c:557 > 557 ierr = pc_gamg->ops->optprolongator(pc, Aarr[level], &Prol11);CHKERRQ(ierr); > (gdb) up > #10 0x00007fb264d8535b in PCSetUp (pc=0x12f3d30) at /home/sg/petsc-3.13.2/src/ksp/pc/interface/precon.c:898 > 898 ierr = (*pc->ops->setup)(pc);CHKERRQ(ierr); > (gdb) up > #11 0x00007fb264e01a93 in KSPSetUp (ksp=0x128dd80) at /home/sg/petsc-3.13.2/src/ksp/ksp/interface/itfunc.c:376 > 376 ierr = PCSetUp(ksp->pc);CHKERRQ(ierr); > (gdb) up > #12 0x00007fb264e057af in KSPSolve_Private (ksp=0x128dd80, b=0x1259f30, x=0x125d910) at /home/sg/petsc-3.13.2/src/ksp/ksp/interface/itfunc.c:633 > 633 ierr = KSPSetUp(ksp);CHKERRQ(ierr); > (gdb) up > #13 0x00007fb264e086b9 in KSPSolve (ksp=0x128dd80, b=0x1259f30, x=0x125d910) at /home/sg/petsc-3.13.2/src/ksp/ksp/interface/itfunc.c:853 > 853 ierr = KSPSolve_Private(ksp,b,x);CHKERRQ(ierr); > (gdb) up > #14 0x00007fb264e46216 in kspsolve_ (ksp=0x832670 <__pfeapc_MOD_kspsol>, b=0x832698 <__pfeapc_MOD_rhs>, x=0x8326a0 <__pfeapc_MOD_sol>, __ierr=0x7ffc5010f358) > at /home/sg/petsc-3.13.2/src/ksp/ksp/interface/ftn-auto/itfuncf.c:266 > 266 *__ierr = KSPSolve( > (gdb) up > #15 0x000000000043298d in usolve (flags=..., b=...) at usolve.F:313 > 313 call KSPSolve (kspsol, rhs, sol, ierr) > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Sat Jun 13 12:13:35 2020 From: mfadams at lbl.gov (Mark Adams) Date: Sat, 13 Jun 2020 13:13:35 -0400 Subject: [petsc-users] Debugging suggestions: GAMG In-Reply-To: <87h7vf3ywo.fsf@jedbrown.org> References: <87h7vf3ywo.fsf@jedbrown.org> Message-ID: > > > > If SPD then you definitely want '-pc_gamg_esteig_ksp_type cg'. CG > converges > > faster and is more robust. > > Mark, this comes up a lot. Should we select it by default if MAT_SPD is > set? > I didn't know about MAT_SPD. Yea, I'll add it. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Sat Jun 13 12:40:03 2020 From: mfadams at lbl.gov (Mark Adams) Date: Sat, 13 Jun 2020 13:40:03 -0400 Subject: [petsc-users] Debugging suggestions: GAMG In-Reply-To: References: <87h7vf3ywo.fsf@jedbrown.org> Message-ID: On Sat, Jun 13, 2020 at 1:13 PM Mark Adams wrote: > >> > If SPD then you definitely want '-pc_gamg_esteig_ksp_type cg'. CG >> converges >> > faster and is more robust. >> >> Mark, this comes up a lot. Should we select it by default if MAT_SPD is >> set? >> > > I didn't know about MAT_SPD. Yea, I'll add it. > I guess I should do this for cheby also. -------------- next part -------------- An HTML attachment was scrubbed... URL: From s_g at berkeley.edu Sat Jun 13 14:34:00 2020 From: s_g at berkeley.edu (Sanjay Govindjee) Date: Sat, 13 Jun 2020 12:34:00 -0700 Subject: [petsc-users] Debugging suggestions: GAMG In-Reply-To: <24AC8444-94DD-4885-8E04-9107DD7DB673@petsc.dev> References: <24AC8444-94DD-4885-8E04-9107DD7DB673@petsc.dev> Message-ID: <62cf1d2b-054a-b440-02ef-e8a8b7861653@berkeley.edu> Machine details: ???? Fedora Core 30:? 5.6.13-100.fc30.x86_64 ???? gcc (GCC) 9.3.1 20200408 (Red Hat 9.3.1-2) ???? GNU Fortran (GCC) 9.3.1 20200408 (Red Hat 9.3.1-2) Lapack/BLAS: Are whatever came with the machine and are in /usr/lib64.? I did not compile them myself I'll try two things: (1) Rebuil with a different BLAS/LAPACK and (2) set a stop in ieee_handler( ) to see when and where it is getting called. Also just for completeness here are the rest of the error messages from the run: Thread 1 "feap" received signal SIGFPE, Arithmetic exception. 0x00007f0fe77e5be1 in ieeeck_ () from /lib64/liblapack.so.3 [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Caught signal number 8 FPE: Floating Point Exception,probably divide by zero [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [0]PETSC ERROR: likely location of problem given in stack below [0]PETSC ERROR: ---------------------? Stack Frames ------------------------------------ [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [0]PETSC ERROR:?????? INSTEAD the line number of the start of the function [0]PETSC ERROR:?????? is given. [0]PETSC ERROR: [0] LAPACKgesvd line 32 /home/sg/petsc-3.13.2/src/ksp/ksp/impls/gmres/gmreig.c [0]PETSC ERROR: [0] KSPComputeExtremeSingularValues_GMRES line 14 /home/sg/petsc-3.13.2/src/ksp/ksp/impls/gmres/gmreig.c [0]PETSC ERROR: [0] KSPComputeExtremeSingularValues line 57 /home/sg/petsc-3.13.2/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: [0] PCGAMGOptProlongator_AGG line 1107 /home/sg/petsc-3.13.2/src/ksp/pc/impls/gamg/agg.c [0]PETSC ERROR: User provided function() line 0 in? unknown file (null) Program received signal SIGABRT: Process abort signal. On 6/13/20 9:04 AM, Barry Smith wrote: > > ? ?The LAPACK routine?ieeeck_ intentionally does a divide by zero to > check if the system can handle it without generating an exception. It > doesn't have anything to do > with the particular matrix data passed to LAPACK. > > ? ? In?KSPComputeExtremeSingularValues_GMRES() we have the code structure > > ? ierr = PetscFPTrapPush(PETSC_FP_TRAP_OFF);CHKERRQ(ierr); > #if !defined(PETSC_USE_COMPLEX) > PetscStackCallBLAS("LAPACKgesvd",LAPACKgesvd_("N","N",&bn,&bn,R,&bN,realpart,&sdummy,&idummy,&sdummy,&idummy,work,&lwork,&lierr)); > #else > PetscStackCallBLAS("LAPACKgesvd",LAPACKgesvd_("N","N",&bn,&bn,R,&bN,realpart,&sdummy,&idummy,&sdummy,&idummy,work,&lwork,realpart+N,&lierr)); > #endif > ? if (lierr) SETERRQ1(PETSC_COMM_SELF,PETSC_ERR_LIB,"Error in SVD > Lapack routine %d",(int)lierr); > ? ierr = PetscFPTrapPop();CHKERRQ(ierr); > > ? ?So PETSc tries to turn off trapping of floating point exceptions > before calling the LAPACK routines that eventually lead to the exception. > > PetscErrorCode PetscFPTrapPush(PetscFPTrap trap) > { > ? PetscErrorCode ? ? ? ? ierr; > ? struct PetscFPTrapLink *link; > > ? PetscFunctionBegin; > ? ierr ? ? ? ? ? = PetscNew(&link);CHKERRQ(ierr); > ? link->trapmode = _trapmode; > ? link->next ? ? = _trapstack; > ? _trapstack ? ? = link; > ? if (trap != _trapmode) {ierr = PetscSetFPTrap(trap);CHKERRQ(ierr);} > ? PetscFunctionReturn(0); > } > > PetscErrorCode PetscSetFPTrap(PetscFPTrap flag) > { > ? char *out; > > ? PetscFunctionBegin; > ? /* Clear accumulated exceptions. ?Used to suppress meaningless > messages from f77 programs */ > ? (void) ieee_flags("clear","exception","all",&out); > ? if (flag == PETSC_FP_TRAP_ON) { > ? ? /* > ? ? ? To trap more fp exceptions, including underflow, change the line > below to > ? ? ? if (ieee_handler("set","all",PetscDefaultFPTrap)) { > ? ? */ > ? ? if (ieee_handler("set","common",PetscDefaultFPTrap)) > ?(*PetscErrorPrintf)("Can't set floatingpoint handler\n"); > ? } else if (ieee_handler("clear","common",PetscDefaultFPTrap)) > (*PetscErrorPrintf)("Can't clear floatingpoint handler\n"); > > ? _trapmode = flag; > ? PetscFunctionReturn(0); > } > > ? So either the ieee_handler clear is not working for your system or > some other code, AFTER PETSc calls ieee_handler sets the ?ieee_handler > to trap divide by zero. > > ? A git grep -i ieee_handler ?shows that the reference BLAS/LAPACK and > OpenBLAS never seem to call the ieee_handler. > > ? We need to know what lapack/blas you are using and how they were > compiled. > > ? Some Fortran compilers/linkers set nonstandard exception handlers, > but since PETSc clears them I don't know how they could get set again > > ? You could try in gdb to put a break point in ieee_handler and find > all the places it gets called, maybe this will lead to the location of > the cause. > > ? Barry > > >> On Jun 13, 2020, at 1:30 AM, Sanjay Govindjee > > wrote: >> >> I have a FEA problem that I am trying to solve with GAMG.? The >> problem solves >> just fine with direct solvers (mumps, superlu) and iterative solvers >> (gmres, ml, hypre-boomer) etc. >> >> However with GAMG I am getting a divide by zero that I am having >> trouble tracking down.? Below >> is the gdb stack trace and the source lines going up the stack. >> >> When I run in valgrind the problem runs fine (and gets the correct >> answer). >> Valgrind reports nothing of note (just lots of indirectly lost >> blocks? related to PMP_INIT). >> >> I'm only running on one processor. >> >> Any suggestions on where to start to trace the problem? >> >> -sanjay >> >> #0? 0x00007fb262dc5be1 in ieeeck_ () from /lib64/liblapack.so.3 >> #1? 0x00007fb262dc5332 in ilaenv_ () from /lib64/liblapack.so.3 >> #2? 0x00007fb262dbbcef in dlasq2_ () from /lib64/liblapack.so.3 >> #3? 0x00007fb262dbb78c in dlasq1_ () from /lib64/liblapack.so.3 >> #4? 0x00007fb262da1e2e in dbdsqr_ () from /lib64/liblapack.so.3 >> #5? 0x00007fb262960110 in dgesvd_ () from /lib64/liblapack.so.3 >> #6? 0x00007fb264e74b66 in KSPComputeExtremeSingularValues_GMRES >> (ksp=0x1816560, emax=0x7ffc5010e7c8, emin=0x7ffc5010e7d0) at >> /home/sg/petsc-3.13.2/src/ksp/ksp/impls/gmres/gmreig.c:32 >> #7? 0x00007fb264dfe69a in KSPComputeExtremeSingularValues >> (ksp=0x1816560, emax=0x7ffc5010e7c8, emin=0x7ffc5010e7d0) at >> /home/sg/petsc-3.13.2/src/ksp/ksp/interface/itfunc.c:64 >> #8? 0x00007fb264b44a1f in PCGAMGOptProlongator_AGG (pc=0x12f3d30, >> Amat=0x11a2630, a_P=0x7ffc5010ebe0) at >> /home/sg/petsc-3.13.2/src/ksp/pc/impls/gamg/agg.c:1145 >> #9? 0x00007fb264b248a1 in PCSetUp_GAMG (pc=0x12f3d30) at >> /home/sg/petsc-3.13.2/src/ksp/pc/impls/gamg/gamg.c:557 >> #10 0x00007fb264d8535b in PCSetUp (pc=0x12f3d30) at >> /home/sg/petsc-3.13.2/src/ksp/pc/interface/precon.c:898 >> #11 0x00007fb264e01a93 in KSPSetUp (ksp=0x128dd80) at >> /home/sg/petsc-3.13.2/src/ksp/ksp/interface/itfunc.c:376 >> #12 0x00007fb264e057af in KSPSolve_Private (ksp=0x128dd80, >> b=0x1259f30, x=0x125d910) at >> /home/sg/petsc-3.13.2/src/ksp/ksp/interface/itfunc.c:633 >> #13 0x00007fb264e086b9 in KSPSolve (ksp=0x128dd80, b=0x1259f30, >> x=0x125d910) at >> /home/sg/petsc-3.13.2/src/ksp/ksp/interface/itfunc.c:853 >> #14 0x00007fb264e46216 in kspsolve_ (ksp=0x832670 >> <__pfeapc_MOD_kspsol>, b=0x832698 <__pfeapc_MOD_rhs>, x=0x8326a0 >> <__pfeapc_MOD_sol>, __ierr=0x7ffc5010f358) >> ??? at >> /home/sg/petsc-3.13.2/src/ksp/ksp/interface/ftn-auto/itfuncf.c:266 >> #15 0x000000000043298d in usolve (flags=..., b=...) at usolve.F:313 >> #16 0x000000000044afba in psolve (stype=-3, b=..., fp=..., >> factor=.TRUE., solve=.TRUE., cfr=.FALSE., prnt=.TRUE.) at >> psolve.f:212 >> #17 0x00000000006b7393 in pmacr1 (lct=..., ct=..., j=3, >> _lct=_lct at entry=15) at pmacr1.f:578 >> #18 0x00000000005c247b in pmacr (initf=.FALSE.) at pmacr.f:578 >> #19 0x000000000044ff20 in pcontr () at pcontr.f:1307 >> #20 0x0000000000404d9b in feap () at feap86.f:162 >> #21 main (argc=, argv=) at feap86.f:168 >> #22 0x00007fb261aaef43 in __libc_start_main () from /lib64/libc.so.6 >> #23 0x0000000000404dde in _start () >> >> (gdb) list >> 1?????? : No such file or directory. >> (gdb) up >> #1? 0x00007fb262dc5332 in ilaenv_ () from /lib64/liblapack.so.3 >> (gdb) up >> #2? 0x00007fb262dbbcef in dlasq2_ () from /lib64/liblapack.so.3 >> (gdb) up >> #3? 0x00007fb262dbb78c in dlasq1_ () from /lib64/liblapack.so.3 >> (gdb) up >> #4? 0x00007fb262da1e2e in dbdsqr_ () from /lib64/liblapack.so.3 >> (gdb) up >> #5? 0x00007fb262960110 in dgesvd_ () from /lib64/liblapack.so.3 >> (gdb) up >> #6? 0x00007fb264e74b66 in KSPComputeExtremeSingularValues_GMRES >> (ksp=0x1816560, emax=0x7ffc5010e7c8, emin=0x7ffc5010e7d0) at >> /home/sg/petsc-3.13.2/src/ksp/ksp/impls/gmres/gmreig.c:32 >> 32 >> PetscStackCallBLAS("LAPACKgesvd",LAPACKgesvd_("N","N",&bn,&bn,R,&bN,realpart,&sdummy,&idummy,&sdummy,&idummy,work,&lwork,&lierr)); >> (gdb) up >> #7? 0x00007fb264dfe69a in KSPComputeExtremeSingularValues >> (ksp=0x1816560, emax=0x7ffc5010e7c8, emin=0x7ffc5010e7d0) at >> /home/sg/petsc-3.13.2/src/ksp/ksp/interface/itfunc.c:64 >> 64????????? ierr = >> (*ksp->ops->computeextremesingularvalues)(ksp,emax,emin);CHKERRQ(ierr); >> (gdb) up >> #8? 0x00007fb264b44a1f in PCGAMGOptProlongator_AGG (pc=0x12f3d30, >> Amat=0x11a2630, a_P=0x7ffc5010ebe0) at >> /home/sg/petsc-3.13.2/src/ksp/pc/impls/gamg/agg.c:1145 >> 1145????????? ierr = KSPComputeExtremeSingularValues(eksp, &emax, >> &emin);CHKERRQ(ierr); >> (gdb) up >> #9? 0x00007fb264b248a1 in PCSetUp_GAMG (pc=0x12f3d30) at >> /home/sg/petsc-3.13.2/src/ksp/pc/impls/gamg/gamg.c:557 >> 557?????????????? ierr = pc_gamg->ops->optprolongator(pc, >> Aarr[level], &Prol11);CHKERRQ(ierr); >> (gdb) up >> #10 0x00007fb264d8535b in PCSetUp (pc=0x12f3d30) at >> /home/sg/petsc-3.13.2/src/ksp/pc/interface/precon.c:898 >> 898???????? ierr = (*pc->ops->setup)(pc);CHKERRQ(ierr); >> (gdb) up >> #11 0x00007fb264e01a93 in KSPSetUp (ksp=0x128dd80) at >> /home/sg/petsc-3.13.2/src/ksp/ksp/interface/itfunc.c:376 >> 376?????? ierr = PCSetUp(ksp->pc);CHKERRQ(ierr); >> (gdb) up >> #12 0x00007fb264e057af in KSPSolve_Private (ksp=0x128dd80, >> b=0x1259f30, x=0x125d910) at >> /home/sg/petsc-3.13.2/src/ksp/ksp/interface/itfunc.c:633 >> 633?????? ierr = KSPSetUp(ksp);CHKERRQ(ierr); >> (gdb) up >> #13 0x00007fb264e086b9 in KSPSolve (ksp=0x128dd80, b=0x1259f30, >> x=0x125d910) at >> /home/sg/petsc-3.13.2/src/ksp/ksp/interface/itfunc.c:853 >> 853?????? ierr = KSPSolve_Private(ksp,b,x);CHKERRQ(ierr); >> (gdb) up >> #14 0x00007fb264e46216 in kspsolve_ (ksp=0x832670 >> <__pfeapc_MOD_kspsol>, b=0x832698 <__pfeapc_MOD_rhs>, x=0x8326a0 >> <__pfeapc_MOD_sol>, __ierr=0x7ffc5010f358) >> ??? at >> /home/sg/petsc-3.13.2/src/ksp/ksp/interface/ftn-auto/itfuncf.c:266 >> 266???? *__ierr = KSPSolve( >> (gdb) up >> #15 0x000000000043298d in usolve (flags=..., b=...) at usolve.F:313 >> 313?????????????? call KSPSolve???????? (kspsol, rhs, sol, ierr) >> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From s_g at berkeley.edu Sat Jun 13 14:48:06 2020 From: s_g at berkeley.edu (Sanjay Govindjee) Date: Sat, 13 Jun 2020 12:48:06 -0700 Subject: [petsc-users] Debugging suggestions: GAMG In-Reply-To: References: Message-ID: <46d67cf7-94fa-a7ef-585e-e3629c2ba49c@berkeley.edu> Mark, ? The problem is a 2D plane elasticity problem, definitely SPD. If I add -pc_gamg_esteig_ksp_type cg, it works perfectly. ? I also tried? adding -info but that caused some other type of weird crash; the options I used were -ksp_type cg -ksp_monitor -log_view -pc_type gamg -pc_gamg_type agg -pc_gamg_agg_nsmooths 1 -pc_gamg_esteig_ksp_type cg -info? -options_left The resulting error was: [0] petscinitialize_internal(): (Fortran):PETSc successfully started: procs 1 [0] petscinitialize_internal(): Running on machine: vulture.ce.berkeley.edu ?--> ERRORS OCCURRED: For details see file: Ofo [0] PetscFinalize(): PetscFinalize() called [0] PetscCommDuplicate(): Duplicating a communicator 140367846615616 28408096 max tags = 2147483647 ************************************************************************************************************************ ***???????????? WIDEN YOUR WINDOW TO 120 CHARACTERS.? Use 'enscript -r -fCourier9' to print this document??????????? *** ************************************************************************************************************************ ---------------------------------------------- PETSc Performance Summary: ---------------------------------------------- ????? ########################################################## ????? #??????????????????????????????????????????????????????? # ????? #?????????????????????? WARNING!!!?????????????????????? # ????? #??????????????????????????????????????????????????????? # ????? #?? This code was compiled with a debugging option.????? # ????? #?? To get timing results run ./configure??????????????? # ????? #?? using --with-debugging=no, the performance will????? # ????? #?? be generally two or three times faster.????????????? # ????? #??????????????????????????????????????????????????????? # ????? ########################################################## /home/sg/Feap/ver86/parfeap/feap on a gnu named vulture.ce.berkeley.edu with 1 processor, by sg Sat Jun 13 12:47:06 2020 Using Petsc Release Version 3.13.2, unknown ???????????????????????? Max?????? Max/Min???? Avg?????? Total Time (sec):?????????? 1.516e-03???? 1.000?? 1.516e-03 Objects:????????????? 1.000e+00???? 1.000?? 1.000e+00 Flop:???????????????? 0.000e+00???? 0.000?? 0.000e+00? 0.000e+00 Flop/sec:???????????? 0.000e+00???? 0.000?? 0.000e+00? 0.000e+00 Memory:?????????????? 5.122e+04???? 1.000?? 5.122e+04? 5.122e+04 MPI Messages:???????? 0.000e+00???? 0.000?? 0.000e+00? 0.000e+00 MPI Message Lengths:? 0.000e+00???? 0.000?? 0.000e+00? 0.000e+00 MPI Reductions:?????? 0.000e+00???? 0.000 Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract) ??????????????????????????? e.g., VecAXPY() for real vectors of length N --> 2N flop ??????????????????????????? and VecAXPY() for complex vectors of length N --> 8N flop Summary of Stages:?? ----- Time ------? ----- Flop ------? --- Messages ---? -- Message Lengths --? -- Reductions -- ??????????????????????? Avg???? %Total???? Avg???? %Total Count?? %Total???? Avg???????? %Total??? Count?? %Total ?0:????? Main Stage: 1.5061e-03? 99.3%? 0.0000e+00?? 0.0% 0.000e+00?? 0.0%? 0.000e+00??????? 0.0%? 0.000e+00?? 0.0% ------------------------------------------------------------------------------------------------------------------------ See the 'Profiling' chapter of the users' manual for details on interpreting output. Phase summary info: ?? Count: number of times phase was executed ?? Time and Flop: Max - maximum over all processors ????????????????? Ratio - ratio of maximum to minimum over all processors ?? Mess: number of messages sent ?? AvgLen: average message length (bytes) ?? Reduct: number of global reductions ?? Global: entire computation ?? Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop(). ????? %T - percent time in this phase???????? %F - percent flop in this phase ????? %M - percent messages in this phase???? %L - percent message lengths in this phase ????? %R - percent reductions in this phase ?? Total Mflop/s: 10e-6 * (sum of flop over all processors)/(max time over all processors) ------------------------------------------------------------------------------------------------------------------------ ????? ########################################################## ????? #??????????????????????????????????????????????????????? # ????? #?????????????????????? WARNING!!!?????????????????????? # ????? #??????????????????????????????????????????????????????? # ????? #?? This code was compiled with a debugging option.????? # ????? #?? To get timing results run ./configure??????????????? # ????? #?? using --with-debugging=no, the performance will????? # ????? #?? be generally two or three times faster.????????????? # ????? #??????????????????????????????????????????????????????? # ????? ########################################################## Event??????????????? Count????? Time (sec) Flop????????????????????????????? --- Global ---? --- Stage ---- Total ?????????????????? Max Ratio? Max???? Ratio?? Max? Ratio? Mess AvgLen? Reduct? %T %F %M %L %R? %T %F %M %L %R Mflop/s ------------------------------------------------------------------------------------------------------------------------ --- Event Stage 0: Main Stage ------------------------------------------------------------------------------------------------------------------------ Memory usage is given in bytes: Object Type????????? Creations?? Destructions???? Memory Descendants' Mem. Reports information only for process 0. --- Event Stage 0: Main Stage ????????????? Viewer???? 1????????????? 0??????????? 0???? 0. ======================================================================================================================== Average time to get PetscTime(): 4.38071e-08 #PETSc Option Table entries: -info -ksp_monitor -ksp_type cg -log_view -options_left -pc_gamg_agg_nsmooths 1 -pc_gamg_esteig_ksp_type cg -pc_gamg_type agg -pc_type gamg #End of PETSc Option Table entries Compiled without FORTRAN kernels Compiled with full precision matrices (default) sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4 Configure options: --download-parmetis --download-superlu_dist --download-openmpi --download-ml --download-hypre --download-metis --download-mumps --download-scalapack --download-blacs ----------------------------------------- Libraries compiled on 2020-06-13 05:13:15 on vulture.ce.berkeley.edu Machine characteristics: Linux-5.6.13-100.fc30.x86_64-x86_64-with-fedora-30-Thirty Using PETSc directory: /home/sg/petsc-3.13.2 Using PETSc arch: gnu ----------------------------------------- Using C compiler: /home/sg/petsc-3.13.2/gnu/bin/mpicc? -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -fstack-protector -fvisibility=hidden -g3 Using Fortran compiler: /home/sg/petsc-3.13.2/gnu/bin/mpif90 -fPIC -Wall -ffree-line-length-0 -Wno-unused-dummy-argument -g ----------------------------------------- Using include paths: -I/home/sg/petsc-3.13.2/include -I/home/sg/petsc-3.13.2/gnu/include ----------------------------------------- Using C linker: /home/sg/petsc-3.13.2/gnu/bin/mpicc Using Fortran linker: /home/sg/petsc-3.13.2/gnu/bin/mpif90 Using libraries: -Wl,-rpath,/home/sg/petsc-3.13.2/gnu/lib -L/home/sg/petsc-3.13.2/gnu/lib -lpetsc -Wl,-rpath,/home/sg/petsc-3.13.2/gnu/lib -L/home/sg/petsc-3.13.2/gnu/lib -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/9 -L/usr/lib/gcc/x86_64-redhat-linux/9 -lHYPRE -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lscalapack -lsuperlu_dist -lml -llapack -lblas -lX11 -lparmetis -lmetis -lm -lstdc++ -ldl -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi -lgfortran -lm -lgfortran -lm -lgcc_s -lquadmath -lpthread -lstdc++ -ldl ----------------------------------------- ????? ########################################################## ????? #??????????????????????????????????????????????????????? # ????? #?????????????????????? WARNING!!!?????????????????????? # ????? #??????????????????????????????????????????????????????? # ????? #?? This code was compiled with a debugging option.????? # ????? #?? To get timing results run ./configure??????????????? # ????? #?? using --with-debugging=no, the performance will????? # ????? #?? be generally two or three times faster.????????????? # ????? #??????????????????????????????????????????????????????? # ????? ########################################################## [0] Petsc_DelViewer(): Removing viewer data attribute in an MPI_Comm 28408096 [0] Petsc_OuterComm_Attr_Delete_Fn(): Removing reference to PETSc communicator embedded in a user MPI_Comm 28408096 [0] Petsc_InnerComm_Attr_Delete_Fn(): User MPI_Comm 140367846615616 is being unlinked from inner PETSc comm 28408096 [0] PetscCommDestroy(): Deleting PETSc MPI_Comm 28408096 [0] Petsc_Counter_Attr_Delete_Fn(): Deleting counter data in an MPI_Comm 28408096 [0] PetscCommDuplicate(): Duplicating a communicator 140367846615616 28408096 max tags = 2147483647 #PETSc Option Table entries: -info -ksp_monitor -ksp_type cg -log_view -options_left -pc_gamg_agg_nsmooths 1 -pc_gamg_esteig_ksp_type cg -pc_gamg_type agg -pc_type gamg #End of PETSc Option Table entries [0] Petsc_OuterComm_Attr_Delete_Fn(): Removing reference to PETSc communicator embedded in a user MPI_Comm 28408096 [0] Petsc_InnerComm_Attr_Delete_Fn(): User MPI_Comm 140367846615616 is being unlinked from inner PETSc comm 28408096 [0] PetscCommDestroy(): Deleting PETSc MPI_Comm 28408096 [0] Petsc_Counter_Attr_Delete_Fn(): Deleting counter data in an MPI_Comm 28408096 WARNING! There are options you set that were not used! WARNING! could be spelling mistake, etc! There are 6 unused database options. They are: Option left: name:-ksp_monitor (no value) Option left: name:-ksp_type value: cg Option left: name:-pc_gamg_agg_nsmooths value: 1 Option left: name:-pc_gamg_esteig_ksp_type value: cg Option left: name:-pc_gamg_type value: agg Option left: name:-pc_type value: gamg [sg at vulture main]$ On 6/13/20 5:10 AM, Mark Adams wrote: > That is odd. Are these problems symmetric?positive?definite? > > Eigen estimates are a pain in practice but I've never seen this. Hypre > has (better) smoothers that don't need this and the AMG algorithm does > not need them either. I think ML does pretty much the same thing as me. > > If SPD then you definitely?want '-pc_gamg_esteig_ksp_type cg'. CG > converges faster and is more robust. Even if not strictly?SPD like > with large deformation or plasticity?CG is probably better. You can > run with -info and grep on GAMG to see what eigen (high) estimates it > came up with for each level. They should be >1 and < 4, or so. > > I wonder if your LAPACK is funky. You might try a fresh build (delete > the "arch" directory) and download LAPACK, but this problem will go > away with CG. > > Note, if you use Chebyshev smoothing then it needs eigen estimates > also. THis is usually where eigen estimate problems come up. If you > use jacobi PC in the smoother, GAMG will use the eigen estimate from > the smoothed aggregation smoothing (what is failing for you here). > -pc_gamg_use_sa_esteig will force it to use, or not use, it. For > instance, if you sor/ilu/asm then the eigen estimates are probably > fine to reuse but GAMG does not by default. If the cheby?eigen > estimates are low then the solver fails. THere is a safety?factor to > increase the eigen estimate in chychev to avoid this failure. > > And, -pc_gamg_esteig_ksp_max_it X sets the number of iterations in the > estimator. You can increase this and you should see your eigen > estimates go up and converge. You should see that CG converges much > faster than GMRES (the default). > > > > > On Sat, Jun 13, 2020 at 2:32 AM Sanjay Govindjee > wrote: > > I have a FEA problem that I am trying to solve with GAMG.? The > problem solves > just fine with direct solvers (mumps, superlu) and iterative > solvers (gmres, ml, hypre-boomer) etc. > > However with GAMG I am getting a divide by zero that I am having > trouble tracking down.? Below > is the gdb stack trace and the source lines going up the stack. > > When I run in valgrind the problem runs fine (and gets the correct > answer). > Valgrind reports nothing of note (just lots of indirectly lost > blocks? related to PMP_INIT). > > I'm only running on one processor. > > Any suggestions on where to start to trace the problem? > > -sanjay > > #0? 0x00007fb262dc5be1 in ieeeck_ () from /lib64/liblapack.so.3 > #1? 0x00007fb262dc5332 in ilaenv_ () from /lib64/liblapack.so.3 > #2? 0x00007fb262dbbcef in dlasq2_ () from /lib64/liblapack.so.3 > #3? 0x00007fb262dbb78c in dlasq1_ () from /lib64/liblapack.so.3 > #4? 0x00007fb262da1e2e in dbdsqr_ () from /lib64/liblapack.so.3 > #5? 0x00007fb262960110 in dgesvd_ () from /lib64/liblapack.so.3 > #6? 0x00007fb264e74b66 in > KSPComputeExtremeSingularValues_GMRES (ksp=0x1816560, > emax=0x7ffc5010e7c8, emin=0x7ffc5010e7d0) at > /home/sg/petsc-3.13.2/src/ksp/ksp/impls/gmres/gmreig.c:32 > #7? 0x00007fb264dfe69a in KSPComputeExtremeSingularValues > (ksp=0x1816560, emax=0x7ffc5010e7c8, emin=0x7ffc5010e7d0) at > /home/sg/petsc-3.13.2/src/ksp/ksp/interface/itfunc.c:64 > #8? 0x00007fb264b44a1f in PCGAMGOptProlongator_AGG > (pc=0x12f3d30, Amat=0x11a2630, a_P=0x7ffc5010ebe0) at > /home/sg/petsc-3.13.2/src/ksp/pc/impls/gamg/agg.c:1145 > #9? 0x00007fb264b248a1 in PCSetUp_GAMG (pc=0x12f3d30) at > /home/sg/petsc-3.13.2/src/ksp/pc/impls/gamg/gamg.c:557 > #10 0x00007fb264d8535b in PCSetUp (pc=0x12f3d30) at > /home/sg/petsc-3.13.2/src/ksp/pc/interface/precon.c:898 > #11 0x00007fb264e01a93 in KSPSetUp (ksp=0x128dd80) at > /home/sg/petsc-3.13.2/src/ksp/ksp/interface/itfunc.c:376 > #12 0x00007fb264e057af in KSPSolve_Private (ksp=0x128dd80, > b=0x1259f30, x=0x125d910) at > /home/sg/petsc-3.13.2/src/ksp/ksp/interface/itfunc.c:633 > #13 0x00007fb264e086b9 in KSPSolve (ksp=0x128dd80, > b=0x1259f30, x=0x125d910) at > /home/sg/petsc-3.13.2/src/ksp/ksp/interface/itfunc.c:853 > #14 0x00007fb264e46216 in kspsolve_ (ksp=0x832670 > <__pfeapc_MOD_kspsol>, b=0x832698 <__pfeapc_MOD_rhs>, > x=0x8326a0 <__pfeapc_MOD_sol>, __ierr=0x7ffc5010f358) > ??? at > /home/sg/petsc-3.13.2/src/ksp/ksp/interface/ftn-auto/itfuncf.c:266 > #15 0x000000000043298d in usolve (flags=..., b=...) at > usolve.F:313 > #16 0x000000000044afba in psolve (stype=-3, b=..., fp=..., > factor=.TRUE., solve=.TRUE., cfr=.FALSE., prnt=.TRUE.) at > psolve.f:212 > #17 0x00000000006b7393 in pmacr1 (lct=..., ct=..., j=3, > _lct=_lct at entry=15) at pmacr1.f:578 > #18 0x00000000005c247b in pmacr (initf=.FALSE.) at pmacr.f:578 > #19 0x000000000044ff20 in pcontr () at pcontr.f:1307 > #20 0x0000000000404d9b in feap () at feap86.f:162 > #21 main (argc=, argv=) at > feap86.f:168 > #22 0x00007fb261aaef43 in __libc_start_main () from > /lib64/libc.so.6 > #23 0x0000000000404dde in _start () > > (gdb) list > 1?????? : No such file or directory. > (gdb) up > #1? 0x00007fb262dc5332 in ilaenv_ () from /lib64/liblapack.so.3 > (gdb) up > #2? 0x00007fb262dbbcef in dlasq2_ () from /lib64/liblapack.so.3 > (gdb) up > #3? 0x00007fb262dbb78c in dlasq1_ () from /lib64/liblapack.so.3 > (gdb) up > #4? 0x00007fb262da1e2e in dbdsqr_ () from /lib64/liblapack.so.3 > (gdb) up > #5? 0x00007fb262960110 in dgesvd_ () from /lib64/liblapack.so.3 > (gdb) up > #6? 0x00007fb264e74b66 in > KSPComputeExtremeSingularValues_GMRES (ksp=0x1816560, > emax=0x7ffc5010e7c8, emin=0x7ffc5010e7d0) at > /home/sg/petsc-3.13.2/src/ksp/ksp/impls/gmres/gmreig.c:32 > 32 > PetscStackCallBLAS("LAPACKgesvd",LAPACKgesvd_("N","N",&bn,&bn,R,&bN,realpart,&sdummy,&idummy,&sdummy,&idummy,work,&lwork,&lierr)); > (gdb) up > #7? 0x00007fb264dfe69a in KSPComputeExtremeSingularValues > (ksp=0x1816560, emax=0x7ffc5010e7c8, emin=0x7ffc5010e7d0) at > /home/sg/petsc-3.13.2/src/ksp/ksp/interface/itfunc.c:64 > 64????????? ierr = > (*ksp->ops->computeextremesingularvalues)(ksp,emax,emin);CHKERRQ(ierr); > (gdb) up > #8? 0x00007fb264b44a1f in PCGAMGOptProlongator_AGG > (pc=0x12f3d30, Amat=0x11a2630, a_P=0x7ffc5010ebe0) at > /home/sg/petsc-3.13.2/src/ksp/pc/impls/gamg/agg.c:1145 > 1145????????? ierr = KSPComputeExtremeSingularValues(eksp, > &emax, &emin);CHKERRQ(ierr); > (gdb) up > #9? 0x00007fb264b248a1 in PCSetUp_GAMG (pc=0x12f3d30) at > /home/sg/petsc-3.13.2/src/ksp/pc/impls/gamg/gamg.c:557 > 557?????????????? ierr = pc_gamg->ops->optprolongator(pc, > Aarr[level], &Prol11);CHKERRQ(ierr); > (gdb) up > #10 0x00007fb264d8535b in PCSetUp (pc=0x12f3d30) at > /home/sg/petsc-3.13.2/src/ksp/pc/interface/precon.c:898 > 898???????? ierr = (*pc->ops->setup)(pc);CHKERRQ(ierr); > (gdb) up > #11 0x00007fb264e01a93 in KSPSetUp (ksp=0x128dd80) at > /home/sg/petsc-3.13.2/src/ksp/ksp/interface/itfunc.c:376 > 376?????? ierr = PCSetUp(ksp->pc);CHKERRQ(ierr); > (gdb) up > #12 0x00007fb264e057af in KSPSolve_Private (ksp=0x128dd80, > b=0x1259f30, x=0x125d910) at > /home/sg/petsc-3.13.2/src/ksp/ksp/interface/itfunc.c:633 > 633?????? ierr = KSPSetUp(ksp);CHKERRQ(ierr); > (gdb) up > #13 0x00007fb264e086b9 in KSPSolve (ksp=0x128dd80, > b=0x1259f30, x=0x125d910) at > /home/sg/petsc-3.13.2/src/ksp/ksp/interface/itfunc.c:853 > 853?????? ierr = KSPSolve_Private(ksp,b,x);CHKERRQ(ierr); > (gdb) up > #14 0x00007fb264e46216 in kspsolve_ (ksp=0x832670 > <__pfeapc_MOD_kspsol>, b=0x832698 <__pfeapc_MOD_rhs>, > x=0x8326a0 <__pfeapc_MOD_sol>, __ierr=0x7ffc5010f358) > ??? at > /home/sg/petsc-3.13.2/src/ksp/ksp/interface/ftn-auto/itfuncf.c:266 > 266???? *__ierr = KSPSolve( > (gdb) up > #15 0x000000000043298d in usolve (flags=..., b=...) at > usolve.F:313 > 313?????????????? call KSPSolve???????? (kspsol, rhs, sol, ierr) > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From s_g at berkeley.edu Sat Jun 13 15:06:07 2020 From: s_g at berkeley.edu (Sanjay Govindjee) Date: Sat, 13 Jun 2020 13:06:07 -0700 Subject: [petsc-users] Debugging suggestions: GAMG In-Reply-To: <24AC8444-94DD-4885-8E04-9107DD7DB673@petsc.dev> References: <24AC8444-94DD-4885-8E04-9107DD7DB673@petsc.dev> Message-ID: Not a lot of luck here; I guess I don't know how to set the break point correctly? (gdb) b ieee_handler Function "ieee_handler" not defined. Make breakpoint pending on future shared library load? (y or [n]) y Breakpoint 1 (ieee_handler) pending. (gdb) c Continuing. Thread 1 "feap" received signal SIGFPE, Arithmetic exception. 0x00007f8d1a422be1 in ieeeck_ () from /lib64/liblapack.so.3 On 6/13/20 9:04 AM, Barry Smith wrote: > You could try in gdb to put a break point in ieee_handler and find all > the places it gets called, maybe this will lead to the location of the > cause. -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Sat Jun 13 15:35:03 2020 From: bsmith at petsc.dev (Barry Smith) Date: Sat, 13 Jun 2020 15:35:03 -0500 Subject: [petsc-users] Debugging suggestions: GAMG In-Reply-To: References: <24AC8444-94DD-4885-8E04-9107DD7DB673@petsc.dev> Message-ID: I can reproduce the crash, so I will debug it and let you know the results. Thanks for all your help Barry > On Jun 13, 2020, at 3:06 PM, Sanjay Govindjee wrote: > > Not a lot of luck here; I guess I don't know how to set the break point correctly? > > (gdb) b ieee_handler > Function "ieee_handler" not defined. > Make breakpoint pending on future shared library load? (y or [n]) y > Breakpoint 1 (ieee_handler) pending. > (gdb) c > Continuing. > > Thread 1 "feap" received signal SIGFPE, Arithmetic exception. > 0x00007f8d1a422be1 in ieeeck_ () from /lib64/liblapack.so.3 > > > > On 6/13/20 9:04 AM, Barry Smith wrote: >> You could try in gdb to put a break point in ieee_handler and find all the places it gets called, maybe this will lead to the location of the cause. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Sat Jun 13 15:39:17 2020 From: jed at jedbrown.org (Jed Brown) Date: Sat, 13 Jun 2020 14:39:17 -0600 Subject: [petsc-users] Parallel writing in HDF5-1.12.0 when some processors have no data to write In-Reply-To: References: <360F2E38-6D7C-496E-877A-9177B38D1F6C@gmail.com> <871rml6jsi.fsf@jedbrown.org> <8A7B62D0-0861-455A-8660-C4ECB392B88F@gmail.com> <87k10c5t9w.fsf@jedbrown.org> Message-ID: <87366y4s5m.fsf@jedbrown.org> Can you reproduce in C? You're missing three extra arguments that exist in the Fortran interface. https://support.hdfgroup.org/HDF5/doc/RM/RM_H5D.html#Dataset-Create Danyang Su writes: > Hi Jed, > > Attached is the example for your test. > > This example uses H5Sset_none to tell H5Dwrite call that there will be no data. 4-th process HAS to participate we are in a collective mode. > The code is ported and modified based on the C example from https://support.hdfgroup.org/ftp/HDF5/examples/parallel/coll_test.c > > The compiling flags in the makefile are same as those used in my own code. > > To compile the code, please run 'make all' > To test the code, please run 'mpiexec -n 4 ./hdf5_zero_data'. Any number of processors larger than 4 should help to detect the problem. > > The code may crash on HDF5 1.12.0 but works fine on HDF5 1.10.x. > > The following platforms have been tested: > Macos-Mojave + GNU-8.2 + HDF5-1.12.0 -> Works fine > Ubuntu-16.04 + GNU-5.4 + HDF5-1.12.0 -> Crashes > Ubuntu-16.04 + GNU-7.5 + HDF5-1.12.0 -> Crashes > Ubuntu-16.04 + GNU-5.4 + HDF5-1.10.x -> Works fine > Centos-7 + Intel2018 + HDF5-12.0 -> Works fine > > Possible error when code crashes > At line 6686 of file H5_gen.F90 > Fortran runtime error: Index '1' of dimension 1 of array 'buf' above upper bound of 0 > > Thanks, > > Danyang > > ?On 2020-06-12, 6:05 AM, "Jed Brown" wrote: > > Danyang Su writes: > > > Hi Jed, > > > > Thanks for your double check. > > > > The HDF 1.10.6 version also works. But versions from 1.12.x stop working. > > I'd suggest making a reduced test case in order to submit a bug report. > > This was the relevant change in PETSc for hdf5-1.12. > > https://gitlab.com/petsc/petsc/commit/806daeb7de397195b5132278177f4d5553f9f612 > > > Attached is the code section where I have problem. > > > > !c write the dataset collectively > > !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! > > !!!! CODE CRASHES HERE IF SOME PROCESSORS HAVE NO DATA TO WRITE!!!! > > !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! > > call h5dwrite_f(dset_id, H5T_NATIVE_DOUBLE, dataset, hdf5_dsize, & > > hdf5_ierr, file_space_id=filespace, & > > mem_space_id=memspace, xfer_prp = xlist_id) > > > > Please let me know if there is something wrong in the code that causes the problem. > > !c > !c This example uses H5Sset_none to tell H5Dwrite call that > !c there will be no data. 4-th process HAS to participate since > !c we are in a collective mode. > !c > !c The code is ported and modified based on the C example > !c from https://support.hdfgroup.org/ftp/HDF5/examples/parallel/coll_test.c > !c by Danyang Su on June 12, 2020. > !c > !c To compile the code, please run 'make all' > !c To test the code, please run 'mpiexec -n 4 ./hdf5_zero_data' > !c > !c IMPORTNANT NOTE > !c The code may crash on HDF5 1.12.0 but works fine on HDF5 1.10.x. > !c > !c The following platforms have been tested: > !c Macos-Mojave + GNU-8.2 + HDF5-1.12.0 -> Works fine > !c Ubuntu-16.04 + GNU-5.4 + HDF5-1.12.0 -> Crashes > !c Ubuntu-16.04 + GNU-7.5 + HDF5-1.12.0 -> Crashes > !c Ubuntu-16.04 + GNU-5.4 + HDF5-1.10.x -> Works fine > !c Centos-7 + Intel2018 + HDF5-12.0 -> Works fine > !c > !c Possible error when code crashes > !c At line 6686 of file H5_gen.F90 > !c Fortran runtime error: Index '1' of dimension 1 of array 'buf' above upper bound of 0 > !c > > program hdf5_zero_data > > > #include > > use petscsys > use hdf5 > > implicit none > > character(len=10), parameter :: h5File_Name = "SDS_row.h5" > character(len=8), parameter :: DatasetName = "IntArray" > integer, parameter :: nx = 8, ny = 5, ndim = 2 > > integer :: i > integer :: hdf5_ierr ! HDF5 error code > PetscErrorCode :: ierr > > integer(HID_T) :: file_id ! File identifier > integer(HID_T) :: dset_id ! Dataset identifier > integer(HID_T) :: space_id ! Space identifier > integer(HID_T) :: plist_id ! Property list identifier > integer(HID_T) :: FileSpace ! File identifier > integer(HID_T) :: MemSpace ! Dataset identifier > > integer(HSIZE_T) :: dimsf(2) ! Dataset dimensions > integer(HSIZE_T) :: hcount(2) ! hyperslab selection parameters > integer(HSIZE_T) :: offset(2) ! hyperslab selection parameters > > integer, allocatable :: data(:) ! Dataset to write > > integer :: mpi_size, mpi_rank; > > !c Initialize MPI. > call PetscInitialize(Petsc_Null_Character,ierr) > CHKERRQ(ierr) > call MPI_Comm_rank(Petsc_Comm_World,mpi_rank,ierr) > CHKERRQ(ierr) > call MPI_Comm_size(Petsc_Comm_World,mpi_size,ierr) > CHKERRQ(ierr) > > !c initialize fortran interface > call h5open_f(hdf5_ierr) > > !c Set up file access property list with parallel I/O access. > call h5pcreate_f(H5P_FILE_ACCESS_F, plist_id, hdf5_ierr) > call h5pset_fapl_mpio_f(plist_id, Petsc_Comm_World, & > MPI_INFO_NULL, hdf5_ierr) > > !c Create a new file collectively and release property list identifier. > call h5fcreate_f(h5File_Name, H5F_ACC_TRUNC_F, file_id, & > hdf5_ierr, access_prp = plist_id) > call h5pclose_f(plist_id, hdf5_ierr) > > !c Create the dataspace for the dataset. > dimsf(1) = nx > dimsf(2) = ny > call h5screate_simple_f(ndim, dimsf, FileSpace, hdf5_ierr) > > !c Create the dataset with default properties and close filespace. > call h5dcreate_f(file_id, DatasetName, H5T_NATIVE_INTEGER, & > FileSpace, dset_id, hdf5_ierr) > call h5sclose_f(FileSpace, hdf5_ierr) > > !c Each process defines dataset in memory and writes it to the hyperslab > !c in the file. > if (mpi_rank == 3) then > hcount(1) = 0 > hcount(2) = dimsf(2) > else > hcount(1) = dimsf(1)/mpi_size > hcount(2) = dimsf(2) > end if > > offset(1) = mpi_rank*hcount(1) > offset(2) = 0 > call h5screate_simple_f(ndim, hcount, MemSpace, hdf5_ierr) > > if (mpi_rank == 3) then > call h5sselect_none_f(MemSpace,hdf5_ierr) > end if > > !c Select hyperslab in the file. > call h5dget_space_f(dset_id,FileSpace,hdf5_ierr) > call h5sselect_hyperslab_f(FileSpace, H5S_SELECT_SET_F, & > offset, hcount, hdf5_ierr) > > if (mpi_rank == 3) then > call h5sselect_none_f(FileSpace,hdf5_ierr) > end if > > !c Initialize data buffer. > if (mpi_rank == 3) then > allocate(data(0)) > write(*,*) 'Null data space for processor ',mpi_rank > else > allocate(data(hcount(1)*hcount(2))) > write(*,*) 'Data space for processor ',mpi_rank, 'size',size(data) > do i = 1, hcount(1)*hcount(2) > data(i) = mpi_rank+10 > end do > end if > > !c Create property list for collective dataset write. > call h5pcreate_f(H5P_DATASET_XFER_F, plist_id, hdf5_ierr) > call h5pset_dxpl_mpio_f(plist_id, H5FD_MPIO_COLLECTIVE_F, & > hdf5_ierr) > !c write the dataset collectively > call h5dwrite_f(dset_id, H5T_NATIVE_INTEGER, data, hcount, & > hdf5_ierr, file_space_id=FileSpace, & > mem_space_id=MemSpace, xfer_prp = plist_id) > > !c free data. > deallocate(data) > > !c Close/release resources. > call h5dclose_f(dset_id, hdf5_ierr) > call h5sclose_f(FileSpace, hdf5_ierr) > call h5sclose_f(MemSpace, hdf5_ierr) > call h5pclose_f(plist_id, hdf5_ierr) > call h5fclose_f(file_id, hdf5_ierr) > > call PetscFinalize(ierr) > CHKERRQ(ierr) > > return > > end program hdf5_zero_data From danyang.su at gmail.com Sat Jun 13 17:27:10 2020 From: danyang.su at gmail.com (Danyang Su) Date: Sat, 13 Jun 2020 15:27:10 -0700 Subject: [petsc-users] Parallel writing in HDF5-1.12.0 when some processors have no data to write In-Reply-To: <87366y4s5m.fsf@jedbrown.org> References: <360F2E38-6D7C-496E-877A-9177B38D1F6C@gmail.com> <871rml6jsi.fsf@jedbrown.org> <8A7B62D0-0861-455A-8660-C4ECB392B88F@gmail.com> <87k10c5t9w.fsf@jedbrown.org> <87366y4s5m.fsf@jedbrown.org> Message-ID: Hi Jed, These three extra arguments in h5dcreate_f are optional so I am not sure if these three arguments cause the problem. The error is from h5dwrite_f where all the arguments are provided. I will try to see if I can reproduce the problem in C. Thanks, Danyang ?On 2020-06-13, 1:39 PM, "Jed Brown" wrote: Can you reproduce in C? You're missing three extra arguments that exist in the Fortran interface. https://support.hdfgroup.org/HDF5/doc/RM/RM_H5D.html#Dataset-Create Danyang Su writes: > Hi Jed, > > Attached is the example for your test. > > This example uses H5Sset_none to tell H5Dwrite call that there will be no data. 4-th process HAS to participate we are in a collective mode. > The code is ported and modified based on the C example from https://support.hdfgroup.org/ftp/HDF5/examples/parallel/coll_test.c > > The compiling flags in the makefile are same as those used in my own code. > > To compile the code, please run 'make all' > To test the code, please run 'mpiexec -n 4 ./hdf5_zero_data'. Any number of processors larger than 4 should help to detect the problem. > > The code may crash on HDF5 1.12.0 but works fine on HDF5 1.10.x. > > The following platforms have been tested: > Macos-Mojave + GNU-8.2 + HDF5-1.12.0 -> Works fine > Ubuntu-16.04 + GNU-5.4 + HDF5-1.12.0 -> Crashes > Ubuntu-16.04 + GNU-7.5 + HDF5-1.12.0 -> Crashes > Ubuntu-16.04 + GNU-5.4 + HDF5-1.10.x -> Works fine > Centos-7 + Intel2018 + HDF5-12.0 -> Works fine > > Possible error when code crashes > At line 6686 of file H5_gen.F90 > Fortran runtime error: Index '1' of dimension 1 of array 'buf' above upper bound of 0 > > Thanks, > > Danyang > > ?On 2020-06-12, 6:05 AM, "Jed Brown" wrote: > > Danyang Su writes: > > > Hi Jed, > > > > Thanks for your double check. > > > > The HDF 1.10.6 version also works. But versions from 1.12.x stop working. > > I'd suggest making a reduced test case in order to submit a bug report. > > This was the relevant change in PETSc for hdf5-1.12. > > https://gitlab.com/petsc/petsc/commit/806daeb7de397195b5132278177f4d5553f9f612 > > > Attached is the code section where I have problem. > > > > !c write the dataset collectively > > !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! > > !!!! CODE CRASHES HERE IF SOME PROCESSORS HAVE NO DATA TO WRITE!!!! > > !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! > > call h5dwrite_f(dset_id, H5T_NATIVE_DOUBLE, dataset, hdf5_dsize, & > > hdf5_ierr, file_space_id=filespace, & > > mem_space_id=memspace, xfer_prp = xlist_id) > > > > Please let me know if there is something wrong in the code that causes the problem. > > !c > !c This example uses H5Sset_none to tell H5Dwrite call that > !c there will be no data. 4-th process HAS to participate since > !c we are in a collective mode. > !c > !c The code is ported and modified based on the C example > !c from https://support.hdfgroup.org/ftp/HDF5/examples/parallel/coll_test.c > !c by Danyang Su on June 12, 2020. > !c > !c To compile the code, please run 'make all' > !c To test the code, please run 'mpiexec -n 4 ./hdf5_zero_data' > !c > !c IMPORTNANT NOTE > !c The code may crash on HDF5 1.12.0 but works fine on HDF5 1.10.x. > !c > !c The following platforms have been tested: > !c Macos-Mojave + GNU-8.2 + HDF5-1.12.0 -> Works fine > !c Ubuntu-16.04 + GNU-5.4 + HDF5-1.12.0 -> Crashes > !c Ubuntu-16.04 + GNU-7.5 + HDF5-1.12.0 -> Crashes > !c Ubuntu-16.04 + GNU-5.4 + HDF5-1.10.x -> Works fine > !c Centos-7 + Intel2018 + HDF5-12.0 -> Works fine > !c > !c Possible error when code crashes > !c At line 6686 of file H5_gen.F90 > !c Fortran runtime error: Index '1' of dimension 1 of array 'buf' above upper bound of 0 > !c > > program hdf5_zero_data > > > #include > > use petscsys > use hdf5 > > implicit none > > character(len=10), parameter :: h5File_Name = "SDS_row.h5" > character(len=8), parameter :: DatasetName = "IntArray" > integer, parameter :: nx = 8, ny = 5, ndim = 2 > > integer :: i > integer :: hdf5_ierr ! HDF5 error code > PetscErrorCode :: ierr > > integer(HID_T) :: file_id ! File identifier > integer(HID_T) :: dset_id ! Dataset identifier > integer(HID_T) :: space_id ! Space identifier > integer(HID_T) :: plist_id ! Property list identifier > integer(HID_T) :: FileSpace ! File identifier > integer(HID_T) :: MemSpace ! Dataset identifier > > integer(HSIZE_T) :: dimsf(2) ! Dataset dimensions > integer(HSIZE_T) :: hcount(2) ! hyperslab selection parameters > integer(HSIZE_T) :: offset(2) ! hyperslab selection parameters > > integer, allocatable :: data(:) ! Dataset to write > > integer :: mpi_size, mpi_rank; > > !c Initialize MPI. > call PetscInitialize(Petsc_Null_Character,ierr) > CHKERRQ(ierr) > call MPI_Comm_rank(Petsc_Comm_World,mpi_rank,ierr) > CHKERRQ(ierr) > call MPI_Comm_size(Petsc_Comm_World,mpi_size,ierr) > CHKERRQ(ierr) > > !c initialize fortran interface > call h5open_f(hdf5_ierr) > > !c Set up file access property list with parallel I/O access. > call h5pcreate_f(H5P_FILE_ACCESS_F, plist_id, hdf5_ierr) > call h5pset_fapl_mpio_f(plist_id, Petsc_Comm_World, & > MPI_INFO_NULL, hdf5_ierr) > > !c Create a new file collectively and release property list identifier. > call h5fcreate_f(h5File_Name, H5F_ACC_TRUNC_F, file_id, & > hdf5_ierr, access_prp = plist_id) > call h5pclose_f(plist_id, hdf5_ierr) > > !c Create the dataspace for the dataset. > dimsf(1) = nx > dimsf(2) = ny > call h5screate_simple_f(ndim, dimsf, FileSpace, hdf5_ierr) > > !c Create the dataset with default properties and close filespace. > call h5dcreate_f(file_id, DatasetName, H5T_NATIVE_INTEGER, & > FileSpace, dset_id, hdf5_ierr) > call h5sclose_f(FileSpace, hdf5_ierr) > > !c Each process defines dataset in memory and writes it to the hyperslab > !c in the file. > if (mpi_rank == 3) then > hcount(1) = 0 > hcount(2) = dimsf(2) > else > hcount(1) = dimsf(1)/mpi_size > hcount(2) = dimsf(2) > end if > > offset(1) = mpi_rank*hcount(1) > offset(2) = 0 > call h5screate_simple_f(ndim, hcount, MemSpace, hdf5_ierr) > > if (mpi_rank == 3) then > call h5sselect_none_f(MemSpace,hdf5_ierr) > end if > > !c Select hyperslab in the file. > call h5dget_space_f(dset_id,FileSpace,hdf5_ierr) > call h5sselect_hyperslab_f(FileSpace, H5S_SELECT_SET_F, & > offset, hcount, hdf5_ierr) > > if (mpi_rank == 3) then > call h5sselect_none_f(FileSpace,hdf5_ierr) > end if > > !c Initialize data buffer. > if (mpi_rank == 3) then > allocate(data(0)) > write(*,*) 'Null data space for processor ',mpi_rank > else > allocate(data(hcount(1)*hcount(2))) > write(*,*) 'Data space for processor ',mpi_rank, 'size',size(data) > do i = 1, hcount(1)*hcount(2) > data(i) = mpi_rank+10 > end do > end if > > !c Create property list for collective dataset write. > call h5pcreate_f(H5P_DATASET_XFER_F, plist_id, hdf5_ierr) > call h5pset_dxpl_mpio_f(plist_id, H5FD_MPIO_COLLECTIVE_F, & > hdf5_ierr) > !c write the dataset collectively > call h5dwrite_f(dset_id, H5T_NATIVE_INTEGER, data, hcount, & > hdf5_ierr, file_space_id=FileSpace, & > mem_space_id=MemSpace, xfer_prp = plist_id) > > !c free data. > deallocate(data) > > !c Close/release resources. > call h5dclose_f(dset_id, hdf5_ierr) > call h5sclose_f(FileSpace, hdf5_ierr) > call h5sclose_f(MemSpace, hdf5_ierr) > call h5pclose_f(plist_id, hdf5_ierr) > call h5fclose_f(file_id, hdf5_ierr) > > call PetscFinalize(ierr) > CHKERRQ(ierr) > > return > > end program hdf5_zero_data From knepley at gmail.com Sat Jun 13 18:41:28 2020 From: knepley at gmail.com (Matthew Knepley) Date: Sat, 13 Jun 2020 19:41:28 -0400 Subject: [petsc-users] Implementing periodicity using DMPlex In-Reply-To: References: Message-ID: On Fri, Jun 12, 2020 at 3:19 PM Shashwat Tiwari wrote: > Hi, > I am writing a first order 2D solver for unstructured grids with periodic > boundaries using DMPlex. After generating the mesh, I use > "DMSetPeriodicity" function to set periodicity in both directions. After > which I partition the mesh (DMPlexDistribute), construct ghost cells > (DMPlexConstructGhostCells), > These ghost cells are for FVM boundary conditions. If you want cells to be shared across parallel partitions, then you want to give overlap=1 to DMPlexDIstribute(). Is that what you want? Thanks, Matt > create a section, and set some initial values in the global vector. Then I > use "VecGhostUpdateBegin" to start updating the boundary ghost cell values, > but, I get the following error in case I use multiple processors: > > [0]PETSC ERROR: Invalid argument > [0]PETSC ERROR: Vector is not ghosted > [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > > if I run with a single process, there is no error but the values remain > empty (zero) and are not updated. Kindly let me know, if I am missing some > crucial step before I can update the ghost values in order to implement the > periodic bc, or if there is any other approach to achieve it. I am > attaching a small code to demonstrate the issue for your reference. > > Regards, > Shashwat > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From danyang.su at gmail.com Sun Jun 14 01:31:30 2020 From: danyang.su at gmail.com (Danyang Su) Date: Sat, 13 Jun 2020 23:31:30 -0700 Subject: [petsc-users] Parallel writing in HDF5-1.12.0 when some processors have no data to write In-Reply-To: <87366y4s5m.fsf@jedbrown.org> References: <360F2E38-6D7C-496E-877A-9177B38D1F6C@gmail.com> <871rml6jsi.fsf@jedbrown.org> <8A7B62D0-0861-455A-8660-C4ECB392B88F@gmail.com> <87k10c5t9w.fsf@jedbrown.org> <87366y4s5m.fsf@jedbrown.org> Message-ID: <0FFE8772-A490-4DA0-8A8E-4AB6D97BA7F4@gmail.com> Hi Jed, I cannot reproduce the same problem in C (attached example). The C code works fine on HDF 1.12.0 without problem. I am still confused why this happens. Anyway, will use HDF5-1.10.6 with PETSc-3.13 at this moment. Thanks, Danyang ?On 2020-06-13, 1:39 PM, "Jed Brown" wrote: Can you reproduce in C? You're missing three extra arguments that exist in the Fortran interface. https://support.hdfgroup.org/HDF5/doc/RM/RM_H5D.html#Dataset-Create Danyang Su writes: > Hi Jed, > > Attached is the example for your test. > > This example uses H5Sset_none to tell H5Dwrite call that there will be no data. 4-th process HAS to participate we are in a collective mode. > The code is ported and modified based on the C example from https://support.hdfgroup.org/ftp/HDF5/examples/parallel/coll_test.c > > The compiling flags in the makefile are same as those used in my own code. > > To compile the code, please run 'make all' > To test the code, please run 'mpiexec -n 4 ./hdf5_zero_data'. Any number of processors larger than 4 should help to detect the problem. > > The code may crash on HDF5 1.12.0 but works fine on HDF5 1.10.x. > > The following platforms have been tested: > Macos-Mojave + GNU-8.2 + HDF5-1.12.0 -> Works fine > Ubuntu-16.04 + GNU-5.4 + HDF5-1.12.0 -> Crashes > Ubuntu-16.04 + GNU-7.5 + HDF5-1.12.0 -> Crashes > Ubuntu-16.04 + GNU-5.4 + HDF5-1.10.x -> Works fine > Centos-7 + Intel2018 + HDF5-12.0 -> Works fine > > Possible error when code crashes > At line 6686 of file H5_gen.F90 > Fortran runtime error: Index '1' of dimension 1 of array 'buf' above upper bound of 0 > > Thanks, > > Danyang > > ?On 2020-06-12, 6:05 AM, "Jed Brown" wrote: > > Danyang Su writes: > > > Hi Jed, > > > > Thanks for your double check. > > > > The HDF 1.10.6 version also works. But versions from 1.12.x stop working. > > I'd suggest making a reduced test case in order to submit a bug report. > > This was the relevant change in PETSc for hdf5-1.12. > > https://gitlab.com/petsc/petsc/commit/806daeb7de397195b5132278177f4d5553f9f612 > > > Attached is the code section where I have problem. > > > > !c write the dataset collectively > > !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! > > !!!! CODE CRASHES HERE IF SOME PROCESSORS HAVE NO DATA TO WRITE!!!! > > !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! > > call h5dwrite_f(dset_id, H5T_NATIVE_DOUBLE, dataset, hdf5_dsize, & > > hdf5_ierr, file_space_id=filespace, & > > mem_space_id=memspace, xfer_prp = xlist_id) > > > > Please let me know if there is something wrong in the code that causes the problem. > > !c > !c This example uses H5Sset_none to tell H5Dwrite call that > !c there will be no data. 4-th process HAS to participate since > !c we are in a collective mode. > !c > !c The code is ported and modified based on the C example > !c from https://support.hdfgroup.org/ftp/HDF5/examples/parallel/coll_test.c > !c by Danyang Su on June 12, 2020. > !c > !c To compile the code, please run 'make all' > !c To test the code, please run 'mpiexec -n 4 ./hdf5_zero_data' > !c > !c IMPORTNANT NOTE > !c The code may crash on HDF5 1.12.0 but works fine on HDF5 1.10.x. > !c > !c The following platforms have been tested: > !c Macos-Mojave + GNU-8.2 + HDF5-1.12.0 -> Works fine > !c Ubuntu-16.04 + GNU-5.4 + HDF5-1.12.0 -> Crashes > !c Ubuntu-16.04 + GNU-7.5 + HDF5-1.12.0 -> Crashes > !c Ubuntu-16.04 + GNU-5.4 + HDF5-1.10.x -> Works fine > !c Centos-7 + Intel2018 + HDF5-12.0 -> Works fine > !c > !c Possible error when code crashes > !c At line 6686 of file H5_gen.F90 > !c Fortran runtime error: Index '1' of dimension 1 of array 'buf' above upper bound of 0 > !c > > program hdf5_zero_data > > > #include > > use petscsys > use hdf5 > > implicit none > > character(len=10), parameter :: h5File_Name = "SDS_row.h5" > character(len=8), parameter :: DatasetName = "IntArray" > integer, parameter :: nx = 8, ny = 5, ndim = 2 > > integer :: i > integer :: hdf5_ierr ! HDF5 error code > PetscErrorCode :: ierr > > integer(HID_T) :: file_id ! File identifier > integer(HID_T) :: dset_id ! Dataset identifier > integer(HID_T) :: space_id ! Space identifier > integer(HID_T) :: plist_id ! Property list identifier > integer(HID_T) :: FileSpace ! File identifier > integer(HID_T) :: MemSpace ! Dataset identifier > > integer(HSIZE_T) :: dimsf(2) ! Dataset dimensions > integer(HSIZE_T) :: hcount(2) ! hyperslab selection parameters > integer(HSIZE_T) :: offset(2) ! hyperslab selection parameters > > integer, allocatable :: data(:) ! Dataset to write > > integer :: mpi_size, mpi_rank; > > !c Initialize MPI. > call PetscInitialize(Petsc_Null_Character,ierr) > CHKERRQ(ierr) > call MPI_Comm_rank(Petsc_Comm_World,mpi_rank,ierr) > CHKERRQ(ierr) > call MPI_Comm_size(Petsc_Comm_World,mpi_size,ierr) > CHKERRQ(ierr) > > !c initialize fortran interface > call h5open_f(hdf5_ierr) > > !c Set up file access property list with parallel I/O access. > call h5pcreate_f(H5P_FILE_ACCESS_F, plist_id, hdf5_ierr) > call h5pset_fapl_mpio_f(plist_id, Petsc_Comm_World, & > MPI_INFO_NULL, hdf5_ierr) > > !c Create a new file collectively and release property list identifier. > call h5fcreate_f(h5File_Name, H5F_ACC_TRUNC_F, file_id, & > hdf5_ierr, access_prp = plist_id) > call h5pclose_f(plist_id, hdf5_ierr) > > !c Create the dataspace for the dataset. > dimsf(1) = nx > dimsf(2) = ny > call h5screate_simple_f(ndim, dimsf, FileSpace, hdf5_ierr) > > !c Create the dataset with default properties and close filespace. > call h5dcreate_f(file_id, DatasetName, H5T_NATIVE_INTEGER, & > FileSpace, dset_id, hdf5_ierr) > call h5sclose_f(FileSpace, hdf5_ierr) > > !c Each process defines dataset in memory and writes it to the hyperslab > !c in the file. > if (mpi_rank == 3) then > hcount(1) = 0 > hcount(2) = dimsf(2) > else > hcount(1) = dimsf(1)/mpi_size > hcount(2) = dimsf(2) > end if > > offset(1) = mpi_rank*hcount(1) > offset(2) = 0 > call h5screate_simple_f(ndim, hcount, MemSpace, hdf5_ierr) > > if (mpi_rank == 3) then > call h5sselect_none_f(MemSpace,hdf5_ierr) > end if > > !c Select hyperslab in the file. > call h5dget_space_f(dset_id,FileSpace,hdf5_ierr) > call h5sselect_hyperslab_f(FileSpace, H5S_SELECT_SET_F, & > offset, hcount, hdf5_ierr) > > if (mpi_rank == 3) then > call h5sselect_none_f(FileSpace,hdf5_ierr) > end if > > !c Initialize data buffer. > if (mpi_rank == 3) then > allocate(data(0)) > write(*,*) 'Null data space for processor ',mpi_rank > else > allocate(data(hcount(1)*hcount(2))) > write(*,*) 'Data space for processor ',mpi_rank, 'size',size(data) > do i = 1, hcount(1)*hcount(2) > data(i) = mpi_rank+10 > end do > end if > > !c Create property list for collective dataset write. > call h5pcreate_f(H5P_DATASET_XFER_F, plist_id, hdf5_ierr) > call h5pset_dxpl_mpio_f(plist_id, H5FD_MPIO_COLLECTIVE_F, & > hdf5_ierr) > !c write the dataset collectively > call h5dwrite_f(dset_id, H5T_NATIVE_INTEGER, data, hcount, & > hdf5_ierr, file_space_id=FileSpace, & > mem_space_id=MemSpace, xfer_prp = plist_id) > > !c free data. > deallocate(data) > > !c Close/release resources. > call h5dclose_f(dset_id, hdf5_ierr) > call h5sclose_f(FileSpace, hdf5_ierr) > call h5sclose_f(MemSpace, hdf5_ierr) > call h5pclose_f(plist_id, hdf5_ierr) > call h5fclose_f(file_id, hdf5_ierr) > > call PetscFinalize(ierr) > CHKERRQ(ierr) > > return > > end program hdf5_zero_data -------------- next part -------------- A non-text attachment was scrubbed... Name: coll_test.c Type: application/octet-stream Size: 3349 bytes Desc: not available URL: From bsmith at petsc.dev Sun Jun 14 02:19:05 2020 From: bsmith at petsc.dev (Barry Smith) Date: Sun, 14 Jun 2020 02:19:05 -0500 Subject: [petsc-users] Debugging suggestions: GAMG In-Reply-To: References: <24AC8444-94DD-4885-8E04-9107DD7DB673@petsc.dev> Message-ID: <4FC9EF0D-C6BC-4344-A9F9-2C802B023078@petsc.dev> You should be able to use the option -fp_trap on to resolve this problem. I am working on a general fix but it is extensive and has to go into the master branch and cannot be back ported. Barry > On Jun 13, 2020, at 3:06 PM, Sanjay Govindjee wrote: > > Not a lot of luck here; I guess I don't know how to set the break point correctly? > > (gdb) b ieee_handler > Function "ieee_handler" not defined. > Make breakpoint pending on future shared library load? (y or [n]) y > Breakpoint 1 (ieee_handler) pending. > (gdb) c > Continuing. > > Thread 1 "feap" received signal SIGFPE, Arithmetic exception. > 0x00007f8d1a422be1 in ieeeck_ () from /lib64/liblapack.so.3 > > > > On 6/13/20 9:04 AM, Barry Smith wrote: >> You could try in gdb to put a break point in ieee_handler and find all the places it gets called, maybe this will lead to the location of the cause. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From s_g at berkeley.edu Sun Jun 14 02:29:55 2020 From: s_g at berkeley.edu (Sanjay Govindjee) Date: Sun, 14 Jun 2020 00:29:55 -0700 Subject: [petsc-users] Debugging suggestions: GAMG In-Reply-To: <4FC9EF0D-C6BC-4344-A9F9-2C802B023078@petsc.dev> References: <24AC8444-94DD-4885-8E04-9107DD7DB673@petsc.dev> <4FC9EF0D-C6BC-4344-A9F9-2C802B023078@petsc.dev> Message-ID: <785e5462-7d4c-d10d-aeee-411621662292@berkeley.edu> That worked.? Thanks. On 6/14/20 12:19 AM, Barry Smith wrote: > > ? ?You should be able to use the option > > ?-fp_trap on > > to resolve this problem. I am working on a general fix but it is > extensive and has to go into the master branch and cannot be back ported. > > ? Barry > > >> On Jun 13, 2020, at 3:06 PM, Sanjay Govindjee > > wrote: >> >> Not a lot of luck here; I guess I don't know how to set the break >> point correctly? >> >> (gdb) b ieee_handler >> Function "ieee_handler" not defined. >> Make breakpoint pending on future shared library load? (y or [n]) y >> Breakpoint 1 (ieee_handler) pending. >> (gdb) c >> Continuing. >> >> Thread 1 "feap" received signal SIGFPE, Arithmetic exception. >> 0x00007f8d1a422be1 in ieeeck_ () from /lib64/liblapack.so.3 >> >> >> On 6/13/20 9:04 AM, Barry Smith wrote: >>> You could try in gdb to put a break point in ieee_handler and find >>> all the places it gets called, maybe this will lead to the location >>> of the cause. >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From shaswat121994 at gmail.com Mon Jun 15 04:09:54 2020 From: shaswat121994 at gmail.com (Shashwat Tiwari) Date: Mon, 15 Jun 2020 14:39:54 +0530 Subject: [petsc-users] Implementing periodicity using DMPlex In-Reply-To: References: Message-ID: The way I'm trying to implement periodic bc is, when I loop over the boundary faces, say, on the left boundary of the domain to compute flux and residual, I need solution values from the two cells neighbouring the face, i.e. the left cell and the right cell (the face normal pointing from the left cell to the right cell and the right cell being a boundary ghost cell for boundary faces). For the boundary to be periodic, I need the value that I get from the right cell (boundary ghost cell) to be the solution value at its periodic counterpart, i.e. the solution value at left cell of the face on right boundary of the domain in this case. My question is how do I update the value at a boundary ghost cell with the value at the real cell which is its periodic counterpart from the other side of the domain. Is there some kind of mapping of the boundary ghost cells to their corresponding real cells which I can use to update the solution values at ghost cells? Regards, Shashwat On Sun, Jun 14, 2020 at 5:11 AM Matthew Knepley wrote: > On Fri, Jun 12, 2020 at 3:19 PM Shashwat Tiwari > wrote: > >> Hi, >> I am writing a first order 2D solver for unstructured grids with periodic >> boundaries using DMPlex. After generating the mesh, I use >> "DMSetPeriodicity" function to set periodicity in both directions. After >> which I partition the mesh (DMPlexDistribute), construct ghost cells >> (DMPlexConstructGhostCells), >> > > These ghost cells are for FVM boundary conditions. If you want cells to be > shared across parallel partitions, then you want to give overlap=1 > to DMPlexDIstribute(). Is that what you want? > > Thanks, > > Matt > > >> create a section, and set some initial values in the global vector. Then >> I use "VecGhostUpdateBegin" to start updating the boundary ghost cell >> values, but, I get the following error in case I use multiple processors: >> >> [0]PETSC ERROR: Invalid argument >> [0]PETSC ERROR: Vector is not ghosted >> [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html >> for trouble shooting. >> >> if I run with a single process, there is no error but the values remain >> empty (zero) and are not updated. Kindly let me know, if I am missing some >> crucial step before I can update the ghost values in order to implement the >> periodic bc, or if there is any other approach to achieve it. I am >> attaching a small code to demonstrate the issue for your reference. >> >> Regards, >> Shashwat >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefano.zampini at gmail.com Mon Jun 15 04:24:33 2020 From: stefano.zampini at gmail.com (Stefano Zampini) Date: Mon, 15 Jun 2020 12:24:33 +0300 Subject: [petsc-users] Implementing periodicity using DMPlex In-Reply-To: References: Message-ID: <4E40F31A-AD04-4F7C-AA82-B43C9DBEA074@gmail.com> It is enough if you use DMPlexDistribute with 1 level of overlap, set the local section for your dofs, and call DMGlobalToLocal . The local vector will contain data for what you call ?ghost? cells. I call them ?not-owned? cells. > On Jun 15, 2020, at 12:09 PM, Shashwat Tiwari > wrote: > > The way I'm trying to implement periodic bc is, when I loop over the boundary faces, say, on the left boundary of the domain to compute flux and residual, I need solution values from the two cells neighbouring the face, i.e. the left cell and the right cell (the face normal pointing from the left cell to the right cell and the right cell being a boundary ghost cell for boundary faces). For the boundary to be periodic, I need the value that I get from the right cell (boundary ghost cell) to be the solution value at its periodic counterpart, i.e. the solution value at left cell of the face on right boundary of the domain in this case. My question is how do I update the value at a boundary ghost cell with the value at the real cell which is its periodic counterpart from the other side of the domain. Is there some kind of mapping of the boundary ghost cells to their corresponding real cells which I can use to update the solution values at ghost cells? > > Regards, > Shashwat > > On Sun, Jun 14, 2020 at 5:11 AM Matthew Knepley > wrote: > On Fri, Jun 12, 2020 at 3:19 PM Shashwat Tiwari > wrote: > Hi, > I am writing a first order 2D solver for unstructured grids with periodic boundaries using DMPlex. After generating the mesh, I use "DMSetPeriodicity" function to set periodicity in both directions. After which I partition the mesh (DMPlexDistribute), construct ghost cells (DMPlexConstructGhostCells), > > These ghost cells are for FVM boundary conditions. If you want cells to be shared across parallel partitions, then you want to give overlap=1 > to DMPlexDIstribute(). Is that what you want? > > Thanks, > > Matt > > create a section, and set some initial values in the global vector. Then I use "VecGhostUpdateBegin" to start updating the boundary ghost cell values, but, I get the following error in case I use multiple processors: > > [0]PETSC ERROR: Invalid argument > [0]PETSC ERROR: Vector is not ghosted > [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > > if I run with a single process, there is no error but the values remain empty (zero) and are not updated. Kindly let me know, if I am missing some crucial step before I can update the ghost values in order to implement the periodic bc, or if there is any other approach to achieve it. I am attaching a small code to demonstrate the issue for your reference. > > Regards, > Shashwat > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From KJiao at slb.com Mon Jun 15 12:11:43 2020 From: KJiao at slb.com (Kun Jiao) Date: Mon, 15 Jun 2020 17:11:43 +0000 Subject: [petsc-users] configure error Message-ID: Hi, Lately I run into a configure error with intel compiler+openmpi+lapack, Error message as: =============================================================================== TESTING: checkLib from config.packages.BlasLapack(config/BuildSystem/config/packages/BlasLapack.py:114) ******************************************************************************* UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): ------------------------------------------------------------------------------- You set a value for --with-blaslapack-lib=, but ['-L/mnt/disks/intel-2018-3-222-blade-runtime-env-2018-1-07-08-2018-132838/2019.3/intel/compilers_and_libraries_2019.3.199/linux/mkl/lib/intel64', '-Wl,', '--no-as-needed', '-lmkl_intel_ilp64', '-lmkl_intel_thread', '-lmkl_core', '-liomp5', '-lpthread', '-lm', '-ldl'] cannot be used ******************************************************************************* Here is my configure parameters: ./configure --configModules=PETSc.Configure --optionsModule=config.compilerOptions --configModules=PETSc.Configure --optionsModule=config.compilerOptions PETSC_ARCH=linux-gnu-intel --with-precision=single --with-cc=icc --with-cxx=icc --with-fc=ifort --with-mpi-include=/opt/openmpi_4.0.3_intel2019.3/include --with-mpi-lib="-L/opt/openmpi_4.0.3_intel2019.3/lib -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi" --with-blaslapack-include="/mnt/disks/intel-2018-3-222-blade-runtime-env-2018-1-07-08-2018-132838/2019.3/intel/compilers_and_libraries_2019.3.199/linux/mkl/include" --with-blaslapack-lib="-L/mnt/disks/intel-2018-3-222-blade-runtime-env-2018-1-07-08-2018-132838/2019.3/intel/compilers_and_libraries_2019.3.199/linux/mkl/lib/intel64 -Wl, --no-as-needed -lmkl_intel_ilp64 -lmkl_intel_thread -lmkl_core -liomp5 -lpthread -lm -ldl" --with-scalapack-lib="-L/mnt/disks/intel-2018-3-222-blade-runtime-env-2018-1-07-08-2018-132838/2019.3/intel/compilers_and_libraries_2019.3.199/linux/mkl/lib/intel64 -lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64" --with-scalapack-include=/mnt/disks/intel-2018-3-222-blade-runtime-env-2018-1-07-08-2018-132838/2019.3/intel/compilers_and_libraries_2019.3.199/linux/mkl/include --with-cmake=1 --prefix=/opt/petsc_3.10.4_intel2019.3 --known-endian=big --with-debugging=0 --COPTFLAGS=" -Ofast -xHost" --CXXOPTFLAGS=" -Ofast -xHost" --FOPTLAGS=" -Ofast -xHost" --with-x=0 --with-64-bit-indices And configure.log attached. Any suggestion will be great? Regards, Kun Schlumberger-Private -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: configure.log Type: application/octet-stream Size: 2569809 bytes Desc: configure.log URL: From knepley at gmail.com Mon Jun 15 12:25:02 2020 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 15 Jun 2020 12:25:02 -0500 Subject: [petsc-users] configure error In-Reply-To: References: Message-ID: On Mon, Jun 15, 2020 at 12:13 PM Kun Jiao via petsc-users < petsc-users at mcs.anl.gov> wrote: > Hi, > > > > Lately I run into a configure error with intel compiler+openmpi+lapack, > > > > Error message as: > > =============================================================================== > TESTING: checkLib from > config.packages.BlasLapack(config/BuildSystem/config/packages/BlasLapack.py:114) > ******************************************************************************* > > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for > details): > > > ------------------------------------------------------------------------------- > > You set a value for --with-blaslapack-lib=, but > ['-L/mnt/disks/intel-2018-3-222-blade-runtime-env-2018-1-07-08-2018-132838/2019.3/intel/compilers_and_libraries_2019.3.199/linux/mkl/lib/intel64', > '-Wl,', '--no-as-needed', '-lmkl_intel_ilp64', '-lmkl_intel_thread', > '-lmkl_core', '-liomp5', '-lpthread', '-lm', '-ldl'] cannot be used > > > ******************************************************************************* > You input looks wrong: Executing: icc -o /tmp/petsc-srHpJ6/config.libraries/conftest -fPIC -wd1572 -Ofast -xHost /tmp/petsc-srHpJ6/config.libraries/conftest.o -L/mnt/disks/intel-2018-3-222-blade-runtime-env-2018-1-07-08-2018-132838/2019.3/intel/compilers_and_libraries_2019.3.199/linux/mkl/lib/intel64 -Wl, --no-as-needed -lmkl_intel_ilp64 -lmkl_intel_thread -lmkl_core -liomp5 -lpthread -lm -ldl -lstdc++ -ldl -Wl,-rpath,/mnt/disks/intel-2018-3-222-blade-runtime-env-2018-1-07-08-2018-132838/2019.3/intel/compilers_and_libraries_2019.3.199/linux/mpi/intel64/libfabric/lib -L/mnt/disks/intel-2018-3-222-blade-runtime-env-2018-1-07-08-2018-132838/2019.3/intel/compilers_and_libraries_2019.3.199/linux/mpi/intel64/libfabric/lib -Wl,-rpath,/mnt/disks/intel-2018-3-222-blade-runtime-env-2018-1-07-08-2018-132838/2019.3/intel/compilers_and_libraries_2019.3.199/linux/compiler/lib/intel64_lin -L/mnt/disks/intel-2018-3-222-blade-runtime-env-2018-1-07-08-2018-132838/2019.3/intel/compilers_and_libraries_2019.3.199/linux/compiler/lib/intel64_lin -Wl,-rpath,/mnt/disks/intel-2018-3-222-blade-runtime-env-2018-1-07-08-2018-132838/2019.3/intel/compilers_and_libraries_2019.3.199/linux/mkl/lib/intel64_lin -L/mnt/disks/intel-2018-3-222-blade-runtime-env-2018-1-07-08-2018-132838/2019.3/intel/compilers_and_libraries_2019.3.199/linux/mkl/lib/intel64_lin -Wl,-rpath,/opt/2019.3/compilers_and_libraries_2019.3.199/linux/compiler/lib/intel64_lin -L/opt/2019.3/compilers_and_libraries_2019.3.199/linux/compiler/lib/intel64_lin -Wl,-rpath,/opt/gcc-4.9.2/lib/gcc/x86_64-unknown-linux-gnu/4.9.2 -L/opt/gcc-4.9.2/lib/gcc/x86_64-unknown-linux-gnu/4.9.2 -Wl,-rpath,/opt/gcc-4.9.2/lib64 -L/opt/gcc-4.9.2/lib64 -Wl,-rpath,/opt/gcc-4.9.2/lib -L/opt/gcc-4.9.2/lib -lifport -lifcoremt_pic -limf -lsvml -lm -lipgo -lirc -lpthread -lgcc_s -lirc_s -lstdc++ -ldl Possible ERROR while running linker: exit code 256 stderr: icc: command line warning #10006: ignoring unknown option '-fno-as-needed' ld: cannot find -lmkl_intel_ilp64 Thanks, Matt > Here is my configure parameters: > > > > ./configure --configModules=PETSc.Configure > --optionsModule=config.compilerOptions --configModules=PETSc.Configure > --optionsModule=config.compilerOptions PETSC_ARCH=linux-gnu-intel > --with-precision=single --with-cc=icc --with-cxx=icc --with-fc=ifort > --with-mpi-include=/opt/openmpi_4.0.3_intel2019.3/include > --with-mpi-lib="-L/opt/openmpi_4.0.3_intel2019.3/lib -lmpi_usempif08 > -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi" > --with-blaslapack-include="/mnt/disks/intel-2018-3-222-blade-runtime-env-2018-1-07-08-2018-132838/2019.3/intel/compilers_and_libraries_2019.3.199/linux/mkl/include" > --with-blaslapack-lib="-L/mnt/disks/intel-2018-3-222-blade-runtime-env-2018-1-07-08-2018-132838/2019.3/intel/compilers_and_libraries_2019.3.199/linux/mkl/lib/intel64 > -Wl, --no-as-needed -lmkl_intel_ilp64 -lmkl_intel_thread -lmkl_core -liomp5 > -lpthread -lm -ldl" > --with-scalapack-lib="-L/mnt/disks/intel-2018-3-222-blade-runtime-env-2018-1-07-08-2018-132838/2019.3/intel/compilers_and_libraries_2019.3.199/linux/mkl/lib/intel64 > -lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64" > --with-scalapack-include=/mnt/disks/intel-2018-3-222-blade-runtime-env-2018-1-07-08-2018-132838/2019.3/intel/compilers_and_libraries_2019.3.199/linux/mkl/include > --with-cmake=1 --prefix=/opt/petsc_3.10.4_intel2019.3 --known-endian=big > --with-debugging=0 --COPTFLAGS=" -Ofast -xHost" --CXXOPTFLAGS=" -Ofast > -xHost" --FOPTLAGS=" -Ofast -xHost" --with-x=0 --with-64-bit-indices > > > > And configure.log attached. > > > > Any suggestion will be great? > > > > Regards, > > Kun > > > > Schlumberger-Private > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From KJiao at slb.com Mon Jun 15 12:27:54 2020 From: KJiao at slb.com (Kun Jiao) Date: Mon, 15 Jun 2020 17:27:54 +0000 Subject: [petsc-users] [Ext] Re: configure error In-Reply-To: References: Message-ID: Thanks. Let me fix that. Try it again. From: Matthew Knepley Sent: Monday, June 15, 2020 12:25 PM To: Kun Jiao Cc: petsc-users at mcs.anl.gov Subject: [Ext] Re: [petsc-users] configure error On Mon, Jun 15, 2020 at 12:13 PM Kun Jiao via petsc-users > wrote: Hi, Lately I run into a configure error with intel compiler+openmpi+lapack, Error message as: =============================================================================== TESTING: checkLib from config.packages.BlasLapack(config/BuildSystem/config/packages/BlasLapack.py:114) ******************************************************************************* UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): ------------------------------------------------------------------------------- You set a value for --with-blaslapack-lib=, but ['-L/mnt/disks/intel-2018-3-222-blade-runtime-env-2018-1-07-08-2018-132838/2019.3/intel/compilers_and_libraries_2019.3.199/linux/mkl/lib/intel64', '-Wl,', '--no-as-needed', '-lmkl_intel_ilp64', '-lmkl_intel_thread', '-lmkl_core', '-liomp5', '-lpthread', '-lm', '-ldl'] cannot be used ******************************************************************************* You input looks wrong: Executing: icc -o /tmp/petsc-srHpJ6/config.libraries/conftest -fPIC -wd1572 -Ofast -xHost /tmp/petsc-srHpJ6/config.libraries/conftest.o -L/mnt/disks/intel-2018-3-222-blade-runtime-env-2018-1-07-08-2018-132838/2019.3/intel/compilers_and_libraries_2019.3.199/linux/mkl/lib/intel64 -Wl, --no-as-needed -lmkl_intel_ilp64 -lmkl_intel_thread -lmkl_core -liomp5 -lpthread -lm -ldl -lstdc++ -ldl -Wl,-rpath,/mnt/disks/intel-2018-3-222-blade-runtime-env-2018-1-07-08-2018-132838/2019.3/intel/compilers_and_libraries_2019.3.199/linux/mpi/intel64/libfabric/lib -L/mnt/disks/intel-2018-3-222-blade-runtime-env-2018-1-07-08-2018-132838/2019.3/intel/compilers_and_libraries_2019.3.199/linux/mpi/intel64/libfabric/lib -Wl,-rpath,/mnt/disks/intel-2018-3-222-blade-runtime-env-2018-1-07-08-2018-132838/2019.3/intel/compilers_and_libraries_2019.3.199/linux/compiler/lib/intel64_lin -L/mnt/disks/intel-2018-3-222-blade-runtime-env-2018-1-07-08-2018-132838/2019.3/intel/compilers_and_libraries_2019.3.199/linux/compiler/lib/intel64_lin -Wl,-rpath,/mnt/disks/intel-2018-3-222-blade-runtime-env-2018-1-07-08-2018-132838/2019.3/intel/compilers_and_libraries_2019.3.199/linux/mkl/lib/intel64_lin -L/mnt/disks/intel-2018-3-222-blade-runtime-env-2018-1-07-08-2018-132838/2019.3/intel/compilers_and_libraries_2019.3.199/linux/mkl/lib/intel64_lin -Wl,-rpath,/opt/2019.3/compilers_and_libraries_2019.3.199/linux/compiler/lib/intel64_lin -L/opt/2019.3/compilers_and_libraries_2019.3.199/linux/compiler/lib/intel64_lin -Wl,-rpath,/opt/gcc-4.9.2/lib/gcc/x86_64-unknown-linux-gnu/4.9.2 -L/opt/gcc-4.9.2/lib/gcc/x86_64-unknown-linux-gnu/4.9.2 -Wl,-rpath,/opt/gcc-4.9.2/lib64 -L/opt/gcc-4.9.2/lib64 -Wl,-rpath,/opt/gcc-4.9.2/lib -L/opt/gcc-4.9.2/lib -lifport -lifcoremt_pic -limf -lsvml -lm -lipgo -lirc -lpthread -lgcc_s -lirc_s -lstdc++ -ldl Possible ERROR while running linker: exit code 256 stderr: icc: command line warning #10006: ignoring unknown option '-fno-as-needed' ld: cannot find -lmkl_intel_ilp64 Thanks, Matt Here is my configure parameters: ./configure --configModules=PETSc.Configure --optionsModule=config.compilerOptions --configModules=PETSc.Configure --optionsModule=config.compilerOptions PETSC_ARCH=linux-gnu-intel --with-precision=single --with-cc=icc --with-cxx=icc --with-fc=ifort --with-mpi-include=/opt/openmpi_4.0.3_intel2019.3/include --with-mpi-lib="-L/opt/openmpi_4.0.3_intel2019.3/lib -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi" --with-blaslapack-include="/mnt/disks/intel-2018-3-222-blade-runtime-env-2018-1-07-08-2018-132838/2019.3/intel/compilers_and_libraries_2019.3.199/linux/mkl/include" --with-blaslapack-lib="-L/mnt/disks/intel-2018-3-222-blade-runtime-env-2018-1-07-08-2018-132838/2019.3/intel/compilers_and_libraries_2019.3.199/linux/mkl/lib/intel64 -Wl, --no-as-needed -lmkl_intel_ilp64 -lmkl_intel_thread -lmkl_core -liomp5 -lpthread -lm -ldl" --with-scalapack-lib="-L/mnt/disks/intel-2018-3-222-blade-runtime-env-2018-1-07-08-2018-132838/2019.3/intel/compilers_and_libraries_2019.3.199/linux/mkl/lib/intel64 -lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64" --with-scalapack-include=/mnt/disks/intel-2018-3-222-blade-runtime-env-2018-1-07-08-2018-132838/2019.3/intel/compilers_and_libraries_2019.3.199/linux/mkl/include --with-cmake=1 --prefix=/opt/petsc_3.10.4_intel2019.3 --known-endian=big --with-debugging=0 --COPTFLAGS=" -Ofast -xHost" --CXXOPTFLAGS=" -Ofast -xHost" --FOPTLAGS=" -Ofast -xHost" --with-x=0 --with-64-bit-indices And configure.log attached. Any suggestion will be great? Regards, Kun Schlumberger-Private -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ Schlumberger-Private -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Mon Jun 15 12:28:48 2020 From: bsmith at petsc.dev (Barry Smith) Date: Mon, 15 Jun 2020 12:28:48 -0500 Subject: [petsc-users] configure error In-Reply-To: References: Message-ID: It is reporting ld: cannot find -lmkl_intel_ilp64 Please send the output from ls /mnt/disks/intel-2018-3-222-blade-runtime-env-2018-1-07-08-2018-132838/2019.3/intel/compilers_and_libraries_2019.3.199/linux/mkl ls /mnt/disks/intel-2018-3-222-blade-runtime-env-2018-1-07-08-2018-132838/2019.3/intel/compilers_and_libraries_2019.3.199/linux/mkl/lib ls /mnt/disks/intel-2018-3-222-blade-runtime-env-2018-1-07-08-2018-132838/2019.3/intel/compilers_and_libraries_2019.3.199/linux/mkl/lib/intel64 Also you are mixing using the Intel MPI and OpenMPI, I am not sure this will work. /opt/openmpi_4.0.3_intel2019.3/lib > -lmkl_blacs_intelmpi_lp64 > On Jun 15, 2020, at 12:11 PM, Kun Jiao via petsc-users wrote: > > Hi, > > Lately I run into a configure error with intel compiler+openmpi+lapack, > > Error message as: > =============================================================================== TESTING: checkLib from config.packages.BlasLapack(config/BuildSystem/config/packages/BlasLapack.py:114) ******************************************************************************* > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): > ------------------------------------------------------------------------------- > You set a value for --with-blaslapack-lib=, but ['-L/mnt/disks/intel-2018-3-222-blade-runtime-env-2018-1-07-08-2018-132838/2019.3/intel/compilers_and_libraries_2019.3.199/linux/mkl/lib/intel64', '-Wl,', '--no-as-needed', '-lmkl_intel_ilp64', '-lmkl_intel_thread', '-lmkl_core', '-liomp5', '-lpthread', '-lm', '-ldl'] cannot be used > ******************************************************************************* > > > Here is my configure parameters: > > ./configure --configModules=PETSc.Configure --optionsModule=config.compilerOptions --configModules=PETSc.Configure --optionsModule=config.compilerOptions PETSC_ARCH=linux-gnu-intel --with-precision=single --with-cc=icc --with-cxx=icc --with-fc=ifort --with-mpi-include=/opt/openmpi_4.0.3_intel2019.3/include --with-mpi-lib="-L/opt/openmpi_4.0.3_intel2019.3/lib -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi" --with-blaslapack-include="/mnt/disks/intel-2018-3-222-blade-runtime-env-2018-1-07-08-2018-132838/2019.3/intel/compilers_and_libraries_2019.3.199/linux/mkl/include" --with-blaslapack-lib="-L/mnt/disks/intel-2018-3-222-blade-runtime-env-2018-1-07-08-2018-132838/2019.3/intel/compilers_and_libraries_2019.3.199/linux/mkl/lib/intel64 -Wl, --no-as-needed -lmkl_intel_ilp64 -lmkl_intel_thread -lmkl_core -liomp5 -lpthread -lm -ldl" --with-scalapack-lib="-L/mnt/disks/intel-2018-3-222-blade-runtime-env-2018-1-07-08-2018-132838/2019.3/intel/compilers_and_libraries_2019.3.199/linux/mkl/lib/intel64 -lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64" --with-scalapack-include=/mnt/disks/intel-2018-3-222-blade-runtime-env-2018-1-07-08-2018-132838/2019.3/intel/compilers_and_libraries_2019.3.199/linux/mkl/include --with-cmake=1 --prefix=/opt/petsc_3.10.4_intel2019.3 --known-endian=big --with-debugging=0 --COPTFLAGS=" -Ofast -xHost" --CXXOPTFLAGS=" -Ofast -xHost" --FOPTLAGS=" -Ofast -xHost" --with-x=0 --with-64-bit-indices > > And configure.log attached. > > Any suggestion will be great? > > Regards, > Kun > > > Schlumberger-Private > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Jun 15 12:30:05 2020 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 15 Jun 2020 12:30:05 -0500 Subject: [petsc-users] Adaptive remeshing/refining using DMPlex In-Reply-To: References: Message-ID: On Sat, Jun 13, 2020 at 3:42 AM Prateek Gupta wrote: > Hi, > I intend to implement adaptive remeshing and refining in my FE code. > Currently, I am not using the DMPlex class of PETSc. As anyone would be > able to guess, renumbering of mesh nodes and then , in some cases, load > balancing via repartitioning requires a lot of book keeping. > > 1. I am wondering if DMPlex has built in functions which could be used to > automatically renumber the mesh upon operations like edge-flipping in > tetrahedra? > You normally do not need to renumber here, since you start and end with the same number of entities, but the topology changes. This is fairly easy to do in Plex (DMPlexSetCone), but the problem is that these algorithms routinely run over parallel partitions, which is difficult to manage. > 2. If not, then does DAG representation of the grid offer any > ease/performance benefits, if I had to go down and implement? > I think so. Some papers are listed here: https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DMPLEX/index.html > 3. Upon adaptive refining and coarsening, is there a way to dynamically > update my global solution vectors (defined on nodes)? Or do I have to > destroy old vectors and reallocate? > Destroy and reallocate. This is not normally a performance bottleneck, and it would be interesting to see a case where it was. We have interfaces to p4est (mature) and Pragmatic (not mature). Thanks, Matt > Any help/pointers are greatly appreciated :) > > Thank you. > Sincerely, > Prateek Gupta, PhD > D-MAVT, ETHZ > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From KJiao at slb.com Mon Jun 15 12:31:47 2020 From: KJiao at slb.com (Kun Jiao) Date: Mon, 15 Jun 2020 17:31:47 +0000 Subject: [petsc-users] [Ext] Re: configure error In-Reply-To: References: Message-ID: Actually, that is a question I should ask. When I use --with-64-bit-indices option to config petsc, do I need to use 64 integer option in mkl which means -lmkl_intel_ilp64, or I can use 32integer option for mkl which is -lmkl_intel_lp64? Schlumberger-Private From: Barry Smith Sent: Monday, June 15, 2020 12:29 PM To: Kun Jiao Cc: petsc-users at mcs.anl.gov Subject: [Ext] Re: [petsc-users] configure error It is reporting ld: cannot find -lmkl_intel_ilp64 Please send the output from ls /mnt/disks/intel-2018-3-222-blade-runtime-env-2018-1-07-08-2018-132838/2019.3/intel/compilers_and_libraries_2019.3.199/linux/mkl ls /mnt/disks/intel-2018-3-222-blade-runtime-env-2018-1-07-08-2018-132838/2019.3/intel/compilers_and_libraries_2019.3.199/linux/mkl/lib ls /mnt/disks/intel-2018-3-222-blade-runtime-env-2018-1-07-08-2018-132838/2019.3/intel/compilers_and_libraries_2019.3.199/linux/mkl/lib/intel64 Also you are mixing using the Intel MPI and OpenMPI, I am not sure this will work. /opt/openmpi_4.0.3_intel2019.3/lib -lmkl_blacs_intelmpi_lp64 On Jun 15, 2020, at 12:11 PM, Kun Jiao via petsc-users > wrote: Hi, Lately I run into a configure error with intel compiler+openmpi+lapack, Error message as: =============================================================================== TESTING: checkLib from config.packages.BlasLapack(config/BuildSystem/config/packages/BlasLapack.py:114) ******************************************************************************* UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): ------------------------------------------------------------------------------- You set a value for --with-blaslapack-lib=, but ['-L/mnt/disks/intel-2018-3-222-blade-runtime-env-2018-1-07-08-2018-132838/2019.3/intel/compilers_and_libraries_2019.3.199/linux/mkl/lib/intel64', '-Wl,', '--no-as-needed', '-lmkl_intel_ilp64', '-lmkl_intel_thread', '-lmkl_core', '-liomp5', '-lpthread', '-lm', '-ldl'] cannot be used ******************************************************************************* Here is my configure parameters: ./configure --configModules=PETSc.Configure --optionsModule=config.compilerOptions --configModules=PETSc.Configure --optionsModule=config.compilerOptions PETSC_ARCH=linux-gnu-intel --with-precision=single --with-cc=icc --with-cxx=icc --with-fc=ifort --with-mpi-include=/opt/openmpi_4.0.3_intel2019.3/include --with-mpi-lib="-L/opt/openmpi_4.0.3_intel2019.3/lib -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi" --with-blaslapack-include="/mnt/disks/intel-2018-3-222-blade-runtime-env-2018-1-07-08-2018-132838/2019.3/intel/compilers_and_libraries_2019.3.199/linux/mkl/include" --with-blaslapack-lib="-L/mnt/disks/intel-2018-3-222-blade-runtime-env-2018-1-07-08-2018-132838/2019.3/intel/compilers_and_libraries_2019.3.199/linux/mkl/lib/intel64 -Wl, --no-as-needed -lmkl_intel_ilp64 -lmkl_intel_thread -lmkl_core -liomp5 -lpthread -lm -ldl" --with-scalapack-lib="-L/mnt/disks/intel-2018-3-222-blade-runtime-env-2018-1-07-08-2018-132838/2019.3/intel/compilers_and_libraries_2019.3.199/linux/mkl/lib/intel64 -lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64" --with-scalapack-include=/mnt/disks/intel-2018-3-222-blade-runtime-env-2018-1-07-08-2018-132838/2019.3/intel/compilers_and_libraries_2019.3.199/linux/mkl/include --with-cmake=1 --prefix=/opt/petsc_3.10.4_intel2019.3 --known-endian=big --with-debugging=0 --COPTFLAGS=" -Ofast -xHost" --CXXOPTFLAGS=" -Ofast -xHost" --FOPTLAGS=" -Ofast -xHost" --with-x=0 --with-64-bit-indices And configure.log attached. Any suggestion will be great? Regards, Kun Schlumberger-Private -------------- next part -------------- An HTML attachment was scrubbed... URL: From KJiao at slb.com Mon Jun 15 12:35:21 2020 From: KJiao at slb.com (Kun Jiao) Date: Mon, 15 Jun 2020 17:35:21 +0000 Subject: [petsc-users] [Ext] Re: configure error In-Reply-To: References: Message-ID: Great, it fix everything. Sorry, my bad I did not catch this silly error. Regards, Kun From: Matthew Knepley Sent: Monday, June 15, 2020 12:25 PM To: Kun Jiao Cc: petsc-users at mcs.anl.gov Subject: [Ext] Re: [petsc-users] configure error On Mon, Jun 15, 2020 at 12:13 PM Kun Jiao via petsc-users > wrote: Hi, Lately I run into a configure error with intel compiler+openmpi+lapack, Error message as: =============================================================================== TESTING: checkLib from config.packages.BlasLapack(config/BuildSystem/config/packages/BlasLapack.py:114) ******************************************************************************* UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): ------------------------------------------------------------------------------- You set a value for --with-blaslapack-lib=, but ['-L/mnt/disks/intel-2018-3-222-blade-runtime-env-2018-1-07-08-2018-132838/2019.3/intel/compilers_and_libraries_2019.3.199/linux/mkl/lib/intel64', '-Wl,', '--no-as-needed', '-lmkl_intel_ilp64', '-lmkl_intel_thread', '-lmkl_core', '-liomp5', '-lpthread', '-lm', '-ldl'] cannot be used ******************************************************************************* You input looks wrong: Executing: icc -o /tmp/petsc-srHpJ6/config.libraries/conftest -fPIC -wd1572 -Ofast -xHost /tmp/petsc-srHpJ6/config.libraries/conftest.o -L/mnt/disks/intel-2018-3-222-blade-runtime-env-2018-1-07-08-2018-132838/2019.3/intel/compilers_and_libraries_2019.3.199/linux/mkl/lib/intel64 -Wl, --no-as-needed -lmkl_intel_ilp64 -lmkl_intel_thread -lmkl_core -liomp5 -lpthread -lm -ldl -lstdc++ -ldl -Wl,-rpath,/mnt/disks/intel-2018-3-222-blade-runtime-env-2018-1-07-08-2018-132838/2019.3/intel/compilers_and_libraries_2019.3.199/linux/mpi/intel64/libfabric/lib -L/mnt/disks/intel-2018-3-222-blade-runtime-env-2018-1-07-08-2018-132838/2019.3/intel/compilers_and_libraries_2019.3.199/linux/mpi/intel64/libfabric/lib -Wl,-rpath,/mnt/disks/intel-2018-3-222-blade-runtime-env-2018-1-07-08-2018-132838/2019.3/intel/compilers_and_libraries_2019.3.199/linux/compiler/lib/intel64_lin -L/mnt/disks/intel-2018-3-222-blade-runtime-env-2018-1-07-08-2018-132838/2019.3/intel/compilers_and_libraries_2019.3.199/linux/compiler/lib/intel64_lin -Wl,-rpath,/mnt/disks/intel-2018-3-222-blade-runtime-env-2018-1-07-08-2018-132838/2019.3/intel/compilers_and_libraries_2019.3.199/linux/mkl/lib/intel64_lin -L/mnt/disks/intel-2018-3-222-blade-runtime-env-2018-1-07-08-2018-132838/2019.3/intel/compilers_and_libraries_2019.3.199/linux/mkl/lib/intel64_lin -Wl,-rpath,/opt/2019.3/compilers_and_libraries_2019.3.199/linux/compiler/lib/intel64_lin -L/opt/2019.3/compilers_and_libraries_2019.3.199/linux/compiler/lib/intel64_lin -Wl,-rpath,/opt/gcc-4.9.2/lib/gcc/x86_64-unknown-linux-gnu/4.9.2 -L/opt/gcc-4.9.2/lib/gcc/x86_64-unknown-linux-gnu/4.9.2 -Wl,-rpath,/opt/gcc-4.9.2/lib64 -L/opt/gcc-4.9.2/lib64 -Wl,-rpath,/opt/gcc-4.9.2/lib -L/opt/gcc-4.9.2/lib -lifport -lifcoremt_pic -limf -lsvml -lm -lipgo -lirc -lpthread -lgcc_s -lirc_s -lstdc++ -ldl Possible ERROR while running linker: exit code 256 stderr: icc: command line warning #10006: ignoring unknown option '-fno-as-needed' ld: cannot find -lmkl_intel_ilp64 Thanks, Matt Here is my configure parameters: ./configure --configModules=PETSc.Configure --optionsModule=config.compilerOptions --configModules=PETSc.Configure --optionsModule=config.compilerOptions PETSC_ARCH=linux-gnu-intel --with-precision=single --with-cc=icc --with-cxx=icc --with-fc=ifort --with-mpi-include=/opt/openmpi_4.0.3_intel2019.3/include --with-mpi-lib="-L/opt/openmpi_4.0.3_intel2019.3/lib -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi" --with-blaslapack-include="/mnt/disks/intel-2018-3-222-blade-runtime-env-2018-1-07-08-2018-132838/2019.3/intel/compilers_and_libraries_2019.3.199/linux/mkl/include" --with-blaslapack-lib="-L/mnt/disks/intel-2018-3-222-blade-runtime-env-2018-1-07-08-2018-132838/2019.3/intel/compilers_and_libraries_2019.3.199/linux/mkl/lib/intel64 -Wl, --no-as-needed -lmkl_intel_ilp64 -lmkl_intel_thread -lmkl_core -liomp5 -lpthread -lm -ldl" --with-scalapack-lib="-L/mnt/disks/intel-2018-3-222-blade-runtime-env-2018-1-07-08-2018-132838/2019.3/intel/compilers_and_libraries_2019.3.199/linux/mkl/lib/intel64 -lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64" --with-scalapack-include=/mnt/disks/intel-2018-3-222-blade-runtime-env-2018-1-07-08-2018-132838/2019.3/intel/compilers_and_libraries_2019.3.199/linux/mkl/include --with-cmake=1 --prefix=/opt/petsc_3.10.4_intel2019.3 --known-endian=big --with-debugging=0 --COPTFLAGS=" -Ofast -xHost" --CXXOPTFLAGS=" -Ofast -xHost" --FOPTLAGS=" -Ofast -xHost" --with-x=0 --with-64-bit-indices And configure.log attached. Any suggestion will be great? Regards, Kun Schlumberger-Private -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ Schlumberger-Private -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Mon Jun 15 12:40:13 2020 From: balay at mcs.anl.gov (Satish Balay) Date: Mon, 15 Jun 2020 12:40:13 -0500 (CDT) Subject: [petsc-users] [Ext] Re: configure error In-Reply-To: References: Message-ID: You need regular MKL. ILP64 can be used with --with-64-bit-blas-indices option. Configure Options: --configModules=PETSc.Configure --optionsModule=config.compilerOptions --configModules=PETSc.Configure --optionsModule=config.compilerOptions --configModules=PETSc.Configure --optionsModule=config.compilerOptions PETSC_ARCH=linux-gnu-intel --with-precision=single --with-cc=icc --with-cxx=icc --with-fc=ifort --with-mpi-include=/opt/openmpi_4.0.3_intel2019.3/include --with-mpi-lib="-L/opt/openmpi_4.0.3_intel2019.3/lib -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi" --with-blaslapack-include=/mnt/disks/intel-2018-3-222-blade-runtime-env-2018-1-07-08-2018-132838/2019.3/intel/compilers_and_libraries_2019.3.199/linux/mkl/include --with-blaslapack-lib="-L/mnt/disks/intel-2018-3-222-blade-runtime-env-2018-1-07-08-2018-132838/2019.3/intel/compilers_and_libraries_2019.3.199/linux/mkl/lib/intel64 -Wl, --no-as-needed -lmkl_intel_ilp64 -lmkl_intel_thread -lmkl_core -liomp5 -lpthread -lm -ldl" --with-scalapack-lib="-L/mnt/disks/intel-2018-3-222-blade-runtime-e nv-2018-1-07-08-2018-132838/2019.3/intel/compilers_and_libraries_2019.3.199/linux/mkl/lib/intel64 -lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64" --with-scalapack-include=/mnt/disks/intel-2018-3-222-blade-runtime-env-2018-1-07-08-2018-132838/2019.3/intel/compilers_and_libraries_2019.3.199/linux/mkl/include --with-cmake=1 --prefix=/opt/petsc_3.10.4_intel2019.3 --known-endian=big --with-debugging=0 --COPTFLAGS=" -Ofast -xHost" --CXXOPTFLAGS=" -Ofast -xHost" --FOPTLAGS=" -Ofast -xHost" --with-x=0 --with-64-bit-indices This is a complicated build command. What are you trying to do? > --with-precision=single Are you sure this is what you want? > --with-scalapack-lib= PETSc does not use scalapack. Its a dependency for MUMPS [apart from the mixing MPI issue] Its best to use appropriate mpicc etc wrappers so that one doesn't files from multiple MPIs.. And its probably better to use: --with-blaslapack-dir=/mnt/disks/intel-2018-3-222-blade-runtime-env-2018-1-07-08-2018-132838/2019.3/intel/compilers_and_libraries_2019.3.199/linux/mkl Satish On Mon, 15 Jun 2020, Kun Jiao via petsc-users wrote: > Actually, that is a question I should ask. When I use --with-64-bit-indices option to config petsc, do I need to use 64 integer option in mkl which means -lmkl_intel_ilp64, or I can use 32integer option for mkl which is -lmkl_intel_lp64? > > > > Schlumberger-Private > From: Barry Smith > Sent: Monday, June 15, 2020 12:29 PM > To: Kun Jiao > Cc: petsc-users at mcs.anl.gov > Subject: [Ext] Re: [petsc-users] configure error > > > It is reporting > > ld: cannot find -lmkl_intel_ilp64 > > Please send the output from > > ls /mnt/disks/intel-2018-3-222-blade-runtime-env-2018-1-07-08-2018-132838/2019.3/intel/compilers_and_libraries_2019.3.199/linux/mkl > ls /mnt/disks/intel-2018-3-222-blade-runtime-env-2018-1-07-08-2018-132838/2019.3/intel/compilers_and_libraries_2019.3.199/linux/mkl/lib > ls /mnt/disks/intel-2018-3-222-blade-runtime-env-2018-1-07-08-2018-132838/2019.3/intel/compilers_and_libraries_2019.3.199/linux/mkl/lib/intel64 > > Also you are mixing using the Intel MPI and OpenMPI, I am not sure this will work. > > /opt/openmpi_4.0.3_intel2019.3/lib > -lmkl_blacs_intelmpi_lp64 > > > > > On Jun 15, 2020, at 12:11 PM, Kun Jiao via petsc-users > wrote: > > Hi, > > Lately I run into a configure error with intel compiler+openmpi+lapack, > > Error message as: > =============================================================================== TESTING: checkLib from config.packages.BlasLapack(config/BuildSystem/config/packages/BlasLapack.py:114) ******************************************************************************* > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): > ------------------------------------------------------------------------------- > You set a value for --with-blaslapack-lib=, but ['-L/mnt/disks/intel-2018-3-222-blade-runtime-env-2018-1-07-08-2018-132838/2019.3/intel/compilers_and_libraries_2019.3.199/linux/mkl/lib/intel64', '-Wl,', '--no-as-needed', '-lmkl_intel_ilp64', '-lmkl_intel_thread', '-lmkl_core', '-liomp5', '-lpthread', '-lm', '-ldl'] cannot be used > ******************************************************************************* > > > Here is my configure parameters: > > ./configure --configModules=PETSc.Configure --optionsModule=config.compilerOptions --configModules=PETSc.Configure --optionsModule=config.compilerOptions PETSC_ARCH=linux-gnu-intel --with-precision=single --with-cc=icc --with-cxx=icc --with-fc=ifort --with-mpi-include=/opt/openmpi_4.0.3_intel2019.3/include --with-mpi-lib="-L/opt/openmpi_4.0.3_intel2019.3/lib -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi" --with-blaslapack-include="/mnt/disks/intel-2018-3-222-blade-runtime-env-2018-1-07-08-2018-132838/2019.3/intel/compilers_and_libraries_2019.3.199/linux/mkl/include" --with-blaslapack-lib="-L/mnt/disks/intel-2018-3-222-blade-runtime-env-2018-1-07-08-2018-132838/2019.3/intel/compilers_and_libraries_2019.3.199/linux/mkl/lib/intel64 -Wl, --no-as-needed -lmkl_intel_ilp64 -lmkl_intel_thread -lmkl_core -liomp5 -lpthread -lm -ldl" --with-scalapack-lib="-L/mnt/disks/intel-2018-3-222-blade-runtime-env-2018-1-07-08-2018-132838/2019.3/intel/compilers_and_libraries_2019.3.19 9/linux/mkl/lib/intel64 -lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64" --with-scalapack-include=/mnt/disks/intel-2018-3-222-blade-runtime-env-2018-1-07-08-2018-132838/2019.3/intel/compilers_and_libraries_2019.3.199/linux/mkl/include --with-cmake=1 --prefix=/opt/petsc_3.10.4_intel2019.3 --known-endian=big --with-debugging=0 --COPTFLAGS=" -Ofast -xHost" --CXXOPTFLAGS=" -Ofast -xHost" --FOPTLAGS=" -Ofast -xHost" --with-x=0 --with-64-bit-indices > > And configure.log attached. > > Any suggestion will be great? > > Regards, > Kun > > > Schlumberger-Private > > > From KJiao at slb.com Mon Jun 15 12:45:15 2020 From: KJiao at slb.com (Kun Jiao) Date: Mon, 15 Jun 2020 17:45:15 +0000 Subject: [petsc-users] [Ext] Re: configure error In-Reply-To: References: Message-ID: I see. Thanks. We are computing everything on float. Therefore I choose --with-precision=single. Regards, Kun Schlumberger-Private -----Original Message----- From: Satish Balay Sent: Monday, June 15, 2020 12:40 PM To: Kun Jiao Cc: Barry Smith ; petsc-users at mcs.anl.gov Subject: Re: [petsc-users] [Ext] Re: configure error You need regular MKL. ILP64 can be used with --with-64-bit-blas-indices option. Configure Options: --configModules=PETSc.Configure --optionsModule=config.compilerOptions --configModules=PETSc.Configure --optionsModule=config.compilerOptions --configModules=PETSc.Configure --optionsModule=config.compilerOptions PETSC_ARCH=linux-gnu-intel --with-precision=single --with-cc=icc --with-cxx=icc --with-fc=ifort --with-mpi-include=/opt/openmpi_4.0.3_intel2019.3/include --with-mpi-lib="-L/opt/openmpi_4.0.3_intel2019.3/lib -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi" --with-blaslapack-include=/mnt/disks/intel-2018-3-222-blade-runtime-env-2018-1-07-08-2018-132838/2019.3/intel/compilers_and_libraries_2019.3.199/linux/mkl/include --with-blaslapack-lib="-L/mnt/disks/intel-2018-3-222-blade-runtime-env-2018-1-07-08-2018-132838/2019.3/intel/compilers_and_libraries_2019.3.199/linux/mkl/lib/intel64 -Wl, --no-as-needed -lmkl_intel_ilp64 -lmkl_intel_thread -lmkl_core -liomp5 -lpthread -lm -ldl" --with-scalapack-lib="-L/mnt/disks/intel-2018-3-222-blade-runtime-e nv-2018-1-07-08-2018-132838/2019.3/intel/compilers_and_libraries_2019.3.199/linux/mkl/lib/intel64 -lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64" --with-scalapack-include=/mnt/disks/intel-2018-3-222-blade-runtime-env-2018-1-07-08-2018-132838/2019.3/intel/compilers_and_libraries_2019.3.199/linux/mkl/include --with-cmake=1 --prefix=/opt/petsc_3.10.4_intel2019.3 --known-endian=big --with-debugging=0 --COPTFLAGS=" -Ofast -xHost" --CXXOPTFLAGS=" -Ofast -xHost" --FOPTLAGS=" -Ofast -xHost" --with-x=0 --with-64-bit-indices This is a complicated build command. What are you trying to do? > --with-precision=single Are you sure this is what you want? > --with-scalapack-lib= PETSc does not use scalapack. Its a dependency for MUMPS [apart from the mixing MPI issue] Its best to use appropriate mpicc etc wrappers so that one doesn't files from multiple MPIs.. And its probably better to use: --with-blaslapack-dir=/mnt/disks/intel-2018-3-222-blade-runtime-env-2018-1-07-08-2018-132838/2019.3/intel/compilers_and_libraries_2019.3.199/linux/mkl Satish On Mon, 15 Jun 2020, Kun Jiao via petsc-users wrote: > Actually, that is a question I should ask. When I use --with-64-bit-indices option to config petsc, do I need to use 64 integer option in mkl which means -lmkl_intel_ilp64, or I can use 32integer option for mkl which is -lmkl_intel_lp64? > > > > Schlumberger-Private > From: Barry Smith > Sent: Monday, June 15, 2020 12:29 PM > To: Kun Jiao > Cc: petsc-users at mcs.anl.gov > Subject: [Ext] Re: [petsc-users] configure error > > > It is reporting > > ld: cannot find -lmkl_intel_ilp64 > > Please send the output from > > ls > /mnt/disks/intel-2018-3-222-blade-runtime-env-2018-1-07-08-2018-132838 > /2019.3/intel/compilers_and_libraries_2019.3.199/linux/mkl > ls > /mnt/disks/intel-2018-3-222-blade-runtime-env-2018-1-07-08-2018-132838 > /2019.3/intel/compilers_and_libraries_2019.3.199/linux/mkl/lib > ls > /mnt/disks/intel-2018-3-222-blade-runtime-env-2018-1-07-08-2018-132838 > /2019.3/intel/compilers_and_libraries_2019.3.199/linux/mkl/lib/intel64 > > Also you are mixing using the Intel MPI and OpenMPI, I am not sure this will work. > > /opt/openmpi_4.0.3_intel2019.3/lib > -lmkl_blacs_intelmpi_lp64 > > > > > On Jun 15, 2020, at 12:11 PM, Kun Jiao via petsc-users > wrote: > > Hi, > > Lately I run into a configure error with intel > compiler+openmpi+lapack, > > Error message as: > =============================================================================== TESTING: checkLib from config.packages.BlasLapack(config/BuildSystem/config/packages/BlasLapack.py:114) ******************************************************************************* > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): > ---------------------------------------------------------------------- > --------- You set a value for --with-blaslapack-lib=, but > ['-L/mnt/disks/intel-2018-3-222-blade-runtime-env-2018-1-07-08-2018-13 > 2838/2019.3/intel/compilers_and_libraries_2019.3.199/linux/mkl/lib/int > el64', '-Wl,', '--no-as-needed', '-lmkl_intel_ilp64', > '-lmkl_intel_thread', '-lmkl_core', '-liomp5', '-lpthread', '-lm', > '-ldl'] cannot be used > ********************************************************************** > ********* > > > Here is my configure parameters: > > ./configure --configModules=PETSc.Configure > --optionsModule=config.compilerOptions --configModules=PETSc.Configure > --optionsModule=config.compilerOptions PETSC_ARCH=linux-gnu-intel > --with-precision=single --with-cc=icc --with-cxx=icc --with-fc=ifort > --with-mpi-include=/opt/openmpi_4.0.3_intel2019.3/include > --with-mpi-lib="-L/opt/openmpi_4.0.3_intel2019.3/lib -lmpi_usempif08 > -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi" > --with-blaslapack-include="/mnt/disks/intel-2018-3-222-blade-runtime-e > nv-2018-1-07-08-2018-132838/2019.3/intel/compilers_and_libraries_2019. > 3.199/linux/mkl/include" > --with-blaslapack-lib="-L/mnt/disks/intel-2018-3-222-blade-runtime-env > -2018-1-07-08-2018-132838/2019.3/intel/compilers_and_libraries_2019.3. > 199/linux/mkl/lib/intel64 -Wl, --no-as-needed -lmkl_intel_ilp64 > -lmkl_intel_thread -lmkl_core -liomp5 -lpthread -lm -ldl" > --with-scalapack-lib="-L/mnt/disks/intel-2018-3-222-blade-runtime-env- > 2018-1-07-08-2018-132838/2019.3/intel/compilers_and_libraries_2019.3.1 > 9 9/linux/mkl/lib/intel64 -lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64" --with-scalapack-include=/mnt/disks/intel-2018-3-222-blade-runtime-env-2018-1-07-08-2018-132838/2019.3/intel/compilers_and_libraries_2019.3.199/linux/mkl/include --with-cmake=1 --prefix=/opt/petsc_3.10.4_intel2019.3 --known-endian=big --with-debugging=0 --COPTFLAGS=" -Ofast -xHost" --CXXOPTFLAGS=" -Ofast -xHost" --FOPTLAGS=" -Ofast -xHost" --with-x=0 --with-64-bit-indices > > And configure.log attached. > > Any suggestion will be great? > > Regards, > Kun > > > Schlumberger-Private > > > From eda.oktay at metu.edu.tr Tue Jun 16 11:24:22 2020 From: eda.oktay at metu.edu.tr (Eda Oktay) Date: Tue, 16 Jun 2020 19:24:22 +0300 Subject: [petsc-users] Finding mincut after k-means clustering Message-ID: Hello everyone, I am solving a graph partitioning problem. I found an unnormalized Laplacian matrix of a graph, then since I have 4 processes and I am using spectral partitioning algorithm, I calculated 4 eigenvectors corresponding to 4 smallest eigenvalues of this Laplacian matrix. Then, since I am using the k-means clustering algorithm, I formed a matrix U whose columns are these eigenvectors. After that, I clustered each row of U according to the k-means algorithm. Now, I have different row vectors at different processes, as I want. In other words, my eigenvectors have 72 elements. So, right now, I have 72 different row vectors with 4 elements. Those 72 row vectors were clustered but not in a serial order. For instance, the 4th vector is in the first process and the 5th one is in the 4th process. As the last part of my work, I want to find mincut of this partitioning but I couldn't understand how to do it. If I didn't use k-means algorithm, I would use MatPartitioningApply but now, since I clustered row vectors of U according to the index set I obtained from k-means algorithm, I don't know how to use partitioning routines. Should I form another matrix by reordering these vectors and then partition? But then, how can I be sure the vectors stay in the same process as before? Besides MatPartitioning routines, is there any routine I can use for this? I thought MatColoring can work but I guess it won't. Thanks a lot! Eda From junchao.zhang at gmail.com Tue Jun 16 13:47:56 2020 From: junchao.zhang at gmail.com (Junchao Zhang) Date: Tue, 16 Jun 2020 13:47:56 -0500 Subject: [petsc-users] Possible bug PETSc+Complex+CUDA In-Reply-To: References: Message-ID: Hello, Jorge, The problem is fixed in petsc master and you need to add --download-thrust to your configure options to use this feature. Thanks for reporting. --Junchao Zhang On Fri, May 29, 2020 at 7:52 AM Jorge Chiva Segura < jorge.chiva.segura at gmail.com> wrote: > Dear all, > > I just wanted to add what we found in case that it can help to solve this > problem: > > * Examples ex2.c, ex11.c and ex32.c under src/ksp/ksp/tutorials all of > them seg-fault with "-mat_type mpiaijcusparse" or "-vec_type mpicuda" with > CUDA 9.2 and 10.2. > Examples ex2.c and ex32.c work fine if PETSc scalar type is set to real > instead of complex. PETSc has been compiled with gcc-6.4.0 > > * Here: > https://dynamite.readthedocs.io/en/latest/tips.html#gpu-support > it is mentioned that there is some problem for CUDA versions 8 and > above. > > * It seems that the same problem was mentioned in this thread: > > https://lists.mcs.anl.gov/mailman/htdig/petsc-users/2019-February/037748.html > > Please, let me know if you need any extra information. > > Thank you very much for your help. > > Best regards, > > Jorge > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fdkong.jd at gmail.com Wed Jun 17 00:21:09 2020 From: fdkong.jd at gmail.com (Fande Kong) Date: Tue, 16 Jun 2020 23:21:09 -0600 Subject: [petsc-users] Make stream In-Reply-To: <87h7vf68qr.fsf@jedbrown.org> References: <874krkdjjg.fsf@jedbrown.org> <87pna7d0p3.fsf@jedbrown.org> <87h7vf68qr.fsf@jedbrown.org> Message-ID: Thanks, Jed, It is fascinating. I will try to check if I can do anything to have this kind of improvement as well. Thanks, Fande, On Fri, Jun 12, 2020 at 7:43 PM Jed Brown wrote: > Jed Brown writes: > > > Fande Kong writes: > > > >>> There's a lot more to AMG setup than memory bandwidth (architecture > >>> matters a lot, even between different generation CPUs). > >> > >> > >> Could you elaborate a bit more on this? From my understanding, one big > part > >> of AMG SetUp is RAP that should be pretty much bandwidth. > > > > The RAP isn't "pretty much bandwidth". See below for some > > Skylake/POWER9/EPYC results and analysis (copied from an off-list > > thread). I'll leave in some other bandwidth comments that may or may > > not be relevant to you. The short story is that Skylake and EPYC are > > both much better than POWER9 at MatPtAP despite POWER9 having similar > > bandwidth as EPYC and thus being significantly faster than Skylake for > > MatMult/smoothing. > > > > > > Jed Brown writes: > > > >> I'm attaching a log from my machine (Noether), which is 2-socket EPYC > >> 7452 (32 cores each). Each socket has 8xDDR4-3200 and 128 MB of L3 > >> cache. This is the same node architecture as the new BER/E3SM machine > >> being installed at Argonne (though that one will probably have > >> higher-clocked and/or more cores per socket). Note that these CPUs are > >> about $2k each while Skylake 8180 are about $10k. > >> > >> Some excerpts/comments below. > >> > > > > [...] > > > > In addition to the notes below, I'd like to call out how important > > streaming stores are on EPYC. With vanilla code or _mm256_store_pd, we > > get the following performance > > > > $ mpiexec -n 64 --bind-to core --map-by core:1 > src/benchmarks/streams/MPIVersion > > Copy 162609.2392 Scale 159119.8259 Add 174687.6250 Triad > 175840.1587 > > > > but replacing _mm256_store_pd with _mm256_stream_pd gives this > > > > $ mpiexec -n 64 --bind-to core --map-by core:1 > src/benchmarks/streams/MPIVersion > > Copy 259951.9936 Scale 259381.0589 Add 250216.3389 Triad > 249292.9701 > > I turned on NPS4 (a BIOS setting that creates a NUMA node for each pair > of memory channels) and get a modest performance boost. > > $ mpiexec -n 64 --bind-to core --map-by core:1 > src/benchmarks/streams/MPIVersion > > Copy 289645.3776 Scale 289186.2783 Add 273220.0133 Triad 272911.2263 > > On this architecture, best performance comes from one process per 4-core > CCX (shared L3). > > $ mpiexec -n 16 --bind-to core --map-by core:4 > src/benchmarks/streams/MPIVersion > > Copy 300704.8859 Scale 304556.3380 Add 295970.1132 Triad 298891.3821 > > > This is just preposterously huge, but very repeatable using gcc and > > clang, and inspecting the assembly. This suggests that it would be > > useful for vector kernels to have streaming and non-streaming variants. > > That is, if I drop the vector length by 20 (so the working set is 2.3 > > MB/core instead of 46 MB in the default version), then we get 2.4 TB/s > > Triad with _mm256_store_pd: > > > > $ mpiexec -n 64 --bind-to core --map-by core:1 > src/benchmarks/streams/MPIVersion > > Copy 2159915.7058 Scale 2212671.7087 Add 2414758.2757 Triad > 2402671.1178 > > > > and a thoroughly embarrassing 353 GB/s with _mm256_stream_pd: > > > > $ mpiexec -n 64 --bind-to core --map-by core:1 > src/benchmarks/streams/MPIVersion > > Copy 235934.6653 Scale 237446.8507 Add 352805.7288 Triad > 352992.9692 > > > > > > I don't know a good way to automatically determine whether to expect the > > memory to be in cache, but we could make it a global (or per-object) > > run-time selection. > > > >> Jed Brown writes: > >> > >>> "Smith, Barry F." writes: > >>> > >>>> Thanks. The PowerPC is pretty crappy compared to Skylake. > >>> > >>> Compare the MGSmooth times. The POWER9 is faster than the Skylake > >>> because it has more memory bandwidth. > >>> > >>> $ rg 'MGInterp Level 4|MGSmooth Level 4' ex56* > >>> ex56-JLSE-skylake-56ranks-converged.txt > >>> 254:MGSmooth Level 4 68 1.0 1.8808e+00 1.2 7.93e+08 1.3 3.6e+04 > 1.9e+04 3.4e+01 8 29 10 16 3 62 60 18 54 25 22391 > >>> 256:MGInterp Level 4 68 1.0 4.0043e-01 1.8 1.45e+08 1.3 2.2e+04 > 2.5e+03 0.0e+00 1 5 6 1 0 9 11 11 4 0 19109 > >>> > >>> ex56-summit-cpu-36ranks-converged.txt > >>> 265:MGSmooth Level 4 68 1.0 1.1531e+00 1.1 1.22e+09 1.2 2.3e+04 > 2.6e+04 3.4e+01 3 29 7 13 3 61 60 12 54 25 36519 0 0 > 0.00e+00 0 0.00e+00 0 > >>> 267:MGInterp Level 4 68 1.0 2.0749e-01 1.1 2.23e+08 1.2 1.4e+04 > 3.4e+03 0.0e+00 0 5 4 1 0 11 11 7 4 0 36925 0 0 > 0.00e+00 0 0.00e+00 0 > >>> > >>> ex56-summit-gpu-24ranks-converged.txt > >>> 275:MGSmooth Level 4 68 1.0 1.4499e-01 1.2 1.85e+09 1.2 1.0e+04 > 5.3e+04 3.4e+01 0 29 7 13 3 26 60 12 55 25 299156 940881 115 > 2.46e+01 116 8.64e+01 100 > >>> 277:MGInterp Level 4 68 1.0 1.7674e-01 1.0 3.23e+08 1.2 6.1e+03 > 6.7e+03 0.0e+00 0 5 4 1 0 33 11 7 4 0 42715 621223 36 > 2.98e+01 136 3.95e+00 100 > >>> > >>> ex56-summit-gpu-36ranks-converged.txt > >>> 275:MGSmooth Level 4 68 1.0 1.4877e-01 1.2 1.25e+09 1.2 2.3e+04 > 2.6e+04 3.4e+01 0 29 7 13 3 19 60 12 54 25 291548 719522 115 > 1.83e+01 116 5.80e+01 100 > >>> 277:MGInterp Level 4 68 1.0 2.4317e-01 1.0 2.20e+08 1.2 1.4e+04 > 3.4e+03 0.0e+00 0 5 4 1 0 33 11 7 4 0 31062 586044 36 > 1.99e+01 136 2.82e+00 100 > >> > >> 258:MGSmooth Level 4 68 1.0 9.6950e-01 1.3 6.15e+08 1.3 4.0e+04 > 1.4e+04 2.0e+00 6 28 10 15 0 59 59 18 54 25 39423 > >> 260:MGInterp Level 4 68 1.0 2.5707e-01 1.5 1.23e+08 1.2 2.7e+04 > 1.9e+03 0.0e+00 1 5 7 1 0 13 12 12 5 0 29294 > >> > >> Epyc is faster than Power9 is faster than Sklake. > >> > >>> > >>> The Skylake is a lot faster at PtAP. It'd be interesting to better > >>> understand why. Perhaps it has to do with caching or aggressiveness of > >>> out-of-order execution. > >>> > >>> $ rg 'PtAP' ex56* > >>> ex56-JLSE-skylake-56ranks-converged.txt > >>> 164:MatPtAP 4 1.0 1.4214e+00 1.0 3.94e+08 1.5 1.1e+04 > 7.4e+04 4.4e+01 6 13 3 20 4 8 28 8 39 5 13754 > >>> 165:MatPtAPSymbolic 4 1.0 8.3981e-01 1.0 0.00e+00 0.0 6.5e+03 > 7.3e+04 2.8e+01 4 0 2 12 2 5 0 5 23 3 0 > >>> 166:MatPtAPNumeric 4 1.0 5.8402e-01 1.0 3.94e+08 1.5 4.5e+03 > 7.5e+04 1.6e+01 2 13 1 8 1 3 28 3 16 2 33474 > >>> > >>> ex56-summit-cpu-36ranks-converged.txt > >>> 164:MatPtAP 4 1.0 3.9077e+00 1.0 5.89e+08 1.4 1.6e+04 > 7.4e+04 4.4e+01 9 13 5 26 4 11 28 12 46 5 4991 0 0 > 0.00e+00 0 0.00e+00 0 > >>> 165:MatPtAPSymbolic 4 1.0 1.9525e+00 1.0 0.00e+00 0.0 1.2e+04 > 7.3e+04 2.8e+01 5 0 4 19 3 5 0 9 34 3 0 0 0 > 0.00e+00 0 0.00e+00 0 > >>> 166:MatPtAPNumeric 4 1.0 1.9621e+00 1.0 5.89e+08 1.4 4.0e+03 > 7.5e+04 1.6e+01 5 13 1 7 1 5 28 3 12 2 9940 0 0 > 0.00e+00 0 0.00e+00 0 > >>> > >>> ex56-summit-gpu-24ranks-converged.txt > >>> 167:MatPtAP 4 1.0 5.7210e+00 1.0 8.48e+08 1.3 7.5e+03 > 1.3e+05 4.4e+01 8 13 5 25 4 11 28 12 46 5 3415 0 16 > 3.36e+01 4 6.30e-02 0 > >>> 168:MatPtAPSymbolic 4 1.0 2.8717e+00 1.0 0.00e+00 0.0 5.5e+03 > 1.3e+05 2.8e+01 4 0 4 19 3 5 0 9 34 3 0 0 0 > 0.00e+00 0 0.00e+00 0 > >>> 169:MatPtAPNumeric 4 1.0 2.8537e+00 1.0 8.48e+08 1.3 2.0e+03 > 1.3e+05 1.6e+01 4 13 1 7 1 5 28 3 12 2 6846 0 16 > 3.36e+01 4 6.30e-02 0 > >>> > >>> ex56-summit-gpu-36ranks-converged.txt > >>> 167:MatPtAP 4 1.0 4.0340e+00 1.0 5.89e+08 1.4 1.6e+04 > 7.4e+04 4.4e+01 8 13 5 26 4 11 28 12 46 5 4835 0 16 > 2.30e+01 4 5.18e-02 0 > >>> 168:MatPtAPSymbolic 4 1.0 2.0355e+00 1.0 0.00e+00 0.0 1.2e+04 > 7.3e+04 2.8e+01 4 0 4 19 3 5 0 9 34 3 0 0 0 > 0.00e+00 0 0.00e+00 0 > >>> 169:MatPtAPNumeric 4 1.0 2.0050e+00 1.0 5.89e+08 1.4 4.0e+03 > 7.5e+04 1.6e+01 4 13 1 7 1 5 28 3 12 2 9728 0 16 > 2.30e+01 4 5.18e-02 0 > >> > >> 153:MatPtAPSymbolic 4 1.0 7.6053e-01 1.0 0.00e+00 0.0 7.6e+03 > 5.8e+04 2.8e+01 5 0 2 12 2 6 0 5 22 3 0 > >> 154:MatPtAPNumeric 4 1.0 6.5172e-01 1.0 3.21e+08 1.4 6.4e+03 > 4.8e+04 2.4e+01 4 14 2 8 2 5 27 4 16 2 28861 > >> > >> Epyc similar to Skylake here. > >> > >>> I'd really like to compare an EPYC for these operations. I bet it's > >>> pretty good. (More bandwidth than Skylake, bigger caches, but no > >>> AVX512.) > >>> > >>>> So the biggest consumer is MatPtAP I guess that should be done > first. > >>>> > >>>> It would be good to have these results exclude the Jacobian and > Function evaluation which really dominate the time and add clutter making > it difficult to see the problems with the rest of SNESSolve. > >>>> > >>>> > >>>> Did you notice: > >>>> > >>>> MGInterp Level 4 68 1.0 1.7674e-01 1.0 3.23e+08 1.2 6.1e+03 > 6.7e+03 0.0e+00 0 5 4 1 0 33 11 7 4 0 42715 621223 36 > 2.98e+01 136 3.95e+00 100 > >>>> > >>>> it is terrible! Well over half of the KSPSolve time is in this one > relatively minor routine. All of the interps are terribly slow. Is it > related to the transpose multiple or something? > >>> > >>> Yes, it's definitely the MatMultTranspose, which must be about 3x more > >>> expensive than restriction even on the CPU. PCMG/PCGAMG should > >>> explicitly transpose (unless the user sets an option to aggressively > >>> minimize memory usage). > >>> > >>> $ rg 'MGInterp|MultTrans' ex56* > >>> ex56-JLSE-skylake-56ranks-converged.txt > >>> 222:MatMultTranspose 136 1.0 3.5105e-01 3.7 7.91e+07 1.3 2.5e+04 > 1.3e+03 0.0e+00 1 3 7 1 0 5 6 13 3 0 11755 > >>> 247:MGInterp Level 1 68 1.0 3.3894e-04 2.2 2.35e+05 0.0 0.0e+00 > 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 693 > >>> 250:MGInterp Level 2 68 1.0 1.1212e-0278.0 1.17e+06 0.0 1.8e+03 > 7.7e+02 0.0e+00 0 0 1 0 0 0 0 1 0 0 2172 > >>> 253:MGInterp Level 3 68 1.0 6.7105e-02 5.3 1.23e+07 1.8 2.7e+04 > 4.2e+02 0.0e+00 0 0 8 0 0 1 1 14 1 0 8594 > >>> 256:MGInterp Level 4 68 1.0 4.0043e-01 1.8 1.45e+08 1.3 2.2e+04 > 2.5e+03 0.0e+00 1 5 6 1 0 9 11 11 4 0 19109 > >>> > >>> ex56-summit-cpu-36ranks-converged.txt > >>> 229:MatMultTranspose 136 1.0 1.4832e-01 1.4 1.21e+08 1.2 1.9e+04 > 1.5e+03 0.0e+00 0 3 6 1 0 6 6 10 3 0 27842 0 0 > 0.00e+00 0 0.00e+00 0 > >>> 258:MGInterp Level 1 68 1.0 2.9145e-04 1.5 1.08e+05 0.0 0.0e+00 > 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 370 0 0 > 0.00e+00 0 0.00e+00 0 > >>> 261:MGInterp Level 2 68 1.0 5.7095e-03 1.5 9.16e+05 2.5 2.4e+03 > 7.1e+02 0.0e+00 0 0 1 0 0 0 0 1 0 0 4093 0 0 > 0.00e+00 0 0.00e+00 0 > >>> 264:MGInterp Level 3 68 1.0 3.5654e-02 2.8 1.77e+07 1.5 2.3e+04 > 3.9e+02 0.0e+00 0 0 7 0 0 1 1 12 1 0 16095 0 0 > 0.00e+00 0 0.00e+00 0 > >>> 267:MGInterp Level 4 68 1.0 2.0749e-01 1.1 2.23e+08 1.2 1.4e+04 > 3.4e+03 0.0e+00 0 5 4 1 0 11 11 7 4 0 36925 0 0 > 0.00e+00 0 0.00e+00 0 > >>> > >>> ex56-summit-gpu-24ranks-converged.txt > >>> 236:MatMultTranspose 136 1.0 2.1445e-01 1.0 1.72e+08 1.2 9.5e+03 > 2.6e+03 0.0e+00 0 3 6 1 0 39 6 11 3 0 18719 451131 8 > 3.11e+01 272 2.19e+00 100 > >>> 268:MGInterp Level 1 68 1.0 4.0388e-03 2.8 1.08e+05 0.0 0.0e+00 > 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 27 79 37 > 5.84e-04 68 6.80e-05 100 > >>> 271:MGInterp Level 2 68 1.0 2.9033e-02 2.9 1.25e+06 1.9 1.6e+03 > 7.8e+02 0.0e+00 0 0 1 0 0 5 0 2 0 0 812 11539 36 > 1.14e-01 136 5.41e-02 100 > >>> 274:MGInterp Level 3 68 1.0 4.9503e-02 1.1 2.50e+07 1.4 1.1e+04 > 6.3e+02 0.0e+00 0 0 7 0 0 9 1 13 1 0 11476 100889 36 > 2.29e+00 136 3.74e-01 100 > >>> 277:MGInterp Level 4 68 1.0 1.7674e-01 1.0 3.23e+08 1.2 6.1e+03 > 6.7e+03 0.0e+00 0 5 4 1 0 33 11 7 4 0 42715 621223 36 > 2.98e+01 136 3.95e+00 100 > >>> > >>> ex56-summit-gpu-36ranks-converged.txt > >>> 236:MatMultTranspose 136 1.0 2.9692e-01 1.0 1.17e+08 1.2 1.9e+04 > 1.5e+03 0.0e+00 1 3 6 1 0 40 6 10 3 0 13521 336701 8 > 2.08e+01 272 1.59e+00 100 > >>> 268:MGInterp Level 1 68 1.0 3.8752e-03 2.5 1.03e+05 0.0 0.0e+00 > 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 27 79 37 > 3.95e-04 68 4.53e-05 100 > >>> 271:MGInterp Level 2 68 1.0 3.5465e-02 2.2 9.12e+05 2.5 2.4e+03 > 7.1e+02 0.0e+00 0 0 1 0 0 4 0 1 0 0 655 5989 36 > 8.16e-02 136 4.89e-02 100 > >>> 274:MGInterp Level 3 68 1.0 6.7101e-02 1.1 1.75e+07 1.5 2.3e+04 > 3.9e+02 0.0e+00 0 0 7 0 0 9 1 12 1 0 8455 56175 36 > 1.55e+00 136 3.03e-01 100 > >>> 277:MGInterp Level 4 68 1.0 2.4317e-01 1.0 2.20e+08 1.2 1.4e+04 > 3.4e+03 0.0e+00 0 5 4 1 0 33 11 7 4 0 31062 586044 36 > 1.99e+01 136 2.82e+00 100 > >> > >> 223:MatMultTranspose 136 1.0 2.0702e-01 2.9 6.59e+07 1.2 2.7e+04 > 1.1e+03 0.0e+00 1 3 7 1 0 7 6 12 3 0 19553 > >> 251:MGInterp Level 1 68 1.0 2.8062e-04 1.5 9.79e+04 0.0 0.0e+00 > 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 349 > >> 254:MGInterp Level 2 68 1.0 6.2506e-0331.9 9.69e+05 0.0 2.1e+03 > 6.3e+02 0.0e+00 0 0 1 0 0 0 0 1 0 0 3458 > >> 257:MGInterp Level 3 68 1.0 4.8159e-02 6.5 9.62e+06 1.5 2.5e+04 > 4.2e+02 0.0e+00 0 0 6 0 0 1 1 11 1 0 11199 > >> 260:MGInterp Level 4 68 1.0 2.5707e-01 1.5 1.23e+08 1.2 2.7e+04 > 1.9e+03 0.0e+00 1 5 7 1 0 13 12 12 5 0 29294 > >> > >> Power9 still has an edge here. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Bruce.Palmer at pnnl.gov Wed Jun 17 09:31:06 2020 From: Bruce.Palmer at pnnl.gov (Palmer, Bruce J) Date: Wed, 17 Jun 2020 14:31:06 +0000 Subject: [petsc-users] Suppress printing in PETSc Message-ID: Hi, I think I may have asked this in the past, but I can?t seem to find the answer in the archive. Is there a way to suppress all printing from PETSc? Bruce Palmer Senior Research Scientist Pacific Northwest National Laboratory Richland, WA 99352 (509) 375-3899 -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Wed Jun 17 10:03:41 2020 From: bsmith at petsc.dev (Barry Smith) Date: Wed, 17 Jun 2020 10:03:41 -0500 Subject: [petsc-users] Suppress printing in PETSc In-Reply-To: References: Message-ID: <7C8281C4-DAAD-43F9-99E1-63598893DD5D@petsc.dev> Bruce, Normally PETSc does does not print output unless you ask it, except error messages. In C and C++ you can set the global variables FILE *PETSC_STDOUT and FILE *PETSC_STDERR to any file you like, for example /dev/null. In this way anything PETSc prints will just disappear. Barry > On Jun 17, 2020, at 9:31 AM, Palmer, Bruce J via petsc-users wrote: > > Hi, > > I think I may have asked this in the past, but I can?t seem to find the answer in the archive. Is there a way to suppress all printing from PETSc? > > Bruce Palmer > Senior Research Scientist > Pacific Northwest National Laboratory > Richland, WA 99352 > (509) 375-3899 -------------- next part -------------- An HTML attachment was scrubbed... URL: From flw at rzg.mpg.de Wed Jun 17 10:49:33 2020 From: flw at rzg.mpg.de (flw at rzg.mpg.de) Date: Wed, 17 Jun 2020 17:49:33 +0200 Subject: [petsc-users] SuperLU_Dist in single Message-ID: <20200617174933.Horde.-E8Gv8ZH_qLIsnFfkNT8P3d@webmail.mpcdf.mpg.de> Dear PETSc Team, is it just me or is SuperLU_Dist only available in double precision so far? If not, is this something that could be addressed in future releases? With best regards, Felix Wilms From balay at mcs.anl.gov Wed Jun 17 11:05:27 2020 From: balay at mcs.anl.gov (Satish Balay) Date: Wed, 17 Jun 2020 11:05:27 -0500 (CDT) Subject: [petsc-users] SuperLU_Dist in single In-Reply-To: <20200617174933.Horde.-E8Gv8ZH_qLIsnFfkNT8P3d@webmail.mpcdf.mpg.de> References: <20200617174933.Horde.-E8Gv8ZH_qLIsnFfkNT8P3d@webmail.mpcdf.mpg.de> Message-ID: Its just double precision. cc: Sherry Satish On Wed, 17 Jun 2020, flw at rzg.mpg.de wrote: > Dear PETSc Team, > is it just me or is SuperLU_Dist only available in double precision so far? If > not, is this something that could be addressed in future releases? > > With best regards, > Felix Wilms > From xsli at lbl.gov Wed Jun 17 11:27:50 2020 From: xsli at lbl.gov (Xiaoye S. Li) Date: Wed, 17 Jun 2020 09:27:50 -0700 Subject: [petsc-users] SuperLU_Dist in single In-Reply-To: References: <20200617174933.Horde.-E8Gv8ZH_qLIsnFfkNT8P3d@webmail.mpcdf.mpg.de> Message-ID: We plan to release single precision version soon. -Sherry On Wed, Jun 17, 2020 at 9:11 AM Satish Balay wrote: > Its just double precision. > > cc: Sherry > > Satish > > On Wed, 17 Jun 2020, flw at rzg.mpg.de wrote: > > > Dear PETSc Team, > > is it just me or is SuperLU_Dist only available in double precision so > far? If > > not, is this something that could be addressed in future releases? > > > > With best regards, > > Felix Wilms > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mukkundsunjii at gmail.com Wed Jun 17 12:32:10 2020 From: mukkundsunjii at gmail.com (MUKKUND SUNJII) Date: Wed, 17 Jun 2020 19:32:10 +0200 Subject: [petsc-users] Regarding P4est Message-ID: Greetings, I am a master?s student working on the shallow water model of the TS example 'ex11.c' as part of my thesis. Therefore, I am working with DMForest for the implementation of adaptive grids. I have a question and an observation. I am trying to find relevant information about interpolation that takes place through the routine DMForestTransferVec. Perhaps it could be my inability to find it, but I am unable to locate the implementation of the routine (forest->transfervec)(dmIn,vecIn,dmOut,vecOut,useBCs,time). Any information on this particular routine is highly appreciated. Furthermore, I have developed a well balanced Riemann Solver that includes topography in the model. In the process of testing both the non-adaptive and adaptive version, I found that my results differed when I changed the type of DM. For instance, when I run a scenario in a fixed, non-adaptive grid with a DM of type 'P4est', I find that the well balanced nature is lost due to small perturbations all across the domain. However, this does not occur when I use a DM of type ?plex?. Is there a radical change in the routines between the two DM?s? This is not as much of a question as it is an observation. Thank you for all of your suggestions! Regards, Mukkund -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Wed Jun 17 13:47:21 2020 From: mfadams at lbl.gov (Mark Adams) Date: Wed, 17 Jun 2020 14:47:21 -0400 Subject: [petsc-users] Regarding P4est In-Reply-To: References: Message-ID: So you get this noise with a regular grid in p4est. So the same grid as will Plex, and you are not getting the same results. I don't know of any difference from p4est on a non-adapted grid. Can you reproduce this with ex11? Matt and Toby could answer this better. On Wed, Jun 17, 2020 at 1:33 PM MUKKUND SUNJII wrote: > Greetings, > > I am a master?s student working on the shallow water model of the TS > example 'ex11.c' as part of my thesis. Therefore, I am working with > DMForest for the implementation of adaptive grids. I have a question and an > observation. > > I am trying to find relevant information about interpolation that takes > place through the routine DMForestTransferVec. Perhaps it could be my > inability to find it, but I am unable to locate the implementation of the > routine > > (forest->transfervec)(dmIn,vecIn,dmOut,vecOut,useBCs,time). > > Any information on this particular routine is highly appreciated. > > Furthermore, I have developed a well balanced Riemann Solver that includes > topography in the model. In the process of testing both the non-adaptive > and adaptive version, I found that my results differed when I changed the > type of DM. For instance, when I run a scenario in a fixed, non-adaptive > grid with a DM of type 'P4est', I find that the well balanced nature is > lost due to small perturbations all across the domain. However, this does > not occur when I use a DM of type ?plex?. Is there a radical change in the > routines between the two DM?s? This is not as much of a question as it is > an observation. > > Thank you for all of your suggestions! > > Regards, > > Mukkund > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mukkundsunjii at gmail.com Wed Jun 17 14:20:29 2020 From: mukkundsunjii at gmail.com (MUKKUND SUNJII) Date: Wed, 17 Jun 2020 21:20:29 +0200 Subject: [petsc-users] Regarding P4est In-Reply-To: References: Message-ID: Yes, precisely! I am not sure how I can replicate using the original version of ex11.c because it does not support bathymetry. Regardless, to demonstrate the discrepancy, I have uploaded three plots. The scenario is a lake at rest. Essentially, you have a varying bathymetry but a level water surface. If the model is well balanced, then the water surface height must not change. The description of the files are below 1) Bathymetry.png : It shows you the bathymetry profile (z(x)) and the water surface height (H = h+z(x)) at t = 0. 2) Plex.png : This is the water surface height after 1 time step (0.007055 sec) and the dm type is Plex. As you can see, the water surface height is undisturbed as expected. 3) P4est.png : This is the result after 1 time step (same final time) if I set the dm type as p4est. The noise is in the order of 1e-3 to be a little more specific. Since its not specifically at the boundaries and more or less spread throughout, it could indeed be noise introduced. But of course I could be wrong. Maybe this paints a better picture. Regards, Mukkund For your reference, the Riemann Solver is a modified version of the HLL solver: A simple well-balanced and positive numerical scheme for the shallow-water system by Emmanuel Audusse, Christophe Chalons, Philippe Ung. (https://www.intlpress.com/site/pub/files/_fulltext/journals/cms/2015/0013/0005/CMS-2015-0013-0005-a011.pdf ) > On 17 Jun 2020, at 20:47, Mark Adams wrote: > > So you get this noise with a regular grid in p4est. So the same grid as will Plex, and you are not getting the same results. > > I don't know of any difference from p4est on a non-adapted grid. Can you reproduce this with ex11? > > Matt and Toby could answer this better. > > > On Wed, Jun 17, 2020 at 1:33 PM MUKKUND SUNJII wrote: > Greetings, > > I am a master?s student working on the shallow water model of the TS example 'ex11.c' as part of my thesis. Therefore, I am working with DMForest for the implementation of adaptive grids. I have a question and an observation. > > I am trying to find relevant information about interpolation that takes place through the routine DMForestTransferVec. Perhaps it could be my inability to find it, but I am unable to locate the implementation of the routine > > (forest->transfervec)(dmIn,vecIn,dmOut,vecOut,useBCs,time). > > Any information on this particular routine is highly appreciated. > > Furthermore, I have developed a well balanced Riemann Solver that includes topography in the model. In the process of testing both the non-adaptive and adaptive version, I found that my results differed when I changed the type of DM. For instance, when I run a scenario in a fixed, non-adaptive grid with a DM of type 'P4est', I find that the well balanced nature is lost due to small perturbations all across the domain. However, this does not occur when I use a DM of type ?plex?. Is there a radical change in the routines between the two DM?s? This is not as much of a question as it is an observation. > > Thank you for all of your suggestions! > > Regards, > > Mukkund > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Bathymetry.png Type: image/png Size: 79963 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Plex.png Type: image/png Size: 68539 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: p4est.png Type: image/png Size: 150800 bytes Desc: not available URL: From jacob.fai at gmail.com Wed Jun 17 15:03:59 2020 From: jacob.fai at gmail.com (Jacob Faibussowitsch) Date: Wed, 17 Jun 2020 15:03:59 -0500 Subject: [petsc-users] Cohesive Element Support Message-ID: <7C00A8A6-F7F2-4D62-852B-45ADCD317B98@gmail.com> Hello, I am looking to perform large scale fracture and crack propagation simulations and have a few questions regarding PETSc support for this. Specifically I am looking for cohesive surface element support with a few twists: 1. Is there support for zero thickness surface elements? For example modeling virtually flat patches of adhesives holding together two larger structures being pulled apart. 2. Is there support for ?joining? two possibly distinct meshes with cohesive surface elements? For example say I have two distinct cylinders representing fibers which would ?touch" to form an X shape. 3. In a similar vein, is there support for a mesh to fracture entirely along a crack formed through the cohesive elements? Imagine the aforementioned X configuration separating entirely into two separate cylinders again. 4. Is there a mechanism by which you can classify existing elements as cohesive elements? 5. Is there an already implemented way of imposing tie-constraints between independent meshes? This would potentially be used to tie high order cohesive cells which would have a non-conforming interface to the ?regular? mesh. From googling I have come across DMPlexCreateHybridMesh(), DMPlexConstructCohesiveCells(), and DMPlexCreateCohesiveSubmesh(). While these do implement cohesive cells these functions don?t at first glance seem to allow one to implement the above. Best regards, Jacob Faibussowitsch (Jacob Fai - booss - oh - vitch) Cell: (312) 694-3391 -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave.mayhem23 at gmail.com Wed Jun 17 15:53:52 2020 From: dave.mayhem23 at gmail.com (Dave May) Date: Wed, 17 Jun 2020 22:53:52 +0200 Subject: [petsc-users] Regarding P4est In-Reply-To: References: Message-ID: On Wed 17. Jun 2020 at 21:21, MUKKUND SUNJII wrote: > Yes, precisely! I am not sure how I can replicate using the original > version of ex11.c because it does not support bathymetry. > > Regardless, to demonstrate the discrepancy, I have uploaded three plots. > The scenario is a lake at rest. Essentially, you have a varying bathymetry > but a level water surface. If the model is well balanced, then the water > surface height must not change. The description of the files are below > > 1) Bathymetry.png : It shows you the bathymetry profile (z(x)) and the > water surface height (H = h+z(x)) at t = 0. > > 2) Plex.png : This is the water surface height after 1 time step (0.007055 > sec) and the dm type is Plex. As you can see, the water surface height > is undisturbed as expected. > > 3) P4est.png : This is the result after 1 time step (same final time) if I > set the dm type as p4est. The noise is in the order of 1e-3 to be a little > more specific. Since its not specifically at the boundaries and more or > less spread throughout, it could indeed be noise introduced. But of course > I could be wrong. > > The (wrong) result has seemingly a lot of structure. Have you verified your code using p4est is valgrind clean? This looks too much like a weird indexing bug for me to not ask this question. Thanks, Dave Maybe this paints a better picture. > > Regards, > > Mukkund > > For your reference, the Riemann Solver is a modified version of the HLL > solver: *A simple well-balanced and positive numerical scheme for the > shallow-water system by **Emmanuel Audusse, Christophe Chalons, Philippe > Ung. * > ( > https://www.intlpress.com/site/pub/files/_fulltext/journals/cms/2015/0013/0005/CMS-2015-0013-0005-a011.pdf > ) > > On 17 Jun 2020, at 20:47, Mark Adams wrote: > > So you get this noise with a regular grid in p4est. So the same grid as > will Plex, and you are not getting the same results. > > I don't know of any difference from p4est on a non-adapted grid. Can you > reproduce this with ex11? > > Matt and Toby could answer this better. > > > On Wed, Jun 17, 2020 at 1:33 PM MUKKUND SUNJII > wrote: > Greetings, > > I am a master?s student working on the shallow water model of the TS > example 'ex11.c' as part of my thesis. Therefore, I am working with > DMForest for the implementation of adaptive grids. I have a question and an > observation. > > I am trying to find relevant information about interpolation that takes > place through the routine DMForestTransferVec. Perhaps it could be my > inability to find it, but I am unable to locate the implementation of the > routine > > (forest->transfervec)(dmIn,vecIn,dmOut,vecOut,useBCs,time). > > Any information on this particular routine is highly appreciated. > > Furthermore, I have developed a well balanced Riemann Solver that includes > topography in the model. In the process of testing both the non-adaptive > and adaptive version, I found that my results differed when I changed the > type of DM. For instance, when I run a scenario in a fixed, non-adaptive > grid with a DM of type 'P4est', I find that the well balanced nature is > lost due to small perturbations all across the domain. However, this does > not occur when I use a DM of type ?plex?. Is there a radical change in the > routines between the two DM?s? This is not as much of a question as it is > an observation. > > Thank you for all of your suggestions! > > Regards, > > Mukkund > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Plex.png Type: image/png Size: 68539 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Bathymetry.png Type: image/png Size: 79963 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: p4est.png Type: image/png Size: 150800 bytes Desc: not available URL: From mukkundsunjii at gmail.com Wed Jun 17 17:25:45 2020 From: mukkundsunjii at gmail.com (MUKKUND SUNJII) Date: Thu, 18 Jun 2020 00:25:45 +0200 Subject: [petsc-users] Regarding P4est In-Reply-To: References: Message-ID: <6F1DD449-EA9C-4333-B5AD-FD2C73EF373C@gmail.com> I agree with the structured nature of the noise. I did play around with the PetscFV implementation a bit to allow for the computation of different fluxes left and right side of every interface. Nevertheless it is indeed strange that the problem disappears when I use a PLEX dm. Regards, Mukkund > On 17 Jun 2020, at 22:53, Dave May wrote: > > > > On Wed 17. Jun 2020 at 21:21, MUKKUND SUNJII > wrote: > Yes, precisely! I am not sure how I can replicate using the original version of ex11.c because it does not support bathymetry. > > Regardless, to demonstrate the discrepancy, I have uploaded three plots. The scenario is a lake at rest. Essentially, you have a varying bathymetry but a level water surface. If the model is well balanced, then the water surface height must not change. The description of the files are below > > 1) Bathymetry.png : It shows you the bathymetry profile (z(x)) and the water surface height (H = h+z(x)) at t = 0. > > > 2) Plex.png : This is the water surface height after 1 time step (0.007055 sec) and the dm type is Plex. As you can see, the water surface height is undisturbed as expected. > > > 3) P4est.png : This is the result after 1 time step (same final time) if I set the dm type as p4est. The noise is in the order of 1e-3 to be a little more specific. Since its not specifically at the boundaries and more or less spread throughout, it could indeed be noise introduced. But of course I could be wrong. > > > > The (wrong) result has seemingly a lot of structure. Have you verified your code using p4est is valgrind clean? This looks too much like a weird indexing bug for me to not ask this question. > > Thanks, > Dave > > > Maybe this paints a better picture. > > Regards, > > Mukkund > > For your reference, the Riemann Solver is a modified version of the HLL solver: A simple well-balanced and positive numerical scheme for the shallow-water system by Emmanuel Audusse, Christophe Chalons, Philippe Ung. > (https://www.intlpress.com/site/pub/files/_fulltext/journals/cms/2015/0013/0005/CMS-2015-0013-0005-a011.pdf ) > >> On 17 Jun 2020, at 20:47, Mark Adams > wrote: >> >> So you get this noise with a regular grid in p4est. So the same grid as will Plex, and you are not getting the same results. >> >> I don't know of any difference from p4est on a non-adapted grid. Can you reproduce this with ex11? >> >> Matt and Toby could answer this better. >> >> >> On Wed, Jun 17, 2020 at 1:33 PM MUKKUND SUNJII > wrote: >> Greetings, >> >> I am a master?s student working on the shallow water model of the TS example 'ex11.c' as part of my thesis. Therefore, I am working with DMForest for the implementation of adaptive grids. I have a question and an observation. >> >> I am trying to find relevant information about interpolation that takes place through the routine DMForestTransferVec. Perhaps it could be my inability to find it, but I am unable to locate the implementation of the routine >> >> (forest->transfervec)(dmIn,vecIn,dmOut,vecOut,useBCs,time). >> >> Any information on this particular routine is highly appreciated. >> >> Furthermore, I have developed a well balanced Riemann Solver that includes topography in the model. In the process of testing both the non-adaptive and adaptive version, I found that my results differed when I changed the type of DM. For instance, when I run a scenario in a fixed, non-adaptive grid with a DM of type 'P4est', I find that the well balanced nature is lost due to small perturbations all across the domain. However, this does not occur when I use a DM of type ?plex?. Is there a radical change in the routines between the two DM?s? This is not as much of a question as it is an observation. >> >> Thank you for all of your suggestions! >> >> Regards, >> >> Mukkund -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave.mayhem23 at gmail.com Wed Jun 17 17:43:39 2020 From: dave.mayhem23 at gmail.com (Dave May) Date: Wed, 17 Jun 2020 23:43:39 +0100 Subject: [petsc-users] Regarding P4est In-Reply-To: <6F1DD449-EA9C-4333-B5AD-FD2C73EF373C@gmail.com> References: <6F1DD449-EA9C-4333-B5AD-FD2C73EF373C@gmail.com> Message-ID: Is the code valgrind clean? On Wed, 17 Jun 2020 at 23:25, MUKKUND SUNJII wrote: > I agree with the structured nature of the noise. I did play around with > the PetscFV implementation a bit to allow for the computation of different > fluxes left and right side of every interface. > > Nevertheless it is indeed strange that the problem disappears when I use a > PLEX dm. > > Regards, > > Mukkund > > On 17 Jun 2020, at 22:53, Dave May wrote: > > > > On Wed 17. Jun 2020 at 21:21, MUKKUND SUNJII > wrote: > >> Yes, precisely! I am not sure how I can replicate using the original >> version of ex11.c because it does not support bathymetry. >> >> Regardless, to demonstrate the discrepancy, I have uploaded three plots. >> The scenario is a lake at rest. Essentially, you have a varying bathymetry >> but a level water surface. If the model is well balanced, then the water >> surface height must not change. The description of the files are below >> >> 1) Bathymetry.png : It shows you the bathymetry profile (z(x)) and the >> water surface height (H = h+z(x)) at t = 0. >> >> >> 2) Plex.png : This is the water surface height after 1 time step (0.007055 >> sec) and the dm type is Plex. As you can see, the water surface height >> is undisturbed as expected. >> >> >> 3) P4est.png : This is the result after 1 time step (same final time) if >> I set the dm type as p4est. The noise is in the order of 1e-3 to be a >> little more specific. Since its not specifically at the boundaries and more >> or less spread throughout, it could indeed be noise introduced. But of >> course I could be wrong. >> >> >> > The (wrong) result has seemingly a lot of structure. Have you verified > your code using p4est is valgrind clean? This looks too much like a weird > indexing bug for me to not ask this question. > > Thanks, > Dave > > > Maybe this paints a better picture. >> >> Regards, >> >> Mukkund >> >> For your reference, the Riemann Solver is a modified version of the HLL >> solver: *A simple well-balanced and positive numerical scheme for the >> shallow-water system by **Emmanuel Audusse, Christophe Chalons, Philippe >> Ung. * >> ( >> https://www.intlpress.com/site/pub/files/_fulltext/journals/cms/2015/0013/0005/CMS-2015-0013-0005-a011.pdf >> ) >> >> On 17 Jun 2020, at 20:47, Mark Adams wrote: >> >> So you get this noise with a regular grid in p4est. So the same grid as >> will Plex, and you are not getting the same results. >> >> I don't know of any difference from p4est on a non-adapted grid. Can you >> reproduce this with ex11? >> >> Matt and Toby could answer this better. >> >> >> On Wed, Jun 17, 2020 at 1:33 PM MUKKUND SUNJII >> wrote: >> Greetings, >> >> I am a master?s student working on the shallow water model of the TS >> example 'ex11.c' as part of my thesis. Therefore, I am working with >> DMForest for the implementation of adaptive grids. I have a question and an >> observation. >> >> I am trying to find relevant information about interpolation that takes >> place through the routine DMForestTransferVec. Perhaps it could be my >> inability to find it, but I am unable to locate the implementation of the >> routine >> >> (forest->transfervec)(dmIn,vecIn,dmOut,vecOut,useBCs,time). >> >> Any information on this particular routine is highly appreciated. >> >> Furthermore, I have developed a well balanced Riemann Solver that >> includes topography in the model. In the process of testing both the >> non-adaptive and adaptive version, I found that my results differed when I >> changed the type of DM. For instance, when I run a scenario in a fixed, >> non-adaptive grid with a DM of type 'P4est', I find that the well balanced >> nature is lost due to small perturbations all across the domain. However, >> this does not occur when I use a DM of type ?plex?. Is there a radical >> change in the routines between the two DM?s? This is not as much of a >> question as it is an observation. >> >> Thank you for all of your suggestions! >> >> Regards, >> >> Mukkund >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mukkundsunjii at gmail.com Wed Jun 17 17:47:37 2020 From: mukkundsunjii at gmail.com (MUKKUND SUNJII) Date: Thu, 18 Jun 2020 00:47:37 +0200 Subject: [petsc-users] Regarding P4est In-Reply-To: References: <6F1DD449-EA9C-4333-B5AD-FD2C73EF373C@gmail.com> Message-ID: <18AF3007-DE3E-46E7-BEC8-9C182D8A1A8D@gmail.com> No, I have not checked it using Valgrind. Perhaps it will help me trace the problem. Regards, Mukkund > On 18 Jun 2020, at 00:43, Dave May wrote: > > Is the code valgrind clean? > > On Wed, 17 Jun 2020 at 23:25, MUKKUND SUNJII > wrote: > I agree with the structured nature of the noise. I did play around with the PetscFV implementation a bit to allow for the computation of different fluxes left and right side of every interface. > > Nevertheless it is indeed strange that the problem disappears when I use a PLEX dm. > > Regards, > > Mukkund > >> On 17 Jun 2020, at 22:53, Dave May > wrote: >> >> >> >> On Wed 17. Jun 2020 at 21:21, MUKKUND SUNJII > wrote: >> Yes, precisely! I am not sure how I can replicate using the original version of ex11.c because it does not support bathymetry. >> >> Regardless, to demonstrate the discrepancy, I have uploaded three plots. The scenario is a lake at rest. Essentially, you have a varying bathymetry but a level water surface. If the model is well balanced, then the water surface height must not change. The description of the files are below >> >> 1) Bathymetry.png : It shows you the bathymetry profile (z(x)) and the water surface height (H = h+z(x)) at t = 0. >> >> >> 2) Plex.png : This is the water surface height after 1 time step (0.007055 sec) and the dm type is Plex. As you can see, the water surface height is undisturbed as expected. >> >> >> 3) P4est.png : This is the result after 1 time step (same final time) if I set the dm type as p4est. The noise is in the order of 1e-3 to be a little more specific. Since its not specifically at the boundaries and more or less spread throughout, it could indeed be noise introduced. But of course I could be wrong. >> >> >> >> The (wrong) result has seemingly a lot of structure. Have you verified your code using p4est is valgrind clean? This looks too much like a weird indexing bug for me to not ask this question. >> >> Thanks, >> Dave >> >> >> Maybe this paints a better picture. >> >> Regards, >> >> Mukkund >> >> For your reference, the Riemann Solver is a modified version of the HLL solver: A simple well-balanced and positive numerical scheme for the shallow-water system by Emmanuel Audusse, Christophe Chalons, Philippe Ung. >> (https://www.intlpress.com/site/pub/files/_fulltext/journals/cms/2015/0013/0005/CMS-2015-0013-0005-a011.pdf ) >> >>> On 17 Jun 2020, at 20:47, Mark Adams > wrote: >>> >>> So you get this noise with a regular grid in p4est. So the same grid as will Plex, and you are not getting the same results. >>> >>> I don't know of any difference from p4est on a non-adapted grid. Can you reproduce this with ex11? >>> >>> Matt and Toby could answer this better. >>> >>> >>> On Wed, Jun 17, 2020 at 1:33 PM MUKKUND SUNJII > wrote: >>> Greetings, >>> >>> I am a master?s student working on the shallow water model of the TS example 'ex11.c' as part of my thesis. Therefore, I am working with DMForest for the implementation of adaptive grids. I have a question and an observation. >>> >>> I am trying to find relevant information about interpolation that takes place through the routine DMForestTransferVec. Perhaps it could be my inability to find it, but I am unable to locate the implementation of the routine >>> >>> (forest->transfervec)(dmIn,vecIn,dmOut,vecOut,useBCs,time). >>> >>> Any information on this particular routine is highly appreciated. >>> >>> Furthermore, I have developed a well balanced Riemann Solver that includes topography in the model. In the process of testing both the non-adaptive and adaptive version, I found that my results differed when I changed the type of DM. For instance, when I run a scenario in a fixed, non-adaptive grid with a DM of type 'P4est', I find that the well balanced nature is lost due to small perturbations all across the domain. However, this does not occur when I use a DM of type ?plex?. Is there a radical change in the routines between the two DM?s? This is not as much of a question as it is an observation. >>> >>> Thank you for all of your suggestions! >>> >>> Regards, >>> >>> Mukkund > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Wed Jun 17 18:20:15 2020 From: mfadams at lbl.gov (Mark Adams) Date: Wed, 17 Jun 2020 19:20:15 -0400 Subject: [petsc-users] Regarding P4est In-Reply-To: <18AF3007-DE3E-46E7-BEC8-9C182D8A1A8D@gmail.com> References: <6F1DD449-EA9C-4333-B5AD-FD2C73EF373C@gmail.com> <18AF3007-DE3E-46E7-BEC8-9C182D8A1A8D@gmail.com> Message-ID: PETSc does take pains to keep it clean in Valgrind, to make it more useful ... And yes there is tree structure to this error, and p4est is a tree code. Try with uniform bathymetry, maybe your mapping is messed up by some recording by p4est. On Wed, Jun 17, 2020 at 6:47 PM MUKKUND SUNJII wrote: > No, I have not checked it using Valgrind. Perhaps it will help me trace > the problem. > > Regards, > > Mukkund > > On 18 Jun 2020, at 00:43, Dave May wrote: > > Is the code valgrind clean? > > On Wed, 17 Jun 2020 at 23:25, MUKKUND SUNJII > wrote: > >> I agree with the structured nature of the noise. I did play around with >> the PetscFV implementation a bit to allow for the computation of different >> fluxes left and right side of every interface. >> >> Nevertheless it is indeed strange that the problem disappears when I use >> a PLEX dm. >> >> Regards, >> >> Mukkund >> >> On 17 Jun 2020, at 22:53, Dave May wrote: >> >> >> >> On Wed 17. Jun 2020 at 21:21, MUKKUND SUNJII >> wrote: >> >>> Yes, precisely! I am not sure how I can replicate using the original >>> version of ex11.c because it does not support bathymetry. >>> >>> Regardless, to demonstrate the discrepancy, I have uploaded three plots. >>> The scenario is a lake at rest. Essentially, you have a varying bathymetry >>> but a level water surface. If the model is well balanced, then the water >>> surface height must not change. The description of the files are below >>> >>> 1) Bathymetry.png : It shows you the bathymetry profile (z(x)) and the >>> water surface height (H = h+z(x)) at t = 0. >>> >>> >>> 2) Plex.png : This is the water surface height after 1 time step (0.007055 >>> sec) and the dm type is Plex. As you can see, the water surface height >>> is undisturbed as expected. >>> >>> >>> 3) P4est.png : This is the result after 1 time step (same final time) if >>> I set the dm type as p4est. The noise is in the order of 1e-3 to be a >>> little more specific. Since its not specifically at the boundaries and more >>> or less spread throughout, it could indeed be noise introduced. But of >>> course I could be wrong. >>> >>> >>> >> The (wrong) result has seemingly a lot of structure. Have you verified >> your code using p4est is valgrind clean? This looks too much like a weird >> indexing bug for me to not ask this question. >> >> Thanks, >> Dave >> >> >> Maybe this paints a better picture. >>> >>> Regards, >>> >>> Mukkund >>> >>> For your reference, the Riemann Solver is a modified version of the HLL >>> solver: *A simple well-balanced and positive numerical scheme for the >>> shallow-water system by **Emmanuel Audusse, Christophe Chalons, >>> Philippe Ung. * >>> ( >>> https://www.intlpress.com/site/pub/files/_fulltext/journals/cms/2015/0013/0005/CMS-2015-0013-0005-a011.pdf >>> ) >>> >>> On 17 Jun 2020, at 20:47, Mark Adams wrote: >>> >>> So you get this noise with a regular grid in p4est. So the same grid as >>> will Plex, and you are not getting the same results. >>> >>> I don't know of any difference from p4est on a non-adapted grid. Can you >>> reproduce this with ex11? >>> >>> Matt and Toby could answer this better. >>> >>> >>> On Wed, Jun 17, 2020 at 1:33 PM MUKKUND SUNJII >>> wrote: >>> Greetings, >>> >>> I am a master?s student working on the shallow water model of the TS >>> example 'ex11.c' as part of my thesis. Therefore, I am working with >>> DMForest for the implementation of adaptive grids. I have a question and an >>> observation. >>> >>> I am trying to find relevant information about interpolation that takes >>> place through the routine DMForestTransferVec. Perhaps it could be my >>> inability to find it, but I am unable to locate the implementation of the >>> routine >>> >>> (forest->transfervec)(dmIn,vecIn,dmOut,vecOut,useBCs,time). >>> >>> Any information on this particular routine is highly appreciated. >>> >>> Furthermore, I have developed a well balanced Riemann Solver that >>> includes topography in the model. In the process of testing both the >>> non-adaptive and adaptive version, I found that my results differed when I >>> changed the type of DM. For instance, when I run a scenario in a fixed, >>> non-adaptive grid with a DM of type 'P4est', I find that the well balanced >>> nature is lost due to small perturbations all across the domain. However, >>> this does not occur when I use a DM of type ?plex?. Is there a radical >>> change in the routines between the two DM?s? This is not as much of a >>> question as it is an observation. >>> >>> Thank you for all of your suggestions! >>> >>> Regards, >>> >>> Mukkund >>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave.mayhem23 at gmail.com Thu Jun 18 01:19:56 2020 From: dave.mayhem23 at gmail.com (Dave May) Date: Thu, 18 Jun 2020 08:19:56 +0200 Subject: [petsc-users] Regarding P4est In-Reply-To: References: <6F1DD449-EA9C-4333-B5AD-FD2C73EF373C@gmail.com> <18AF3007-DE3E-46E7-BEC8-9C182D8A1A8D@gmail.com> Message-ID: On Thu 18. Jun 2020 at 01:20, Mark Adams wrote: > PETSc does take pains to keep it clean in Valgrind, to make it more > useful ... > Yes of course! As I understood, the code being discussed was derived / based on ex11, and not identical to ex11 (eg flux definitions have changed). Hence there?s some user code in the mix which is not guaranteed to be valgrind clean. > And yes there is tree structure to this error, and p4est is a tree code. > > Try with uniform bathymetry, maybe your mapping is messed up by some > recording by p4est. > > > On Wed, Jun 17, 2020 at 6:47 PM MUKKUND SUNJII > wrote: > >> No, I have not checked it using Valgrind. Perhaps it will help me trace >> the problem. >> >> Regards, >> >> Mukkund >> >> On 18 Jun 2020, at 00:43, Dave May wrote: >> >> Is the code valgrind clean? >> >> On Wed, 17 Jun 2020 at 23:25, MUKKUND SUNJII >> wrote: >> >>> I agree with the structured nature of the noise. I did play around with >>> the PetscFV implementation a bit to allow for the computation of different >>> fluxes left and right side of every interface. >>> >>> Nevertheless it is indeed strange that the problem disappears when I use >>> a PLEX dm. >>> >>> Regards, >>> >>> Mukkund >>> >>> On 17 Jun 2020, at 22:53, Dave May wrote: >>> >>> >>> >>> On Wed 17. Jun 2020 at 21:21, MUKKUND SUNJII >>> wrote: >>> >>>> Yes, precisely! I am not sure how I can replicate using the original >>>> version of ex11.c because it does not support bathymetry. >>>> >>>> Regardless, to demonstrate the discrepancy, I have uploaded three >>>> plots. The scenario is a lake at rest. Essentially, you have a varying >>>> bathymetry but a level water surface. If the model is well balanced, then >>>> the water surface height must not change. The description of the files are >>>> below >>>> >>>> 1) Bathymetry.png : It shows you the bathymetry profile (z(x)) and the >>>> water surface height (H = h+z(x)) at t = 0. >>>> >>>> >>>> 2) Plex.png : This is the water surface height after 1 time step (0.007055 >>>> sec) and the dm type is Plex. As you can see, the water surface >>>> height is undisturbed as expected. >>>> >>>> >>>> 3) P4est.png : This is the result after 1 time step (same final time) >>>> if I set the dm type as p4est. The noise is in the order of 1e-3 to be a >>>> little more specific. Since its not specifically at the boundaries and more >>>> or less spread throughout, it could indeed be noise introduced. But of >>>> course I could be wrong. >>>> >>>> >>>> >>> The (wrong) result has seemingly a lot of structure. Have you verified >>> your code using p4est is valgrind clean? This looks too much like a weird >>> indexing bug for me to not ask this question. >>> >>> Thanks, >>> Dave >>> >>> >>> Maybe this paints a better picture. >>>> >>>> Regards, >>>> >>>> Mukkund >>>> >>>> For your reference, the Riemann Solver is a modified version of the HLL >>>> solver: *A simple well-balanced and positive numerical scheme for the >>>> shallow-water system by **Emmanuel Audusse, Christophe Chalons, >>>> Philippe Ung. * >>>> ( >>>> https://www.intlpress.com/site/pub/files/_fulltext/journals/cms/2015/0013/0005/CMS-2015-0013-0005-a011.pdf >>>> ) >>>> >>>> On 17 Jun 2020, at 20:47, Mark Adams wrote: >>>> >>>> So you get this noise with a regular grid in p4est. So the same grid as >>>> will Plex, and you are not getting the same results. >>>> >>>> I don't know of any difference from p4est on a non-adapted grid. Can >>>> you reproduce this with ex11? >>>> >>>> Matt and Toby could answer this better. >>>> >>>> >>>> On Wed, Jun 17, 2020 at 1:33 PM MUKKUND SUNJII >>>> wrote: >>>> Greetings, >>>> >>>> I am a master?s student working on the shallow water model of the TS >>>> example 'ex11.c' as part of my thesis. Therefore, I am working with >>>> DMForest for the implementation of adaptive grids. I have a question and an >>>> observation. >>>> >>>> I am trying to find relevant information about interpolation that takes >>>> place through the routine DMForestTransferVec. Perhaps it could be my >>>> inability to find it, but I am unable to locate the implementation of the >>>> routine >>>> >>>> (forest->transfervec)(dmIn,vecIn,dmOut,vecOut,useBCs,time). >>>> >>>> Any information on this particular routine is highly appreciated. >>>> >>>> Furthermore, I have developed a well balanced Riemann Solver that >>>> includes topography in the model. In the process of testing both the >>>> non-adaptive and adaptive version, I found that my results differed when I >>>> changed the type of DM. For instance, when I run a scenario in a fixed, >>>> non-adaptive grid with a DM of type 'P4est', I find that the well balanced >>>> nature is lost due to small perturbations all across the domain. However, >>>> this does not occur when I use a DM of type ?plex?. Is there a radical >>>> change in the routines between the two DM?s? This is not as much of a >>>> question as it is an observation. >>>> >>>> Thank you for all of your suggestions! >>>> >>>> Regards, >>>> >>>> Mukkund >>>> >>>> >>> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Jun 18 05:28:49 2020 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 18 Jun 2020 06:28:49 -0400 Subject: [petsc-users] Cohesive Element Support In-Reply-To: <7C00A8A6-F7F2-4D62-852B-45ADCD317B98@gmail.com> References: <7C00A8A6-F7F2-4D62-852B-45ADCD317B98@gmail.com> Message-ID: On Wed, Jun 17, 2020 at 4:05 PM Jacob Faibussowitsch wrote: > Hello, > > I am looking to perform large scale fracture and crack propagation > simulations and have a few questions regarding PETSc support for this. > Specifically I am looking for cohesive surface element support with a few > twists: > > 1. Is there support for zero thickness surface elements? For example > modeling virtually flat patches of adhesives holding together two larger > structures being pulled apart. > This is how PyLith works: https://github.com/geodynamics/pylith > 2. Is there support for ?joining? two possibly distinct meshes with > cohesive surface elements? For example say I have two distinct cylinders > representing fibers which would ?touch" to form an X shape. > No, it would have to be coded. > 3. In a similar vein, is there support for a mesh to fracture entirely > along a crack formed through the cohesive elements? Imagine the > aforementioned X configuration separating entirely into two separate > cylinders again. > No, it would have to be coded. > 4. Is there a mechanism by which you can classify existing elements as > cohesive elements? > See 1. > 5. Is there an already implemented way of imposing tie-constraints between > independent meshes? This would potentially be used to tie high order > cohesive cells which would have a non-conforming interface to the ?regular? > mesh. > There is nothing for non-conforming interfaces. > From googling I have come across DMPlexCreateHybridMesh(), > DMPlexConstructCohesiveCells(), and DMPlexCreateCohesiveSubmesh(). While > these do implement cohesive cells these functions don?t at first glance > seem to allow one to implement the above. > Having worked with cohesive elements for more than a decade, I would be cautious about a new code using them for fracture. To me, it appears that variational fracture codes, like those from Blaise Bourdin and J. J. Marigo's group have much better geometric flexibility, and Maurini's work on the solver clears up the hardest part. Thanks, Matt > Best regards, > > Jacob Faibussowitsch > (Jacob Fai - booss - oh - vitch) > Cell: (312) 694-3391 > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From jorge.chiva.segura at gmail.com Thu Jun 18 06:26:28 2020 From: jorge.chiva.segura at gmail.com (Jorge Chiva Segura) Date: Thu, 18 Jun 2020 13:26:28 +0200 Subject: [petsc-users] Possible bug PETSc+Complex+CUDA In-Reply-To: References: Message-ID: Hello Junchao, I have seen the merge request: https://gitlab.com/petsc/petsc/-/merge_requests/2822 It is very detailed and interesting. Amazing job!!! :D I tried the first option that you mentioned in the merge request (clonning thrust, especifying '--with-clanguage=cxx' and adding "-I/path-to-thrust" to CFLAGS, CXXFLAGS and CUDAFLAGS) and it solved the problem. Then I tried with petsc master and '--with-thrust-dir=/path-to-folder-that-contains-a-folder-named-include-that-contains-thrust' and it also worked ^_^ Thank you very much for your help! Best regards, Jorge On Tue, Jun 16, 2020 at 8:48 PM Junchao Zhang wrote: > Hello, Jorge, > The problem is fixed in petsc master and you need to add > --download-thrust to your configure options to use this feature. > Thanks for reporting. > --Junchao Zhang > > > On Fri, May 29, 2020 at 7:52 AM Jorge Chiva Segura < > jorge.chiva.segura at gmail.com> wrote: > >> Dear all, >> >> I just wanted to add what we found in case that it can help to solve this >> problem: >> >> * Examples ex2.c, ex11.c and ex32.c under src/ksp/ksp/tutorials all of >> them seg-fault with "-mat_type mpiaijcusparse" or "-vec_type mpicuda" with >> CUDA 9.2 and 10.2. >> Examples ex2.c and ex32.c work fine if PETSc scalar type is set to >> real instead of complex. PETSc has been compiled with gcc-6.4.0 >> >> * Here: >> https://dynamite.readthedocs.io/en/latest/tips.html#gpu-support >> it is mentioned that there is some problem for CUDA versions 8 and >> above. >> >> * It seems that the same problem was mentioned in this thread: >> >> https://lists.mcs.anl.gov/mailman/htdig/petsc-users/2019-February/037748.html >> >> Please, let me know if you need any extra information. >> >> Thank you very much for your help. >> >> Best regards, >> >> Jorge >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From shaswat121994 at gmail.com Thu Jun 18 12:20:34 2020 From: shaswat121994 at gmail.com (Shashwat Tiwari) Date: Thu, 18 Jun 2020 22:50:34 +0530 Subject: [petsc-users] Implementing periodicity using DMPlex In-Reply-To: <4E40F31A-AD04-4F7C-AA82-B43C9DBEA074@gmail.com> References: <4E40F31A-AD04-4F7C-AA82-B43C9DBEA074@gmail.com> Message-ID: Thank you for the help. I followed your suggestion and removed the "DMPlexConstructGhostCells" function and now I am using a gmsh generated mesh with periodicity in both directions, and distributing the mesh with 1 level of overlap. I can see that the periodic faces now have two neighbouring cells on the opposite side of the domain which I can use to get the periodic cell values using "DMGlobalToLocal" without the need to construct ghost cells. There are two main issues that I am facing now, regarding geometric information needed for implementing periodic bc. One, I may need the centroid of periodic counterpart of a cell which is possible to get from a "ghost" cell created on the periodic face, and containing the solution data from periodic counterpart. But now, I would need to shift the coordinates of the neighbouring cell accordingly. If you see the sample output given below along with the screenshot of the sample 2D mesh that I have used (please find attached), while reconstructing the left and right solution values on face 32 for a second order scheme, I need the centroid of a ghost cell on left side of the face which is at (x, y) = (-1.25, 8.75). I wanted to know if there is some way to get these shifted "ghost" coordinates directly from the geometric information obtained using "PetscFV" class. Kindly let me know if you have some suggestions regarding this. Sample Output: face: 32, centroid = 0.000000, 8.750000, normal = 2.500000, -0.000000 support[0] = 0, cent = 1.250000, 8.750000, area = 6.250000 support[1] = 3, cent = 8.750000, 8.750000, area = 6.250000 face: 33, centroid = 1.250000, 7.500000, normal = -0.000000, -2.500000 support[0] = 0, cent = 1.250000, 8.750000, area = 6.250000 support[1] = 4, cent = 1.250000, 6.250000, area = 6.250000 Second, the face normals of the periodic faces obtained using the face geometry data from "DMPlexGetDataFVM" function seem to be in the wrong direction i.e., from the right cell "support[1]" to the left cell "support[0]", while the other faces have the opposite direction. For face: 32, left cell is 0 and the right cell is 3, but, the normal points the opposite way (cell 3 would be on the left side of cell 0 on periodic boundary), while, for face: 33 which is an interior face, the normal direction is from left to right cell. Also, I am getting incorrect normal values for some of the faces (including some interior faces). Here is a sample output of the same: face: 59, centroid = 3.750000, 2.500000, normal = 0.000000, -7.500000 support[0] = 11, cent = 8.750000, 3.750000, area = 6.250000 support[1] = 15, cent = 8.750000, 1.250000, area = 6.250000 this face has the correct direction (from 11 to 15), but the value is wrong (all the faces are 2.5 units). This problem does not occur when I ignore periodicity by setting -dm_plex_gmsh_periodic to 0. These issues occur regardless of the number of processes. I am attaching the test code that I am using along with the gmsh file for your reference. Please suggest to me, what might be the issue here. Regards, Shashwat On Mon, Jun 15, 2020 at 2:54 PM Stefano Zampini wrote: > It is enough if you use DMPlexDistribute with 1 level of overlap, set the > local section for your dofs, and call DMGlobalToLocal . The local vector > will contain data for what you call ?ghost? cells. I call them ?not-owned? > cells. > > On Jun 15, 2020, at 12:09 PM, Shashwat Tiwari > wrote: > > The way I'm trying to implement periodic bc is, when I loop over the > boundary faces, say, on the left boundary of the domain to compute flux and > residual, I need solution values from the two cells neighbouring the face, > i.e. the left cell and the right cell (the face normal pointing from the > left cell to the right cell and the right cell being a boundary ghost cell > for boundary faces). For the boundary to be periodic, I need the value that > I get from the right cell (boundary ghost cell) to be the solution value at > its periodic counterpart, i.e. the solution value at left cell of the face > on right boundary of the domain in this case. My question is how do I > update the value at a boundary ghost cell with the value at the real cell > which is its periodic counterpart from the other side of the domain. Is > there some kind of mapping of the boundary ghost cells to their > corresponding real cells which I can use to update the solution values at > ghost cells? > > Regards, > Shashwat > > On Sun, Jun 14, 2020 at 5:11 AM Matthew Knepley wrote: > >> On Fri, Jun 12, 2020 at 3:19 PM Shashwat Tiwari >> wrote: >> >>> Hi, >>> I am writing a first order 2D solver for unstructured grids with >>> periodic boundaries using DMPlex. After generating the mesh, I use >>> "DMSetPeriodicity" function to set periodicity in both directions. After >>> which I partition the mesh (DMPlexDistribute), construct ghost cells >>> (DMPlexConstructGhostCells), >>> >> >> These ghost cells are for FVM boundary conditions. If you want cells to >> be shared across parallel partitions, then you want to give overlap=1 >> to DMPlexDIstribute(). Is that what you want? >> >> Thanks, >> >> Matt >> >> >>> create a section, and set some initial values in the global vector. Then >>> I use "VecGhostUpdateBegin" to start updating the boundary ghost cell >>> values, but, I get the following error in case I use multiple processors: >>> >>> [0]PETSC ERROR: Invalid argument >>> [0]PETSC ERROR: Vector is not ghosted >>> [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html >>> for trouble shooting. >>> >>> if I run with a single process, there is no error but the values remain >>> empty (zero) and are not updated. Kindly let me know, if I am missing some >>> crucial step before I can update the ghost values in order to implement the >>> periodic bc, or if there is any other approach to achieve it. I am >>> attaching a small code to demonstrate the issue for your reference. >>> >>> Regards, >>> Shashwat >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: mesh.png Type: image/png Size: 18621 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: test.c Type: text/x-csrc Size: 5358 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ug_periodic.geo Type: application/octet-stream Size: 518 bytes Desc: not available URL: From mukkundsunjii at gmail.com Thu Jun 18 13:13:23 2020 From: mukkundsunjii at gmail.com (MUKKUND SUNJII) Date: Thu, 18 Jun 2020 20:13:23 +0200 Subject: [petsc-users] Regarding P4est In-Reply-To: References: <6F1DD449-EA9C-4333-B5AD-FD2C73EF373C@gmail.com> <18AF3007-DE3E-46E7-BEC8-9C182D8A1A8D@gmail.com> Message-ID: I agree with your points. It could be the case there might be some mapping error. Although I am not quite sure how to narrow down the problem. In any case, thank you for your valuable suggestions. >>>> I am trying to find relevant information about interpolation that takes place through the routine DMForestTransferVec. Perhaps it could be my inability to find it, but I am unable to locate the implementation of the routine >>>> >>>> (forest->transfervec)(dmIn,vecIn,dmOut,vecOut,useBCs,time). Unfortunately, my first question still remains a mystery. Hopefully, someone could take a few moments to point me in the right direction. Regards, Mukkund > On 18 Jun 2020, at 08:19, Dave May wrote: > > > > On Thu 18. Jun 2020 at 01:20, Mark Adams > wrote: > PETSc does take pains to keep it clean in Valgrind, to make it more useful ... > > Yes of course! > > As I understood, the code being discussed was derived / based on ex11, and not identical to ex11 (eg flux definitions have changed). Hence there?s some user code in the mix which is not guaranteed to be valgrind clean. > > > > > > And yes there is tree structure to this error, and p4est is a tree code. > > Try with uniform bathymetry, maybe your mapping is messed up by some recording by p4est. > > > On Wed, Jun 17, 2020 at 6:47 PM MUKKUND SUNJII > wrote: > No, I have not checked it using Valgrind. Perhaps it will help me trace the problem. > > Regards, > > Mukkund > >> On 18 Jun 2020, at 00:43, Dave May > wrote: >> >> Is the code valgrind clean? >> >> On Wed, 17 Jun 2020 at 23:25, MUKKUND SUNJII > wrote: >> I agree with the structured nature of the noise. I did play around with the PetscFV implementation a bit to allow for the computation of different fluxes left and right side of every interface. >> >> Nevertheless it is indeed strange that the problem disappears when I use a PLEX dm. >> >> Regards, >> >> Mukkund >> >>> On 17 Jun 2020, at 22:53, Dave May > wrote: >>> >>> >>> >>> On Wed 17. Jun 2020 at 21:21, MUKKUND SUNJII > wrote: >>> Yes, precisely! I am not sure how I can replicate using the original version of ex11.c because it does not support bathymetry. >>> >>> Regardless, to demonstrate the discrepancy, I have uploaded three plots. The scenario is a lake at rest. Essentially, you have a varying bathymetry but a level water surface. If the model is well balanced, then the water surface height must not change. The description of the files are below >>> >>> 1) Bathymetry.png : It shows you the bathymetry profile (z(x)) and the water surface height (H = h+z(x)) at t = 0. >>> >>> >>> 2) Plex.png : This is the water surface height after 1 time step (0.007055 sec) and the dm type is Plex. As you can see, the water surface height is undisturbed as expected. >>> >>> >>> 3) P4est.png : This is the result after 1 time step (same final time) if I set the dm type as p4est. The noise is in the order of 1e-3 to be a little more specific. Since its not specifically at the boundaries and more or less spread throughout, it could indeed be noise introduced. But of course I could be wrong. >>> >>> >>> >>> The (wrong) result has seemingly a lot of structure. Have you verified your code using p4est is valgrind clean? This looks too much like a weird indexing bug for me to not ask this question. >>> >>> Thanks, >>> Dave >>> >>> >>> Maybe this paints a better picture. >>> >>> Regards, >>> >>> Mukkund >>> >>> For your reference, the Riemann Solver is a modified version of the HLL solver: A simple well-balanced and positive numerical scheme for the shallow-water system by Emmanuel Audusse, Christophe Chalons, Philippe Ung. >>> (https://www.intlpress.com/site/pub/files/_fulltext/journals/cms/2015/0013/0005/CMS-2015-0013-0005-a011.pdf ) >>> >>>> On 17 Jun 2020, at 20:47, Mark Adams > wrote: >>>> >>>> So you get this noise with a regular grid in p4est. So the same grid as will Plex, and you are not getting the same results. >>>> >>>> I don't know of any difference from p4est on a non-adapted grid. Can you reproduce this with ex11? >>>> >>>> Matt and Toby could answer this better. >>>> >>>> >>>> On Wed, Jun 17, 2020 at 1:33 PM MUKKUND SUNJII > wrote: >>>> Greetings, >>>> >>>> I am a master?s student working on the shallow water model of the TS example 'ex11.c' as part of my thesis. Therefore, I am working with DMForest for the implementation of adaptive grids. I have a question and an observation. >>>> >>>> I am trying to find relevant information about interpolation that takes place through the routine DMForestTransferVec. Perhaps it could be my inability to find it, but I am unable to locate the implementation of the routine >>>> >>>> (forest->transfervec)(dmIn,vecIn,dmOut,vecOut,useBCs,time). >>>> >>>> Any information on this particular routine is highly appreciated. >>>> >>>> Furthermore, I have developed a well balanced Riemann Solver that includes topography in the model. In the process of testing both the non-adaptive and adaptive version, I found that my results differed when I changed the type of DM. For instance, when I run a scenario in a fixed, non-adaptive grid with a DM of type 'P4est', I find that the well balanced nature is lost due to small perturbations all across the domain. However, this does not occur when I use a DM of type ?plex?. Is there a radical change in the routines between the two DM?s? This is not as much of a question as it is an observation. >>>> >>>> Thank you for all of your suggestions! >>>> >>>> Regards, >>>> >>>> Mukkund >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Thu Jun 18 13:20:12 2020 From: balay at mcs.anl.gov (Satish Balay) Date: Thu, 18 Jun 2020 13:20:12 -0500 (CDT) Subject: [petsc-users] Regarding P4est In-Reply-To: References: <6F1DD449-EA9C-4333-B5AD-FD2C73EF373C@gmail.com> <18AF3007-DE3E-46E7-BEC8-9C182D8A1A8D@gmail.com> Message-ID: On Thu, 18 Jun 2020, MUKKUND SUNJII wrote: > > >>>> I am trying to find relevant information about interpolation that takes place through the routine DMForestTransferVec. Perhaps it could be my inability to find it, but I am unable to locate the implementation of the routine > >>>> > >>>> (forest->transfervec)(dmIn,vecIn,dmOut,vecOut,useBCs,time). > > Unfortunately, my first question still remains a mystery. Hopefully, someone could take a few moments to point me in the right direction. $ git grep transfervec include/petsc/private/dmforestimpl.h: PetscErrorCode (*transfervec)(DM,Vec,DM,Vec,PetscBool,PetscReal); include/petsc/private/dmforestimpl.h: PetscErrorCode (*transfervecfrombase)(DM,Vec,Vec); src/dm/impls/forest/forest.c: if (!forest->transfervec) SETERRQ(PetscObjectComm((PetscObject)dmIn),PETSC_ERR_SUP,"DMForestTransferVec() not implemented"); src/dm/impls/forest/forest.c: ierr = (forest->transfervec)(dmIn,vecIn,dmOut,vecOut,useBCs,time);CHKERRQ(ierr); src/dm/impls/forest/forest.c: if (!forest->transfervecfrombase) SETERRQ(PetscObjectComm((PetscObject)dm),PETSC_ERR_SUP,"DMForestTransferVecFromBase() not implemented"); src/dm/impls/forest/forest.c: ierr = (forest->transfervecfrombase)(dm,vecIn,vecOut);CHKERRQ(ierr); src/dm/impls/forest/p4est/pforest.c: forest->transfervec = DMForestTransferVec_pforest; src/dm/impls/forest/p4est/pforest.c: forest->transfervecfrombase = DMForestTransferVecFromBase_pforest; Likely DMForestTransferVec_pforest Its best to run in debugger - and verify which routine is getting called here... Satish From bsmith at petsc.dev Thu Jun 18 14:21:12 2020 From: bsmith at petsc.dev (Barry Smith) Date: Thu, 18 Jun 2020 14:21:12 -0500 Subject: [petsc-users] Regarding P4est In-Reply-To: References: <6F1DD449-EA9C-4333-B5AD-FD2C73EF373C@gmail.com> <18AF3007-DE3E-46E7-BEC8-9C182D8A1A8D@gmail.com> Message-ID: <9592AAFD-8BA6-4FF4-AD5F-80E1EF337566@petsc.dev> In PETSc the implementation names are always XXX_classname so you can use whatever tools your development environment has for finding all the implementations. For example, git grep or Emacs etags, etc Barry > On Jun 18, 2020, at 1:20 PM, Satish Balay via petsc-users wrote: > > On Thu, 18 Jun 2020, MUKKUND SUNJII wrote: > >> >>>>>> I am trying to find relevant information about interpolation that takes place through the routine DMForestTransferVec. Perhaps it could be my inability to find it, but I am unable to locate the implementation of the routine >>>>>> >>>>>> (forest->transfervec)(dmIn,vecIn,dmOut,vecOut,useBCs,time). >> >> Unfortunately, my first question still remains a mystery. Hopefully, someone could take a few moments to point me in the right direction. > > $ git grep transfervec > include/petsc/private/dmforestimpl.h: PetscErrorCode (*transfervec)(DM,Vec,DM,Vec,PetscBool,PetscReal); > include/petsc/private/dmforestimpl.h: PetscErrorCode (*transfervecfrombase)(DM,Vec,Vec); > src/dm/impls/forest/forest.c: if (!forest->transfervec) SETERRQ(PetscObjectComm((PetscObject)dmIn),PETSC_ERR_SUP,"DMForestTransferVec() not implemented"); > src/dm/impls/forest/forest.c: ierr = (forest->transfervec)(dmIn,vecIn,dmOut,vecOut,useBCs,time);CHKERRQ(ierr); > src/dm/impls/forest/forest.c: if (!forest->transfervecfrombase) SETERRQ(PetscObjectComm((PetscObject)dm),PETSC_ERR_SUP,"DMForestTransferVecFromBase() not implemented"); > src/dm/impls/forest/forest.c: ierr = (forest->transfervecfrombase)(dm,vecIn,vecOut);CHKERRQ(ierr); > src/dm/impls/forest/p4est/pforest.c: forest->transfervec = DMForestTransferVec_pforest; > src/dm/impls/forest/p4est/pforest.c: forest->transfervecfrombase = DMForestTransferVecFromBase_pforest; > > Likely DMForestTransferVec_pforest > > Its best to run in debugger - and verify which routine is getting called here... > > Satish From rhalder at umich.edu Thu Jun 18 15:14:04 2020 From: rhalder at umich.edu (Rakesh Halder) Date: Thu, 18 Jun 2020 16:14:04 -0400 Subject: [petsc-users] Using MatCreateSubMatrix Message-ID: Hi all, I'm writing an algorithm using PETSc that requires part of an (N x k) matrix, where k << N. Specifically, my algorithm loops from j = 1 to k, and uses j columns of the matrix of a time. I've been creating a new matrix for each iteration, and use MatGetColumnVector to get the values for each column vector, scatter them into a new vector, and then use MatSetValue to populate my matrix. MatGetColumnVector is very slow, and I was thinking of using MatCreateSubMatrix. However, I'm not entirely sure how to get the correct IS that would allow me to create a submatrix using the first j columns of the matrix at a time. I'm running in parallel, and the matrix is an MPIAIJ matrix. I've looked at examples of MatCreateSubMatrix as well as examples including the use of IS, but I'm still not very sure how to use them. Thanks, Rakesh -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Jun 18 15:22:27 2020 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 18 Jun 2020 16:22:27 -0400 Subject: [petsc-users] Regarding P4est In-Reply-To: References: Message-ID: On Wed, Jun 17, 2020 at 3:21 PM MUKKUND SUNJII wrote: > Yes, precisely! I am not sure how I can replicate using the original > version of ex11.c because it does not support bathymetry. > > Regardless, to demonstrate the discrepancy, I have uploaded three plots. > The scenario is a lake at rest. Essentially, you have a varying bathymetry > but a level water surface. If the model is well balanced, then the water > surface height must not change. The description of the files are below > > 1) Bathymetry.png : It shows you the bathymetry profile (z(x)) and the > water surface height (H = h+z(x)) at t = 0. > Hmm, that does look like a bug. The purpose of TransferVec is to preserve moments when transferring from one grid to another. Here it looks like it is artificially diffusive, but I do not see a reason for this pattern if you did the refinement initially, and not with p4est, since then the base grid is the same as the Plex. Is that what your code does? Thanks, Matt > > 2) Plex.png : This is the water surface height after 1 time step (0.007055 > sec) and the dm type is Plex. As you can see, the water surface height > is undisturbed as expected. > > 3) P4est.png : This is the result after 1 time step (same final time) if I > set the dm type as p4est. The noise is in the order of 1e-3 to be a little > more specific. Since its not specifically at the boundaries and more or > less spread throughout, it could indeed be noise introduced. But of course > I could be wrong. > > Maybe this paints a better picture. > > Regards, > > Mukkund > > For your reference, the Riemann Solver is a modified version of the HLL > solver: *A simple well-balanced and positive numerical scheme for the > shallow-water system by **Emmanuel Audusse, Christophe Chalons, Philippe > Ung. * > ( > https://www.intlpress.com/site/pub/files/_fulltext/journals/cms/2015/0013/0005/CMS-2015-0013-0005-a011.pdf > ) > > On 17 Jun 2020, at 20:47, Mark Adams wrote: > > So you get this noise with a regular grid in p4est. So the same grid as > will Plex, and you are not getting the same results. > > I don't know of any difference from p4est on a non-adapted grid. Can you > reproduce this with ex11? > > Matt and Toby could answer this better. > > > On Wed, Jun 17, 2020 at 1:33 PM MUKKUND SUNJII > wrote: > Greetings, > > I am a master?s student working on the shallow water model of the TS > example 'ex11.c' as part of my thesis. Therefore, I am working with > DMForest for the implementation of adaptive grids. I have a question and an > observation. > > I am trying to find relevant information about interpolation that takes > place through the routine DMForestTransferVec. Perhaps it could be my > inability to find it, but I am unable to locate the implementation of the > routine > > (forest->transfervec)(dmIn,vecIn,dmOut,vecOut,useBCs,time). > > Any information on this particular routine is highly appreciated. > > Furthermore, I have developed a well balanced Riemann Solver that includes > topography in the model. In the process of testing both the non-adaptive > and adaptive version, I found that my results differed when I changed the > type of DM. For instance, when I run a scenario in a fixed, non-adaptive > grid with a DM of type 'P4est', I find that the well balanced nature is > lost due to small perturbations all across the domain. However, this does > not occur when I use a DM of type ?plex?. Is there a radical change in the > routines between the two DM?s? This is not as much of a question as it is > an observation. > > Thank you for all of your suggestions! > > Regards, > > Mukkund > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Bathymetry.png Type: image/png Size: 79963 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Plex.png Type: image/png Size: 68539 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: p4est.png Type: image/png Size: 150800 bytes Desc: not available URL: From knepley at gmail.com Thu Jun 18 15:24:28 2020 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 18 Jun 2020 16:24:28 -0400 Subject: [petsc-users] Using MatCreateSubMatrix In-Reply-To: References: Message-ID: On Thu, Jun 18, 2020 at 4:22 PM Rakesh Halder wrote: > Hi all, > > I'm writing an algorithm using PETSc that requires part of an (N x k) > matrix, where k << N. Specifically, my algorithm loops from j = 1 to k, and > uses j columns of the matrix of a time. I've been creating a new matrix for > each iteration, and use MatGetColumnVector to get the values for each > column vector, scatter them into a new vector, and then use MatSetValue to > populate my matrix. MatGetColumnVector is very slow, and I was thinking of > using MatCreateSubMatrix. However, I'm not entirely sure how to get the > correct IS that would allow me to create a submatrix using the first j > columns of the matrix at a time. I'm running in parallel, and the matrix is > an MPIAIJ matrix. I've looked at examples of MatCreateSubMatrix as well as > examples including the use of IS, but I'm still not very sure how to use > them. > Use a ISStride of [0, N) for an MxN matrix for the column IS and you will get everything in the row, which is what you want. Thanks, Matt > Thanks, > > Rakesh > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefano.zampini at gmail.com Thu Jun 18 15:40:08 2020 From: stefano.zampini at gmail.com (Stefano Zampini) Date: Thu, 18 Jun 2020 23:40:08 +0300 Subject: [petsc-users] Regarding P4est In-Reply-To: References: Message-ID: Also, when using p4est, the orientation of the cells are different than the one generated when refining with DMPlex alone. I would double-check the code if you are not making any assumption ons it Il giorno gio 18 giu 2020 alle ore 23:24 Matthew Knepley ha scritto: > On Wed, Jun 17, 2020 at 3:21 PM MUKKUND SUNJII > wrote: > >> Yes, precisely! I am not sure how I can replicate using the original >> version of ex11.c because it does not support bathymetry. >> >> Regardless, to demonstrate the discrepancy, I have uploaded three plots. >> The scenario is a lake at rest. Essentially, you have a varying bathymetry >> but a level water surface. If the model is well balanced, then the water >> surface height must not change. The description of the files are below >> >> 1) Bathymetry.png : It shows you the bathymetry profile (z(x)) and the >> water surface height (H = h+z(x)) at t = 0. >> > > Hmm, that does look like a bug. The purpose of TransferVec is to preserve > moments when transferring from one grid to another. Here it looks like it > is artificially diffusive, but I do not > see a reason for this pattern if you did the refinement initially, and not > with p4est, since then the base grid is the same as the Plex. Is that > what your code does? > > Thanks, > > Matt > > >> >> 2) Plex.png : This is the water surface height after 1 time step (0.007055 >> sec) and the dm type is Plex. As you can see, the water surface height >> is undisturbed as expected. >> >> 3) P4est.png : This is the result after 1 time step (same final time) if >> I set the dm type as p4est. The noise is in the order of 1e-3 to be a >> little more specific. Since its not specifically at the boundaries and more >> or less spread throughout, it could indeed be noise introduced. But of >> course I could be wrong. >> >> Maybe this paints a better picture. >> >> Regards, >> >> Mukkund >> >> For your reference, the Riemann Solver is a modified version of the HLL >> solver: *A simple well-balanced and positive numerical scheme for the >> shallow-water system by **Emmanuel Audusse, Christophe Chalons, Philippe >> Ung. * >> ( >> https://www.intlpress.com/site/pub/files/_fulltext/journals/cms/2015/0013/0005/CMS-2015-0013-0005-a011.pdf >> ) >> >> On 17 Jun 2020, at 20:47, Mark Adams wrote: >> >> So you get this noise with a regular grid in p4est. So the same grid as >> will Plex, and you are not getting the same results. >> >> I don't know of any difference from p4est on a non-adapted grid. Can you >> reproduce this with ex11? >> >> Matt and Toby could answer this better. >> >> >> On Wed, Jun 17, 2020 at 1:33 PM MUKKUND SUNJII >> wrote: >> Greetings, >> >> I am a master?s student working on the shallow water model of the TS >> example 'ex11.c' as part of my thesis. Therefore, I am working with >> DMForest for the implementation of adaptive grids. I have a question and an >> observation. >> >> I am trying to find relevant information about interpolation that takes >> place through the routine DMForestTransferVec. Perhaps it could be my >> inability to find it, but I am unable to locate the implementation of the >> routine >> >> (forest->transfervec)(dmIn,vecIn,dmOut,vecOut,useBCs,time). >> >> Any information on this particular routine is highly appreciated. >> >> Furthermore, I have developed a well balanced Riemann Solver that >> includes topography in the model. In the process of testing both the >> non-adaptive and adaptive version, I found that my results differed when I >> changed the type of DM. For instance, when I run a scenario in a fixed, >> non-adaptive grid with a DM of type 'P4est', I find that the well balanced >> nature is lost due to small perturbations all across the domain. However, >> this does not occur when I use a DM of type ?plex?. Is there a radical >> change in the routines between the two DM?s? This is not as much of a >> question as it is an observation. >> >> Thank you for all of your suggestions! >> >> Regards, >> >> Mukkund >> >> >> >> > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -- Stefano -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Bathymetry.png Type: image/png Size: 79963 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Plex.png Type: image/png Size: 68539 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: p4est.png Type: image/png Size: 150800 bytes Desc: not available URL: From mukkundsunjii at gmail.com Thu Jun 18 15:49:33 2020 From: mukkundsunjii at gmail.com (MUKKUND SUNJII) Date: Thu, 18 Jun 2020 22:49:33 +0200 Subject: [petsc-users] Regarding P4est In-Reply-To: References: Message-ID: <1A1B2B5B-FE8A-4BE4-9294-0380A6C3C70D@gmail.com> Greeting, >> transferring from one grid to another. Here it looks like it is artificially diffusive, but I do not >> see a reason for this pattern if you did the refinement initially, and not with p4est, since then the base grid is the same as the Plex. Is that what your code does? I am not sure how to differentiate the refining method. For the results that does not have this pattern, I provide the following command line options: -dm_type plex -grid_size 32,32. For the one that shows this odd pattern, the command line options I provide are: -dm_type p4est -dm_refine 5. > Also, when using p4est, the orientation of the cells are different than the one generated when refining with DMPlex alone. I would double-check the code if you are not making any assumption ons it I suppose this could indeed be the case as I do fiddle around with the Riemann solver interface in petscfv.c Regards, Mukkund Sunjii Sent on the go. > On 18 Jun 2020, at 10:40 PM, Stefano Zampini wrote: > > ? > Also, when using p4est, the orientation of the cells are different than the one generated when refining with DMPlex alone. I would double-check the code if you are not making any assumption ons it > >> Il giorno gio 18 giu 2020 alle ore 23:24 Matthew Knepley ha scritto: >>> On Wed, Jun 17, 2020 at 3:21 PM MUKKUND SUNJII wrote: >> >>> Yes, precisely! I am not sure how I can replicate using the original version of ex11.c because it does not support bathymetry. >>> >>> Regardless, to demonstrate the discrepancy, I have uploaded three plots. The scenario is a lake at rest. Essentially, you have a varying bathymetry but a level water surface. If the model is well balanced, then the water surface height must not change. The description of the files are below >>> >>> 1) Bathymetry.png : It shows you the bathymetry profile (z(x)) and the water surface height (H = h+z(x)) at t = 0. >> >> Hmm, that does look like a bug. The purpose of TransferVec is to preserve moments when transferring from one grid to another. Here it looks like it is artificially diffusive, but I do not >> see a reason for this pattern if you did the refinement initially, and not with p4est, since then the base grid is the same as the Plex. Is that what your code does? >> >> Thanks, >> >> Matt >> >>> >>> >>> 2) Plex.png : This is the water surface height after 1 time step (0.007055 sec) and the dm type is Plex. As you can see, the water surface height is undisturbed as expected. >>> >>> >>> 3) P4est.png : This is the result after 1 time step (same final time) if I set the dm type as p4est. The noise is in the order of 1e-3 to be a little more specific. Since its not specifically at the boundaries and more or less spread throughout, it could indeed be noise introduced. But of course I could be wrong. >>> >>> >>> Maybe this paints a better picture. >>> >>> Regards, >>> >>> Mukkund >>> >>> For your reference, the Riemann Solver is a modified version of the HLL solver: A simple well-balanced and positive numerical scheme for the shallow-water system by Emmanuel Audusse, Christophe Chalons, Philippe Ung. >>> (https://www.intlpress.com/site/pub/files/_fulltext/journals/cms/2015/0013/0005/CMS-2015-0013-0005-a011.pdf) >>> >>>> On 17 Jun 2020, at 20:47, Mark Adams wrote: >>>> >>>> So you get this noise with a regular grid in p4est. So the same grid as will Plex, and you are not getting the same results. >>>> >>>> I don't know of any difference from p4est on a non-adapted grid. Can you reproduce this with ex11? >>>> >>>> Matt and Toby could answer this better. >>>> >>>> >>>> On Wed, Jun 17, 2020 at 1:33 PM MUKKUND SUNJII wrote: >>>> Greetings, >>>> >>>> I am a master?s student working on the shallow water model of the TS example 'ex11.c' as part of my thesis. Therefore, I am working with DMForest for the implementation of adaptive grids. I have a question and an observation. >>>> >>>> I am trying to find relevant information about interpolation that takes place through the routine DMForestTransferVec. Perhaps it could be my inability to find it, but I am unable to locate the implementation of the routine >>>> >>>> (forest->transfervec)(dmIn,vecIn,dmOut,vecOut,useBCs,time). >>>> >>>> Any information on this particular routine is highly appreciated. >>>> >>>> Furthermore, I have developed a well balanced Riemann Solver that includes topography in the model. In the process of testing both the non-adaptive and adaptive version, I found that my results differed when I changed the type of DM. For instance, when I run a scenario in a fixed, non-adaptive grid with a DM of type 'P4est', I find that the well balanced nature is lost due to small perturbations all across the domain. However, this does not occur when I use a DM of type ?plex?. Is there a radical change in the routines between the two DM?s? This is not as much of a question as it is an observation. >>>> >>>> Thank you for all of your suggestions! >>>> >>>> Regards, >>>> >>>> Mukkund >>>> >>>> >>> >> >> >> -- >> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ > > > -- > Stefano -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Thu Jun 18 15:57:25 2020 From: mfadams at lbl.gov (Mark Adams) Date: Thu, 18 Jun 2020 16:57:25 -0400 Subject: [petsc-users] Problem with GPU configure at NERSC Message-ID: I am trying to configure the GPU nodes at NERSC and the instructions are pretty slim. Any idea what is wrong here? Thanks, Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: configure.log Type: application/octet-stream Size: 90595 bytes Desc: not available URL: From jed at jedbrown.org Thu Jun 18 16:11:14 2020 From: jed at jedbrown.org (Jed Brown) Date: Thu, 18 Jun 2020 15:11:14 -0600 Subject: [petsc-users] Finding mincut after k-means clustering In-Reply-To: References: Message-ID: <87zh90w03x.fsf@jedbrown.org> Are you just looking for metrics related to the partition you've created, or are you wanting to redistribute some other data according to the partition? Eda Oktay writes: > Hello everyone, > > I am solving a graph partitioning problem. I found an unnormalized > Laplacian matrix of a graph, then since I have 4 processes and I am > using spectral partitioning algorithm, I calculated 4 eigenvectors > corresponding to 4 smallest eigenvalues of this Laplacian matrix. > > Then, since I am using the k-means clustering algorithm, I formed a > matrix U whose columns are these eigenvectors. After that, I clustered > each row of U according to the k-means algorithm. Now, I have > different row vectors at different processes, as I want. > > In other words, my eigenvectors have 72 elements. So, right now, I > have 72 different row vectors with 4 elements. Those 72 row vectors > were clustered but not in a serial order. For instance, the 4th vector > is in the first process and the 5th one is in the 4th process. > > As the last part of my work, I want to find mincut of this > partitioning but I couldn't understand how to do it. If I didn't use > k-means algorithm, I would use MatPartitioningApply but now, since I > clustered row vectors of U according to the index set I obtained from > k-means algorithm, I don't know how to use partitioning routines. > Should I form another matrix by reordering these vectors and then > partition? But then, how can I be sure the vectors stay in the same > process as before? Besides MatPartitioning routines, is there any > routine I can use for this? I thought MatColoring can work but I guess > it won't. > > Thanks a lot! > > Eda From mfadams at lbl.gov Thu Jun 18 16:17:18 2020 From: mfadams at lbl.gov (Mark Adams) Date: Thu, 18 Jun 2020 17:17:18 -0400 Subject: [petsc-users] Regarding P4est In-Reply-To: <1A1B2B5B-FE8A-4BE4-9294-0380A6C3C70D@gmail.com> References: <1A1B2B5B-FE8A-4BE4-9294-0380A6C3C70D@gmail.com> Message-ID: I would put a print statement in your point method (Riemann solver) and print all the incoming values, face center, normal, L/R states, etc. Then 'sort' the output, because the ordering will be different, and 'diff' the good and bad runs. See what changes. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mukkundsunjii at gmail.com Thu Jun 18 16:40:57 2020 From: mukkundsunjii at gmail.com (MUKKUND SUNJII) Date: Thu, 18 Jun 2020 23:40:57 +0200 Subject: [petsc-users] Regarding P4est In-Reply-To: References: <1A1B2B5B-FE8A-4BE4-9294-0380A6C3C70D@gmail.com> Message-ID: <69FBE951-E355-4E26-9D53-49305950909D@gmail.com> Thats an interesting suggestion. I can certainly try that out. > On 18 Jun 2020, at 23:17, Mark Adams wrote: > > I would put a print statement in your point method (Riemann solver) and print all the incoming values, face center, normal, L/R states, etc. Then 'sort' the output, because the ordering will be different, and 'diff' the good and bad runs. See what changes. -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Thu Jun 18 18:30:45 2020 From: bsmith at petsc.dev (Barry Smith) Date: Thu, 18 Jun 2020 18:30:45 -0500 Subject: [petsc-users] Problem with GPU configure at NERSC In-Reply-To: References: Message-ID: <4CBEA683-26A3-4B4D-B47E-23BAC1BA4E03@petsc.dev> Mark, When it calls the C compiler it prints Preprocess stderr before filtering:No supported cpu target is set, CRAY_CPU_TARGET=x86-64 will be used. Load a valid targeting module or set CRAY_CPU_TARGET the C compiler is outputting its message on stderr and that is triggering the configure failure. Since PETSc does not know this is a harmless error message. I will update the filter code to ignore this "error message" but you should just set that environmental variable before you run configure or use the modules in a different way. Barry > On Jun 18, 2020, at 3:57 PM, Mark Adams wrote: > > I am trying to configure the GPU nodes at NERSC and the instructions are pretty slim. > Any idea what is wrong here? > Thanks, > Mark > From bsmith at petsc.dev Thu Jun 18 18:44:04 2020 From: bsmith at petsc.dev (Barry Smith) Date: Thu, 18 Jun 2020 18:44:04 -0500 Subject: [petsc-users] Problem with GPU configure at NERSC In-Reply-To: References: Message-ID: <73503BDA-21D9-44BC-9A5C-B2C699756615@petsc.dev> Mark, Please try the branch barry/2020-06-18/filter-nersc-cray/maint and DO NOT set the variable CRAY_CPU_TARGET (to test the branch, if it works it will be put into maint) Thanks Barry > On Jun 18, 2020, at 3:57 PM, Mark Adams wrote: > > I am trying to configure the GPU nodes at NERSC and the instructions are pretty slim. > Any idea what is wrong here? > Thanks, > Mark > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Thu Jun 18 18:49:26 2020 From: mfadams at lbl.gov (Mark Adams) Date: Thu, 18 Jun 2020 19:49:26 -0400 Subject: [petsc-users] Problem with GPU configure at NERSC In-Reply-To: <73503BDA-21D9-44BC-9A5C-B2C699756615@petsc.dev> References: <73503BDA-21D9-44BC-9A5C-B2C699756615@petsc.dev> Message-ID: THank, still not working. And they do have this wrt cross compiling: Nearly all software provided by Cray (cray-petsc, cray-fftw, cray-hdf5, etc.) is not usable on the Cori GPU nodes. This is because the GPU nodes have different hardware and run a different OS. Only a select subset of modules available on Cori are designed to work on the GPU nodes. This means you will likely need to compile your own software directly on the GPU nodes themselves, rather than cross-compiling for the GPU nodes on a login node. On Thu, Jun 18, 2020 at 7:44 PM Barry Smith wrote: > > Mark, > > Please try the branch *barry/2020-06-18/filter-nersc-cray/maint and > DO NOT set the variable **CRAY_CPU_TARGET (to test the branch, if it > works it will be put into maint)* > > * Thanks* > > * Barry* > > > On Jun 18, 2020, at 3:57 PM, Mark Adams wrote: > > I am trying to configure the GPU nodes at NERSC and the instructions are > pretty slim. > Any idea what is wrong here? > Thanks, > Mark > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: configure.log Type: application/octet-stream Size: 90994 bytes Desc: not available URL: From bsmith at petsc.dev Thu Jun 18 19:32:56 2020 From: bsmith at petsc.dev (Barry Smith) Date: Thu, 18 Jun 2020 19:32:56 -0500 Subject: [petsc-users] Problem with GPU configure at NERSC In-Reply-To: References: <73503BDA-21D9-44BC-9A5C-B2C699756615@petsc.dev> Message-ID: <3AAA1390-B3A2-4BA5-8558-F5AFCB4B518D@petsc.dev> Sorry, my mistake. Please try again when you have the change with a new pull. Thanks Barry > On Jun 18, 2020, at 6:49 PM, Mark Adams wrote: > > THank, still not working. > > And they do have this wrt cross compiling: > > Nearly all software provided by Cray (cray-petsc, cray-fftw, cray-hdf5, etc.) is not usable on the Cori GPU nodes. This is because the GPU nodes have different hardware and run a different OS. Only a select subset of modules available on Cori are designed to work on the GPU nodes. > > This means you will likely need to compile your own software directly on the GPU nodes themselves, rather than cross-compiling for the GPU nodes on a login node. > > > On Thu, Jun 18, 2020 at 7:44 PM Barry Smith > wrote: > > Mark, > > Please try the branch barry/2020-06-18/filter-nersc-cray/maint and DO NOT set the variable CRAY_CPU_TARGET (to test the branch, if it works it will be put into maint) > > Thanks > > Barry > > >> On Jun 18, 2020, at 3:57 PM, Mark Adams > wrote: >> >> I am trying to configure the GPU nodes at NERSC and the instructions are pretty slim. >> Any idea what is wrong here? >> Thanks, >> Mark >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bourdin at lsu.edu Fri Jun 19 10:15:38 2020 From: bourdin at lsu.edu (Blaise A Bourdin) Date: Fri, 19 Jun 2020 15:15:38 +0000 Subject: [petsc-users] Cohesive Element Support In-Reply-To: References: <7C00A8A6-F7F2-4D62-852B-45ADCD317B98@gmail.com> Message-ID: On Jun 18, 2020, at 5:28 AM, Matthew Knepley > wrote: On Wed, Jun 17, 2020 at 4:05 PM Jacob Faibussowitsch > wrote: Hello, I am looking to perform large scale fracture and crack propagation simulations and have a few questions regarding PETSc support for this. Specifically I am looking for cohesive surface element support with a few twists: 1. Is there support for zero thickness surface elements? For example modeling virtually flat patches of adhesives holding together two larger structures being pulled apart. This is how PyLith works: https://github.com/geodynamics/pylith 2. Is there support for ?joining? two possibly distinct meshes with cohesive surface elements? For example say I have two distinct cylinders representing fibers which would ?touch" to form an X shape. No, it would have to be coded. 3. In a similar vein, is there support for a mesh to fracture entirely along a crack formed through the cohesive elements? Imagine the aforementioned X configuration separating entirely into two separate cylinders again. No, it would have to be coded. 4. Is there a mechanism by which you can classify existing elements as cohesive elements? See 1. 5. Is there an already implemented way of imposing tie-constraints between independent meshes? This would potentially be used to tie high order cohesive cells which would have a non-conforming interface to the ?regular? mesh. There is nothing for non-conforming interfaces. From googling I have come across DMPlexCreateHybridMesh(), DMPlexConstructCohesiveCells(), and DMPlexCreateCohesiveSubmesh(). While these do implement cohesive cells these functions don?t at first glance seem to allow one to implement the above. Having worked with cohesive elements for more than a decade, I would be cautious about a new code using them for fracture. To me, it appears that variational fracture codes, like those from Blaise Bourdin and J. J. Marigo's group have much better geometric flexibility, and Maurini's work on the solver clears up the hardest part. I definitely concur with this and would be happy to help? Blaise -- A.K. & Shirley Barton Professor of Mathematics Adjunct Professor of Mechanical Engineering Adjunct of the Center for Computation & Technology Louisiana State University, Lockett Hall Room 344, Baton Rouge, LA 70803, USA Tel. +1 (225) 578 1612, Fax +1 (225) 578 4276 Web http://www.math.lsu.edu/~bourdin -------------- next part -------------- An HTML attachment was scrubbed... URL: From eda.oktay at metu.edu.tr Fri Jun 19 10:22:47 2020 From: eda.oktay at metu.edu.tr (Eda Oktay) Date: Fri, 19 Jun 2020 18:22:47 +0300 Subject: [petsc-users] MatFindOffBlockDiagonalEntries error Message-ID: Hi all, I am trying to find off block diagonal entries of a matrix and I am trying to use MatFindOffBlockDiagonalEntries. However, although my matrix is not NULL, I am getting an error message as follows: 1]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [1]PETSC ERROR: [2]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [2]PETSC ERROR: Null argument, when expecting valid pointer [2]PETSC ERROR: Null Object: Parameter # 1 [2]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [2]PETSC ERROR: Petsc Release Version 3.11.1, Apr, 12, 2019 [2]PETSC ERROR: [3]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [3]PETSC ERROR: Null argument, when expecting valid pointer [3]PETSC ERROR: Null Object: Parameter # 1 [3]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [3]PETSC ERROR: Null argument, when expecting valid pointer [1]PETSC ERROR: Null Object: Parameter # 1 [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [1]PETSC ERROR: Petsc Release Version 3.11.1, Apr, 12, 2019 [1]PETSC ERROR: ./approx_cut_deneme_clustering_son_final_edgecut_without_parmetis on a arch-linux2-c-debug named b342.wls.metu.edu.tr by edaoktay Fri Jun 19 18:19:11 2020 [1]PETSC ERROR: Configure options --download-mpich --download-openblas --download-slepc --download-metis --download-parmetis --download-chaco --with-X=1 [1]PETSC ERROR: #1 ISGetSize() line 510 in /home/edaoktay/petsc-3.11.1/src/vec/is/is/interface/index.c [1]PETSC ERROR: ./approx_cut_deneme_clustering_son_final_edgecut_without_parmetis on a arch-linux2-c-debug named b342.wls.metu.edu.tr by edaoktay Fri Jun 19 18:19:11 2020 [2]PETSC ERROR: Configure options --download-mpich --download-openblas --download-slepc --download-metis --download-parmetis --download-chaco --with-X=1 [2]PETSC ERROR: #1 ISGetSize() line 510 in /home/edaoktay/petsc-3.11.1/src/vec/is/is/interface/index.c [2]PETSC ERROR: #2 MatFindOffBlockDiagonalEntries_MPIAIJ() line 211 in /home/edaoktay/petsc-3.11.1/src/mat/impls/aij/mpi/mpiaij.c [2]PETSC ERROR: #2 MatFindOffBlockDiagonalEntries_MPIAIJ() line 211 in /home/edaoktay/petsc-3.11.1/src/mat/impls/aij/mpi/mpiaij.c [1]PETSC ERROR: #3 MatFindOffBlockDiagonalEntries() line 10502 in /home/edaoktay/petsc-3.11.1/src/mat/interface/matrix.c #3 MatFindOffBlockDiagonalEntries() line 10502 in /home/edaoktay/petsc-3.11.1/src/mat/interface/matrix.c Petsc Release Version 3.11.1, Apr, 12, 2019 [3]PETSC ERROR: ./approx_cut_deneme_clustering_son_final_edgecut_without_parmetis on a arch-linux2-c-debug named b342.wls.metu.edu.tr by edaoktay Fri Jun 19 18:19:11 2020 [3]PETSC ERROR: Configure options --download-mpich --download-openblas --download-slepc --download-metis --download-parmetis --download-chaco --with-X=1 [3]PETSC ERROR: #1 ISGetSize() line 510 in /home/edaoktay/petsc-3.11.1/src/vec/is/is/interface/index.c [3]PETSC ERROR: #2 MatFindOffBlockDiagonalEntries_MPIAIJ() line 211 in /home/edaoktay/petsc-3.11.1/src/mat/impls/aij/mpi/mpiaij.c [3]PETSC ERROR: #3 MatFindOffBlockDiagonalEntries() line 10502 in /home/edaoktay/petsc-3.11.1/src/mat/interface/matrix.c And the program is still running. The part of my program is : Mat Ais; MatCreateSubMatrix(A,idUi,idUi,MAT_INITIAL_MATRIX,&Ais); ISSetPermutation(idUi); MatPermute(Ais,idUi,idUi,&PL); IS offd; MatFindOffBlockDiagonalEntries(PL,&offd); Since there isn't any examples for this function, I couldn't understand what should I do. Since it says in the error that it is about parameter 1, the matrix, I don't know how to solve the problem. My matrix PL is parallel sparse. I know there shouldn't be any problem in PL since I can view PL by using MatView. Thanks! Eda From mfadams at lbl.gov Fri Jun 19 10:37:59 2020 From: mfadams at lbl.gov (Mark Adams) Date: Fri, 19 Jun 2020 11:37:59 -0400 Subject: [petsc-users] MatFindOffBlockDiagonalEntries error In-Reply-To: References: Message-ID: I don't know what is going on here. There was an update to this function about a year ago, so that might fix your problem. We would need you to test with a current version. Mark On Fri, Jun 19, 2020 at 11:23 AM Eda Oktay wrote: > Hi all, > > I am trying to find off block diagonal entries of a matrix and I am > trying to use MatFindOffBlockDiagonalEntries. However, although my > matrix is not NULL, I am getting an error message as follows: > > 1]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [1]PETSC ERROR: [2]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [2]PETSC ERROR: Null argument, when expecting valid pointer > [2]PETSC ERROR: Null Object: Parameter # 1 > [2]PETSC ERROR: See > http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble > shooting. > [2]PETSC ERROR: Petsc Release Version 3.11.1, Apr, 12, 2019 > [2]PETSC ERROR: [3]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [3]PETSC ERROR: Null argument, when expecting valid pointer > [3]PETSC ERROR: Null Object: Parameter # 1 > [3]PETSC ERROR: See > http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble > shooting. > [3]PETSC ERROR: Null argument, when expecting valid pointer > [1]PETSC ERROR: Null Object: Parameter # 1 > [1]PETSC ERROR: See > http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble > shooting. > [1]PETSC ERROR: Petsc Release Version 3.11.1, Apr, 12, 2019 > [1]PETSC ERROR: > ./approx_cut_deneme_clustering_son_final_edgecut_without_parmetis on a > arch-linux2-c-debug named b342.wls.metu.edu.tr by edaoktay Fri Jun 19 > 18:19:11 2020 > [1]PETSC ERROR: Configure options --download-mpich --download-openblas > --download-slepc --download-metis --download-parmetis --download-chaco > --with-X=1 > [1]PETSC ERROR: #1 ISGetSize() line 510 in > /home/edaoktay/petsc-3.11.1/src/vec/is/is/interface/index.c > [1]PETSC ERROR: > ./approx_cut_deneme_clustering_son_final_edgecut_without_parmetis on a > arch-linux2-c-debug named b342.wls.metu.edu.tr by edaoktay Fri Jun 19 > 18:19:11 2020 > [2]PETSC ERROR: Configure options --download-mpich --download-openblas > --download-slepc --download-metis --download-parmetis --download-chaco > --with-X=1 > [2]PETSC ERROR: #1 ISGetSize() line 510 in > /home/edaoktay/petsc-3.11.1/src/vec/is/is/interface/index.c > [2]PETSC ERROR: #2 MatFindOffBlockDiagonalEntries_MPIAIJ() line 211 in > /home/edaoktay/petsc-3.11.1/src/mat/impls/aij/mpi/mpiaij.c > [2]PETSC ERROR: #2 MatFindOffBlockDiagonalEntries_MPIAIJ() line 211 in > /home/edaoktay/petsc-3.11.1/src/mat/impls/aij/mpi/mpiaij.c > [1]PETSC ERROR: #3 MatFindOffBlockDiagonalEntries() line 10502 in > /home/edaoktay/petsc-3.11.1/src/mat/interface/matrix.c > #3 MatFindOffBlockDiagonalEntries() line 10502 in > /home/edaoktay/petsc-3.11.1/src/mat/interface/matrix.c > Petsc Release Version 3.11.1, Apr, 12, 2019 > [3]PETSC ERROR: > ./approx_cut_deneme_clustering_son_final_edgecut_without_parmetis on a > arch-linux2-c-debug named b342.wls.metu.edu.tr by edaoktay Fri Jun 19 > 18:19:11 2020 > [3]PETSC ERROR: Configure options --download-mpich --download-openblas > --download-slepc --download-metis --download-parmetis --download-chaco > --with-X=1 > [3]PETSC ERROR: #1 ISGetSize() line 510 in > /home/edaoktay/petsc-3.11.1/src/vec/is/is/interface/index.c > [3]PETSC ERROR: #2 MatFindOffBlockDiagonalEntries_MPIAIJ() line 211 in > /home/edaoktay/petsc-3.11.1/src/mat/impls/aij/mpi/mpiaij.c > [3]PETSC ERROR: #3 MatFindOffBlockDiagonalEntries() line 10502 in > /home/edaoktay/petsc-3.11.1/src/mat/interface/matrix.c > > And the program is still running. > > The part of my program is : > > Mat Ais; > MatCreateSubMatrix(A,idUi,idUi,MAT_INITIAL_MATRIX,&Ais); > ISSetPermutation(idUi); > MatPermute(Ais,idUi,idUi,&PL); > IS offd; > MatFindOffBlockDiagonalEntries(PL,&offd); > > Since there isn't any examples for this function, I couldn't > understand what should I do. Since it says in the error that it is > about parameter 1, the matrix, I don't know how to solve the problem. > My matrix PL is parallel sparse. I know there shouldn't be any problem > in PL since I can view PL by using MatView. > > Thanks! > > Eda > -------------- next part -------------- An HTML attachment was scrubbed... URL: From eda.oktay at metu.edu.tr Fri Jun 19 11:46:47 2020 From: eda.oktay at metu.edu.tr (Eda Oktay) Date: Fri, 19 Jun 2020 19:46:47 +0300 Subject: [petsc-users] MatFindOffBlockDiagonalEntries error In-Reply-To: References: Message-ID: Dear Mark, I updated Petsc and again, here's the error: [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Arguments must have same communicators [0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [1]PETSC ERROR: Arguments must have same communicators [1]PETSC ERROR: Different communicators in the two objects: Argument # 1 and 2 flag 3 [1]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [1]PETSC ERROR: Petsc Release Version 3.13.2, Jun 02, 2020 [1]PETSC ERROR: [2]PETSC ERROR: [3]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [3]PETSC ERROR: Arguments must have same communicators [3]PETSC ERROR: Different communicators in the two objects: Argument # 1 and 2 flag 3 [3]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [3]PETSC ERROR: Petsc Release Version 3.13.2, Jun 02, 2020 [3]PETSC ERROR: ./approx_cut_deneme_clustering_son_final_edgecut_without_parmetis on a arch-linux2-c-debug named b342.wls.metu.edu.tr by edaoktay Fri Jun 19 19:44:41 2020 [3]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --download-mpich --download-openblas --download-slepc --download-metis --download-parmetis --download-chaco --with-X=1 Different communicators in the two objects: Argument # 1 and 2 flag 3 [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.13.2, Jun 02, 2020 [0]PETSC ERROR: ./approx_cut_deneme_clustering_son_final_edgecut_without_parmetis on a arch-linux2-c-debug named b342.wls.metu.edu.tr by edaoktay Fri Jun 19 19:44:41 2020 [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --download-mpich --download-openblas --download-slepc --download-metis --download-parmetis --download-chaco --with-X=1 [0]PETSC ERROR: #1 MatCopy() line 4044 in /home/edaoktay/petsc-3.13.2/src/mat/interface/matrix.c ./approx_cut_deneme_clustering_son_final_edgecut_without_parmetis on a arch-linux2-c-debug named b342.wls.metu.edu.tr by edaoktay Fri Jun 19 19:44:41 2020 [1]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --download-mpich --download-openblas --download-slepc --download-metis --download-parmetis --download-chaco --with-X=1 [1]PETSC ERROR: #1 MatCopy() line 4044 in /home/edaoktay/petsc-3.13.2/src/mat/interface/matrix.c [3]PETSC ERROR: #1 MatCopy() line 4044 in /home/edaoktay/petsc-3.13.2/src/mat/interface/matrix.c [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Nonconforming object sizes [1]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [1]PETSC ERROR: Nonconforming object sizes [1]PETSC ERROR: Non conforming matrix add: global sizes 1 x 72, 1 x 72 [3]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [3]PETSC ERROR: Nonconforming object sizes [3]PETSC ERROR: Non conforming matrix add: global sizes 16 x 72, 16 x 72 [3]PETSC ERROR: [0]PETSC ERROR: Non conforming matrix add: global sizes 52 x 72, 52 x 72 [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.13.2, Jun 02, 2020 [0]PETSC ERROR: [1]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [1]PETSC ERROR: Petsc Release Version 3.13.2, Jun 02, 2020 [1]PETSC ERROR: ./approx_cut_deneme_clustering_son_final_edgecut_without_parmetis on a arch-linux2-c-debug named b342.wls.metu.edu.tr by edaoktay Fri Jun 19 19:44:41 2020 See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [3]PETSC ERROR: Petsc Release Version 3.13.2, Jun 02, 2020 [3]PETSC ERROR: ./approx_cut_deneme_clustering_son_final_edgecut_without_parmetis on a arch-linux2-c-debug named b342.wls.metu.edu.tr by edaoktay Fri Jun 19 19:44:41 2020 [3]PETSC ERROR: ./approx_cut_deneme_clustering_son_final_edgecut_without_parmetis on a arch-linux2-c-debug named b342.wls.metu.edu.tr by edaoktay Fri Jun 19 19:44:41 2020 [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --download-mpich --download-openblas --download-slepc --download-metis --download-parmetis --download-chaco --with-X=1 [0]PETSC ERROR: [1]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --download-mpich --download-openblas --download-slepc --download-metis --download-parmetis --download-chaco --with-X=1 [1]PETSC ERROR: #2 MatAXPY() line 72 in /home/edaoktay/petsc-3.13.2/src/mat/utils/axpy.c Configure options --with-cc=gcc --with-cxx=g++ --download-mpich --download-openblas --download-slepc --download-metis --download-parmetis --download-chaco --with-X=1 [3]PETSC ERROR: #2 MatAXPY() line 72 in /home/edaoktay/petsc-3.13.2/src/mat/utils/axpy.c #2 MatAXPY() line 72 in /home/edaoktay/petsc-3.13.2/src/mat/utils/axpy.c Elapsed time: 2.5e-01 --------------------- Error Message -------------------------------------------------------------- [2]PETSC ERROR: Arguments must have same communicators [2]PETSC ERROR: Different communicators in the two objects: Argument # 1 and 2 flag 3 [2]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [2]PETSC ERROR: Petsc Release Version 3.13.2, Jun 02, 2020 [2]PETSC ERROR: ./approx_cut_deneme_clustering_son_final_edgecut_without_parmetis on a arch-linux2-c-debug named b342.wls.metu.edu.tr by edaoktay Fri Jun 19 19:44:41 2020 [2]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --download-mpich --download-openblas --download-slepc --download-metis --download-parmetis --download-chaco --with-X=1 [2]PETSC ERROR: #1 MatCopy() line 4044 in /home/edaoktay/petsc-3.13.2/src/mat/interface/matrix.c [2]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [2]PETSC ERROR: Nonconforming object sizes [2]PETSC ERROR: Non conforming matrix add: global sizes 3 x 72, 3 x 72 [2]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [2]PETSC ERROR: Petsc Release Version 3.13.2, Jun 02, 2020 [2]PETSC ERROR: ./approx_cut_deneme_clustering_son_final_edgecut_without_parmetis on a arch-linux2-c-debug named b342.wls.metu.edu.tr by edaoktay Fri Jun 19 19:44:41 2020 [2]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --download-mpich --download-openblas --download-slepc --download-metis --download-parmetis --download-chaco --with-X=1 [2]PETSC ERROR: #2 MatAXPY() line 72 in /home/edaoktay/petsc-3.13.2/src/mat/utils/axpy.c What can be the reason? Mark Adams , 19 Haz 2020 Cum, 18:41 tarihinde ?unu yazd?: > > I don't know what is going on here. There was an update to this function about a year ago, so that might fix your problem. > > We would need you to test with a current version. > > Mark > > On Fri, Jun 19, 2020 at 11:23 AM Eda Oktay wrote: >> >> Hi all, >> >> I am trying to find off block diagonal entries of a matrix and I am >> trying to use MatFindOffBlockDiagonalEntries. However, although my >> matrix is not NULL, I am getting an error message as follows: >> >> 1]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> [1]PETSC ERROR: [2]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> [2]PETSC ERROR: Null argument, when expecting valid pointer >> [2]PETSC ERROR: Null Object: Parameter # 1 >> [2]PETSC ERROR: See >> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble >> shooting. >> [2]PETSC ERROR: Petsc Release Version 3.11.1, Apr, 12, 2019 >> [2]PETSC ERROR: [3]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> [3]PETSC ERROR: Null argument, when expecting valid pointer >> [3]PETSC ERROR: Null Object: Parameter # 1 >> [3]PETSC ERROR: See >> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble >> shooting. >> [3]PETSC ERROR: Null argument, when expecting valid pointer >> [1]PETSC ERROR: Null Object: Parameter # 1 >> [1]PETSC ERROR: See >> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble >> shooting. >> [1]PETSC ERROR: Petsc Release Version 3.11.1, Apr, 12, 2019 >> [1]PETSC ERROR: >> ./approx_cut_deneme_clustering_son_final_edgecut_without_parmetis on a >> arch-linux2-c-debug named b342.wls.metu.edu.tr by edaoktay Fri Jun 19 >> 18:19:11 2020 >> [1]PETSC ERROR: Configure options --download-mpich --download-openblas >> --download-slepc --download-metis --download-parmetis --download-chaco >> --with-X=1 >> [1]PETSC ERROR: #1 ISGetSize() line 510 in >> /home/edaoktay/petsc-3.11.1/src/vec/is/is/interface/index.c >> [1]PETSC ERROR: >> ./approx_cut_deneme_clustering_son_final_edgecut_without_parmetis on a >> arch-linux2-c-debug named b342.wls.metu.edu.tr by edaoktay Fri Jun 19 >> 18:19:11 2020 >> [2]PETSC ERROR: Configure options --download-mpich --download-openblas >> --download-slepc --download-metis --download-parmetis --download-chaco >> --with-X=1 >> [2]PETSC ERROR: #1 ISGetSize() line 510 in >> /home/edaoktay/petsc-3.11.1/src/vec/is/is/interface/index.c >> [2]PETSC ERROR: #2 MatFindOffBlockDiagonalEntries_MPIAIJ() line 211 in >> /home/edaoktay/petsc-3.11.1/src/mat/impls/aij/mpi/mpiaij.c >> [2]PETSC ERROR: #2 MatFindOffBlockDiagonalEntries_MPIAIJ() line 211 in >> /home/edaoktay/petsc-3.11.1/src/mat/impls/aij/mpi/mpiaij.c >> [1]PETSC ERROR: #3 MatFindOffBlockDiagonalEntries() line 10502 in >> /home/edaoktay/petsc-3.11.1/src/mat/interface/matrix.c >> #3 MatFindOffBlockDiagonalEntries() line 10502 in >> /home/edaoktay/petsc-3.11.1/src/mat/interface/matrix.c >> Petsc Release Version 3.11.1, Apr, 12, 2019 >> [3]PETSC ERROR: >> ./approx_cut_deneme_clustering_son_final_edgecut_without_parmetis on a >> arch-linux2-c-debug named b342.wls.metu.edu.tr by edaoktay Fri Jun 19 >> 18:19:11 2020 >> [3]PETSC ERROR: Configure options --download-mpich --download-openblas >> --download-slepc --download-metis --download-parmetis --download-chaco >> --with-X=1 >> [3]PETSC ERROR: #1 ISGetSize() line 510 in >> /home/edaoktay/petsc-3.11.1/src/vec/is/is/interface/index.c >> [3]PETSC ERROR: #2 MatFindOffBlockDiagonalEntries_MPIAIJ() line 211 in >> /home/edaoktay/petsc-3.11.1/src/mat/impls/aij/mpi/mpiaij.c >> [3]PETSC ERROR: #3 MatFindOffBlockDiagonalEntries() line 10502 in >> /home/edaoktay/petsc-3.11.1/src/mat/interface/matrix.c >> >> And the program is still running. >> >> The part of my program is : >> >> Mat Ais; >> MatCreateSubMatrix(A,idUi,idUi,MAT_INITIAL_MATRIX,&Ais); >> ISSetPermutation(idUi); >> MatPermute(Ais,idUi,idUi,&PL); >> IS offd; >> MatFindOffBlockDiagonalEntries(PL,&offd); >> >> Since there isn't any examples for this function, I couldn't >> understand what should I do. Since it says in the error that it is >> about parameter 1, the matrix, I don't know how to solve the problem. >> My matrix PL is parallel sparse. I know there shouldn't be any problem >> in PL since I can view PL by using MatView. >> >> Thanks! >> >> Eda From junchao.zhang at gmail.com Fri Jun 19 11:58:38 2020 From: junchao.zhang at gmail.com (Junchao Zhang) Date: Fri, 19 Jun 2020 11:58:38 -0500 Subject: [petsc-users] MatFindOffBlockDiagonalEntries error In-Reply-To: References: Message-ID: Do you have a small test example? --Junchao Zhang On Fri, Jun 19, 2020 at 11:47 AM Eda Oktay wrote: > Dear Mark, > > I updated Petsc and again, here's the error: > > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Arguments must have same communicators > [0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [1]PETSC ERROR: Arguments must have same communicators > [1]PETSC ERROR: Different communicators in the two objects: Argument # > 1 and 2 flag 3 > [1]PETSC ERROR: See > https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble > shooting. > [1]PETSC ERROR: Petsc Release Version 3.13.2, Jun 02, 2020 > [1]PETSC ERROR: [2]PETSC ERROR: [3]PETSC ERROR: --------------------- > Error Message > -------------------------------------------------------------- > [3]PETSC ERROR: Arguments must have same communicators > [3]PETSC ERROR: Different communicators in the two objects: Argument # > 1 and 2 flag 3 > [3]PETSC ERROR: See > https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble > shooting. > [3]PETSC ERROR: Petsc Release Version 3.13.2, Jun 02, 2020 > [3]PETSC ERROR: > ./approx_cut_deneme_clustering_son_final_edgecut_without_parmetis on a > arch-linux2-c-debug named b342.wls.metu.edu.tr by edaoktay Fri Jun 19 > 19:44:41 2020 > [3]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ > --download-mpich --download-openblas --download-slepc --download-metis > --download-parmetis --download-chaco --with-X=1 > Different communicators in the two objects: Argument # 1 and 2 flag 3 > [0]PETSC ERROR: See > https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble > shooting. > [0]PETSC ERROR: Petsc Release Version 3.13.2, Jun 02, 2020 > [0]PETSC ERROR: > ./approx_cut_deneme_clustering_son_final_edgecut_without_parmetis on a > arch-linux2-c-debug named b342.wls.metu.edu.tr by edaoktay Fri Jun 19 > 19:44:41 2020 > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ > --download-mpich --download-openblas --download-slepc --download-metis > --download-parmetis --download-chaco --with-X=1 > [0]PETSC ERROR: #1 MatCopy() line 4044 in > /home/edaoktay/petsc-3.13.2/src/mat/interface/matrix.c > ./approx_cut_deneme_clustering_son_final_edgecut_without_parmetis on a > arch-linux2-c-debug named b342.wls.metu.edu.tr by edaoktay Fri Jun 19 > 19:44:41 2020 > [1]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ > --download-mpich --download-openblas --download-slepc --download-metis > --download-parmetis --download-chaco --with-X=1 > [1]PETSC ERROR: #1 MatCopy() line 4044 in > /home/edaoktay/petsc-3.13.2/src/mat/interface/matrix.c > [3]PETSC ERROR: #1 MatCopy() line 4044 in > /home/edaoktay/petsc-3.13.2/src/mat/interface/matrix.c > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Nonconforming object sizes > [1]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [1]PETSC ERROR: Nonconforming object sizes > [1]PETSC ERROR: Non conforming matrix add: global sizes 1 x 72, 1 x 72 > [3]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [3]PETSC ERROR: Nonconforming object sizes > [3]PETSC ERROR: Non conforming matrix add: global sizes 16 x 72, 16 x 72 > [3]PETSC ERROR: [0]PETSC ERROR: Non conforming matrix add: global > sizes 52 x 72, 52 x 72 > [0]PETSC ERROR: See > https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble > shooting. > [0]PETSC ERROR: Petsc Release Version 3.13.2, Jun 02, 2020 > [0]PETSC ERROR: [1]PETSC ERROR: See > https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble > shooting. > [1]PETSC ERROR: Petsc Release Version 3.13.2, Jun 02, 2020 > [1]PETSC ERROR: > ./approx_cut_deneme_clustering_son_final_edgecut_without_parmetis on a > arch-linux2-c-debug named b342.wls.metu.edu.tr by edaoktay Fri Jun 19 > 19:44:41 2020 > See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble > shooting. > [3]PETSC ERROR: Petsc Release Version 3.13.2, Jun 02, 2020 > [3]PETSC ERROR: > ./approx_cut_deneme_clustering_son_final_edgecut_without_parmetis on a > arch-linux2-c-debug named b342.wls.metu.edu.tr by edaoktay Fri Jun 19 > 19:44:41 2020 > [3]PETSC ERROR: > ./approx_cut_deneme_clustering_son_final_edgecut_without_parmetis on a > arch-linux2-c-debug named b342.wls.metu.edu.tr by edaoktay Fri Jun 19 > 19:44:41 2020 > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ > --download-mpich --download-openblas --download-slepc --download-metis > --download-parmetis --download-chaco --with-X=1 > [0]PETSC ERROR: [1]PETSC ERROR: Configure options --with-cc=gcc > --with-cxx=g++ --download-mpich --download-openblas --download-slepc > --download-metis --download-parmetis --download-chaco --with-X=1 > [1]PETSC ERROR: #2 MatAXPY() line 72 in > /home/edaoktay/petsc-3.13.2/src/mat/utils/axpy.c > Configure options --with-cc=gcc --with-cxx=g++ --download-mpich > --download-openblas --download-slepc --download-metis > --download-parmetis --download-chaco --with-X=1 > [3]PETSC ERROR: #2 MatAXPY() line 72 in > /home/edaoktay/petsc-3.13.2/src/mat/utils/axpy.c > #2 MatAXPY() line 72 in /home/edaoktay/petsc-3.13.2/src/mat/utils/axpy.c > > Elapsed time: 2.5e-01 > > --------------------- Error Message > -------------------------------------------------------------- > [2]PETSC ERROR: Arguments must have same communicators > [2]PETSC ERROR: Different communicators in the two objects: Argument # > 1 and 2 flag 3 > [2]PETSC ERROR: See > https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble > shooting. > [2]PETSC ERROR: Petsc Release Version 3.13.2, Jun 02, 2020 > [2]PETSC ERROR: > ./approx_cut_deneme_clustering_son_final_edgecut_without_parmetis on a > arch-linux2-c-debug named b342.wls.metu.edu.tr by edaoktay Fri Jun 19 > 19:44:41 2020 > [2]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ > --download-mpich --download-openblas --download-slepc --download-metis > --download-parmetis --download-chaco --with-X=1 > [2]PETSC ERROR: #1 MatCopy() line 4044 in > /home/edaoktay/petsc-3.13.2/src/mat/interface/matrix.c > [2]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [2]PETSC ERROR: Nonconforming object sizes > [2]PETSC ERROR: Non conforming matrix add: global sizes 3 x 72, 3 x 72 > [2]PETSC ERROR: See > https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble > shooting. > [2]PETSC ERROR: Petsc Release Version 3.13.2, Jun 02, 2020 > [2]PETSC ERROR: > ./approx_cut_deneme_clustering_son_final_edgecut_without_parmetis on a > arch-linux2-c-debug named b342.wls.metu.edu.tr by edaoktay Fri Jun 19 > 19:44:41 2020 > [2]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ > --download-mpich --download-openblas --download-slepc --download-metis > --download-parmetis --download-chaco --with-X=1 > [2]PETSC ERROR: #2 MatAXPY() line 72 in > /home/edaoktay/petsc-3.13.2/src/mat/utils/axpy.c > > What can be the reason? > > Mark Adams , 19 Haz 2020 Cum, 18:41 tarihinde ?unu yazd?: > > > > I don't know what is going on here. There was an update to this function > about a year ago, so that might fix your problem. > > > > We would need you to test with a current version. > > > > Mark > > > > On Fri, Jun 19, 2020 at 11:23 AM Eda Oktay > wrote: > >> > >> Hi all, > >> > >> I am trying to find off block diagonal entries of a matrix and I am > >> trying to use MatFindOffBlockDiagonalEntries. However, although my > >> matrix is not NULL, I am getting an error message as follows: > >> > >> 1]PETSC ERROR: --------------------- Error Message > >> -------------------------------------------------------------- > >> [1]PETSC ERROR: [2]PETSC ERROR: --------------------- Error Message > >> -------------------------------------------------------------- > >> [2]PETSC ERROR: Null argument, when expecting valid pointer > >> [2]PETSC ERROR: Null Object: Parameter # 1 > >> [2]PETSC ERROR: See > >> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble > >> shooting. > >> [2]PETSC ERROR: Petsc Release Version 3.11.1, Apr, 12, 2019 > >> [2]PETSC ERROR: [3]PETSC ERROR: --------------------- Error Message > >> -------------------------------------------------------------- > >> [3]PETSC ERROR: Null argument, when expecting valid pointer > >> [3]PETSC ERROR: Null Object: Parameter # 1 > >> [3]PETSC ERROR: See > >> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble > >> shooting. > >> [3]PETSC ERROR: Null argument, when expecting valid pointer > >> [1]PETSC ERROR: Null Object: Parameter # 1 > >> [1]PETSC ERROR: See > >> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble > >> shooting. > >> [1]PETSC ERROR: Petsc Release Version 3.11.1, Apr, 12, 2019 > >> [1]PETSC ERROR: > >> ./approx_cut_deneme_clustering_son_final_edgecut_without_parmetis on a > >> arch-linux2-c-debug named b342.wls.metu.edu.tr by edaoktay Fri Jun 19 > >> 18:19:11 2020 > >> [1]PETSC ERROR: Configure options --download-mpich --download-openblas > >> --download-slepc --download-metis --download-parmetis --download-chaco > >> --with-X=1 > >> [1]PETSC ERROR: #1 ISGetSize() line 510 in > >> /home/edaoktay/petsc-3.11.1/src/vec/is/is/interface/index.c > >> [1]PETSC ERROR: > >> ./approx_cut_deneme_clustering_son_final_edgecut_without_parmetis on a > >> arch-linux2-c-debug named b342.wls.metu.edu.tr by edaoktay Fri Jun 19 > >> 18:19:11 2020 > >> [2]PETSC ERROR: Configure options --download-mpich --download-openblas > >> --download-slepc --download-metis --download-parmetis --download-chaco > >> --with-X=1 > >> [2]PETSC ERROR: #1 ISGetSize() line 510 in > >> /home/edaoktay/petsc-3.11.1/src/vec/is/is/interface/index.c > >> [2]PETSC ERROR: #2 MatFindOffBlockDiagonalEntries_MPIAIJ() line 211 in > >> /home/edaoktay/petsc-3.11.1/src/mat/impls/aij/mpi/mpiaij.c > >> [2]PETSC ERROR: #2 MatFindOffBlockDiagonalEntries_MPIAIJ() line 211 in > >> /home/edaoktay/petsc-3.11.1/src/mat/impls/aij/mpi/mpiaij.c > >> [1]PETSC ERROR: #3 MatFindOffBlockDiagonalEntries() line 10502 in > >> /home/edaoktay/petsc-3.11.1/src/mat/interface/matrix.c > >> #3 MatFindOffBlockDiagonalEntries() line 10502 in > >> /home/edaoktay/petsc-3.11.1/src/mat/interface/matrix.c > >> Petsc Release Version 3.11.1, Apr, 12, 2019 > >> [3]PETSC ERROR: > >> ./approx_cut_deneme_clustering_son_final_edgecut_without_parmetis on a > >> arch-linux2-c-debug named b342.wls.metu.edu.tr by edaoktay Fri Jun 19 > >> 18:19:11 2020 > >> [3]PETSC ERROR: Configure options --download-mpich --download-openblas > >> --download-slepc --download-metis --download-parmetis --download-chaco > >> --with-X=1 > >> [3]PETSC ERROR: #1 ISGetSize() line 510 in > >> /home/edaoktay/petsc-3.11.1/src/vec/is/is/interface/index.c > >> [3]PETSC ERROR: #2 MatFindOffBlockDiagonalEntries_MPIAIJ() line 211 in > >> /home/edaoktay/petsc-3.11.1/src/mat/impls/aij/mpi/mpiaij.c > >> [3]PETSC ERROR: #3 MatFindOffBlockDiagonalEntries() line 10502 in > >> /home/edaoktay/petsc-3.11.1/src/mat/interface/matrix.c > >> > >> And the program is still running. > >> > >> The part of my program is : > >> > >> Mat Ais; > >> MatCreateSubMatrix(A,idUi,idUi,MAT_INITIAL_MATRIX,&Ais); > >> ISSetPermutation(idUi); > >> MatPermute(Ais,idUi,idUi,&PL); > >> IS offd; > >> MatFindOffBlockDiagonalEntries(PL,&offd); > >> > >> Since there isn't any examples for this function, I couldn't > >> understand what should I do. Since it says in the error that it is > >> about parameter 1, the matrix, I don't know how to solve the problem. > >> My matrix PL is parallel sparse. I know there shouldn't be any problem > >> in PL since I can view PL by using MatView. > >> > >> Thanks! > >> > >> Eda > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Fri Jun 19 12:35:03 2020 From: bsmith at petsc.dev (Barry Smith) Date: Fri, 19 Jun 2020 12:35:03 -0500 Subject: [petsc-users] MatFindOffBlockDiagonalEntries error In-Reply-To: References: Message-ID: <997A6BBA-96ED-4C94-A449-694D8F4E0BD2@petsc.dev> Looks like a completely different location now. Are you calling MatAXPY ? If so perhaps the arguments are wrong. One is likely a parallel matrix and one sequential. > On Jun 19, 2020, at 11:46 AM, Eda Oktay wrote: > > Dear Mark, > > I updated Petsc and again, here's the error: > > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Arguments must have same communicators > [0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [1]PETSC ERROR: Arguments must have same communicators > [1]PETSC ERROR: Different communicators in the two objects: Argument # > 1 and 2 flag 3 > [1]PETSC ERROR: See > https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble > shooting. > [1]PETSC ERROR: Petsc Release Version 3.13.2, Jun 02, 2020 > [1]PETSC ERROR: [2]PETSC ERROR: [3]PETSC ERROR: --------------------- > Error Message -------------------------------------------------------------- > [3]PETSC ERROR: Arguments must have same communicators > [3]PETSC ERROR: Different communicators in the two objects: Argument # > 1 and 2 flag 3 > [3]PETSC ERROR: See > https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble > shooting. > [3]PETSC ERROR: Petsc Release Version 3.13.2, Jun 02, 2020 > [3]PETSC ERROR: > ./approx_cut_deneme_clustering_son_final_edgecut_without_parmetis on a > arch-linux2-c-debug named b342.wls.metu.edu.tr by edaoktay Fri Jun 19 > 19:44:41 2020 > [3]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ > --download-mpich --download-openblas --download-slepc --download-metis > --download-parmetis --download-chaco --with-X=1 > Different communicators in the two objects: Argument # 1 and 2 flag 3 > [0]PETSC ERROR: See > https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble > shooting. > [0]PETSC ERROR: Petsc Release Version 3.13.2, Jun 02, 2020 > [0]PETSC ERROR: > ./approx_cut_deneme_clustering_son_final_edgecut_without_parmetis on a > arch-linux2-c-debug named b342.wls.metu.edu.tr by edaoktay Fri Jun 19 > 19:44:41 2020 > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ > --download-mpich --download-openblas --download-slepc --download-metis > --download-parmetis --download-chaco --with-X=1 > [0]PETSC ERROR: #1 MatCopy() line 4044 in > /home/edaoktay/petsc-3.13.2/src/mat/interface/matrix.c > ./approx_cut_deneme_clustering_son_final_edgecut_without_parmetis on a > arch-linux2-c-debug named b342.wls.metu.edu.tr by edaoktay Fri Jun 19 > 19:44:41 2020 > [1]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ > --download-mpich --download-openblas --download-slepc --download-metis > --download-parmetis --download-chaco --with-X=1 > [1]PETSC ERROR: #1 MatCopy() line 4044 in > /home/edaoktay/petsc-3.13.2/src/mat/interface/matrix.c > [3]PETSC ERROR: #1 MatCopy() line 4044 in > /home/edaoktay/petsc-3.13.2/src/mat/interface/matrix.c > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Nonconforming object sizes > [1]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [1]PETSC ERROR: Nonconforming object sizes > [1]PETSC ERROR: Non conforming matrix add: global sizes 1 x 72, 1 x 72 > [3]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [3]PETSC ERROR: Nonconforming object sizes > [3]PETSC ERROR: Non conforming matrix add: global sizes 16 x 72, 16 x 72 > [3]PETSC ERROR: [0]PETSC ERROR: Non conforming matrix add: global > sizes 52 x 72, 52 x 72 > [0]PETSC ERROR: See > https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble > shooting. > [0]PETSC ERROR: Petsc Release Version 3.13.2, Jun 02, 2020 > [0]PETSC ERROR: [1]PETSC ERROR: See > https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble > shooting. > [1]PETSC ERROR: Petsc Release Version 3.13.2, Jun 02, 2020 > [1]PETSC ERROR: > ./approx_cut_deneme_clustering_son_final_edgecut_without_parmetis on a > arch-linux2-c-debug named b342.wls.metu.edu.tr by edaoktay Fri Jun 19 > 19:44:41 2020 > See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [3]PETSC ERROR: Petsc Release Version 3.13.2, Jun 02, 2020 > [3]PETSC ERROR: > ./approx_cut_deneme_clustering_son_final_edgecut_without_parmetis on a > arch-linux2-c-debug named b342.wls.metu.edu.tr by edaoktay Fri Jun 19 > 19:44:41 2020 > [3]PETSC ERROR: > ./approx_cut_deneme_clustering_son_final_edgecut_without_parmetis on a > arch-linux2-c-debug named b342.wls.metu.edu.tr by edaoktay Fri Jun 19 > 19:44:41 2020 > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ > --download-mpich --download-openblas --download-slepc --download-metis > --download-parmetis --download-chaco --with-X=1 > [0]PETSC ERROR: [1]PETSC ERROR: Configure options --with-cc=gcc > --with-cxx=g++ --download-mpich --download-openblas --download-slepc > --download-metis --download-parmetis --download-chaco --with-X=1 > [1]PETSC ERROR: #2 MatAXPY() line 72 in > /home/edaoktay/petsc-3.13.2/src/mat/utils/axpy.c > Configure options --with-cc=gcc --with-cxx=g++ --download-mpich > --download-openblas --download-slepc --download-metis > --download-parmetis --download-chaco --with-X=1 > [3]PETSC ERROR: #2 MatAXPY() line 72 in > /home/edaoktay/petsc-3.13.2/src/mat/utils/axpy.c > #2 MatAXPY() line 72 in /home/edaoktay/petsc-3.13.2/src/mat/utils/axpy.c > > Elapsed time: 2.5e-01 > > --------------------- Error Message > -------------------------------------------------------------- > [2]PETSC ERROR: Arguments must have same communicators > [2]PETSC ERROR: Different communicators in the two objects: Argument # > 1 and 2 flag 3 > [2]PETSC ERROR: See > https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble > shooting. > [2]PETSC ERROR: Petsc Release Version 3.13.2, Jun 02, 2020 > [2]PETSC ERROR: > ./approx_cut_deneme_clustering_son_final_edgecut_without_parmetis on a > arch-linux2-c-debug named b342.wls.metu.edu.tr by edaoktay Fri Jun 19 > 19:44:41 2020 > [2]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ > --download-mpich --download-openblas --download-slepc --download-metis > --download-parmetis --download-chaco --with-X=1 > [2]PETSC ERROR: #1 MatCopy() line 4044 in > /home/edaoktay/petsc-3.13.2/src/mat/interface/matrix.c > [2]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [2]PETSC ERROR: Nonconforming object sizes > [2]PETSC ERROR: Non conforming matrix add: global sizes 3 x 72, 3 x 72 > [2]PETSC ERROR: See > https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble > shooting. > [2]PETSC ERROR: Petsc Release Version 3.13.2, Jun 02, 2020 > [2]PETSC ERROR: > ./approx_cut_deneme_clustering_son_final_edgecut_without_parmetis on a > arch-linux2-c-debug named b342.wls.metu.edu.tr by edaoktay Fri Jun 19 > 19:44:41 2020 > [2]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ > --download-mpich --download-openblas --download-slepc --download-metis > --download-parmetis --download-chaco --with-X=1 > [2]PETSC ERROR: #2 MatAXPY() line 72 in > /home/edaoktay/petsc-3.13.2/src/mat/utils/axpy.c > > What can be the reason? > > Mark Adams , 19 Haz 2020 Cum, 18:41 tarihinde ?unu yazd?: >> >> I don't know what is going on here. There was an update to this function about a year ago, so that might fix your problem. >> >> We would need you to test with a current version. >> >> Mark >> >> On Fri, Jun 19, 2020 at 11:23 AM Eda Oktay wrote: >>> >>> Hi all, >>> >>> I am trying to find off block diagonal entries of a matrix and I am >>> trying to use MatFindOffBlockDiagonalEntries. However, although my >>> matrix is not NULL, I am getting an error message as follows: >>> >>> 1]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> [1]PETSC ERROR: [2]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> [2]PETSC ERROR: Null argument, when expecting valid pointer >>> [2]PETSC ERROR: Null Object: Parameter # 1 >>> [2]PETSC ERROR: See >>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble >>> shooting. >>> [2]PETSC ERROR: Petsc Release Version 3.11.1, Apr, 12, 2019 >>> [2]PETSC ERROR: [3]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> [3]PETSC ERROR: Null argument, when expecting valid pointer >>> [3]PETSC ERROR: Null Object: Parameter # 1 >>> [3]PETSC ERROR: See >>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble >>> shooting. >>> [3]PETSC ERROR: Null argument, when expecting valid pointer >>> [1]PETSC ERROR: Null Object: Parameter # 1 >>> [1]PETSC ERROR: See >>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble >>> shooting. >>> [1]PETSC ERROR: Petsc Release Version 3.11.1, Apr, 12, 2019 >>> [1]PETSC ERROR: >>> ./approx_cut_deneme_clustering_son_final_edgecut_without_parmetis on a >>> arch-linux2-c-debug named b342.wls.metu.edu.tr by edaoktay Fri Jun 19 >>> 18:19:11 2020 >>> [1]PETSC ERROR: Configure options --download-mpich --download-openblas >>> --download-slepc --download-metis --download-parmetis --download-chaco >>> --with-X=1 >>> [1]PETSC ERROR: #1 ISGetSize() line 510 in >>> /home/edaoktay/petsc-3.11.1/src/vec/is/is/interface/index.c >>> [1]PETSC ERROR: >>> ./approx_cut_deneme_clustering_son_final_edgecut_without_parmetis on a >>> arch-linux2-c-debug named b342.wls.metu.edu.tr by edaoktay Fri Jun 19 >>> 18:19:11 2020 >>> [2]PETSC ERROR: Configure options --download-mpich --download-openblas >>> --download-slepc --download-metis --download-parmetis --download-chaco >>> --with-X=1 >>> [2]PETSC ERROR: #1 ISGetSize() line 510 in >>> /home/edaoktay/petsc-3.11.1/src/vec/is/is/interface/index.c >>> [2]PETSC ERROR: #2 MatFindOffBlockDiagonalEntries_MPIAIJ() line 211 in >>> /home/edaoktay/petsc-3.11.1/src/mat/impls/aij/mpi/mpiaij.c >>> [2]PETSC ERROR: #2 MatFindOffBlockDiagonalEntries_MPIAIJ() line 211 in >>> /home/edaoktay/petsc-3.11.1/src/mat/impls/aij/mpi/mpiaij.c >>> [1]PETSC ERROR: #3 MatFindOffBlockDiagonalEntries() line 10502 in >>> /home/edaoktay/petsc-3.11.1/src/mat/interface/matrix.c >>> #3 MatFindOffBlockDiagonalEntries() line 10502 in >>> /home/edaoktay/petsc-3.11.1/src/mat/interface/matrix.c >>> Petsc Release Version 3.11.1, Apr, 12, 2019 >>> [3]PETSC ERROR: >>> ./approx_cut_deneme_clustering_son_final_edgecut_without_parmetis on a >>> arch-linux2-c-debug named b342.wls.metu.edu.tr by edaoktay Fri Jun 19 >>> 18:19:11 2020 >>> [3]PETSC ERROR: Configure options --download-mpich --download-openblas >>> --download-slepc --download-metis --download-parmetis --download-chaco >>> --with-X=1 >>> [3]PETSC ERROR: #1 ISGetSize() line 510 in >>> /home/edaoktay/petsc-3.11.1/src/vec/is/is/interface/index.c >>> [3]PETSC ERROR: #2 MatFindOffBlockDiagonalEntries_MPIAIJ() line 211 in >>> /home/edaoktay/petsc-3.11.1/src/mat/impls/aij/mpi/mpiaij.c >>> [3]PETSC ERROR: #3 MatFindOffBlockDiagonalEntries() line 10502 in >>> /home/edaoktay/petsc-3.11.1/src/mat/interface/matrix.c >>> >>> And the program is still running. >>> >>> The part of my program is : >>> >>> Mat Ais; >>> MatCreateSubMatrix(A,idUi,idUi,MAT_INITIAL_MATRIX,&Ais); >>> ISSetPermutation(idUi); >>> MatPermute(Ais,idUi,idUi,&PL); >>> IS offd; >>> MatFindOffBlockDiagonalEntries(PL,&offd); >>> >>> Since there isn't any examples for this function, I couldn't >>> understand what should I do. Since it says in the error that it is >>> about parameter 1, the matrix, I don't know how to solve the problem. >>> My matrix PL is parallel sparse. I know there shouldn't be any problem >>> in PL since I can view PL by using MatView. >>> >>> Thanks! >>> >>> Eda From mfadams at lbl.gov Fri Jun 19 12:35:36 2020 From: mfadams at lbl.gov (Mark Adams) Date: Fri, 19 Jun 2020 13:35:36 -0400 Subject: [petsc-users] MatFindOffBlockDiagonalEntries error In-Reply-To: References: Message-ID: This is more useful and you are getting a new error. Maybe the old error is fixed. It looks like you are trying to copy a matrix into another but they have different communicators. For example, one is created with MPI_COMM_WOLD and the other with PETSC_COMM_WORLD. On Fri, Jun 19, 2020 at 12:48 PM Eda Oktay wrote: > Dear Mark, > > I updated Petsc and again, here's the error: > > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Arguments must have same communicators > [0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [1]PETSC ERROR: Arguments must have same communicators > [1]PETSC ERROR: Different communicators in the two objects: Argument # > 1 and 2 flag 3 > [1]PETSC ERROR: See > https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble > shooting. > [1]PETSC ERROR: Petsc Release Version 3.13.2, Jun 02, 2020 > [1]PETSC ERROR: [2]PETSC ERROR: [3]PETSC ERROR: --------------------- > Error Message > -------------------------------------------------------------- > [3]PETSC ERROR: Arguments must have same communicators > [3]PETSC ERROR: Different communicators in the two objects: Argument # > 1 and 2 flag 3 > [3]PETSC ERROR: See > https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble > shooting. > [3]PETSC ERROR: Petsc Release Version 3.13.2, Jun 02, 2020 > [3]PETSC ERROR: > ./approx_cut_deneme_clustering_son_final_edgecut_without_parmetis on a > arch-linux2-c-debug named b342.wls.metu.edu.tr by edaoktay Fri Jun 19 > 19:44:41 2020 > [3]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ > --download-mpich --download-openblas --download-slepc --download-metis > --download-parmetis --download-chaco --with-X=1 > Different communicators in the two objects: Argument # 1 and 2 flag 3 > [0]PETSC ERROR: See > https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble > shooting. > [0]PETSC ERROR: Petsc Release Version 3.13.2, Jun 02, 2020 > [0]PETSC ERROR: > ./approx_cut_deneme_clustering_son_final_edgecut_without_parmetis on a > arch-linux2-c-debug named b342.wls.metu.edu.tr by edaoktay Fri Jun 19 > 19:44:41 2020 > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ > --download-mpich --download-openblas --download-slepc --download-metis > --download-parmetis --download-chaco --with-X=1 > [0]PETSC ERROR: #1 MatCopy() line 4044 in > /home/edaoktay/petsc-3.13.2/src/mat/interface/matrix.c > ./approx_cut_deneme_clustering_son_final_edgecut_without_parmetis on a > arch-linux2-c-debug named b342.wls.metu.edu.tr by edaoktay Fri Jun 19 > 19:44:41 2020 > [1]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ > --download-mpich --download-openblas --download-slepc --download-metis > --download-parmetis --download-chaco --with-X=1 > [1]PETSC ERROR: #1 MatCopy() line 4044 in > /home/edaoktay/petsc-3.13.2/src/mat/interface/matrix.c > [3]PETSC ERROR: #1 MatCopy() line 4044 in > /home/edaoktay/petsc-3.13.2/src/mat/interface/matrix.c > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Nonconforming object sizes > [1]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [1]PETSC ERROR: Nonconforming object sizes > [1]PETSC ERROR: Non conforming matrix add: global sizes 1 x 72, 1 x 72 > [3]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [3]PETSC ERROR: Nonconforming object sizes > [3]PETSC ERROR: Non conforming matrix add: global sizes 16 x 72, 16 x 72 > [3]PETSC ERROR: [0]PETSC ERROR: Non conforming matrix add: global > sizes 52 x 72, 52 x 72 > [0]PETSC ERROR: See > https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble > shooting. > [0]PETSC ERROR: Petsc Release Version 3.13.2, Jun 02, 2020 > [0]PETSC ERROR: [1]PETSC ERROR: See > https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble > shooting. > [1]PETSC ERROR: Petsc Release Version 3.13.2, Jun 02, 2020 > [1]PETSC ERROR: > ./approx_cut_deneme_clustering_son_final_edgecut_without_parmetis on a > arch-linux2-c-debug named b342.wls.metu.edu.tr by edaoktay Fri Jun 19 > 19:44:41 2020 > See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble > shooting. > [3]PETSC ERROR: Petsc Release Version 3.13.2, Jun 02, 2020 > [3]PETSC ERROR: > ./approx_cut_deneme_clustering_son_final_edgecut_without_parmetis on a > arch-linux2-c-debug named b342.wls.metu.edu.tr by edaoktay Fri Jun 19 > 19:44:41 2020 > [3]PETSC ERROR: > ./approx_cut_deneme_clustering_son_final_edgecut_without_parmetis on a > arch-linux2-c-debug named b342.wls.metu.edu.tr by edaoktay Fri Jun 19 > 19:44:41 2020 > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ > --download-mpich --download-openblas --download-slepc --download-metis > --download-parmetis --download-chaco --with-X=1 > [0]PETSC ERROR: [1]PETSC ERROR: Configure options --with-cc=gcc > --with-cxx=g++ --download-mpich --download-openblas --download-slepc > --download-metis --download-parmetis --download-chaco --with-X=1 > [1]PETSC ERROR: #2 MatAXPY() line 72 in > /home/edaoktay/petsc-3.13.2/src/mat/utils/axpy.c > Configure options --with-cc=gcc --with-cxx=g++ --download-mpich > --download-openblas --download-slepc --download-metis > --download-parmetis --download-chaco --with-X=1 > [3]PETSC ERROR: #2 MatAXPY() line 72 in > /home/edaoktay/petsc-3.13.2/src/mat/utils/axpy.c > #2 MatAXPY() line 72 in /home/edaoktay/petsc-3.13.2/src/mat/utils/axpy.c > > Elapsed time: 2.5e-01 > > --------------------- Error Message > -------------------------------------------------------------- > [2]PETSC ERROR: Arguments must have same communicators > [2]PETSC ERROR: Different communicators in the two objects: Argument # > 1 and 2 flag 3 > [2]PETSC ERROR: See > https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble > shooting. > [2]PETSC ERROR: Petsc Release Version 3.13.2, Jun 02, 2020 > [2]PETSC ERROR: > ./approx_cut_deneme_clustering_son_final_edgecut_without_parmetis on a > arch-linux2-c-debug named b342.wls.metu.edu.tr by edaoktay Fri Jun 19 > 19:44:41 2020 > [2]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ > --download-mpich --download-openblas --download-slepc --download-metis > --download-parmetis --download-chaco --with-X=1 > [2]PETSC ERROR: #1 MatCopy() line 4044 in > /home/edaoktay/petsc-3.13.2/src/mat/interface/matrix.c > [2]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [2]PETSC ERROR: Nonconforming object sizes > [2]PETSC ERROR: Non conforming matrix add: global sizes 3 x 72, 3 x 72 > [2]PETSC ERROR: See > https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble > shooting. > [2]PETSC ERROR: Petsc Release Version 3.13.2, Jun 02, 2020 > [2]PETSC ERROR: > ./approx_cut_deneme_clustering_son_final_edgecut_without_parmetis on a > arch-linux2-c-debug named b342.wls.metu.edu.tr by edaoktay Fri Jun 19 > 19:44:41 2020 > [2]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ > --download-mpich --download-openblas --download-slepc --download-metis > --download-parmetis --download-chaco --with-X=1 > [2]PETSC ERROR: #2 MatAXPY() line 72 in > /home/edaoktay/petsc-3.13.2/src/mat/utils/axpy.c > > What can be the reason? > > Mark Adams , 19 Haz 2020 Cum, 18:41 tarihinde ?unu yazd?: > > > > I don't know what is going on here. There was an update to this function > about a year ago, so that might fix your problem. > > > > We would need you to test with a current version. > > > > Mark > > > > On Fri, Jun 19, 2020 at 11:23 AM Eda Oktay > wrote: > >> > >> Hi all, > >> > >> I am trying to find off block diagonal entries of a matrix and I am > >> trying to use MatFindOffBlockDiagonalEntries. However, although my > >> matrix is not NULL, I am getting an error message as follows: > >> > >> 1]PETSC ERROR: --------------------- Error Message > >> -------------------------------------------------------------- > >> [1]PETSC ERROR: [2]PETSC ERROR: --------------------- Error Message > >> -------------------------------------------------------------- > >> [2]PETSC ERROR: Null argument, when expecting valid pointer > >> [2]PETSC ERROR: Null Object: Parameter # 1 > >> [2]PETSC ERROR: See > >> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble > >> shooting. > >> [2]PETSC ERROR: Petsc Release Version 3.11.1, Apr, 12, 2019 > >> [2]PETSC ERROR: [3]PETSC ERROR: --------------------- Error Message > >> -------------------------------------------------------------- > >> [3]PETSC ERROR: Null argument, when expecting valid pointer > >> [3]PETSC ERROR: Null Object: Parameter # 1 > >> [3]PETSC ERROR: See > >> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble > >> shooting. > >> [3]PETSC ERROR: Null argument, when expecting valid pointer > >> [1]PETSC ERROR: Null Object: Parameter # 1 > >> [1]PETSC ERROR: See > >> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble > >> shooting. > >> [1]PETSC ERROR: Petsc Release Version 3.11.1, Apr, 12, 2019 > >> [1]PETSC ERROR: > >> ./approx_cut_deneme_clustering_son_final_edgecut_without_parmetis on a > >> arch-linux2-c-debug named b342.wls.metu.edu.tr by edaoktay Fri Jun 19 > >> 18:19:11 2020 > >> [1]PETSC ERROR: Configure options --download-mpich --download-openblas > >> --download-slepc --download-metis --download-parmetis --download-chaco > >> --with-X=1 > >> [1]PETSC ERROR: #1 ISGetSize() line 510 in > >> /home/edaoktay/petsc-3.11.1/src/vec/is/is/interface/index.c > >> [1]PETSC ERROR: > >> ./approx_cut_deneme_clustering_son_final_edgecut_without_parmetis on a > >> arch-linux2-c-debug named b342.wls.metu.edu.tr by edaoktay Fri Jun 19 > >> 18:19:11 2020 > >> [2]PETSC ERROR: Configure options --download-mpich --download-openblas > >> --download-slepc --download-metis --download-parmetis --download-chaco > >> --with-X=1 > >> [2]PETSC ERROR: #1 ISGetSize() line 510 in > >> /home/edaoktay/petsc-3.11.1/src/vec/is/is/interface/index.c > >> [2]PETSC ERROR: #2 MatFindOffBlockDiagonalEntries_MPIAIJ() line 211 in > >> /home/edaoktay/petsc-3.11.1/src/mat/impls/aij/mpi/mpiaij.c > >> [2]PETSC ERROR: #2 MatFindOffBlockDiagonalEntries_MPIAIJ() line 211 in > >> /home/edaoktay/petsc-3.11.1/src/mat/impls/aij/mpi/mpiaij.c > >> [1]PETSC ERROR: #3 MatFindOffBlockDiagonalEntries() line 10502 in > >> /home/edaoktay/petsc-3.11.1/src/mat/interface/matrix.c > >> #3 MatFindOffBlockDiagonalEntries() line 10502 in > >> /home/edaoktay/petsc-3.11.1/src/mat/interface/matrix.c > >> Petsc Release Version 3.11.1, Apr, 12, 2019 > >> [3]PETSC ERROR: > >> ./approx_cut_deneme_clustering_son_final_edgecut_without_parmetis on a > >> arch-linux2-c-debug named b342.wls.metu.edu.tr by edaoktay Fri Jun 19 > >> 18:19:11 2020 > >> [3]PETSC ERROR: Configure options --download-mpich --download-openblas > >> --download-slepc --download-metis --download-parmetis --download-chaco > >> --with-X=1 > >> [3]PETSC ERROR: #1 ISGetSize() line 510 in > >> /home/edaoktay/petsc-3.11.1/src/vec/is/is/interface/index.c > >> [3]PETSC ERROR: #2 MatFindOffBlockDiagonalEntries_MPIAIJ() line 211 in > >> /home/edaoktay/petsc-3.11.1/src/mat/impls/aij/mpi/mpiaij.c > >> [3]PETSC ERROR: #3 MatFindOffBlockDiagonalEntries() line 10502 in > >> /home/edaoktay/petsc-3.11.1/src/mat/interface/matrix.c > >> > >> And the program is still running. > >> > >> The part of my program is : > >> > >> Mat Ais; > >> MatCreateSubMatrix(A,idUi,idUi,MAT_INITIAL_MATRIX,&Ais); > >> ISSetPermutation(idUi); > >> MatPermute(Ais,idUi,idUi,&PL); > >> IS offd; > >> MatFindOffBlockDiagonalEntries(PL,&offd); > >> > >> Since there isn't any examples for this function, I couldn't > >> understand what should I do. Since it says in the error that it is > >> about parameter 1, the matrix, I don't know how to solve the problem. > >> My matrix PL is parallel sparse. I know there shouldn't be any problem > >> in PL since I can view PL by using MatView. > >> > >> Thanks! > >> > >> Eda > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jacob.fai at gmail.com Fri Jun 19 12:53:56 2020 From: jacob.fai at gmail.com (Jacob Faibussowitsch) Date: Fri, 19 Jun 2020 12:53:56 -0500 Subject: [petsc-users] Cohesive Element Support In-Reply-To: References: <7C00A8A6-F7F2-4D62-852B-45ADCD317B98@gmail.com> Message-ID: <2997A784-5B72-4114-A5B8-00EC7A8E4CAA@gmail.com> Hello, Thank you both for your comprehensive replies. Matt: Thanks for the rundown, I will take a look at PyLith and its cohesive element impls. Blaise: The project is still in very early stages as you can tell, so everything is up in the air. Currently we are weighing using a cohesive element approach vs an element extinction approach but we can certainly add a variational model into the mix. We had leaned towards the cohesive element approach as we also need to link Quantum Monte Carlo generated data to the model, which seemed logically the easiest to do with cohesive elements. Are there any examples of your codes online? The only reference I could find online was in this link https://www.math.lsu.edu/~bourdin/defectmechanics/ Best regards, Jacob Faibussowitsch (Jacob Fai - booss - oh - vitch) Cell: (312) 694-3391 > On Jun 19, 2020, at 10:15 AM, Blaise A Bourdin wrote: > > > >> On Jun 18, 2020, at 5:28 AM, Matthew Knepley > wrote: >> >> On Wed, Jun 17, 2020 at 4:05 PM Jacob Faibussowitsch > wrote: >> Hello, >> >> I am looking to perform large scale fracture and crack propagation simulations and have a few questions regarding PETSc support for this. Specifically I am looking for cohesive surface element support with a few twists: >> >> 1. Is there support for zero thickness surface elements? For example modeling virtually flat patches of adhesives holding together two larger structures being pulled apart. >> >> This is how PyLith works: https://github.com/geodynamics/pylith >> >> 2. Is there support for ?joining? two possibly distinct meshes with cohesive surface elements? For example say I have two distinct cylinders representing fibers which would ?touch" to form an X shape. >> >> No, it would have to be coded. >> >> 3. In a similar vein, is there support for a mesh to fracture entirely along a crack formed through the cohesive elements? Imagine the aforementioned X configuration separating entirely into two separate cylinders again. >> >> No, it would have to be coded. >> >> 4. Is there a mechanism by which you can classify existing elements as cohesive elements? >> >> See 1. >> >> 5. Is there an already implemented way of imposing tie-constraints between independent meshes? This would potentially be used to tie high order cohesive cells which would have a non-conforming interface to the ?regular? mesh. >> >> There is nothing for non-conforming interfaces. >> >> From googling I have come across DMPlexCreateHybridMesh(), DMPlexConstructCohesiveCells(), and DMPlexCreateCohesiveSubmesh(). While these do implement cohesive cells these functions don?t at first glance seem to allow one to implement the above. >> >> Having worked with cohesive elements for more than a decade, I would be cautious about a new code using them for fracture. To me, it appears >> that variational fracture codes, like those from Blaise Bourdin and J. J. Marigo's group have much better geometric flexibility, and Maurini's work on >> the solver clears up the hardest part. > > I definitely concur with this and would be happy to help? > > Blaise > > -- > A.K. & Shirley Barton Professor of Mathematics > Adjunct Professor of Mechanical Engineering > Adjunct of the Center for Computation & Technology > Louisiana State University, Lockett Hall Room 344, Baton Rouge, LA 70803, USA > Tel. +1 (225) 578 1612, Fax +1 (225) 578 4276 Web http://www.math.lsu.edu/~bourdin -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Jun 19 16:11:09 2020 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 19 Jun 2020 17:11:09 -0400 Subject: [petsc-users] Implementing periodicity using DMPlex In-Reply-To: References: <4E40F31A-AD04-4F7C-AA82-B43C9DBEA074@gmail.com> Message-ID: On Thu, Jun 18, 2020 at 1:20 PM Shashwat Tiwari wrote: > Thank you for the help. I followed your suggestion and removed the > "DMPlexConstructGhostCells" function and now I am using a gmsh generated > mesh with periodicity in both directions, and distributing the mesh with 1 > level of overlap. I can see that the periodic faces now have two > neighbouring cells on the opposite side of the domain which I can use to > get the periodic cell values using "DMGlobalToLocal" without the need to > construct ghost cells. There are two main issues that I am facing now, > regarding geometric information needed for implementing periodic bc. > > One, I may need the centroid of periodic counterpart of a cell which is > possible to get from a "ghost" cell created on the periodic face, and > containing the solution data from periodic counterpart. But now, I would > need to shift the coordinates of the neighbouring cell accordingly. If you > see the sample output given below along with the screenshot of the sample > 2D mesh that I have used (please find attached), while reconstructing the > left and right solution values on face 32 for a second order scheme, I need > the centroid of a ghost cell on left side of the face which is at (x, y) = > (-1.25, 8.75). I wanted to know if there is some way to get these shifted > "ghost" coordinates directly from the geometric information obtained using > "PetscFV" class. Kindly let me know if you have some suggestions regarding > this. > I do not completely understand the question. However, I think you want something like https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DM/DMLocalizeCoordinate.html There are many variants of this functionality. > Sample Output: > > face: 32, centroid = 0.000000, 8.750000, normal = 2.500000, -0.000000 > support[0] = 0, cent = 1.250000, 8.750000, area = 6.250000 > support[1] = 3, cent = 8.750000, 8.750000, area = 6.250000 > > face: 33, centroid = 1.250000, 7.500000, normal = -0.000000, -2.500000 > support[0] = 0, cent = 1.250000, 8.750000, area = 6.250000 > support[1] = 4, cent = 1.250000, 6.250000, area = 6.250000 > > Second, the face normals of the periodic faces obtained using the face > geometry data from "DMPlexGetDataFVM" function seem to be in the wrong > direction i.e., from the right cell "support[1]" to the left cell > "support[0]", while the other faces have the opposite direction. For face: > 32, left cell is 0 and the right cell is 3, but, the normal points the > opposite way (cell 3 would be on the left side of cell 0 on periodic > boundary), while, for face: 33 which is an interior face, the normal > direction is from left to right cell. > This should be easy to check. The strange thing is that if this were broken, you would expect other examples to fail. > Also, I am getting incorrect normal values for some of the faces > (including some interior faces). Here is a sample output of the same: > > face: 59, centroid = 3.750000, 2.500000, normal = 0.000000, -7.500000 > support[0] = 11, cent = 8.750000, 3.750000, area = 6.250000 > support[1] = 15, cent = 8.750000, 1.250000, area = 6.250000 > > this face has the correct direction (from 11 to 15), but the value is > wrong (all the faces are 2.5 units). This problem does not occur when I > ignore periodicity by setting -dm_plex_gmsh_periodic to 0. > > These issues occur regardless of the number of processes. I am attaching > the test code that I am using along with the gmsh file for your reference. > Please suggest to me, what might be the issue here. > I will take a look. Thanks, Matt > Regards, > Shashwat > > > On Mon, Jun 15, 2020 at 2:54 PM Stefano Zampini > wrote: > >> It is enough if you use DMPlexDistribute with 1 level of overlap, set the >> local section for your dofs, and call DMGlobalToLocal . The local vector >> will contain data for what you call ?ghost? cells. I call them ?not-owned? >> cells. >> >> On Jun 15, 2020, at 12:09 PM, Shashwat Tiwari >> wrote: >> >> The way I'm trying to implement periodic bc is, when I loop over the >> boundary faces, say, on the left boundary of the domain to compute flux and >> residual, I need solution values from the two cells neighbouring the face, >> i.e. the left cell and the right cell (the face normal pointing from the >> left cell to the right cell and the right cell being a boundary ghost cell >> for boundary faces). For the boundary to be periodic, I need the value that >> I get from the right cell (boundary ghost cell) to be the solution value at >> its periodic counterpart, i.e. the solution value at left cell of the face >> on right boundary of the domain in this case. My question is how do I >> update the value at a boundary ghost cell with the value at the real cell >> which is its periodic counterpart from the other side of the domain. Is >> there some kind of mapping of the boundary ghost cells to their >> corresponding real cells which I can use to update the solution values at >> ghost cells? >> >> Regards, >> Shashwat >> >> On Sun, Jun 14, 2020 at 5:11 AM Matthew Knepley >> wrote: >> >>> On Fri, Jun 12, 2020 at 3:19 PM Shashwat Tiwari >>> wrote: >>> >>>> Hi, >>>> I am writing a first order 2D solver for unstructured grids with >>>> periodic boundaries using DMPlex. After generating the mesh, I use >>>> "DMSetPeriodicity" function to set periodicity in both directions. After >>>> which I partition the mesh (DMPlexDistribute), construct ghost cells >>>> (DMPlexConstructGhostCells), >>>> >>> >>> These ghost cells are for FVM boundary conditions. If you want cells to >>> be shared across parallel partitions, then you want to give overlap=1 >>> to DMPlexDIstribute(). Is that what you want? >>> >>> Thanks, >>> >>> Matt >>> >>> >>>> create a section, and set some initial values in the global vector. >>>> Then I use "VecGhostUpdateBegin" to start updating the boundary ghost cell >>>> values, but, I get the following error in case I use multiple processors: >>>> >>>> [0]PETSC ERROR: Invalid argument >>>> [0]PETSC ERROR: Vector is not ghosted >>>> [0]PETSC ERROR: See >>>> https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble >>>> shooting. >>>> >>>> if I run with a single process, there is no error but the values remain >>>> empty (zero) and are not updated. Kindly let me know, if I am missing some >>>> crucial step before I can update the ghost values in order to implement the >>>> periodic bc, or if there is any other approach to achieve it. I am >>>> attaching a small code to demonstrate the issue for your reference. >>>> >>>> Regards, >>>> Shashwat >>>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> https://www.cse.buffalo.edu/~knepley/ >>> >>> >> >> -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ashish.patel at onscale.com Fri Jun 19 17:15:03 2020 From: ashish.patel at onscale.com (Ashish Patel) Date: Fri, 19 Jun 2020 15:15:03 -0700 Subject: [petsc-users] SNES Line search: fixing the non zero solution values of dirichlet nodes during line search Message-ID: Dear PETSc users, We use PETSc as part of a finite element method program and we are trying to properly implement Dirichlet boundary conditions for non-linear, transient problems. We find that when we use a line search method it also changes the non-zero solution value of Dirichlet nodes as it steps through the line search iteration. I was wondering if there is a way to freeze some index sets of a vector from changing during the line search operation? We are using the TS framework to setup the problem and use 'MatZeroRowsColumns' to set the diagonal of the jacobian to 1 for the dirichlet nodes and set the RHS as increment of (Dirichlet_value-solution_value). This works when the line search method is turned off by using '-snes_linesearch_type basic' however using the default 'bt' linesearch, the TS diverges with error shown below. In a separate implementation if we overwrote the dirichlet nodes of the solution vector in TS residual function with the Dirichlet values then the 'bt' line search method converged to the right solution. However we would like to avoid modifying the internal PETSc vector in our implementation. 0 TS dt 1. time 0. 0 SNES Function norm 2.378549386020e+03 Line search: gnorm after quadratic fit 4.369235425165e+03 Line search: Cubic step no good, shrinking lambda, current gnorm 2.385369069060e+03 lambda=1.0000000000000002e-02 Line search: Cubically determined step, current gnorm 2.373925008934e+03 lambda=3.8846250444606093e-03 1 SNES Function norm 2.373925008934e+03 Line search: gnorm after quadratic fit 5.006914179995e+03 Line search: Cubic step no good, shrinking lambda, current gnorm 2.420957096780e+03 lambda=1.0000000000000002e-02 Line search: Cubic step no good, shrinking lambda, current gnorm 2.376034946750e+03 lambda=1.6129422079664700e-03 Line search: Cubic step no good, shrinking lambda, current gnorm 2.374313344729e+03 lambda=4.8465026740690043e-04 Line search: Cubic step no good, shrinking lambda, current gnorm 2.373999473242e+03 lambda=1.5857251532828948e-04 Line search: Cubically determined step, current gnorm 2.373921668024e+03 lambda=5.8116507162753387e-05 2 SNES Function norm 2.373921668024e+03 Line search: gnorm after quadratic fit 4.771035112853e+03 Line search: Cubic step no good, shrinking lambda, current gnorm 2.410650718394e+03 lambda=1.0000000000000002e-02 Line search: Cubic step no good, shrinking lambda, current gnorm 2.375104094198e+03 lambda=1.8983783738011522e-03 Line search: Cubic step no good, shrinking lambda, current gnorm 2.374049151562e+03 lambda=7.0688528086485479e-04 Line search: Cubic step no good, shrinking lambda, current gnorm 2.373935090907e+03 lambda=2.9132722794896019e-04 Line search: Cubically determined step, current gnorm 2.373921032081e+03 lambda=1.2527602265373028e-04 3 SNES Function norm 2.373921032081e+03 Line search: gnorm after quadratic fit 5.117914832660e+03 Line search: Cubic step no good, shrinking lambda, current gnorm 2.422635362094e+03 lambda=1.0000000000000002e-02 Line search: Cubic step no good, shrinking lambda, current gnorm 2.375923870970e+03 lambda=1.5769887508300913e-03 Line search: Cubic step no good, shrinking lambda, current gnorm 2.374287081592e+03 lambda=4.8018017100729705e-04 Line search: Cubic step no good, shrinking lambda, current gnorm 2.373999131160e+03 lambda=1.5908966655977892e-04 Line search: Cubic step no good, shrinking lambda, current gnorm 2.373930608274e+03 lambda=5.7603977147935371e-05 Line search: Cubic step no good, shrinking lambda, current gnorm 2.373922022038e+03 lambda=2.3884787050507805e-05 Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921396636e+03 lambda=1.0282405393471519e-05 Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921173719e+03 lambda=4.3868704034012554e-06 Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921091304e+03 lambda=1.8774991151727107e-06 Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921057168e+03 lambda=8.0331628193813397e-07 Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921042769e+03 lambda=3.4377811174560817e-07 Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921036645e+03 lambda=1.4712421886880395e-07 Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921034032e+03 lambda=6.2965363212312132e-08 Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921032916e+03 lambda=2.6947718250469261e-08 Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921032438e+03 lambda=1.1533043989179318e-08 Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921032233e+03 lambda=4.9359018284334258e-09 Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921032146e+03 lambda=2.1124641857409436e-09 Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921032109e+03 lambda=9.0409137304143975e-10 Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921032093e+03 lambda=3.8693264359674819e-10 Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921032086e+03 lambda=1.6559911204766632e-10 Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921032083e+03 lambda=7.0873187020534353e-11 Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921032081e+03 lambda=3.0332192901757729e-11 Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921032081e+03 lambda=1.2981600435512875e-11 Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921032081e+03 lambda=5.5559212521628090e-12 Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921032081e+03 lambda=2.3777380405336958e-12 Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921032081e+03 lambda=1.0176091649134191e-12 Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921032081e+03 lambda=4.3555419789721080e-13 Line search: unable to find good step length! After 27 tries Line search: fnorm=2.3739210320805191e+03, gnorm=2.3739210320805323e+03, ynorm=8.5698020038772756e+03, minlambda=9.9999999999999998e-13, lambda=4.3555419789721080e-13, initial slope=-5.6355010665542409e+06 SNES Object: 1 MPI processes type: newtonls maximum iterations=50, maximum function evaluations=10000 tolerances: relative=1e-08, absolute=1e-50, solution=1e-08 total number of linear solver iterations=4 total number of function evaluations=48 norm schedule ALWAYS SNESLineSearch Object: 1 MPI processes type: bt interpolation: cubic alpha=1.000000e-04 maxstep=1.000000e+08, minlambda=1.000000e-12 tolerances: relative=1.000000e-08, absolute=1.000000e-15, lambda=1.000000e-08 maximum iterations=40 KSP Object: 1 MPI processes type: preonly maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000. left preconditioning using NONE norm type for convergence test PC Object: 1 MPI processes type: cholesky out-of-place factorization tolerance for zero pivot 2.22045e-14 matrix ordering: natural factor fill ratio given 0., needed 0. Factored matrix follows: Mat Object: 1 MPI processes type: mumps rows=38154, cols=38154 package used to perform factorization: mumps total: nonzeros=7080060, allocated nonzeros=7080060 total number of mallocs used during MatSetValues calls=0 MUMPS run parameters: SYM (matrix type): 2 PAR (host participation): 1 ICNTL(1) (output for error): 6 ICNTL(2) (output of diagnostic msg): 0 ICNTL(3) (output for global info): 0 ICNTL(4) (level of printing): 0 ICNTL(5) (input mat struct): 0 ICNTL(6) (matrix prescaling): 7 ICNTL(7) (sequential matrix ordering):7 ICNTL(8) (scaling strategy): 77 ICNTL(10) (max num of refinements): 0 ICNTL(11) (error analysis): 0 ICNTL(12) (efficiency control): 0 ICNTL(13) (efficiency control): 0 ICNTL(14) (percentage of estimated workspace increase): 20 ICNTL(18) (input mat struct): 0 ICNTL(19) (Schur complement info): 0 ICNTL(20) (rhs sparse pattern): 0 ICNTL(21) (solution struct): 0 ICNTL(22) (in-core/out-of-core facility): 0 ICNTL(23) (max size of memory can be allocated locally):0 ICNTL(24) (detection of null pivot rows): 0 ICNTL(25) (computation of a null space basis): 0 ICNTL(26) (Schur options for rhs or solution): 0 ICNTL(27) (experimental parameter): -32 ICNTL(28) (use parallel or sequential ordering): 1 ICNTL(29) (parallel ordering): 0 ICNTL(30) (user-specified set of entries in inv(A)): 0 ICNTL(31) (factors is discarded in the solve phase): 0 ICNTL(33) (compute determinant): 0 ICNTL(35) (activate BLR based factorization): 0 ICNTL(36) (choice of BLR factorization variant): 0 ICNTL(38) (estimated compression rate of LU factors): 333 CNTL(1) (relative pivoting threshold): 0.01 CNTL(2) (stopping criterion of refinement): 1.49012e-08 CNTL(3) (absolute pivoting threshold): 0. CNTL(4) (value of static pivoting): -1. CNTL(5) (fixation for null pivots): 0. CNTL(7) (dropping parameter for BLR): 0. RINFO(1) (local estimated flops for the elimination after analysis): [0] 2.73979e+09 RINFO(2) (local estimated flops for the assembly after factorization): [0] 1.08826e+07 RINFO(3) (local estimated flops for the elimination after factorization): [0] 2.73979e+09 INFO(15) (estimated size of (in MB) MUMPS internal data for running numerical factorization): [0] 94 INFO(16) (size of (in MB) MUMPS internal data used during numerical factorization): [0] 94 INFO(23) (num of pivots eliminated on this processor after factorization): [0] 38154 RINFOG(1) (global estimated flops for the elimination after analysis): 2.73979e+09 RINFOG(2) (global estimated flops for the assembly after factorization): 1.08826e+07 RINFOG(3) (global estimated flops for the elimination after factorization): 2.73979e+09 (RINFOG(12) RINFOG(13))*2^INFOG(34) (determinant): (0.,0.)*(2^0) INFOG(3) (estimated real workspace for factors on all processors after analysis): 8377336 INFOG(4) (estimated integer workspace for factors on all processors after analysis): 447902 INFOG(5) (estimated maximum front size in the complete tree): 990 INFOG(6) (number of nodes in the complete tree): 2730 INFOG(7) (ordering option effectively use after analysis): 5 INFOG(8) (structural symmetry in percent of the permuted matrix after analysis): 100 INFOG(9) (total real/complex workspace to store the matrix factors after factorization): 8377336 INFOG(10) (total integer space store the matrix factors after factorization): 447902 INFOG(11) (order of largest frontal matrix after factorization): 990 INFOG(12) (number of off-diagonal pivots): 10 INFOG(13) (number of delayed pivots after factorization): 0 INFOG(14) (number of memory compress after factorization): 0 INFOG(15) (number of steps of iterative refinement after solution): 0 INFOG(16) (estimated size (in MB) of all MUMPS internal data for factorization after analysis: value on the most memory consuming processor): 94 INFOG(17) (estimated size of all MUMPS internal data for factorization after analysis: sum over all processors): 94 INFOG(18) (size of all MUMPS internal data allocated during factorization: value on the most memory consuming processor): 94 INFOG(19) (size of all MUMPS internal data allocated during factorization: sum over all processors): 94 INFOG(20) (estimated number of entries in the factors): 7080060 INFOG(21) (size in MB of memory effectively used during factorization - value on the most memory consuming processor): 80 INFOG(22) (size in MB of memory effectively used during factorization - sum over all processors): 80 INFOG(23) (after analysis: value of ICNTL(6) effectively used): 0 INFOG(24) (after analysis: value of ICNTL(12) effectively used): 1 INFOG(25) (after factorization: number of pivots modified by static pivoting): 0 INFOG(28) (after factorization: number of null pivots encountered): 0 INFOG(29) (after factorization: effective number of entries in the factors (sum over all processors)): 7080060 INFOG(30, 31) (after solution: size in Mbytes of memory used during solution phase): 92, 92 INFOG(32) (after analysis: type of analysis done): 1 INFOG(33) (value used for ICNTL(8)): 7 INFOG(34) (exponent of the determinant if determinant is requested): 0 INFOG(35) (after factorization: number of entries taking into account BLR factor compression - sum over all processors): 7080060 INFOG(36) (after analysis: estimated size of all MUMPS internal data for running BLR in-core - value on the most memory consuming processor): 0 INFOG(37) (after analysis: estimated size of all MUMPS internal data for running BLR in-core - sum over all processors): 0 INFOG(38) (after analysis: estimated size of all MUMPS internal data for running BLR out-of-core - value on the most memory consuming processor): 0 INFOG(39) (after analysis: estimated size of all MUMPS internal data for running BLR out-of-core - sum over all processors): 0 linear system matrix = precond matrix: Mat Object: 1 MPI processes type: seqaij rows=38154, cols=38154 total: nonzeros=973446, allocated nonzeros=973446 total number of mallocs used during MatSetValues calls=0 not using I-node routines [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: [0]PETSC ERROR: TSStep has failed due to DIVERGED_NONLINEAR_SOLVE, increase -ts_max_snes_failures or make negative to attempt recovery [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.12.3, Jan, 03, 2020 Thanks, Ashish Scientific Computing Division OnScale CA, USA -------------- next part -------------- An HTML attachment was scrubbed... URL: From sam.guo at cd-adapco.com Sat Jun 20 00:32:05 2020 From: sam.guo at cd-adapco.com (Sam Guo) Date: Fri, 19 Jun 2020 22:32:05 -0700 Subject: [petsc-users] PETSc initialization error Message-ID: Dear PETSc team, When I called SLEPc multiple time, I eventually got following error: MPI operation not supported by PETSc's sequential MPI wrappers [0]PETSC ERROR: #1 PetscInitialize() line 967 in ../../../petsc/src/sys/objects/pinit.c [0]PETSC ERROR: #2 SlepcInitialize() line 262 in ../../../slepc/src/sys/slepcinit.c [0]PETSC ERROR: #3 SlepcInitializeNoPointers() line 359 in ../../../slepc/src/sys/slepcinit.c PETSC ERROR: Logging has not been enabled. You might have forgotten to call PetscInitialize(). I debugged: it is because of following in petsc/src/sys/mpiuni/mpi.c if (num_attr >= MAX_ATTR) in function int MPI_Comm_create_keyval(MPI_Copy_function *copy_fn,MPI_Delete_function *delete_fn,int *keyval,void *extra_state) num_attr is declared static and keeps increasing every time MPI_Comm_create_keyval is called. I am using petsc 3.11.3 but found 3.13.2 has the same logic. Is this a bug or I didn't use it correctly? Thanks, Sam -------------- next part -------------- An HTML attachment was scrubbed... URL: From sam.guo at cd-adapco.com Sat Jun 20 00:46:55 2020 From: sam.guo at cd-adapco.com (Sam Guo) Date: Fri, 19 Jun 2020 22:46:55 -0700 Subject: [petsc-users] PETSc initialization error In-Reply-To: References: Message-ID: To clarify, I call PETSc initialize and PETSc finalize everytime I call SLEPc: PetscInitializeNoPointers(argc,args,nullptr,nullptr); SlepcInitialize(&argc,&args,static_cast(nullptr),help); //calling slepc SlepcFinalize(); PetscFinalize(); On Fri, Jun 19, 2020 at 10:32 PM Sam Guo wrote: > Dear PETSc team, > When I called SLEPc multiple time, I eventually got following error: > > MPI operation not supported by PETSc's sequential MPI wrappers > [0]PETSC ERROR: #1 PetscInitialize() line 967 in > ../../../petsc/src/sys/objects/pinit.c > [0]PETSC ERROR: #2 SlepcInitialize() line 262 in > ../../../slepc/src/sys/slepcinit.c > [0]PETSC ERROR: #3 SlepcInitializeNoPointers() line 359 in > ../../../slepc/src/sys/slepcinit.c > PETSC ERROR: Logging has not been enabled. > You might have forgotten to call PetscInitialize(). > > I debugged: it is because of following in petsc/src/sys/mpiuni/mpi.c > > if (num_attr >= MAX_ATTR) > > in function int MPI_Comm_create_keyval(MPI_Copy_function > *copy_fn,MPI_Delete_function *delete_fn,int *keyval,void *extra_state) > > num_attr is declared static and keeps increasing every > time MPI_Comm_create_keyval is called. > > I am using petsc 3.11.3 but found 3.13.2 has the same logic. > > Is this a bug or I didn't use it correctly? > > Thanks, > Sam > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefano.zampini at gmail.com Sat Jun 20 06:40:41 2020 From: stefano.zampini at gmail.com (Stefano Zampini) Date: Sat, 20 Jun 2020 14:40:41 +0300 Subject: [petsc-users] SNES Line search: fixing the non zero solution values of dirichlet nodes during line search In-Reply-To: References: Message-ID: I think you want to use SNESLineSearchSetPostCheck, see https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/SNES/SNESLineSearchSetPostCheck.html Il giorno sab 20 giu 2020 alle ore 01:16 Ashish Patel < ashish.patel at onscale.com> ha scritto: > Dear PETSc users, > > We use PETSc as part of a finite element method program and we are trying > to properly implement Dirichlet boundary conditions for non-linear, > transient problems. We find that when we use a line search method it also > changes the non-zero solution value of Dirichlet nodes as it steps through > the line search iteration. I was wondering if there is a way to freeze > some index sets of a vector from changing during the line search operation? > > We are using the TS framework to setup the problem and use > 'MatZeroRowsColumns' to set the diagonal of the jacobian to 1 for the > dirichlet nodes and set the RHS as increment of > (Dirichlet_value-solution_value). This works when the line search method is > turned off by using '-snes_linesearch_type basic' however using the default > 'bt' linesearch, the TS diverges with error shown below. In a separate > implementation if we overwrote the dirichlet nodes of the solution vector > in TS residual function with the Dirichlet values then the 'bt' line > search method converged to the right solution. However we would like to > avoid modifying the internal PETSc vector in our implementation. > > 0 TS dt 1. time 0. > 0 SNES Function norm 2.378549386020e+03 > Line search: gnorm after quadratic fit 4.369235425165e+03 > Line search: Cubic step no good, shrinking lambda, current gnorm > 2.385369069060e+03 lambda=1.0000000000000002e-02 > Line search: Cubically determined step, current gnorm > 2.373925008934e+03 lambda=3.8846250444606093e-03 > 1 SNES Function norm 2.373925008934e+03 > Line search: gnorm after quadratic fit 5.006914179995e+03 > Line search: Cubic step no good, shrinking lambda, current gnorm > 2.420957096780e+03 lambda=1.0000000000000002e-02 > Line search: Cubic step no good, shrinking lambda, current gnorm > 2.376034946750e+03 lambda=1.6129422079664700e-03 > Line search: Cubic step no good, shrinking lambda, current gnorm > 2.374313344729e+03 lambda=4.8465026740690043e-04 > Line search: Cubic step no good, shrinking lambda, current gnorm > 2.373999473242e+03 lambda=1.5857251532828948e-04 > Line search: Cubically determined step, current gnorm > 2.373921668024e+03 lambda=5.8116507162753387e-05 > 2 SNES Function norm 2.373921668024e+03 > Line search: gnorm after quadratic fit 4.771035112853e+03 > Line search: Cubic step no good, shrinking lambda, current gnorm > 2.410650718394e+03 lambda=1.0000000000000002e-02 > Line search: Cubic step no good, shrinking lambda, current gnorm > 2.375104094198e+03 lambda=1.8983783738011522e-03 > Line search: Cubic step no good, shrinking lambda, current gnorm > 2.374049151562e+03 lambda=7.0688528086485479e-04 > Line search: Cubic step no good, shrinking lambda, current gnorm > 2.373935090907e+03 lambda=2.9132722794896019e-04 > Line search: Cubically determined step, current gnorm > 2.373921032081e+03 lambda=1.2527602265373028e-04 > 3 SNES Function norm 2.373921032081e+03 > Line search: gnorm after quadratic fit 5.117914832660e+03 > Line search: Cubic step no good, shrinking lambda, current gnorm > 2.422635362094e+03 lambda=1.0000000000000002e-02 > Line search: Cubic step no good, shrinking lambda, current gnorm > 2.375923870970e+03 lambda=1.5769887508300913e-03 > Line search: Cubic step no good, shrinking lambda, current gnorm > 2.374287081592e+03 lambda=4.8018017100729705e-04 > Line search: Cubic step no good, shrinking lambda, current gnorm > 2.373999131160e+03 lambda=1.5908966655977892e-04 > Line search: Cubic step no good, shrinking lambda, current gnorm > 2.373930608274e+03 lambda=5.7603977147935371e-05 > Line search: Cubic step no good, shrinking lambda, current gnorm > 2.373922022038e+03 lambda=2.3884787050507805e-05 > Line search: Cubic step no good, shrinking lambda, current gnorm > 2.373921396636e+03 lambda=1.0282405393471519e-05 > Line search: Cubic step no good, shrinking lambda, current gnorm > 2.373921173719e+03 lambda=4.3868704034012554e-06 > Line search: Cubic step no good, shrinking lambda, current gnorm > 2.373921091304e+03 lambda=1.8774991151727107e-06 > Line search: Cubic step no good, shrinking lambda, current gnorm > 2.373921057168e+03 lambda=8.0331628193813397e-07 > Line search: Cubic step no good, shrinking lambda, current gnorm > 2.373921042769e+03 lambda=3.4377811174560817e-07 > Line search: Cubic step no good, shrinking lambda, current gnorm > 2.373921036645e+03 lambda=1.4712421886880395e-07 > Line search: Cubic step no good, shrinking lambda, current gnorm > 2.373921034032e+03 lambda=6.2965363212312132e-08 > Line search: Cubic step no good, shrinking lambda, current gnorm > 2.373921032916e+03 lambda=2.6947718250469261e-08 > Line search: Cubic step no good, shrinking lambda, current gnorm > 2.373921032438e+03 lambda=1.1533043989179318e-08 > Line search: Cubic step no good, shrinking lambda, current gnorm > 2.373921032233e+03 lambda=4.9359018284334258e-09 > Line search: Cubic step no good, shrinking lambda, current gnorm > 2.373921032146e+03 lambda=2.1124641857409436e-09 > Line search: Cubic step no good, shrinking lambda, current gnorm > 2.373921032109e+03 lambda=9.0409137304143975e-10 > Line search: Cubic step no good, shrinking lambda, current gnorm > 2.373921032093e+03 lambda=3.8693264359674819e-10 > Line search: Cubic step no good, shrinking lambda, current gnorm > 2.373921032086e+03 lambda=1.6559911204766632e-10 > Line search: Cubic step no good, shrinking lambda, current gnorm > 2.373921032083e+03 lambda=7.0873187020534353e-11 > Line search: Cubic step no good, shrinking lambda, current gnorm > 2.373921032081e+03 lambda=3.0332192901757729e-11 > Line search: Cubic step no good, shrinking lambda, current gnorm > 2.373921032081e+03 lambda=1.2981600435512875e-11 > Line search: Cubic step no good, shrinking lambda, current gnorm > 2.373921032081e+03 lambda=5.5559212521628090e-12 > Line search: Cubic step no good, shrinking lambda, current gnorm > 2.373921032081e+03 lambda=2.3777380405336958e-12 > Line search: Cubic step no good, shrinking lambda, current gnorm > 2.373921032081e+03 lambda=1.0176091649134191e-12 > Line search: Cubic step no good, shrinking lambda, current gnorm > 2.373921032081e+03 lambda=4.3555419789721080e-13 > Line search: unable to find good step length! After 27 tries > Line search: fnorm=2.3739210320805191e+03, > gnorm=2.3739210320805323e+03, ynorm=8.5698020038772756e+03, > minlambda=9.9999999999999998e-13, lambda=4.3555419789721080e-13, initial > slope=-5.6355010665542409e+06 > SNES Object: 1 MPI processes > type: newtonls > maximum iterations=50, maximum function evaluations=10000 > tolerances: relative=1e-08, absolute=1e-50, solution=1e-08 > total number of linear solver iterations=4 > total number of function evaluations=48 > norm schedule ALWAYS > SNESLineSearch Object: 1 MPI processes > type: bt > interpolation: cubic > alpha=1.000000e-04 > maxstep=1.000000e+08, minlambda=1.000000e-12 > tolerances: relative=1.000000e-08, absolute=1.000000e-15, > lambda=1.000000e-08 > maximum iterations=40 > KSP Object: 1 MPI processes > type: preonly > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > left preconditioning > using NONE norm type for convergence test > PC Object: 1 MPI processes > type: cholesky > out-of-place factorization > tolerance for zero pivot 2.22045e-14 > matrix ordering: natural > factor fill ratio given 0., needed 0. > Factored matrix follows: > Mat Object: 1 MPI processes > type: mumps > rows=38154, cols=38154 > package used to perform factorization: mumps > total: nonzeros=7080060, allocated nonzeros=7080060 > total number of mallocs used during MatSetValues calls=0 > MUMPS run parameters: > SYM (matrix type): 2 > PAR (host participation): 1 > ICNTL(1) (output for error): 6 > ICNTL(2) (output of diagnostic msg): 0 > ICNTL(3) (output for global info): 0 > ICNTL(4) (level of printing): 0 > ICNTL(5) (input mat struct): 0 > ICNTL(6) (matrix prescaling): 7 > ICNTL(7) (sequential matrix ordering):7 > ICNTL(8) (scaling strategy): 77 > ICNTL(10) (max num of refinements): 0 > ICNTL(11) (error analysis): 0 > ICNTL(12) (efficiency control): 0 > ICNTL(13) (efficiency control): 0 > ICNTL(14) (percentage of estimated workspace increase): 20 > ICNTL(18) (input mat struct): 0 > ICNTL(19) (Schur complement info): 0 > ICNTL(20) (rhs sparse pattern): 0 > ICNTL(21) (solution struct): 0 > ICNTL(22) (in-core/out-of-core facility): 0 > ICNTL(23) (max size of memory can be allocated locally):0 > ICNTL(24) (detection of null pivot rows): 0 > ICNTL(25) (computation of a null space basis): 0 > ICNTL(26) (Schur options for rhs or solution): 0 > ICNTL(27) (experimental parameter): > -32 > ICNTL(28) (use parallel or sequential ordering): 1 > ICNTL(29) (parallel ordering): 0 > ICNTL(30) (user-specified set of entries in inv(A)): 0 > ICNTL(31) (factors is discarded in the solve phase): 0 > ICNTL(33) (compute determinant): 0 > ICNTL(35) (activate BLR based factorization): 0 > ICNTL(36) (choice of BLR factorization variant): 0 > ICNTL(38) (estimated compression rate of LU factors): > 333 > CNTL(1) (relative pivoting threshold): 0.01 > CNTL(2) (stopping criterion of refinement): 1.49012e-08 > CNTL(3) (absolute pivoting threshold): 0. > CNTL(4) (value of static pivoting): -1. > CNTL(5) (fixation for null pivots): 0. > CNTL(7) (dropping parameter for BLR): 0. > RINFO(1) (local estimated flops for the elimination after > analysis): > [0] 2.73979e+09 > RINFO(2) (local estimated flops for the assembly after > factorization): > [0] 1.08826e+07 > RINFO(3) (local estimated flops for the elimination after > factorization): > [0] 2.73979e+09 > INFO(15) (estimated size of (in MB) MUMPS internal data > for running numerical factorization): > [0] 94 > INFO(16) (size of (in MB) MUMPS internal data used during > numerical factorization): > [0] 94 > INFO(23) (num of pivots eliminated on this processor after > factorization): > [0] 38154 > RINFOG(1) (global estimated flops for the elimination > after analysis): 2.73979e+09 > RINFOG(2) (global estimated flops for the assembly after > factorization): 1.08826e+07 > RINFOG(3) (global estimated flops for the elimination > after factorization): 2.73979e+09 > (RINFOG(12) RINFOG(13))*2^INFOG(34) (determinant): > (0.,0.)*(2^0) > INFOG(3) (estimated real workspace for factors on all > processors after analysis): 8377336 > INFOG(4) (estimated integer workspace for factors on all > processors after analysis): 447902 > INFOG(5) (estimated maximum front size in the complete > tree): 990 > INFOG(6) (number of nodes in the complete tree): 2730 > INFOG(7) (ordering option effectively use after analysis): > 5 > INFOG(8) (structural symmetry in percent of the permuted > matrix after analysis): 100 > INFOG(9) (total real/complex workspace to store the matrix > factors after factorization): 8377336 > INFOG(10) (total integer space store the matrix factors > after factorization): 447902 > INFOG(11) (order of largest frontal matrix after > factorization): 990 > INFOG(12) (number of off-diagonal pivots): 10 > INFOG(13) (number of delayed pivots after factorization): > 0 > INFOG(14) (number of memory compress after factorization): > 0 > INFOG(15) (number of steps of iterative refinement after > solution): 0 > INFOG(16) (estimated size (in MB) of all MUMPS internal > data for factorization after analysis: value on the most memory consuming > processor): 94 > INFOG(17) (estimated size of all MUMPS internal data for > factorization after analysis: sum over all processors): 94 > INFOG(18) (size of all MUMPS internal data allocated > during factorization: value on the most memory consuming processor): 94 > INFOG(19) (size of all MUMPS internal data allocated > during factorization: sum over all processors): 94 > INFOG(20) (estimated number of entries in the factors): > 7080060 > INFOG(21) (size in MB of memory effectively used during > factorization - value on the most memory consuming processor): 80 > INFOG(22) (size in MB of memory effectively used during > factorization - sum over all processors): 80 > INFOG(23) (after analysis: value of ICNTL(6) effectively > used): 0 > INFOG(24) (after analysis: value of ICNTL(12) effectively > used): 1 > INFOG(25) (after factorization: number of pivots modified > by static pivoting): 0 > INFOG(28) (after factorization: number of null pivots > encountered): 0 > INFOG(29) (after factorization: effective number of > entries in the factors (sum over all processors)): 7080060 > INFOG(30, 31) (after solution: size in Mbytes of memory > used during solution phase): 92, 92 > INFOG(32) (after analysis: type of analysis done): 1 > INFOG(33) (value used for ICNTL(8)): 7 > INFOG(34) (exponent of the determinant if determinant is > requested): 0 > INFOG(35) (after factorization: number of entries taking > into account BLR factor compression - sum over all processors): 7080060 > INFOG(36) (after analysis: estimated size of all MUMPS > internal data for running BLR in-core - value on the most memory consuming > processor): 0 > INFOG(37) (after analysis: estimated size of all MUMPS > internal data for running BLR in-core - sum over all processors): 0 > INFOG(38) (after analysis: estimated size of all MUMPS > internal data for running BLR out-of-core - value on the most memory > consuming processor): 0 > INFOG(39) (after analysis: estimated size of all MUMPS > internal data for running BLR out-of-core - sum over all processors): 0 > linear system matrix = precond matrix: > Mat Object: 1 MPI processes > type: seqaij > rows=38154, cols=38154 > total: nonzeros=973446, allocated nonzeros=973446 > total number of mallocs used during MatSetValues calls=0 > not using I-node routines > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: > [0]PETSC ERROR: TSStep has failed due to DIVERGED_NONLINEAR_SOLVE, > increase -ts_max_snes_failures or make negative to attempt recovery > [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.12.3, Jan, 03, 2020 > > Thanks, > Ashish > Scientific Computing Division > OnScale > CA, USA > -- Stefano -------------- next part -------------- An HTML attachment was scrubbed... URL: From junchao.zhang at gmail.com Sat Jun 20 08:53:38 2020 From: junchao.zhang at gmail.com (Junchao Zhang) Date: Sat, 20 Jun 2020 08:53:38 -0500 Subject: [petsc-users] PETSc initialization error In-Reply-To: References: Message-ID: The counter num_attr should be recycled. But first try to call PETSc initialize/Finalize only once to see it fixes the error. --Junchao Zhang On Sat, Jun 20, 2020 at 12:48 AM Sam Guo wrote: > To clarify, I call PETSc initialize and PETSc finalize everytime I call > SLEPc: > > PetscInitializeNoPointers(argc,args,nullptr,nullptr); > > SlepcInitialize(&argc,&args,static_cast(nullptr),help); > > //calling slepc > > SlepcFinalize(); > > PetscFinalize(); > > > > On Fri, Jun 19, 2020 at 10:32 PM Sam Guo wrote: > >> Dear PETSc team, >> When I called SLEPc multiple time, I eventually got following error: >> >> MPI operation not supported by PETSc's sequential MPI wrappers >> [0]PETSC ERROR: #1 PetscInitialize() line 967 in >> ../../../petsc/src/sys/objects/pinit.c >> [0]PETSC ERROR: #2 SlepcInitialize() line 262 in >> ../../../slepc/src/sys/slepcinit.c >> [0]PETSC ERROR: #3 SlepcInitializeNoPointers() line 359 in >> ../../../slepc/src/sys/slepcinit.c >> PETSC ERROR: Logging has not been enabled. >> You might have forgotten to call PetscInitialize(). >> >> I debugged: it is because of following in petsc/src/sys/mpiuni/mpi.c >> >> if (num_attr >= MAX_ATTR) >> >> in function int MPI_Comm_create_keyval(MPI_Copy_function >> *copy_fn,MPI_Delete_function *delete_fn,int *keyval,void *extra_state) >> >> num_attr is declared static and keeps increasing every >> time MPI_Comm_create_keyval is called. >> >> I am using petsc 3.11.3 but found 3.13.2 has the same logic. >> >> Is this a bug or I didn't use it correctly? >> >> Thanks, >> Sam >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sam.guo at cd-adapco.com Sat Jun 20 10:45:37 2020 From: sam.guo at cd-adapco.com (Sam Guo) Date: Sat, 20 Jun 2020 08:45:37 -0700 Subject: [petsc-users] PETSc initialization error In-Reply-To: References: Message-ID: Assuming finalizer is only needed once for entire session(?), I can put initializer into the static block to call it once but where do I call finalizer? On Saturday, June 20, 2020, Junchao Zhang wrote: > The counter num_attr should be recycled. But first try to call PETSc > initialize/Finalize only once to see it fixes the error. > --Junchao Zhang > > > On Sat, Jun 20, 2020 at 12:48 AM Sam Guo wrote: > >> To clarify, I call PETSc initialize and PETSc finalize everytime I call >> SLEPc: >> >> PetscInitializeNoPointers(argc,args,nullptr,nullptr); >> >> SlepcInitialize(&argc,&args,static_cast(nullptr),help); >> >> //calling slepc >> >> SlepcFinalize(); >> >> PetscFinalize(); >> >> >> >> On Fri, Jun 19, 2020 at 10:32 PM Sam Guo wrote: >> >>> Dear PETSc team, >>> When I called SLEPc multiple time, I eventually got following error: >>> >>> MPI operation not supported by PETSc's sequential MPI wrappers >>> [0]PETSC ERROR: #1 PetscInitialize() line 967 in ../../../petsc/src/sys/ >>> objects/pinit.c >>> [0]PETSC ERROR: #2 SlepcInitialize() line 262 in ../../../slepc/src/sys/ >>> slepcinit.c >>> [0]PETSC ERROR: #3 SlepcInitializeNoPointers() line 359 in >>> ../../../slepc/src/sys/slepcinit.c >>> PETSC ERROR: Logging has not been enabled. >>> You might have forgotten to call PetscInitialize(). >>> >>> I debugged: it is because of following in petsc/src/sys/mpiuni/mpi.c >>> >>> if (num_attr >= MAX_ATTR) >>> >>> in function int MPI_Comm_create_keyval(MPI_Copy_function >>> *copy_fn,MPI_Delete_function *delete_fn,int *keyval,void *extra_state) >>> >>> num_attr is declared static and keeps increasing every >>> time MPI_Comm_create_keyval is called. >>> >>> I am using petsc 3.11.3 but found 3.13.2 has the same logic. >>> >>> Is this a bug or I didn't use it correctly? >>> >>> Thanks, >>> Sam >>> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From sam.guo at cd-adapco.com Sat Jun 20 10:48:43 2020 From: sam.guo at cd-adapco.com (Sam Guo) Date: Sat, 20 Jun 2020 08:48:43 -0700 Subject: [petsc-users] PETSc initialization error In-Reply-To: References: Message-ID: Typo: I mean ?Assuming initializer is only needed once for entire session? On Saturday, June 20, 2020, Sam Guo wrote: > Assuming finalizer is only needed once for entire session(?), I can put > initializer into the static block to call it once but where do I call > finalizer? > > > On Saturday, June 20, 2020, Junchao Zhang wrote: > >> The counter num_attr should be recycled. But first try to call PETSc >> initialize/Finalize only once to see it fixes the error. >> --Junchao Zhang >> >> >> On Sat, Jun 20, 2020 at 12:48 AM Sam Guo wrote: >> >>> To clarify, I call PETSc initialize and PETSc finalize everytime I call >>> SLEPc: >>> >>> PetscInitializeNoPointers(argc,args,nullptr,nullptr); >>> >>> SlepcInitialize(&argc,&args,static_cast(nullptr),help); >>> >>> //calling slepc >>> >>> SlepcFinalize(); >>> >>> PetscFinalize(); >>> >>> >>> >>> On Fri, Jun 19, 2020 at 10:32 PM Sam Guo wrote: >>> >>>> Dear PETSc team, >>>> When I called SLEPc multiple time, I eventually got following error: >>>> >>>> MPI operation not supported by PETSc's sequential MPI wrappers >>>> [0]PETSC ERROR: #1 PetscInitialize() line 967 in >>>> ../../../petsc/src/sys/objects/pinit.c >>>> [0]PETSC ERROR: #2 SlepcInitialize() line 262 in >>>> ../../../slepc/src/sys/slepcinit.c >>>> [0]PETSC ERROR: #3 SlepcInitializeNoPointers() line 359 in >>>> ../../../slepc/src/sys/slepcinit.c >>>> PETSC ERROR: Logging has not been enabled. >>>> You might have forgotten to call PetscInitialize(). >>>> >>>> I debugged: it is because of following in petsc/src/sys/mpiuni/mpi.c >>>> >>>> if (num_attr >= MAX_ATTR) >>>> >>>> in function int MPI_Comm_create_keyval(MPI_Copy_function >>>> *copy_fn,MPI_Delete_function *delete_fn,int *keyval,void *extra_state) >>>> >>>> num_attr is declared static and keeps increasing every >>>> time MPI_Comm_create_keyval is called. >>>> >>>> I am using petsc 3.11.3 but found 3.13.2 has the same logic. >>>> >>>> Is this a bug or I didn't use it correctly? >>>> >>>> Thanks, >>>> Sam >>>> >>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From junchao.zhang at gmail.com Sat Jun 20 10:54:16 2020 From: junchao.zhang at gmail.com (Junchao Zhang) Date: Sat, 20 Jun 2020 10:54:16 -0500 Subject: [petsc-users] PETSc initialization error In-Reply-To: References: Message-ID: I don't understand what your session means. Let's try this patch diff --git a/src/sys/mpiuni/mpi.c b/src/sys/mpiuni/mpi.c index d559a513..c058265d 100644 --- a/src/sys/mpiuni/mpi.c +++ b/src/sys/mpiuni/mpi.c @@ -283,6 +283,7 @@ int MPI_Finalize(void) MPI_Comm_free(&comm); comm = MPI_COMM_SELF; MPI_Comm_free(&comm); + num_attr = 1; /* reset the counter */ MPI_was_finalized = 1; return MPI_SUCCESS; } --Junchao Zhang On Sat, Jun 20, 2020 at 10:48 AM Sam Guo wrote: > Typo: I mean ?Assuming initializer is only needed once for entire session? > > On Saturday, June 20, 2020, Sam Guo wrote: > >> Assuming finalizer is only needed once for entire session(?), I can put >> initializer into the static block to call it once but where do I call >> finalizer? >> >> >> On Saturday, June 20, 2020, Junchao Zhang >> wrote: >> >>> The counter num_attr should be recycled. But first try to call PETSc >>> initialize/Finalize only once to see it fixes the error. >>> --Junchao Zhang >>> >>> >>> On Sat, Jun 20, 2020 at 12:48 AM Sam Guo wrote: >>> >>>> To clarify, I call PETSc initialize and PETSc finalize everytime I call >>>> SLEPc: >>>> >>>> PetscInitializeNoPointers(argc,args,nullptr,nullptr); >>>> >>>> SlepcInitialize(&argc,&args,static_cast(nullptr),help); >>>> >>>> //calling slepc >>>> >>>> SlepcFinalize(); >>>> >>>> PetscFinalize(); >>>> >>>> >>>> >>>> On Fri, Jun 19, 2020 at 10:32 PM Sam Guo wrote: >>>> >>>>> Dear PETSc team, >>>>> When I called SLEPc multiple time, I eventually got following >>>>> error: >>>>> >>>>> MPI operation not supported by PETSc's sequential MPI wrappers >>>>> [0]PETSC ERROR: #1 PetscInitialize() line 967 in >>>>> ../../../petsc/src/sys/objects/pinit.c >>>>> [0]PETSC ERROR: #2 SlepcInitialize() line 262 in >>>>> ../../../slepc/src/sys/slepcinit.c >>>>> [0]PETSC ERROR: #3 SlepcInitializeNoPointers() line 359 in >>>>> ../../../slepc/src/sys/slepcinit.c >>>>> PETSC ERROR: Logging has not been enabled. >>>>> You might have forgotten to call PetscInitialize(). >>>>> >>>>> I debugged: it is because of following in petsc/src/sys/mpiuni/mpi.c >>>>> >>>>> if (num_attr >= MAX_ATTR) >>>>> >>>>> in function int MPI_Comm_create_keyval(MPI_Copy_function >>>>> *copy_fn,MPI_Delete_function *delete_fn,int *keyval,void *extra_state) >>>>> >>>>> num_attr is declared static and keeps increasing every >>>>> time MPI_Comm_create_keyval is called. >>>>> >>>>> I am using petsc 3.11.3 but found 3.13.2 has the same logic. >>>>> >>>>> Is this a bug or I didn't use it correctly? >>>>> >>>>> Thanks, >>>>> Sam >>>>> >>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Sat Jun 20 12:00:28 2020 From: bsmith at petsc.dev (Barry Smith) Date: Sat, 20 Jun 2020 12:00:28 -0500 Subject: [petsc-users] SNES Line search: fixing the non zero solution values of dirichlet nodes during line search In-Reply-To: References: Message-ID: Ashish, Based on this " RHS as increment of (Dirichlet_value-solution_value). " I am guessing your inhomogeneous Dirichlet boundary conditions are time dependent, that is u(t) = b(t) for a given b? If so your problem is really a DAE, not an ODE. You can solve it in several ways: if b(t) is something you can differentiate then write new boundary conditions as \dot u = \dot b and solve the equations with this new form otherwise: solve as a DAE I think just with TSARKIMEX and TS_EQ_DAE_IMPLICIT_INDEX1 with TSSetEquationType and TSARKIMEXSetFullyImplicit but you will need to use TSSetIFunction()/Jaocbian instead of TSRHSFunction/Jacobian I am cc:ing a couple of others who know much more about ODE integrators than I and may have better approaches. I don't see any real harm in your in current approach, but there isn't something we can easily add to PETSc "do it for you". I think you will lose some accuracy in your approach because you are just interpolating the know b() at the one new time-steps losing any information about it in the stage values while the approaches above integrate u at the stage locations. Barry > On Jun 19, 2020, at 5:15 PM, Ashish Patel wrote: > > Dear PETSc users, > > We use PETSc as part of a finite element method program and we are trying to properly implement Dirichlet boundary conditions for non-linear, transient problems. We find that when we use a line search method it also changes the non-zero solution value of Dirichlet nodes as it steps through the line search iteration. I was wondering if there is a way to freeze some index sets of a vector from changing during the line search operation? > > We are using the TS framework to setup the problem and use 'MatZeroRowsColumns' to set the diagonal of the jacobian to 1 for the dirichlet nodes and set the RHS as increment of (Dirichlet_value-solution_value). This works when the line search method is turned off by using '-snes_linesearch_type basic' however using the default 'bt' linesearch, the TS diverges with error shown below. In a separate implementation if we overwrote the dirichlet nodes of the solution vector in TS residual function with the Dirichlet values then the 'bt' line search method converged to the right solution. However we would like to avoid modifying the internal PETSc vector in our implementation. > > 0 TS dt 1. time 0. > 0 SNES Function norm 2.378549386020e+03 > Line search: gnorm after quadratic fit 4.369235425165e+03 > Line search: Cubic step no good, shrinking lambda, current gnorm 2.385369069060e+03 lambda=1.0000000000000002e-02 > Line search: Cubically determined step, current gnorm 2.373925008934e+03 lambda=3.8846250444606093e-03 > 1 SNES Function norm 2.373925008934e+03 > Line search: gnorm after quadratic fit 5.006914179995e+03 > Line search: Cubic step no good, shrinking lambda, current gnorm 2.420957096780e+03 lambda=1.0000000000000002e-02 > Line search: Cubic step no good, shrinking lambda, current gnorm 2.376034946750e+03 lambda=1.6129422079664700e-03 > Line search: Cubic step no good, shrinking lambda, current gnorm 2.374313344729e+03 lambda=4.8465026740690043e-04 > Line search: Cubic step no good, shrinking lambda, current gnorm 2.373999473242e+03 lambda=1.5857251532828948e-04 > Line search: Cubically determined step, current gnorm 2.373921668024e+03 lambda=5.8116507162753387e-05 > 2 SNES Function norm 2.373921668024e+03 > Line search: gnorm after quadratic fit 4.771035112853e+03 > Line search: Cubic step no good, shrinking lambda, current gnorm 2.410650718394e+03 lambda=1.0000000000000002e-02 > Line search: Cubic step no good, shrinking lambda, current gnorm 2.375104094198e+03 lambda=1.8983783738011522e-03 > Line search: Cubic step no good, shrinking lambda, current gnorm 2.374049151562e+03 lambda=7.0688528086485479e-04 > Line search: Cubic step no good, shrinking lambda, current gnorm 2.373935090907e+03 lambda=2.9132722794896019e-04 > Line search: Cubically determined step, current gnorm 2.373921032081e+03 lambda=1.2527602265373028e-04 > 3 SNES Function norm 2.373921032081e+03 > Line search: gnorm after quadratic fit 5.117914832660e+03 > Line search: Cubic step no good, shrinking lambda, current gnorm 2.422635362094e+03 lambda=1.0000000000000002e-02 > Line search: Cubic step no good, shrinking lambda, current gnorm 2.375923870970e+03 lambda=1.5769887508300913e-03 > Line search: Cubic step no good, shrinking lambda, current gnorm 2.374287081592e+03 lambda=4.8018017100729705e-04 > Line search: Cubic step no good, shrinking lambda, current gnorm 2.373999131160e+03 lambda=1.5908966655977892e-04 > Line search: Cubic step no good, shrinking lambda, current gnorm 2.373930608274e+03 lambda=5.7603977147935371e-05 > Line search: Cubic step no good, shrinking lambda, current gnorm 2.373922022038e+03 lambda=2.3884787050507805e-05 > Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921396636e+03 lambda=1.0282405393471519e-05 > Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921173719e+03 lambda=4.3868704034012554e-06 > Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921091304e+03 lambda=1.8774991151727107e-06 > Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921057168e+03 lambda=8.0331628193813397e-07 > Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921042769e+03 lambda=3.4377811174560817e-07 > Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921036645e+03 lambda=1.4712421886880395e-07 > Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921034032e+03 lambda=6.2965363212312132e-08 > Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921032916e+03 lambda=2.6947718250469261e-08 > Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921032438e+03 lambda=1.1533043989179318e-08 > Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921032233e+03 lambda=4.9359018284334258e-09 > Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921032146e+03 lambda=2.1124641857409436e-09 > Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921032109e+03 lambda=9.0409137304143975e-10 > Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921032093e+03 lambda=3.8693264359674819e-10 > Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921032086e+03 lambda=1.6559911204766632e-10 > Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921032083e+03 lambda=7.0873187020534353e-11 > Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921032081e+03 lambda=3.0332192901757729e-11 > Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921032081e+03 lambda=1.2981600435512875e-11 > Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921032081e+03 lambda=5.5559212521628090e-12 > Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921032081e+03 lambda=2.3777380405336958e-12 > Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921032081e+03 lambda=1.0176091649134191e-12 > Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921032081e+03 lambda=4.3555419789721080e-13 > Line search: unable to find good step length! After 27 tries > Line search: fnorm=2.3739210320805191e+03, gnorm=2.3739210320805323e+03, ynorm=8.5698020038772756e+03, minlambda=9.9999999999999998e-13, lambda=4.3555419789721080e-13, initial slope=-5.6355010665542409e+06 > SNES Object: 1 MPI processes > type: newtonls > maximum iterations=50, maximum function evaluations=10000 > tolerances: relative=1e-08, absolute=1e-50, solution=1e-08 > total number of linear solver iterations=4 > total number of function evaluations=48 > norm schedule ALWAYS > SNESLineSearch Object: 1 MPI processes > type: bt > interpolation: cubic > alpha=1.000000e-04 > maxstep=1.000000e+08, minlambda=1.000000e-12 > tolerances: relative=1.000000e-08, absolute=1.000000e-15, lambda=1.000000e-08 > maximum iterations=40 > KSP Object: 1 MPI processes > type: preonly > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > left preconditioning > using NONE norm type for convergence test > PC Object: 1 MPI processes > type: cholesky > out-of-place factorization > tolerance for zero pivot 2.22045e-14 > matrix ordering: natural > factor fill ratio given 0., needed 0. > Factored matrix follows: > Mat Object: 1 MPI processes > type: mumps > rows=38154, cols=38154 > package used to perform factorization: mumps > total: nonzeros=7080060, allocated nonzeros=7080060 > total number of mallocs used during MatSetValues calls=0 > MUMPS run parameters: > SYM (matrix type): 2 > PAR (host participation): 1 > ICNTL(1) (output for error): 6 > ICNTL(2) (output of diagnostic msg): 0 > ICNTL(3) (output for global info): 0 > ICNTL(4) (level of printing): 0 > ICNTL(5) (input mat struct): 0 > ICNTL(6) (matrix prescaling): 7 > ICNTL(7) (sequential matrix ordering):7 > ICNTL(8) (scaling strategy): 77 > ICNTL(10) (max num of refinements): 0 > ICNTL(11) (error analysis): 0 > ICNTL(12) (efficiency control): 0 > ICNTL(13) (efficiency control): 0 > ICNTL(14) (percentage of estimated workspace increase): 20 > ICNTL(18) (input mat struct): 0 > ICNTL(19) (Schur complement info): 0 > ICNTL(20) (rhs sparse pattern): 0 > ICNTL(21) (solution struct): 0 > ICNTL(22) (in-core/out-of-core facility): 0 > ICNTL(23) (max size of memory can be allocated locally):0 > ICNTL(24) (detection of null pivot rows): 0 > ICNTL(25) (computation of a null space basis): 0 > ICNTL(26) (Schur options for rhs or solution): 0 > ICNTL(27) (experimental parameter): -32 > ICNTL(28) (use parallel or sequential ordering): 1 > ICNTL(29) (parallel ordering): 0 > ICNTL(30) (user-specified set of entries in inv(A)): 0 > ICNTL(31) (factors is discarded in the solve phase): 0 > ICNTL(33) (compute determinant): 0 > ICNTL(35) (activate BLR based factorization): 0 > ICNTL(36) (choice of BLR factorization variant): 0 > ICNTL(38) (estimated compression rate of LU factors): 333 > CNTL(1) (relative pivoting threshold): 0.01 > CNTL(2) (stopping criterion of refinement): 1.49012e-08 > CNTL(3) (absolute pivoting threshold): 0. > CNTL(4) (value of static pivoting): -1. > CNTL(5) (fixation for null pivots): 0. > CNTL(7) (dropping parameter for BLR): 0. > RINFO(1) (local estimated flops for the elimination after analysis): > [0] 2.73979e+09 > RINFO(2) (local estimated flops for the assembly after factorization): > [0] 1.08826e+07 > RINFO(3) (local estimated flops for the elimination after factorization): > [0] 2.73979e+09 > INFO(15) (estimated size of (in MB) MUMPS internal data for running numerical factorization): > [0] 94 > INFO(16) (size of (in MB) MUMPS internal data used during numerical factorization): > [0] 94 > INFO(23) (num of pivots eliminated on this processor after factorization): > [0] 38154 > RINFOG(1) (global estimated flops for the elimination after analysis): 2.73979e+09 > RINFOG(2) (global estimated flops for the assembly after factorization): 1.08826e+07 > RINFOG(3) (global estimated flops for the elimination after factorization): 2.73979e+09 > (RINFOG(12) RINFOG(13))*2^INFOG(34) (determinant): (0.,0.)*(2^0) > INFOG(3) (estimated real workspace for factors on all processors after analysis): 8377336 > INFOG(4) (estimated integer workspace for factors on all processors after analysis): 447902 > INFOG(5) (estimated maximum front size in the complete tree): 990 > INFOG(6) (number of nodes in the complete tree): 2730 > INFOG(7) (ordering option effectively use after analysis): 5 > INFOG(8) (structural symmetry in percent of the permuted matrix after analysis): 100 > INFOG(9) (total real/complex workspace to store the matrix factors after factorization): 8377336 > INFOG(10) (total integer space store the matrix factors after factorization): 447902 > INFOG(11) (order of largest frontal matrix after factorization): 990 > INFOG(12) (number of off-diagonal pivots): 10 > INFOG(13) (number of delayed pivots after factorization): 0 > INFOG(14) (number of memory compress after factorization): 0 > INFOG(15) (number of steps of iterative refinement after solution): 0 > INFOG(16) (estimated size (in MB) of all MUMPS internal data for factorization after analysis: value on the most memory consuming processor): 94 > INFOG(17) (estimated size of all MUMPS internal data for factorization after analysis: sum over all processors): 94 > INFOG(18) (size of all MUMPS internal data allocated during factorization: value on the most memory consuming processor): 94 > INFOG(19) (size of all MUMPS internal data allocated during factorization: sum over all processors): 94 > INFOG(20) (estimated number of entries in the factors): 7080060 > INFOG(21) (size in MB of memory effectively used during factorization - value on the most memory consuming processor): 80 > INFOG(22) (size in MB of memory effectively used during factorization - sum over all processors): 80 > INFOG(23) (after analysis: value of ICNTL(6) effectively used): 0 > INFOG(24) (after analysis: value of ICNTL(12) effectively used): 1 > INFOG(25) (after factorization: number of pivots modified by static pivoting): 0 > INFOG(28) (after factorization: number of null pivots encountered): 0 > INFOG(29) (after factorization: effective number of entries in the factors (sum over all processors)): 7080060 > INFOG(30, 31) (after solution: size in Mbytes of memory used during solution phase): 92, 92 > INFOG(32) (after analysis: type of analysis done): 1 > INFOG(33) (value used for ICNTL(8)): 7 > INFOG(34) (exponent of the determinant if determinant is requested): 0 > INFOG(35) (after factorization: number of entries taking into account BLR factor compression - sum over all processors): 7080060 > INFOG(36) (after analysis: estimated size of all MUMPS internal data for running BLR in-core - value on the most memory consuming processor): 0 > INFOG(37) (after analysis: estimated size of all MUMPS internal data for running BLR in-core - sum over all processors): 0 > INFOG(38) (after analysis: estimated size of all MUMPS internal data for running BLR out-of-core - value on the most memory consuming processor): 0 > INFOG(39) (after analysis: estimated size of all MUMPS internal data for running BLR out-of-core - sum over all processors): 0 > linear system matrix = precond matrix: > Mat Object: 1 MPI processes > type: seqaij > rows=38154, cols=38154 > total: nonzeros=973446, allocated nonzeros=973446 > total number of mallocs used during MatSetValues calls=0 > not using I-node routines > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [0]PETSC ERROR: > [0]PETSC ERROR: TSStep has failed due to DIVERGED_NONLINEAR_SOLVE, increase -ts_max_snes_failures or make negative to attempt recovery > [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.12.3, Jan, 03, 2020 > > Thanks, > Ashish > Scientific Computing Division > OnScale > CA, USA -------------- next part -------------- An HTML attachment was scrubbed... URL: From sam.guo at cd-adapco.com Sat Jun 20 12:08:50 2020 From: sam.guo at cd-adapco.com (Sam Guo) Date: Sat, 20 Jun 2020 10:08:50 -0700 Subject: [petsc-users] PETSc initialization error In-Reply-To: References: Message-ID: Hi Junchao, Thanks for the quick response. By "session" I mean the program. I tried your patch but it does not help. This is the serial. I guess finalizer may not be called? Thanks, Sam On Sat, Jun 20, 2020 at 8:54 AM Junchao Zhang wrote: > I don't understand what your session means. Let's try this patch > > diff --git a/src/sys/mpiuni/mpi.c b/src/sys/mpiuni/mpi.c > index d559a513..c058265d 100644 > --- a/src/sys/mpiuni/mpi.c > +++ b/src/sys/mpiuni/mpi.c > @@ -283,6 +283,7 @@ int MPI_Finalize(void) > MPI_Comm_free(&comm); > comm = MPI_COMM_SELF; > MPI_Comm_free(&comm); > + num_attr = 1; /* reset the counter */ > MPI_was_finalized = 1; > return MPI_SUCCESS; > } > > > --Junchao Zhang > > > On Sat, Jun 20, 2020 at 10:48 AM Sam Guo wrote: > >> Typo: I mean ?Assuming initializer is only needed once for entire session? >> >> On Saturday, June 20, 2020, Sam Guo wrote: >> >>> Assuming finalizer is only needed once for entire session(?), I can put >>> initializer into the static block to call it once but where do I call >>> finalizer? >>> >>> >>> On Saturday, June 20, 2020, Junchao Zhang >>> wrote: >>> >>>> The counter num_attr should be recycled. But first try to call PETSc >>>> initialize/Finalize only once to see it fixes the error. >>>> --Junchao Zhang >>>> >>>> >>>> On Sat, Jun 20, 2020 at 12:48 AM Sam Guo wrote: >>>> >>>>> To clarify, I call PETSc initialize and PETSc finalize everytime I >>>>> call SLEPc: >>>>> >>>>> PetscInitializeNoPointers(argc,args,nullptr,nullptr); >>>>> >>>>> SlepcInitialize(&argc,&args,static_cast(nullptr),help); >>>>> >>>>> //calling slepc >>>>> >>>>> SlepcFinalize(); >>>>> >>>>> PetscFinalize(); >>>>> >>>>> >>>>> >>>>> On Fri, Jun 19, 2020 at 10:32 PM Sam Guo >>>>> wrote: >>>>> >>>>>> Dear PETSc team, >>>>>> When I called SLEPc multiple time, I eventually got following >>>>>> error: >>>>>> >>>>>> MPI operation not supported by PETSc's sequential MPI wrappers >>>>>> [0]PETSC ERROR: #1 PetscInitialize() line 967 in >>>>>> ../../../petsc/src/sys/objects/pinit.c >>>>>> [0]PETSC ERROR: #2 SlepcInitialize() line 262 in >>>>>> ../../../slepc/src/sys/slepcinit.c >>>>>> [0]PETSC ERROR: #3 SlepcInitializeNoPointers() line 359 in >>>>>> ../../../slepc/src/sys/slepcinit.c >>>>>> PETSC ERROR: Logging has not been enabled. >>>>>> You might have forgotten to call PetscInitialize(). >>>>>> >>>>>> I debugged: it is because of following in petsc/src/sys/mpiuni/mpi.c >>>>>> >>>>>> if (num_attr >= MAX_ATTR) >>>>>> >>>>>> in function int MPI_Comm_create_keyval(MPI_Copy_function >>>>>> *copy_fn,MPI_Delete_function *delete_fn,int *keyval,void *extra_state) >>>>>> >>>>>> num_attr is declared static and keeps increasing every >>>>>> time MPI_Comm_create_keyval is called. >>>>>> >>>>>> I am using petsc 3.11.3 but found 3.13.2 has the same logic. >>>>>> >>>>>> Is this a bug or I didn't use it correctly? >>>>>> >>>>>> Thanks, >>>>>> Sam >>>>>> >>>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Sat Jun 20 12:24:45 2020 From: bsmith at petsc.dev (Barry Smith) Date: Sat, 20 Jun 2020 12:24:45 -0500 Subject: [petsc-users] PETSc initialization error In-Reply-To: References: Message-ID: <004EA4EB-815F-4EEE-8988-CA2179304157@petsc.dev> Junchao, This is a good bug fix. It solves the problem when PETSc initialize is called many times. There is another fix you can do to limit PETSc mpiuni running out of attributes inside a single PETSc run: int MPI_Comm_create_keyval(MPI_Copy_function *copy_fn,MPI_Delete_function *delete_fn,int *keyval,void *extra_state) { if (num_attr >= MAX_ATTR){ for (i=0; i On Jun 20, 2020, at 10:54 AM, Junchao Zhang wrote: > > I don't understand what your session means. Let's try this patch > > diff --git a/src/sys/mpiuni/mpi.c b/src/sys/mpiuni/mpi.c > index d559a513..c058265d 100644 > --- a/src/sys/mpiuni/mpi.c > +++ b/src/sys/mpiuni/mpi.c > @@ -283,6 +283,7 @@ int MPI_Finalize(void) > MPI_Comm_free(&comm); > comm = MPI_COMM_SELF; > MPI_Comm_free(&comm); > + num_attr = 1; /* reset the counter */ > MPI_was_finalized = 1; > return MPI_SUCCESS; > } > > > --Junchao Zhang > > > On Sat, Jun 20, 2020 at 10:48 AM Sam Guo > wrote: > Typo: I mean ?Assuming initializer is only needed once for entire session? > > On Saturday, June 20, 2020, Sam Guo > wrote: > Assuming finalizer is only needed once for entire session(?), I can put initializer into the static block to call it once but where do I call finalizer? > > > On Saturday, June 20, 2020, Junchao Zhang > wrote: > The counter num_attr should be recycled. But first try to call PETSc initialize/Finalize only once to see it fixes the error. > --Junchao Zhang > > > On Sat, Jun 20, 2020 at 12:48 AM Sam Guo > wrote: > To clarify, I call PETSc initialize and PETSc finalize everytime I call SLEPc: > > PetscInitializeNoPointers(argc,args,nullptr,nullptr); > SlepcInitialize(&argc,&args,static_cast(nullptr),help); > //calling slepc > SlepcFinalize(); > PetscFinalize(); > > > On Fri, Jun 19, 2020 at 10:32 PM Sam Guo > wrote: > Dear PETSc team, > When I called SLEPc multiple time, I eventually got following error: > > MPI operation not supported by PETSc's sequential MPI wrappers > [0]PETSC ERROR: #1 PetscInitialize() line 967 in ../../../petsc/src/sys/objects/pinit.c > [0]PETSC ERROR: #2 SlepcInitialize() line 262 in ../../../slepc/src/sys/slepcinit.c > [0]PETSC ERROR: #3 SlepcInitializeNoPointers() line 359 in ../../../slepc/src/sys/slepcinit.c > PETSC ERROR: Logging has not been enabled. > You might have forgotten to call PetscInitialize(). > > I debugged: it is because of following in petsc/src/sys/mpiuni/mpi.c > > if (num_attr >= MAX_ATTR) > > in function int MPI_Comm_create_keyval(MPI_Copy_function *copy_fn,MPI_Delete_function *delete_fn,int *keyval,void *extra_state) > > num_attr is declared static and keeps increasing every time MPI_Comm_create_keyval is called. > > I am using petsc 3.11.3 but found 3.13.2 has the same logic. > > Is this a bug or I didn't use it correctly? > > Thanks, > Sam -------------- next part -------------- An HTML attachment was scrubbed... URL: From sam.guo at cd-adapco.com Sat Jun 20 13:12:05 2020 From: sam.guo at cd-adapco.com (Sam Guo) Date: Sat, 20 Jun 2020 11:12:05 -0700 Subject: [petsc-users] PETSc initialization error In-Reply-To: <004EA4EB-815F-4EEE-8988-CA2179304157@petsc.dev> References: <004EA4EB-815F-4EEE-8988-CA2179304157@petsc.dev> Message-ID: Junchao, I debugged: MPI_Finalize is not called for serial. Barry, I tried your patch and it seems better but eventually got following error: [0]PETSC ERROR: #1 PetscCommDuplicate() line 160 in ../../../petsc/src/sys/objects/tagm.c [0]PETSC ERROR: #2 PetscHeaderCreate_Private() line 64 in ../../../petsc/src/sys/objects/inherit.c [0]PETSC ERROR: #3 MatCreate() line 91 in ../../../petsc/src/mat/utils/gcreate.c [0]PETSC ERROR: #4 MatCreateShell() line 787 in ../../../petsc/src/mat/impls/shell/shell.c [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Null argument, when expecting valid pointer [0]PETSC ERROR: Null Object: Parameter # 1 [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.11.3, Jun, 26, 2019 [0]PETSC ERROR: #6 MatShellSetOperation() line 1052 in ../../../petsc/src/mat/impls/shell/shell.c On Sat, Jun 20, 2020 at 10:24 AM Barry Smith wrote: > > Junchao, > > This is a good bug fix. It solves the problem when PETSc initialize > is called many times. > > There is another fix you can do to limit PETSc mpiuni running out of > attributes inside a single PETSc run: > > > int MPI_Comm_create_keyval(MPI_Copy_function *copy_fn,MPI_Delete_function > *delete_fn,int *keyval,void *extra_state) > { > > if (num_attr >= MAX_ATTR){ > for (i=0; i if (!attr_keyval[i].extra_state) { > /* reuse this slot */ > attr_keyval[i].extra_state = extra_state; > attr_keyval[i.]del = delete_fn; > *keyval = i; > return MPI_SUCCESS; > } > } > return MPIUni_Abort(MPI_COMM_WORLD,1); > } > return MPIUni_Abort(MPI_COMM_WORLD,1); > attr_keyval[num_attr].extra_state = extra_state; > attr_keyval[num_attr].del = delete_fn; > *keyval = num_attr++; > return MPI_SUCCESS; > } > > This will work if the user creates tons of attributes but is constantly > deleting some as they new ones. So long as the number outstanding at one > time is < MAX_ATTR) > > Barry > > > > > > On Jun 20, 2020, at 10:54 AM, Junchao Zhang > wrote: > > I don't understand what your session means. Let's try this patch > > diff --git a/src/sys/mpiuni/mpi.c b/src/sys/mpiuni/mpi.c > index d559a513..c058265d 100644 > --- a/src/sys/mpiuni/mpi.c > +++ b/src/sys/mpiuni/mpi.c > @@ -283,6 +283,7 @@ int MPI_Finalize(void) > MPI_Comm_free(&comm); > comm = MPI_COMM_SELF; > MPI_Comm_free(&comm); > + num_attr = 1; /* reset the counter */ > MPI_was_finalized = 1; > return MPI_SUCCESS; > } > > > --Junchao Zhang > > > On Sat, Jun 20, 2020 at 10:48 AM Sam Guo wrote: > >> Typo: I mean ?Assuming initializer is only needed once for entire session? >> >> On Saturday, June 20, 2020, Sam Guo wrote: >> >>> Assuming finalizer is only needed once for entire session(?), I can put >>> initializer into the static block to call it once but where do I call >>> finalizer? >>> >>> >>> On Saturday, June 20, 2020, Junchao Zhang >>> wrote: >>> >>>> The counter num_attr should be recycled. But first try to call PETSc >>>> initialize/Finalize only once to see it fixes the error. >>>> --Junchao Zhang >>>> >>>> >>>> On Sat, Jun 20, 2020 at 12:48 AM Sam Guo wrote: >>>> >>>>> To clarify, I call PETSc initialize and PETSc finalize everytime I >>>>> call SLEPc: >>>>> >>>>> PetscInitializeNoPointers(argc,args,nullptr,nullptr); >>>>> >>>>> SlepcInitialize(&argc,&args,static_cast(nullptr),help); >>>>> >>>>> //calling slepc >>>>> >>>>> SlepcFinalize(); >>>>> >>>>> PetscFinalize(); >>>>> >>>>> >>>>> >>>>> On Fri, Jun 19, 2020 at 10:32 PM Sam Guo >>>>> wrote: >>>>> >>>>>> Dear PETSc team, >>>>>> When I called SLEPc multiple time, I eventually got following >>>>>> error: >>>>>> >>>>>> MPI operation not supported by PETSc's sequential MPI wrappers >>>>>> [0]PETSC ERROR: #1 PetscInitialize() line 967 in >>>>>> ../../../petsc/src/sys/objects/pinit.c >>>>>> [0]PETSC ERROR: #2 SlepcInitialize() line 262 in >>>>>> ../../../slepc/src/sys/slepcinit.c >>>>>> [0]PETSC ERROR: #3 SlepcInitializeNoPointers() line 359 in >>>>>> ../../../slepc/src/sys/slepcinit.c >>>>>> PETSC ERROR: Logging has not been enabled. >>>>>> You might have forgotten to call PetscInitialize(). >>>>>> >>>>>> I debugged: it is because of following in petsc/src/sys/mpiuni/mpi.c >>>>>> >>>>>> if (num_attr >= MAX_ATTR) >>>>>> >>>>>> in function int MPI_Comm_create_keyval(MPI_Copy_function >>>>>> *copy_fn,MPI_Delete_function *delete_fn,int *keyval,void *extra_state) >>>>>> >>>>>> num_attr is declared static and keeps increasing every >>>>>> time MPI_Comm_create_keyval is called. >>>>>> >>>>>> I am using petsc 3.11.3 but found 3.13.2 has the same logic. >>>>>> >>>>>> Is this a bug or I didn't use it correctly? >>>>>> >>>>>> Thanks, >>>>>> Sam >>>>>> >>>>> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From junchao.zhang at gmail.com Sat Jun 20 13:40:25 2020 From: junchao.zhang at gmail.com (Junchao Zhang) Date: Sat, 20 Jun 2020 13:40:25 -0500 Subject: [petsc-users] PETSc initialization error In-Reply-To: References: <004EA4EB-815F-4EEE-8988-CA2179304157@petsc.dev> Message-ID: Sam, There are more problems. I am working on a fix. Please wait an hour. Thanks. --Junchao Zhang On Sat, Jun 20, 2020 at 1:12 PM Sam Guo wrote: > Junchao, > I debugged: MPI_Finalize is not called for serial. > > Barry, > I tried your patch and it seems better but eventually got following > error: > > [0]PETSC ERROR: #1 PetscCommDuplicate() line 160 in > ../../../petsc/src/sys/objects/tagm.c > [0]PETSC ERROR: #2 PetscHeaderCreate_Private() line 64 in > ../../../petsc/src/sys/objects/inherit.c > [0]PETSC ERROR: #3 MatCreate() line 91 in > ../../../petsc/src/mat/utils/gcreate.c > [0]PETSC ERROR: #4 MatCreateShell() line 787 in > ../../../petsc/src/mat/impls/shell/shell.c > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Null argument, when expecting valid pointer > [0]PETSC ERROR: Null Object: Parameter # 1 > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.11.3, Jun, 26, 2019 > [0]PETSC ERROR: #6 MatShellSetOperation() line 1052 in > ../../../petsc/src/mat/impls/shell/shell.c > > On Sat, Jun 20, 2020 at 10:24 AM Barry Smith wrote: > >> >> Junchao, >> >> This is a good bug fix. It solves the problem when PETSc initialize >> is called many times. >> >> There is another fix you can do to limit PETSc mpiuni running out of >> attributes inside a single PETSc run: >> >> >> int MPI_Comm_create_keyval(MPI_Copy_function *copy_fn,MPI_Delete_function >> *delete_fn,int *keyval,void *extra_state) >> { >> >> if (num_attr >= MAX_ATTR){ >> for (i=0; i> if (!attr_keyval[i].extra_state) { >> /* reuse this slot */ >> attr_keyval[i].extra_state = extra_state; >> attr_keyval[i.]del = delete_fn; >> *keyval = i; >> return MPI_SUCCESS; >> } >> } >> return MPIUni_Abort(MPI_COMM_WORLD,1); >> } >> return MPIUni_Abort(MPI_COMM_WORLD,1); >> attr_keyval[num_attr].extra_state = extra_state; >> attr_keyval[num_attr].del = delete_fn; >> *keyval = num_attr++; >> return MPI_SUCCESS; >> } >> >> This will work if the user creates tons of attributes but is constantly >> deleting some as they new ones. So long as the number outstanding at one >> time is < MAX_ATTR) >> >> Barry >> >> >> >> >> >> On Jun 20, 2020, at 10:54 AM, Junchao Zhang >> wrote: >> >> I don't understand what your session means. Let's try this patch >> >> diff --git a/src/sys/mpiuni/mpi.c b/src/sys/mpiuni/mpi.c >> index d559a513..c058265d 100644 >> --- a/src/sys/mpiuni/mpi.c >> +++ b/src/sys/mpiuni/mpi.c >> @@ -283,6 +283,7 @@ int MPI_Finalize(void) >> MPI_Comm_free(&comm); >> comm = MPI_COMM_SELF; >> MPI_Comm_free(&comm); >> + num_attr = 1; /* reset the counter */ >> MPI_was_finalized = 1; >> return MPI_SUCCESS; >> } >> >> >> --Junchao Zhang >> >> >> On Sat, Jun 20, 2020 at 10:48 AM Sam Guo wrote: >> >>> Typo: I mean ?Assuming initializer is only needed once for entire >>> session? >>> >>> On Saturday, June 20, 2020, Sam Guo wrote: >>> >>>> Assuming finalizer is only needed once for entire session(?), I can put >>>> initializer into the static block to call it once but where do I call >>>> finalizer? >>>> >>>> >>>> On Saturday, June 20, 2020, Junchao Zhang >>>> wrote: >>>> >>>>> The counter num_attr should be recycled. But first try to call PETSc >>>>> initialize/Finalize only once to see it fixes the error. >>>>> --Junchao Zhang >>>>> >>>>> >>>>> On Sat, Jun 20, 2020 at 12:48 AM Sam Guo >>>>> wrote: >>>>> >>>>>> To clarify, I call PETSc initialize and PETSc finalize everytime I >>>>>> call SLEPc: >>>>>> >>>>>> PetscInitializeNoPointers(argc,args,nullptr,nullptr); >>>>>> >>>>>> SlepcInitialize(&argc,&args,static_cast(nullptr),help); >>>>>> >>>>>> //calling slepc >>>>>> >>>>>> SlepcFinalize(); >>>>>> >>>>>> PetscFinalize(); >>>>>> >>>>>> >>>>>> >>>>>> On Fri, Jun 19, 2020 at 10:32 PM Sam Guo >>>>>> wrote: >>>>>> >>>>>>> Dear PETSc team, >>>>>>> When I called SLEPc multiple time, I eventually got following >>>>>>> error: >>>>>>> >>>>>>> MPI operation not supported by PETSc's sequential MPI wrappers >>>>>>> [0]PETSC ERROR: #1 PetscInitialize() line 967 in >>>>>>> ../../../petsc/src/sys/objects/pinit.c >>>>>>> [0]PETSC ERROR: #2 SlepcInitialize() line 262 in >>>>>>> ../../../slepc/src/sys/slepcinit.c >>>>>>> [0]PETSC ERROR: #3 SlepcInitializeNoPointers() line 359 in >>>>>>> ../../../slepc/src/sys/slepcinit.c >>>>>>> PETSC ERROR: Logging has not been enabled. >>>>>>> You might have forgotten to call PetscInitialize(). >>>>>>> >>>>>>> I debugged: it is because of following in >>>>>>> petsc/src/sys/mpiuni/mpi.c >>>>>>> >>>>>>> if (num_attr >= MAX_ATTR) >>>>>>> >>>>>>> in function int MPI_Comm_create_keyval(MPI_Copy_function >>>>>>> *copy_fn,MPI_Delete_function *delete_fn,int *keyval,void *extra_state) >>>>>>> >>>>>>> num_attr is declared static and keeps increasing every >>>>>>> time MPI_Comm_create_keyval is called. >>>>>>> >>>>>>> I am using petsc 3.11.3 but found 3.13.2 has the same logic. >>>>>>> >>>>>>> Is this a bug or I didn't use it correctly? >>>>>>> >>>>>>> Thanks, >>>>>>> Sam >>>>>>> >>>>>> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From sam.guo at cd-adapco.com Sat Jun 20 13:49:27 2020 From: sam.guo at cd-adapco.com (Sam Guo) Date: Sat, 20 Jun 2020 11:49:27 -0700 Subject: [petsc-users] PETSc initialization error In-Reply-To: References: <004EA4EB-815F-4EEE-8988-CA2179304157@petsc.dev> Message-ID: Thanks a lot. On Sat, Jun 20, 2020 at 11:40 AM Junchao Zhang wrote: > Sam, > There are more problems. I am working on a fix. Please wait an hour. > Thanks. > > --Junchao Zhang > > > On Sat, Jun 20, 2020 at 1:12 PM Sam Guo wrote: > >> Junchao, >> I debugged: MPI_Finalize is not called for serial. >> >> Barry, >> I tried your patch and it seems better but eventually got following >> error: >> >> [0]PETSC ERROR: #1 PetscCommDuplicate() line 160 in >> ../../../petsc/src/sys/objects/tagm.c >> [0]PETSC ERROR: #2 PetscHeaderCreate_Private() line 64 in >> ../../../petsc/src/sys/objects/inherit.c >> [0]PETSC ERROR: #3 MatCreate() line 91 in >> ../../../petsc/src/mat/utils/gcreate.c >> [0]PETSC ERROR: #4 MatCreateShell() line 787 in >> ../../../petsc/src/mat/impls/shell/shell.c >> [0]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> [0]PETSC ERROR: Null argument, when expecting valid pointer >> [0]PETSC ERROR: Null Object: Parameter # 1 >> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >> for trouble shooting. >> [0]PETSC ERROR: Petsc Release Version 3.11.3, Jun, 26, 2019 >> [0]PETSC ERROR: #6 MatShellSetOperation() line 1052 in >> ../../../petsc/src/mat/impls/shell/shell.c >> >> On Sat, Jun 20, 2020 at 10:24 AM Barry Smith wrote: >> >>> >>> Junchao, >>> >>> This is a good bug fix. It solves the problem when PETSc initialize >>> is called many times. >>> >>> There is another fix you can do to limit PETSc mpiuni running out >>> of attributes inside a single PETSc run: >>> >>> >>> int MPI_Comm_create_keyval(MPI_Copy_function >>> *copy_fn,MPI_Delete_function *delete_fn,int *keyval,void *extra_state) >>> { >>> >>> if (num_attr >= MAX_ATTR){ >>> for (i=0; i>> if (!attr_keyval[i].extra_state) { >>> /* reuse this slot */ >>> attr_keyval[i].extra_state = extra_state; >>> attr_keyval[i.]del = delete_fn; >>> *keyval = i; >>> return MPI_SUCCESS; >>> } >>> } >>> return MPIUni_Abort(MPI_COMM_WORLD,1); >>> } >>> return MPIUni_Abort(MPI_COMM_WORLD,1); >>> attr_keyval[num_attr].extra_state = extra_state; >>> attr_keyval[num_attr].del = delete_fn; >>> *keyval = num_attr++; >>> return MPI_SUCCESS; >>> } >>> >>> This will work if the user creates tons of attributes but is >>> constantly deleting some as they new ones. So long as the number >>> outstanding at one time is < MAX_ATTR) >>> >>> Barry >>> >>> >>> >>> >>> >>> On Jun 20, 2020, at 10:54 AM, Junchao Zhang >>> wrote: >>> >>> I don't understand what your session means. Let's try this patch >>> >>> diff --git a/src/sys/mpiuni/mpi.c b/src/sys/mpiuni/mpi.c >>> index d559a513..c058265d 100644 >>> --- a/src/sys/mpiuni/mpi.c >>> +++ b/src/sys/mpiuni/mpi.c >>> @@ -283,6 +283,7 @@ int MPI_Finalize(void) >>> MPI_Comm_free(&comm); >>> comm = MPI_COMM_SELF; >>> MPI_Comm_free(&comm); >>> + num_attr = 1; /* reset the counter */ >>> MPI_was_finalized = 1; >>> return MPI_SUCCESS; >>> } >>> >>> >>> --Junchao Zhang >>> >>> >>> On Sat, Jun 20, 2020 at 10:48 AM Sam Guo wrote: >>> >>>> Typo: I mean ?Assuming initializer is only needed once for entire >>>> session? >>>> >>>> On Saturday, June 20, 2020, Sam Guo wrote: >>>> >>>>> Assuming finalizer is only needed once for entire session(?), I can >>>>> put initializer into the static block to call it once but where do I call >>>>> finalizer? >>>>> >>>>> >>>>> On Saturday, June 20, 2020, Junchao Zhang >>>>> wrote: >>>>> >>>>>> The counter num_attr should be recycled. But first try to call PETSc >>>>>> initialize/Finalize only once to see it fixes the error. >>>>>> --Junchao Zhang >>>>>> >>>>>> >>>>>> On Sat, Jun 20, 2020 at 12:48 AM Sam Guo >>>>>> wrote: >>>>>> >>>>>>> To clarify, I call PETSc initialize and PETSc finalize everytime I >>>>>>> call SLEPc: >>>>>>> >>>>>>> PetscInitializeNoPointers(argc,args,nullptr,nullptr); >>>>>>> >>>>>>> SlepcInitialize(&argc,&args,static_cast(nullptr),help); >>>>>>> >>>>>>> //calling slepc >>>>>>> >>>>>>> SlepcFinalize(); >>>>>>> >>>>>>> PetscFinalize(); >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Fri, Jun 19, 2020 at 10:32 PM Sam Guo >>>>>>> wrote: >>>>>>> >>>>>>>> Dear PETSc team, >>>>>>>> When I called SLEPc multiple time, I eventually got following >>>>>>>> error: >>>>>>>> >>>>>>>> MPI operation not supported by PETSc's sequential MPI wrappers >>>>>>>> [0]PETSC ERROR: #1 PetscInitialize() line 967 in >>>>>>>> ../../../petsc/src/sys/objects/pinit.c >>>>>>>> [0]PETSC ERROR: #2 SlepcInitialize() line 262 in >>>>>>>> ../../../slepc/src/sys/slepcinit.c >>>>>>>> [0]PETSC ERROR: #3 SlepcInitializeNoPointers() line 359 in >>>>>>>> ../../../slepc/src/sys/slepcinit.c >>>>>>>> PETSC ERROR: Logging has not been enabled. >>>>>>>> You might have forgotten to call PetscInitialize(). >>>>>>>> >>>>>>>> I debugged: it is because of following in >>>>>>>> petsc/src/sys/mpiuni/mpi.c >>>>>>>> >>>>>>>> if (num_attr >= MAX_ATTR) >>>>>>>> >>>>>>>> in function int MPI_Comm_create_keyval(MPI_Copy_function >>>>>>>> *copy_fn,MPI_Delete_function *delete_fn,int *keyval,void *extra_state) >>>>>>>> >>>>>>>> num_attr is declared static and keeps increasing every >>>>>>>> time MPI_Comm_create_keyval is called. >>>>>>>> >>>>>>>> I am using petsc 3.11.3 but found 3.13.2 has the same logic. >>>>>>>> >>>>>>>> Is this a bug or I didn't use it correctly? >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Sam >>>>>>>> >>>>>>> >>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From junchao.zhang at gmail.com Sat Jun 20 18:06:09 2020 From: junchao.zhang at gmail.com (Junchao Zhang) Date: Sat, 20 Jun 2020 18:06:09 -0500 Subject: [petsc-users] PETSc initialization error In-Reply-To: References: <004EA4EB-815F-4EEE-8988-CA2179304157@petsc.dev> Message-ID: Hi, Sam, Sorry I am late. Please try branch jczhang/fix-mpiuni, based off petsc maint, or directly apply the attached patch. Under your PETSC_DIR, git apply patch.txt Thanks. --Junchao Zhang On Sat, Jun 20, 2020 at 1:40 PM Junchao Zhang wrote: > Sam, > There are more problems. I am working on a fix. Please wait an hour. > Thanks. > > --Junchao Zhang > > > On Sat, Jun 20, 2020 at 1:12 PM Sam Guo wrote: > >> Junchao, >> I debugged: MPI_Finalize is not called for serial. >> >> Barry, >> I tried your patch and it seems better but eventually got following >> error: >> >> [0]PETSC ERROR: #1 PetscCommDuplicate() line 160 in >> ../../../petsc/src/sys/objects/tagm.c >> [0]PETSC ERROR: #2 PetscHeaderCreate_Private() line 64 in >> ../../../petsc/src/sys/objects/inherit.c >> [0]PETSC ERROR: #3 MatCreate() line 91 in >> ../../../petsc/src/mat/utils/gcreate.c >> [0]PETSC ERROR: #4 MatCreateShell() line 787 in >> ../../../petsc/src/mat/impls/shell/shell.c >> [0]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> [0]PETSC ERROR: Null argument, when expecting valid pointer >> [0]PETSC ERROR: Null Object: Parameter # 1 >> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >> for trouble shooting. >> [0]PETSC ERROR: Petsc Release Version 3.11.3, Jun, 26, 2019 >> [0]PETSC ERROR: #6 MatShellSetOperation() line 1052 in >> ../../../petsc/src/mat/impls/shell/shell.c >> >> On Sat, Jun 20, 2020 at 10:24 AM Barry Smith wrote: >> >>> >>> Junchao, >>> >>> This is a good bug fix. It solves the problem when PETSc initialize >>> is called many times. >>> >>> There is another fix you can do to limit PETSc mpiuni running out >>> of attributes inside a single PETSc run: >>> >>> >>> int MPI_Comm_create_keyval(MPI_Copy_function >>> *copy_fn,MPI_Delete_function *delete_fn,int *keyval,void *extra_state) >>> { >>> >>> if (num_attr >= MAX_ATTR){ >>> for (i=0; i>> if (!attr_keyval[i].extra_state) { >>> /* reuse this slot */ >>> attr_keyval[i].extra_state = extra_state; >>> attr_keyval[i.]del = delete_fn; >>> *keyval = i; >>> return MPI_SUCCESS; >>> } >>> } >>> return MPIUni_Abort(MPI_COMM_WORLD,1); >>> } >>> return MPIUni_Abort(MPI_COMM_WORLD,1); >>> attr_keyval[num_attr].extra_state = extra_state; >>> attr_keyval[num_attr].del = delete_fn; >>> *keyval = num_attr++; >>> return MPI_SUCCESS; >>> } >>> >>> This will work if the user creates tons of attributes but is >>> constantly deleting some as they new ones. So long as the number >>> outstanding at one time is < MAX_ATTR) >>> >>> Barry >>> >>> >>> >>> >>> >>> On Jun 20, 2020, at 10:54 AM, Junchao Zhang >>> wrote: >>> >>> I don't understand what your session means. Let's try this patch >>> >>> diff --git a/src/sys/mpiuni/mpi.c b/src/sys/mpiuni/mpi.c >>> index d559a513..c058265d 100644 >>> --- a/src/sys/mpiuni/mpi.c >>> +++ b/src/sys/mpiuni/mpi.c >>> @@ -283,6 +283,7 @@ int MPI_Finalize(void) >>> MPI_Comm_free(&comm); >>> comm = MPI_COMM_SELF; >>> MPI_Comm_free(&comm); >>> + num_attr = 1; /* reset the counter */ >>> MPI_was_finalized = 1; >>> return MPI_SUCCESS; >>> } >>> >>> >>> --Junchao Zhang >>> >>> >>> On Sat, Jun 20, 2020 at 10:48 AM Sam Guo wrote: >>> >>>> Typo: I mean ?Assuming initializer is only needed once for entire >>>> session? >>>> >>>> On Saturday, June 20, 2020, Sam Guo wrote: >>>> >>>>> Assuming finalizer is only needed once for entire session(?), I can >>>>> put initializer into the static block to call it once but where do I call >>>>> finalizer? >>>>> >>>>> >>>>> On Saturday, June 20, 2020, Junchao Zhang >>>>> wrote: >>>>> >>>>>> The counter num_attr should be recycled. But first try to call PETSc >>>>>> initialize/Finalize only once to see it fixes the error. >>>>>> --Junchao Zhang >>>>>> >>>>>> >>>>>> On Sat, Jun 20, 2020 at 12:48 AM Sam Guo >>>>>> wrote: >>>>>> >>>>>>> To clarify, I call PETSc initialize and PETSc finalize everytime I >>>>>>> call SLEPc: >>>>>>> >>>>>>> PetscInitializeNoPointers(argc,args,nullptr,nullptr); >>>>>>> >>>>>>> SlepcInitialize(&argc,&args,static_cast(nullptr),help); >>>>>>> >>>>>>> //calling slepc >>>>>>> >>>>>>> SlepcFinalize(); >>>>>>> >>>>>>> PetscFinalize(); >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Fri, Jun 19, 2020 at 10:32 PM Sam Guo >>>>>>> wrote: >>>>>>> >>>>>>>> Dear PETSc team, >>>>>>>> When I called SLEPc multiple time, I eventually got following >>>>>>>> error: >>>>>>>> >>>>>>>> MPI operation not supported by PETSc's sequential MPI wrappers >>>>>>>> [0]PETSC ERROR: #1 PetscInitialize() line 967 in >>>>>>>> ../../../petsc/src/sys/objects/pinit.c >>>>>>>> [0]PETSC ERROR: #2 SlepcInitialize() line 262 in >>>>>>>> ../../../slepc/src/sys/slepcinit.c >>>>>>>> [0]PETSC ERROR: #3 SlepcInitializeNoPointers() line 359 in >>>>>>>> ../../../slepc/src/sys/slepcinit.c >>>>>>>> PETSC ERROR: Logging has not been enabled. >>>>>>>> You might have forgotten to call PetscInitialize(). >>>>>>>> >>>>>>>> I debugged: it is because of following in >>>>>>>> petsc/src/sys/mpiuni/mpi.c >>>>>>>> >>>>>>>> if (num_attr >= MAX_ATTR) >>>>>>>> >>>>>>>> in function int MPI_Comm_create_keyval(MPI_Copy_function >>>>>>>> *copy_fn,MPI_Delete_function *delete_fn,int *keyval,void *extra_state) >>>>>>>> >>>>>>>> num_attr is declared static and keeps increasing every >>>>>>>> time MPI_Comm_create_keyval is called. >>>>>>>> >>>>>>>> I am using petsc 3.11.3 but found 3.13.2 has the same logic. >>>>>>>> >>>>>>>> Is this a bug or I didn't use it correctly? >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Sam >>>>>>>> >>>>>>> >>> -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- diff --git a/src/sys/mpiuni/mpi.c b/src/sys/mpiuni/mpi.c index d559a513..29299b21 100644 --- a/src/sys/mpiuni/mpi.c +++ b/src/sys/mpiuni/mpi.c @@ -23,8 +23,6 @@ void *MPIUNI_TMP = NULL; #define MAX_ATTR 256 #define MAX_COMM 128 -static int MaxComm = 2; - typedef struct { void *attribute_val; int active; @@ -37,10 +35,15 @@ typedef struct { static MPI_Attr_keyval attr_keyval[MAX_ATTR]; static MPI_Attr attr[MAX_COMM][MAX_ATTR]; -static int comm_active[MAX_COMM]; -static int num_attr = 1,mpi_tag_ub = 100000000; +static int comm_active[MAX_COMM]; /* Boolean array indicating which comms are in use */ +static int keyval_active[MAX_ATTR]; /* Boolean array indicating which keyvals are in use */ +static int mpi_tag_ub = 100000000; +static int num_attr = 1; /* Maximal number of keyvals/attributes ever created, including the predefined MPI_TAG_UB attribute. */ +static int MaxComm = 2; /* Maximal number of communicators ever created, including comm_self(1), comm_world(2), but not comm_null(0) */ static void* MPIUNIF_mpi_in_place = 0; +#define CommIdx(comm) ((comm)-1) /* the communicator's interal index used in attr[idx][] and comm_active[idx]. comm_null does not occupy slots in attr[][] */ + #if defined(__cplusplus) extern "C" { #endif @@ -131,20 +134,31 @@ int MPI_Type_get_contents(MPI_Datatype datatype,int max_integers,int max_address */ static int Keyval_setup(void) { - attr[MPI_COMM_WORLD-1][0].active = 1; - attr[MPI_COMM_WORLD-1][0].attribute_val = &mpi_tag_ub; - attr[MPI_COMM_SELF-1][0].active = 1; - attr[MPI_COMM_SELF-1][0].attribute_val = &mpi_tag_ub; + attr[CommIdx(MPI_COMM_WORLD)][0].active = 1; + attr[CommIdx(MPI_COMM_WORLD)][0].attribute_val = &mpi_tag_ub; + attr[CommIdx(MPI_COMM_SELF )][0].active = 1; + attr[CommIdx(MPI_COMM_SELF )][0].attribute_val = &mpi_tag_ub; + keyval_active[0] = 1; return MPI_SUCCESS; } int MPI_Comm_create_keyval(MPI_Copy_function *copy_fn,MPI_Delete_function *delete_fn,int *keyval,void *extra_state) { + int i,keyid; + for (i=1; i= MAX_ATTR) return MPIUni_Abort(MPI_COMM_WORLD,1); + keyid = num_attr++; - attr_keyval[num_attr].extra_state = extra_state; - attr_keyval[num_attr].del = delete_fn; - *keyval = num_attr++; +found: + attr_keyval[keyid].extra_state = extra_state; + attr_keyval[keyid].del = delete_fn; + *keyval = keyid; + keyval_active[keyid] = 1; return MPI_SUCCESS; } @@ -152,26 +166,28 @@ int MPI_Comm_free_keyval(int *keyval) { attr_keyval[*keyval].extra_state = 0; attr_keyval[*keyval].del = 0; - + keyval_active[*keyval] = 0; *keyval = 0; return MPI_SUCCESS; } int MPI_Comm_set_attr(MPI_Comm comm,int keyval,void *attribute_val) { - if (comm-1 < 0 || comm-1 > MaxComm) return MPI_FAILURE; - attr[comm-1][keyval].active = 1; - attr[comm-1][keyval].attribute_val = attribute_val; + int idx = CommIdx(comm); + if (comm < 1 || comm > MaxComm) return MPI_FAILURE; + attr[idx][keyval].active = 1; + attr[idx][keyval].attribute_val = attribute_val; return MPI_SUCCESS; } int MPI_Comm_delete_attr(MPI_Comm comm,int keyval) { - if (comm-1 < 0 || comm-1 > MaxComm) return MPI_FAILURE; - if (attr[comm-1][keyval].active && attr_keyval[keyval].del) { - void *save_attribute_val = attr[comm-1][keyval].attribute_val; - attr[comm-1][keyval].active = 0; - attr[comm-1][keyval].attribute_val = 0; + int idx = CommIdx(comm); + if (comm < 1 || comm > MaxComm) return MPI_FAILURE; + if (attr[idx][keyval].active && attr_keyval[keyval].del) { + void *save_attribute_val = attr[idx][keyval].attribute_val; + attr[idx][keyval].active = 0; + attr[idx][keyval].attribute_val = 0; (*(attr_keyval[keyval].del))(comm,keyval,save_attribute_val,attr_keyval[keyval].extra_state); } return MPI_SUCCESS; @@ -179,72 +195,74 @@ int MPI_Comm_delete_attr(MPI_Comm comm,int keyval) int MPI_Comm_get_attr(MPI_Comm comm,int keyval,void *attribute_val,int *flag) { - if (comm-1 < 0 || comm-1 > MaxComm) return MPI_FAILURE; + int idx = CommIdx(comm); + if (comm < 1 || comm > MaxComm) return MPI_FAILURE; if (!keyval) Keyval_setup(); - *flag = attr[comm-1][keyval].active; - *(void**)attribute_val = attr[comm-1][keyval].attribute_val; + *flag = attr[idx][keyval].active; + *(void**)attribute_val = attr[idx][keyval].attribute_val; return MPI_SUCCESS; } int MPI_Comm_create(MPI_Comm comm,MPI_Group group,MPI_Comm *newcomm) { int j; - if (comm-1 < 0 || comm-1 > MaxComm) return MPI_FAILURE; - for (j=3; j MaxComm) return MPI_FAILURE; + for (j=3; j<=MaxComm; j++) { + if (!comm_active[CommIdx(j)]) { + comm_active[CommIdx(j)] = 1; *newcomm = j; return MPI_SUCCESS; } } - if (MaxComm > MAX_COMM) return MPI_FAILURE; - *newcomm = MaxComm++; - comm_active[*newcomm-1] = 1; + if (MaxComm >= MAX_COMM) return MPI_FAILURE; + *newcomm = ++MaxComm; + comm_active[CommIdx(*newcomm)] = 1; return MPI_SUCCESS; } int MPI_Comm_dup(MPI_Comm comm,MPI_Comm *out) { int j; - if (comm-1 < 0 || comm-1 > MaxComm) return MPI_FAILURE; - for (j=3; j MaxComm) return MPI_FAILURE; + for (j=3; j<=MaxComm; j++) { + if (!comm_active[CommIdx(j)]) { + comm_active[CommIdx(j)] = 1; *out = j; return MPI_SUCCESS; } } - if (MaxComm > MAX_COMM) return MPI_FAILURE; - *out = MaxComm++; - comm_active[*out-1] = 1; + if (MaxComm >= MAX_COMM) return MPI_FAILURE; + *out = ++MaxComm; + comm_active[CommIdx(*out)] = 1; return MPI_SUCCESS; } int MPI_Comm_free(MPI_Comm *comm) { int i; + int idx = CommIdx(*comm); - if (*comm-1 < 0 || *comm-1 > MaxComm) return MPI_FAILURE; + if (*comm < 1 || *comm > MaxComm) return MPI_FAILURE; for (i=0; i= 3) comm_active[*comm-1] = 0; + if (*comm >= 3) comm_active[idx] = 0; *comm = 0; return MPI_SUCCESS; } int MPI_Comm_size(MPI_Comm comm, int *size) { - if (comm-1 < 0 || comm-1 > MaxComm) return MPI_FAILURE; + if (comm < 1 || comm > MaxComm) return MPI_FAILURE; *size=1; return MPI_SUCCESS; } int MPI_Comm_rank(MPI_Comm comm, int *rank) { - if (comm-1 < 0 || comm-1 > MaxComm) return MPI_FAILURE; + if (comm < 1 || comm > MaxComm) return MPI_FAILURE; *rank=0; return MPI_SUCCESS; } @@ -269,7 +287,7 @@ static int MPI_was_finalized = 0; int MPI_Init(int *argc, char ***argv) { if (MPI_was_initialized) return MPI_FAILURE; - if (MPI_was_finalized) return MPI_FAILURE; + if (MPI_was_finalized) return MPI_FAILURE; /* MPI standard: once MPI_FINALIZE returns, no MPI routine (not even MPI_INIT) may be called, except ... */ MPI_was_initialized = 1; return MPI_SUCCESS; } @@ -283,6 +301,17 @@ int MPI_Finalize(void) MPI_Comm_free(&comm); comm = MPI_COMM_SELF; MPI_Comm_free(&comm); +#if defined(PETSC_USE_DEBUG) + { + int i; + for (i=3; i<=MaxComm; i++) { + if (comm_active[CommIdx(i)]) printf("Warning: MPI communicator %d is not freed before MPI_Finalize()\n", i); + } + } +#endif + /* reset counters */ + MaxComm = 2; + num_attr = 1; MPI_was_finalized = 1; return MPI_SUCCESS; } diff --git a/src/sys/tests/ex53.c b/src/sys/tests/ex53.c new file mode 100644 index 00000000..4dd598ff --- /dev/null +++ b/src/sys/tests/ex53.c @@ -0,0 +1,55 @@ +static char help[] = "Test resource recyling and MPI_Comm and keyval creation in mpi or mpiuni\n"; + +#include + +#define CHKMPIERR(ierr) do {if (ierr) MPI_Abort(MPI_COMM_WORLD, ierr);} while(0) + +int main(int argc,char **argv) +{ + PetscErrorCode ierr; + PetscInt i; + PetscMPIInt key1,key2,attr1=100,attr2=200,*attr,flag; + MPI_Comm newcomm; + + ierr = MPI_Init(&argc,&argv);CHKMPIERR(ierr); + + /* Repeated keyval create/free should not blow up MPI */ + for (i=0; i<500; i++) { + ierr = MPI_Comm_create_keyval(MPI_COMM_NULL_COPY_FN,MPI_COMM_NULL_DELETE_FN,&key1,NULL);CHKMPIERR(ierr); + ierr = MPI_Comm_free_keyval(&key1);CHKMPIERR(ierr); + } + + /* The following keyval/attr code exposes a bug in old mpiuni code, where it had wrong newcomm returned in MPI_Comm_dup. */ + ierr = MPI_Comm_create_keyval(MPI_COMM_NULL_COPY_FN,MPI_COMM_NULL_DELETE_FN,&key1,NULL);CHKMPIERR(ierr); + ierr = MPI_Comm_create_keyval(MPI_COMM_NULL_COPY_FN,MPI_COMM_NULL_DELETE_FN,&key2,NULL);CHKMPIERR(ierr); + ierr = MPI_Comm_dup(MPI_COMM_WORLD,&newcomm);CHKMPIERR(ierr); + if (MPI_COMM_WORLD == newcomm) printf("Error: wrong newcomm returned by MPI_Comm_dup()\n"); + + ierr = MPI_Comm_set_attr(MPI_COMM_WORLD,key1,&attr1);CHKMPIERR(ierr); + ierr = MPI_Comm_set_attr(newcomm,key2,&attr2);CHKMPIERR(ierr); + ierr = MPI_Comm_get_attr(newcomm,key1,&attr,&flag);CHKMPIERR(ierr); + if (flag) printf("Error: newcomm should not have attribute for keyval %d\n", (int)key1); + ierr = MPI_Comm_get_attr(MPI_COMM_WORLD,key1,&attr,&flag);CHKMPIERR(ierr); + if (*attr != attr1) printf("Error: expected attribute %d, but got %d\n", (int)attr1, (int)*attr); + ierr = MPI_Comm_get_attr(newcomm,key2,&attr,&flag);CHKMPIERR(ierr); + if (*attr != attr2) printf("Error: expected attribute %d, but got %d\n", (int)attr2, (int)*attr); + + ierr = MPI_Comm_delete_attr(MPI_COMM_WORLD,key1);CHKMPIERR(ierr); + ierr = MPI_Comm_delete_attr(newcomm,key2);CHKMPIERR(ierr); + ierr = MPI_Comm_free_keyval(&key1);CHKMPIERR(ierr); + ierr = MPI_Comm_free_keyval(&key2);CHKMPIERR(ierr); + ierr = MPI_Comm_free(&newcomm);CHKMPIERR(ierr); + + /* Init/Finalize PETSc multiple times when MPI is initialized */ + for (i=0; i<500; i++) { + ierr = PetscInitialize(&argc,&argv,(char*)0,help);if (ierr) return ierr; + ierr = PetscFinalize();if (ierr) return ierr; + } + + ierr = MPI_Finalize(); + return ierr; +} + +/*TEST + test: +TEST*/ diff --git a/src/sys/tests/makefile b/src/sys/tests/makefile index 5bd93ddc..5dcdb9bf 100644 --- a/src/sys/tests/makefile +++ b/src/sys/tests/makefile @@ -8,7 +8,7 @@ EXAMPLESC = ex1.c ex2.c ex3.c ex6.c ex7.c ex8.c ex9.c ex10.c ex11.c ex12.c ex14.c ex16.c ex18.c ex19.c ex20.c ex21.c \ ex22.c ex23.c ex24.c ex25.c ex26.c ex27.c ex28.c ex29.c ex30.c ex31.c ex32.c ex35.c ex37.c \ ex44.cxx ex45.cxx ex46.cxx ex47.c ex49.c \ - ex50.c ex51.c ex52.c + ex50.c ex51.c ex52.c ex53.c EXAMPLESF = ex1f.F90 ex5f.F ex6f.F ex17f.F ex36f.F90 ex38f.F90 ex47f.F90 ex48f90.F90 ex49f.F90 MANSEC = Sys diff --git a/src/sys/tests/output/ex53_1.out b/src/sys/tests/output/ex53_1.out new file mode 100644 index 00000000..e69de29b From junchao.zhang at gmail.com Sat Jun 20 18:23:36 2020 From: junchao.zhang at gmail.com (Junchao Zhang) Date: Sat, 20 Jun 2020 18:23:36 -0500 Subject: [petsc-users] PETSc initialization error In-Reply-To: <004EA4EB-815F-4EEE-8988-CA2179304157@petsc.dev> References: <004EA4EB-815F-4EEE-8988-CA2179304157@petsc.dev> Message-ID: On Sat, Jun 20, 2020 at 12:24 PM Barry Smith wrote: > > Junchao, > > This is a good bug fix. It solves the problem when PETSc initialize > is called many times. > > There is another fix you can do to limit PETSc mpiuni running out of > attributes inside a single PETSc run: > > > int MPI_Comm_create_keyval(MPI_Copy_function *copy_fn,MPI_Delete_function > *delete_fn,int *keyval,void *extra_state) > { > > if (num_attr >= MAX_ATTR){ > for (i=0; i if (!attr_keyval[i].extra_state) { > attr_keyval[i].extra_state is provided by user (could be NULL). We can not rely on it. > /* reuse this slot */ > attr_keyval[i].extra_state = extra_state; > attr_keyval[i.]del = delete_fn; > *keyval = i; > return MPI_SUCCESS; > } > } > return MPIUni_Abort(MPI_COMM_WORLD,1); > } > return MPIUni_Abort(MPI_COMM_WORLD,1); > attr_keyval[num_attr].extra_state = extra_state; > attr_keyval[num_attr].del = delete_fn; > *keyval = num_attr++; > return MPI_SUCCESS; > } > > This will work if the user creates tons of attributes but is constantly > deleting some as they new ones. So long as the number outstanding at one > time is < MAX_ATTR) > > Barry > > > > > > On Jun 20, 2020, at 10:54 AM, Junchao Zhang > wrote: > > I don't understand what your session means. Let's try this patch > > diff --git a/src/sys/mpiuni/mpi.c b/src/sys/mpiuni/mpi.c > index d559a513..c058265d 100644 > --- a/src/sys/mpiuni/mpi.c > +++ b/src/sys/mpiuni/mpi.c > @@ -283,6 +283,7 @@ int MPI_Finalize(void) > MPI_Comm_free(&comm); > comm = MPI_COMM_SELF; > MPI_Comm_free(&comm); > + num_attr = 1; /* reset the counter */ > MPI_was_finalized = 1; > return MPI_SUCCESS; > } > > > --Junchao Zhang > > > On Sat, Jun 20, 2020 at 10:48 AM Sam Guo wrote: > >> Typo: I mean ?Assuming initializer is only needed once for entire session? >> >> On Saturday, June 20, 2020, Sam Guo wrote: >> >>> Assuming finalizer is only needed once for entire session(?), I can put >>> initializer into the static block to call it once but where do I call >>> finalizer? >>> >>> >>> On Saturday, June 20, 2020, Junchao Zhang >>> wrote: >>> >>>> The counter num_attr should be recycled. But first try to call PETSc >>>> initialize/Finalize only once to see it fixes the error. >>>> --Junchao Zhang >>>> >>>> >>>> On Sat, Jun 20, 2020 at 12:48 AM Sam Guo wrote: >>>> >>>>> To clarify, I call PETSc initialize and PETSc finalize everytime I >>>>> call SLEPc: >>>>> >>>>> PetscInitializeNoPointers(argc,args,nullptr,nullptr); >>>>> >>>>> SlepcInitialize(&argc,&args,static_cast(nullptr),help); >>>>> >>>>> //calling slepc >>>>> >>>>> SlepcFinalize(); >>>>> >>>>> PetscFinalize(); >>>>> >>>>> >>>>> >>>>> On Fri, Jun 19, 2020 at 10:32 PM Sam Guo >>>>> wrote: >>>>> >>>>>> Dear PETSc team, >>>>>> When I called SLEPc multiple time, I eventually got following >>>>>> error: >>>>>> >>>>>> MPI operation not supported by PETSc's sequential MPI wrappers >>>>>> [0]PETSC ERROR: #1 PetscInitialize() line 967 in >>>>>> ../../../petsc/src/sys/objects/pinit.c >>>>>> [0]PETSC ERROR: #2 SlepcInitialize() line 262 in >>>>>> ../../../slepc/src/sys/slepcinit.c >>>>>> [0]PETSC ERROR: #3 SlepcInitializeNoPointers() line 359 in >>>>>> ../../../slepc/src/sys/slepcinit.c >>>>>> PETSC ERROR: Logging has not been enabled. >>>>>> You might have forgotten to call PetscInitialize(). >>>>>> >>>>>> I debugged: it is because of following in petsc/src/sys/mpiuni/mpi.c >>>>>> >>>>>> if (num_attr >= MAX_ATTR) >>>>>> >>>>>> in function int MPI_Comm_create_keyval(MPI_Copy_function >>>>>> *copy_fn,MPI_Delete_function *delete_fn,int *keyval,void *extra_state) >>>>>> >>>>>> num_attr is declared static and keeps increasing every >>>>>> time MPI_Comm_create_keyval is called. >>>>>> >>>>>> I am using petsc 3.11.3 but found 3.13.2 has the same logic. >>>>>> >>>>>> Is this a bug or I didn't use it correctly? >>>>>> >>>>>> Thanks, >>>>>> Sam >>>>>> >>>>> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sam.guo at cd-adapco.com Sat Jun 20 21:35:10 2020 From: sam.guo at cd-adapco.com (Sam Guo) Date: Sat, 20 Jun 2020 19:35:10 -0700 Subject: [petsc-users] PETSc initialization error In-Reply-To: References: <004EA4EB-815F-4EEE-8988-CA2179304157@petsc.dev> Message-ID: Hi Junchao, Your patch works. Thanks, Sam On Sat, Jun 20, 2020 at 4:23 PM Junchao Zhang wrote: > > > On Sat, Jun 20, 2020 at 12:24 PM Barry Smith wrote: > >> >> Junchao, >> >> This is a good bug fix. It solves the problem when PETSc initialize >> is called many times. >> >> There is another fix you can do to limit PETSc mpiuni running out of >> attributes inside a single PETSc run: >> >> >> int MPI_Comm_create_keyval(MPI_Copy_function *copy_fn,MPI_Delete_function >> *delete_fn,int *keyval,void *extra_state) >> { >> >> if (num_attr >= MAX_ATTR){ >> for (i=0; i> if (!attr_keyval[i].extra_state) { >> > attr_keyval[i].extra_state is provided by user (could be NULL). We can not > rely on it. > >> /* reuse this slot */ >> attr_keyval[i].extra_state = extra_state; >> attr_keyval[i.]del = delete_fn; >> *keyval = i; >> return MPI_SUCCESS; >> } >> } >> return MPIUni_Abort(MPI_COMM_WORLD,1); >> } >> return MPIUni_Abort(MPI_COMM_WORLD,1); >> attr_keyval[num_attr].extra_state = extra_state; >> attr_keyval[num_attr].del = delete_fn; >> *keyval = num_attr++; >> return MPI_SUCCESS; >> } >> >> This will work if the user creates tons of attributes but is constantly >> deleting some as they new ones. So long as the number outstanding at one >> time is < MAX_ATTR) >> >> Barry >> >> >> >> >> >> On Jun 20, 2020, at 10:54 AM, Junchao Zhang >> wrote: >> >> I don't understand what your session means. Let's try this patch >> >> diff --git a/src/sys/mpiuni/mpi.c b/src/sys/mpiuni/mpi.c >> index d559a513..c058265d 100644 >> --- a/src/sys/mpiuni/mpi.c >> +++ b/src/sys/mpiuni/mpi.c >> @@ -283,6 +283,7 @@ int MPI_Finalize(void) >> MPI_Comm_free(&comm); >> comm = MPI_COMM_SELF; >> MPI_Comm_free(&comm); >> + num_attr = 1; /* reset the counter */ >> MPI_was_finalized = 1; >> return MPI_SUCCESS; >> } >> >> >> --Junchao Zhang >> >> >> On Sat, Jun 20, 2020 at 10:48 AM Sam Guo wrote: >> >>> Typo: I mean ?Assuming initializer is only needed once for entire >>> session? >>> >>> On Saturday, June 20, 2020, Sam Guo wrote: >>> >>>> Assuming finalizer is only needed once for entire session(?), I can put >>>> initializer into the static block to call it once but where do I call >>>> finalizer? >>>> >>>> >>>> On Saturday, June 20, 2020, Junchao Zhang >>>> wrote: >>>> >>>>> The counter num_attr should be recycled. But first try to call PETSc >>>>> initialize/Finalize only once to see it fixes the error. >>>>> --Junchao Zhang >>>>> >>>>> >>>>> On Sat, Jun 20, 2020 at 12:48 AM Sam Guo >>>>> wrote: >>>>> >>>>>> To clarify, I call PETSc initialize and PETSc finalize everytime I >>>>>> call SLEPc: >>>>>> >>>>>> PetscInitializeNoPointers(argc,args,nullptr,nullptr); >>>>>> >>>>>> SlepcInitialize(&argc,&args,static_cast(nullptr),help); >>>>>> >>>>>> //calling slepc >>>>>> >>>>>> SlepcFinalize(); >>>>>> >>>>>> PetscFinalize(); >>>>>> >>>>>> >>>>>> >>>>>> On Fri, Jun 19, 2020 at 10:32 PM Sam Guo >>>>>> wrote: >>>>>> >>>>>>> Dear PETSc team, >>>>>>> When I called SLEPc multiple time, I eventually got following >>>>>>> error: >>>>>>> >>>>>>> MPI operation not supported by PETSc's sequential MPI wrappers >>>>>>> [0]PETSC ERROR: #1 PetscInitialize() line 967 in >>>>>>> ../../../petsc/src/sys/objects/pinit.c >>>>>>> [0]PETSC ERROR: #2 SlepcInitialize() line 262 in >>>>>>> ../../../slepc/src/sys/slepcinit.c >>>>>>> [0]PETSC ERROR: #3 SlepcInitializeNoPointers() line 359 in >>>>>>> ../../../slepc/src/sys/slepcinit.c >>>>>>> PETSC ERROR: Logging has not been enabled. >>>>>>> You might have forgotten to call PetscInitialize(). >>>>>>> >>>>>>> I debugged: it is because of following in >>>>>>> petsc/src/sys/mpiuni/mpi.c >>>>>>> >>>>>>> if (num_attr >= MAX_ATTR) >>>>>>> >>>>>>> in function int MPI_Comm_create_keyval(MPI_Copy_function >>>>>>> *copy_fn,MPI_Delete_function *delete_fn,int *keyval,void *extra_state) >>>>>>> >>>>>>> num_attr is declared static and keeps increasing every >>>>>>> time MPI_Comm_create_keyval is called. >>>>>>> >>>>>>> I am using petsc 3.11.3 but found 3.13.2 has the same logic. >>>>>>> >>>>>>> Is this a bug or I didn't use it correctly? >>>>>>> >>>>>>> Thanks, >>>>>>> Sam >>>>>>> >>>>>> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From hongzhang at anl.gov Sun Jun 21 20:23:04 2020 From: hongzhang at anl.gov (Zhang, Hong) Date: Mon, 22 Jun 2020 01:23:04 +0000 Subject: [petsc-users] SNES Line search: fixing the non zero solution values of dirichlet nodes during line search In-Reply-To: References: Message-ID: <6F862511-22D9-42D0-9802-1CC36E80182C@anl.gov> Can you elaborate a bit on how you "set the RHS as increment of (Dirichlet_value-solution_value)?? It seems that the boundary condition is imposed as algebraic equations (e.g. 0 = u(t)-b(t) ), and the boundary nodes are included in the computational domain. This approach should work theoretically and is used in some TS examples such as ex17.c, you might want to check if the algebraic equations are implemented properly in your IFunction() and IJacobian(). An alternative approach you can try is to remove the boundary nodes from the unknown variables, see the example ex25.c. In this way you solve ODEs instead of DAEs and do not need to worry about SNES modifying certain values. Hong (Mr.) On Jun 19, 2020, at 5:15 PM, Ashish Patel > wrote: Dear PETSc users, We use PETSc as part of a finite element method program and we are trying to properly implement Dirichlet boundary conditions for non-linear, transient problems. We find that when we use a line search method it also changes the non-zero solution value of Dirichlet nodes as it steps through the line search iteration. I was wondering if there is a way to freeze some index sets of a vector from changing during the line search operation? We are using the TS framework to setup the problem and use 'MatZeroRowsColumns' to set the diagonal of the jacobian to 1 for the dirichlet nodes and set the RHS as increment of (Dirichlet_value-solution_value). This works when the line search method is turned off by using '-snes_linesearch_type basic' however using the default 'bt' linesearch, the TS diverges with error shown below. In a separate implementation if we overwrote the dirichlet nodes of the solution vector in TS residual function with the Dirichlet values then the 'bt' line search method converged to the right solution. However we would like to avoid modifying the internal PETSc vector in our implementation. 0 TS dt 1. time 0. 0 SNES Function norm 2.378549386020e+03 Line search: gnorm after quadratic fit 4.369235425165e+03 Line search: Cubic step no good, shrinking lambda, current gnorm 2.385369069060e+03 lambda=1.0000000000000002e-02 Line search: Cubically determined step, current gnorm 2.373925008934e+03 lambda=3.8846250444606093e-03 1 SNES Function norm 2.373925008934e+03 Line search: gnorm after quadratic fit 5.006914179995e+03 Line search: Cubic step no good, shrinking lambda, current gnorm 2.420957096780e+03 lambda=1.0000000000000002e-02 Line search: Cubic step no good, shrinking lambda, current gnorm 2.376034946750e+03 lambda=1.6129422079664700e-03 Line search: Cubic step no good, shrinking lambda, current gnorm 2.374313344729e+03 lambda=4.8465026740690043e-04 Line search: Cubic step no good, shrinking lambda, current gnorm 2.373999473242e+03 lambda=1.5857251532828948e-04 Line search: Cubically determined step, current gnorm 2.373921668024e+03 lambda=5.8116507162753387e-05 2 SNES Function norm 2.373921668024e+03 Line search: gnorm after quadratic fit 4.771035112853e+03 Line search: Cubic step no good, shrinking lambda, current gnorm 2.410650718394e+03 lambda=1.0000000000000002e-02 Line search: Cubic step no good, shrinking lambda, current gnorm 2.375104094198e+03 lambda=1.8983783738011522e-03 Line search: Cubic step no good, shrinking lambda, current gnorm 2.374049151562e+03 lambda=7.0688528086485479e-04 Line search: Cubic step no good, shrinking lambda, current gnorm 2.373935090907e+03 lambda=2.9132722794896019e-04 Line search: Cubically determined step, current gnorm 2.373921032081e+03 lambda=1.2527602265373028e-04 3 SNES Function norm 2.373921032081e+03 Line search: gnorm after quadratic fit 5.117914832660e+03 Line search: Cubic step no good, shrinking lambda, current gnorm 2.422635362094e+03 lambda=1.0000000000000002e-02 Line search: Cubic step no good, shrinking lambda, current gnorm 2.375923870970e+03 lambda=1.5769887508300913e-03 Line search: Cubic step no good, shrinking lambda, current gnorm 2.374287081592e+03 lambda=4.8018017100729705e-04 Line search: Cubic step no good, shrinking lambda, current gnorm 2.373999131160e+03 lambda=1.5908966655977892e-04 Line search: Cubic step no good, shrinking lambda, current gnorm 2.373930608274e+03 lambda=5.7603977147935371e-05 Line search: Cubic step no good, shrinking lambda, current gnorm 2.373922022038e+03 lambda=2.3884787050507805e-05 Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921396636e+03 lambda=1.0282405393471519e-05 Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921173719e+03 lambda=4.3868704034012554e-06 Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921091304e+03 lambda=1.8774991151727107e-06 Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921057168e+03 lambda=8.0331628193813397e-07 Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921042769e+03 lambda=3.4377811174560817e-07 Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921036645e+03 lambda=1.4712421886880395e-07 Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921034032e+03 lambda=6.2965363212312132e-08 Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921032916e+03 lambda=2.6947718250469261e-08 Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921032438e+03 lambda=1.1533043989179318e-08 Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921032233e+03 lambda=4.9359018284334258e-09 Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921032146e+03 lambda=2.1124641857409436e-09 Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921032109e+03 lambda=9.0409137304143975e-10 Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921032093e+03 lambda=3.8693264359674819e-10 Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921032086e+03 lambda=1.6559911204766632e-10 Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921032083e+03 lambda=7.0873187020534353e-11 Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921032081e+03 lambda=3.0332192901757729e-11 Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921032081e+03 lambda=1.2981600435512875e-11 Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921032081e+03 lambda=5.5559212521628090e-12 Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921032081e+03 lambda=2.3777380405336958e-12 Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921032081e+03 lambda=1.0176091649134191e-12 Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921032081e+03 lambda=4.3555419789721080e-13 Line search: unable to find good step length! After 27 tries Line search: fnorm=2.3739210320805191e+03, gnorm=2.3739210320805323e+03, ynorm=8.5698020038772756e+03, minlambda=9.9999999999999998e-13, lambda=4.3555419789721080e-13, initial slope=-5.6355010665542409e+06 SNES Object: 1 MPI processes type: newtonls maximum iterations=50, maximum function evaluations=10000 tolerances: relative=1e-08, absolute=1e-50, solution=1e-08 total number of linear solver iterations=4 total number of function evaluations=48 norm schedule ALWAYS SNESLineSearch Object: 1 MPI processes type: bt interpolation: cubic alpha=1.000000e-04 maxstep=1.000000e+08, minlambda=1.000000e-12 tolerances: relative=1.000000e-08, absolute=1.000000e-15, lambda=1.000000e-08 maximum iterations=40 KSP Object: 1 MPI processes type: preonly maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000. left preconditioning using NONE norm type for convergence test PC Object: 1 MPI processes type: cholesky out-of-place factorization tolerance for zero pivot 2.22045e-14 matrix ordering: natural factor fill ratio given 0., needed 0. Factored matrix follows: Mat Object: 1 MPI processes type: mumps rows=38154, cols=38154 package used to perform factorization: mumps total: nonzeros=7080060, allocated nonzeros=7080060 total number of mallocs used during MatSetValues calls=0 MUMPS run parameters: SYM (matrix type): 2 PAR (host participation): 1 ICNTL(1) (output for error): 6 ICNTL(2) (output of diagnostic msg): 0 ICNTL(3) (output for global info): 0 ICNTL(4) (level of printing): 0 ICNTL(5) (input mat struct): 0 ICNTL(6) (matrix prescaling): 7 ICNTL(7) (sequential matrix ordering):7 ICNTL(8) (scaling strategy): 77 ICNTL(10) (max num of refinements): 0 ICNTL(11) (error analysis): 0 ICNTL(12) (efficiency control): 0 ICNTL(13) (efficiency control): 0 ICNTL(14) (percentage of estimated workspace increase): 20 ICNTL(18) (input mat struct): 0 ICNTL(19) (Schur complement info): 0 ICNTL(20) (rhs sparse pattern): 0 ICNTL(21) (solution struct): 0 ICNTL(22) (in-core/out-of-core facility): 0 ICNTL(23) (max size of memory can be allocated locally):0 ICNTL(24) (detection of null pivot rows): 0 ICNTL(25) (computation of a null space basis): 0 ICNTL(26) (Schur options for rhs or solution): 0 ICNTL(27) (experimental parameter): -32 ICNTL(28) (use parallel or sequential ordering): 1 ICNTL(29) (parallel ordering): 0 ICNTL(30) (user-specified set of entries in inv(A)): 0 ICNTL(31) (factors is discarded in the solve phase): 0 ICNTL(33) (compute determinant): 0 ICNTL(35) (activate BLR based factorization): 0 ICNTL(36) (choice of BLR factorization variant): 0 ICNTL(38) (estimated compression rate of LU factors): 333 CNTL(1) (relative pivoting threshold): 0.01 CNTL(2) (stopping criterion of refinement): 1.49012e-08 CNTL(3) (absolute pivoting threshold): 0. CNTL(4) (value of static pivoting): -1. CNTL(5) (fixation for null pivots): 0. CNTL(7) (dropping parameter for BLR): 0. RINFO(1) (local estimated flops for the elimination after analysis): [0] 2.73979e+09 RINFO(2) (local estimated flops for the assembly after factorization): [0] 1.08826e+07 RINFO(3) (local estimated flops for the elimination after factorization): [0] 2.73979e+09 INFO(15) (estimated size of (in MB) MUMPS internal data for running numerical factorization): [0] 94 INFO(16) (size of (in MB) MUMPS internal data used during numerical factorization): [0] 94 INFO(23) (num of pivots eliminated on this processor after factorization): [0] 38154 RINFOG(1) (global estimated flops for the elimination after analysis): 2.73979e+09 RINFOG(2) (global estimated flops for the assembly after factorization): 1.08826e+07 RINFOG(3) (global estimated flops for the elimination after factorization): 2.73979e+09 (RINFOG(12) RINFOG(13))*2^INFOG(34) (determinant): (0.,0.)*(2^0) INFOG(3) (estimated real workspace for factors on all processors after analysis): 8377336 INFOG(4) (estimated integer workspace for factors on all processors after analysis): 447902 INFOG(5) (estimated maximum front size in the complete tree): 990 INFOG(6) (number of nodes in the complete tree): 2730 INFOG(7) (ordering option effectively use after analysis): 5 INFOG(8) (structural symmetry in percent of the permuted matrix after analysis): 100 INFOG(9) (total real/complex workspace to store the matrix factors after factorization): 8377336 INFOG(10) (total integer space store the matrix factors after factorization): 447902 INFOG(11) (order of largest frontal matrix after factorization): 990 INFOG(12) (number of off-diagonal pivots): 10 INFOG(13) (number of delayed pivots after factorization): 0 INFOG(14) (number of memory compress after factorization): 0 INFOG(15) (number of steps of iterative refinement after solution): 0 INFOG(16) (estimated size (in MB) of all MUMPS internal data for factorization after analysis: value on the most memory consuming processor): 94 INFOG(17) (estimated size of all MUMPS internal data for factorization after analysis: sum over all processors): 94 INFOG(18) (size of all MUMPS internal data allocated during factorization: value on the most memory consuming processor): 94 INFOG(19) (size of all MUMPS internal data allocated during factorization: sum over all processors): 94 INFOG(20) (estimated number of entries in the factors): 7080060 INFOG(21) (size in MB of memory effectively used during factorization - value on the most memory consuming processor): 80 INFOG(22) (size in MB of memory effectively used during factorization - sum over all processors): 80 INFOG(23) (after analysis: value of ICNTL(6) effectively used): 0 INFOG(24) (after analysis: value of ICNTL(12) effectively used): 1 INFOG(25) (after factorization: number of pivots modified by static pivoting): 0 INFOG(28) (after factorization: number of null pivots encountered): 0 INFOG(29) (after factorization: effective number of entries in the factors (sum over all processors)): 7080060 INFOG(30, 31) (after solution: size in Mbytes of memory used during solution phase): 92, 92 INFOG(32) (after analysis: type of analysis done): 1 INFOG(33) (value used for ICNTL(8)): 7 INFOG(34) (exponent of the determinant if determinant is requested): 0 INFOG(35) (after factorization: number of entries taking into account BLR factor compression - sum over all processors): 7080060 INFOG(36) (after analysis: estimated size of all MUMPS internal data for running BLR in-core - value on the most memory consuming processor): 0 INFOG(37) (after analysis: estimated size of all MUMPS internal data for running BLR in-core - sum over all processors): 0 INFOG(38) (after analysis: estimated size of all MUMPS internal data for running BLR out-of-core - value on the most memory consuming processor): 0 INFOG(39) (after analysis: estimated size of all MUMPS internal data for running BLR out-of-core - sum over all processors): 0 linear system matrix = precond matrix: Mat Object: 1 MPI processes type: seqaij rows=38154, cols=38154 total: nonzeros=973446, allocated nonzeros=973446 total number of mallocs used during MatSetValues calls=0 not using I-node routines [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: [0]PETSC ERROR: TSStep has failed due to DIVERGED_NONLINEAR_SOLVE, increase -ts_max_snes_failures or make negative to attempt recovery [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.12.3, Jan, 03, 2020 Thanks, Ashish Scientific Computing Division OnScale CA, USA -------------- next part -------------- An HTML attachment was scrubbed... URL: From eda.oktay at metu.edu.tr Mon Jun 22 04:20:19 2020 From: eda.oktay at metu.edu.tr (Eda Oktay) Date: Mon, 22 Jun 2020 12:20:19 +0300 Subject: [petsc-users] MatFindOffBlockDiagonalEntries error In-Reply-To: References: Message-ID: I am so sorry for the inconvenience, the last error I sent you was a different one, I forgot to change the codes. I am sending you the actual error where I used MatFindOffBlockDiagonalEntries again in the newest version. [1]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [1]PETSC ERROR: Null argument, when expecting valid pointer [2]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [2]PETSC ERROR: Null argument, when expecting valid pointer [2]PETSC ERROR: Null Object: Parameter # 1 [2]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [2]PETSC ERROR: Petsc Release Version 3.13.2, Jun 02, 2020 [2]PETSC ERROR: [3]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [3]PETSC ERROR: Null argument, when expecting valid pointer [3]PETSC ERROR: Null Object: Parameter # 1 [3]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [3]PETSC ERROR: Petsc Release Version 3.13.2, Jun 02, 2020 [3]PETSC ERROR: ./approx_cut_deneme_clustering_son_final_edgecut_without_parmetis on a arch-linux2-c-debug named b342.wls.metu.edu.tr by edaoktay Mon Jun 22 12:11:44 2020 [1]PETSC ERROR: Null Object: Parameter # 1 [1]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [1]PETSC ERROR: Petsc Release Version 3.13.2, Jun 02, 2020 [1]PETSC ERROR: ./approx_cut_deneme_clustering_son_final_edgecut_without_parmetis on a arch-linux2-c-debug named b342.wls.metu.edu.tr by edaoktay Mon Jun 22 12:11:44 2020 [1]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --download-mpich --download-openblas --download-slepc --download-metis --download-parmetis --download-chaco --with-X=1 [1]PETSC ERROR: #1 ISGetSize() line 1101 in /home/edaoktay/petsc-3.13.2/src/vec/is/is/interface/index.c ./approx_cut_deneme_clustering_son_final_edgecut_without_parmetis on a arch-linux2-c-debug named b342.wls.metu.edu.tr by edaoktay Mon Jun 22 12:11:44 2020 [2]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --download-mpich --download-openblas --download-slepc --download-metis --download-parmetis --download-chaco --with-X=1 [2]PETSC ERROR: #1 ISGetSize() line 1101 in /home/edaoktay/petsc-3.13.2/src/vec/is/is/interface/index.c [2]PETSC ERROR: [3]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --download-mpich --download-openblas --download-slepc --download-metis --download-parmetis --download-chaco --with-X=1 [3]PETSC ERROR: #1 ISGetSize() line 1101 in /home/edaoktay/petsc-3.13.2/src/vec/is/is/interface/index.c [3]PETSC ERROR: #2 MatFindOffBlockDiagonalEntries_MPIAIJ() line 229 in /home/edaoktay/petsc-3.13.2/src/mat/impls/aij/mpi/mpiaij.c [1]PETSC ERROR: #2 MatFindOffBlockDiagonalEntries_MPIAIJ() line 229 in /home/edaoktay/petsc-3.13.2/src/mat/impls/aij/mpi/mpiaij.c [1]PETSC ERROR: #3 MatFindOffBlockDiagonalEntries() line 9824 in /home/edaoktay/petsc-3.13.2/src/mat/interface/matrix.c #2 MatFindOffBlockDiagonalEntries_MPIAIJ() line 229 in /home/edaoktay/petsc-3.13.2/src/mat/impls/aij/mpi/mpiaij.c [2]PETSC ERROR: #3 MatFindOffBlockDiagonalEntries() line 9824 in /home/edaoktay/petsc-3.13.2/src/mat/interface/matrix.c [3]PETSC ERROR: #3 MatFindOffBlockDiagonalEntries() line 9824 in /home/edaoktay/petsc-3.13.2/src/mat/interface/matrix.c I think this is again the same error when I used an older version. Again, even though I am using a valid matrix and an index set, I don't know why I am getting this error. About the small test example, I couldn't understand what Junchao ment but this is the first time I am trying to use this function and my current matrix is quite small 72*72. Mat Ais; MatCreateSubMatrix(A,idUi,idUi,MAT_INITIAL_MATRIX,&Ais); ISSetPermutation(idUi); MatPermute(Ais,idUi,idUi,&PL); IS offd; MatFindOffBlockDiagonalEntries(PL,&offd); My matrix PL is working, I know that since I viewed it before the line: IS offd. And although it says null object for parameter 1, I believe that other lines below this error are about IS. Should I create IS before using it in MatFindOffBlockDiagonalEntries? Mark Adams , 19 Haz 2020 Cum, 20:37 tarihinde ?unu yazd?: > > This is more useful and you are getting a new error. Maybe the old error is fixed. > It looks like you are trying to copy a matrix into another but they have different communicators. For example, one is created with MPI_COMM_WOLD and the other with PETSC_COMM_WORLD. > > On Fri, Jun 19, 2020 at 12:48 PM Eda Oktay wrote: >> >> Dear Mark, >> >> I updated Petsc and again, here's the error: >> >> [0]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> [0]PETSC ERROR: Arguments must have same communicators >> [0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> [1]PETSC ERROR: Arguments must have same communicators >> [1]PETSC ERROR: Different communicators in the two objects: Argument # >> 1 and 2 flag 3 >> [1]PETSC ERROR: See >> https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble >> shooting. >> [1]PETSC ERROR: Petsc Release Version 3.13.2, Jun 02, 2020 >> [1]PETSC ERROR: [2]PETSC ERROR: [3]PETSC ERROR: --------------------- >> Error Message -------------------------------------------------------------- >> [3]PETSC ERROR: Arguments must have same communicators >> [3]PETSC ERROR: Different communicators in the two objects: Argument # >> 1 and 2 flag 3 >> [3]PETSC ERROR: See >> https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble >> shooting. >> [3]PETSC ERROR: Petsc Release Version 3.13.2, Jun 02, 2020 >> [3]PETSC ERROR: >> ./approx_cut_deneme_clustering_son_final_edgecut_without_parmetis on a >> arch-linux2-c-debug named b342.wls.metu.edu.tr by edaoktay Fri Jun 19 >> 19:44:41 2020 >> [3]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ >> --download-mpich --download-openblas --download-slepc --download-metis >> --download-parmetis --download-chaco --with-X=1 >> Different communicators in the two objects: Argument # 1 and 2 flag 3 >> [0]PETSC ERROR: See >> https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble >> shooting. >> [0]PETSC ERROR: Petsc Release Version 3.13.2, Jun 02, 2020 >> [0]PETSC ERROR: >> ./approx_cut_deneme_clustering_son_final_edgecut_without_parmetis on a >> arch-linux2-c-debug named b342.wls.metu.edu.tr by edaoktay Fri Jun 19 >> 19:44:41 2020 >> [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ >> --download-mpich --download-openblas --download-slepc --download-metis >> --download-parmetis --download-chaco --with-X=1 >> [0]PETSC ERROR: #1 MatCopy() line 4044 in >> /home/edaoktay/petsc-3.13.2/src/mat/interface/matrix.c >> ./approx_cut_deneme_clustering_son_final_edgecut_without_parmetis on a >> arch-linux2-c-debug named b342.wls.metu.edu.tr by edaoktay Fri Jun 19 >> 19:44:41 2020 >> [1]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ >> --download-mpich --download-openblas --download-slepc --download-metis >> --download-parmetis --download-chaco --with-X=1 >> [1]PETSC ERROR: #1 MatCopy() line 4044 in >> /home/edaoktay/petsc-3.13.2/src/mat/interface/matrix.c >> [3]PETSC ERROR: #1 MatCopy() line 4044 in >> /home/edaoktay/petsc-3.13.2/src/mat/interface/matrix.c >> [0]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> [0]PETSC ERROR: Nonconforming object sizes >> [1]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> [1]PETSC ERROR: Nonconforming object sizes >> [1]PETSC ERROR: Non conforming matrix add: global sizes 1 x 72, 1 x 72 >> [3]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> [3]PETSC ERROR: Nonconforming object sizes >> [3]PETSC ERROR: Non conforming matrix add: global sizes 16 x 72, 16 x 72 >> [3]PETSC ERROR: [0]PETSC ERROR: Non conforming matrix add: global >> sizes 52 x 72, 52 x 72 >> [0]PETSC ERROR: See >> https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble >> shooting. >> [0]PETSC ERROR: Petsc Release Version 3.13.2, Jun 02, 2020 >> [0]PETSC ERROR: [1]PETSC ERROR: See >> https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble >> shooting. >> [1]PETSC ERROR: Petsc Release Version 3.13.2, Jun 02, 2020 >> [1]PETSC ERROR: >> ./approx_cut_deneme_clustering_son_final_edgecut_without_parmetis on a >> arch-linux2-c-debug named b342.wls.metu.edu.tr by edaoktay Fri Jun 19 >> 19:44:41 2020 >> See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. >> [3]PETSC ERROR: Petsc Release Version 3.13.2, Jun 02, 2020 >> [3]PETSC ERROR: >> ./approx_cut_deneme_clustering_son_final_edgecut_without_parmetis on a >> arch-linux2-c-debug named b342.wls.metu.edu.tr by edaoktay Fri Jun 19 >> 19:44:41 2020 >> [3]PETSC ERROR: >> ./approx_cut_deneme_clustering_son_final_edgecut_without_parmetis on a >> arch-linux2-c-debug named b342.wls.metu.edu.tr by edaoktay Fri Jun 19 >> 19:44:41 2020 >> [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ >> --download-mpich --download-openblas --download-slepc --download-metis >> --download-parmetis --download-chaco --with-X=1 >> [0]PETSC ERROR: [1]PETSC ERROR: Configure options --with-cc=gcc >> --with-cxx=g++ --download-mpich --download-openblas --download-slepc >> --download-metis --download-parmetis --download-chaco --with-X=1 >> [1]PETSC ERROR: #2 MatAXPY() line 72 in >> /home/edaoktay/petsc-3.13.2/src/mat/utils/axpy.c >> Configure options --with-cc=gcc --with-cxx=g++ --download-mpich >> --download-openblas --download-slepc --download-metis >> --download-parmetis --download-chaco --with-X=1 >> [3]PETSC ERROR: #2 MatAXPY() line 72 in >> /home/edaoktay/petsc-3.13.2/src/mat/utils/axpy.c >> #2 MatAXPY() line 72 in /home/edaoktay/petsc-3.13.2/src/mat/utils/axpy.c >> >> Elapsed time: 2.5e-01 >> >> --------------------- Error Message >> -------------------------------------------------------------- >> [2]PETSC ERROR: Arguments must have same communicators >> [2]PETSC ERROR: Different communicators in the two objects: Argument # >> 1 and 2 flag 3 >> [2]PETSC ERROR: See >> https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble >> shooting. >> [2]PETSC ERROR: Petsc Release Version 3.13.2, Jun 02, 2020 >> [2]PETSC ERROR: >> ./approx_cut_deneme_clustering_son_final_edgecut_without_parmetis on a >> arch-linux2-c-debug named b342.wls.metu.edu.tr by edaoktay Fri Jun 19 >> 19:44:41 2020 >> [2]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ >> --download-mpich --download-openblas --download-slepc --download-metis >> --download-parmetis --download-chaco --with-X=1 >> [2]PETSC ERROR: #1 MatCopy() line 4044 in >> /home/edaoktay/petsc-3.13.2/src/mat/interface/matrix.c >> [2]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> [2]PETSC ERROR: Nonconforming object sizes >> [2]PETSC ERROR: Non conforming matrix add: global sizes 3 x 72, 3 x 72 >> [2]PETSC ERROR: See >> https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble >> shooting. >> [2]PETSC ERROR: Petsc Release Version 3.13.2, Jun 02, 2020 >> [2]PETSC ERROR: >> ./approx_cut_deneme_clustering_son_final_edgecut_without_parmetis on a >> arch-linux2-c-debug named b342.wls.metu.edu.tr by edaoktay Fri Jun 19 >> 19:44:41 2020 >> [2]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ >> --download-mpich --download-openblas --download-slepc --download-metis >> --download-parmetis --download-chaco --with-X=1 >> [2]PETSC ERROR: #2 MatAXPY() line 72 in >> /home/edaoktay/petsc-3.13.2/src/mat/utils/axpy.c >> >> What can be the reason? >> >> Mark Adams , 19 Haz 2020 Cum, 18:41 tarihinde ?unu yazd?: >> > >> > I don't know what is going on here. There was an update to this function about a year ago, so that might fix your problem. >> > >> > We would need you to test with a current version. >> > >> > Mark >> > >> > On Fri, Jun 19, 2020 at 11:23 AM Eda Oktay wrote: >> >> >> >> Hi all, >> >> >> >> I am trying to find off block diagonal entries of a matrix and I am >> >> trying to use MatFindOffBlockDiagonalEntries. However, although my >> >> matrix is not NULL, I am getting an error message as follows: >> >> >> >> 1]PETSC ERROR: --------------------- Error Message >> >> -------------------------------------------------------------- >> >> [1]PETSC ERROR: [2]PETSC ERROR: --------------------- Error Message >> >> -------------------------------------------------------------- >> >> [2]PETSC ERROR: Null argument, when expecting valid pointer >> >> [2]PETSC ERROR: Null Object: Parameter # 1 >> >> [2]PETSC ERROR: See >> >> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble >> >> shooting. >> >> [2]PETSC ERROR: Petsc Release Version 3.11.1, Apr, 12, 2019 >> >> [2]PETSC ERROR: [3]PETSC ERROR: --------------------- Error Message >> >> -------------------------------------------------------------- >> >> [3]PETSC ERROR: Null argument, when expecting valid pointer >> >> [3]PETSC ERROR: Null Object: Parameter # 1 >> >> [3]PETSC ERROR: See >> >> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble >> >> shooting. >> >> [3]PETSC ERROR: Null argument, when expecting valid pointer >> >> [1]PETSC ERROR: Null Object: Parameter # 1 >> >> [1]PETSC ERROR: See >> >> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble >> >> shooting. >> >> [1]PETSC ERROR: Petsc Release Version 3.11.1, Apr, 12, 2019 >> >> [1]PETSC ERROR: >> >> ./approx_cut_deneme_clustering_son_final_edgecut_without_parmetis on a >> >> arch-linux2-c-debug named b342.wls.metu.edu.tr by edaoktay Fri Jun 19 >> >> 18:19:11 2020 >> >> [1]PETSC ERROR: Configure options --download-mpich --download-openblas >> >> --download-slepc --download-metis --download-parmetis --download-chaco >> >> --with-X=1 >> >> [1]PETSC ERROR: #1 ISGetSize() line 510 in >> >> /home/edaoktay/petsc-3.11.1/src/vec/is/is/interface/index.c >> >> [1]PETSC ERROR: >> >> ./approx_cut_deneme_clustering_son_final_edgecut_without_parmetis on a >> >> arch-linux2-c-debug named b342.wls.metu.edu.tr by edaoktay Fri Jun 19 >> >> 18:19:11 2020 >> >> [2]PETSC ERROR: Configure options --download-mpich --download-openblas >> >> --download-slepc --download-metis --download-parmetis --download-chaco >> >> --with-X=1 >> >> [2]PETSC ERROR: #1 ISGetSize() line 510 in >> >> /home/edaoktay/petsc-3.11.1/src/vec/is/is/interface/index.c >> >> [2]PETSC ERROR: #2 MatFindOffBlockDiagonalEntries_MPIAIJ() line 211 in >> >> /home/edaoktay/petsc-3.11.1/src/mat/impls/aij/mpi/mpiaij.c >> >> [2]PETSC ERROR: #2 MatFindOffBlockDiagonalEntries_MPIAIJ() line 211 in >> >> /home/edaoktay/petsc-3.11.1/src/mat/impls/aij/mpi/mpiaij.c >> >> [1]PETSC ERROR: #3 MatFindOffBlockDiagonalEntries() line 10502 in >> >> /home/edaoktay/petsc-3.11.1/src/mat/interface/matrix.c >> >> #3 MatFindOffBlockDiagonalEntries() line 10502 in >> >> /home/edaoktay/petsc-3.11.1/src/mat/interface/matrix.c >> >> Petsc Release Version 3.11.1, Apr, 12, 2019 >> >> [3]PETSC ERROR: >> >> ./approx_cut_deneme_clustering_son_final_edgecut_without_parmetis on a >> >> arch-linux2-c-debug named b342.wls.metu.edu.tr by edaoktay Fri Jun 19 >> >> 18:19:11 2020 >> >> [3]PETSC ERROR: Configure options --download-mpich --download-openblas >> >> --download-slepc --download-metis --download-parmetis --download-chaco >> >> --with-X=1 >> >> [3]PETSC ERROR: #1 ISGetSize() line 510 in >> >> /home/edaoktay/petsc-3.11.1/src/vec/is/is/interface/index.c >> >> [3]PETSC ERROR: #2 MatFindOffBlockDiagonalEntries_MPIAIJ() line 211 in >> >> /home/edaoktay/petsc-3.11.1/src/mat/impls/aij/mpi/mpiaij.c >> >> [3]PETSC ERROR: #3 MatFindOffBlockDiagonalEntries() line 10502 in >> >> /home/edaoktay/petsc-3.11.1/src/mat/interface/matrix.c >> >> >> >> And the program is still running. >> >> >> >> The part of my program is : >> >> >> >> Mat Ais; >> >> MatCreateSubMatrix(A,idUi,idUi,MAT_INITIAL_MATRIX,&Ais); >> >> ISSetPermutation(idUi); >> >> MatPermute(Ais,idUi,idUi,&PL); >> >> IS offd; >> >> MatFindOffBlockDiagonalEntries(PL,&offd); >> >> >> >> Since there isn't any examples for this function, I couldn't >> >> understand what should I do. Since it says in the error that it is >> >> about parameter 1, the matrix, I don't know how to solve the problem. >> >> My matrix PL is parallel sparse. I know there shouldn't be any problem >> >> in PL since I can view PL by using MatView. >> >> >> >> Thanks! >> >> >> >> Eda From stefano.zampini at gmail.com Mon Jun 22 04:33:35 2020 From: stefano.zampini at gmail.com (Stefano Zampini) Date: Mon, 22 Jun 2020 12:33:35 +0300 Subject: [petsc-users] MatFindOffBlockDiagonalEntries error In-Reply-To: References: Message-ID: It seems MatFindOffBlockDiagonalEntries_MPIAIJ is broken for a special case. Can you send a MWE to reproduce problem so that we can fix it properly and add it to the testsuite? Il giorno lun 22 giu 2020 alle ore 12:20 Eda Oktay ha scritto: > I am so sorry for the inconvenience, the last error I sent you was a > different one, I forgot to change the codes. I am sending you the > actual error where I used MatFindOffBlockDiagonalEntries again in the > newest version. > > [1]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [1]PETSC ERROR: Null argument, when expecting valid pointer > [2]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [2]PETSC ERROR: Null argument, when expecting valid pointer > [2]PETSC ERROR: Null Object: Parameter # 1 > [2]PETSC ERROR: See > https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble > shooting. > [2]PETSC ERROR: Petsc Release Version 3.13.2, Jun 02, 2020 > [2]PETSC ERROR: [3]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [3]PETSC ERROR: Null argument, when expecting valid pointer > [3]PETSC ERROR: Null Object: Parameter # 1 > [3]PETSC ERROR: See > https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble > shooting. > [3]PETSC ERROR: Petsc Release Version 3.13.2, Jun 02, 2020 > [3]PETSC ERROR: > ./approx_cut_deneme_clustering_son_final_edgecut_without_parmetis on a > arch-linux2-c-debug named b342.wls.metu.edu.tr by edaoktay Mon Jun 22 > 12:11:44 2020 > [1]PETSC ERROR: Null Object: Parameter # 1 > [1]PETSC ERROR: See > https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble > shooting. > [1]PETSC ERROR: Petsc Release Version 3.13.2, Jun 02, 2020 > [1]PETSC ERROR: > ./approx_cut_deneme_clustering_son_final_edgecut_without_parmetis on a > arch-linux2-c-debug named b342.wls.metu.edu.tr by edaoktay Mon Jun 22 > 12:11:44 2020 > [1]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ > --download-mpich --download-openblas --download-slepc --download-metis > --download-parmetis --download-chaco --with-X=1 > [1]PETSC ERROR: #1 ISGetSize() line 1101 in > /home/edaoktay/petsc-3.13.2/src/vec/is/is/interface/index.c > ./approx_cut_deneme_clustering_son_final_edgecut_without_parmetis on a > arch-linux2-c-debug named b342.wls.metu.edu.tr by edaoktay Mon Jun 22 > 12:11:44 2020 > [2]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ > --download-mpich --download-openblas --download-slepc --download-metis > --download-parmetis --download-chaco --with-X=1 > [2]PETSC ERROR: #1 ISGetSize() line 1101 in > /home/edaoktay/petsc-3.13.2/src/vec/is/is/interface/index.c > [2]PETSC ERROR: [3]PETSC ERROR: Configure options --with-cc=gcc > --with-cxx=g++ --download-mpich --download-openblas --download-slepc > --download-metis --download-parmetis --download-chaco --with-X=1 > [3]PETSC ERROR: #1 ISGetSize() line 1101 in > /home/edaoktay/petsc-3.13.2/src/vec/is/is/interface/index.c > [3]PETSC ERROR: #2 MatFindOffBlockDiagonalEntries_MPIAIJ() line 229 in > /home/edaoktay/petsc-3.13.2/src/mat/impls/aij/mpi/mpiaij.c > [1]PETSC ERROR: #2 MatFindOffBlockDiagonalEntries_MPIAIJ() line 229 in > /home/edaoktay/petsc-3.13.2/src/mat/impls/aij/mpi/mpiaij.c > [1]PETSC ERROR: #3 MatFindOffBlockDiagonalEntries() line 9824 in > /home/edaoktay/petsc-3.13.2/src/mat/interface/matrix.c > #2 MatFindOffBlockDiagonalEntries_MPIAIJ() line 229 in > /home/edaoktay/petsc-3.13.2/src/mat/impls/aij/mpi/mpiaij.c > [2]PETSC ERROR: #3 MatFindOffBlockDiagonalEntries() line 9824 in > /home/edaoktay/petsc-3.13.2/src/mat/interface/matrix.c > [3]PETSC ERROR: #3 MatFindOffBlockDiagonalEntries() line 9824 in > /home/edaoktay/petsc-3.13.2/src/mat/interface/matrix.c > > I think this is again the same error when I used an older version. > Again, even though I am using a valid matrix and an index set, I don't > know why I am getting this error. > > About the small test example, I couldn't understand what Junchao ment > but this is the first time I am trying to use this function and my > current matrix is quite small 72*72. > > Mat Ais; > MatCreateSubMatrix(A,idUi,idUi,MAT_INITIAL_MATRIX,&Ais); > ISSetPermutation(idUi); > MatPermute(Ais,idUi,idUi,&PL); > IS offd; > MatFindOffBlockDiagonalEntries(PL,&offd); > > My matrix PL is working, I know that since I viewed it before the > line: IS offd. And although it says null object for parameter 1, I > believe that other lines below this error are about IS. Should I > create IS before using it in MatFindOffBlockDiagonalEntries? > > Mark Adams , 19 Haz 2020 Cum, 20:37 tarihinde ?unu yazd?: > > > > This is more useful and you are getting a new error. Maybe the old error > is fixed. > > It looks like you are trying to copy a matrix into another but they have > different communicators. For example, one is created with MPI_COMM_WOLD > and the other with PETSC_COMM_WORLD. > > > > On Fri, Jun 19, 2020 at 12:48 PM Eda Oktay > wrote: > >> > >> Dear Mark, > >> > >> I updated Petsc and again, here's the error: > >> > >> [0]PETSC ERROR: --------------------- Error Message > >> -------------------------------------------------------------- > >> [0]PETSC ERROR: Arguments must have same communicators > >> [0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error Message > >> -------------------------------------------------------------- > >> [1]PETSC ERROR: Arguments must have same communicators > >> [1]PETSC ERROR: Different communicators in the two objects: Argument # > >> 1 and 2 flag 3 > >> [1]PETSC ERROR: See > >> https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble > >> shooting. > >> [1]PETSC ERROR: Petsc Release Version 3.13.2, Jun 02, 2020 > >> [1]PETSC ERROR: [2]PETSC ERROR: [3]PETSC ERROR: --------------------- > >> Error Message > -------------------------------------------------------------- > >> [3]PETSC ERROR: Arguments must have same communicators > >> [3]PETSC ERROR: Different communicators in the two objects: Argument # > >> 1 and 2 flag 3 > >> [3]PETSC ERROR: See > >> https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble > >> shooting. > >> [3]PETSC ERROR: Petsc Release Version 3.13.2, Jun 02, 2020 > >> [3]PETSC ERROR: > >> ./approx_cut_deneme_clustering_son_final_edgecut_without_parmetis on a > >> arch-linux2-c-debug named b342.wls.metu.edu.tr by edaoktay Fri Jun 19 > >> 19:44:41 2020 > >> [3]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ > >> --download-mpich --download-openblas --download-slepc --download-metis > >> --download-parmetis --download-chaco --with-X=1 > >> Different communicators in the two objects: Argument # 1 and 2 flag 3 > >> [0]PETSC ERROR: See > >> https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble > >> shooting. > >> [0]PETSC ERROR: Petsc Release Version 3.13.2, Jun 02, 2020 > >> [0]PETSC ERROR: > >> ./approx_cut_deneme_clustering_son_final_edgecut_without_parmetis on a > >> arch-linux2-c-debug named b342.wls.metu.edu.tr by edaoktay Fri Jun 19 > >> 19:44:41 2020 > >> [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ > >> --download-mpich --download-openblas --download-slepc --download-metis > >> --download-parmetis --download-chaco --with-X=1 > >> [0]PETSC ERROR: #1 MatCopy() line 4044 in > >> /home/edaoktay/petsc-3.13.2/src/mat/interface/matrix.c > >> ./approx_cut_deneme_clustering_son_final_edgecut_without_parmetis on a > >> arch-linux2-c-debug named b342.wls.metu.edu.tr by edaoktay Fri Jun 19 > >> 19:44:41 2020 > >> [1]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ > >> --download-mpich --download-openblas --download-slepc --download-metis > >> --download-parmetis --download-chaco --with-X=1 > >> [1]PETSC ERROR: #1 MatCopy() line 4044 in > >> /home/edaoktay/petsc-3.13.2/src/mat/interface/matrix.c > >> [3]PETSC ERROR: #1 MatCopy() line 4044 in > >> /home/edaoktay/petsc-3.13.2/src/mat/interface/matrix.c > >> [0]PETSC ERROR: --------------------- Error Message > >> -------------------------------------------------------------- > >> [0]PETSC ERROR: Nonconforming object sizes > >> [1]PETSC ERROR: --------------------- Error Message > >> -------------------------------------------------------------- > >> [1]PETSC ERROR: Nonconforming object sizes > >> [1]PETSC ERROR: Non conforming matrix add: global sizes 1 x 72, 1 x 72 > >> [3]PETSC ERROR: --------------------- Error Message > >> -------------------------------------------------------------- > >> [3]PETSC ERROR: Nonconforming object sizes > >> [3]PETSC ERROR: Non conforming matrix add: global sizes 16 x 72, 16 x 72 > >> [3]PETSC ERROR: [0]PETSC ERROR: Non conforming matrix add: global > >> sizes 52 x 72, 52 x 72 > >> [0]PETSC ERROR: See > >> https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble > >> shooting. > >> [0]PETSC ERROR: Petsc Release Version 3.13.2, Jun 02, 2020 > >> [0]PETSC ERROR: [1]PETSC ERROR: See > >> https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble > >> shooting. > >> [1]PETSC ERROR: Petsc Release Version 3.13.2, Jun 02, 2020 > >> [1]PETSC ERROR: > >> ./approx_cut_deneme_clustering_son_final_edgecut_without_parmetis on a > >> arch-linux2-c-debug named b342.wls.metu.edu.tr by edaoktay Fri Jun 19 > >> 19:44:41 2020 > >> See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble > shooting. > >> [3]PETSC ERROR: Petsc Release Version 3.13.2, Jun 02, 2020 > >> [3]PETSC ERROR: > >> ./approx_cut_deneme_clustering_son_final_edgecut_without_parmetis on a > >> arch-linux2-c-debug named b342.wls.metu.edu.tr by edaoktay Fri Jun 19 > >> 19:44:41 2020 > >> [3]PETSC ERROR: > >> ./approx_cut_deneme_clustering_son_final_edgecut_without_parmetis on a > >> arch-linux2-c-debug named b342.wls.metu.edu.tr by edaoktay Fri Jun 19 > >> 19:44:41 2020 > >> [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ > >> --download-mpich --download-openblas --download-slepc --download-metis > >> --download-parmetis --download-chaco --with-X=1 > >> [0]PETSC ERROR: [1]PETSC ERROR: Configure options --with-cc=gcc > >> --with-cxx=g++ --download-mpich --download-openblas --download-slepc > >> --download-metis --download-parmetis --download-chaco --with-X=1 > >> [1]PETSC ERROR: #2 MatAXPY() line 72 in > >> /home/edaoktay/petsc-3.13.2/src/mat/utils/axpy.c > >> Configure options --with-cc=gcc --with-cxx=g++ --download-mpich > >> --download-openblas --download-slepc --download-metis > >> --download-parmetis --download-chaco --with-X=1 > >> [3]PETSC ERROR: #2 MatAXPY() line 72 in > >> /home/edaoktay/petsc-3.13.2/src/mat/utils/axpy.c > >> #2 MatAXPY() line 72 in /home/edaoktay/petsc-3.13.2/src/mat/utils/axpy.c > >> > >> Elapsed time: 2.5e-01 > >> > >> --------------------- Error Message > >> -------------------------------------------------------------- > >> [2]PETSC ERROR: Arguments must have same communicators > >> [2]PETSC ERROR: Different communicators in the two objects: Argument # > >> 1 and 2 flag 3 > >> [2]PETSC ERROR: See > >> https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble > >> shooting. > >> [2]PETSC ERROR: Petsc Release Version 3.13.2, Jun 02, 2020 > >> [2]PETSC ERROR: > >> ./approx_cut_deneme_clustering_son_final_edgecut_without_parmetis on a > >> arch-linux2-c-debug named b342.wls.metu.edu.tr by edaoktay Fri Jun 19 > >> 19:44:41 2020 > >> [2]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ > >> --download-mpich --download-openblas --download-slepc --download-metis > >> --download-parmetis --download-chaco --with-X=1 > >> [2]PETSC ERROR: #1 MatCopy() line 4044 in > >> /home/edaoktay/petsc-3.13.2/src/mat/interface/matrix.c > >> [2]PETSC ERROR: --------------------- Error Message > >> -------------------------------------------------------------- > >> [2]PETSC ERROR: Nonconforming object sizes > >> [2]PETSC ERROR: Non conforming matrix add: global sizes 3 x 72, 3 x 72 > >> [2]PETSC ERROR: See > >> https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble > >> shooting. > >> [2]PETSC ERROR: Petsc Release Version 3.13.2, Jun 02, 2020 > >> [2]PETSC ERROR: > >> ./approx_cut_deneme_clustering_son_final_edgecut_without_parmetis on a > >> arch-linux2-c-debug named b342.wls.metu.edu.tr by edaoktay Fri Jun 19 > >> 19:44:41 2020 > >> [2]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ > >> --download-mpich --download-openblas --download-slepc --download-metis > >> --download-parmetis --download-chaco --with-X=1 > >> [2]PETSC ERROR: #2 MatAXPY() line 72 in > >> /home/edaoktay/petsc-3.13.2/src/mat/utils/axpy.c > >> > >> What can be the reason? > >> > >> Mark Adams , 19 Haz 2020 Cum, 18:41 tarihinde ?unu > yazd?: > >> > > >> > I don't know what is going on here. There was an update to this > function about a year ago, so that might fix your problem. > >> > > >> > We would need you to test with a current version. > >> > > >> > Mark > >> > > >> > On Fri, Jun 19, 2020 at 11:23 AM Eda Oktay > wrote: > >> >> > >> >> Hi all, > >> >> > >> >> I am trying to find off block diagonal entries of a matrix and I am > >> >> trying to use MatFindOffBlockDiagonalEntries. However, although my > >> >> matrix is not NULL, I am getting an error message as follows: > >> >> > >> >> 1]PETSC ERROR: --------------------- Error Message > >> >> -------------------------------------------------------------- > >> >> [1]PETSC ERROR: [2]PETSC ERROR: --------------------- Error Message > >> >> -------------------------------------------------------------- > >> >> [2]PETSC ERROR: Null argument, when expecting valid pointer > >> >> [2]PETSC ERROR: Null Object: Parameter # 1 > >> >> [2]PETSC ERROR: See > >> >> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble > >> >> shooting. > >> >> [2]PETSC ERROR: Petsc Release Version 3.11.1, Apr, 12, 2019 > >> >> [2]PETSC ERROR: [3]PETSC ERROR: --------------------- Error Message > >> >> -------------------------------------------------------------- > >> >> [3]PETSC ERROR: Null argument, when expecting valid pointer > >> >> [3]PETSC ERROR: Null Object: Parameter # 1 > >> >> [3]PETSC ERROR: See > >> >> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble > >> >> shooting. > >> >> [3]PETSC ERROR: Null argument, when expecting valid pointer > >> >> [1]PETSC ERROR: Null Object: Parameter # 1 > >> >> [1]PETSC ERROR: See > >> >> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble > >> >> shooting. > >> >> [1]PETSC ERROR: Petsc Release Version 3.11.1, Apr, 12, 2019 > >> >> [1]PETSC ERROR: > >> >> ./approx_cut_deneme_clustering_son_final_edgecut_without_parmetis on > a > >> >> arch-linux2-c-debug named b342.wls.metu.edu.tr by edaoktay Fri Jun > 19 > >> >> 18:19:11 2020 > >> >> [1]PETSC ERROR: Configure options --download-mpich > --download-openblas > >> >> --download-slepc --download-metis --download-parmetis > --download-chaco > >> >> --with-X=1 > >> >> [1]PETSC ERROR: #1 ISGetSize() line 510 in > >> >> /home/edaoktay/petsc-3.11.1/src/vec/is/is/interface/index.c > >> >> [1]PETSC ERROR: > >> >> ./approx_cut_deneme_clustering_son_final_edgecut_without_parmetis on > a > >> >> arch-linux2-c-debug named b342.wls.metu.edu.tr by edaoktay Fri Jun > 19 > >> >> 18:19:11 2020 > >> >> [2]PETSC ERROR: Configure options --download-mpich > --download-openblas > >> >> --download-slepc --download-metis --download-parmetis > --download-chaco > >> >> --with-X=1 > >> >> [2]PETSC ERROR: #1 ISGetSize() line 510 in > >> >> /home/edaoktay/petsc-3.11.1/src/vec/is/is/interface/index.c > >> >> [2]PETSC ERROR: #2 MatFindOffBlockDiagonalEntries_MPIAIJ() line 211 > in > >> >> /home/edaoktay/petsc-3.11.1/src/mat/impls/aij/mpi/mpiaij.c > >> >> [2]PETSC ERROR: #2 MatFindOffBlockDiagonalEntries_MPIAIJ() line 211 > in > >> >> /home/edaoktay/petsc-3.11.1/src/mat/impls/aij/mpi/mpiaij.c > >> >> [1]PETSC ERROR: #3 MatFindOffBlockDiagonalEntries() line 10502 in > >> >> /home/edaoktay/petsc-3.11.1/src/mat/interface/matrix.c > >> >> #3 MatFindOffBlockDiagonalEntries() line 10502 in > >> >> /home/edaoktay/petsc-3.11.1/src/mat/interface/matrix.c > >> >> Petsc Release Version 3.11.1, Apr, 12, 2019 > >> >> [3]PETSC ERROR: > >> >> ./approx_cut_deneme_clustering_son_final_edgecut_without_parmetis on > a > >> >> arch-linux2-c-debug named b342.wls.metu.edu.tr by edaoktay Fri Jun > 19 > >> >> 18:19:11 2020 > >> >> [3]PETSC ERROR: Configure options --download-mpich > --download-openblas > >> >> --download-slepc --download-metis --download-parmetis > --download-chaco > >> >> --with-X=1 > >> >> [3]PETSC ERROR: #1 ISGetSize() line 510 in > >> >> /home/edaoktay/petsc-3.11.1/src/vec/is/is/interface/index.c > >> >> [3]PETSC ERROR: #2 MatFindOffBlockDiagonalEntries_MPIAIJ() line 211 > in > >> >> /home/edaoktay/petsc-3.11.1/src/mat/impls/aij/mpi/mpiaij.c > >> >> [3]PETSC ERROR: #3 MatFindOffBlockDiagonalEntries() line 10502 in > >> >> /home/edaoktay/petsc-3.11.1/src/mat/interface/matrix.c > >> >> > >> >> And the program is still running. > >> >> > >> >> The part of my program is : > >> >> > >> >> Mat Ais; > >> >> MatCreateSubMatrix(A,idUi,idUi,MAT_INITIAL_MATRIX,&Ais); > >> >> ISSetPermutation(idUi); > >> >> MatPermute(Ais,idUi,idUi,&PL); > >> >> IS offd; > >> >> MatFindOffBlockDiagonalEntries(PL,&offd); > >> >> > >> >> Since there isn't any examples for this function, I couldn't > >> >> understand what should I do. Since it says in the error that it is > >> >> about parameter 1, the matrix, I don't know how to solve the problem. > >> >> My matrix PL is parallel sparse. I know there shouldn't be any > problem > >> >> in PL since I can view PL by using MatView. > >> >> > >> >> Thanks! > >> >> > >> >> Eda > -- Stefano -------------- next part -------------- An HTML attachment was scrubbed... URL: From eda.oktay at metu.edu.tr Mon Jun 22 06:14:08 2020 From: eda.oktay at metu.edu.tr (Eda Oktay) Date: Mon, 22 Jun 2020 14:14:08 +0300 Subject: [petsc-users] Create MATSEQAIJ from MATMPIAIJ Message-ID: Hello everyone, I am trying to find elements in off diagonal blocks of a parallel sparse matrix. That is why, I want to use MatGetDiagonalBlock and from the matrix I obtain, I want to obtain off diagonal elements by subtracting it from my original matrix by using MatAXPY. However, since MatGetDiagonalBlock gives a sequential matrix and my original one is parallel, I can't use MatAXPY. That's why I want to change the type of one of the matrices. How can I change a MATSEQAIJ to MATMPIAIJ or vice versa? Thanks! From knepley at gmail.com Mon Jun 22 06:43:44 2020 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 22 Jun 2020 07:43:44 -0400 Subject: [petsc-users] Create MATSEQAIJ from MATMPIAIJ In-Reply-To: References: Message-ID: On Mon, Jun 22, 2020 at 7:14 AM Eda Oktay wrote: > Hello everyone, > > I am trying to find elements in off diagonal blocks of a parallel > sparse matrix. That is why, I want to use MatGetDiagonalBlock and from > the matrix I obtain, I want to obtain off diagonal elements by > subtracting it from my original matrix by using MatAXPY. > > However, since MatGetDiagonalBlock gives a sequential matrix and my > original one is parallel, I can't use MatAXPY. That's why I want to > change the type of one of the matrices. > > How can I change a MATSEQAIJ to MATMPIAIJ or vice versa? > I assume you want a parallel matrix with the element in the diagonal block removed. I can think of at least two ways to do this which sound easier to me: 1) Make a copy and then zero out the diagonal block is a way similar to MatChop: https://www.mcs.anl.gov/petsc/petsc-current/src/mat/utils/axpy.c.html#MatChop 2) Use MatGetSubMatrix() and exclude any columns from the diagonal block on each process. Thanks, Matt > Thanks! > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From junchao.zhang at gmail.com Mon Jun 22 10:00:55 2020 From: junchao.zhang at gmail.com (Junchao Zhang) Date: Mon, 22 Jun 2020 10:00:55 -0500 Subject: [petsc-users] Create MATSEQAIJ from MATMPIAIJ In-Reply-To: References: Message-ID: The resulting matrix is dependent on the number of processes you use (e.g., if you run with only one process, you will get a zero matrix). Think it over if this is really what you want. --Junchao Zhang On Mon, Jun 22, 2020 at 6:14 AM Eda Oktay wrote: > Hello everyone, > > I am trying to find elements in off diagonal blocks of a parallel > sparse matrix. That is why, I want to use MatGetDiagonalBlock and from > the matrix I obtain, I want to obtain off diagonal elements by > subtracting it from my original matrix by using MatAXPY. > > However, since MatGetDiagonalBlock gives a sequential matrix and my > original one is parallel, I can't use MatAXPY. That's why I want to > change the type of one of the matrices. > > How can I change a MATSEQAIJ to MATMPIAIJ or vice versa? > > Thanks! > -------------- next part -------------- An HTML attachment was scrubbed... URL: From eda.oktay at metu.edu.tr Mon Jun 22 10:39:41 2020 From: eda.oktay at metu.edu.tr (Eda Oktay) Date: Mon, 22 Jun 2020 18:39:41 +0300 Subject: [petsc-users] Create MATSEQAIJ from MATMPIAIJ In-Reply-To: References: Message-ID: Matthew Knepley , 22 Haz 2020 Pzt, 14:43 tarihinde ?unu yazd?: > > On Mon, Jun 22, 2020 at 7:14 AM Eda Oktay wrote: >> >> Hello everyone, >> >> I am trying to find elements in off diagonal blocks of a parallel >> sparse matrix. That is why, I want to use MatGetDiagonalBlock and from >> the matrix I obtain, I want to obtain off diagonal elements by >> subtracting it from my original matrix by using MatAXPY. >> >> However, since MatGetDiagonalBlock gives a sequential matrix and my >> The original one is parallel, I can't use MatAXPY. That's why I want to >> change the type of one of the matrices. >> >> How can I change a MATSEQAIJ to MATMPIAIJ or vice versa? > > > I assume you want a parallel matrix with the element in the diagonal block removed. I can think > of at least two ways to do this which sound easier to me: > > 1) Make a copy and then zero out the diagonal block is a way similar to MatChop: https://www.mcs.anl.gov/petsc/petsc-current/src/mat/utils/axpy.c.html#MatChop I read MatChop before, however I couldn't understand how to decide tolerance. What I understood from this function is to eliminate entries less than a number which is set to be the tolerance. But the entries in the diagonal blocks can be less than ones in offdiagonal blocks. What am I missing? > > 2) Use MatGetSubMatrix() and exclude any columns from the diagonal block on each process. > > Thanks, > > Matt > >> >> Thanks! > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ From knepley at gmail.com Mon Jun 22 11:10:14 2020 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 22 Jun 2020 12:10:14 -0400 Subject: [petsc-users] Create MATSEQAIJ from MATMPIAIJ In-Reply-To: References: Message-ID: On Mon, Jun 22, 2020 at 11:39 AM Eda Oktay wrote: > Matthew Knepley , 22 Haz 2020 Pzt, 14:43 tarihinde > ?unu yazd?: > > > > On Mon, Jun 22, 2020 at 7:14 AM Eda Oktay wrote: > >> > >> Hello everyone, > >> > >> I am trying to find elements in off diagonal blocks of a parallel > >> sparse matrix. That is why, I want to use MatGetDiagonalBlock and from > >> the matrix I obtain, I want to obtain off diagonal elements by > >> subtracting it from my original matrix by using MatAXPY. > >> > >> However, since MatGetDiagonalBlock gives a sequential matrix and my > >> The original one is parallel, I can't use MatAXPY. That's why I want to > >> change the type of one of the matrices. > >> > >> How can I change a MATSEQAIJ to MATMPIAIJ or vice versa? > > > > > > I assume you want a parallel matrix with the element in the diagonal > block removed. I can think > > of at least two ways to do this which sound easier to me: > > > > 1) Make a copy and then zero out the diagonal block is a way similar > to MatChop: > https://www.mcs.anl.gov/petsc/petsc-current/src/mat/utils/axpy.c.html#MatChop > > I read MatChop before, however I couldn't understand how to decide > tolerance. What I understood from this function is to eliminate > entries less than a number which is set to be the tolerance. But the > entries in the diagonal blocks can be less than ones in offdiagonal > blocks. What am I missing? > You _change_ the code. MatChop decides to make an element zero by looking at how big it is. You would decide to make an element zero by looking at what column it is in. Thanks, Matt > > > > 2) Use MatGetSubMatrix() and exclude any columns from the diagonal > block on each process. > > > > Thanks, > > > > Matt > > > >> > >> Thanks! > > > > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > > -- Norbert Wiener > > > > https://www.cse.buffalo.edu/~knepley/ > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From bourdin at lsu.edu Mon Jun 22 11:22:30 2020 From: bourdin at lsu.edu (Blaise A Bourdin) Date: Mon, 22 Jun 2020 16:22:30 +0000 Subject: [petsc-users] Cohesive Element Support In-Reply-To: <2997A784-5B72-4114-A5B8-00EC7A8E4CAA@gmail.com> References: <7C00A8A6-F7F2-4D62-852B-45ADCD317B98@gmail.com> <2997A784-5B72-4114-A5B8-00EC7A8E4CAA@gmail.com> Message-ID: <2CF050EA-6364-467F-8A26-2F05EB151B20@lsu.edu> On Jun 19, 2020, at 12:53 PM, Jacob Faibussowitsch > wrote: Hello, Thank you both for your comprehensive replies. Matt: Thanks for the rundown, I will take a look at PyLith and its cohesive element impls. Blaise: The project is still in very early stages as you can tell, so everything is up in the air. Currently we are weighing using a cohesive element approach vs an element extinction approach but we can certainly add a variational model into the mix. We had leaned towards the cohesive element approach as we also need to link Quantum Monte Carlo generated data to the model, which seemed logically the easiest to do with cohesive elements. Are there any examples of your codes online? The only reference I could find online was in this link https://www.math.lsu.edu/~bourdin/defectmechanics/ So far, you are listing numerical approach. What is your fracture _model_? Both cohesive elements and element erosion approach require some a priori knowledge of the crack path. Is this acceptable for you? My code is released under a BSD license at https://github.com/bourdin/mef90 Regards, Blaise Best regards, Jacob Faibussowitsch (Jacob Fai - booss - oh - vitch) Cell: (312) 694-3391 On Jun 19, 2020, at 10:15 AM, Blaise A Bourdin > wrote: On Jun 18, 2020, at 5:28 AM, Matthew Knepley > wrote: On Wed, Jun 17, 2020 at 4:05 PM Jacob Faibussowitsch > wrote: Hello, I am looking to perform large scale fracture and crack propagation simulations and have a few questions regarding PETSc support for this. Specifically I am looking for cohesive surface element support with a few twists: 1. Is there support for zero thickness surface elements? For example modeling virtually flat patches of adhesives holding together two larger structures being pulled apart. This is how PyLith works: https://github.com/geodynamics/pylith 2. Is there support for ?joining? two possibly distinct meshes with cohesive surface elements? For example say I have two distinct cylinders representing fibers which would ?touch" to form an X shape. No, it would have to be coded. 3. In a similar vein, is there support for a mesh to fracture entirely along a crack formed through the cohesive elements? Imagine the aforementioned X configuration separating entirely into two separate cylinders again. No, it would have to be coded. 4. Is there a mechanism by which you can classify existing elements as cohesive elements? See 1. 5. Is there an already implemented way of imposing tie-constraints between independent meshes? This would potentially be used to tie high order cohesive cells which would have a non-conforming interface to the ?regular? mesh. There is nothing for non-conforming interfaces. From googling I have come across DMPlexCreateHybridMesh(), DMPlexConstructCohesiveCells(), and DMPlexCreateCohesiveSubmesh(). While these do implement cohesive cells these functions don?t at first glance seem to allow one to implement the above. Having worked with cohesive elements for more than a decade, I would be cautious about a new code using them for fracture. To me, it appears that variational fracture codes, like those from Blaise Bourdin and J. J. Marigo's group have much better geometric flexibility, and Maurini's work on the solver clears up the hardest part. I definitely concur with this and would be happy to help? Blaise -- A.K. & Shirley Barton Professor of Mathematics Adjunct Professor of Mechanical Engineering Adjunct of the Center for Computation & Technology Louisiana State University, Lockett Hall Room 344, Baton Rouge, LA 70803, USA Tel. +1 (225) 578 1612, Fax +1 (225) 578 4276 Web http://www.math.lsu.edu/~bourdin -------------- next part -------------- An HTML attachment was scrubbed... URL: From eda.oktay at metu.edu.tr Mon Jun 22 11:25:27 2020 From: eda.oktay at metu.edu.tr (Eda Oktay) Date: Mon, 22 Jun 2020 19:25:27 +0300 Subject: [petsc-users] Create MATSEQAIJ from MATMPIAIJ In-Reply-To: References: Message-ID: Oh okay, I misunderstood before. I will try that. Thanks! Matthew Knepley , 22 Haz 2020 Pzt, 19:10 tarihinde ?unu yazd?: > > On Mon, Jun 22, 2020 at 11:39 AM Eda Oktay wrote: >> >> Matthew Knepley , 22 Haz 2020 Pzt, 14:43 tarihinde >> ?unu yazd?: >> > >> > On Mon, Jun 22, 2020 at 7:14 AM Eda Oktay wrote: >> >> >> >> Hello everyone, >> >> >> >> I am trying to find elements in off diagonal blocks of a parallel >> >> sparse matrix. That is why, I want to use MatGetDiagonalBlock and from >> >> the matrix I obtain, I want to obtain off diagonal elements by >> >> subtracting it from my original matrix by using MatAXPY. >> >> >> >> However, since MatGetDiagonalBlock gives a sequential matrix and my >> >> The original one is parallel, I can't use MatAXPY. That's why I want to >> >> change the type of one of the matrices. >> >> >> >> How can I change a MATSEQAIJ to MATMPIAIJ or vice versa? >> > >> > >> > I assume you want a parallel matrix with the element in the diagonal block removed. I can think >> > of at least two ways to do this which sound easier to me: >> > >> > 1) Make a copy and then zero out the diagonal block is a way similar to MatChop: https://www.mcs.anl.gov/petsc/petsc-current/src/mat/utils/axpy.c.html#MatChop >> >> I read MatChop before, however I couldn't understand how to decide >> tolerance. What I understood from this function is to eliminate >> entries less than a number which is set to be the tolerance. But the >> entries in the diagonal blocks can be less than ones in offdiagonal >> blocks. What am I missing? > > > You _change_ the code. MatChop decides to make an element zero by looking at how big it is. You would decide > to make an element zero by looking at what column it is in. > > Thanks, > > Matt > >> >> > >> > 2) Use MatGetSubMatrix() and exclude any columns from the diagonal block on each process. >> > >> > Thanks, >> > >> > Matt >> > >> >> >> >> Thanks! >> > >> > >> > >> > -- >> > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >> > -- Norbert Wiener >> > >> > https://www.cse.buffalo.edu/~knepley/ > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ From ashish.patel at onscale.com Mon Jun 22 11:56:43 2020 From: ashish.patel at onscale.com (Ashish Patel) Date: Mon, 22 Jun 2020 09:56:43 -0700 Subject: [petsc-users] SNES Line search: fixing the non zero solution values of dirichlet nodes during line search In-Reply-To: <6F862511-22D9-42D0-9802-1CC36E80182C@anl.gov> References: <6F862511-22D9-42D0-9802-1CC36E80182C@anl.gov> Message-ID: Thanks Stefano, Barry and Hong for your feedback. Stefano, the option that you suggested is helpful to freeze the dirichlet values. However, it did not completely solve the convergence issue for nonlinear problems using linesearch since the direction vector which is calculated before any Solve uses the residual which is not yet updated for the new Dirichlet values (u(t) =b(t)). The update happens only after the solve at SNES iteration 0. To fix this I now update the residual for nodes adjacent to Dirichlet nodes without depending on any Solve. The Solve stage only updates the solution for dirichlet nodes. Calculating the residual this way resolves the issue. Barry, Hong yes you are right, the dirichlet boundary conditions are time dependent. We do solve the equations implicitly and use TSIFunction and TSIJacobian to set up the problem. The dirichlet nodes are part of computational nodes. The jacobian for dirchlet nodes has diagonal 1 and rows & colums 0. For setting the residual in TSIFunction the residual for dirichlet nodes are set to (solution_value-Dirichlet_value). Regards Ashish On Sun, Jun 21, 2020 at 6:23 PM Zhang, Hong wrote: > Can you elaborate a bit on how you "set the RHS as increment of > (Dirichlet_value-solution_value)?? > > It seems that the boundary condition is imposed as algebraic equations > (e.g. 0 = u(t)-b(t) ), and the boundary nodes are included in the > computational domain. This approach should work theoretically and is used > in some TS examples such as ex17.c, you might want to check if the > algebraic equations are implemented properly in your IFunction() and > IJacobian(). An alternative approach you can try is to remove the boundary > nodes from the unknown variables, see the example ex25.c. In this way you > solve ODEs instead of DAEs and do not need to worry about SNES modifying > certain values. > > Hong (Mr.) > > On Jun 19, 2020, at 5:15 PM, Ashish Patel > wrote: > > Dear PETSc users, > > We use PETSc as part of a finite element method program and we are trying > to properly implement Dirichlet boundary conditions for non-linear, > transient problems. We find that when we use a line search method it also > changes the non-zero solution value of Dirichlet nodes as it steps through > the line search iteration. I was wondering if there is a way to freeze > some index sets of a vector from changing during the line search operation? > > We are using the TS framework to setup the problem and use > 'MatZeroRowsColumns' to set the diagonal of the jacobian to 1 for the > dirichlet nodes and set the RHS as increment of > (Dirichlet_value-solution_value). This works when the line search method is > turned off by using '-snes_linesearch_type basic' however using the default > 'bt' linesearch, the TS diverges with error shown below. In a separate > implementation if we overwrote the dirichlet nodes of the solution vector > in TS residual function with the Dirichlet values then the 'bt' line > search method converged to the right solution. However we would like to > avoid modifying the internal PETSc vector in our implementation. > > 0 TS dt 1. time 0. > 0 SNES Function norm 2.378549386020e+03 > Line search: gnorm after quadratic fit 4.369235425165e+03 > Line search: Cubic step no good, shrinking lambda, current gnorm > 2.385369069060e+03 lambda=1.0000000000000002e-02 > Line search: Cubically determined step, current gnorm > 2.373925008934e+03 lambda=3.8846250444606093e-03 > 1 SNES Function norm 2.373925008934e+03 > Line search: gnorm after quadratic fit 5.006914179995e+03 > Line search: Cubic step no good, shrinking lambda, current gnorm > 2.420957096780e+03 lambda=1.0000000000000002e-02 > Line search: Cubic step no good, shrinking lambda, current gnorm > 2.376034946750e+03 lambda=1.6129422079664700e-03 > Line search: Cubic step no good, shrinking lambda, current gnorm > 2.374313344729e+03 lambda=4.8465026740690043e-04 > Line search: Cubic step no good, shrinking lambda, current gnorm > 2.373999473242e+03 lambda=1.5857251532828948e-04 > Line search: Cubically determined step, current gnorm > 2.373921668024e+03 lambda=5.8116507162753387e-05 > 2 SNES Function norm 2.373921668024e+03 > Line search: gnorm after quadratic fit 4.771035112853e+03 > Line search: Cubic step no good, shrinking lambda, current gnorm > 2.410650718394e+03 lambda=1.0000000000000002e-02 > Line search: Cubic step no good, shrinking lambda, current gnorm > 2.375104094198e+03 lambda=1.8983783738011522e-03 > Line search: Cubic step no good, shrinking lambda, current gnorm > 2.374049151562e+03 lambda=7.0688528086485479e-04 > Line search: Cubic step no good, shrinking lambda, current gnorm > 2.373935090907e+03 lambda=2.9132722794896019e-04 > Line search: Cubically determined step, current gnorm > 2.373921032081e+03 lambda=1.2527602265373028e-04 > 3 SNES Function norm 2.373921032081e+03 > Line search: gnorm after quadratic fit 5.117914832660e+03 > Line search: Cubic step no good, shrinking lambda, current gnorm > 2.422635362094e+03 lambda=1.0000000000000002e-02 > Line search: Cubic step no good, shrinking lambda, current gnorm > 2.375923870970e+03 lambda=1.5769887508300913e-03 > Line search: Cubic step no good, shrinking lambda, current gnorm > 2.374287081592e+03 lambda=4.8018017100729705e-04 > Line search: Cubic step no good, shrinking lambda, current gnorm > 2.373999131160e+03 lambda=1.5908966655977892e-04 > Line search: Cubic step no good, shrinking lambda, current gnorm > 2.373930608274e+03 lambda=5.7603977147935371e-05 > Line search: Cubic step no good, shrinking lambda, current gnorm > 2.373922022038e+03 lambda=2.3884787050507805e-05 > Line search: Cubic step no good, shrinking lambda, current gnorm > 2.373921396636e+03 lambda=1.0282405393471519e-05 > Line search: Cubic step no good, shrinking lambda, current gnorm > 2.373921173719e+03 lambda=4.3868704034012554e-06 > Line search: Cubic step no good, shrinking lambda, current gnorm > 2.373921091304e+03 lambda=1.8774991151727107e-06 > Line search: Cubic step no good, shrinking lambda, current gnorm > 2.373921057168e+03 lambda=8.0331628193813397e-07 > Line search: Cubic step no good, shrinking lambda, current gnorm > 2.373921042769e+03 lambda=3.4377811174560817e-07 > Line search: Cubic step no good, shrinking lambda, current gnorm > 2.373921036645e+03 lambda=1.4712421886880395e-07 > Line search: Cubic step no good, shrinking lambda, current gnorm > 2.373921034032e+03 lambda=6.2965363212312132e-08 > Line search: Cubic step no good, shrinking lambda, current gnorm > 2.373921032916e+03 lambda=2.6947718250469261e-08 > Line search: Cubic step no good, shrinking lambda, current gnorm > 2.373921032438e+03 lambda=1.1533043989179318e-08 > Line search: Cubic step no good, shrinking lambda, current gnorm > 2.373921032233e+03 lambda=4.9359018284334258e-09 > Line search: Cubic step no good, shrinking lambda, current gnorm > 2.373921032146e+03 lambda=2.1124641857409436e-09 > Line search: Cubic step no good, shrinking lambda, current gnorm > 2.373921032109e+03 lambda=9.0409137304143975e-10 > Line search: Cubic step no good, shrinking lambda, current gnorm > 2.373921032093e+03 lambda=3.8693264359674819e-10 > Line search: Cubic step no good, shrinking lambda, current gnorm > 2.373921032086e+03 lambda=1.6559911204766632e-10 > Line search: Cubic step no good, shrinking lambda, current gnorm > 2.373921032083e+03 lambda=7.0873187020534353e-11 > Line search: Cubic step no good, shrinking lambda, current gnorm > 2.373921032081e+03 lambda=3.0332192901757729e-11 > Line search: Cubic step no good, shrinking lambda, current gnorm > 2.373921032081e+03 lambda=1.2981600435512875e-11 > Line search: Cubic step no good, shrinking lambda, current gnorm > 2.373921032081e+03 lambda=5.5559212521628090e-12 > Line search: Cubic step no good, shrinking lambda, current gnorm > 2.373921032081e+03 lambda=2.3777380405336958e-12 > Line search: Cubic step no good, shrinking lambda, current gnorm > 2.373921032081e+03 lambda=1.0176091649134191e-12 > Line search: Cubic step no good, shrinking lambda, current gnorm > 2.373921032081e+03 lambda=4.3555419789721080e-13 > Line search: unable to find good step length! After 27 tries > Line search: fnorm=2.3739210320805191e+03, > gnorm=2.3739210320805323e+03, ynorm=8.5698020038772756e+03, > minlambda=9.9999999999999998e-13, lambda=4.3555419789721080e-13, initial > slope=-5.6355010665542409e+06 > SNES Object: 1 MPI processes > type: newtonls > maximum iterations=50, maximum function evaluations=10000 > tolerances: relative=1e-08, absolute=1e-50, solution=1e-08 > total number of linear solver iterations=4 > total number of function evaluations=48 > norm schedule ALWAYS > SNESLineSearch Object: 1 MPI processes > type: bt > interpolation: cubic > alpha=1.000000e-04 > maxstep=1.000000e+08, minlambda=1.000000e-12 > tolerances: relative=1.000000e-08, absolute=1.000000e-15, > lambda=1.000000e-08 > maximum iterations=40 > KSP Object: 1 MPI processes > type: preonly > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > left preconditioning > using NONE norm type for convergence test > PC Object: 1 MPI processes > type: cholesky > out-of-place factorization > tolerance for zero pivot 2.22045e-14 > matrix ordering: natural > factor fill ratio given 0., needed 0. > Factored matrix follows: > Mat Object: 1 MPI processes > type: mumps > rows=38154, cols=38154 > package used to perform factorization: mumps > total: nonzeros=7080060, allocated nonzeros=7080060 > total number of mallocs used during MatSetValues calls=0 > MUMPS run parameters: > SYM (matrix type): 2 > PAR (host participation): 1 > ICNTL(1) (output for error): 6 > ICNTL(2) (output of diagnostic msg): 0 > ICNTL(3) (output for global info): 0 > ICNTL(4) (level of printing): 0 > ICNTL(5) (input mat struct): 0 > ICNTL(6) (matrix prescaling): 7 > ICNTL(7) (sequential matrix ordering):7 > ICNTL(8) (scaling strategy): 77 > ICNTL(10) (max num of refinements): 0 > ICNTL(11) (error analysis): 0 > ICNTL(12) (efficiency control): 0 > ICNTL(13) (efficiency control): 0 > ICNTL(14) (percentage of estimated workspace increase): 20 > ICNTL(18) (input mat struct): 0 > ICNTL(19) (Schur complement info): 0 > ICNTL(20) (rhs sparse pattern): 0 > ICNTL(21) (solution struct): 0 > ICNTL(22) (in-core/out-of-core facility): 0 > ICNTL(23) (max size of memory can be allocated locally):0 > ICNTL(24) (detection of null pivot rows): 0 > ICNTL(25) (computation of a null space basis): 0 > ICNTL(26) (Schur options for rhs or solution): 0 > ICNTL(27) (experimental parameter): > -32 > ICNTL(28) (use parallel or sequential ordering): 1 > ICNTL(29) (parallel ordering): 0 > ICNTL(30) (user-specified set of entries in inv(A)): 0 > ICNTL(31) (factors is discarded in the solve phase): 0 > ICNTL(33) (compute determinant): 0 > ICNTL(35) (activate BLR based factorization): 0 > ICNTL(36) (choice of BLR factorization variant): 0 > ICNTL(38) (estimated compression rate of LU factors): > 333 > CNTL(1) (relative pivoting threshold): 0.01 > CNTL(2) (stopping criterion of refinement): 1.49012e-08 > CNTL(3) (absolute pivoting threshold): 0. > CNTL(4) (value of static pivoting): -1. > CNTL(5) (fixation for null pivots): 0. > CNTL(7) (dropping parameter for BLR): 0. > RINFO(1) (local estimated flops for the elimination after > analysis): > [0] 2.73979e+09 > RINFO(2) (local estimated flops for the assembly after > factorization): > [0] 1.08826e+07 > RINFO(3) (local estimated flops for the elimination after > factorization): > [0] 2.73979e+09 > INFO(15) (estimated size of (in MB) MUMPS internal data > for running numerical factorization): > [0] 94 > INFO(16) (size of (in MB) MUMPS internal data used during > numerical factorization): > [0] 94 > INFO(23) (num of pivots eliminated on this processor after > factorization): > [0] 38154 > RINFOG(1) (global estimated flops for the elimination > after analysis): 2.73979e+09 > RINFOG(2) (global estimated flops for the assembly after > factorization): 1.08826e+07 > RINFOG(3) (global estimated flops for the elimination > after factorization): 2.73979e+09 > (RINFOG(12) RINFOG(13))*2^INFOG(34) (determinant): > (0.,0.)*(2^0) > INFOG(3) (estimated real workspace for factors on all > processors after analysis): 8377336 > INFOG(4) (estimated integer workspace for factors on all > processors after analysis): 447902 > INFOG(5) (estimated maximum front size in the complete > tree): 990 > INFOG(6) (number of nodes in the complete tree): 2730 > INFOG(7) (ordering option effectively use after analysis): > 5 > INFOG(8) (structural symmetry in percent of the permuted > matrix after analysis): 100 > INFOG(9) (total real/complex workspace to store the matrix > factors after factorization): 8377336 > INFOG(10) (total integer space store the matrix factors > after factorization): 447902 > INFOG(11) (order of largest frontal matrix after > factorization): 990 > INFOG(12) (number of off-diagonal pivots): 10 > INFOG(13) (number of delayed pivots after factorization): > 0 > INFOG(14) (number of memory compress after factorization): > 0 > INFOG(15) (number of steps of iterative refinement after > solution): 0 > INFOG(16) (estimated size (in MB) of all MUMPS internal > data for factorization after analysis: value on the most memory consuming > processor): 94 > INFOG(17) (estimated size of all MUMPS internal data for > factorization after analysis: sum over all processors): 94 > INFOG(18) (size of all MUMPS internal data allocated > during factorization: value on the most memory consuming processor): 94 > INFOG(19) (size of all MUMPS internal data allocated > during factorization: sum over all processors): 94 > INFOG(20) (estimated number of entries in the factors): > 7080060 > INFOG(21) (size in MB of memory effectively used during > factorization - value on the most memory consuming processor): 80 > INFOG(22) (size in MB of memory effectively used during > factorization - sum over all processors): 80 > INFOG(23) (after analysis: value of ICNTL(6) effectively > used): 0 > INFOG(24) (after analysis: value of ICNTL(12) effectively > used): 1 > INFOG(25) (after factorization: number of pivots modified > by static pivoting): 0 > INFOG(28) (after factorization: number of null pivots > encountered): 0 > INFOG(29) (after factorization: effective number of > entries in the factors (sum over all processors)): 7080060 > INFOG(30, 31) (after solution: size in Mbytes of memory > used during solution phase): 92, 92 > INFOG(32) (after analysis: type of analysis done): 1 > INFOG(33) (value used for ICNTL(8)): 7 > INFOG(34) (exponent of the determinant if determinant is > requested): 0 > INFOG(35) (after factorization: number of entries taking > into account BLR factor compression - sum over all processors): 7080060 > INFOG(36) (after analysis: estimated size of all MUMPS > internal data for running BLR in-core - value on the most memory consuming > processor): 0 > INFOG(37) (after analysis: estimated size of all MUMPS > internal data for running BLR in-core - sum over all processors): 0 > INFOG(38) (after analysis: estimated size of all MUMPS > internal data for running BLR out-of-core - value on the most memory > consuming processor): 0 > INFOG(39) (after analysis: estimated size of all MUMPS > internal data for running BLR out-of-core - sum over all processors): 0 > linear system matrix = precond matrix: > Mat Object: 1 MPI processes > type: seqaij > rows=38154, cols=38154 > total: nonzeros=973446, allocated nonzeros=973446 > total number of mallocs used during MatSetValues calls=0 > not using I-node routines > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: > [0]PETSC ERROR: TSStep has failed due to DIVERGED_NONLINEAR_SOLVE, > increase -ts_max_snes_failures or make negative to attempt recovery > [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.12.3, Jan, 03, 2020 > > Thanks, > Ashish > Scientific Computing Division > OnScale > CA, USA > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Mon Jun 22 12:11:54 2020 From: bsmith at petsc.dev (Barry Smith) Date: Mon, 22 Jun 2020 12:11:54 -0500 Subject: [petsc-users] SNES Line search: fixing the non zero solution values of dirichlet nodes during line search In-Reply-To: References: <6F862511-22D9-42D0-9802-1CC36E80182C@anl.gov> Message-ID: > On Jun 22, 2020, at 11:56 AM, Ashish Patel wrote: > > Thanks Stefano, Barry and Hong for your feedback. > Stefano, the option that you suggested is helpful to freeze the dirichlet values. However, it did not completely solve the convergence issue for nonlinear problems using linesearch since the direction vector which is calculated before any Solve uses the residual which is not yet updated for the new Dirichlet values (u(t) =b(t)). The update happens only after the solve at SNES iteration 0. To fix this I now update the residual for nodes adjacent to Dirichlet nodes without depending on any Solve. The Solve stage only updates the solution for dirichlet nodes. Calculating the residual this way resolves the issue. > > Barry, Hong yes you are right, the dirichlet boundary conditions are time dependent. We do solve the equations implicitly and use TSIFunction and TSIJacobian to set up the problem. The dirichlet nodes are part of computational nodes. The jacobian for dirchlet nodes has diagonal 1 and rows & colums 0. For setting the residual in TSIFunction the residual for dirichlet nodes are set to (solution_value-Dirichlet_value). Understood. The reason the line search cannot succeed well but everything is fine with no line search is that you are putting into the Dirichlet nodes (solution_value-Dirichlet_value) for the next time-step but with multistage integrators and line searches the nonlinear function being evaluated is not simply at the new time step. hence for all those other function evaluations the solution you put in (solution_value-Dirichlet_value) is meaningless. That is your "function" on the Dirichlet nodes is not a function because it doesn't depend on the current input. As I indicated before there are two approaches to putting the simulation on solid footing. Rewriting the boundary condition as an ODE if possible so TS manages the handling of the boundary just like the interior points or telling TS you are solving a DAE, this means when you provide the IFunction and IJacobian you take into account that there is no U_t on the left handside for the boundary nodes. > > Regards > Ashish > > On Sun, Jun 21, 2020 at 6:23 PM Zhang, Hong > wrote: > Can you elaborate a bit on how you "set the RHS as increment of (Dirichlet_value-solution_value)?? > > It seems that the boundary condition is imposed as algebraic equations (e.g. 0 = u(t)-b(t) ), and the boundary nodes are included in the computational domain. This approach should work theoretically and is used in some TS examples such as ex17.c, you might want to check if the algebraic equations are implemented properly in your IFunction() and IJacobian(). An alternative approach you can try is to remove the boundary nodes from the unknown variables, see the example ex25.c. In this way you solve ODEs instead of DAEs and do not need to worry about SNES modifying certain values. > > Hong (Mr.) > >> On Jun 19, 2020, at 5:15 PM, Ashish Patel > wrote: >> >> Dear PETSc users, >> >> We use PETSc as part of a finite element method program and we are trying to properly implement Dirichlet boundary conditions for non-linear, transient problems. We find that when we use a line search method it also changes the non-zero solution value of Dirichlet nodes as it steps through the line search iteration. I was wondering if there is a way to freeze some index sets of a vector from changing during the line search operation? >> >> We are using the TS framework to setup the problem and use 'MatZeroRowsColumns' to set the diagonal of the jacobian to 1 for the dirichlet nodes and set the RHS as increment of (Dirichlet_value-solution_value). This works when the line search method is turned off by using '-snes_linesearch_type basic' however using the default 'bt' linesearch, the TS diverges with error shown below. In a separate implementation if we overwrote the dirichlet nodes of the solution vector in TS residual function with the Dirichlet values then the 'bt' line search method converged to the right solution. However we would like to avoid modifying the internal PETSc vector in our implementation. >> >> 0 TS dt 1. time 0. >> 0 SNES Function norm 2.378549386020e+03 >> Line search: gnorm after quadratic fit 4.369235425165e+03 >> Line search: Cubic step no good, shrinking lambda, current gnorm 2.385369069060e+03 lambda=1.0000000000000002e-02 >> Line search: Cubically determined step, current gnorm 2.373925008934e+03 lambda=3.8846250444606093e-03 >> 1 SNES Function norm 2.373925008934e+03 >> Line search: gnorm after quadratic fit 5.006914179995e+03 >> Line search: Cubic step no good, shrinking lambda, current gnorm 2.420957096780e+03 lambda=1.0000000000000002e-02 >> Line search: Cubic step no good, shrinking lambda, current gnorm 2.376034946750e+03 lambda=1.6129422079664700e-03 >> Line search: Cubic step no good, shrinking lambda, current gnorm 2.374313344729e+03 lambda=4.8465026740690043e-04 >> Line search: Cubic step no good, shrinking lambda, current gnorm 2.373999473242e+03 lambda=1.5857251532828948e-04 >> Line search: Cubically determined step, current gnorm 2.373921668024e+03 lambda=5.8116507162753387e-05 >> 2 SNES Function norm 2.373921668024e+03 >> Line search: gnorm after quadratic fit 4.771035112853e+03 >> Line search: Cubic step no good, shrinking lambda, current gnorm 2.410650718394e+03 lambda=1.0000000000000002e-02 >> Line search: Cubic step no good, shrinking lambda, current gnorm 2.375104094198e+03 lambda=1.8983783738011522e-03 >> Line search: Cubic step no good, shrinking lambda, current gnorm 2.374049151562e+03 lambda=7.0688528086485479e-04 >> Line search: Cubic step no good, shrinking lambda, current gnorm 2.373935090907e+03 lambda=2.9132722794896019e-04 >> Line search: Cubically determined step, current gnorm 2.373921032081e+03 lambda=1.2527602265373028e-04 >> 3 SNES Function norm 2.373921032081e+03 >> Line search: gnorm after quadratic fit 5.117914832660e+03 >> Line search: Cubic step no good, shrinking lambda, current gnorm 2.422635362094e+03 lambda=1.0000000000000002e-02 >> Line search: Cubic step no good, shrinking lambda, current gnorm 2.375923870970e+03 lambda=1.5769887508300913e-03 >> Line search: Cubic step no good, shrinking lambda, current gnorm 2.374287081592e+03 lambda=4.8018017100729705e-04 >> Line search: Cubic step no good, shrinking lambda, current gnorm 2.373999131160e+03 lambda=1.5908966655977892e-04 >> Line search: Cubic step no good, shrinking lambda, current gnorm 2.373930608274e+03 lambda=5.7603977147935371e-05 >> Line search: Cubic step no good, shrinking lambda, current gnorm 2.373922022038e+03 lambda=2.3884787050507805e-05 >> Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921396636e+03 lambda=1.0282405393471519e-05 >> Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921173719e+03 lambda=4.3868704034012554e-06 >> Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921091304e+03 lambda=1.8774991151727107e-06 >> Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921057168e+03 lambda=8.0331628193813397e-07 >> Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921042769e+03 lambda=3.4377811174560817e-07 >> Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921036645e+03 lambda=1.4712421886880395e-07 >> Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921034032e+03 lambda=6.2965363212312132e-08 >> Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921032916e+03 lambda=2.6947718250469261e-08 >> Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921032438e+03 lambda=1.1533043989179318e-08 >> Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921032233e+03 lambda=4.9359018284334258e-09 >> Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921032146e+03 lambda=2.1124641857409436e-09 >> Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921032109e+03 lambda=9.0409137304143975e-10 >> Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921032093e+03 lambda=3.8693264359674819e-10 >> Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921032086e+03 lambda=1.6559911204766632e-10 >> Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921032083e+03 lambda=7.0873187020534353e-11 >> Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921032081e+03 lambda=3.0332192901757729e-11 >> Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921032081e+03 lambda=1.2981600435512875e-11 >> Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921032081e+03 lambda=5.5559212521628090e-12 >> Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921032081e+03 lambda=2.3777380405336958e-12 >> Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921032081e+03 lambda=1.0176091649134191e-12 >> Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921032081e+03 lambda=4.3555419789721080e-13 >> Line search: unable to find good step length! After 27 tries >> Line search: fnorm=2.3739210320805191e+03, gnorm=2.3739210320805323e+03, ynorm=8.5698020038772756e+03, minlambda=9.9999999999999998e-13, lambda=4.3555419789721080e-13, initial slope=-5.6355010665542409e+06 >> SNES Object: 1 MPI processes >> type: newtonls >> maximum iterations=50, maximum function evaluations=10000 >> tolerances: relative=1e-08, absolute=1e-50, solution=1e-08 >> total number of linear solver iterations=4 >> total number of function evaluations=48 >> norm schedule ALWAYS >> SNESLineSearch Object: 1 MPI processes >> type: bt >> interpolation: cubic >> alpha=1.000000e-04 >> maxstep=1.000000e+08, minlambda=1.000000e-12 >> tolerances: relative=1.000000e-08, absolute=1.000000e-15, lambda=1.000000e-08 >> maximum iterations=40 >> KSP Object: 1 MPI processes >> type: preonly >> maximum iterations=10000, initial guess is zero >> tolerances: relative=1e-05, absolute=1e-50, divergence=10000. >> left preconditioning >> using NONE norm type for convergence test >> PC Object: 1 MPI processes >> type: cholesky >> out-of-place factorization >> tolerance for zero pivot 2.22045e-14 >> matrix ordering: natural >> factor fill ratio given 0., needed 0. >> Factored matrix follows: >> Mat Object: 1 MPI processes >> type: mumps >> rows=38154, cols=38154 >> package used to perform factorization: mumps >> total: nonzeros=7080060, allocated nonzeros=7080060 >> total number of mallocs used during MatSetValues calls=0 >> MUMPS run parameters: >> SYM (matrix type): 2 >> PAR (host participation): 1 >> ICNTL(1) (output for error): 6 >> ICNTL(2) (output of diagnostic msg): 0 >> ICNTL(3) (output for global info): 0 >> ICNTL(4) (level of printing): 0 >> ICNTL(5) (input mat struct): 0 >> ICNTL(6) (matrix prescaling): 7 >> ICNTL(7) (sequential matrix ordering):7 >> ICNTL(8) (scaling strategy): 77 >> ICNTL(10) (max num of refinements): 0 >> ICNTL(11) (error analysis): 0 >> ICNTL(12) (efficiency control): 0 >> ICNTL(13) (efficiency control): 0 >> ICNTL(14) (percentage of estimated workspace increase): 20 >> ICNTL(18) (input mat struct): 0 >> ICNTL(19) (Schur complement info): 0 >> ICNTL(20) (rhs sparse pattern): 0 >> ICNTL(21) (solution struct): 0 >> ICNTL(22) (in-core/out-of-core facility): 0 >> ICNTL(23) (max size of memory can be allocated locally):0 >> ICNTL(24) (detection of null pivot rows): 0 >> ICNTL(25) (computation of a null space basis): 0 >> ICNTL(26) (Schur options for rhs or solution): 0 >> ICNTL(27) (experimental parameter): -32 >> ICNTL(28) (use parallel or sequential ordering): 1 >> ICNTL(29) (parallel ordering): 0 >> ICNTL(30) (user-specified set of entries in inv(A)): 0 >> ICNTL(31) (factors is discarded in the solve phase): 0 >> ICNTL(33) (compute determinant): 0 >> ICNTL(35) (activate BLR based factorization): 0 >> ICNTL(36) (choice of BLR factorization variant): 0 >> ICNTL(38) (estimated compression rate of LU factors): 333 >> CNTL(1) (relative pivoting threshold): 0.01 >> CNTL(2) (stopping criterion of refinement): 1.49012e-08 >> CNTL(3) (absolute pivoting threshold): 0. >> CNTL(4) (value of static pivoting): -1. >> CNTL(5) (fixation for null pivots): 0. >> CNTL(7) (dropping parameter for BLR): 0. >> RINFO(1) (local estimated flops for the elimination after analysis): >> [0] 2.73979e+09 >> RINFO(2) (local estimated flops for the assembly after factorization): >> [0] 1.08826e+07 >> RINFO(3) (local estimated flops for the elimination after factorization): >> [0] 2.73979e+09 >> INFO(15) (estimated size of (in MB) MUMPS internal data for running numerical factorization): >> [0] 94 >> INFO(16) (size of (in MB) MUMPS internal data used during numerical factorization): >> [0] 94 >> INFO(23) (num of pivots eliminated on this processor after factorization): >> [0] 38154 >> RINFOG(1) (global estimated flops for the elimination after analysis): 2.73979e+09 >> RINFOG(2) (global estimated flops for the assembly after factorization): 1.08826e+07 >> RINFOG(3) (global estimated flops for the elimination after factorization): 2.73979e+09 >> (RINFOG(12) RINFOG(13))*2^INFOG(34) (determinant): (0.,0.)*(2^0) >> INFOG(3) (estimated real workspace for factors on all processors after analysis): 8377336 >> INFOG(4) (estimated integer workspace for factors on all processors after analysis): 447902 >> INFOG(5) (estimated maximum front size in the complete tree): 990 >> INFOG(6) (number of nodes in the complete tree): 2730 >> INFOG(7) (ordering option effectively use after analysis): 5 >> INFOG(8) (structural symmetry in percent of the permuted matrix after analysis): 100 >> INFOG(9) (total real/complex workspace to store the matrix factors after factorization): 8377336 >> INFOG(10) (total integer space store the matrix factors after factorization): 447902 >> INFOG(11) (order of largest frontal matrix after factorization): 990 >> INFOG(12) (number of off-diagonal pivots): 10 >> INFOG(13) (number of delayed pivots after factorization): 0 >> INFOG(14) (number of memory compress after factorization): 0 >> INFOG(15) (number of steps of iterative refinement after solution): 0 >> INFOG(16) (estimated size (in MB) of all MUMPS internal data for factorization after analysis: value on the most memory consuming processor): 94 >> INFOG(17) (estimated size of all MUMPS internal data for factorization after analysis: sum over all processors): 94 >> INFOG(18) (size of all MUMPS internal data allocated during factorization: value on the most memory consuming processor): 94 >> INFOG(19) (size of all MUMPS internal data allocated during factorization: sum over all processors): 94 >> INFOG(20) (estimated number of entries in the factors): 7080060 >> INFOG(21) (size in MB of memory effectively used during factorization - value on the most memory consuming processor): 80 >> INFOG(22) (size in MB of memory effectively used during factorization - sum over all processors): 80 >> INFOG(23) (after analysis: value of ICNTL(6) effectively used): 0 >> INFOG(24) (after analysis: value of ICNTL(12) effectively used): 1 >> INFOG(25) (after factorization: number of pivots modified by static pivoting): 0 >> INFOG(28) (after factorization: number of null pivots encountered): 0 >> INFOG(29) (after factorization: effective number of entries in the factors (sum over all processors)): 7080060 >> INFOG(30, 31) (after solution: size in Mbytes of memory used during solution phase): 92, 92 >> INFOG(32) (after analysis: type of analysis done): 1 >> INFOG(33) (value used for ICNTL(8)): 7 >> INFOG(34) (exponent of the determinant if determinant is requested): 0 >> INFOG(35) (after factorization: number of entries taking into account BLR factor compression - sum over all processors): 7080060 >> INFOG(36) (after analysis: estimated size of all MUMPS internal data for running BLR in-core - value on the most memory consuming processor): 0 >> INFOG(37) (after analysis: estimated size of all MUMPS internal data for running BLR in-core - sum over all processors): 0 >> INFOG(38) (after analysis: estimated size of all MUMPS internal data for running BLR out-of-core - value on the most memory consuming processor): 0 >> INFOG(39) (after analysis: estimated size of all MUMPS internal data for running BLR out-of-core - sum over all processors): 0 >> linear system matrix = precond matrix: >> Mat Object: 1 MPI processes >> type: seqaij >> rows=38154, cols=38154 >> total: nonzeros=973446, allocated nonzeros=973446 >> total number of mallocs used during MatSetValues calls=0 >> not using I-node routines >> [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >> [0]PETSC ERROR: >> [0]PETSC ERROR: TSStep has failed due to DIVERGED_NONLINEAR_SOLVE, increase -ts_max_snes_failures or make negative to attempt recovery >> [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. >> [0]PETSC ERROR: Petsc Release Version 3.12.3, Jan, 03, 2020 >> >> Thanks, >> Ashish >> Scientific Computing Division >> OnScale >> CA, USA > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Jun 22 12:19:25 2020 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 22 Jun 2020 13:19:25 -0400 Subject: [petsc-users] SNES Line search: fixing the non zero solution values of dirichlet nodes during line search In-Reply-To: References: <6F862511-22D9-42D0-9802-1CC36E80182C@anl.gov> Message-ID: On Mon, Jun 22, 2020 at 1:13 PM Barry Smith wrote: > > > On Jun 22, 2020, at 11:56 AM, Ashish Patel > wrote: > > Thanks Stefano, Barry and Hong for your feedback. > Stefano, the option that you suggested is helpful to freeze the dirichlet > values. However, it did not completely solve the convergence issue for > nonlinear problems using linesearch since the direction vector which is > calculated before any Solve uses the residual which is not yet updated for > the new Dirichlet values (u(t) =b(t)). The update happens only after the > solve at SNES iteration 0. To fix this I now update the residual for nodes > adjacent to Dirichlet nodes without depending on any Solve. The Solve stage > only updates the solution for dirichlet nodes. Calculating the residual > this way resolves the issue. > > Barry, Hong yes you are right, the dirichlet boundary conditions are time > dependent. We do solve the equations implicitly and use TSIFunction and > TSIJacobian to set up the problem. The dirichlet nodes are part of > computational nodes. The jacobian for dirchlet nodes has diagonal 1 and > rows & colums 0. For setting the residual in TSIFunction the residual for > dirichlet nodes are set to (solution_value-Dirichlet_value). > > > Understood. The reason the line search cannot succeed well but > everything is fine with no line search is that you are putting into the > Dirichlet nodes (solution_value-Dirichlet_value) for the next time-step but > with multistage integrators and line searches the nonlinear function being > evaluated is not simply at the new time step. hence for all those other > function evaluations the solution you put in > (solution_value-Dirichlet_value) is meaningless. That is your "function" on > the Dirichlet nodes is not a function because it doesn't depend on the > current input. > > As I indicated before there are two approaches to putting the simulation > on solid footing. Rewriting the boundary condition as an ODE if possible so > TS manages the handling of the boundary just like the interior points or > telling TS you are solving a DAE, this means when you provide the IFunction > and IJacobian you take into account that there is no U_t on the left > handside for the boundary nodes. > There is a third strategy, which is the one employed by DMPlex for all BC, including time dependent. It eliminates the dofs constrained by boundary conditions for the system. For a simple example, see TS ex45 or ex46. There, time-dependent boundary conditions are used. Thanks, Matt > Regards > Ashish > > On Sun, Jun 21, 2020 at 6:23 PM Zhang, Hong wrote: > >> Can you elaborate a bit on how you "set the RHS as increment of >> (Dirichlet_value-solution_value)?? >> >> It seems that the boundary condition is imposed as algebraic equations >> (e.g. 0 = u(t)-b(t) ), and the boundary nodes are included in the >> computational domain. This approach should work theoretically and is used >> in some TS examples such as ex17.c, you might want to check if the >> algebraic equations are implemented properly in your IFunction() and >> IJacobian(). An alternative approach you can try is to remove the boundary >> nodes from the unknown variables, see the example ex25.c. In this way you >> solve ODEs instead of DAEs and do not need to worry about SNES modifying >> certain values. >> >> Hong (Mr.) >> >> On Jun 19, 2020, at 5:15 PM, Ashish Patel >> wrote: >> >> Dear PETSc users, >> >> We use PETSc as part of a finite element method program and we are trying >> to properly implement Dirichlet boundary conditions for non-linear, >> transient problems. We find that when we use a line search method it also >> changes the non-zero solution value of Dirichlet nodes as it steps through >> the line search iteration. I was wondering if there is a way to freeze >> some index sets of a vector from changing during the line search operation? >> >> We are using the TS framework to setup the problem and use >> 'MatZeroRowsColumns' to set the diagonal of the jacobian to 1 for the >> dirichlet nodes and set the RHS as increment of >> (Dirichlet_value-solution_value). This works when the line search method is >> turned off by using '-snes_linesearch_type basic' however using the default >> 'bt' linesearch, the TS diverges with error shown below. In a separate >> implementation if we overwrote the dirichlet nodes of the solution vector >> in TS residual function with the Dirichlet values then the 'bt' line >> search method converged to the right solution. However we would like to >> avoid modifying the internal PETSc vector in our implementation. >> >> 0 TS dt 1. time 0. >> 0 SNES Function norm 2.378549386020e+03 >> Line search: gnorm after quadratic fit 4.369235425165e+03 >> Line search: Cubic step no good, shrinking lambda, current gnorm >> 2.385369069060e+03 lambda=1.0000000000000002e-02 >> Line search: Cubically determined step, current gnorm >> 2.373925008934e+03 lambda=3.8846250444606093e-03 >> 1 SNES Function norm 2.373925008934e+03 >> Line search: gnorm after quadratic fit 5.006914179995e+03 >> Line search: Cubic step no good, shrinking lambda, current gnorm >> 2.420957096780e+03 lambda=1.0000000000000002e-02 >> Line search: Cubic step no good, shrinking lambda, current gnorm >> 2.376034946750e+03 lambda=1.6129422079664700e-03 >> Line search: Cubic step no good, shrinking lambda, current gnorm >> 2.374313344729e+03 lambda=4.8465026740690043e-04 >> Line search: Cubic step no good, shrinking lambda, current gnorm >> 2.373999473242e+03 lambda=1.5857251532828948e-04 >> Line search: Cubically determined step, current gnorm >> 2.373921668024e+03 lambda=5.8116507162753387e-05 >> 2 SNES Function norm 2.373921668024e+03 >> Line search: gnorm after quadratic fit 4.771035112853e+03 >> Line search: Cubic step no good, shrinking lambda, current gnorm >> 2.410650718394e+03 lambda=1.0000000000000002e-02 >> Line search: Cubic step no good, shrinking lambda, current gnorm >> 2.375104094198e+03 lambda=1.8983783738011522e-03 >> Line search: Cubic step no good, shrinking lambda, current gnorm >> 2.374049151562e+03 lambda=7.0688528086485479e-04 >> Line search: Cubic step no good, shrinking lambda, current gnorm >> 2.373935090907e+03 lambda=2.9132722794896019e-04 >> Line search: Cubically determined step, current gnorm >> 2.373921032081e+03 lambda=1.2527602265373028e-04 >> 3 SNES Function norm 2.373921032081e+03 >> Line search: gnorm after quadratic fit 5.117914832660e+03 >> Line search: Cubic step no good, shrinking lambda, current gnorm >> 2.422635362094e+03 lambda=1.0000000000000002e-02 >> Line search: Cubic step no good, shrinking lambda, current gnorm >> 2.375923870970e+03 lambda=1.5769887508300913e-03 >> Line search: Cubic step no good, shrinking lambda, current gnorm >> 2.374287081592e+03 lambda=4.8018017100729705e-04 >> Line search: Cubic step no good, shrinking lambda, current gnorm >> 2.373999131160e+03 lambda=1.5908966655977892e-04 >> Line search: Cubic step no good, shrinking lambda, current gnorm >> 2.373930608274e+03 lambda=5.7603977147935371e-05 >> Line search: Cubic step no good, shrinking lambda, current gnorm >> 2.373922022038e+03 lambda=2.3884787050507805e-05 >> Line search: Cubic step no good, shrinking lambda, current gnorm >> 2.373921396636e+03 lambda=1.0282405393471519e-05 >> Line search: Cubic step no good, shrinking lambda, current gnorm >> 2.373921173719e+03 lambda=4.3868704034012554e-06 >> Line search: Cubic step no good, shrinking lambda, current gnorm >> 2.373921091304e+03 lambda=1.8774991151727107e-06 >> Line search: Cubic step no good, shrinking lambda, current gnorm >> 2.373921057168e+03 lambda=8.0331628193813397e-07 >> Line search: Cubic step no good, shrinking lambda, current gnorm >> 2.373921042769e+03 lambda=3.4377811174560817e-07 >> Line search: Cubic step no good, shrinking lambda, current gnorm >> 2.373921036645e+03 lambda=1.4712421886880395e-07 >> Line search: Cubic step no good, shrinking lambda, current gnorm >> 2.373921034032e+03 lambda=6.2965363212312132e-08 >> Line search: Cubic step no good, shrinking lambda, current gnorm >> 2.373921032916e+03 lambda=2.6947718250469261e-08 >> Line search: Cubic step no good, shrinking lambda, current gnorm >> 2.373921032438e+03 lambda=1.1533043989179318e-08 >> Line search: Cubic step no good, shrinking lambda, current gnorm >> 2.373921032233e+03 lambda=4.9359018284334258e-09 >> Line search: Cubic step no good, shrinking lambda, current gnorm >> 2.373921032146e+03 lambda=2.1124641857409436e-09 >> Line search: Cubic step no good, shrinking lambda, current gnorm >> 2.373921032109e+03 lambda=9.0409137304143975e-10 >> Line search: Cubic step no good, shrinking lambda, current gnorm >> 2.373921032093e+03 lambda=3.8693264359674819e-10 >> Line search: Cubic step no good, shrinking lambda, current gnorm >> 2.373921032086e+03 lambda=1.6559911204766632e-10 >> Line search: Cubic step no good, shrinking lambda, current gnorm >> 2.373921032083e+03 lambda=7.0873187020534353e-11 >> Line search: Cubic step no good, shrinking lambda, current gnorm >> 2.373921032081e+03 lambda=3.0332192901757729e-11 >> Line search: Cubic step no good, shrinking lambda, current gnorm >> 2.373921032081e+03 lambda=1.2981600435512875e-11 >> Line search: Cubic step no good, shrinking lambda, current gnorm >> 2.373921032081e+03 lambda=5.5559212521628090e-12 >> Line search: Cubic step no good, shrinking lambda, current gnorm >> 2.373921032081e+03 lambda=2.3777380405336958e-12 >> Line search: Cubic step no good, shrinking lambda, current gnorm >> 2.373921032081e+03 lambda=1.0176091649134191e-12 >> Line search: Cubic step no good, shrinking lambda, current gnorm >> 2.373921032081e+03 lambda=4.3555419789721080e-13 >> Line search: unable to find good step length! After 27 tries >> Line search: fnorm=2.3739210320805191e+03, >> gnorm=2.3739210320805323e+03, ynorm=8.5698020038772756e+03, >> minlambda=9.9999999999999998e-13, lambda=4.3555419789721080e-13, initial >> slope=-5.6355010665542409e+06 >> SNES Object: 1 MPI processes >> type: newtonls >> maximum iterations=50, maximum function evaluations=10000 >> tolerances: relative=1e-08, absolute=1e-50, solution=1e-08 >> total number of linear solver iterations=4 >> total number of function evaluations=48 >> norm schedule ALWAYS >> SNESLineSearch Object: 1 MPI processes >> type: bt >> interpolation: cubic >> alpha=1.000000e-04 >> maxstep=1.000000e+08, minlambda=1.000000e-12 >> tolerances: relative=1.000000e-08, absolute=1.000000e-15, >> lambda=1.000000e-08 >> maximum iterations=40 >> KSP Object: 1 MPI processes >> type: preonly >> maximum iterations=10000, initial guess is zero >> tolerances: relative=1e-05, absolute=1e-50, divergence=10000. >> left preconditioning >> using NONE norm type for convergence test >> PC Object: 1 MPI processes >> type: cholesky >> out-of-place factorization >> tolerance for zero pivot 2.22045e-14 >> matrix ordering: natural >> factor fill ratio given 0., needed 0. >> Factored matrix follows: >> Mat Object: 1 MPI processes >> type: mumps >> rows=38154, cols=38154 >> package used to perform factorization: mumps >> total: nonzeros=7080060, allocated nonzeros=7080060 >> total number of mallocs used during MatSetValues calls=0 >> MUMPS run parameters: >> SYM (matrix type): 2 >> PAR (host participation): 1 >> ICNTL(1) (output for error): 6 >> ICNTL(2) (output of diagnostic msg): 0 >> ICNTL(3) (output for global info): 0 >> ICNTL(4) (level of printing): 0 >> ICNTL(5) (input mat struct): 0 >> ICNTL(6) (matrix prescaling): 7 >> ICNTL(7) (sequential matrix ordering):7 >> ICNTL(8) (scaling strategy): 77 >> ICNTL(10) (max num of refinements): 0 >> ICNTL(11) (error analysis): 0 >> ICNTL(12) (efficiency control): 0 >> ICNTL(13) (efficiency control): 0 >> ICNTL(14) (percentage of estimated workspace increase): >> 20 >> ICNTL(18) (input mat struct): 0 >> ICNTL(19) (Schur complement info): 0 >> ICNTL(20) (rhs sparse pattern): 0 >> ICNTL(21) (solution struct): 0 >> ICNTL(22) (in-core/out-of-core facility): 0 >> ICNTL(23) (max size of memory can be allocated locally):0 >> ICNTL(24) (detection of null pivot rows): 0 >> ICNTL(25) (computation of a null space basis): 0 >> ICNTL(26) (Schur options for rhs or solution): 0 >> ICNTL(27) (experimental parameter): >> -32 >> ICNTL(28) (use parallel or sequential ordering): 1 >> ICNTL(29) (parallel ordering): 0 >> ICNTL(30) (user-specified set of entries in inv(A)): 0 >> ICNTL(31) (factors is discarded in the solve phase): 0 >> ICNTL(33) (compute determinant): 0 >> ICNTL(35) (activate BLR based factorization): 0 >> ICNTL(36) (choice of BLR factorization variant): 0 >> ICNTL(38) (estimated compression rate of LU factors): >> 333 >> CNTL(1) (relative pivoting threshold): 0.01 >> CNTL(2) (stopping criterion of refinement): 1.49012e-08 >> CNTL(3) (absolute pivoting threshold): 0. >> CNTL(4) (value of static pivoting): -1. >> CNTL(5) (fixation for null pivots): 0. >> CNTL(7) (dropping parameter for BLR): 0. >> RINFO(1) (local estimated flops for the elimination after >> analysis): >> [0] 2.73979e+09 >> RINFO(2) (local estimated flops for the assembly after >> factorization): >> [0] 1.08826e+07 >> RINFO(3) (local estimated flops for the elimination after >> factorization): >> [0] 2.73979e+09 >> INFO(15) (estimated size of (in MB) MUMPS internal data >> for running numerical factorization): >> [0] 94 >> INFO(16) (size of (in MB) MUMPS internal data used during >> numerical factorization): >> [0] 94 >> INFO(23) (num of pivots eliminated on this processor >> after factorization): >> [0] 38154 >> RINFOG(1) (global estimated flops for the elimination >> after analysis): 2.73979e+09 >> RINFOG(2) (global estimated flops for the assembly after >> factorization): 1.08826e+07 >> RINFOG(3) (global estimated flops for the elimination >> after factorization): 2.73979e+09 >> (RINFOG(12) RINFOG(13))*2^INFOG(34) (determinant): >> (0.,0.)*(2^0) >> INFOG(3) (estimated real workspace for factors on all >> processors after analysis): 8377336 >> INFOG(4) (estimated integer workspace for factors on all >> processors after analysis): 447902 >> INFOG(5) (estimated maximum front size in the complete >> tree): 990 >> INFOG(6) (number of nodes in the complete tree): 2730 >> INFOG(7) (ordering option effectively use after >> analysis): 5 >> INFOG(8) (structural symmetry in percent of the permuted >> matrix after analysis): 100 >> INFOG(9) (total real/complex workspace to store the >> matrix factors after factorization): 8377336 >> INFOG(10) (total integer space store the matrix factors >> after factorization): 447902 >> INFOG(11) (order of largest frontal matrix after >> factorization): 990 >> INFOG(12) (number of off-diagonal pivots): 10 >> INFOG(13) (number of delayed pivots after factorization): >> 0 >> INFOG(14) (number of memory compress after >> factorization): 0 >> INFOG(15) (number of steps of iterative refinement after >> solution): 0 >> INFOG(16) (estimated size (in MB) of all MUMPS internal >> data for factorization after analysis: value on the most memory consuming >> processor): 94 >> INFOG(17) (estimated size of all MUMPS internal data for >> factorization after analysis: sum over all processors): 94 >> INFOG(18) (size of all MUMPS internal data allocated >> during factorization: value on the most memory consuming processor): 94 >> INFOG(19) (size of all MUMPS internal data allocated >> during factorization: sum over all processors): 94 >> INFOG(20) (estimated number of entries in the factors): >> 7080060 >> INFOG(21) (size in MB of memory effectively used during >> factorization - value on the most memory consuming processor): 80 >> INFOG(22) (size in MB of memory effectively used during >> factorization - sum over all processors): 80 >> INFOG(23) (after analysis: value of ICNTL(6) effectively >> used): 0 >> INFOG(24) (after analysis: value of ICNTL(12) effectively >> used): 1 >> INFOG(25) (after factorization: number of pivots modified >> by static pivoting): 0 >> INFOG(28) (after factorization: number of null pivots >> encountered): 0 >> INFOG(29) (after factorization: effective number of >> entries in the factors (sum over all processors)): 7080060 >> INFOG(30, 31) (after solution: size in Mbytes of memory >> used during solution phase): 92, 92 >> INFOG(32) (after analysis: type of analysis done): 1 >> INFOG(33) (value used for ICNTL(8)): 7 >> INFOG(34) (exponent of the determinant if determinant is >> requested): 0 >> INFOG(35) (after factorization: number of entries taking >> into account BLR factor compression - sum over all processors): 7080060 >> INFOG(36) (after analysis: estimated size of all MUMPS >> internal data for running BLR in-core - value on the most memory consuming >> processor): 0 >> INFOG(37) (after analysis: estimated size of all MUMPS >> internal data for running BLR in-core - sum over all processors): 0 >> INFOG(38) (after analysis: estimated size of all MUMPS >> internal data for running BLR out-of-core - value on the most memory >> consuming processor): 0 >> INFOG(39) (after analysis: estimated size of all MUMPS >> internal data for running BLR out-of-core - sum over all processors): 0 >> linear system matrix = precond matrix: >> Mat Object: 1 MPI processes >> type: seqaij >> rows=38154, cols=38154 >> total: nonzeros=973446, allocated nonzeros=973446 >> total number of mallocs used during MatSetValues calls=0 >> not using I-node routines >> [0]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> [0]PETSC ERROR: >> [0]PETSC ERROR: TSStep has failed due to DIVERGED_NONLINEAR_SOLVE, >> increase -ts_max_snes_failures or make negative to attempt recovery >> [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html >> for trouble shooting. >> [0]PETSC ERROR: Petsc Release Version 3.12.3, Jan, 03, 2020 >> >> Thanks, >> Ashish >> Scientific Computing Division >> OnScale >> CA, USA >> >> >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ashish.patel at onscale.com Mon Jun 22 19:48:58 2020 From: ashish.patel at onscale.com (Ashish Patel) Date: Mon, 22 Jun 2020 17:48:58 -0700 Subject: [petsc-users] SNES Line search: fixing the non zero solution values of dirichlet nodes during line search In-Reply-To: References: <6F862511-22D9-42D0-9802-1CC36E80182C@anl.gov> Message-ID: Thanks for the suggestions Barry and Matt. DMAddBoundary surely seems interesting for further improvement. Currently we settled on setting up the problem as DAE and using the time available from the function evaluation routine of TSIFunction for Dirichlet_value evaluation. Also the residual for nodes adjacent to Dirichlet nodes are now updated using the Dirichlet values, instead of the solution value which is updated only after the first SNES iteration, This change makes the line search method more robust for us. Thanks Ashish On Mon, Jun 22, 2020 at 10:19 AM Matthew Knepley wrote: > On Mon, Jun 22, 2020 at 1:13 PM Barry Smith wrote: > >> >> >> On Jun 22, 2020, at 11:56 AM, Ashish Patel >> wrote: >> >> Thanks Stefano, Barry and Hong for your feedback. >> Stefano, the option that you suggested is helpful to freeze the dirichlet >> values. However, it did not completely solve the convergence issue for >> nonlinear problems using linesearch since the direction vector which is >> calculated before any Solve uses the residual which is not yet updated for >> the new Dirichlet values (u(t) =b(t)). The update happens only after the >> solve at SNES iteration 0. To fix this I now update the residual for nodes >> adjacent to Dirichlet nodes without depending on any Solve. The Solve stage >> only updates the solution for dirichlet nodes. Calculating the residual >> this way resolves the issue. >> >> Barry, Hong yes you are right, the dirichlet boundary conditions are time >> dependent. We do solve the equations implicitly and use TSIFunction and >> TSIJacobian to set up the problem. The dirichlet nodes are part of >> computational nodes. The jacobian for dirchlet nodes has diagonal 1 and >> rows & colums 0. For setting the residual in TSIFunction the residual for >> dirichlet nodes are set to (solution_value-Dirichlet_value). >> >> >> Understood. The reason the line search cannot succeed well but >> everything is fine with no line search is that you are putting into the >> Dirichlet nodes (solution_value-Dirichlet_value) for the next time-step but >> with multistage integrators and line searches the nonlinear function being >> evaluated is not simply at the new time step. hence for all those other >> function evaluations the solution you put in >> (solution_value-Dirichlet_value) is meaningless. That is your "function" on >> the Dirichlet nodes is not a function because it doesn't depend on the >> current input. >> >> As I indicated before there are two approaches to putting the >> simulation on solid footing. Rewriting the boundary condition as an ODE if >> possible so TS manages the handling of the boundary just like the interior >> points or telling TS you are solving a DAE, this means when you provide the >> IFunction and IJacobian you take into account that there is no U_t on the >> left handside for the boundary nodes. >> > > There is a third strategy, which is the one employed by DMPlex for all BC, > including time dependent. It eliminates the dofs constrained by boundary > conditions > for the system. For a simple example, see TS ex45 or ex46. There, > time-dependent boundary conditions are used. > > Thanks, > > Matt > > >> Regards >> Ashish >> >> On Sun, Jun 21, 2020 at 6:23 PM Zhang, Hong wrote: >> >>> Can you elaborate a bit on how you "set the RHS as increment of >>> (Dirichlet_value-solution_value)?? >>> >>> It seems that the boundary condition is imposed as algebraic equations >>> (e.g. 0 = u(t)-b(t) ), and the boundary nodes are included in the >>> computational domain. This approach should work theoretically and is used >>> in some TS examples such as ex17.c, you might want to check if the >>> algebraic equations are implemented properly in your IFunction() and >>> IJacobian(). An alternative approach you can try is to remove the boundary >>> nodes from the unknown variables, see the example ex25.c. In this way you >>> solve ODEs instead of DAEs and do not need to worry about SNES modifying >>> certain values. >>> >>> Hong (Mr.) >>> >>> On Jun 19, 2020, at 5:15 PM, Ashish Patel >>> wrote: >>> >>> Dear PETSc users, >>> >>> We use PETSc as part of a finite element method program and we are >>> trying to properly implement Dirichlet boundary conditions for non-linear, >>> transient problems. We find that when we use a line search method it also >>> changes the non-zero solution value of Dirichlet nodes as it steps through >>> the line search iteration. I was wondering if there is a way to freeze >>> some index sets of a vector from changing during the line search operation? >>> >>> We are using the TS framework to setup the problem and use >>> 'MatZeroRowsColumns' to set the diagonal of the jacobian to 1 for the >>> dirichlet nodes and set the RHS as increment of >>> (Dirichlet_value-solution_value). This works when the line search method is >>> turned off by using '-snes_linesearch_type basic' however using the default >>> 'bt' linesearch, the TS diverges with error shown below. In a separate >>> implementation if we overwrote the dirichlet nodes of the solution vector >>> in TS residual function with the Dirichlet values then the 'bt' line >>> search method converged to the right solution. However we would like to >>> avoid modifying the internal PETSc vector in our implementation. >>> >>> 0 TS dt 1. time 0. >>> 0 SNES Function norm 2.378549386020e+03 >>> Line search: gnorm after quadratic fit 4.369235425165e+03 >>> Line search: Cubic step no good, shrinking lambda, current gnorm >>> 2.385369069060e+03 lambda=1.0000000000000002e-02 >>> Line search: Cubically determined step, current gnorm >>> 2.373925008934e+03 lambda=3.8846250444606093e-03 >>> 1 SNES Function norm 2.373925008934e+03 >>> Line search: gnorm after quadratic fit 5.006914179995e+03 >>> Line search: Cubic step no good, shrinking lambda, current gnorm >>> 2.420957096780e+03 lambda=1.0000000000000002e-02 >>> Line search: Cubic step no good, shrinking lambda, current gnorm >>> 2.376034946750e+03 lambda=1.6129422079664700e-03 >>> Line search: Cubic step no good, shrinking lambda, current gnorm >>> 2.374313344729e+03 lambda=4.8465026740690043e-04 >>> Line search: Cubic step no good, shrinking lambda, current gnorm >>> 2.373999473242e+03 lambda=1.5857251532828948e-04 >>> Line search: Cubically determined step, current gnorm >>> 2.373921668024e+03 lambda=5.8116507162753387e-05 >>> 2 SNES Function norm 2.373921668024e+03 >>> Line search: gnorm after quadratic fit 4.771035112853e+03 >>> Line search: Cubic step no good, shrinking lambda, current gnorm >>> 2.410650718394e+03 lambda=1.0000000000000002e-02 >>> Line search: Cubic step no good, shrinking lambda, current gnorm >>> 2.375104094198e+03 lambda=1.8983783738011522e-03 >>> Line search: Cubic step no good, shrinking lambda, current gnorm >>> 2.374049151562e+03 lambda=7.0688528086485479e-04 >>> Line search: Cubic step no good, shrinking lambda, current gnorm >>> 2.373935090907e+03 lambda=2.9132722794896019e-04 >>> Line search: Cubically determined step, current gnorm >>> 2.373921032081e+03 lambda=1.2527602265373028e-04 >>> 3 SNES Function norm 2.373921032081e+03 >>> Line search: gnorm after quadratic fit 5.117914832660e+03 >>> Line search: Cubic step no good, shrinking lambda, current gnorm >>> 2.422635362094e+03 lambda=1.0000000000000002e-02 >>> Line search: Cubic step no good, shrinking lambda, current gnorm >>> 2.375923870970e+03 lambda=1.5769887508300913e-03 >>> Line search: Cubic step no good, shrinking lambda, current gnorm >>> 2.374287081592e+03 lambda=4.8018017100729705e-04 >>> Line search: Cubic step no good, shrinking lambda, current gnorm >>> 2.373999131160e+03 lambda=1.5908966655977892e-04 >>> Line search: Cubic step no good, shrinking lambda, current gnorm >>> 2.373930608274e+03 lambda=5.7603977147935371e-05 >>> Line search: Cubic step no good, shrinking lambda, current gnorm >>> 2.373922022038e+03 lambda=2.3884787050507805e-05 >>> Line search: Cubic step no good, shrinking lambda, current gnorm >>> 2.373921396636e+03 lambda=1.0282405393471519e-05 >>> Line search: Cubic step no good, shrinking lambda, current gnorm >>> 2.373921173719e+03 lambda=4.3868704034012554e-06 >>> Line search: Cubic step no good, shrinking lambda, current gnorm >>> 2.373921091304e+03 lambda=1.8774991151727107e-06 >>> Line search: Cubic step no good, shrinking lambda, current gnorm >>> 2.373921057168e+03 lambda=8.0331628193813397e-07 >>> Line search: Cubic step no good, shrinking lambda, current gnorm >>> 2.373921042769e+03 lambda=3.4377811174560817e-07 >>> Line search: Cubic step no good, shrinking lambda, current gnorm >>> 2.373921036645e+03 lambda=1.4712421886880395e-07 >>> Line search: Cubic step no good, shrinking lambda, current gnorm >>> 2.373921034032e+03 lambda=6.2965363212312132e-08 >>> Line search: Cubic step no good, shrinking lambda, current gnorm >>> 2.373921032916e+03 lambda=2.6947718250469261e-08 >>> Line search: Cubic step no good, shrinking lambda, current gnorm >>> 2.373921032438e+03 lambda=1.1533043989179318e-08 >>> Line search: Cubic step no good, shrinking lambda, current gnorm >>> 2.373921032233e+03 lambda=4.9359018284334258e-09 >>> Line search: Cubic step no good, shrinking lambda, current gnorm >>> 2.373921032146e+03 lambda=2.1124641857409436e-09 >>> Line search: Cubic step no good, shrinking lambda, current gnorm >>> 2.373921032109e+03 lambda=9.0409137304143975e-10 >>> Line search: Cubic step no good, shrinking lambda, current gnorm >>> 2.373921032093e+03 lambda=3.8693264359674819e-10 >>> Line search: Cubic step no good, shrinking lambda, current gnorm >>> 2.373921032086e+03 lambda=1.6559911204766632e-10 >>> Line search: Cubic step no good, shrinking lambda, current gnorm >>> 2.373921032083e+03 lambda=7.0873187020534353e-11 >>> Line search: Cubic step no good, shrinking lambda, current gnorm >>> 2.373921032081e+03 lambda=3.0332192901757729e-11 >>> Line search: Cubic step no good, shrinking lambda, current gnorm >>> 2.373921032081e+03 lambda=1.2981600435512875e-11 >>> Line search: Cubic step no good, shrinking lambda, current gnorm >>> 2.373921032081e+03 lambda=5.5559212521628090e-12 >>> Line search: Cubic step no good, shrinking lambda, current gnorm >>> 2.373921032081e+03 lambda=2.3777380405336958e-12 >>> Line search: Cubic step no good, shrinking lambda, current gnorm >>> 2.373921032081e+03 lambda=1.0176091649134191e-12 >>> Line search: Cubic step no good, shrinking lambda, current gnorm >>> 2.373921032081e+03 lambda=4.3555419789721080e-13 >>> Line search: unable to find good step length! After 27 tries >>> Line search: fnorm=2.3739210320805191e+03, >>> gnorm=2.3739210320805323e+03, ynorm=8.5698020038772756e+03, >>> minlambda=9.9999999999999998e-13, lambda=4.3555419789721080e-13, initial >>> slope=-5.6355010665542409e+06 >>> SNES Object: 1 MPI processes >>> type: newtonls >>> maximum iterations=50, maximum function evaluations=10000 >>> tolerances: relative=1e-08, absolute=1e-50, solution=1e-08 >>> total number of linear solver iterations=4 >>> total number of function evaluations=48 >>> norm schedule ALWAYS >>> SNESLineSearch Object: 1 MPI processes >>> type: bt >>> interpolation: cubic >>> alpha=1.000000e-04 >>> maxstep=1.000000e+08, minlambda=1.000000e-12 >>> tolerances: relative=1.000000e-08, absolute=1.000000e-15, >>> lambda=1.000000e-08 >>> maximum iterations=40 >>> KSP Object: 1 MPI processes >>> type: preonly >>> maximum iterations=10000, initial guess is zero >>> tolerances: relative=1e-05, absolute=1e-50, divergence=10000. >>> left preconditioning >>> using NONE norm type for convergence test >>> PC Object: 1 MPI processes >>> type: cholesky >>> out-of-place factorization >>> tolerance for zero pivot 2.22045e-14 >>> matrix ordering: natural >>> factor fill ratio given 0., needed 0. >>> Factored matrix follows: >>> Mat Object: 1 MPI processes >>> type: mumps >>> rows=38154, cols=38154 >>> package used to perform factorization: mumps >>> total: nonzeros=7080060, allocated nonzeros=7080060 >>> total number of mallocs used during MatSetValues calls=0 >>> MUMPS run parameters: >>> SYM (matrix type): 2 >>> PAR (host participation): 1 >>> ICNTL(1) (output for error): 6 >>> ICNTL(2) (output of diagnostic msg): 0 >>> ICNTL(3) (output for global info): 0 >>> ICNTL(4) (level of printing): 0 >>> ICNTL(5) (input mat struct): 0 >>> ICNTL(6) (matrix prescaling): 7 >>> ICNTL(7) (sequential matrix ordering):7 >>> ICNTL(8) (scaling strategy): 77 >>> ICNTL(10) (max num of refinements): 0 >>> ICNTL(11) (error analysis): 0 >>> ICNTL(12) (efficiency control): >>> 0 >>> ICNTL(13) (efficiency control): >>> 0 >>> ICNTL(14) (percentage of estimated workspace increase): >>> 20 >>> ICNTL(18) (input mat struct): >>> 0 >>> ICNTL(19) (Schur complement info): >>> 0 >>> ICNTL(20) (rhs sparse pattern): >>> 0 >>> ICNTL(21) (solution struct): >>> 0 >>> ICNTL(22) (in-core/out-of-core facility): >>> 0 >>> ICNTL(23) (max size of memory can be allocated >>> locally):0 >>> ICNTL(24) (detection of null pivot rows): >>> 0 >>> ICNTL(25) (computation of a null space basis): >>> 0 >>> ICNTL(26) (Schur options for rhs or solution): >>> 0 >>> ICNTL(27) (experimental parameter): >>> -32 >>> ICNTL(28) (use parallel or sequential ordering): >>> 1 >>> ICNTL(29) (parallel ordering): >>> 0 >>> ICNTL(30) (user-specified set of entries in inv(A)): >>> 0 >>> ICNTL(31) (factors is discarded in the solve phase): >>> 0 >>> ICNTL(33) (compute determinant): >>> 0 >>> ICNTL(35) (activate BLR based factorization): >>> 0 >>> ICNTL(36) (choice of BLR factorization variant): >>> 0 >>> ICNTL(38) (estimated compression rate of LU factors): >>> 333 >>> CNTL(1) (relative pivoting threshold): 0.01 >>> CNTL(2) (stopping criterion of refinement): 1.49012e-08 >>> CNTL(3) (absolute pivoting threshold): 0. >>> CNTL(4) (value of static pivoting): -1. >>> CNTL(5) (fixation for null pivots): 0. >>> CNTL(7) (dropping parameter for BLR): 0. >>> RINFO(1) (local estimated flops for the elimination >>> after analysis): >>> [0] 2.73979e+09 >>> RINFO(2) (local estimated flops for the assembly after >>> factorization): >>> [0] 1.08826e+07 >>> RINFO(3) (local estimated flops for the elimination >>> after factorization): >>> [0] 2.73979e+09 >>> INFO(15) (estimated size of (in MB) MUMPS internal data >>> for running numerical factorization): >>> [0] 94 >>> INFO(16) (size of (in MB) MUMPS internal data used >>> during numerical factorization): >>> [0] 94 >>> INFO(23) (num of pivots eliminated on this processor >>> after factorization): >>> [0] 38154 >>> RINFOG(1) (global estimated flops for the elimination >>> after analysis): 2.73979e+09 >>> RINFOG(2) (global estimated flops for the assembly after >>> factorization): 1.08826e+07 >>> RINFOG(3) (global estimated flops for the elimination >>> after factorization): 2.73979e+09 >>> (RINFOG(12) RINFOG(13))*2^INFOG(34) (determinant): >>> (0.,0.)*(2^0) >>> INFOG(3) (estimated real workspace for factors on all >>> processors after analysis): 8377336 >>> INFOG(4) (estimated integer workspace for factors on all >>> processors after analysis): 447902 >>> INFOG(5) (estimated maximum front size in the complete >>> tree): 990 >>> INFOG(6) (number of nodes in the complete tree): 2730 >>> INFOG(7) (ordering option effectively use after >>> analysis): 5 >>> INFOG(8) (structural symmetry in percent of the permuted >>> matrix after analysis): 100 >>> INFOG(9) (total real/complex workspace to store the >>> matrix factors after factorization): 8377336 >>> INFOG(10) (total integer space store the matrix factors >>> after factorization): 447902 >>> INFOG(11) (order of largest frontal matrix after >>> factorization): 990 >>> INFOG(12) (number of off-diagonal pivots): 10 >>> INFOG(13) (number of delayed pivots after >>> factorization): 0 >>> INFOG(14) (number of memory compress after >>> factorization): 0 >>> INFOG(15) (number of steps of iterative refinement after >>> solution): 0 >>> INFOG(16) (estimated size (in MB) of all MUMPS internal >>> data for factorization after analysis: value on the most memory consuming >>> processor): 94 >>> INFOG(17) (estimated size of all MUMPS internal data for >>> factorization after analysis: sum over all processors): 94 >>> INFOG(18) (size of all MUMPS internal data allocated >>> during factorization: value on the most memory consuming processor): 94 >>> INFOG(19) (size of all MUMPS internal data allocated >>> during factorization: sum over all processors): 94 >>> INFOG(20) (estimated number of entries in the factors): >>> 7080060 >>> INFOG(21) (size in MB of memory effectively used during >>> factorization - value on the most memory consuming processor): 80 >>> INFOG(22) (size in MB of memory effectively used during >>> factorization - sum over all processors): 80 >>> INFOG(23) (after analysis: value of ICNTL(6) effectively >>> used): 0 >>> INFOG(24) (after analysis: value of ICNTL(12) >>> effectively used): 1 >>> INFOG(25) (after factorization: number of pivots >>> modified by static pivoting): 0 >>> INFOG(28) (after factorization: number of null pivots >>> encountered): 0 >>> INFOG(29) (after factorization: effective number of >>> entries in the factors (sum over all processors)): 7080060 >>> INFOG(30, 31) (after solution: size in Mbytes of memory >>> used during solution phase): 92, 92 >>> INFOG(32) (after analysis: type of analysis done): 1 >>> INFOG(33) (value used for ICNTL(8)): 7 >>> INFOG(34) (exponent of the determinant if determinant is >>> requested): 0 >>> INFOG(35) (after factorization: number of entries taking >>> into account BLR factor compression - sum over all processors): 7080060 >>> INFOG(36) (after analysis: estimated size of all MUMPS >>> internal data for running BLR in-core - value on the most memory consuming >>> processor): 0 >>> INFOG(37) (after analysis: estimated size of all MUMPS >>> internal data for running BLR in-core - sum over all processors): 0 >>> INFOG(38) (after analysis: estimated size of all MUMPS >>> internal data for running BLR out-of-core - value on the most memory >>> consuming processor): 0 >>> INFOG(39) (after analysis: estimated size of all MUMPS >>> internal data for running BLR out-of-core - sum over all processors): 0 >>> linear system matrix = precond matrix: >>> Mat Object: 1 MPI processes >>> type: seqaij >>> rows=38154, cols=38154 >>> total: nonzeros=973446, allocated nonzeros=973446 >>> total number of mallocs used during MatSetValues calls=0 >>> not using I-node routines >>> [0]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> [0]PETSC ERROR: >>> [0]PETSC ERROR: TSStep has failed due to DIVERGED_NONLINEAR_SOLVE, >>> increase -ts_max_snes_failures or make negative to attempt recovery >>> [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html >>> for trouble shooting. >>> [0]PETSC ERROR: Petsc Release Version 3.12.3, Jan, 03, 2020 >>> >>> Thanks, >>> Ashish >>> Scientific Computing Division >>> OnScale >>> CA, USA >>> >>> >>> >> > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bantingl at myumanitoba.ca Tue Jun 23 07:55:52 2020 From: bantingl at myumanitoba.ca (Lucas Banting) Date: Tue, 23 Jun 2020 12:55:52 +0000 Subject: [petsc-users] Question about HDF5 Viewer and Datasets Message-ID: Hello, I am trying to use the HDF5 viewer to store the solution of my system at every iteration. I am storing the solutions in order to compare two different results, one from an old sequential program and one from a new parallel program that is parallelized using the DMDA structure. I am using matlab to compare the results. In the sequential program, the vector for the solution is just a simple one dimensional vector made from VecCreate. In the parallel program, the vector is made from DMCreateGlobalVector. In both versions of the program, I create the viewer as: call PetscViewerHDF5Open(PETSC_COMM_WORLD,"filename.hdf5",& FILE_MODE_WRITE,hdf5viewer,ierr) When I load the files into MATLAB after words, the sequential program creates a new dataset each iteration, while the parallel version using DMCreateGlobalVector only stores a single dataset. Is there a way I can force both viewers to create a new dataset in the same way each time vecview is called? Or is there someway I can manually create a new dataset? I am using Open MPI 3.1.4, Petsc 3.12.4, Intel Fortran 2019.5, and hdf5/1.10.5. Thanks, Lucas Banting -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Tue Jun 23 08:10:52 2020 From: bsmith at petsc.dev (Barry Smith) Date: Tue, 23 Jun 2020 08:10:52 -0500 Subject: [petsc-users] Question about HDF5 Viewer and Datasets In-Reply-To: References: Message-ID: <68385B3D-4EEA-4CFD-A980-56D67D1BB17C@petsc.dev> I looked at the code VecView_MPI_HDF5 and VecView_MPI_HDF5_DA they both seem to use the timestep variable to allow putting multiple datasets into the fire. Perhaps you forgot call PetscViewerHDF5SetTimestep for the DMDA case? Otherwise it looks like there must be a bug in VecView_MPI_HDF5_DA where it does not properly save multiple fields. You would need to step through that routine in the debugger. Barry > On Jun 23, 2020, at 7:55 AM, Lucas Banting wrote: > > Hello, > > I am trying to use the HDF5 viewer to store the solution of my system at every iteration. > I am storing the solutions in order to compare two different results, one from an old sequential program and one from a new parallel program that is parallelized using the DMDA structure. > I am using matlab to compare the results. > > In the sequential program, the vector for the solution is just a simple one dimensional vector made from VecCreate. In the parallel program, the vector is made from DMCreateGlobalVector. > > In both versions of the program, I create the viewer as: > > call PetscViewerHDF5Open(PETSC_COMM_WORLD,"filename.hdf5",& > FILE_MODE_WRITE,hdf5viewer,ierr) > > When I load the files into MATLAB after words, the sequential program creates a new dataset each iteration, while the parallel version using DMCreateGlobalVector only stores a single dataset. > Is there a way I can force both viewers to create a new dataset in the same way each time vecview is called? Or is there someway I can manually create a new dataset? > > I am using Open MPI 3.1.4, Petsc 3.12.4, Intel Fortran 2019.5, and hdf5/1.10.5. > > Thanks, > > Lucas Banting -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Jun 23 08:15:24 2020 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 23 Jun 2020 09:15:24 -0400 Subject: [petsc-users] Question about HDF5 Viewer and Datasets In-Reply-To: References: Message-ID: On Tue, Jun 23, 2020 at 8:56 AM Lucas Banting wrote: > Hello, > > I am trying to use the HDF5 viewer to store the solution of my system at > every iteration. > I am storing the solutions in order to compare two different results, one > from an old sequential program and one from a new parallel program that is > parallelized using the DMDA structure. > I am using matlab to compare the results. > > In the sequential program, the vector for the solution is just a simple > one dimensional vector made from VecCreate. In the parallel program, the > vector is made from DMCreateGlobalVector. > > In both versions of the program, I create the viewer as: > > call PetscViewerHDF5Open(PETSC_COMM_WORLD,"filename.hdf5",& > FILE_MODE_WRITE,hdf5viewer,ierr) > > When I load the files into MATLAB after words, the sequential program > creates a new dataset each iteration, while the parallel version using > DMCreateGlobalVector only stores a single dataset. > Is there a way I can force both viewers to create a new dataset in the > same way each time vecview is called? Or is there someway I can manually > create a new dataset? > You can see inside the file using 'h5ls' and 'h5dump'. I believe that the Vec is stored in HDF5 using the name (of the PetscObject). I think that is happening is that you are not setting the name, which means it defaults to the pointer, and that pointer is the same for your global vector. I believe if you change the name at each iteration, they will all be saved. You can also use the Timestep mechanism as Barry suggests. Thanks, Matt > I am using Open MPI 3.1.4, Petsc 3.12.4, Intel Fortran 2019.5, and hdf5/ > 1.10.5. > > Thanks, > > Lucas Banting > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Tue Jun 23 08:49:07 2020 From: mfadams at lbl.gov (Mark Adams) Date: Tue, 23 Jun 2020 09:49:07 -0400 Subject: [petsc-users] CUDA error Message-ID: My code runs OK on SUMMIT but ex56 does not. ex56 runs in serial but I get this segv in parallel. I also see these memcheck messages from PetscSFBcastAndOpBegin in my code and ex56. I ran this in DDT and was able to get a stack trace and look at variables. THe segv is on sfbasic.c:148: ierr = MPI_Startall_isend(bas->rootbuflen[PETSCSF_REMOTE],unit,bas->nrootreqs,rootreqs);CHKERRQ(ierr); I did not see anything wrong with the variables here. The segv is on processor 1 of 2 (so the last process). Any ideas? Thanks, Mark #18 main (argc=, args=) at /ccs/home/adams/petsc-old/src/snes/tutorials/ex56.c:477 (at 0x0000000010006224) #17 SNESSolve (snes=0x31064ac0, b=0x3241ac40, x=) at /autofs/nccs-svm1_home1/adams/petsc-old/src/snes/interface/snes.c:4515 (at 0x0000200000de3498) #16 SNESSolve_NEWTONLS (snes=0x31064ac0) at /autofs/nccs-svm1_home1/adams/petsc-old/src/snes/impls/ls/ls.c:175 (at 0x0000200000e1b344) #15 SNESComputeFunction (snes=0x31064ac0, x=0x312a3170, y=0x4a464bd0) at /autofs/nccs-svm1_home1/adams/petsc-old/src/snes/interface/snes.c:2378 (at 0x0000200000dd5024) #14 SNESComputeFunction_DMLocal (snes=0x31064ac0, X=0x312a3170, F=0x4a464bd0, ctx=0x4a3a7ed0) at /autofs/nccs-svm1_home1/adams/petsc-old/src/snes/utils/dmlocalsnes.c:71 (at 0x0000200000dab058) #13 DMGlobalToLocalBegin (dm=0x30fac020, g=0x312a3170, mode=, l=0x4a444c40) at /autofs/nccs-svm1_home1/adams/petsc-old/src/dm/interface/dm.c:2407 (at 0x0000200000b28a38) #12 PetscSFBcastBegin (leafdata=0x200073c00a00, rootdata=0x200073a00000, unit=, sf=0x30f28980) at /ccs/home/adams/petsc-old/include/petscsf.h:189 (at 0x0000200000b28a38) #11 PetscSFBcastAndOpBegin (sf=0x30f28980, unit=0x200021879ed0, rootdata=0x200073a00000, leafdata=0x200073c00a00, op=0x200021889c70) at /autofs/nccs-svm1_home1/adams/petsc-old/src/vec/is/sf/interface/sf.c:1337 (at 0x000020000045a230) #10 PetscSFBcastAndOpBegin_Basic (sf=0x30f28980, unit=0x200021879ed0, rootmtype=, rootdata=0x200073a00000, leafmtype=, leafdata=0x200073c00a00, op=0x200021889c70) at /autofs/nccs-svm1_home1/adams/petsc-old/src/vec/is/sf/impls/basic/sfbasic.c:148 (at 0x00002000003b1d9c) #9 PMPI_Startall () from /autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-6.4.0/spectrum-mpi-10.3.1.2-20200121-awz2q5brde7wgdqqw4ugalrkukeub4eb/container/../lib/libmpi_ibm.so.3 (at 0x00002000217e3d98) #8 mca_pml_pami_start () from /autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-6.4.0/spectrum-mpi-10.3.1.2-20200121-awz2q5brde7wgdqqw4ugalrkukeub4eb/container/../lib/spectrum_mpi/mca_pml_pami.so (at 0x000020002555e6e0) #7 pml_pami_persis_send_start () from /autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-6.4.0/spectrum-mpi-10.3.1.2-20200121-awz2q5brde7wgdqqw4ugalrkukeub4eb/container/../lib/spectrum_mpi/mca_pml_pami.so (at 0x000020002555e29c) #6 pml_pami_send () from /autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-6.4.0/spectrum-mpi-10.3.1.2-20200121-awz2q5brde7wgdqqw4ugalrkukeub4eb/container/../lib/spectrum_mpi/mca_pml_pami.so (at 0x000020002555f69c) #5 PAMI_Send_immediate () from /autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-6.4.0/spectrum-mpi-10.3.1.2-20200121-awz2q5brde7wgdqqw4ugalrkukeub4eb/container/../lib/pami_port/libpami.so.3 (at 0x0000200025725814) #4 PAMI::Protocol::Send::Eager, PAMI::Counter::IndirectBounded, 256u>, PAMI::Counter::Indirect, PAMI::Device::Shmem::CMAShaddr, 256u, 512u> >, PAMI::Device::IBV::PacketModel >::EagerImpl<(PAMI::Protocol::Send::configuration_t)5, true>::immediate(pami_send_immediate_t*) () from /autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-6.4.0/spectrum-mpi-10.3.1.2-20200121-awz2q5brde7wgdqqw4ugalrkukeub4eb/container/../lib/pami_port/libpami.so.3 (at 0x00002000257e7bac) #3 PAMI::Protocol::Send::EagerSimple, PAMI::Counter::IndirectBounded, 256u>, PAMI::Counter::Indirect, PAMI::Device::Shmem::CMAShaddr, 256u, 512u> >, (PAMI::Protocol::Send::configuration_t)5>::immediate_impl(pami_send_immediate_t*) () from /autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-6.4.0/spectrum-mpi-10.3.1.2-20200121-awz2q5brde7wgdqqw4ugalrkukeub4eb/container/../lib/pami_port/libpami.so.3 (at 0x00002000257e7824) #2 bool PAMI::Device::Interface::PacketModel, PAMI::Counter::IndirectBounded, 256u>, PAMI::Counter::Indirect, PAMI::Device::Shmem::CMAShaddr, 256u, 512u> > >::postPacket<2u>(unsigned long, unsigned long, void*, unsigned long, iovec (&) [2u]) () from /autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-6.4.0/spectrum-mpi-10.3.1.2-20200121-awz2q5brde7wgdqqw4ugalrkukeub4eb/container/../lib/pami_port/libpami.so.3 (at 0x00002000257e6c18) #1 PAMI::Device::Shmem::Packet >::writePayload(PAMI::Fifo::FifoPacket<64u, 4096u>&, iovec*, unsigned long) () from /autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-6.4.0/spectrum-mpi-10.3.1.2-20200121-awz2q5brde7wgdqqw4ugalrkukeub4eb/container/../lib/pami_port/libpami.so.3 (at 0x00002000257c5a7c) #0 __memcpy_power7 () from /lib64/libc.so.6 (at 0x00002000219eb84c) ========= Program hit CUDA_ERROR_INVALID_VALUE (error 1) due to "invalid argument" on CUDA API call to cuPointerGetAttribute. ========= Saved host backtrace up to driver entry point at error ========= Host Frame:/lib64/libcuda.so.1 (cuPointerGetAttribute + 0x178) [0x2d14a8] ========= Host Frame:/ccs/home/adams/petsc-old/arch-summit-opt-gnu-cuda-omp/lib/libpetsc.so.3.013 (PetscSFBcastAndOpBegin + 0xd8) [0x3ba1a0] ========= Host Frame:/ccs/home/adams/petsc-old/arch-summit-opt-gnu-cuda-omp/lib/libpetsc.so.3.013 (PetscSectionCreateGlobalSection + 0x948) [0x3c5b24] ========= Host Frame:/ccs/home/adams/petsc-old/arch-summit-opt-gnu-cuda-omp/lib/libpetsc.so.3.013 (DMGetGlobalSection + 0x98) [0xa85e5c] ========= Host Frame:/ccs/home/adams/petsc-old/arch-summit-opt-gnu-cuda-omp/lib/libpetsc.so.3.013 (DMPlexCreateRigidBody + 0xc4) [0x9a9ce4] ========= Host Frame:./ex56 [0x5aec] ========= Host Frame:/lib64/libc.so.6 [0x25200] ========= Host Frame:/lib64/libc.so.6 (__libc_start_main + 0xc4) [0x253f4] ========= ========= Program hit CUDA_ERROR_INVALID_VALUE (error 1) due to "invalid argument" on CUDA API call to cuPointerGetAttribute. ========= Saved host backtrace up to driver entry point at error ========= Host Frame:/lib64/libcuda.so.1 (cuPointerGetAttribute + 0x178) [0x2d14a8] ========= Host Frame:/ccs/home/adams/petsc-old/arch-summit-opt-gnu-cuda-omp/lib/libpetsc[1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [1]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [1]PETSC ERROR: or see https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [1]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [1]PETSC ERROR: to get more information on the crash. [1]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [1]PETSC ERROR: Signal received [1]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [1]PETSC ERROR: Petsc Development GIT revision: v3.13.2-421-gab8fa13 GIT Date: 2020-06-22 13:25:32 -0400 [1]PETSC ERROR: ./ex56 on a arch-summit-opt-gnu-cuda-omp named h35n05 by adams Tue Jun 23 09:06:20 2020 [1]PETSC ERROR: Configure options --with-fc=0 --COPTFLAGS="-g -O -fPIC -DFP_DIM=2" --CXXOPTFLAGS="-g -O -fPIC " --FOPTFLAGS="-g -O -fPIC " --CUDAOPTFLAGS="-g -O -Xcompiler -rdynamic -lineinfo--with-ssl=0" --with-batch=0 --with-cxx=mpicxx --with-mpiexec="jsrun -g1" --with-cuda=1 --with-cudac=nvcc --download-p4est=1 --download-zlib --download-hdf5=1 --download-metis --download-parmetis --download-triangle --with-blaslapack-lib="-L/autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-6.4.0/netlib-lapack-3.8.0-wcabdyqhdi5rooxbkqa6x5d7hxyxwdkm/lib64 -lblas -llapack" --with-cc=mpicc --with-shared-libraries=1 --with-x=0 --with-64-bit-indices=0 --with-debugging=0 PETSC_ARCH=arch-summit-opt-gnu-cuda-omp --with-openmp=1 --with-threadsaftey=1 --with-log=1 PETSC_DIR=/ccs/home/adams/petsc-old --force [1]PETSC ERROR: #1 User provided function() line 0 in unknown file -------------------------------------------------------------------------- -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Tue Jun 23 08:54:11 2020 From: jed at jedbrown.org (Jed Brown) Date: Tue, 23 Jun 2020 07:54:11 -0600 Subject: [petsc-users] CUDA error In-Reply-To: References: Message-ID: <871rm5zy4c.fsf@jedbrown.org> Did you use --smpiargs=-gpu, and have you tried if the error is still there with -use_gpu_aware_mpi 0? I assume you're using -dm_vec_type cuda? Mark Adams writes: > My code runs OK on SUMMIT but ex56 does not. ex56 runs in serial but I get > this segv in parallel. I also see these memcheck messages > from PetscSFBcastAndOpBegin in my code and ex56. > > I ran this in DDT and was able to get a stack trace and look at variables. > THe segv is on sfbasic.c:148: > > ierr = > MPI_Startall_isend(bas->rootbuflen[PETSCSF_REMOTE],unit,bas->nrootreqs,rootreqs);CHKERRQ(ierr); > > I did not see anything wrong with the variables here. The segv is on > processor 1 of 2 (so the last process). > > Any ideas? > Thanks, > Mark > > #18 main (argc=, args=) at > /ccs/home/adams/petsc-old/src/snes/tutorials/ex56.c:477 (at > 0x0000000010006224) > #17 SNESSolve (snes=0x31064ac0, b=0x3241ac40, x=) at > /autofs/nccs-svm1_home1/adams/petsc-old/src/snes/interface/snes.c:4515 (at > 0x0000200000de3498) > #16 SNESSolve_NEWTONLS (snes=0x31064ac0) at > /autofs/nccs-svm1_home1/adams/petsc-old/src/snes/impls/ls/ls.c:175 (at > 0x0000200000e1b344) > #15 SNESComputeFunction (snes=0x31064ac0, x=0x312a3170, y=0x4a464bd0) at > /autofs/nccs-svm1_home1/adams/petsc-old/src/snes/interface/snes.c:2378 (at > 0x0000200000dd5024) > #14 SNESComputeFunction_DMLocal (snes=0x31064ac0, X=0x312a3170, > F=0x4a464bd0, ctx=0x4a3a7ed0) at > /autofs/nccs-svm1_home1/adams/petsc-old/src/snes/utils/dmlocalsnes.c:71 (at > 0x0000200000dab058) > #13 DMGlobalToLocalBegin (dm=0x30fac020, g=0x312a3170, mode= out>, l=0x4a444c40) at > /autofs/nccs-svm1_home1/adams/petsc-old/src/dm/interface/dm.c:2407 (at > 0x0000200000b28a38) > #12 PetscSFBcastBegin (leafdata=0x200073c00a00, rootdata=0x200073a00000, > unit=, sf=0x30f28980) at > /ccs/home/adams/petsc-old/include/petscsf.h:189 (at 0x0000200000b28a38) > #11 PetscSFBcastAndOpBegin (sf=0x30f28980, unit=0x200021879ed0, > rootdata=0x200073a00000, leafdata=0x200073c00a00, op=0x200021889c70) at > /autofs/nccs-svm1_home1/adams/petsc-old/src/vec/is/sf/interface/sf.c:1337 > (at 0x000020000045a230) > #10 PetscSFBcastAndOpBegin_Basic (sf=0x30f28980, unit=0x200021879ed0, > rootmtype=, rootdata=0x200073a00000, leafmtype= out>, leafdata=0x200073c00a00, op=0x200021889c70) at > /autofs/nccs-svm1_home1/adams/petsc-old/src/vec/is/sf/impls/basic/sfbasic.c:148 > (at 0x00002000003b1d9c) > #9 PMPI_Startall () from > /autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-6.4.0/spectrum-mpi-10.3.1.2-20200121-awz2q5brde7wgdqqw4ugalrkukeub4eb/container/../lib/libmpi_ibm.so.3 > (at 0x00002000217e3d98) > #8 mca_pml_pami_start () from > /autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-6.4.0/spectrum-mpi-10.3.1.2-20200121-awz2q5brde7wgdqqw4ugalrkukeub4eb/container/../lib/spectrum_mpi/mca_pml_pami.so > (at 0x000020002555e6e0) > #7 pml_pami_persis_send_start () from > /autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-6.4.0/spectrum-mpi-10.3.1.2-20200121-awz2q5brde7wgdqqw4ugalrkukeub4eb/container/../lib/spectrum_mpi/mca_pml_pami.so > (at 0x000020002555e29c) > #6 pml_pami_send () from > /autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-6.4.0/spectrum-mpi-10.3.1.2-20200121-awz2q5brde7wgdqqw4ugalrkukeub4eb/container/../lib/spectrum_mpi/mca_pml_pami.so > (at 0x000020002555f69c) > #5 PAMI_Send_immediate () from > /autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-6.4.0/spectrum-mpi-10.3.1.2-20200121-awz2q5brde7wgdqqw4ugalrkukeub4eb/container/../lib/pami_port/libpami.so.3 > (at 0x0000200025725814) > #4 > PAMI::Protocol::Send::Eager 4096u>, PAMI::Counter::IndirectBounded, 256u>, > PAMI::Counter::Indirect, > PAMI::Device::Shmem::CMAShaddr, 256u, 512u> >, > PAMI::Device::IBV::PacketModel >>::EagerImpl<(PAMI::Protocol::Send::configuration_t)5, > true>::immediate(pami_send_immediate_t*) () from > /autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-6.4.0/spectrum-mpi-10.3.1.2-20200121-awz2q5brde7wgdqqw4ugalrkukeub4eb/container/../lib/pami_port/libpami.so.3 > (at 0x00002000257e7bac) > #3 > PAMI::Protocol::Send::EagerSimple 4096u>, PAMI::Counter::IndirectBounded, 256u>, > PAMI::Counter::Indirect, > PAMI::Device::Shmem::CMAShaddr, 256u, 512u> >, > (PAMI::Protocol::Send::configuration_t)5>::immediate_impl(pami_send_immediate_t*) > () from > /autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-6.4.0/spectrum-mpi-10.3.1.2-20200121-awz2q5brde7wgdqqw4ugalrkukeub4eb/container/../lib/pami_port/libpami.so.3 > (at 0x00002000257e7824) > #2 bool > PAMI::Device::Interface::PacketModel 4096u>, PAMI::Counter::IndirectBounded, 256u>, > PAMI::Counter::Indirect, > PAMI::Device::Shmem::CMAShaddr, 256u, 512u> > >::postPacket<2u>(unsigned > long, unsigned long, void*, unsigned long, iovec (&) [2u]) () from > /autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-6.4.0/spectrum-mpi-10.3.1.2-20200121-awz2q5brde7wgdqqw4ugalrkukeub4eb/container/../lib/pami_port/libpami.so.3 > (at 0x00002000257e6c18) > #1 PAMI::Device::Shmem::Packet >>::writePayload(PAMI::Fifo::FifoPacket<64u, 4096u>&, iovec*, unsigned long) > () from > /autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-6.4.0/spectrum-mpi-10.3.1.2-20200121-awz2q5brde7wgdqqw4ugalrkukeub4eb/container/../lib/pami_port/libpami.so.3 > (at 0x00002000257c5a7c) > #0 __memcpy_power7 () from /lib64/libc.so.6 (at 0x00002000219eb84c) > > > ========= Program hit CUDA_ERROR_INVALID_VALUE (error 1) due to "invalid > argument" on CUDA API call to cuPointerGetAttribute. > ========= Saved host backtrace up to driver entry point at error > ========= Host Frame:/lib64/libcuda.so.1 (cuPointerGetAttribute + > 0x178) [0x2d14a8] > ========= Host > Frame:/ccs/home/adams/petsc-old/arch-summit-opt-gnu-cuda-omp/lib/libpetsc.so.3.013 > (PetscSFBcastAndOpBegin + 0xd8) [0x3ba1a0] > ========= Host > Frame:/ccs/home/adams/petsc-old/arch-summit-opt-gnu-cuda-omp/lib/libpetsc.so.3.013 > (PetscSectionCreateGlobalSection + 0x948) [0x3c5b24] > ========= Host > Frame:/ccs/home/adams/petsc-old/arch-summit-opt-gnu-cuda-omp/lib/libpetsc.so.3.013 > (DMGetGlobalSection + 0x98) [0xa85e5c] > ========= Host > Frame:/ccs/home/adams/petsc-old/arch-summit-opt-gnu-cuda-omp/lib/libpetsc.so.3.013 > (DMPlexCreateRigidBody + 0xc4) [0x9a9ce4] > ========= Host Frame:./ex56 [0x5aec] > ========= Host Frame:/lib64/libc.so.6 [0x25200] > ========= Host Frame:/lib64/libc.so.6 (__libc_start_main + 0xc4) > [0x253f4] > ========= > ========= Program hit CUDA_ERROR_INVALID_VALUE (error 1) due to "invalid > argument" on CUDA API call to cuPointerGetAttribute. > ========= Saved host backtrace up to driver entry point at error > ========= Host Frame:/lib64/libcuda.so.1 (cuPointerGetAttribute + > 0x178) [0x2d14a8] > ========= Host > Frame:/ccs/home/adams/petsc-old/arch-summit-opt-gnu-cuda-omp/lib/libpetsc[1]PETSC > ERROR: > ------------------------------------------------------------------------ > [1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, > probably memory access out of range > [1]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [1]PETSC ERROR: or see > https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > [1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X > to find memory corruption errors > [1]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and > run > [1]PETSC ERROR: to get more information on the crash. > [1]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [1]PETSC ERROR: Signal received > [1]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [1]PETSC ERROR: Petsc Development GIT revision: v3.13.2-421-gab8fa13 GIT > Date: 2020-06-22 13:25:32 -0400 > [1]PETSC ERROR: ./ex56 on a arch-summit-opt-gnu-cuda-omp named h35n05 by > adams Tue Jun 23 09:06:20 2020 > [1]PETSC ERROR: Configure options --with-fc=0 --COPTFLAGS="-g -O -fPIC > -DFP_DIM=2" --CXXOPTFLAGS="-g -O -fPIC " --FOPTFLAGS="-g -O -fPIC " > --CUDAOPTFLAGS="-g -O -Xcompiler -rdynamic -lineinfo--with-ssl=0" > --with-batch=0 --with-cxx=mpicxx --with-mpiexec="jsrun -g1" --with-cuda=1 > --with-cudac=nvcc --download-p4est=1 --download-zlib --download-hdf5=1 > --download-metis --download-parmetis --download-triangle > --with-blaslapack-lib="-L/autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-6.4.0/netlib-lapack-3.8.0-wcabdyqhdi5rooxbkqa6x5d7hxyxwdkm/lib64 > -lblas -llapack" --with-cc=mpicc --with-shared-libraries=1 --with-x=0 > --with-64-bit-indices=0 --with-debugging=0 > PETSC_ARCH=arch-summit-opt-gnu-cuda-omp --with-openmp=1 > --with-threadsaftey=1 --with-log=1 PETSC_DIR=/ccs/home/adams/petsc-old > --force > [1]PETSC ERROR: #1 User provided function() line 0 in unknown file > -------------------------------------------------------------------------- From bsmith at petsc.dev Tue Jun 23 09:33:02 2020 From: bsmith at petsc.dev (Barry Smith) Date: Tue, 23 Jun 2020 09:33:02 -0500 Subject: [petsc-users] Open discussion on PETSc's future design and implementation Message-ID: <6BB8BBA2-3964-494F-9412-B0AD34298BB3@petsc.dev> We've open a broad general discussion on PETSc's future design and implementation at https://gitlab.com/petsc/petsc/-/issues/643 . There is a possibility of a strip to the metal and rebuild the design approach. All ideas no matter how large or small, or crazy sounding are welcome. Anything that has bugged you over the years with PETSc is perfect to add to the discussion. In a later stage we will be synthesizing the input and planning for future design changes to PETSc. Thanks for all your support and ideas, Barry -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Tue Jun 23 09:51:31 2020 From: mfadams at lbl.gov (Mark Adams) Date: Tue, 23 Jun 2020 10:51:31 -0400 Subject: [petsc-users] CUDA error In-Reply-To: <871rm5zy4c.fsf@jedbrown.org> References: <871rm5zy4c.fsf@jedbrown.org> Message-ID: On Tue, Jun 23, 2020 at 9:54 AM Jed Brown wrote: > Did you use --smpiargs=-gpu, and have you tried if the error is still > > there with -use_gpu_aware_mpi 0? That did it. Thanks, > I assume you're using -dm_vec_type cuda? > > Mark Adams writes: > > > My code runs OK on SUMMIT but ex56 does not. ex56 runs in serial but I > get > > this segv in parallel. I also see these memcheck messages > > from PetscSFBcastAndOpBegin in my code and ex56. > > > > I ran this in DDT and was able to get a stack trace and look at > variables. > > THe segv is on sfbasic.c:148: > > > > ierr = > > > MPI_Startall_isend(bas->rootbuflen[PETSCSF_REMOTE],unit,bas->nrootreqs,rootreqs);CHKERRQ(ierr); > > > > I did not see anything wrong with the variables here. The segv is on > > processor 1 of 2 (so the last process). > > > > Any ideas? > > Thanks, > > Mark > > > > #18 main (argc=, args=) at > > /ccs/home/adams/petsc-old/src/snes/tutorials/ex56.c:477 (at > > 0x0000000010006224) > > #17 SNESSolve (snes=0x31064ac0, b=0x3241ac40, x=) at > > /autofs/nccs-svm1_home1/adams/petsc-old/src/snes/interface/snes.c:4515 > (at > > 0x0000200000de3498) > > #16 SNESSolve_NEWTONLS (snes=0x31064ac0) at > > /autofs/nccs-svm1_home1/adams/petsc-old/src/snes/impls/ls/ls.c:175 (at > > 0x0000200000e1b344) > > #15 SNESComputeFunction (snes=0x31064ac0, x=0x312a3170, y=0x4a464bd0) at > > /autofs/nccs-svm1_home1/adams/petsc-old/src/snes/interface/snes.c:2378 > (at > > 0x0000200000dd5024) > > #14 SNESComputeFunction_DMLocal (snes=0x31064ac0, X=0x312a3170, > > F=0x4a464bd0, ctx=0x4a3a7ed0) at > > /autofs/nccs-svm1_home1/adams/petsc-old/src/snes/utils/dmlocalsnes.c:71 > (at > > 0x0000200000dab058) > > #13 DMGlobalToLocalBegin (dm=0x30fac020, g=0x312a3170, mode= > out>, l=0x4a444c40) at > > /autofs/nccs-svm1_home1/adams/petsc-old/src/dm/interface/dm.c:2407 (at > > 0x0000200000b28a38) > > #12 PetscSFBcastBegin (leafdata=0x200073c00a00, rootdata=0x200073a00000, > > unit=, sf=0x30f28980) at > > /ccs/home/adams/petsc-old/include/petscsf.h:189 (at 0x0000200000b28a38) > > #11 PetscSFBcastAndOpBegin (sf=0x30f28980, unit=0x200021879ed0, > > rootdata=0x200073a00000, leafdata=0x200073c00a00, op=0x200021889c70) at > > /autofs/nccs-svm1_home1/adams/petsc-old/src/vec/is/sf/interface/sf.c:1337 > > (at 0x000020000045a230) > > #10 PetscSFBcastAndOpBegin_Basic (sf=0x30f28980, unit=0x200021879ed0, > > rootmtype=, rootdata=0x200073a00000, leafmtype= > out>, leafdata=0x200073c00a00, op=0x200021889c70) at > > > /autofs/nccs-svm1_home1/adams/petsc-old/src/vec/is/sf/impls/basic/sfbasic.c:148 > > (at 0x00002000003b1d9c) > > #9 PMPI_Startall () from > > > /autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-6.4.0/spectrum-mpi-10.3.1.2-20200121-awz2q5brde7wgdqqw4ugalrkukeub4eb/container/../lib/libmpi_ibm.so.3 > > (at 0x00002000217e3d98) > > #8 mca_pml_pami_start () from > > > /autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-6.4.0/spectrum-mpi-10.3.1.2-20200121-awz2q5brde7wgdqqw4ugalrkukeub4eb/container/../lib/spectrum_mpi/mca_pml_pami.so > > (at 0x000020002555e6e0) > > #7 pml_pami_persis_send_start () from > > > /autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-6.4.0/spectrum-mpi-10.3.1.2-20200121-awz2q5brde7wgdqqw4ugalrkukeub4eb/container/../lib/spectrum_mpi/mca_pml_pami.so > > (at 0x000020002555e29c) > > #6 pml_pami_send () from > > > /autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-6.4.0/spectrum-mpi-10.3.1.2-20200121-awz2q5brde7wgdqqw4ugalrkukeub4eb/container/../lib/spectrum_mpi/mca_pml_pami.so > > (at 0x000020002555f69c) > > #5 PAMI_Send_immediate () from > > > /autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-6.4.0/spectrum-mpi-10.3.1.2-20200121-awz2q5brde7wgdqqw4ugalrkukeub4eb/container/../lib/pami_port/libpami.so.3 > > (at 0x0000200025725814) > > #4 > > > PAMI::Protocol::Send::Eager > 4096u>, PAMI::Counter::IndirectBounded, > 256u>, > > PAMI::Counter::Indirect, > > PAMI::Device::Shmem::CMAShaddr, 256u, 512u> >, > > PAMI::Device::IBV::PacketModel > >>::EagerImpl<(PAMI::Protocol::Send::configuration_t)5, > > true>::immediate(pami_send_immediate_t*) () from > > > /autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-6.4.0/spectrum-mpi-10.3.1.2-20200121-awz2q5brde7wgdqqw4ugalrkukeub4eb/container/../lib/pami_port/libpami.so.3 > > (at 0x00002000257e7bac) > > #3 > > > PAMI::Protocol::Send::EagerSimple > 4096u>, PAMI::Counter::IndirectBounded, > 256u>, > > PAMI::Counter::Indirect, > > PAMI::Device::Shmem::CMAShaddr, 256u, 512u> >, > > > (PAMI::Protocol::Send::configuration_t)5>::immediate_impl(pami_send_immediate_t*) > > () from > > > /autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-6.4.0/spectrum-mpi-10.3.1.2-20200121-awz2q5brde7wgdqqw4ugalrkukeub4eb/container/../lib/pami_port/libpami.so.3 > > (at 0x00002000257e7824) > > #2 bool > > > PAMI::Device::Interface::PacketModel > 4096u>, PAMI::Counter::IndirectBounded, > 256u>, > > PAMI::Counter::Indirect, > > PAMI::Device::Shmem::CMAShaddr, 256u, 512u> > >::postPacket<2u>(unsigned > > long, unsigned long, void*, unsigned long, iovec (&) [2u]) () from > > > /autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-6.4.0/spectrum-mpi-10.3.1.2-20200121-awz2q5brde7wgdqqw4ugalrkukeub4eb/container/../lib/pami_port/libpami.so.3 > > (at 0x00002000257e6c18) > > #1 PAMI::Device::Shmem::Packet > >>::writePayload(PAMI::Fifo::FifoPacket<64u, 4096u>&, iovec*, unsigned > long) > > () from > > > /autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-6.4.0/spectrum-mpi-10.3.1.2-20200121-awz2q5brde7wgdqqw4ugalrkukeub4eb/container/../lib/pami_port/libpami.so.3 > > (at 0x00002000257c5a7c) > > #0 __memcpy_power7 () from /lib64/libc.so.6 (at 0x00002000219eb84c) > > > > > > ========= Program hit CUDA_ERROR_INVALID_VALUE (error 1) due to "invalid > > argument" on CUDA API call to cuPointerGetAttribute. > > ========= Saved host backtrace up to driver entry point at error > > ========= Host Frame:/lib64/libcuda.so.1 (cuPointerGetAttribute + > > 0x178) [0x2d14a8] > > ========= Host > > > Frame:/ccs/home/adams/petsc-old/arch-summit-opt-gnu-cuda-omp/lib/libpetsc.so.3.013 > > (PetscSFBcastAndOpBegin + 0xd8) [0x3ba1a0] > > ========= Host > > > Frame:/ccs/home/adams/petsc-old/arch-summit-opt-gnu-cuda-omp/lib/libpetsc.so.3.013 > > (PetscSectionCreateGlobalSection + 0x948) [0x3c5b24] > > ========= Host > > > Frame:/ccs/home/adams/petsc-old/arch-summit-opt-gnu-cuda-omp/lib/libpetsc.so.3.013 > > (DMGetGlobalSection + 0x98) [0xa85e5c] > > ========= Host > > > Frame:/ccs/home/adams/petsc-old/arch-summit-opt-gnu-cuda-omp/lib/libpetsc.so.3.013 > > (DMPlexCreateRigidBody + 0xc4) [0x9a9ce4] > > ========= Host Frame:./ex56 [0x5aec] > > ========= Host Frame:/lib64/libc.so.6 [0x25200] > > ========= Host Frame:/lib64/libc.so.6 (__libc_start_main + 0xc4) > > [0x253f4] > > ========= > > ========= Program hit CUDA_ERROR_INVALID_VALUE (error 1) due to "invalid > > argument" on CUDA API call to cuPointerGetAttribute. > > ========= Saved host backtrace up to driver entry point at error > > ========= Host Frame:/lib64/libcuda.so.1 (cuPointerGetAttribute + > > 0x178) [0x2d14a8] > > ========= Host > > > Frame:/ccs/home/adams/petsc-old/arch-summit-opt-gnu-cuda-omp/lib/libpetsc[1]PETSC > > ERROR: > > ------------------------------------------------------------------------ > > [1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, > > probably memory access out of range > > [1]PETSC ERROR: Try option -start_in_debugger or > -on_error_attach_debugger > > [1]PETSC ERROR: or see > > https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > > [1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac > OS X > > to find memory corruption errors > > [1]PETSC ERROR: configure using --with-debugging=yes, recompile, link, > and > > run > > [1]PETSC ERROR: to get more information on the crash. > > [1]PETSC ERROR: --------------------- Error Message > > -------------------------------------------------------------- > > [1]PETSC ERROR: Signal received > > [1]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html > > for trouble shooting. > > [1]PETSC ERROR: Petsc Development GIT revision: v3.13.2-421-gab8fa13 GIT > > Date: 2020-06-22 13:25:32 -0400 > > [1]PETSC ERROR: ./ex56 on a arch-summit-opt-gnu-cuda-omp named h35n05 by > > adams Tue Jun 23 09:06:20 2020 > > [1]PETSC ERROR: Configure options --with-fc=0 --COPTFLAGS="-g -O -fPIC > > -DFP_DIM=2" --CXXOPTFLAGS="-g -O -fPIC " --FOPTFLAGS="-g -O -fPIC " > > --CUDAOPTFLAGS="-g -O -Xcompiler -rdynamic -lineinfo--with-ssl=0" > > --with-batch=0 --with-cxx=mpicxx --with-mpiexec="jsrun -g1" --with-cuda=1 > > --with-cudac=nvcc --download-p4est=1 --download-zlib --download-hdf5=1 > > --download-metis --download-parmetis --download-triangle > > > --with-blaslapack-lib="-L/autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-6.4.0/netlib-lapack-3.8.0-wcabdyqhdi5rooxbkqa6x5d7hxyxwdkm/lib64 > > -lblas -llapack" --with-cc=mpicc --with-shared-libraries=1 --with-x=0 > > --with-64-bit-indices=0 --with-debugging=0 > > PETSC_ARCH=arch-summit-opt-gnu-cuda-omp --with-openmp=1 > > --with-threadsaftey=1 --with-log=1 PETSC_DIR=/ccs/home/adams/petsc-old > > --force > > [1]PETSC ERROR: #1 User provided function() line 0 in unknown file > > > -------------------------------------------------------------------------- > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefano.zampini at gmail.com Tue Jun 23 10:07:07 2020 From: stefano.zampini at gmail.com (Stefano Zampini) Date: Tue, 23 Jun 2020 18:07:07 +0300 Subject: [petsc-users] CUDA error In-Reply-To: References: <871rm5zy4c.fsf@jedbrown.org> Message-ID: What did it? How are you running now to have everything working? Can you post smpiargs and petsc options? Il giorno mar 23 giu 2020 alle ore 17:51 Mark Adams ha scritto: > > > On Tue, Jun 23, 2020 at 9:54 AM Jed Brown wrote: > >> Did you use --smpiargs=-gpu, and have you tried if the error is still >> > > > >> there with -use_gpu_aware_mpi 0? > > > That did it. Thanks, > > >> I assume you're using -dm_vec_type cuda? >> >> Mark Adams writes: >> >> > My code runs OK on SUMMIT but ex56 does not. ex56 runs in serial but I >> get >> > this segv in parallel. I also see these memcheck messages >> > from PetscSFBcastAndOpBegin in my code and ex56. >> > >> > I ran this in DDT and was able to get a stack trace and look at >> variables. >> > THe segv is on sfbasic.c:148: >> > >> > ierr = >> > >> MPI_Startall_isend(bas->rootbuflen[PETSCSF_REMOTE],unit,bas->nrootreqs,rootreqs);CHKERRQ(ierr); >> > >> > I did not see anything wrong with the variables here. The segv is on >> > processor 1 of 2 (so the last process). >> > >> > Any ideas? >> > Thanks, >> > Mark >> > >> > #18 main (argc=, args=) at >> > /ccs/home/adams/petsc-old/src/snes/tutorials/ex56.c:477 (at >> > 0x0000000010006224) >> > #17 SNESSolve (snes=0x31064ac0, b=0x3241ac40, x=) at >> > /autofs/nccs-svm1_home1/adams/petsc-old/src/snes/interface/snes.c:4515 >> (at >> > 0x0000200000de3498) >> > #16 SNESSolve_NEWTONLS (snes=0x31064ac0) at >> > /autofs/nccs-svm1_home1/adams/petsc-old/src/snes/impls/ls/ls.c:175 (at >> > 0x0000200000e1b344) >> > #15 SNESComputeFunction (snes=0x31064ac0, x=0x312a3170, y=0x4a464bd0) at >> > /autofs/nccs-svm1_home1/adams/petsc-old/src/snes/interface/snes.c:2378 >> (at >> > 0x0000200000dd5024) >> > #14 SNESComputeFunction_DMLocal (snes=0x31064ac0, X=0x312a3170, >> > F=0x4a464bd0, ctx=0x4a3a7ed0) at >> > /autofs/nccs-svm1_home1/adams/petsc-old/src/snes/utils/dmlocalsnes.c:71 >> (at >> > 0x0000200000dab058) >> > #13 DMGlobalToLocalBegin (dm=0x30fac020, g=0x312a3170, mode=> > out>, l=0x4a444c40) at >> > /autofs/nccs-svm1_home1/adams/petsc-old/src/dm/interface/dm.c:2407 (at >> > 0x0000200000b28a38) >> > #12 PetscSFBcastBegin (leafdata=0x200073c00a00, rootdata=0x200073a00000, >> > unit=, sf=0x30f28980) at >> > /ccs/home/adams/petsc-old/include/petscsf.h:189 (at 0x0000200000b28a38) >> > #11 PetscSFBcastAndOpBegin (sf=0x30f28980, unit=0x200021879ed0, >> > rootdata=0x200073a00000, leafdata=0x200073c00a00, op=0x200021889c70) at >> > >> /autofs/nccs-svm1_home1/adams/petsc-old/src/vec/is/sf/interface/sf.c:1337 >> > (at 0x000020000045a230) >> > #10 PetscSFBcastAndOpBegin_Basic (sf=0x30f28980, unit=0x200021879ed0, >> > rootmtype=, rootdata=0x200073a00000, leafmtype=> > out>, leafdata=0x200073c00a00, op=0x200021889c70) at >> > >> /autofs/nccs-svm1_home1/adams/petsc-old/src/vec/is/sf/impls/basic/sfbasic.c:148 >> > (at 0x00002000003b1d9c) >> > #9 PMPI_Startall () from >> > >> /autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-6.4.0/spectrum-mpi-10.3.1.2-20200121-awz2q5brde7wgdqqw4ugalrkukeub4eb/container/../lib/libmpi_ibm.so.3 >> > (at 0x00002000217e3d98) >> > #8 mca_pml_pami_start () from >> > >> /autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-6.4.0/spectrum-mpi-10.3.1.2-20200121-awz2q5brde7wgdqqw4ugalrkukeub4eb/container/../lib/spectrum_mpi/mca_pml_pami.so >> > (at 0x000020002555e6e0) >> > #7 pml_pami_persis_send_start () from >> > >> /autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-6.4.0/spectrum-mpi-10.3.1.2-20200121-awz2q5brde7wgdqqw4ugalrkukeub4eb/container/../lib/spectrum_mpi/mca_pml_pami.so >> > (at 0x000020002555e29c) >> > #6 pml_pami_send () from >> > >> /autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-6.4.0/spectrum-mpi-10.3.1.2-20200121-awz2q5brde7wgdqqw4ugalrkukeub4eb/container/../lib/spectrum_mpi/mca_pml_pami.so >> > (at 0x000020002555f69c) >> > #5 PAMI_Send_immediate () from >> > >> /autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-6.4.0/spectrum-mpi-10.3.1.2-20200121-awz2q5brde7wgdqqw4ugalrkukeub4eb/container/../lib/pami_port/libpami.so.3 >> > (at 0x0000200025725814) >> > #4 >> > >> PAMI::Protocol::Send::Eager> > 4096u>, PAMI::Counter::IndirectBounded, >> 256u>, >> > PAMI::Counter::Indirect, >> > PAMI::Device::Shmem::CMAShaddr, 256u, 512u> >, >> > PAMI::Device::IBV::PacketModel >> >>::EagerImpl<(PAMI::Protocol::Send::configuration_t)5, >> > true>::immediate(pami_send_immediate_t*) () from >> > >> /autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-6.4.0/spectrum-mpi-10.3.1.2-20200121-awz2q5brde7wgdqqw4ugalrkukeub4eb/container/../lib/pami_port/libpami.so.3 >> > (at 0x00002000257e7bac) >> > #3 >> > >> PAMI::Protocol::Send::EagerSimple> > 4096u>, PAMI::Counter::IndirectBounded, >> 256u>, >> > PAMI::Counter::Indirect, >> > PAMI::Device::Shmem::CMAShaddr, 256u, 512u> >, >> > >> (PAMI::Protocol::Send::configuration_t)5>::immediate_impl(pami_send_immediate_t*) >> > () from >> > >> /autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-6.4.0/spectrum-mpi-10.3.1.2-20200121-awz2q5brde7wgdqqw4ugalrkukeub4eb/container/../lib/pami_port/libpami.so.3 >> > (at 0x00002000257e7824) >> > #2 bool >> > >> PAMI::Device::Interface::PacketModel> > 4096u>, PAMI::Counter::IndirectBounded, >> 256u>, >> > PAMI::Counter::Indirect, >> > PAMI::Device::Shmem::CMAShaddr, 256u, 512u> > >::postPacket<2u>(unsigned >> > long, unsigned long, void*, unsigned long, iovec (&) [2u]) () from >> > >> /autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-6.4.0/spectrum-mpi-10.3.1.2-20200121-awz2q5brde7wgdqqw4ugalrkukeub4eb/container/../lib/pami_port/libpami.so.3 >> > (at 0x00002000257e6c18) >> > #1 PAMI::Device::Shmem::Packet >> >>::writePayload(PAMI::Fifo::FifoPacket<64u, 4096u>&, iovec*, unsigned >> long) >> > () from >> > >> /autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-6.4.0/spectrum-mpi-10.3.1.2-20200121-awz2q5brde7wgdqqw4ugalrkukeub4eb/container/../lib/pami_port/libpami.so.3 >> > (at 0x00002000257c5a7c) >> > #0 __memcpy_power7 () from /lib64/libc.so.6 (at 0x00002000219eb84c) >> > >> > >> > ========= Program hit CUDA_ERROR_INVALID_VALUE (error 1) due to "invalid >> > argument" on CUDA API call to cuPointerGetAttribute. >> > ========= Saved host backtrace up to driver entry point at error >> > ========= Host Frame:/lib64/libcuda.so.1 (cuPointerGetAttribute + >> > 0x178) [0x2d14a8] >> > ========= Host >> > >> Frame:/ccs/home/adams/petsc-old/arch-summit-opt-gnu-cuda-omp/lib/libpetsc.so.3.013 >> > (PetscSFBcastAndOpBegin + 0xd8) [0x3ba1a0] >> > ========= Host >> > >> Frame:/ccs/home/adams/petsc-old/arch-summit-opt-gnu-cuda-omp/lib/libpetsc.so.3.013 >> > (PetscSectionCreateGlobalSection + 0x948) [0x3c5b24] >> > ========= Host >> > >> Frame:/ccs/home/adams/petsc-old/arch-summit-opt-gnu-cuda-omp/lib/libpetsc.so.3.013 >> > (DMGetGlobalSection + 0x98) [0xa85e5c] >> > ========= Host >> > >> Frame:/ccs/home/adams/petsc-old/arch-summit-opt-gnu-cuda-omp/lib/libpetsc.so.3.013 >> > (DMPlexCreateRigidBody + 0xc4) [0x9a9ce4] >> > ========= Host Frame:./ex56 [0x5aec] >> > ========= Host Frame:/lib64/libc.so.6 [0x25200] >> > ========= Host Frame:/lib64/libc.so.6 (__libc_start_main + 0xc4) >> > [0x253f4] >> > ========= >> > ========= Program hit CUDA_ERROR_INVALID_VALUE (error 1) due to "invalid >> > argument" on CUDA API call to cuPointerGetAttribute. >> > ========= Saved host backtrace up to driver entry point at error >> > ========= Host Frame:/lib64/libcuda.so.1 (cuPointerGetAttribute + >> > 0x178) [0x2d14a8] >> > ========= Host >> > >> Frame:/ccs/home/adams/petsc-old/arch-summit-opt-gnu-cuda-omp/lib/libpetsc[1]PETSC >> > ERROR: >> > ------------------------------------------------------------------------ >> > [1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, >> > probably memory access out of range >> > [1]PETSC ERROR: Try option -start_in_debugger or >> -on_error_attach_debugger >> > [1]PETSC ERROR: or see >> > https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind >> > [1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac >> OS X >> > to find memory corruption errors >> > [1]PETSC ERROR: configure using --with-debugging=yes, recompile, link, >> and >> > run >> > [1]PETSC ERROR: to get more information on the crash. >> > [1]PETSC ERROR: --------------------- Error Message >> > -------------------------------------------------------------- >> > [1]PETSC ERROR: Signal received >> > [1]PETSC ERROR: See >> https://www.mcs.anl.gov/petsc/documentation/faq.html >> > for trouble shooting. >> > [1]PETSC ERROR: Petsc Development GIT revision: v3.13.2-421-gab8fa13 >> GIT >> > Date: 2020-06-22 13:25:32 -0400 >> > [1]PETSC ERROR: ./ex56 on a arch-summit-opt-gnu-cuda-omp named h35n05 by >> > adams Tue Jun 23 09:06:20 2020 >> > [1]PETSC ERROR: Configure options --with-fc=0 --COPTFLAGS="-g -O -fPIC >> > -DFP_DIM=2" --CXXOPTFLAGS="-g -O -fPIC " --FOPTFLAGS="-g -O -fPIC " >> > --CUDAOPTFLAGS="-g -O -Xcompiler -rdynamic -lineinfo--with-ssl=0" >> > --with-batch=0 --with-cxx=mpicxx --with-mpiexec="jsrun -g1" >> --with-cuda=1 >> > --with-cudac=nvcc --download-p4est=1 --download-zlib --download-hdf5=1 >> > --download-metis --download-parmetis --download-triangle >> > >> --with-blaslapack-lib="-L/autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-6.4.0/netlib-lapack-3.8.0-wcabdyqhdi5rooxbkqa6x5d7hxyxwdkm/lib64 >> > -lblas -llapack" --with-cc=mpicc --with-shared-libraries=1 --with-x=0 >> > --with-64-bit-indices=0 --with-debugging=0 >> > PETSC_ARCH=arch-summit-opt-gnu-cuda-omp --with-openmp=1 >> > --with-threadsaftey=1 --with-log=1 PETSC_DIR=/ccs/home/adams/petsc-old >> > --force >> > [1]PETSC ERROR: #1 User provided function() line 0 in unknown file >> > >> -------------------------------------------------------------------------- >> > -- Stefano -------------- next part -------------- An HTML attachment was scrubbed... URL: From junchao.zhang at gmail.com Tue Jun 23 10:08:10 2020 From: junchao.zhang at gmail.com (Junchao Zhang) Date: Tue, 23 Jun 2020 10:08:10 -0500 Subject: [petsc-users] CUDA error In-Reply-To: References: <871rm5zy4c.fsf@jedbrown.org> Message-ID: On Tue, Jun 23, 2020 at 9:51 AM Mark Adams wrote: > > > On Tue, Jun 23, 2020 at 9:54 AM Jed Brown wrote: > >> Did you use --smpiargs=-gpu, and have you tried if the error is still >> > > > >> there with -use_gpu_aware_mpi 0? > > > That did it. Thanks, > Weird. It should work with -use_gpu_aware_mpi 1, if you use jsrun --smpiargs=-gpu > > >> I assume you're using -dm_vec_type cuda? >> >> Mark Adams writes: >> >> > My code runs OK on SUMMIT but ex56 does not. ex56 runs in serial but I >> get >> > this segv in parallel. I also see these memcheck messages >> > from PetscSFBcastAndOpBegin in my code and ex56. >> > >> > I ran this in DDT and was able to get a stack trace and look at >> variables. >> > THe segv is on sfbasic.c:148: >> > >> > ierr = >> > >> MPI_Startall_isend(bas->rootbuflen[PETSCSF_REMOTE],unit,bas->nrootreqs,rootreqs);CHKERRQ(ierr); >> > >> > I did not see anything wrong with the variables here. The segv is on >> > processor 1 of 2 (so the last process). >> > >> > Any ideas? >> > Thanks, >> > Mark >> > >> > #18 main (argc=, args=) at >> > /ccs/home/adams/petsc-old/src/snes/tutorials/ex56.c:477 (at >> > 0x0000000010006224) >> > #17 SNESSolve (snes=0x31064ac0, b=0x3241ac40, x=) at >> > /autofs/nccs-svm1_home1/adams/petsc-old/src/snes/interface/snes.c:4515 >> (at >> > 0x0000200000de3498) >> > #16 SNESSolve_NEWTONLS (snes=0x31064ac0) at >> > /autofs/nccs-svm1_home1/adams/petsc-old/src/snes/impls/ls/ls.c:175 (at >> > 0x0000200000e1b344) >> > #15 SNESComputeFunction (snes=0x31064ac0, x=0x312a3170, y=0x4a464bd0) at >> > /autofs/nccs-svm1_home1/adams/petsc-old/src/snes/interface/snes.c:2378 >> (at >> > 0x0000200000dd5024) >> > #14 SNESComputeFunction_DMLocal (snes=0x31064ac0, X=0x312a3170, >> > F=0x4a464bd0, ctx=0x4a3a7ed0) at >> > /autofs/nccs-svm1_home1/adams/petsc-old/src/snes/utils/dmlocalsnes.c:71 >> (at >> > 0x0000200000dab058) >> > #13 DMGlobalToLocalBegin (dm=0x30fac020, g=0x312a3170, mode=> > out>, l=0x4a444c40) at >> > /autofs/nccs-svm1_home1/adams/petsc-old/src/dm/interface/dm.c:2407 (at >> > 0x0000200000b28a38) >> > #12 PetscSFBcastBegin (leafdata=0x200073c00a00, rootdata=0x200073a00000, >> > unit=, sf=0x30f28980) at >> > /ccs/home/adams/petsc-old/include/petscsf.h:189 (at 0x0000200000b28a38) >> > #11 PetscSFBcastAndOpBegin (sf=0x30f28980, unit=0x200021879ed0, >> > rootdata=0x200073a00000, leafdata=0x200073c00a00, op=0x200021889c70) at >> > >> /autofs/nccs-svm1_home1/adams/petsc-old/src/vec/is/sf/interface/sf.c:1337 >> > (at 0x000020000045a230) >> > #10 PetscSFBcastAndOpBegin_Basic (sf=0x30f28980, unit=0x200021879ed0, >> > rootmtype=, rootdata=0x200073a00000, leafmtype=> > out>, leafdata=0x200073c00a00, op=0x200021889c70) at >> > >> /autofs/nccs-svm1_home1/adams/petsc-old/src/vec/is/sf/impls/basic/sfbasic.c:148 >> > (at 0x00002000003b1d9c) >> > #9 PMPI_Startall () from >> > >> /autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-6.4.0/spectrum-mpi-10.3.1.2-20200121-awz2q5brde7wgdqqw4ugalrkukeub4eb/container/../lib/libmpi_ibm.so.3 >> > (at 0x00002000217e3d98) >> > #8 mca_pml_pami_start () from >> > >> /autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-6.4.0/spectrum-mpi-10.3.1.2-20200121-awz2q5brde7wgdqqw4ugalrkukeub4eb/container/../lib/spectrum_mpi/mca_pml_pami.so >> > (at 0x000020002555e6e0) >> > #7 pml_pami_persis_send_start () from >> > >> /autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-6.4.0/spectrum-mpi-10.3.1.2-20200121-awz2q5brde7wgdqqw4ugalrkukeub4eb/container/../lib/spectrum_mpi/mca_pml_pami.so >> > (at 0x000020002555e29c) >> > #6 pml_pami_send () from >> > >> /autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-6.4.0/spectrum-mpi-10.3.1.2-20200121-awz2q5brde7wgdqqw4ugalrkukeub4eb/container/../lib/spectrum_mpi/mca_pml_pami.so >> > (at 0x000020002555f69c) >> > #5 PAMI_Send_immediate () from >> > >> /autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-6.4.0/spectrum-mpi-10.3.1.2-20200121-awz2q5brde7wgdqqw4ugalrkukeub4eb/container/../lib/pami_port/libpami.so.3 >> > (at 0x0000200025725814) >> > #4 >> > >> PAMI::Protocol::Send::Eager> > 4096u>, PAMI::Counter::IndirectBounded, >> 256u>, >> > PAMI::Counter::Indirect, >> > PAMI::Device::Shmem::CMAShaddr, 256u, 512u> >, >> > PAMI::Device::IBV::PacketModel >> >>::EagerImpl<(PAMI::Protocol::Send::configuration_t)5, >> > true>::immediate(pami_send_immediate_t*) () from >> > >> /autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-6.4.0/spectrum-mpi-10.3.1.2-20200121-awz2q5brde7wgdqqw4ugalrkukeub4eb/container/../lib/pami_port/libpami.so.3 >> > (at 0x00002000257e7bac) >> > #3 >> > >> PAMI::Protocol::Send::EagerSimple> > 4096u>, PAMI::Counter::IndirectBounded, >> 256u>, >> > PAMI::Counter::Indirect, >> > PAMI::Device::Shmem::CMAShaddr, 256u, 512u> >, >> > >> (PAMI::Protocol::Send::configuration_t)5>::immediate_impl(pami_send_immediate_t*) >> > () from >> > >> /autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-6.4.0/spectrum-mpi-10.3.1.2-20200121-awz2q5brde7wgdqqw4ugalrkukeub4eb/container/../lib/pami_port/libpami.so.3 >> > (at 0x00002000257e7824) >> > #2 bool >> > >> PAMI::Device::Interface::PacketModel> > 4096u>, PAMI::Counter::IndirectBounded, >> 256u>, >> > PAMI::Counter::Indirect, >> > PAMI::Device::Shmem::CMAShaddr, 256u, 512u> > >::postPacket<2u>(unsigned >> > long, unsigned long, void*, unsigned long, iovec (&) [2u]) () from >> > >> /autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-6.4.0/spectrum-mpi-10.3.1.2-20200121-awz2q5brde7wgdqqw4ugalrkukeub4eb/container/../lib/pami_port/libpami.so.3 >> > (at 0x00002000257e6c18) >> > #1 PAMI::Device::Shmem::Packet >> >>::writePayload(PAMI::Fifo::FifoPacket<64u, 4096u>&, iovec*, unsigned >> long) >> > () from >> > >> /autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-6.4.0/spectrum-mpi-10.3.1.2-20200121-awz2q5brde7wgdqqw4ugalrkukeub4eb/container/../lib/pami_port/libpami.so.3 >> > (at 0x00002000257c5a7c) >> > #0 __memcpy_power7 () from /lib64/libc.so.6 (at 0x00002000219eb84c) >> > >> > >> > ========= Program hit CUDA_ERROR_INVALID_VALUE (error 1) due to "invalid >> > argument" on CUDA API call to cuPointerGetAttribute. >> > ========= Saved host backtrace up to driver entry point at error >> > ========= Host Frame:/lib64/libcuda.so.1 (cuPointerGetAttribute + >> > 0x178) [0x2d14a8] >> > ========= Host >> > >> Frame:/ccs/home/adams/petsc-old/arch-summit-opt-gnu-cuda-omp/lib/libpetsc.so.3.013 >> > (PetscSFBcastAndOpBegin + 0xd8) [0x3ba1a0] >> > ========= Host >> > >> Frame:/ccs/home/adams/petsc-old/arch-summit-opt-gnu-cuda-omp/lib/libpetsc.so.3.013 >> > (PetscSectionCreateGlobalSection + 0x948) [0x3c5b24] >> > ========= Host >> > >> Frame:/ccs/home/adams/petsc-old/arch-summit-opt-gnu-cuda-omp/lib/libpetsc.so.3.013 >> > (DMGetGlobalSection + 0x98) [0xa85e5c] >> > ========= Host >> > >> Frame:/ccs/home/adams/petsc-old/arch-summit-opt-gnu-cuda-omp/lib/libpetsc.so.3.013 >> > (DMPlexCreateRigidBody + 0xc4) [0x9a9ce4] >> > ========= Host Frame:./ex56 [0x5aec] >> > ========= Host Frame:/lib64/libc.so.6 [0x25200] >> > ========= Host Frame:/lib64/libc.so.6 (__libc_start_main + 0xc4) >> > [0x253f4] >> > ========= >> > ========= Program hit CUDA_ERROR_INVALID_VALUE (error 1) due to "invalid >> > argument" on CUDA API call to cuPointerGetAttribute. >> > ========= Saved host backtrace up to driver entry point at error >> > ========= Host Frame:/lib64/libcuda.so.1 (cuPointerGetAttribute + >> > 0x178) [0x2d14a8] >> > ========= Host >> > >> Frame:/ccs/home/adams/petsc-old/arch-summit-opt-gnu-cuda-omp/lib/libpetsc[1]PETSC >> > ERROR: >> > ------------------------------------------------------------------------ >> > [1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, >> > probably memory access out of range >> > [1]PETSC ERROR: Try option -start_in_debugger or >> -on_error_attach_debugger >> > [1]PETSC ERROR: or see >> > https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind >> > [1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac >> OS X >> > to find memory corruption errors >> > [1]PETSC ERROR: configure using --with-debugging=yes, recompile, link, >> and >> > run >> > [1]PETSC ERROR: to get more information on the crash. >> > [1]PETSC ERROR: --------------------- Error Message >> > -------------------------------------------------------------- >> > [1]PETSC ERROR: Signal received >> > [1]PETSC ERROR: See >> https://www.mcs.anl.gov/petsc/documentation/faq.html >> > for trouble shooting. >> > [1]PETSC ERROR: Petsc Development GIT revision: v3.13.2-421-gab8fa13 >> GIT >> > Date: 2020-06-22 13:25:32 -0400 >> > [1]PETSC ERROR: ./ex56 on a arch-summit-opt-gnu-cuda-omp named h35n05 by >> > adams Tue Jun 23 09:06:20 2020 >> > [1]PETSC ERROR: Configure options --with-fc=0 --COPTFLAGS="-g -O -fPIC >> > -DFP_DIM=2" --CXXOPTFLAGS="-g -O -fPIC " --FOPTFLAGS="-g -O -fPIC " >> > --CUDAOPTFLAGS="-g -O -Xcompiler -rdynamic -lineinfo--with-ssl=0" >> > --with-batch=0 --with-cxx=mpicxx --with-mpiexec="jsrun -g1" >> --with-cuda=1 >> > --with-cudac=nvcc --download-p4est=1 --download-zlib --download-hdf5=1 >> > --download-metis --download-parmetis --download-triangle >> > >> --with-blaslapack-lib="-L/autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-6.4.0/netlib-lapack-3.8.0-wcabdyqhdi5rooxbkqa6x5d7hxyxwdkm/lib64 >> > -lblas -llapack" --with-cc=mpicc --with-shared-libraries=1 --with-x=0 >> > --with-64-bit-indices=0 --with-debugging=0 >> > PETSC_ARCH=arch-summit-opt-gnu-cuda-omp --with-openmp=1 >> > --with-threadsaftey=1 --with-log=1 PETSC_DIR=/ccs/home/adams/petsc-old >> > --force >> > [1]PETSC ERROR: #1 User provided function() line 0 in unknown file >> > >> -------------------------------------------------------------------------- >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Tue Jun 23 10:16:49 2020 From: mfadams at lbl.gov (Mark Adams) Date: Tue, 23 Jun 2020 11:16:49 -0400 Subject: [petsc-users] CUDA error In-Reply-To: References: <871rm5zy4c.fsf@jedbrown.org> Message-ID: I run in an interactive shell on SUMMIT and don't need --smpiargs=-gpu. I have been having a lot of problems getting the cuda test to be reproducible in the pipeline and SUMMIT was failing. -use_gpu_aware_mpi 0 fixed the test on SUMMIT, in that the test ran and it look OK, and I updated the test. Hoping it fixes the pipeline ex56/cuda diffs. On Tue, Jun 23, 2020 at 11:07 AM Stefano Zampini wrote: > What did it? How are you running now to have everything working? Can you > post smpiargs and petsc options? > > Il giorno mar 23 giu 2020 alle ore 17:51 Mark Adams ha > scritto: > >> >> >> On Tue, Jun 23, 2020 at 9:54 AM Jed Brown wrote: >> >>> Did you use --smpiargs=-gpu, and have you tried if the error is still >>> >> >> >> >>> there with -use_gpu_aware_mpi 0? >> >> >> That did it. Thanks, >> >> >>> I assume you're using -dm_vec_type cuda? >>> >>> Mark Adams writes: >>> >>> > My code runs OK on SUMMIT but ex56 does not. ex56 runs in serial but I >>> get >>> > this segv in parallel. I also see these memcheck messages >>> > from PetscSFBcastAndOpBegin in my code and ex56. >>> > >>> > I ran this in DDT and was able to get a stack trace and look at >>> variables. >>> > THe segv is on sfbasic.c:148: >>> > >>> > ierr = >>> > >>> MPI_Startall_isend(bas->rootbuflen[PETSCSF_REMOTE],unit,bas->nrootreqs,rootreqs);CHKERRQ(ierr); >>> > >>> > I did not see anything wrong with the variables here. The segv is on >>> > processor 1 of 2 (so the last process). >>> > >>> > Any ideas? >>> > Thanks, >>> > Mark >>> > >>> > #18 main (argc=, args=) at >>> > /ccs/home/adams/petsc-old/src/snes/tutorials/ex56.c:477 (at >>> > 0x0000000010006224) >>> > #17 SNESSolve (snes=0x31064ac0, b=0x3241ac40, x=) at >>> > /autofs/nccs-svm1_home1/adams/petsc-old/src/snes/interface/snes.c:4515 >>> (at >>> > 0x0000200000de3498) >>> > #16 SNESSolve_NEWTONLS (snes=0x31064ac0) at >>> > /autofs/nccs-svm1_home1/adams/petsc-old/src/snes/impls/ls/ls.c:175 (at >>> > 0x0000200000e1b344) >>> > #15 SNESComputeFunction (snes=0x31064ac0, x=0x312a3170, y=0x4a464bd0) >>> at >>> > /autofs/nccs-svm1_home1/adams/petsc-old/src/snes/interface/snes.c:2378 >>> (at >>> > 0x0000200000dd5024) >>> > #14 SNESComputeFunction_DMLocal (snes=0x31064ac0, X=0x312a3170, >>> > F=0x4a464bd0, ctx=0x4a3a7ed0) at >>> > >>> /autofs/nccs-svm1_home1/adams/petsc-old/src/snes/utils/dmlocalsnes.c:71 (at >>> > 0x0000200000dab058) >>> > #13 DMGlobalToLocalBegin (dm=0x30fac020, g=0x312a3170, mode=>> > out>, l=0x4a444c40) at >>> > /autofs/nccs-svm1_home1/adams/petsc-old/src/dm/interface/dm.c:2407 (at >>> > 0x0000200000b28a38) >>> > #12 PetscSFBcastBegin (leafdata=0x200073c00a00, >>> rootdata=0x200073a00000, >>> > unit=, sf=0x30f28980) at >>> > /ccs/home/adams/petsc-old/include/petscsf.h:189 (at 0x0000200000b28a38) >>> > #11 PetscSFBcastAndOpBegin (sf=0x30f28980, unit=0x200021879ed0, >>> > rootdata=0x200073a00000, leafdata=0x200073c00a00, op=0x200021889c70) at >>> > >>> /autofs/nccs-svm1_home1/adams/petsc-old/src/vec/is/sf/interface/sf.c:1337 >>> > (at 0x000020000045a230) >>> > #10 PetscSFBcastAndOpBegin_Basic (sf=0x30f28980, unit=0x200021879ed0, >>> > rootmtype=, rootdata=0x200073a00000, >>> leafmtype=>> > out>, leafdata=0x200073c00a00, op=0x200021889c70) at >>> > >>> /autofs/nccs-svm1_home1/adams/petsc-old/src/vec/is/sf/impls/basic/sfbasic.c:148 >>> > (at 0x00002000003b1d9c) >>> > #9 PMPI_Startall () from >>> > >>> /autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-6.4.0/spectrum-mpi-10.3.1.2-20200121-awz2q5brde7wgdqqw4ugalrkukeub4eb/container/../lib/libmpi_ibm.so.3 >>> > (at 0x00002000217e3d98) >>> > #8 mca_pml_pami_start () from >>> > >>> /autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-6.4.0/spectrum-mpi-10.3.1.2-20200121-awz2q5brde7wgdqqw4ugalrkukeub4eb/container/../lib/spectrum_mpi/mca_pml_pami.so >>> > (at 0x000020002555e6e0) >>> > #7 pml_pami_persis_send_start () from >>> > >>> /autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-6.4.0/spectrum-mpi-10.3.1.2-20200121-awz2q5brde7wgdqqw4ugalrkukeub4eb/container/../lib/spectrum_mpi/mca_pml_pami.so >>> > (at 0x000020002555e29c) >>> > #6 pml_pami_send () from >>> > >>> /autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-6.4.0/spectrum-mpi-10.3.1.2-20200121-awz2q5brde7wgdqqw4ugalrkukeub4eb/container/../lib/spectrum_mpi/mca_pml_pami.so >>> > (at 0x000020002555f69c) >>> > #5 PAMI_Send_immediate () from >>> > >>> /autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-6.4.0/spectrum-mpi-10.3.1.2-20200121-awz2q5brde7wgdqqw4ugalrkukeub4eb/container/../lib/pami_port/libpami.so.3 >>> > (at 0x0000200025725814) >>> > #4 >>> > >>> PAMI::Protocol::Send::Eager>> > 4096u>, PAMI::Counter::IndirectBounded, >>> 256u>, >>> > PAMI::Counter::Indirect, >>> > PAMI::Device::Shmem::CMAShaddr, 256u, 512u> >, >>> > PAMI::Device::IBV::PacketModel >>> >>::EagerImpl<(PAMI::Protocol::Send::configuration_t)5, >>> > true>::immediate(pami_send_immediate_t*) () from >>> > >>> /autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-6.4.0/spectrum-mpi-10.3.1.2-20200121-awz2q5brde7wgdqqw4ugalrkukeub4eb/container/../lib/pami_port/libpami.so.3 >>> > (at 0x00002000257e7bac) >>> > #3 >>> > >>> PAMI::Protocol::Send::EagerSimple>> > 4096u>, PAMI::Counter::IndirectBounded, >>> 256u>, >>> > PAMI::Counter::Indirect, >>> > PAMI::Device::Shmem::CMAShaddr, 256u, 512u> >, >>> > >>> (PAMI::Protocol::Send::configuration_t)5>::immediate_impl(pami_send_immediate_t*) >>> > () from >>> > >>> /autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-6.4.0/spectrum-mpi-10.3.1.2-20200121-awz2q5brde7wgdqqw4ugalrkukeub4eb/container/../lib/pami_port/libpami.so.3 >>> > (at 0x00002000257e7824) >>> > #2 bool >>> > >>> PAMI::Device::Interface::PacketModel>> > 4096u>, PAMI::Counter::IndirectBounded, >>> 256u>, >>> > PAMI::Counter::Indirect, >>> > PAMI::Device::Shmem::CMAShaddr, 256u, 512u> > >>> >::postPacket<2u>(unsigned >>> > long, unsigned long, void*, unsigned long, iovec (&) [2u]) () from >>> > >>> /autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-6.4.0/spectrum-mpi-10.3.1.2-20200121-awz2q5brde7wgdqqw4ugalrkukeub4eb/container/../lib/pami_port/libpami.so.3 >>> > (at 0x00002000257e6c18) >>> > #1 PAMI::Device::Shmem::Packet >>> >>::writePayload(PAMI::Fifo::FifoPacket<64u, 4096u>&, iovec*, unsigned >>> long) >>> > () from >>> > >>> /autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-6.4.0/spectrum-mpi-10.3.1.2-20200121-awz2q5brde7wgdqqw4ugalrkukeub4eb/container/../lib/pami_port/libpami.so.3 >>> > (at 0x00002000257c5a7c) >>> > #0 __memcpy_power7 () from /lib64/libc.so.6 (at 0x00002000219eb84c) >>> > >>> > >>> > ========= Program hit CUDA_ERROR_INVALID_VALUE (error 1) due to >>> "invalid >>> > argument" on CUDA API call to cuPointerGetAttribute. >>> > ========= Saved host backtrace up to driver entry point at error >>> > ========= Host Frame:/lib64/libcuda.so.1 (cuPointerGetAttribute + >>> > 0x178) [0x2d14a8] >>> > ========= Host >>> > >>> Frame:/ccs/home/adams/petsc-old/arch-summit-opt-gnu-cuda-omp/lib/libpetsc.so.3.013 >>> > (PetscSFBcastAndOpBegin + 0xd8) [0x3ba1a0] >>> > ========= Host >>> > >>> Frame:/ccs/home/adams/petsc-old/arch-summit-opt-gnu-cuda-omp/lib/libpetsc.so.3.013 >>> > (PetscSectionCreateGlobalSection + 0x948) [0x3c5b24] >>> > ========= Host >>> > >>> Frame:/ccs/home/adams/petsc-old/arch-summit-opt-gnu-cuda-omp/lib/libpetsc.so.3.013 >>> > (DMGetGlobalSection + 0x98) [0xa85e5c] >>> > ========= Host >>> > >>> Frame:/ccs/home/adams/petsc-old/arch-summit-opt-gnu-cuda-omp/lib/libpetsc.so.3.013 >>> > (DMPlexCreateRigidBody + 0xc4) [0x9a9ce4] >>> > ========= Host Frame:./ex56 [0x5aec] >>> > ========= Host Frame:/lib64/libc.so.6 [0x25200] >>> > ========= Host Frame:/lib64/libc.so.6 (__libc_start_main + 0xc4) >>> > [0x253f4] >>> > ========= >>> > ========= Program hit CUDA_ERROR_INVALID_VALUE (error 1) due to >>> "invalid >>> > argument" on CUDA API call to cuPointerGetAttribute. >>> > ========= Saved host backtrace up to driver entry point at error >>> > ========= Host Frame:/lib64/libcuda.so.1 (cuPointerGetAttribute + >>> > 0x178) [0x2d14a8] >>> > ========= Host >>> > >>> Frame:/ccs/home/adams/petsc-old/arch-summit-opt-gnu-cuda-omp/lib/libpetsc[1]PETSC >>> > ERROR: >>> > >>> ------------------------------------------------------------------------ >>> > [1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, >>> > probably memory access out of range >>> > [1]PETSC ERROR: Try option -start_in_debugger or >>> -on_error_attach_debugger >>> > [1]PETSC ERROR: or see >>> > https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind >>> > [1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac >>> OS X >>> > to find memory corruption errors >>> > [1]PETSC ERROR: configure using --with-debugging=yes, recompile, link, >>> and >>> > run >>> > [1]PETSC ERROR: to get more information on the crash. >>> > [1]PETSC ERROR: --------------------- Error Message >>> > -------------------------------------------------------------- >>> > [1]PETSC ERROR: Signal received >>> > [1]PETSC ERROR: See >>> https://www.mcs.anl.gov/petsc/documentation/faq.html >>> > for trouble shooting. >>> > [1]PETSC ERROR: Petsc Development GIT revision: v3.13.2-421-gab8fa13 >>> GIT >>> > Date: 2020-06-22 13:25:32 -0400 >>> > [1]PETSC ERROR: ./ex56 on a arch-summit-opt-gnu-cuda-omp named h35n05 >>> by >>> > adams Tue Jun 23 09:06:20 2020 >>> > [1]PETSC ERROR: Configure options --with-fc=0 --COPTFLAGS="-g -O -fPIC >>> > -DFP_DIM=2" --CXXOPTFLAGS="-g -O -fPIC " --FOPTFLAGS="-g -O -fPIC " >>> > --CUDAOPTFLAGS="-g -O -Xcompiler -rdynamic -lineinfo--with-ssl=0" >>> > --with-batch=0 --with-cxx=mpicxx --with-mpiexec="jsrun -g1" >>> --with-cuda=1 >>> > --with-cudac=nvcc --download-p4est=1 --download-zlib --download-hdf5=1 >>> > --download-metis --download-parmetis --download-triangle >>> > >>> --with-blaslapack-lib="-L/autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-6.4.0/netlib-lapack-3.8.0-wcabdyqhdi5rooxbkqa6x5d7hxyxwdkm/lib64 >>> > -lblas -llapack" --with-cc=mpicc --with-shared-libraries=1 --with-x=0 >>> > --with-64-bit-indices=0 --with-debugging=0 >>> > PETSC_ARCH=arch-summit-opt-gnu-cuda-omp --with-openmp=1 >>> > --with-threadsaftey=1 --with-log=1 PETSC_DIR=/ccs/home/adams/petsc-old >>> > --force >>> > [1]PETSC ERROR: #1 User provided function() line 0 in unknown file >>> > >>> -------------------------------------------------------------------------- >>> >> > > -- > Stefano > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bantingl at myumanitoba.ca Tue Jun 23 10:49:43 2020 From: bantingl at myumanitoba.ca (Lucas Banting) Date: Tue, 23 Jun 2020 15:49:43 +0000 Subject: [petsc-users] Question about HDF5 Viewer and Datasets In-Reply-To: References: , Message-ID: Thank you both. The reason the sequential one was storing multiple datasets was that the vector was created and destroyed each time step, so the name was changing in the hdf5 file as the pointer for the vector was changing. Neither version originally was ever calling SetTimeStep, which I have since added and it appears to be working as intended. ________________________________ From: Matthew Knepley Sent: Tuesday, June 23, 2020 8:15 AM To: Lucas Banting Cc: PETSc Subject: Re: [petsc-users] Question about HDF5 Viewer and Datasets On Tue, Jun 23, 2020 at 8:56 AM Lucas Banting > wrote: Hello, I am trying to use the HDF5 viewer to store the solution of my system at every iteration. I am storing the solutions in order to compare two different results, one from an old sequential program and one from a new parallel program that is parallelized using the DMDA structure. I am using matlab to compare the results. In the sequential program, the vector for the solution is just a simple one dimensional vector made from VecCreate. In the parallel program, the vector is made from DMCreateGlobalVector. In both versions of the program, I create the viewer as: call PetscViewerHDF5Open(PETSC_COMM_WORLD,"filename.hdf5",& FILE_MODE_WRITE,hdf5viewer,ierr) When I load the files into MATLAB after words, the sequential program creates a new dataset each iteration, while the parallel version using DMCreateGlobalVector only stores a single dataset. Is there a way I can force both viewers to create a new dataset in the same way each time vecview is called? Or is there someway I can manually create a new dataset? You can see inside the file using 'h5ls' and 'h5dump'. I believe that the Vec is stored in HDF5 using the name (of the PetscObject). I think that is happening is that you are not setting the name, which means it defaults to the pointer, and that pointer is the same for your global vector. I believe if you change the name at each iteration, they will all be saved. You can also use the Timestep mechanism as Barry suggests. Thanks, Matt I am using Open MPI 3.1.4, Petsc 3.12.4, Intel Fortran 2019.5, and hdf5/1.10.5. Thanks, Lucas Banting -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From shaswat121994 at gmail.com Tue Jun 23 13:43:13 2020 From: shaswat121994 at gmail.com (Shashwat Tiwari) Date: Wed, 24 Jun 2020 00:13:13 +0530 Subject: [petsc-users] Implementing periodicity using DMPlex In-Reply-To: References: <4E40F31A-AD04-4F7C-AA82-B43C9DBEA074@gmail.com> Message-ID: On Sat, Jun 20, 2020 at 2:41 AM Matthew Knepley wrote: > On Thu, Jun 18, 2020 at 1:20 PM Shashwat Tiwari > wrote: > >> Thank you for the help. I followed your suggestion and removed the >> "DMPlexConstructGhostCells" function and now I am using a gmsh generated >> mesh with periodicity in both directions, and distributing the mesh with 1 >> level of overlap. I can see that the periodic faces now have two >> neighbouring cells on the opposite side of the domain which I can use to >> get the periodic cell values using "DMGlobalToLocal" without the need to >> construct ghost cells. There are two main issues that I am facing now, >> regarding geometric information needed for implementing periodic bc. >> >> One, I may need the centroid of periodic counterpart of a cell which is >> possible to get from a "ghost" cell created on the periodic face, and >> containing the solution data from periodic counterpart. But now, I would >> need to shift the coordinates of the neighbouring cell accordingly. If you >> see the sample output given below along with the screenshot of the sample >> 2D mesh that I have used (please find attached), while reconstructing the >> left and right solution values on face 32 for a second order scheme, I need >> the centroid of a ghost cell on left side of the face which is at (x, y) = >> (-1.25, 8.75). I wanted to know if there is some way to get these shifted >> "ghost" coordinates directly from the geometric information obtained using >> "PetscFV" class. Kindly let me know if you have some suggestions regarding >> this. >> > > I do not completely understand the question. However, I think you want > something like > > > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DM/DMLocalizeCoordinate.html > > There are many variants of this functionality. > This function seems to be for the purpose of shifting the coordinates which are outside the domain, to inside the domain limit. I want to shift the coordinates of the periodic neighbor of a cell, which is a cell inside the domain. I think I can manage to do that myself manually without much complications. Sample Output: >> >> face: 32, centroid = 0.000000, 8.750000, normal = 2.500000, -0.000000 >> support[0] = 0, cent = 1.250000, 8.750000, area = 6.250000 >> support[1] = 3, cent = 8.750000, 8.750000, area = 6.250000 >> >> face: 33, centroid = 1.250000, 7.500000, normal = -0.000000, -2.500000 >> support[0] = 0, cent = 1.250000, 8.750000, area = 6.250000 >> support[1] = 4, cent = 1.250000, 6.250000, area = 6.250000 >> >> Second, the face normals of the periodic faces obtained using the face >> geometry data from "DMPlexGetDataFVM" function seem to be in the wrong >> direction i.e., from the right cell "support[1]" to the left cell >> "support[0]", while the other faces have the opposite direction. For face: >> 32, left cell is 0 and the right cell is 3, but, the normal points the >> opposite way (cell 3 would be on the left side of cell 0 on periodic >> boundary), while, for face: 33 which is an interior face, the normal >> direction is from left to right cell. >> > > This should be easy to check. The strange thing is that if this were > broken, you would expect other examples to fail. > I am actually unable to get these values right. The normals seem to be in the opposite direction for all the periodic faces and length of some faces gets multiplied by a factor of three. Were you able to look at this? If so, kindly let me know how I can resolve this issue. Regards, Shashwat > > >> Also, I am getting incorrect normal values for some of the faces >> (including some interior faces). Here is a sample output of the same: >> >> face: 59, centroid = 3.750000, 2.500000, normal = 0.000000, -7.500000 >> support[0] = 11, cent = 8.750000, 3.750000, area = 6.250000 >> support[1] = 15, cent = 8.750000, 1.250000, area = 6.250000 >> >> this face has the correct direction (from 11 to 15), but the value is >> wrong (all the faces are 2.5 units). This problem does not occur when I >> ignore periodicity by setting -dm_plex_gmsh_periodic to 0. >> >> These issues occur regardless of the number of processes. I am attaching >> the test code that I am using along with the gmsh file for your reference. >> Please suggest to me, what might be the issue here. >> > > I will take a look. > > Thanks, > > Matt > > >> Regards, >> Shashwat >> >> >> On Mon, Jun 15, 2020 at 2:54 PM Stefano Zampini < >> stefano.zampini at gmail.com> wrote: >> >>> It is enough if you use DMPlexDistribute with 1 level of overlap, set >>> the local section for your dofs, and call DMGlobalToLocal . The local >>> vector will contain data for what you call ?ghost? cells. I call them >>> ?not-owned? cells. >>> >>> On Jun 15, 2020, at 12:09 PM, Shashwat Tiwari >>> wrote: >>> >>> The way I'm trying to implement periodic bc is, when I loop over the >>> boundary faces, say, on the left boundary of the domain to compute flux and >>> residual, I need solution values from the two cells neighbouring the face, >>> i.e. the left cell and the right cell (the face normal pointing from the >>> left cell to the right cell and the right cell being a boundary ghost cell >>> for boundary faces). For the boundary to be periodic, I need the value that >>> I get from the right cell (boundary ghost cell) to be the solution value at >>> its periodic counterpart, i.e. the solution value at left cell of the face >>> on right boundary of the domain in this case. My question is how do I >>> update the value at a boundary ghost cell with the value at the real cell >>> which is its periodic counterpart from the other side of the domain. Is >>> there some kind of mapping of the boundary ghost cells to their >>> corresponding real cells which I can use to update the solution values at >>> ghost cells? >>> >>> Regards, >>> Shashwat >>> >>> On Sun, Jun 14, 2020 at 5:11 AM Matthew Knepley >>> wrote: >>> >>>> On Fri, Jun 12, 2020 at 3:19 PM Shashwat Tiwari < >>>> shaswat121994 at gmail.com> wrote: >>>> >>>>> Hi, >>>>> I am writing a first order 2D solver for unstructured grids with >>>>> periodic boundaries using DMPlex. After generating the mesh, I use >>>>> "DMSetPeriodicity" function to set periodicity in both directions. After >>>>> which I partition the mesh (DMPlexDistribute), construct ghost cells >>>>> (DMPlexConstructGhostCells), >>>>> >>>> >>>> These ghost cells are for FVM boundary conditions. If you want cells to >>>> be shared across parallel partitions, then you want to give overlap=1 >>>> to DMPlexDIstribute(). Is that what you want? >>>> >>>> Thanks, >>>> >>>> Matt >>>> >>>> >>>>> create a section, and set some initial values in the global vector. >>>>> Then I use "VecGhostUpdateBegin" to start updating the boundary ghost cell >>>>> values, but, I get the following error in case I use multiple processors: >>>>> >>>>> [0]PETSC ERROR: Invalid argument >>>>> [0]PETSC ERROR: Vector is not ghosted >>>>> [0]PETSC ERROR: See >>>>> https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble >>>>> shooting. >>>>> >>>>> if I run with a single process, there is no error but the values >>>>> remain empty (zero) and are not updated. Kindly let me know, if I am >>>>> missing some crucial step before I can update the ghost values in order to >>>>> implement the periodic bc, or if there is any other approach to achieve it. >>>>> I am attaching a small code to demonstrate the issue for your reference. >>>>> >>>>> Regards, >>>>> Shashwat >>>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>>> https://www.cse.buffalo.edu/~knepley/ >>>> >>>> >>> >>> > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From EPrudencio at slb.com Wed Jun 24 16:25:17 2020 From: EPrudencio at slb.com (Ernesto Prudencio) Date: Wed, 24 Jun 2020 21:25:17 +0000 Subject: [petsc-users] Question on SLEPc + computing SVD with a "matrix free" matrix Message-ID: Hi, Is it possible to compute a SVD for a "matrix free" matrix? At first, it seems it would be ok with a MatCreateMFFD() and MatMFFDSetFunction(), because one could then provide the routine that computes "A . v" for any given v, which is an operation needed by SLEPc. However, one would also need to set up the SVD object in SLEPc with an implicit transpose. Would that be possible? Thanks in advance, Ernesto. Schlumberger-Private -------------- next part -------------- An HTML attachment was scrubbed... URL: From jroman at dsic.upv.es Thu Jun 25 01:00:31 2020 From: jroman at dsic.upv.es (Jose E. Roman) Date: Thu, 25 Jun 2020 08:00:31 +0200 Subject: [petsc-users] Question on SLEPc + computing SVD with a "matrix free" matrix In-Reply-To: References: Message-ID: Yes, you have to set it with SVDSetImplicitTranspose(), but then the matrix-free matrix should implement both "A.v" and "A'.v" operations. Jose > El 24 jun 2020, a las 23:25, Ernesto Prudencio via petsc-users escribi?: > > Hi, > > Is it possible to compute a SVD for a ?matrix free? matrix? > > At first, it seems it would be ok with a MatCreateMFFD() and MatMFFDSetFunction(), because one could then provide the routine that computes ?A . v? for any given v, which is an operation needed by SLEPc. However, one would also need to set up the SVD object in SLEPc with an implicit transpose. Would that be possible? > > Thanks in advance, > > Ernesto. > > Schlumberger-Private From EPrudencio at slb.com Thu Jun 25 01:23:23 2020 From: EPrudencio at slb.com (Ernesto Prudencio) Date: Thu, 25 Jun 2020 06:23:23 +0000 Subject: [petsc-users] [Ext] Re: Question on SLEPc + computing SVD with a "matrix free" matrix In-Reply-To: References: Message-ID: Thank you, Jose. However, in the case of a "matrix free" matrix, the APIs on PETSc seem to allow just the implementation of A.v, not of A' . w One could create another "matrix free" matrix B which could make the role of computing z = B . w = A' . w. But how could one force the routine MatMultTranspose(A, w, z) to call the routine for B? Here I am assuming that MatMultTranspose(A, w, z) is the routine that SLEPc calls internally in its algorithms when the user sets for implicit transpose. I see two possibilities (for brain storming, since I don't know if such approaches would be acceptable for the PETSc team and/or the SLEPc team): 1) PETSc could add an entry for a second routine that computes [ A' . w ] when calling MatMFFDSetFunction() 2) SLEPc could add an entry for the routine for computing [ A' . w ] when the matrix A is "matrix free" Thanks again, Ernesto. Schlumberger-Private -----Original Message----- From: Jose E. Roman [mailto:jroman at dsic.upv.es] Sent: Thursday, June 25, 2020 1:01 AM To: Ernesto Prudencio Cc: petsc-users at mcs.anl.gov Subject: [Ext] Re: [petsc-users] Question on SLEPc + computing SVD with a "matrix free" matrix Yes, you have to set it with SVDSetImplicitTranspose(), but then the matrix-free matrix should implement both "A.v" and "A'.v" operations. Jose > El 24 jun 2020, a las 23:25, Ernesto Prudencio via petsc-users escribi?: > > Hi, > > Is it possible to compute a SVD for a ?matrix free? matrix? > > At first, it seems it would be ok with a MatCreateMFFD() and MatMFFDSetFunction(), because one could then provide the routine that computes ?A . v? for any given v, which is an operation needed by SLEPc. However, one would also need to set up the SVD object in SLEPc with an implicit transpose. Would that be possible? > > Thanks in advance, > > Ernesto. > > Schlumberger-Private From jroman at dsic.upv.es Thu Jun 25 01:26:20 2020 From: jroman at dsic.upv.es (Jose E. Roman) Date: Thu, 25 Jun 2020 08:26:20 +0200 Subject: [petsc-users] [Ext] Question on SLEPc + computing SVD with a "matrix free" matrix In-Reply-To: References: Message-ID: Another alternative is to create a shell matrix C with a user context that contains both A and B, then MatMult(C) calls MatMult(A) and MatMultTranspose(C) calls MatMult(B). > El 25 jun 2020, a las 8:23, Ernesto Prudencio escribi?: > > Thank you, Jose. > > However, in the case of a "matrix free" matrix, the APIs on PETSc seem to allow just the implementation of A.v, not of A' . w > > One could create another "matrix free" matrix B which could make the role of computing z = B . w = A' . w. But how could one force the routine MatMultTranspose(A, w, z) to call the routine for B? Here I am assuming that MatMultTranspose(A, w, z) is the routine that SLEPc calls internally in its algorithms when the user sets for implicit transpose. > > I see two possibilities (for brain storming, since I don't know if such approaches would be acceptable for the PETSc team and/or the SLEPc team): > 1) PETSc could add an entry for a second routine that computes [ A' . w ] when calling MatMFFDSetFunction() > 2) SLEPc could add an entry for the routine for computing [ A' . w ] when the matrix A is "matrix free" > > Thanks again, > > Ernesto. > > > Schlumberger-Private > > -----Original Message----- > From: Jose E. Roman [mailto:jroman at dsic.upv.es] > Sent: Thursday, June 25, 2020 1:01 AM > To: Ernesto Prudencio > Cc: petsc-users at mcs.anl.gov > Subject: [Ext] Re: [petsc-users] Question on SLEPc + computing SVD with a "matrix free" matrix > > Yes, you have to set it with SVDSetImplicitTranspose(), but then the matrix-free matrix should implement both "A.v" and "A'.v" operations. > Jose > > >> El 24 jun 2020, a las 23:25, Ernesto Prudencio via petsc-users escribi?: >> >> Hi, >> >> Is it possible to compute a SVD for a ?matrix free? matrix? >> >> At first, it seems it would be ok with a MatCreateMFFD() and MatMFFDSetFunction(), because one could then provide the routine that computes ?A . v? for any given v, which is an operation needed by SLEPc. However, one would also need to set up the SVD object in SLEPc with an implicit transpose. Would that be possible? >> >> Thanks in advance, >> >> Ernesto. >> >> Schlumberger-Private From EPrudencio at slb.com Thu Jun 25 01:28:09 2020 From: EPrudencio at slb.com (Ernesto Prudencio) Date: Thu, 25 Jun 2020 06:28:09 +0000 Subject: [petsc-users] [Ext] Question on SLEPc + computing SVD with a "matrix free" matrix In-Reply-To: References: Message-ID: I didn't know about this "shell matrix" concept. I will take a look. Best, Ernesto. Schlumberger-Private -----Original Message----- From: Jose E. Roman [mailto:jroman at dsic.upv.es] Sent: Thursday, June 25, 2020 1:26 AM To: Ernesto Prudencio Cc: petsc-users at mcs.anl.gov Subject: Re: [Ext] [petsc-users] Question on SLEPc + computing SVD with a "matrix free" matrix Another alternative is to create a shell matrix C with a user context that contains both A and B, then MatMult(C) calls MatMult(A) and MatMultTranspose(C) calls MatMult(B). > El 25 jun 2020, a las 8:23, Ernesto Prudencio escribi?: > > Thank you, Jose. > > However, in the case of a "matrix free" matrix, the APIs on PETSc seem to allow just the implementation of A.v, not of A' . w > > One could create another "matrix free" matrix B which could make the role of computing z = B . w = A' . w. But how could one force the routine MatMultTranspose(A, w, z) to call the routine for B? Here I am assuming that MatMultTranspose(A, w, z) is the routine that SLEPc calls internally in its algorithms when the user sets for implicit transpose. > > I see two possibilities (for brain storming, since I don't know if such approaches would be acceptable for the PETSc team and/or the SLEPc team): > 1) PETSc could add an entry for a second routine that computes [ A' . w ] when calling MatMFFDSetFunction() > 2) SLEPc could add an entry for the routine for computing [ A' . w ] when the matrix A is "matrix free" > > Thanks again, > > Ernesto. > > > Schlumberger-Private > > -----Original Message----- > From: Jose E. Roman [mailto:jroman at dsic.upv.es] > Sent: Thursday, June 25, 2020 1:01 AM > To: Ernesto Prudencio > Cc: petsc-users at mcs.anl.gov > Subject: [Ext] Re: [petsc-users] Question on SLEPc + computing SVD with a "matrix free" matrix > > Yes, you have to set it with SVDSetImplicitTranspose(), but then the matrix-free matrix should implement both "A.v" and "A'.v" operations. > Jose > > >> El 24 jun 2020, a las 23:25, Ernesto Prudencio via petsc-users escribi?: >> >> Hi, >> >> Is it possible to compute a SVD for a ?matrix free? matrix? >> >> At first, it seems it would be ok with a MatCreateMFFD() and MatMFFDSetFunction(), because one could then provide the routine that computes ?A . v? for any given v, which is an operation needed by SLEPc. However, one would also need to set up the SVD object in SLEPc with an implicit transpose. Would that be possible? >> >> Thanks in advance, >> >> Ernesto. >> >> Schlumberger-Private From jroman at dsic.upv.es Thu Jun 25 01:37:25 2020 From: jroman at dsic.upv.es (Jose E. Roman) Date: Thu, 25 Jun 2020 08:37:25 +0200 Subject: [petsc-users] [Ext] Question on SLEPc + computing SVD with a "matrix free" matrix In-Reply-To: References: Message-ID: Here is an example that creates a shell matrix with a C struct as a context: https://www.mcs.anl.gov/petsc/petsc-current/src/mat/tests/ex203.c.html Or if you use Fortran: https://www.mcs.anl.gov/petsc/petsc-current/src/mat/tutorials/ex6f.F90.html Jose > El 25 jun 2020, a las 8:28, Ernesto Prudencio escribi?: > > I didn't know about this "shell matrix" concept. I will take a look. > > Best, > > Ernesto. > > > Schlumberger-Private > > -----Original Message----- > From: Jose E. Roman [mailto:jroman at dsic.upv.es] > Sent: Thursday, June 25, 2020 1:26 AM > To: Ernesto Prudencio > Cc: petsc-users at mcs.anl.gov > Subject: Re: [Ext] [petsc-users] Question on SLEPc + computing SVD with a "matrix free" matrix > > Another alternative is to create a shell matrix C with a user context that contains both A and B, then MatMult(C) calls MatMult(A) and MatMultTranspose(C) calls MatMult(B). > > >> El 25 jun 2020, a las 8:23, Ernesto Prudencio escribi?: >> >> Thank you, Jose. >> >> However, in the case of a "matrix free" matrix, the APIs on PETSc seem to allow just the implementation of A.v, not of A' . w >> >> One could create another "matrix free" matrix B which could make the role of computing z = B . w = A' . w. But how could one force the routine MatMultTranspose(A, w, z) to call the routine for B? Here I am assuming that MatMultTranspose(A, w, z) is the routine that SLEPc calls internally in its algorithms when the user sets for implicit transpose. >> >> I see two possibilities (for brain storming, since I don't know if such approaches would be acceptable for the PETSc team and/or the SLEPc team): >> 1) PETSc could add an entry for a second routine that computes [ A' . w ] when calling MatMFFDSetFunction() >> 2) SLEPc could add an entry for the routine for computing [ A' . w ] when the matrix A is "matrix free" >> >> Thanks again, >> >> Ernesto. >> >> >> Schlumberger-Private >> >> -----Original Message----- >> From: Jose E. Roman [mailto:jroman at dsic.upv.es] >> Sent: Thursday, June 25, 2020 1:01 AM >> To: Ernesto Prudencio >> Cc: petsc-users at mcs.anl.gov >> Subject: [Ext] Re: [petsc-users] Question on SLEPc + computing SVD with a "matrix free" matrix >> >> Yes, you have to set it with SVDSetImplicitTranspose(), but then the matrix-free matrix should implement both "A.v" and "A'.v" operations. >> Jose >> >> >>> El 24 jun 2020, a las 23:25, Ernesto Prudencio via petsc-users escribi?: >>> >>> Hi, >>> >>> Is it possible to compute a SVD for a ?matrix free? matrix? >>> >>> At first, it seems it would be ok with a MatCreateMFFD() and MatMFFDSetFunction(), because one could then provide the routine that computes ?A . v? for any given v, which is an operation needed by SLEPc. However, one would also need to set up the SVD object in SLEPc with an implicit transpose. Would that be possible? >>> >>> Thanks in advance, >>> >>> Ernesto. >>> >>> Schlumberger-Private From dave.mayhem23 at gmail.com Thu Jun 25 02:11:51 2020 From: dave.mayhem23 at gmail.com (Dave May) Date: Thu, 25 Jun 2020 09:11:51 +0200 Subject: [petsc-users] [Ext] Re: Question on SLEPc + computing SVD with a "matrix free" matrix In-Reply-To: References: Message-ID: On Thu 25. Jun 2020 at 08:23, Ernesto Prudencio via petsc-users < petsc-users at mcs.anl.gov> wrote: > Thank you, Jose. > > However, in the case of a "matrix free" matrix, the APIs on PETSc seem to > allow just the implementation of A.v, not of A' . w > > One could create another "matrix free" matrix B which could make the role > of computing z = B . w = A' . w. But how could one force the routine > MatMultTranspose(A, w, z) to call the routine for B? Here I am assuming > that MatMultTranspose(A, w, z) is the routine that SLEPc calls internally > in its algorithms when the user sets for implicit transpose. > > I see two possibilities (for brain storming, since I don't know if such > approaches would be acceptable for the PETSc team and/or the SLEPc team): > 1) PETSc could add an entry for a second routine that computes [ A' . w ] > when calling MatMFFDSetFunction() > MatMFFD is designed for providing the action of the jacobian associated with a nonlinear problem F(x)=0. It uses F to provide a finite difference approx of the action J w. It does not support actions J? w. You will need to create a MatShell and supply the methods for MatMult and MatMultTranspose (as Jose suggested). Thanks Dave 2) SLEPc could add an entry for the routine for computing [ A' . w ] when > the matrix A is "matrix free" > > Thanks again, > > Ernesto. > > > Schlumberger-Private > > -----Original Message----- > From: Jose E. Roman [mailto:jroman at dsic.upv.es] > Sent: Thursday, June 25, 2020 1:01 AM > To: Ernesto Prudencio > Cc: petsc-users at mcs.anl.gov > Subject: [Ext] Re: [petsc-users] Question on SLEPc + computing SVD with a > "matrix free" matrix > > Yes, you have to set it with SVDSetImplicitTranspose(), but then the > matrix-free matrix should implement both "A.v" and "A'.v" operations. > Jose > > > > El 24 jun 2020, a las 23:25, Ernesto Prudencio via petsc-users < > petsc-users at mcs.anl.gov> escribi?: > > > > Hi, > > > > Is it possible to compute a SVD for a ?matrix free? matrix? > > > > At first, it seems it would be ok with a MatCreateMFFD() and > MatMFFDSetFunction(), because one could then provide the routine that > computes ?A . v? for any given v, which is an operation needed by SLEPc. > However, one would also need to set up the SVD object in SLEPc with an > implicit transpose. Would that be possible? > > > > Thanks in advance, > > > > Ernesto. > > > > Schlumberger-Private > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Mark.Olesen at esi-group.com Thu Jun 25 04:53:20 2020 From: Mark.Olesen at esi-group.com (Mark Olesen) Date: Thu, 25 Jun 2020 11:53:20 +0200 Subject: [petsc-users] Problem coupling Petsc into OpenFOAM In-Reply-To: References: <56797615-51EA-4C56-AEB4-F6FEC1949348@anl.gov> <673e9f0f-499c-07e3-280f-63ec02f08810@esi-group.com> <12E2C6D2-8152-42E8-9286-C190FD30AC5D@mcs.anl.gov> Message-ID: Late followup, but people may want to know that a PETSc/OpenFOAM interface is included in the pending OpenFOAM-v2006 release as a git submodule: https://develop.openfoam.com/modules/external-solver Cheers, /mark On 2019-05-24 12:06, Matthew Knepley via petsc-users wrote: > On Thu, May 23, 2019 at 10:41 PM Vu Q. Do via petsc-users > > wrote: > > ?Hi all, > > Thanks for your previous suggestion, I have been able to > successfully link Petsc to OpenFOAM. I have written a simple > interface and it works quite well in serial mode, but cannot run in > parallel. I have been thinking about this problem for weeks but > couldn't solve it. So I think maybe you could give me some idea. I > describe my problem below. > > My interface is just a class named "petscSolver/"/, which is used to > convert an openfoam's matrix or blocked matrix to Petsc Mat, then > solve the matrix using Petsc's solver. > To use Petsc, an Openfoam's solver need to be recompiled after > adding the following lines to make file: > > EXE_INC = \ > ... > ? -I$(LIB_SRC)/petscSolver \ > ? -I$(PETSC_DIR)/include \ > ? -I$(PETSC_DIR)/$(PETSC_ARCH)/include > EXE_LIBS = \ > ... > ? -L$(PETSC_DIR)/$(PETSC_ARCH)/lib -lpetsc > > To run an openfoam's case in parallel, first I need to discretize > the domain into subdomains (e.g. 2 subdomains ), then use the > following command: > > mpirun -np 2 mySolver -parallel > > (where mpirun is literally mpiexec) The simulation crashed even > before doing anything and the error message is as in the attached image. > > > Run in the debugger and see where it is crashing. Its possible to screw > up the MPI linking here, so that you > link OPENFOAM with one MPI and PETSc with another, or you call MPIInit() > after you call PetscInitialize(), etc. > > ? Thanks, > > ? ? Matt > > I have tested and realized that the solver can run in parallel as > normal by removing the two lines: > ? -I$(PETSC_DIR)/include \ > ? -I$(PETSC_DIR)/$(PETSC_ARCH)/include > But then it is clearly no longer linked to Petsc. > > I would appreciate any suggestion. > > Screenshot from 2019-05-24 09-22-17.png > > > On Thu, Apr 11, 2019 at 1:37 PM Smith, Barry F. > wrote: > > > ? Mark, > > ? ? Thanks for the clarifying email. My google searches didn't > locate the rheoTool you mention nor "a PRACE project running via > CINECA (Bologna)". > > ? ? It would be nice if someday OpenFOAM had (either directly > or somehow with the modules directory) an interface to the PETSc > solvers. This would allow the use of a variety of other solvers > including hypre BoomerAMG, SuperLU_Dist, MUMPS, and even the use > of PETSc/ViennaCL GPU based solvers automatically from OpenFOAM. > Unfortunately the PETSc group doesn't have the resources or > expertise to develop and support such an interface ourselves. We > would, of course, try to answer emails about PETSc usage and > bugs for such an interface. > > ? ?Barry > > ? If OpenFOAM did have such an interface one thing we could > provide is the CI infrastructure for tracking changes to PETSc > that may effect OpenFOAM. For example we could automatically > build OpenFOAM each day with the latest master of PETSc thus > immediately detecting changes that effect the interface. > > > > > > On Apr 10, 2019, at 4:55 PM, Mark Olesen > > > wrote: > > > > The paper that Barry mentioned gives some generalities, but > probably > > won't help much. There are some PETSc/OpenFOAM interfaces in > rheoTool > > that are probably much more helpful. > > > > As Barry also rightly noted, there are some config files in > the OpenFOAM > > tree that were put in some time ago for helping with setting > up PETSc > > and OpenFOAM. Assuming that you have set the appropriate > values in the > > etc/config.sh/petsc file you will be > able to use those when using wmake. > > For running you will still need to ensure that the > LD_LIBARY_PATH is set > > correctly. For example, what some build scripts exhibit: > > > > wmake(petsc)? : > > ==> Before running, verify that PETSc libraries can be found > > > > Enable in the OpenFOAM etc/bashrc, define manually or try > with the > > following (POSIX shell): > > > >? ? ?eval $(foamEtcFile -sh -config petsc -- -force) > > > > == > > > > > > There is currently a PRACE project running via CINECA > (Bologna) with > > binding in PETSc as a runtime selectable linear solver in > OpenFOAM. This > > is still at the stage of early testing and performance > benchmarking. > > > > Cheers, > > /mark > > > > On 4/10/19 6:37 PM, Smith, Barry F. via petsc-users wrote: > >> > >>? ?We don't know much about OpenFoam but > >> > >> 1)? if I do a > >> > >>? ?git grep -i petsc > >> > >>? ? ?in the > https://develop.openfoam.com/Development/OpenFOAM-plus.git > repository I see various configuration files specifically for PETSc. > >> > >> etc/config.csh/petsc? etc/config.sh/petsc > wmake/scripts/have_petsc > >> > >>? ? ?so it appears that OpenFOAM has the tools to be linked > against PETSc (to me the documentation on how to use them is > rather terse). Are > >>? ? ?you using these?? If you have trouble with them perhaps > you can ask the OpenFOAM user community how to use them. > >> > >> > >>? ?2)? if you are editing the Make/options file directly you > can try changing > >> > >>? ? ? -L$(PETSC_DIR)/$(PETSC_ARCH)/lib -lpetsc > >> > >>? ? ? to > >> > >>? ? ? -Wl,-rpath,$(PETSC_DIR)/$(PETSC_ARCH)/lib > ?-L$(PETSC_DIR)/$(PETSC_ARCH)/lib -lpetsc > >> > >> > >> > >>? ? Note also that simply including petsc.h into the OpenFoam > source code and linking against -lpetsc will not immediately > allow calling the PETSc solvers from OpenFOAM. One needs to > write all the interface code that sets up and calls the PETSc > solvers from OpenFOAM. There is a paper > https://www.researchgate.net/publication/319045499_Insertion_of_PETSc_in_the_OpenFOAM_Framework > that describes at an abstract level how they wrote code that > calls the PETSc solvers from OpenFOAM but the source code that > actually does the work does not appear to be available. > >> > >>? ? Note that PETSc is now at version 3.11 we recommend > working with that version (unless you already have a lot of code > that calls PETSc written with a previous version of PETSc, for > that we recommend first upgrading to petsc 3.11 and then > continuing to add code). > >> > >>? ? Barry > >> > >> > >> > >> > >> > >>> On Apr 10, 2019, at 8:23 AM, Balay, Satish via petsc-users > > wrote: > >>> > >>> Runtime error? You might have to add the path to > $PETSC_ARCH/lib in? LD_LIBRARY_PATH env variable > >>> or - to your link command. If linux/gcc - the linker option > is -Wl,-rpath,$PETSC_ARCH/lib > >>> > >>> If not - send detail logs. > >>> > >>> Satish > >>> > >>> On Wed, 10 Apr 2019, Vu Do Quoc via petsc-users wrote: > >>> > >>>> Hi all, > >>>> > >>>> I am trying to insert Petsc to OpenFOAM opensource software. > >>>> I have been successfully compiling Petsc with an available > solver in > >>>> OpenFOAM by linking it with the shared library > libpetsc.so. However, when I > >>>> call the solver to run a test case, I got an error saying > that: > >>>> "libpetsc.so cannot be found", even though the library > still exists in the > >>>> $PETSC_ARCH/lib folder. > >>>> > >>>> I have been struggling for weeks but still, have not been > able to figure it > >>>> out. Therefore I would be very grateful for any suggestion > to solve this > >>>> problem. > >>>> > >>>> Thanks in advance for your time, > >>>> > >>>> Best regards, > >>>> > >>>> Vu Do > >>>> > > > > -- > *Vu Q. Do > *------------------------------------------------ > /Student of Aeronautical Engineering/ > Programme de Formation d'Ing?nieurs d'Excellence au Vietnam /- PFIEV/ > School of Transportation Engineering/ > / > Hanoi University of Science and Technology > 01 Dai Co Viet Avenue, Hanoi, Vietnam > E-mail: vu.doquochust at gmail.com > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ -- Dr Mark OLESEN Principal Engineer, ESI-OpenCFD Engineering System International GmbH | Einsteinring 24 | 85609 Munich Mob. +49 171 9710 149 www.openfoam.com | www.esi-group.com | mark.olesen at esi-group.com From EPrudencio at slb.com Thu Jun 25 09:14:26 2020 From: EPrudencio at slb.com (Ernesto Prudencio) Date: Thu, 25 Jun 2020 14:14:26 +0000 Subject: [petsc-users] [Ext] Re: Question on SLEPc + computing SVD with a "matrix free" matrix In-Reply-To: References: Message-ID: Thanks, Dave. Ernesto. From: Dave May [mailto:dave.mayhem23 at gmail.com] Sent: Thursday, June 25, 2020 2:12 AM To: Ernesto Prudencio Cc: Jose E. Roman ; petsc-users at mcs.anl.gov Subject: Re: [petsc-users] [Ext] Re: Question on SLEPc + computing SVD with a "matrix free" matrix On Thu 25. Jun 2020 at 08:23, Ernesto Prudencio via petsc-users > wrote: Thank you, Jose. However, in the case of a "matrix free" matrix, the APIs on PETSc seem to allow just the implementation of A.v, not of A' . w One could create another "matrix free" matrix B which could make the role of computing z = B . w = A' . w. But how could one force the routine MatMultTranspose(A, w, z) to call the routine for B? Here I am assuming that MatMultTranspose(A, w, z) is the routine that SLEPc calls internally in its algorithms when the user sets for implicit transpose. I see two possibilities (for brain storming, since I don't know if such approaches would be acceptable for the PETSc team and/or the SLEPc team): 1) PETSc could add an entry for a second routine that computes [ A' . w ] when calling MatMFFDSetFunction() MatMFFD is designed for providing the action of the jacobian associated with a nonlinear problem F(x)=0. It uses F to provide a finite difference approx of the action J w. It does not support actions J' w. You will need to create a MatShell and supply the methods for MatMult and MatMultTranspose (as Jose suggested). Thanks Dave 2) SLEPc could add an entry for the routine for computing [ A' . w ] when the matrix A is "matrix free" Thanks again, Ernesto. Schlumberger-Private -----Original Message----- From: Jose E. Roman [mailto:jroman at dsic.upv.es] Sent: Thursday, June 25, 2020 1:01 AM To: Ernesto Prudencio > Cc: petsc-users at mcs.anl.gov Subject: [Ext] Re: [petsc-users] Question on SLEPc + computing SVD with a "matrix free" matrix Yes, you have to set it with SVDSetImplicitTranspose(), but then the matrix-free matrix should implement both "A.v" and "A'.v" operations. Jose > El 24 jun 2020, a las 23:25, Ernesto Prudencio via petsc-users > escribi?: > > Hi, > > Is it possible to compute a SVD for a "matrix free" matrix? > > At first, it seems it would be ok with a MatCreateMFFD() and MatMFFDSetFunction(), because one could then provide the routine that computes "A . v" for any given v, which is an operation needed by SLEPc. However, one would also need to set up the SVD object in SLEPc with an implicit transpose. Would that be possible? > > Thanks in advance, > > Ernesto. > > Schlumberger-Private Schlumberger-Private -------------- next part -------------- An HTML attachment was scrubbed... URL: From eda.oktay at metu.edu.tr Thu Jun 25 10:09:48 2020 From: eda.oktay at metu.edu.tr (Eda Oktay) Date: Thu, 25 Jun 2020 18:09:48 +0300 Subject: [petsc-users] Gather and Broadcast Parallel Vectors in k-means algorithm In-Reply-To: <738e615c-de8e-88cd-81cd-4d6f35b454d2@anl.gov> References: <0c10fc0a-3d86-e91a-f349-fd7c087ba8ed@anl.gov> <738e615c-de8e-88cd-81cd-4d6f35b454d2@anl.gov> Message-ID: Dear Richard, >From now on, I am actually trying to write a parallel k-means algorithm by using petsc routines (I don't have to use petsc but I believe it will be easier) and I used the algorithm you mentioned before about finding cluster centroids. However, something is bothering me: You said that by using MPI_Allreduce, I should "calculate the global element wise sum of all the local sums and finally divide by the number of members of that cluster to get the centroid." But if I do it this way, I think each cluster centroid will be the same, right? But when I read the k-means algorithm, I thought that each cluster centroid should be different since elements in each cluster are different. Can you enlighten me? Thanks! Eda Mills, Richard Tran , 30 Nis 2020 Per, 02:07 tarihinde ?unu yazd?: > > Hi Eda, > > Thanks for your reply. I'm still trying to understand why you say you need to duplicate the row vectors across all processes. When I have implemented parallel k-means, I don't duplicate the row vectors. (This would be very unscalable and largely defeat the point of doing this with MPI parallelism in the first place.) > > Earlier in this email thread, you said that you have used Matlab to get cluster IDs for each row vector. Are you trying to then use this information to calculate the cluster centroids from inside your PETSc program? If so, you can do this by having each MPI rank do the following: For cluster i in 0 to (k-1), calculate the element-wise sum of all of the local rows that belong to cluster i, then use MPI_Allreduce() to calculate the global elementwise sum of all the local sums (this array will be replicated across all MPI ranks), and finally divide by the number of members of that cluster to get the centroid. Note that MPI_Allreduce() doesn't work on PETSc objects, but simple arrays, so you'll want to use something like MatGetValues() or MatGetRow() to access the elements of your row vectors. > > Let me know if I am misunderstanding what you are aiming to do, or if I am misunderstanding something. > > It sounds like you would benefit from having some routines in PETSc to do k-means (or other) clustering, by the way? > > Best regards, > Richard > > On 4/29/20 3:47 AM, Eda Oktay wrote: > > Dear Richard, > > I am trying to use spectral clustering algorithm by using k-means clustering algorithm at some point. I am doing this by producing a matrix consisting of eigenvectors (of the adjacency matrix of the graph that I want to partition), then forming row vectors of this matrix. This is the part that I am using parallel vector. By using the output from k-means, I am trying to cluster these row vectors. To cluster these vectors, I think I need all row vectors in all processes. I wanted to use sequential vectors, however, I couldn't find a different way that I form row vectors of a matrix. > > I am trying to use VecScatterCreateToAll, however, since my vector is parallel crated by VecDuplicateVecs, my input is not in correct type, so I get error. I still can't get how can I use this function in parallel vector created by VecDuplicateVecs. > > Thank you all for your help. > > Eda > > Mills, Richard Tran , 7 Nis 2020 Sal, 01:51 tarihinde ?unu yazd?: >> >> Hi Eda, >> >> I think that you probably want to use VecScatter routines, as Junchao >> has suggested, instead of the lower level star forest for this. I >> believe that VecScatterCreateToZero() is what you want for the broadcast >> problem you describe, in the second part of your question. I'm not sure >> what you are trying to do in the first part. Taking a parallel vector >> and then copying its entire contents to a sequential vector residing on >> each process is not scalable, and a lot of the design that has gone into >> PETSc is to prevent the user from ever needing to do things like that. >> Can you please tell us what you intend to do with these sequential vectors? >> >> I'm also wondering why, later in your message, you say that you get >> cluster assignments from Matlab, and then "to cluster row vectors >> according to this information, all processors need to have all of the >> row vectors". Do you mean you want to get all of the row vectors copied >> onto all of the processors so that you can compute the cluster >> centroids? If so, computing the cluster centroids can be done without >> copying the row vectors onto all processors if you use a communication >> operation like MPI_Allreduce(). >> >> Lastly, let me add that I've done a fair amount of work implementing >> clustering algorithms on distributed memory parallel machines, but >> outside of PETSc. I was thinking that I should implement some of these >> routines using PETSc. I can't get to this immediately, but I'm wondering >> if you might care to tell me a bit more about the clustering problems >> you need to solve and how having some support for this in PETSc might >> (or might not) help. >> >> Best regards, >> Richard >> >> On 4/4/20 1:39 AM, Eda Oktay wrote: >> > Hi all, >> > >> > I created a parallel vector UV, by using VecDuplicateVecs since I need >> > row vectors of a matrix. However, I need the whole vector be in all >> > processors, which means I need to gather all and broadcast them to all >> > processors. To gather, I tried to use VecStrideGatherAll: >> > >> > Vec UVG; >> > VecStrideGatherAll(UV,UVG,INSERT_VALUES); >> > VecView(UVG,PETSC_VIEWER_STDOUT_WORLD); >> > >> > however when I try to view the vector, I get the following error. >> > >> > [3]PETSC ERROR: Invalid argument >> > [3]PETSC ERROR: Wrong type of object: Parameter # 1 >> > [3]PETSC ERROR: See >> > http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. >> > [3]PETSC ERROR: Petsc Release Version 3.11.1, Apr, 12, 2019 >> > [3]PETSC ERROR: ./clustering_son_final_edgecut_without_parmetis on a >> > arch-linux2-c-debug named localhost.localdomain by edaoktay Sat Apr 4 >> > 11:22:54 2020 >> > [3]PETSC ERROR: Wrong type of object: Parameter # 1 >> > [0]PETSC ERROR: See >> > http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. >> > [0]PETSC ERROR: Petsc Release Version 3.11.1, Apr, 12, 2019 >> > [0]PETSC ERROR: ./clustering_son_final_edgecut_without_parmetis on a >> > arch-linux2-c-debug named localhost.localdomain by edaoktay Sat Apr 4 >> > 11:22:54 2020 >> > [0]PETSC ERROR: Configure options --download-mpich --download-openblas >> > --download-slepc --download-metis --download-parmetis --download-chaco >> > --with-X=1 >> > [0]PETSC ERROR: #1 VecStrideGatherAll() line 646 in >> > /home/edaoktay/petsc-3.11.1/src/vec/vec/utils/vinv.c >> > ./clustering_son_final_edgecut_without_parmetis on a >> > arch-linux2-c-debug named localhost.localdomain by edaoktay Sat Apr 4 >> > 11:22:54 2020 >> > [1]PETSC ERROR: Configure options --download-mpich --download-openblas >> > --download-slepc --download-metis --download-parmetis --download-chaco >> > --with-X=1 >> > [1]PETSC ERROR: #1 VecStrideGatherAll() line 646 in >> > /home/edaoktay/petsc-3.11.1/src/vec/vec/utils/vinv.c >> > Configure options --download-mpich --download-openblas >> > --download-slepc --download-metis --download-parmetis --download-chaco >> > --with-X=1 >> > [3]PETSC ERROR: #1 VecStrideGatherAll() line 646 in >> > /home/edaoktay/petsc-3.11.1/src/vec/vec/utils/vinv.c >> > >> > I couldn't understand why I am getting this error. Is this because of >> > UV being created by VecDuplicateVecs? How can I solve this problem? >> > >> > The other question is broadcasting. After gathering all elements of >> > the vector UV, I need to broadcast them to all processors. I found >> > PetscSFBcastBegin. However, I couldn't understand the PetscSF concept >> > properly. I couldn't adjust my question to the star forest concept. >> > >> > My problem is: If I have 4 processors, I create a matrix whose columns >> > are 4 smallest eigenvectors, say of size 72. Then by defining each row >> > of this matrix as a vector, I cluster them by using k-means >> > clustering algorithm. For now, I cluster them by using MATLAB and I >> > obtain a vector showing which row vector is in which cluster. After >> > getting this vector, to cluster row vectors according to this >> > information, all processors need to have all of the row vectors. >> > >> > According to this problem, how can I use the star forest concept? >> > >> > I will be glad if you can help me about this problem since I don't >> > have enough knowledge about graph theory. An if you have any idea >> > about how can I use k-means algorithm in a more practical way, please >> > let me know. >> > >> > Thanks! >> > >> > Eda > > From knepley at gmail.com Thu Jun 25 10:12:39 2020 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 25 Jun 2020 11:12:39 -0400 Subject: [petsc-users] Gather and Broadcast Parallel Vectors in k-means algorithm In-Reply-To: References: <0c10fc0a-3d86-e91a-f349-fd7c087ba8ed@anl.gov> <738e615c-de8e-88cd-81cd-4d6f35b454d2@anl.gov> Message-ID: On Thu, Jun 25, 2020 at 11:10 AM Eda Oktay wrote: > Dear Richard, > > From now on, I am actually trying to write a parallel k-means > algorithm by using petsc routines (I don't have to use petsc but I > believe it will be easier) and I used the algorithm you mentioned > before about finding cluster centroids. However, something is > bothering me: > > You said that by using MPI_Allreduce, I should "calculate the global > element wise sum of all the local sums and finally divide by the > number of members of that cluster to get the centroid." > > But if I do it this way, I think each cluster centroid will be the > same, right? But when I read the k-means algorithm, I thought that > each cluster centroid should be different since elements in each > cluster are different. > > Can you enlighten me? > I believe Richard means here to calculate the global sum (meaning sum across processes) of the local sums _for that cluster_, since cluster members could appear on more than 1 process. Thanks, Matt > Thanks! > > Eda > > Mills, Richard Tran , 30 Nis 2020 Per, 02:07 > tarihinde ?unu yazd?: > > > > Hi Eda, > > > > Thanks for your reply. I'm still trying to understand why you say you > need to duplicate the row vectors across all processes. When I have > implemented parallel k-means, I don't duplicate the row vectors. (This > would be very unscalable and largely defeat the point of doing this with > MPI parallelism in the first place.) > > > > Earlier in this email thread, you said that you have used Matlab to get > cluster IDs for each row vector. Are you trying to then use this > information to calculate the cluster centroids from inside your PETSc > program? If so, you can do this by having each MPI rank do the following: > For cluster i in 0 to (k-1), calculate the element-wise sum of all of the > local rows that belong to cluster i, then use MPI_Allreduce() to calculate > the global elementwise sum of all the local sums (this array will be > replicated across all MPI ranks), and finally divide by the number of > members of that cluster to get the centroid. Note that MPI_Allreduce() > doesn't work on PETSc objects, but simple arrays, so you'll want to use > something like MatGetValues() or MatGetRow() to access the elements of your > row vectors. > > > > Let me know if I am misunderstanding what you are aiming to do, or if I > am misunderstanding something. > > > > It sounds like you would benefit from having some routines in PETSc to > do k-means (or other) clustering, by the way? > > > > Best regards, > > Richard > > > > On 4/29/20 3:47 AM, Eda Oktay wrote: > > > > Dear Richard, > > > > I am trying to use spectral clustering algorithm by using k-means > clustering algorithm at some point. I am doing this by producing a matrix > consisting of eigenvectors (of the adjacency matrix of the graph that I > want to partition), then forming row vectors of this matrix. This is the > part that I am using parallel vector. By using the output from k-means, I > am trying to cluster these row vectors. To cluster these vectors, I think I > need all row vectors in all processes. I wanted to use sequential vectors, > however, I couldn't find a different way that I form row vectors of a > matrix. > > > > I am trying to use VecScatterCreateToAll, however, since my vector is > parallel crated by VecDuplicateVecs, my input is not in correct type, so I > get error. I still can't get how can I use this function in parallel vector > created by VecDuplicateVecs. > > > > Thank you all for your help. > > > > Eda > > > > Mills, Richard Tran , 7 Nis 2020 Sal, 01:51 tarihinde > ?unu yazd?: > >> > >> Hi Eda, > >> > >> I think that you probably want to use VecScatter routines, as Junchao > >> has suggested, instead of the lower level star forest for this. I > >> believe that VecScatterCreateToZero() is what you want for the broadcast > >> problem you describe, in the second part of your question. I'm not sure > >> what you are trying to do in the first part. Taking a parallel vector > >> and then copying its entire contents to a sequential vector residing on > >> each process is not scalable, and a lot of the design that has gone into > >> PETSc is to prevent the user from ever needing to do things like that. > >> Can you please tell us what you intend to do with these sequential > vectors? > >> > >> I'm also wondering why, later in your message, you say that you get > >> cluster assignments from Matlab, and then "to cluster row vectors > >> according to this information, all processors need to have all of the > >> row vectors". Do you mean you want to get all of the row vectors copied > >> onto all of the processors so that you can compute the cluster > >> centroids? If so, computing the cluster centroids can be done without > >> copying the row vectors onto all processors if you use a communication > >> operation like MPI_Allreduce(). > >> > >> Lastly, let me add that I've done a fair amount of work implementing > >> clustering algorithms on distributed memory parallel machines, but > >> outside of PETSc. I was thinking that I should implement some of these > >> routines using PETSc. I can't get to this immediately, but I'm wondering > >> if you might care to tell me a bit more about the clustering problems > >> you need to solve and how having some support for this in PETSc might > >> (or might not) help. > >> > >> Best regards, > >> Richard > >> > >> On 4/4/20 1:39 AM, Eda Oktay wrote: > >> > Hi all, > >> > > >> > I created a parallel vector UV, by using VecDuplicateVecs since I need > >> > row vectors of a matrix. However, I need the whole vector be in all > >> > processors, which means I need to gather all and broadcast them to all > >> > processors. To gather, I tried to use VecStrideGatherAll: > >> > > >> > Vec UVG; > >> > VecStrideGatherAll(UV,UVG,INSERT_VALUES); > >> > VecView(UVG,PETSC_VIEWER_STDOUT_WORLD); > >> > > >> > however when I try to view the vector, I get the following error. > >> > > >> > [3]PETSC ERROR: Invalid argument > >> > [3]PETSC ERROR: Wrong type of object: Parameter # 1 > >> > [3]PETSC ERROR: See > >> > http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble > shooting. > >> > [3]PETSC ERROR: Petsc Release Version 3.11.1, Apr, 12, 2019 > >> > [3]PETSC ERROR: ./clustering_son_final_edgecut_without_parmetis on a > >> > arch-linux2-c-debug named localhost.localdomain by edaoktay Sat Apr 4 > >> > 11:22:54 2020 > >> > [3]PETSC ERROR: Wrong type of object: Parameter # 1 > >> > [0]PETSC ERROR: See > >> > http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble > shooting. > >> > [0]PETSC ERROR: Petsc Release Version 3.11.1, Apr, 12, 2019 > >> > [0]PETSC ERROR: ./clustering_son_final_edgecut_without_parmetis on a > >> > arch-linux2-c-debug named localhost.localdomain by edaoktay Sat Apr 4 > >> > 11:22:54 2020 > >> > [0]PETSC ERROR: Configure options --download-mpich --download-openblas > >> > --download-slepc --download-metis --download-parmetis --download-chaco > >> > --with-X=1 > >> > [0]PETSC ERROR: #1 VecStrideGatherAll() line 646 in > >> > /home/edaoktay/petsc-3.11.1/src/vec/vec/utils/vinv.c > >> > ./clustering_son_final_edgecut_without_parmetis on a > >> > arch-linux2-c-debug named localhost.localdomain by edaoktay Sat Apr 4 > >> > 11:22:54 2020 > >> > [1]PETSC ERROR: Configure options --download-mpich --download-openblas > >> > --download-slepc --download-metis --download-parmetis --download-chaco > >> > --with-X=1 > >> > [1]PETSC ERROR: #1 VecStrideGatherAll() line 646 in > >> > /home/edaoktay/petsc-3.11.1/src/vec/vec/utils/vinv.c > >> > Configure options --download-mpich --download-openblas > >> > --download-slepc --download-metis --download-parmetis --download-chaco > >> > --with-X=1 > >> > [3]PETSC ERROR: #1 VecStrideGatherAll() line 646 in > >> > /home/edaoktay/petsc-3.11.1/src/vec/vec/utils/vinv.c > >> > > >> > I couldn't understand why I am getting this error. Is this because of > >> > UV being created by VecDuplicateVecs? How can I solve this problem? > >> > > >> > The other question is broadcasting. After gathering all elements of > >> > the vector UV, I need to broadcast them to all processors. I found > >> > PetscSFBcastBegin. However, I couldn't understand the PetscSF concept > >> > properly. I couldn't adjust my question to the star forest concept. > >> > > >> > My problem is: If I have 4 processors, I create a matrix whose columns > >> > are 4 smallest eigenvectors, say of size 72. Then by defining each row > >> > of this matrix as a vector, I cluster them by using k-means > >> > clustering algorithm. For now, I cluster them by using MATLAB and I > >> > obtain a vector showing which row vector is in which cluster. After > >> > getting this vector, to cluster row vectors according to this > >> > information, all processors need to have all of the row vectors. > >> > > >> > According to this problem, how can I use the star forest concept? > >> > > >> > I will be glad if you can help me about this problem since I don't > >> > have enough knowledge about graph theory. An if you have any idea > >> > about how can I use k-means algorithm in a more practical way, please > >> > let me know. > >> > > >> > Thanks! > >> > > >> > Eda > > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From sam.guo at cd-adapco.com Thu Jun 25 14:18:05 2020 From: sam.guo at cd-adapco.com (Sam Guo) Date: Thu, 25 Jun 2020 12:18:05 -0700 Subject: [petsc-users] PETSc initialization error In-Reply-To: References: <004EA4EB-815F-4EEE-8988-CA2179304157@petsc.dev> Message-ID: Hi Junchao, I now encountered the same error with parallel. I am wondering if there is a need for parallel fix as well. [1]PETSC ERROR: #1 PetscInitialize() line 969 in ../../../petsc/src/sys/objects/pinit.c PETSC ERROR: Logging has not been enabled. You might have forgotten to call PetscInitialize(). PETSC ERROR: Logging has not been enabled. You might have forgotten to call PetscInitialize(). On Sat, Jun 20, 2020 at 7:35 PM Sam Guo wrote: > Hi Junchao, > Your patch works. > > Thanks, > Sam > > On Sat, Jun 20, 2020 at 4:23 PM Junchao Zhang > wrote: > >> >> >> On Sat, Jun 20, 2020 at 12:24 PM Barry Smith wrote: >> >>> >>> Junchao, >>> >>> This is a good bug fix. It solves the problem when PETSc initialize >>> is called many times. >>> >>> There is another fix you can do to limit PETSc mpiuni running out >>> of attributes inside a single PETSc run: >>> >>> >>> int MPI_Comm_create_keyval(MPI_Copy_function >>> *copy_fn,MPI_Delete_function *delete_fn,int *keyval,void *extra_state) >>> { >>> >>> if (num_attr >= MAX_ATTR){ >>> for (i=0; i>> if (!attr_keyval[i].extra_state) { >>> >> attr_keyval[i].extra_state is provided by user (could be NULL). We can >> not rely on it. >> >>> /* reuse this slot */ >>> attr_keyval[i].extra_state = extra_state; >>> attr_keyval[i.]del = delete_fn; >>> *keyval = i; >>> return MPI_SUCCESS; >>> } >>> } >>> return MPIUni_Abort(MPI_COMM_WORLD,1); >>> } >>> return MPIUni_Abort(MPI_COMM_WORLD,1); >>> attr_keyval[num_attr].extra_state = extra_state; >>> attr_keyval[num_attr].del = delete_fn; >>> *keyval = num_attr++; >>> return MPI_SUCCESS; >>> } >>> >>> This will work if the user creates tons of attributes but is >>> constantly deleting some as they new ones. So long as the number >>> outstanding at one time is < MAX_ATTR) >>> >>> Barry >>> >>> >>> >>> >>> >>> On Jun 20, 2020, at 10:54 AM, Junchao Zhang >>> wrote: >>> >>> I don't understand what your session means. Let's try this patch >>> >>> diff --git a/src/sys/mpiuni/mpi.c b/src/sys/mpiuni/mpi.c >>> index d559a513..c058265d 100644 >>> --- a/src/sys/mpiuni/mpi.c >>> +++ b/src/sys/mpiuni/mpi.c >>> @@ -283,6 +283,7 @@ int MPI_Finalize(void) >>> MPI_Comm_free(&comm); >>> comm = MPI_COMM_SELF; >>> MPI_Comm_free(&comm); >>> + num_attr = 1; /* reset the counter */ >>> MPI_was_finalized = 1; >>> return MPI_SUCCESS; >>> } >>> >>> >>> --Junchao Zhang >>> >>> >>> On Sat, Jun 20, 2020 at 10:48 AM Sam Guo wrote: >>> >>>> Typo: I mean ?Assuming initializer is only needed once for entire >>>> session? >>>> >>>> On Saturday, June 20, 2020, Sam Guo wrote: >>>> >>>>> Assuming finalizer is only needed once for entire session(?), I can >>>>> put initializer into the static block to call it once but where do I call >>>>> finalizer? >>>>> >>>>> >>>>> On Saturday, June 20, 2020, Junchao Zhang >>>>> wrote: >>>>> >>>>>> The counter num_attr should be recycled. But first try to call PETSc >>>>>> initialize/Finalize only once to see it fixes the error. >>>>>> --Junchao Zhang >>>>>> >>>>>> >>>>>> On Sat, Jun 20, 2020 at 12:48 AM Sam Guo >>>>>> wrote: >>>>>> >>>>>>> To clarify, I call PETSc initialize and PETSc finalize everytime I >>>>>>> call SLEPc: >>>>>>> >>>>>>> PetscInitializeNoPointers(argc,args,nullptr,nullptr); >>>>>>> >>>>>>> SlepcInitialize(&argc,&args,static_cast(nullptr),help); >>>>>>> >>>>>>> //calling slepc >>>>>>> >>>>>>> SlepcFinalize(); >>>>>>> >>>>>>> PetscFinalize(); >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Fri, Jun 19, 2020 at 10:32 PM Sam Guo >>>>>>> wrote: >>>>>>> >>>>>>>> Dear PETSc team, >>>>>>>> When I called SLEPc multiple time, I eventually got following >>>>>>>> error: >>>>>>>> >>>>>>>> MPI operation not supported by PETSc's sequential MPI wrappers >>>>>>>> [0]PETSC ERROR: #1 PetscInitialize() line 967 in >>>>>>>> ../../../petsc/src/sys/objects/pinit.c >>>>>>>> [0]PETSC ERROR: #2 SlepcInitialize() line 262 in >>>>>>>> ../../../slepc/src/sys/slepcinit.c >>>>>>>> [0]PETSC ERROR: #3 SlepcInitializeNoPointers() line 359 in >>>>>>>> ../../../slepc/src/sys/slepcinit.c >>>>>>>> PETSC ERROR: Logging has not been enabled. >>>>>>>> You might have forgotten to call PetscInitialize(). >>>>>>>> >>>>>>>> I debugged: it is because of following in >>>>>>>> petsc/src/sys/mpiuni/mpi.c >>>>>>>> >>>>>>>> if (num_attr >= MAX_ATTR) >>>>>>>> >>>>>>>> in function int MPI_Comm_create_keyval(MPI_Copy_function >>>>>>>> *copy_fn,MPI_Delete_function *delete_fn,int *keyval,void *extra_state) >>>>>>>> >>>>>>>> num_attr is declared static and keeps increasing every >>>>>>>> time MPI_Comm_create_keyval is called. >>>>>>>> >>>>>>>> I am using petsc 3.11.3 but found 3.13.2 has the same logic. >>>>>>>> >>>>>>>> Is this a bug or I didn't use it correctly? >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Sam >>>>>>>> >>>>>>> >>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From rhalder at umich.edu Thu Jun 25 14:43:17 2020 From: rhalder at umich.edu (Rakesh Halder) Date: Thu, 25 Jun 2020 15:43:17 -0400 Subject: [petsc-users] Inaccurate Matrix Inversion using MatMatSolve Message-ID: Hi all, I'm using PETSc for matrix calculations as part of a model order reduction code. An algorithm I'm using requires that I compute the explicit inverse of a matrix, as it needs to be used in matrix-matrix products. The matrix is small (I'll show a 5x5 example), and I first find the LU factorization of it as follows: MatGetOrdering(A,MATORDERINGNATURAL,&perm,&iperm); MatFactorInfoInitialize(&info); MatLUFactor(A,perm,iperm,&info); I then find the inverse of it: MatMatSolve(A,I,B); Where I is the identity matrix, and B is the inverse. The subroutine does calculate an inverse, although not accurately; the matrix A I looked at is: 5.23E-02 1.86E-02 2.67E-02 4.58E-02 3.55E-02 6.37E-03 5.86E-02 5.07E-03 1.64E-02 1.36E-02 -4.07E-02 3.99E-03 5.50E-02 1.77E-02 -3.21E-02 1.96E-02 -5.53E-03 4.02E-02 -5.37E-02 1.80E-02 1.51E-02 1.70E-02 -1.57E-02 -3.75E-03 -5.64E-02 And when I take the product A*B, I don't recover the identity matrix, but rather: 9.97E-01 2.77E-04 -1.66E-03 1.25E-04 -1.10E-03 -6.79E-04 9.99E-01 -1.72E-04 6.24E-04 1.16E-03 2.70E-03 -3.98E-04 9.98E-01 -1.51E-04 6.82E-04 -3.35E-04 -4.82E-04 -1.03E-03 9.99E-01 9.80E-05 -9.50E-04 -8.70E-04 8.37E-04 -3.68E-04 9.99E-01 I believe this is causing large inaccuracies in my program, as the diagonal and off-diagonal entries have large errors associated with them. I am wondering if there is a way to perhaps tighten the tolerance of the subroutine, or if there are other methods I can use. I would like to avoid using KSP and solving for the inverse vector-by-vector if possible. Thanks, Rakesh Halder -------------- next part -------------- An HTML attachment was scrubbed... URL: From hzhang at mcs.anl.gov Thu Jun 25 15:03:41 2020 From: hzhang at mcs.anl.gov (Zhang, Hong) Date: Thu, 25 Jun 2020 20:03:41 +0000 Subject: [petsc-users] Inaccurate Matrix Inversion using MatMatSolve In-Reply-To: References: Message-ID: Rakesh, Can you find the condition number of your matrix A? Hong ________________________________ From: petsc-users on behalf of Rakesh Halder Sent: Thursday, June 25, 2020 2:43 PM To: petsc-users at mcs.anl.gov Subject: [petsc-users] Inaccurate Matrix Inversion using MatMatSolve Hi all, I'm using PETSc for matrix calculations as part of a model order reduction code. An algorithm I'm using requires that I compute the explicit inverse of a matrix, as it needs to be used in matrix-matrix products. The matrix is small (I'll show a 5x5 example), and I first find the LU factorization of it as follows: MatGetOrdering(A,MATORDERINGNATURAL,&perm,&iperm); MatFactorInfoInitialize(&info); MatLUFactor(A,perm,iperm,&info); I then find the inverse of it: MatMatSolve(A,I,B); Where I is the identity matrix, and B is the inverse. The subroutine does calculate an inverse, although not accurately; the matrix A I looked at is: 5.23E-02 1.86E-02 2.67E-02 4.58E-02 3.55E-02 6.37E-03 5.86E-02 5.07E-03 1.64E-02 1.36E-02 -4.07E-02 3.99E-03 5.50E-02 1.77E-02 -3.21E-02 1.96E-02 -5.53E-03 4.02E-02 -5.37E-02 1.80E-02 1.51E-02 1.70E-02 -1.57E-02 -3.75E-03 -5.64E-02 And when I take the product A*B, I don't recover the identity matrix, but rather: 9.97E-01 2.77E-04 -1.66E-03 1.25E-04 -1.10E-03 -6.79E-04 9.99E-01 -1.72E-04 6.24E-04 1.16E-03 2.70E-03 -3.98E-04 9.98E-01 -1.51E-04 6.82E-04 -3.35E-04 -4.82E-04 -1.03E-03 9.99E-01 9.80E-05 -9.50E-04 -8.70E-04 8.37E-04 -3.68E-04 9.99E-01 I believe this is causing large inaccuracies in my program, as the diagonal and off-diagonal entries have large errors associated with them. I am wondering if there is a way to perhaps tighten the tolerance of the subroutine, or if there are other methods I can use. I would like to avoid using KSP and solving for the inverse vector-by-vector if possible. Thanks, Rakesh Halder -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Thu Jun 25 15:51:33 2020 From: bsmith at petsc.dev (Barry Smith) Date: Thu, 25 Jun 2020 15:51:33 -0500 Subject: [petsc-users] Inaccurate Matrix Inversion using MatMatSolve In-Reply-To: References: Message-ID: Please send the code. This should work fine. Barry > On Jun 25, 2020, at 2:43 PM, Rakesh Halder wrote: > > Hi all, > > I'm using PETSc for matrix calculations as part of a model order reduction code. An algorithm I'm using requires that I compute the explicit inverse of a matrix, as it needs to be used in matrix-matrix products. > > The matrix is small (I'll show a 5x5 example), and I first find the LU factorization of it as follows: > > MatGetOrdering(A,MATORDERINGNATURAL,&perm,&iperm); > MatFactorInfoInitialize(&info); > MatLUFactor(A,perm,iperm,&info); > > I then find the inverse of it: > > MatMatSolve(A,I,B); > > Where I is the identity matrix, and B is the inverse. The subroutine does calculate an inverse, although not accurately; the matrix A I looked at is: > > 5.23E-02 1.86E-02 2.67E-02 4.58E-02 3.55E-02 > 6.37E-03 5.86E-02 5.07E-03 1.64E-02 1.36E-02 > -4.07E-02 3.99E-03 5.50E-02 1.77E-02 -3.21E-02 > 1.96E-02 -5.53E-03 4.02E-02 -5.37E-02 1.80E-02 > 1.51E-02 1.70E-02 -1.57E-02 -3.75E-03 -5.64E-02 > > And when I take the product A*B, I don't recover the identity matrix, but rather: > > 9.97E-01 2.77E-04 -1.66E-03 1.25E-04 -1.10E-03 > -6.79E-04 9.99E-01 -1.72E-04 6.24E-04 1.16E-03 > 2.70E-03 -3.98E-04 9.98E-01 -1.51E-04 6.82E-04 > -3.35E-04 -4.82E-04 -1.03E-03 9.99E-01 9.80E-05 > -9.50E-04 -8.70E-04 8.37E-04 -3.68E-04 9.99E-01 > > I believe this is causing large inaccuracies in my program, as the diagonal and off-diagonal entries have large errors associated with them. I am wondering if there is a way to perhaps tighten the tolerance of the subroutine, or if there are other methods I can use. I would like to avoid using KSP and solving for the inverse vector-by-vector if possible. > > Thanks, > > Rakesh Halder -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Thu Jun 25 15:51:33 2020 From: bsmith at petsc.dev (Barry Smith) Date: Thu, 25 Jun 2020 15:51:33 -0500 Subject: [petsc-users] Inaccurate Matrix Inversion using MatMatSolve In-Reply-To: References: Message-ID: Please send the code. This should work fine. Barry > On Jun 25, 2020, at 2:43 PM, Rakesh Halder wrote: > > Hi all, > > I'm using PETSc for matrix calculations as part of a model order reduction code. An algorithm I'm using requires that I compute the explicit inverse of a matrix, as it needs to be used in matrix-matrix products. > > The matrix is small (I'll show a 5x5 example), and I first find the LU factorization of it as follows: > > MatGetOrdering(A,MATORDERINGNATURAL,&perm,&iperm); > MatFactorInfoInitialize(&info); > MatLUFactor(A,perm,iperm,&info); > > I then find the inverse of it: > > MatMatSolve(A,I,B); > > Where I is the identity matrix, and B is the inverse. The subroutine does calculate an inverse, although not accurately; the matrix A I looked at is: > > 5.23E-02 1.86E-02 2.67E-02 4.58E-02 3.55E-02 > 6.37E-03 5.86E-02 5.07E-03 1.64E-02 1.36E-02 > -4.07E-02 3.99E-03 5.50E-02 1.77E-02 -3.21E-02 > 1.96E-02 -5.53E-03 4.02E-02 -5.37E-02 1.80E-02 > 1.51E-02 1.70E-02 -1.57E-02 -3.75E-03 -5.64E-02 > > And when I take the product A*B, I don't recover the identity matrix, but rather: > > 9.97E-01 2.77E-04 -1.66E-03 1.25E-04 -1.10E-03 > -6.79E-04 9.99E-01 -1.72E-04 6.24E-04 1.16E-03 > 2.70E-03 -3.98E-04 9.98E-01 -1.51E-04 6.82E-04 > -3.35E-04 -4.82E-04 -1.03E-03 9.99E-01 9.80E-05 > -9.50E-04 -8.70E-04 8.37E-04 -3.68E-04 9.99E-01 > > I believe this is causing large inaccuracies in my program, as the diagonal and off-diagonal entries have large errors associated with them. I am wondering if there is a way to perhaps tighten the tolerance of the subroutine, or if there are other methods I can use. I would like to avoid using KSP and solving for the inverse vector-by-vector if possible. > > Thanks, > > Rakesh Halder -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Thu Jun 25 20:54:44 2020 From: bsmith at petsc.dev (Barry Smith) Date: Thu, 25 Jun 2020 20:54:44 -0500 Subject: [petsc-users] Problem coupling Petsc into OpenFOAM In-Reply-To: References: <56797615-51EA-4C56-AEB4-F6FEC1949348@anl.gov> <673e9f0f-499c-07e3-280f-63ec02f08810@esi-group.com> <12E2C6D2-8152-42E8-9286-C190FD30AC5D@mcs.anl.gov> Message-ID: <0A1FAFD2-01EB-4EE1-A841-7EDFC824AE34@petsc.dev> Mark, Thanks for letting us know. We have added it to the PETSc package list. Should any issues come up we'll be glad to help support it. Barry > On Jun 25, 2020, at 4:53 AM, Mark Olesen wrote: > > Late followup, but people may want to know that a PETSc/OpenFOAM interface is included in the pending OpenFOAM-v2006 release as a git submodule: > > https://develop.openfoam.com/modules/external-solver > > Cheers, > /mark > > On 2019-05-24 12:06, Matthew Knepley via petsc-users wrote: >> On Thu, May 23, 2019 at 10:41 PM Vu Q. Do via petsc-users > wrote: >> Hi all, >> Thanks for your previous suggestion, I have been able to >> successfully link Petsc to OpenFOAM. I have written a simple >> interface and it works quite well in serial mode, but cannot run in >> parallel. I have been thinking about this problem for weeks but >> couldn't solve it. So I think maybe you could give me some idea. I >> describe my problem below. >> My interface is just a class named "petscSolver/"/, which is used to >> convert an openfoam's matrix or blocked matrix to Petsc Mat, then >> solve the matrix using Petsc's solver. >> To use Petsc, an Openfoam's solver need to be recompiled after >> adding the following lines to make file: >> EXE_INC = \ >> ... >> -I$(LIB_SRC)/petscSolver \ >> -I$(PETSC_DIR)/include \ >> -I$(PETSC_DIR)/$(PETSC_ARCH)/include >> EXE_LIBS = \ >> ... >> -L$(PETSC_DIR)/$(PETSC_ARCH)/lib -lpetsc >> To run an openfoam's case in parallel, first I need to discretize >> the domain into subdomains (e.g. 2 subdomains ), then use the >> following command: >> mpirun -np 2 mySolver -parallel >> (where mpirun is literally mpiexec) The simulation crashed even >> before doing anything and the error message is as in the attached image. >> Run in the debugger and see where it is crashing. Its possible to screw up the MPI linking here, so that you >> link OPENFOAM with one MPI and PETSc with another, or you call MPIInit() after you call PetscInitialize(), etc. >> Thanks, >> Matt >> I have tested and realized that the solver can run in parallel as >> normal by removing the two lines: >> -I$(PETSC_DIR)/include \ >> -I$(PETSC_DIR)/$(PETSC_ARCH)/include >> But then it is clearly no longer linked to Petsc. >> I would appreciate any suggestion. >> Screenshot from 2019-05-24 09-22-17.png >> On Thu, Apr 11, 2019 at 1:37 PM Smith, Barry F. > > wrote: >> Mark, >> Thanks for the clarifying email. My google searches didn't >> locate the rheoTool you mention nor "a PRACE project running via >> CINECA (Bologna)". >> It would be nice if someday OpenFOAM had (either directly >> or somehow with the modules directory) an interface to the PETSc >> solvers. This would allow the use of a variety of other solvers >> including hypre BoomerAMG, SuperLU_Dist, MUMPS, and even the use >> of PETSc/ViennaCL GPU based solvers automatically from OpenFOAM. >> Unfortunately the PETSc group doesn't have the resources or >> expertise to develop and support such an interface ourselves. We >> would, of course, try to answer emails about PETSc usage and >> bugs for such an interface. >> Barry >> If OpenFOAM did have such an interface one thing we could >> provide is the CI infrastructure for tracking changes to PETSc >> that may effect OpenFOAM. For example we could automatically >> build OpenFOAM each day with the latest master of PETSc thus >> immediately detecting changes that effect the interface. >> > On Apr 10, 2019, at 4:55 PM, Mark Olesen >> > >> wrote: >> > >> > The paper that Barry mentioned gives some generalities, but >> probably >> > won't help much. There are some PETSc/OpenFOAM interfaces in >> rheoTool >> > that are probably much more helpful. >> > >> > As Barry also rightly noted, there are some config files in >> the OpenFOAM >> > tree that were put in some time ago for helping with setting >> up PETSc >> > and OpenFOAM. Assuming that you have set the appropriate >> values in the >> > etc/config.sh/petsc file you will be >> able to use those when using wmake. >> > For running you will still need to ensure that the >> LD_LIBARY_PATH is set >> > correctly. For example, what some build scripts exhibit: >> > >> > wmake(petsc) : >> > ==> Before running, verify that PETSc libraries can be found >> > >> > Enable in the OpenFOAM etc/bashrc, define manually or try >> with the >> > following (POSIX shell): >> > >> > eval $(foamEtcFile -sh -config petsc -- -force) >> > >> > == >> > >> > >> > There is currently a PRACE project running via CINECA >> (Bologna) with >> > binding in PETSc as a runtime selectable linear solver in >> OpenFOAM. This >> > is still at the stage of early testing and performance >> benchmarking. >> > >> > Cheers, >> > /mark >> > >> > On 4/10/19 6:37 PM, Smith, Barry F. via petsc-users wrote: >> >> >> >> We don't know much about OpenFoam but >> >> >> >> 1) if I do a >> >> >> >> git grep -i petsc >> >> >> >> in the >> https://develop.openfoam.com/Development/OpenFOAM-plus.git >> repository I see various configuration files specifically for PETSc. >> >> >> >> etc/config.csh/petsc etc/config.sh/petsc >> wmake/scripts/have_petsc >> >> >> >> so it appears that OpenFOAM has the tools to be linked >> against PETSc (to me the documentation on how to use them is >> rather terse). Are >> >> you using these? If you have trouble with them perhaps >> you can ask the OpenFOAM user community how to use them. >> >> >> >> >> >> 2) if you are editing the Make/options file directly you >> can try changing >> >> >> >> -L$(PETSC_DIR)/$(PETSC_ARCH)/lib -lpetsc >> >> >> >> to >> >> >> >> -Wl,-rpath,$(PETSC_DIR)/$(PETSC_ARCH)/lib -L$(PETSC_DIR)/$(PETSC_ARCH)/lib -lpetsc >> >> >> >> >> >> >> >> Note also that simply including petsc.h into the OpenFoam >> source code and linking against -lpetsc will not immediately >> allow calling the PETSc solvers from OpenFOAM. One needs to >> write all the interface code that sets up and calls the PETSc >> solvers from OpenFOAM. There is a paper >> https://www.researchgate.net/publication/319045499_Insertion_of_PETSc_in_the_OpenFOAM_Framework >> that describes at an abstract level how they wrote code that >> calls the PETSc solvers from OpenFOAM but the source code that >> actually does the work does not appear to be available. >> >> >> >> Note that PETSc is now at version 3.11 we recommend >> working with that version (unless you already have a lot of code >> that calls PETSc written with a previous version of PETSc, for >> that we recommend first upgrading to petsc 3.11 and then >> continuing to add code). >> >> >> >> Barry >> >> >> >> >> >> >> >> >> >> >> >>> On Apr 10, 2019, at 8:23 AM, Balay, Satish via petsc-users >> > wrote: >> >>> >> >>> Runtime error? You might have to add the path to >> $PETSC_ARCH/lib in LD_LIBRARY_PATH env variable >> >>> or - to your link command. If linux/gcc - the linker option >> is -Wl,-rpath,$PETSC_ARCH/lib >> >>> >> >>> If not - send detail logs. >> >>> >> >>> Satish >> >>> >> >>> On Wed, 10 Apr 2019, Vu Do Quoc via petsc-users wrote: >> >>> >> >>>> Hi all, >> >>>> >> >>>> I am trying to insert Petsc to OpenFOAM opensource software. >> >>>> I have been successfully compiling Petsc with an available >> solver in >> >>>> OpenFOAM by linking it with the shared library >> libpetsc.so. However, when I >> >>>> call the solver to run a test case, I got an error saying >> that: >> >>>> "libpetsc.so cannot be found", even though the library >> still exists in the >> >>>> $PETSC_ARCH/lib folder. >> >>>> >> >>>> I have been struggling for weeks but still, have not been >> able to figure it >> >>>> out. Therefore I would be very grateful for any suggestion >> to solve this >> >>>> problem. >> >>>> >> >>>> Thanks in advance for your time, >> >>>> >> >>>> Best regards, >> >>>> >> >>>> Vu Do >> >>>> >> -- *Vu Q. Do >> *------------------------------------------------ >> /Student of Aeronautical Engineering/ >> Programme de Formation d'Ing?nieurs d'Excellence au Vietnam /- PFIEV/ >> School of Transportation Engineering/ >> / >> Hanoi University of Science and Technology >> 01 Dai Co Viet Avenue, Hanoi, Vietnam >> E-mail: vu.doquochust at gmail.com >> -- >> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >> -- Norbert Wiener >> https://www.cse.buffalo.edu/~knepley/ > > -- > Dr Mark OLESEN > Principal Engineer, ESI-OpenCFD > Engineering System International GmbH | Einsteinring 24 | 85609 Munich > Mob. +49 171 9710 149 > www.openfoam.com | www.esi-group.com | mark.olesen at esi-group.com From junchao.zhang at gmail.com Thu Jun 25 21:36:55 2020 From: junchao.zhang at gmail.com (Junchao Zhang) Date: Thu, 25 Jun 2020 21:36:55 -0500 Subject: [petsc-users] PETSc initialization error In-Reply-To: References: <004EA4EB-815F-4EEE-8988-CA2179304157@petsc.dev> Message-ID: Sam, The MPI_Comm_create_keyval() error was fixed in maint/master. From the error message, it seems you need to configure --with-log=1 Otherwise, please send your full error stack trace and configure.log. Thanks. --Junchao Zhang On Thu, Jun 25, 2020 at 2:18 PM Sam Guo wrote: > Hi Junchao, > I now encountered the same error with parallel. I am wondering if there > is a need for parallel fix as well. > [1]PETSC ERROR: #1 PetscInitialize() line 969 in > ../../../petsc/src/sys/objects/pinit.c > PETSC ERROR: Logging has not been enabled. > You might have forgotten to call PetscInitialize(). > PETSC ERROR: Logging has not been enabled. > You might have forgotten to call PetscInitialize(). > > On Sat, Jun 20, 2020 at 7:35 PM Sam Guo wrote: > >> Hi Junchao, >> Your patch works. >> >> Thanks, >> Sam >> >> On Sat, Jun 20, 2020 at 4:23 PM Junchao Zhang >> wrote: >> >>> >>> >>> On Sat, Jun 20, 2020 at 12:24 PM Barry Smith wrote: >>> >>>> >>>> Junchao, >>>> >>>> This is a good bug fix. It solves the problem when PETSc >>>> initialize is called many times. >>>> >>>> There is another fix you can do to limit PETSc mpiuni running out >>>> of attributes inside a single PETSc run: >>>> >>>> >>>> int MPI_Comm_create_keyval(MPI_Copy_function >>>> *copy_fn,MPI_Delete_function *delete_fn,int *keyval,void *extra_state) >>>> { >>>> >>>> if (num_attr >= MAX_ATTR){ >>>> for (i=0; i>>> if (!attr_keyval[i].extra_state) { >>>> >>> attr_keyval[i].extra_state is provided by user (could be NULL). We can >>> not rely on it. >>> >>>> /* reuse this slot */ >>>> attr_keyval[i].extra_state = extra_state; >>>> attr_keyval[i.]del = delete_fn; >>>> *keyval = i; >>>> return MPI_SUCCESS; >>>> } >>>> } >>>> return MPIUni_Abort(MPI_COMM_WORLD,1); >>>> } >>>> return MPIUni_Abort(MPI_COMM_WORLD,1); >>>> attr_keyval[num_attr].extra_state = extra_state; >>>> attr_keyval[num_attr].del = delete_fn; >>>> *keyval = num_attr++; >>>> return MPI_SUCCESS; >>>> } >>>> >>>> This will work if the user creates tons of attributes but is >>>> constantly deleting some as they new ones. So long as the number >>>> outstanding at one time is < MAX_ATTR) >>>> >>>> Barry >>>> >>>> >>>> >>>> >>>> >>>> On Jun 20, 2020, at 10:54 AM, Junchao Zhang >>>> wrote: >>>> >>>> I don't understand what your session means. Let's try this patch >>>> >>>> diff --git a/src/sys/mpiuni/mpi.c b/src/sys/mpiuni/mpi.c >>>> index d559a513..c058265d 100644 >>>> --- a/src/sys/mpiuni/mpi.c >>>> +++ b/src/sys/mpiuni/mpi.c >>>> @@ -283,6 +283,7 @@ int MPI_Finalize(void) >>>> MPI_Comm_free(&comm); >>>> comm = MPI_COMM_SELF; >>>> MPI_Comm_free(&comm); >>>> + num_attr = 1; /* reset the counter */ >>>> MPI_was_finalized = 1; >>>> return MPI_SUCCESS; >>>> } >>>> >>>> >>>> --Junchao Zhang >>>> >>>> >>>> On Sat, Jun 20, 2020 at 10:48 AM Sam Guo wrote: >>>> >>>>> Typo: I mean ?Assuming initializer is only needed once for entire >>>>> session? >>>>> >>>>> On Saturday, June 20, 2020, Sam Guo wrote: >>>>> >>>>>> Assuming finalizer is only needed once for entire session(?), I can >>>>>> put initializer into the static block to call it once but where do I call >>>>>> finalizer? >>>>>> >>>>>> >>>>>> On Saturday, June 20, 2020, Junchao Zhang >>>>>> wrote: >>>>>> >>>>>>> The counter num_attr should be recycled. But first try to call PETSc >>>>>>> initialize/Finalize only once to see it fixes the error. >>>>>>> --Junchao Zhang >>>>>>> >>>>>>> >>>>>>> On Sat, Jun 20, 2020 at 12:48 AM Sam Guo >>>>>>> wrote: >>>>>>> >>>>>>>> To clarify, I call PETSc initialize and PETSc finalize everytime I >>>>>>>> call SLEPc: >>>>>>>> >>>>>>>> PetscInitializeNoPointers(argc,args,nullptr,nullptr); >>>>>>>> >>>>>>>> SlepcInitialize(&argc,&args,static_cast(nullptr),help); >>>>>>>> >>>>>>>> //calling slepc >>>>>>>> >>>>>>>> SlepcFinalize(); >>>>>>>> >>>>>>>> PetscFinalize(); >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Fri, Jun 19, 2020 at 10:32 PM Sam Guo >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Dear PETSc team, >>>>>>>>> When I called SLEPc multiple time, I eventually got following >>>>>>>>> error: >>>>>>>>> >>>>>>>>> MPI operation not supported by PETSc's sequential MPI wrappers >>>>>>>>> [0]PETSC ERROR: #1 PetscInitialize() line 967 in >>>>>>>>> ../../../petsc/src/sys/objects/pinit.c >>>>>>>>> [0]PETSC ERROR: #2 SlepcInitialize() line 262 in >>>>>>>>> ../../../slepc/src/sys/slepcinit.c >>>>>>>>> [0]PETSC ERROR: #3 SlepcInitializeNoPointers() line 359 in >>>>>>>>> ../../../slepc/src/sys/slepcinit.c >>>>>>>>> PETSC ERROR: Logging has not been enabled. >>>>>>>>> You might have forgotten to call PetscInitialize(). >>>>>>>>> >>>>>>>>> I debugged: it is because of following in >>>>>>>>> petsc/src/sys/mpiuni/mpi.c >>>>>>>>> >>>>>>>>> if (num_attr >= MAX_ATTR) >>>>>>>>> >>>>>>>>> in function int MPI_Comm_create_keyval(MPI_Copy_function >>>>>>>>> *copy_fn,MPI_Delete_function *delete_fn,int *keyval,void *extra_state) >>>>>>>>> >>>>>>>>> num_attr is declared static and keeps increasing every >>>>>>>>> time MPI_Comm_create_keyval is called. >>>>>>>>> >>>>>>>>> I am using petsc 3.11.3 but found 3.13.2 has the same logic. >>>>>>>>> >>>>>>>>> Is this a bug or I didn't use it correctly? >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Sam >>>>>>>>> >>>>>>>> >>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From Mark.Olesen at esi-group.com Fri Jun 26 02:48:31 2020 From: Mark.Olesen at esi-group.com (Mark Olesen) Date: Fri, 26 Jun 2020 09:48:31 +0200 Subject: [petsc-users] Problem coupling Petsc into OpenFOAM In-Reply-To: <0A1FAFD2-01EB-4EE1-A841-7EDFC824AE34@petsc.dev> References: <56797615-51EA-4C56-AEB4-F6FEC1949348@anl.gov> <673e9f0f-499c-07e3-280f-63ec02f08810@esi-group.com> <12E2C6D2-8152-42E8-9286-C190FD30AC5D@mcs.anl.gov> <0A1FAFD2-01EB-4EE1-A841-7EDFC824AE34@petsc.dev> Message-ID: Thanks Barry for adding it, and for the offer to help with supporting it. In the meantime we have excellent PETSc knowledge available in the form of Stefano Zampini. He is also involved in the OpenFOAM HPC Technical Committee [https://wiki.openfoam.com/High_Performance_Computing_(HPC)_Technical_Committee] so I think there will be an even closer connection with PETSc in the future. Cheers, /mark On 2020-06-26 03:54, Barry Smith wrote: > > Mark, > > Thanks for letting us know. We have added it to the PETSc package list. Should any issues come up we'll be glad to help support it. > > Barry > > >> On Jun 25, 2020, at 4:53 AM, Mark Olesen wrote: >> >> Late followup, but people may want to know that a PETSc/OpenFOAM interface is included in the pending OpenFOAM-v2006 release as a git submodule: >> >> https://develop.openfoam.com/modules/external-solver From sam.guo at cd-adapco.com Fri Jun 26 15:09:55 2020 From: sam.guo at cd-adapco.com (Sam Guo) Date: Fri, 26 Jun 2020 13:09:55 -0700 Subject: [petsc-users] PETSc initialization error In-Reply-To: References: <004EA4EB-815F-4EEE-8988-CA2179304157@petsc.dev> Message-ID: Hi Junchao, Attached please find the configure.log. I also attach the pinit.c which contains your patch (I am currently using 3.11.3. I've applied your patch to 3.11.3). Your patch fixes the serial version. The error now is about the parallel. Here is the error log: [1]PETSC ERROR: #1 PetscInitialize() line 969 in ../../../petsc/src/sys/objects/pinit.c [1]PETSC ERROR: #2 checkError() line 56 in ../../../physics/src/eigensolver/SLEPc.cpp [1]PETSC ERROR: #3 PetscInitialize() line 966 in ../../../petsc/src/sys/objects/pinit.c [1]PETSC ERROR: #4 SlepcInitialize() line 262 in ../../../slepc/src/sys/slepcinit.c [0]PETSC ERROR: #1 PetscInitialize() line 969 in ../../../petsc/src/sys/objects/pinit.c [0]PETSC ERROR: #2 checkError() line 56 in ../../../physics/src/eigensolver/SLEPc.cpp [0]PETSC ERROR: #3 PetscInitialize() line 966 in ../../../petsc/src/sys/objects/pinit.c [0]PETSC ERROR: #4 SlepcInitialize() line 262 in ../../../slepc/src/sys/slepcinit.c PETSC ERROR: Logging has not been enabled. You might have forgotten to call PetscInitialize(). PETSC ERROR: Logging has not been enabled. You might have forgotten to call PetscInitialize(). -------------------------------------------------------------------------- MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD with errorcode 56. NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. You may or may not see output from other processes, depending on exactly when Open MPI kills them. Thanks, Sam On Thu, Jun 25, 2020 at 7:37 PM Junchao Zhang wrote: > Sam, > The MPI_Comm_create_keyval() error was fixed in maint/master. From the > error message, it seems you need to configure --with-log=1 > Otherwise, please send your full error stack trace and configure.log. > Thanks. > --Junchao Zhang > > > On Thu, Jun 25, 2020 at 2:18 PM Sam Guo wrote: > >> Hi Junchao, >> I now encountered the same error with parallel. I am wondering if >> there is a need for parallel fix as well. >> [1]PETSC ERROR: #1 PetscInitialize() line 969 in >> ../../../petsc/src/sys/objects/pinit.c >> PETSC ERROR: Logging has not been enabled. >> You might have forgotten to call PetscInitialize(). >> PETSC ERROR: Logging has not been enabled. >> You might have forgotten to call PetscInitialize(). >> >> On Sat, Jun 20, 2020 at 7:35 PM Sam Guo wrote: >> >>> Hi Junchao, >>> Your patch works. >>> >>> Thanks, >>> Sam >>> >>> On Sat, Jun 20, 2020 at 4:23 PM Junchao Zhang >>> wrote: >>> >>>> >>>> >>>> On Sat, Jun 20, 2020 at 12:24 PM Barry Smith wrote: >>>> >>>>> >>>>> Junchao, >>>>> >>>>> This is a good bug fix. It solves the problem when PETSc >>>>> initialize is called many times. >>>>> >>>>> There is another fix you can do to limit PETSc mpiuni running out >>>>> of attributes inside a single PETSc run: >>>>> >>>>> >>>>> int MPI_Comm_create_keyval(MPI_Copy_function >>>>> *copy_fn,MPI_Delete_function *delete_fn,int *keyval,void *extra_state) >>>>> { >>>>> >>>>> if (num_attr >= MAX_ATTR){ >>>>> for (i=0; i>>>> if (!attr_keyval[i].extra_state) { >>>>> >>>> attr_keyval[i].extra_state is provided by user (could be NULL). We can >>>> not rely on it. >>>> >>>>> /* reuse this slot */ >>>>> attr_keyval[i].extra_state = extra_state; >>>>> attr_keyval[i.]del = delete_fn; >>>>> *keyval = i; >>>>> return MPI_SUCCESS; >>>>> } >>>>> } >>>>> return MPIUni_Abort(MPI_COMM_WORLD,1); >>>>> } >>>>> return MPIUni_Abort(MPI_COMM_WORLD,1); >>>>> attr_keyval[num_attr].extra_state = extra_state; >>>>> attr_keyval[num_attr].del = delete_fn; >>>>> *keyval = num_attr++; >>>>> return MPI_SUCCESS; >>>>> } >>>>> >>>>> This will work if the user creates tons of attributes but is >>>>> constantly deleting some as they new ones. So long as the number >>>>> outstanding at one time is < MAX_ATTR) >>>>> >>>>> Barry >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> On Jun 20, 2020, at 10:54 AM, Junchao Zhang >>>>> wrote: >>>>> >>>>> I don't understand what your session means. Let's try this patch >>>>> >>>>> diff --git a/src/sys/mpiuni/mpi.c b/src/sys/mpiuni/mpi.c >>>>> index d559a513..c058265d 100644 >>>>> --- a/src/sys/mpiuni/mpi.c >>>>> +++ b/src/sys/mpiuni/mpi.c >>>>> @@ -283,6 +283,7 @@ int MPI_Finalize(void) >>>>> MPI_Comm_free(&comm); >>>>> comm = MPI_COMM_SELF; >>>>> MPI_Comm_free(&comm); >>>>> + num_attr = 1; /* reset the counter */ >>>>> MPI_was_finalized = 1; >>>>> return MPI_SUCCESS; >>>>> } >>>>> >>>>> >>>>> --Junchao Zhang >>>>> >>>>> >>>>> On Sat, Jun 20, 2020 at 10:48 AM Sam Guo >>>>> wrote: >>>>> >>>>>> Typo: I mean ?Assuming initializer is only needed once for entire >>>>>> session? >>>>>> >>>>>> On Saturday, June 20, 2020, Sam Guo wrote: >>>>>> >>>>>>> Assuming finalizer is only needed once for entire session(?), I can >>>>>>> put initializer into the static block to call it once but where do I call >>>>>>> finalizer? >>>>>>> >>>>>>> >>>>>>> On Saturday, June 20, 2020, Junchao Zhang >>>>>>> wrote: >>>>>>> >>>>>>>> The counter num_attr should be recycled. But first try to call >>>>>>>> PETSc initialize/Finalize only once to see it fixes the error. >>>>>>>> --Junchao Zhang >>>>>>>> >>>>>>>> >>>>>>>> On Sat, Jun 20, 2020 at 12:48 AM Sam Guo >>>>>>>> wrote: >>>>>>>> >>>>>>>>> To clarify, I call PETSc initialize and PETSc finalize everytime I >>>>>>>>> call SLEPc: >>>>>>>>> >>>>>>>>> PetscInitializeNoPointers(argc,args,nullptr,nullptr); >>>>>>>>> >>>>>>>>> SlepcInitialize(&argc,&args,static_cast(nullptr),help); >>>>>>>>> >>>>>>>>> //calling slepc >>>>>>>>> >>>>>>>>> SlepcFinalize(); >>>>>>>>> >>>>>>>>> PetscFinalize(); >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Fri, Jun 19, 2020 at 10:32 PM Sam Guo >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Dear PETSc team, >>>>>>>>>> When I called SLEPc multiple time, I eventually got following >>>>>>>>>> error: >>>>>>>>>> >>>>>>>>>> MPI operation not supported by PETSc's sequential MPI wrappers >>>>>>>>>> [0]PETSC ERROR: #1 PetscInitialize() line 967 in >>>>>>>>>> ../../../petsc/src/sys/objects/pinit.c >>>>>>>>>> [0]PETSC ERROR: #2 SlepcInitialize() line 262 in >>>>>>>>>> ../../../slepc/src/sys/slepcinit.c >>>>>>>>>> [0]PETSC ERROR: #3 SlepcInitializeNoPointers() line 359 in >>>>>>>>>> ../../../slepc/src/sys/slepcinit.c >>>>>>>>>> PETSC ERROR: Logging has not been enabled. >>>>>>>>>> You might have forgotten to call PetscInitialize(). >>>>>>>>>> >>>>>>>>>> I debugged: it is because of following in >>>>>>>>>> petsc/src/sys/mpiuni/mpi.c >>>>>>>>>> >>>>>>>>>> if (num_attr >= MAX_ATTR) >>>>>>>>>> >>>>>>>>>> in function int MPI_Comm_create_keyval(MPI_Copy_function >>>>>>>>>> *copy_fn,MPI_Delete_function *delete_fn,int *keyval,void *extra_state) >>>>>>>>>> >>>>>>>>>> num_attr is declared static and keeps increasing every >>>>>>>>>> time MPI_Comm_create_keyval is called. >>>>>>>>>> >>>>>>>>>> I am using petsc 3.11.3 but found 3.13.2 has the same logic. >>>>>>>>>> >>>>>>>>>> Is this a bug or I didn't use it correctly? >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> Sam >>>>>>>>>> >>>>>>>>> >>>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: configure.log Type: text/x-log Size: 2853770 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: mpi.c Type: text/x-csrc Size: 25039 bytes Desc: not available URL: From sam.guo at cd-adapco.com Fri Jun 26 15:14:43 2020 From: sam.guo at cd-adapco.com (Sam Guo) Date: Fri, 26 Jun 2020 13:14:43 -0700 Subject: [petsc-users] PETSc initialization error In-Reply-To: References: <004EA4EB-815F-4EEE-8988-CA2179304157@petsc.dev> Message-ID: To clarify, we have a mpi wrapper (so we can switch to different mpi at runtime). I compile petsc using our mpi wrapper. If I just call PETSc initialize once without calling finallize, it is ok. My question to you is that: can I skip finalize? Our program calls mpi_finalize at end anyway. On Fri, Jun 26, 2020 at 1:09 PM Sam Guo wrote: > Hi Junchao, > Attached please find the configure.log. > I also attach the pinit.c which contains your patch (I am currently > using 3.11.3. I've applied your patch to 3.11.3). Your patch fixes the > serial version. The error now is about the parallel. > Here is the error log: > > [1]PETSC ERROR: #1 PetscInitialize() line 969 in > ../../../petsc/src/sys/objects/pinit.c > [1]PETSC ERROR: #2 checkError() line 56 in > ../../../physics/src/eigensolver/SLEPc.cpp > [1]PETSC ERROR: #3 PetscInitialize() line 966 in > ../../../petsc/src/sys/objects/pinit.c > [1]PETSC ERROR: #4 SlepcInitialize() line 262 in > ../../../slepc/src/sys/slepcinit.c > [0]PETSC ERROR: #1 PetscInitialize() line 969 in > ../../../petsc/src/sys/objects/pinit.c > [0]PETSC ERROR: #2 checkError() line 56 in > ../../../physics/src/eigensolver/SLEPc.cpp > [0]PETSC ERROR: #3 PetscInitialize() line 966 in > ../../../petsc/src/sys/objects/pinit.c > [0]PETSC ERROR: #4 SlepcInitialize() line 262 in > ../../../slepc/src/sys/slepcinit.c > PETSC ERROR: Logging has not been enabled. > You might have forgotten to call PetscInitialize(). > PETSC ERROR: Logging has not been enabled. > You might have forgotten to call PetscInitialize(). > -------------------------------------------------------------------------- > MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD > with errorcode 56. > > NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. > You may or may not see output from other processes, depending on > exactly when Open MPI kills them. > > Thanks, > Sam > > On Thu, Jun 25, 2020 at 7:37 PM Junchao Zhang > wrote: > >> Sam, >> The MPI_Comm_create_keyval() error was fixed in maint/master. From the >> error message, it seems you need to configure --with-log=1 >> Otherwise, please send your full error stack trace and configure.log. >> Thanks. >> --Junchao Zhang >> >> >> On Thu, Jun 25, 2020 at 2:18 PM Sam Guo wrote: >> >>> Hi Junchao, >>> I now encountered the same error with parallel. I am wondering if >>> there is a need for parallel fix as well. >>> [1]PETSC ERROR: #1 PetscInitialize() line 969 in >>> ../../../petsc/src/sys/objects/pinit.c >>> PETSC ERROR: Logging has not been enabled. >>> You might have forgotten to call PetscInitialize(). >>> PETSC ERROR: Logging has not been enabled. >>> You might have forgotten to call PetscInitialize(). >>> >>> On Sat, Jun 20, 2020 at 7:35 PM Sam Guo wrote: >>> >>>> Hi Junchao, >>>> Your patch works. >>>> >>>> Thanks, >>>> Sam >>>> >>>> On Sat, Jun 20, 2020 at 4:23 PM Junchao Zhang >>>> wrote: >>>> >>>>> >>>>> >>>>> On Sat, Jun 20, 2020 at 12:24 PM Barry Smith wrote: >>>>> >>>>>> >>>>>> Junchao, >>>>>> >>>>>> This is a good bug fix. It solves the problem when PETSc >>>>>> initialize is called many times. >>>>>> >>>>>> There is another fix you can do to limit PETSc mpiuni running >>>>>> out of attributes inside a single PETSc run: >>>>>> >>>>>> >>>>>> int MPI_Comm_create_keyval(MPI_Copy_function >>>>>> *copy_fn,MPI_Delete_function *delete_fn,int *keyval,void *extra_state) >>>>>> { >>>>>> >>>>>> if (num_attr >= MAX_ATTR){ >>>>>> for (i=0; i>>>>> if (!attr_keyval[i].extra_state) { >>>>>> >>>>> attr_keyval[i].extra_state is provided by user (could be NULL). We can >>>>> not rely on it. >>>>> >>>>>> /* reuse this slot */ >>>>>> attr_keyval[i].extra_state = extra_state; >>>>>> attr_keyval[i.]del = delete_fn; >>>>>> *keyval = i; >>>>>> return MPI_SUCCESS; >>>>>> } >>>>>> } >>>>>> return MPIUni_Abort(MPI_COMM_WORLD,1); >>>>>> } >>>>>> return MPIUni_Abort(MPI_COMM_WORLD,1); >>>>>> attr_keyval[num_attr].extra_state = extra_state; >>>>>> attr_keyval[num_attr].del = delete_fn; >>>>>> *keyval = num_attr++; >>>>>> return MPI_SUCCESS; >>>>>> } >>>>>> >>>>>> This will work if the user creates tons of attributes but is >>>>>> constantly deleting some as they new ones. So long as the number >>>>>> outstanding at one time is < MAX_ATTR) >>>>>> >>>>>> Barry >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Jun 20, 2020, at 10:54 AM, Junchao Zhang >>>>>> wrote: >>>>>> >>>>>> I don't understand what your session means. Let's try this patch >>>>>> >>>>>> diff --git a/src/sys/mpiuni/mpi.c b/src/sys/mpiuni/mpi.c >>>>>> index d559a513..c058265d 100644 >>>>>> --- a/src/sys/mpiuni/mpi.c >>>>>> +++ b/src/sys/mpiuni/mpi.c >>>>>> @@ -283,6 +283,7 @@ int MPI_Finalize(void) >>>>>> MPI_Comm_free(&comm); >>>>>> comm = MPI_COMM_SELF; >>>>>> MPI_Comm_free(&comm); >>>>>> + num_attr = 1; /* reset the counter */ >>>>>> MPI_was_finalized = 1; >>>>>> return MPI_SUCCESS; >>>>>> } >>>>>> >>>>>> >>>>>> --Junchao Zhang >>>>>> >>>>>> >>>>>> On Sat, Jun 20, 2020 at 10:48 AM Sam Guo >>>>>> wrote: >>>>>> >>>>>>> Typo: I mean ?Assuming initializer is only needed once for entire >>>>>>> session? >>>>>>> >>>>>>> On Saturday, June 20, 2020, Sam Guo wrote: >>>>>>> >>>>>>>> Assuming finalizer is only needed once for entire session(?), I can >>>>>>>> put initializer into the static block to call it once but where do I call >>>>>>>> finalizer? >>>>>>>> >>>>>>>> >>>>>>>> On Saturday, June 20, 2020, Junchao Zhang >>>>>>>> wrote: >>>>>>>> >>>>>>>>> The counter num_attr should be recycled. But first try to call >>>>>>>>> PETSc initialize/Finalize only once to see it fixes the error. >>>>>>>>> --Junchao Zhang >>>>>>>>> >>>>>>>>> >>>>>>>>> On Sat, Jun 20, 2020 at 12:48 AM Sam Guo >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> To clarify, I call PETSc initialize and PETSc finalize >>>>>>>>>> everytime I call SLEPc: >>>>>>>>>> >>>>>>>>>> PetscInitializeNoPointers(argc,args,nullptr,nullptr); >>>>>>>>>> >>>>>>>>>> SlepcInitialize(&argc,&args,static_cast(nullptr),help); >>>>>>>>>> >>>>>>>>>> //calling slepc >>>>>>>>>> >>>>>>>>>> SlepcFinalize(); >>>>>>>>>> >>>>>>>>>> PetscFinalize(); >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Fri, Jun 19, 2020 at 10:32 PM Sam Guo >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Dear PETSc team, >>>>>>>>>>> When I called SLEPc multiple time, I eventually got following >>>>>>>>>>> error: >>>>>>>>>>> >>>>>>>>>>> MPI operation not supported by PETSc's sequential MPI wrappers >>>>>>>>>>> [0]PETSC ERROR: #1 PetscInitialize() line 967 in >>>>>>>>>>> ../../../petsc/src/sys/objects/pinit.c >>>>>>>>>>> [0]PETSC ERROR: #2 SlepcInitialize() line 262 in >>>>>>>>>>> ../../../slepc/src/sys/slepcinit.c >>>>>>>>>>> [0]PETSC ERROR: #3 SlepcInitializeNoPointers() line 359 in >>>>>>>>>>> ../../../slepc/src/sys/slepcinit.c >>>>>>>>>>> PETSC ERROR: Logging has not been enabled. >>>>>>>>>>> You might have forgotten to call PetscInitialize(). >>>>>>>>>>> >>>>>>>>>>> I debugged: it is because of following in >>>>>>>>>>> petsc/src/sys/mpiuni/mpi.c >>>>>>>>>>> >>>>>>>>>>> if (num_attr >= MAX_ATTR) >>>>>>>>>>> >>>>>>>>>>> in function int MPI_Comm_create_keyval(MPI_Copy_function >>>>>>>>>>> *copy_fn,MPI_Delete_function *delete_fn,int *keyval,void *extra_state) >>>>>>>>>>> >>>>>>>>>>> num_attr is declared static and keeps increasing every >>>>>>>>>>> time MPI_Comm_create_keyval is called. >>>>>>>>>>> >>>>>>>>>>> I am using petsc 3.11.3 but found 3.13.2 has the same logic. >>>>>>>>>>> >>>>>>>>>>> Is this a bug or I didn't use it correctly? >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> Sam >>>>>>>>>>> >>>>>>>>>> >>>>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Fri Jun 26 15:21:53 2020 From: bsmith at petsc.dev (Barry Smith) Date: Fri, 26 Jun 2020 15:21:53 -0500 Subject: [petsc-users] PETSc initialization error In-Reply-To: References: <004EA4EB-815F-4EEE-8988-CA2179304157@petsc.dev> Message-ID: <9C60917A-5771-434D-BB65-D46750C49966@petsc.dev> Sam, You can skip PetscFinalize() so long as you only call PetscInitialize() once. It is not desirable in general to skip the finalize because PETSc can't free all its data structures and you cannot see the PETSc logging information with -log_view but in terms of the code running correctly you do not need to call PetscFinalize. If your code crashes in PetscFinalize() please send the full error output and we can try to help you debug it. Barry > On Jun 26, 2020, at 3:14 PM, Sam Guo wrote: > > To clarify, we have a mpi wrapper (so we can switch to different mpi at runtime). I compile petsc using our mpi wrapper. > If I just call PETSc initialize once without calling finallize, it is ok. My question to you is that: can I skip finalize? > Our program calls mpi_finalize at end anyway. > > On Fri, Jun 26, 2020 at 1:09 PM Sam Guo > wrote: > Hi Junchao, > Attached please find the configure.log. > I also attach the pinit.c which contains your patch (I am currently using 3.11.3. I've applied your patch to 3.11.3). Your patch fixes the serial version. The error now is about the parallel. > Here is the error log: > > [1]PETSC ERROR: #1 PetscInitialize() line 969 in ../../../petsc/src/sys/objects/pinit.c > [1]PETSC ERROR: #2 checkError() line 56 in ../../../physics/src/eigensolver/SLEPc.cpp > [1]PETSC ERROR: #3 PetscInitialize() line 966 in ../../../petsc/src/sys/objects/pinit.c > [1]PETSC ERROR: #4 SlepcInitialize() line 262 in ../../../slepc/src/sys/slepcinit.c > [0]PETSC ERROR: #1 PetscInitialize() line 969 in ../../../petsc/src/sys/objects/pinit.c > [0]PETSC ERROR: #2 checkError() line 56 in ../../../physics/src/eigensolver/SLEPc.cpp > [0]PETSC ERROR: #3 PetscInitialize() line 966 in ../../../petsc/src/sys/objects/pinit.c > [0]PETSC ERROR: #4 SlepcInitialize() line 262 in ../../../slepc/src/sys/slepcinit.c > PETSC ERROR: Logging has not been enabled. > You might have forgotten to call PetscInitialize(). > PETSC ERROR: Logging has not been enabled. > You might have forgotten to call PetscInitialize(). > -------------------------------------------------------------------------- > MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD > with errorcode 56. > > NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. > You may or may not see output from other processes, depending on > exactly when Open MPI kills them. > > Thanks, > Sam > > On Thu, Jun 25, 2020 at 7:37 PM Junchao Zhang > wrote: > Sam, > The MPI_Comm_create_keyval() error was fixed in maint/master. From the error message, it seems you need to configure --with-log=1 > Otherwise, please send your full error stack trace and configure.log. > Thanks. > --Junchao Zhang > > > On Thu, Jun 25, 2020 at 2:18 PM Sam Guo > wrote: > Hi Junchao, > I now encountered the same error with parallel. I am wondering if there is a need for parallel fix as well. > [1]PETSC ERROR: #1 PetscInitialize() line 969 in ../../../petsc/src/sys/objects/pinit.c > PETSC ERROR: Logging has not been enabled. > You might have forgotten to call PetscInitialize(). > PETSC ERROR: Logging has not been enabled. > You might have forgotten to call PetscInitialize(). > > On Sat, Jun 20, 2020 at 7:35 PM Sam Guo > wrote: > Hi Junchao, > Your patch works. > > Thanks, > Sam > > On Sat, Jun 20, 2020 at 4:23 PM Junchao Zhang > wrote: > > > On Sat, Jun 20, 2020 at 12:24 PM Barry Smith > wrote: > > Junchao, > > This is a good bug fix. It solves the problem when PETSc initialize is called many times. > > There is another fix you can do to limit PETSc mpiuni running out of attributes inside a single PETSc run: > > > int MPI_Comm_create_keyval(MPI_Copy_function *copy_fn,MPI_Delete_function *delete_fn,int *keyval,void *extra_state) > { > > if (num_attr >= MAX_ATTR){ > for (i=0; i if (!attr_keyval[i].extra_state) { > attr_keyval[i].extra_state is provided by user (could be NULL). We can not rely on it. > /* reuse this slot */ > attr_keyval[i].extra_state = extra_state; > attr_keyval[i.]del = delete_fn; > *keyval = i; > return MPI_SUCCESS; > } > } > return MPIUni_Abort(MPI_COMM_WORLD,1); > } > return MPIUni_Abort(MPI_COMM_WORLD,1); > attr_keyval[num_attr].extra_state = extra_state; > attr_keyval[num_attr].del = delete_fn; > *keyval = num_attr++; > return MPI_SUCCESS; > } > > This will work if the user creates tons of attributes but is constantly deleting some as they new ones. So long as the number outstanding at one time is < MAX_ATTR) > > Barry > > > > > >> On Jun 20, 2020, at 10:54 AM, Junchao Zhang > wrote: >> >> I don't understand what your session means. Let's try this patch >> >> diff --git a/src/sys/mpiuni/mpi.c b/src/sys/mpiuni/mpi.c >> index d559a513..c058265d 100644 >> --- a/src/sys/mpiuni/mpi.c >> +++ b/src/sys/mpiuni/mpi.c >> @@ -283,6 +283,7 @@ int MPI_Finalize(void) >> MPI_Comm_free(&comm); >> comm = MPI_COMM_SELF; >> MPI_Comm_free(&comm); >> + num_attr = 1; /* reset the counter */ >> MPI_was_finalized = 1; >> return MPI_SUCCESS; >> } >> >> >> --Junchao Zhang >> >> >> On Sat, Jun 20, 2020 at 10:48 AM Sam Guo > wrote: >> Typo: I mean ?Assuming initializer is only needed once for entire session? >> >> On Saturday, June 20, 2020, Sam Guo > wrote: >> Assuming finalizer is only needed once for entire session(?), I can put initializer into the static block to call it once but where do I call finalizer? >> >> >> On Saturday, June 20, 2020, Junchao Zhang > wrote: >> The counter num_attr should be recycled. But first try to call PETSc initialize/Finalize only once to see it fixes the error. >> --Junchao Zhang >> >> >> On Sat, Jun 20, 2020 at 12:48 AM Sam Guo > wrote: >> To clarify, I call PETSc initialize and PETSc finalize everytime I call SLEPc: >> >> PetscInitializeNoPointers(argc,args,nullptr,nullptr); >> SlepcInitialize(&argc,&args,static_cast(nullptr),help); >> //calling slepc >> SlepcFinalize(); >> PetscFinalize(); >> >> >> On Fri, Jun 19, 2020 at 10:32 PM Sam Guo > wrote: >> Dear PETSc team, >> When I called SLEPc multiple time, I eventually got following error: >> >> MPI operation not supported by PETSc's sequential MPI wrappers >> [0]PETSC ERROR: #1 PetscInitialize() line 967 in ../../../petsc/src/sys/objects/pinit.c >> [0]PETSC ERROR: #2 SlepcInitialize() line 262 in ../../../slepc/src/sys/slepcinit.c >> [0]PETSC ERROR: #3 SlepcInitializeNoPointers() line 359 in ../../../slepc/src/sys/slepcinit.c >> PETSC ERROR: Logging has not been enabled. >> You might have forgotten to call PetscInitialize(). >> >> I debugged: it is because of following in petsc/src/sys/mpiuni/mpi.c >> >> if (num_attr >= MAX_ATTR) >> >> in function int MPI_Comm_create_keyval(MPI_Copy_function *copy_fn,MPI_Delete_function *delete_fn,int *keyval,void *extra_state) >> >> num_attr is declared static and keeps increasing every time MPI_Comm_create_keyval is called. >> >> I am using petsc 3.11.3 but found 3.13.2 has the same logic. >> >> Is this a bug or I didn't use it correctly? >> >> Thanks, >> Sam > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sam.guo at cd-adapco.com Fri Jun 26 15:25:12 2020 From: sam.guo at cd-adapco.com (Sam Guo) Date: Fri, 26 Jun 2020 13:25:12 -0700 Subject: [petsc-users] PETSc initialization error In-Reply-To: <9C60917A-5771-434D-BB65-D46750C49966@petsc.dev> References: <004EA4EB-815F-4EEE-8988-CA2179304157@petsc.dev> <9C60917A-5771-434D-BB65-D46750C49966@petsc.dev> Message-ID: Hi Barry, Thanks for the quick response. I will call PetscInitialize once and skip the PetscFinalize for now to avoid the crash. The crash is actually in PetscInitialize, not PetscFinalize. Thanks, Sam On Fri, Jun 26, 2020 at 1:21 PM Barry Smith wrote: > > Sam, > > You can skip PetscFinalize() so long as you only call PetscInitialize() > once. It is not desirable in general to skip the finalize because PETSc > can't free all its data structures and you cannot see the PETSc logging > information with -log_view but in terms of the code running correctly you > do not need to call PetscFinalize. > > If your code crashes in PetscFinalize() please send the full error > output and we can try to help you debug it. > > > Barry > > On Jun 26, 2020, at 3:14 PM, Sam Guo wrote: > > To clarify, we have a mpi wrapper (so we can switch to different mpi at > runtime). I compile petsc using our mpi wrapper. > If I just call PETSc initialize once without calling finallize, it is ok. > My question to you is that: can I skip finalize? > Our program calls mpi_finalize at end anyway. > > On Fri, Jun 26, 2020 at 1:09 PM Sam Guo wrote: > >> Hi Junchao, >> Attached please find the configure.log. >> I also attach the pinit.c which contains your patch (I am currently >> using 3.11.3. I've applied your patch to 3.11.3). Your patch fixes the >> serial version. The error now is about the parallel. >> Here is the error log: >> >> [1]PETSC ERROR: #1 PetscInitialize() line 969 in >> ../../../petsc/src/sys/objects/pinit.c >> [1]PETSC ERROR: #2 checkError() line 56 in >> ../../../physics/src/eigensolver/SLEPc.cpp >> [1]PETSC ERROR: #3 PetscInitialize() line 966 in >> ../../../petsc/src/sys/objects/pinit.c >> [1]PETSC ERROR: #4 SlepcInitialize() line 262 in >> ../../../slepc/src/sys/slepcinit.c >> [0]PETSC ERROR: #1 PetscInitialize() line 969 in >> ../../../petsc/src/sys/objects/pinit.c >> [0]PETSC ERROR: #2 checkError() line 56 in >> ../../../physics/src/eigensolver/SLEPc.cpp >> [0]PETSC ERROR: #3 PetscInitialize() line 966 in >> ../../../petsc/src/sys/objects/pinit.c >> [0]PETSC ERROR: #4 SlepcInitialize() line 262 in >> ../../../slepc/src/sys/slepcinit.c >> PETSC ERROR: Logging has not been enabled. >> You might have forgotten to call PetscInitialize(). >> PETSC ERROR: Logging has not been enabled. >> You might have forgotten to call PetscInitialize(). >> -------------------------------------------------------------------------- >> MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD >> with errorcode 56. >> >> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. >> You may or may not see output from other processes, depending on >> exactly when Open MPI kills them. >> >> Thanks, >> Sam >> >> On Thu, Jun 25, 2020 at 7:37 PM Junchao Zhang >> wrote: >> >>> Sam, >>> The MPI_Comm_create_keyval() error was fixed in maint/master. From >>> the error message, it seems you need to configure --with-log=1 >>> Otherwise, please send your full error stack trace and configure.log. >>> Thanks. >>> --Junchao Zhang >>> >>> >>> On Thu, Jun 25, 2020 at 2:18 PM Sam Guo wrote: >>> >>>> Hi Junchao, >>>> I now encountered the same error with parallel. I am wondering if >>>> there is a need for parallel fix as well. >>>> [1]PETSC ERROR: #1 PetscInitialize() line 969 in >>>> ../../../petsc/src/sys/objects/pinit.c >>>> PETSC ERROR: Logging has not been enabled. >>>> You might have forgotten to call PetscInitialize(). >>>> PETSC ERROR: Logging has not been enabled. >>>> You might have forgotten to call PetscInitialize(). >>>> >>>> On Sat, Jun 20, 2020 at 7:35 PM Sam Guo wrote: >>>> >>>>> Hi Junchao, >>>>> Your patch works. >>>>> >>>>> Thanks, >>>>> Sam >>>>> >>>>> On Sat, Jun 20, 2020 at 4:23 PM Junchao Zhang >>>>> wrote: >>>>> >>>>>> >>>>>> >>>>>> On Sat, Jun 20, 2020 at 12:24 PM Barry Smith >>>>>> wrote: >>>>>> >>>>>>> >>>>>>> Junchao, >>>>>>> >>>>>>> This is a good bug fix. It solves the problem when PETSc >>>>>>> initialize is called many times. >>>>>>> >>>>>>> There is another fix you can do to limit PETSc mpiuni running >>>>>>> out of attributes inside a single PETSc run: >>>>>>> >>>>>>> >>>>>>> int MPI_Comm_create_keyval(MPI_Copy_function >>>>>>> *copy_fn,MPI_Delete_function *delete_fn,int *keyval,void *extra_state) >>>>>>> { >>>>>>> >>>>>>> if (num_attr >= MAX_ATTR){ >>>>>>> for (i=0; i>>>>>> if (!attr_keyval[i].extra_state) { >>>>>>> >>>>>> attr_keyval[i].extra_state is provided by user (could be NULL). We >>>>>> can not rely on it. >>>>>> >>>>>>> /* reuse this slot */ >>>>>>> attr_keyval[i].extra_state = extra_state; >>>>>>> attr_keyval[i.]del = delete_fn; >>>>>>> *keyval = i; >>>>>>> return MPI_SUCCESS; >>>>>>> } >>>>>>> } >>>>>>> return MPIUni_Abort(MPI_COMM_WORLD,1); >>>>>>> } >>>>>>> return MPIUni_Abort(MPI_COMM_WORLD,1); >>>>>>> attr_keyval[num_attr].extra_state = extra_state; >>>>>>> attr_keyval[num_attr].del = delete_fn; >>>>>>> *keyval = num_attr++; >>>>>>> return MPI_SUCCESS; >>>>>>> } >>>>>>> >>>>>>> This will work if the user creates tons of attributes but is >>>>>>> constantly deleting some as they new ones. So long as the number >>>>>>> outstanding at one time is < MAX_ATTR) >>>>>>> >>>>>>> Barry >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Jun 20, 2020, at 10:54 AM, Junchao Zhang >>>>>>> wrote: >>>>>>> >>>>>>> I don't understand what your session means. Let's try this patch >>>>>>> >>>>>>> diff --git a/src/sys/mpiuni/mpi.c b/src/sys/mpiuni/mpi.c >>>>>>> index d559a513..c058265d 100644 >>>>>>> --- a/src/sys/mpiuni/mpi.c >>>>>>> +++ b/src/sys/mpiuni/mpi.c >>>>>>> @@ -283,6 +283,7 @@ int MPI_Finalize(void) >>>>>>> MPI_Comm_free(&comm); >>>>>>> comm = MPI_COMM_SELF; >>>>>>> MPI_Comm_free(&comm); >>>>>>> + num_attr = 1; /* reset the counter */ >>>>>>> MPI_was_finalized = 1; >>>>>>> return MPI_SUCCESS; >>>>>>> } >>>>>>> >>>>>>> >>>>>>> --Junchao Zhang >>>>>>> >>>>>>> >>>>>>> On Sat, Jun 20, 2020 at 10:48 AM Sam Guo >>>>>>> wrote: >>>>>>> >>>>>>>> Typo: I mean ?Assuming initializer is only needed once for entire >>>>>>>> session? >>>>>>>> >>>>>>>> On Saturday, June 20, 2020, Sam Guo wrote: >>>>>>>> >>>>>>>>> Assuming finalizer is only needed once for entire session(?), I >>>>>>>>> can put initializer into the static block to call it once but where do I >>>>>>>>> call finalizer? >>>>>>>>> >>>>>>>>> >>>>>>>>> On Saturday, June 20, 2020, Junchao Zhang >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> The counter num_attr should be recycled. But first try to call >>>>>>>>>> PETSc initialize/Finalize only once to see it fixes the error. >>>>>>>>>> --Junchao Zhang >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Sat, Jun 20, 2020 at 12:48 AM Sam Guo >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> To clarify, I call PETSc initialize and PETSc finalize >>>>>>>>>>> everytime I call SLEPc: >>>>>>>>>>> >>>>>>>>>>> PetscInitializeNoPointers(argc,args,nullptr,nullptr); >>>>>>>>>>> >>>>>>>>>>> SlepcInitialize(&argc,&args,static_cast(nullptr),help); >>>>>>>>>>> >>>>>>>>>>> //calling slepc >>>>>>>>>>> >>>>>>>>>>> SlepcFinalize(); >>>>>>>>>>> >>>>>>>>>>> PetscFinalize(); >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Fri, Jun 19, 2020 at 10:32 PM Sam Guo >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> Dear PETSc team, >>>>>>>>>>>> When I called SLEPc multiple time, I eventually got >>>>>>>>>>>> following error: >>>>>>>>>>>> >>>>>>>>>>>> MPI operation not supported by PETSc's sequential MPI wrappers >>>>>>>>>>>> [0]PETSC ERROR: #1 PetscInitialize() line 967 in >>>>>>>>>>>> ../../../petsc/src/sys/objects/pinit.c >>>>>>>>>>>> [0]PETSC ERROR: #2 SlepcInitialize() line 262 in >>>>>>>>>>>> ../../../slepc/src/sys/slepcinit.c >>>>>>>>>>>> [0]PETSC ERROR: #3 SlepcInitializeNoPointers() line 359 in >>>>>>>>>>>> ../../../slepc/src/sys/slepcinit.c >>>>>>>>>>>> PETSC ERROR: Logging has not been enabled. >>>>>>>>>>>> You might have forgotten to call PetscInitialize(). >>>>>>>>>>>> >>>>>>>>>>>> I debugged: it is because of following in >>>>>>>>>>>> petsc/src/sys/mpiuni/mpi.c >>>>>>>>>>>> >>>>>>>>>>>> if (num_attr >= MAX_ATTR) >>>>>>>>>>>> >>>>>>>>>>>> in function int MPI_Comm_create_keyval(MPI_Copy_function >>>>>>>>>>>> *copy_fn,MPI_Delete_function *delete_fn,int *keyval,void *extra_state) >>>>>>>>>>>> >>>>>>>>>>>> num_attr is declared static and keeps increasing every >>>>>>>>>>>> time MPI_Comm_create_keyval is called. >>>>>>>>>>>> >>>>>>>>>>>> I am using petsc 3.11.3 but found 3.13.2 has the same logic. >>>>>>>>>>>> >>>>>>>>>>>> Is this a bug or I didn't use it correctly? >>>>>>>>>>>> >>>>>>>>>>>> Thanks, >>>>>>>>>>>> Sam >>>>>>>>>>>> >>>>>>>>>>> >>>>>>> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Fri Jun 26 15:28:25 2020 From: bsmith at petsc.dev (Barry Smith) Date: Fri, 26 Jun 2020 15:28:25 -0500 Subject: [petsc-users] PETSc initialization error In-Reply-To: References: <004EA4EB-815F-4EEE-8988-CA2179304157@petsc.dev> <9C60917A-5771-434D-BB65-D46750C49966@petsc.dev> Message-ID: Ah, so you get the crash the second time you call PetscInitialize()? That is a problem because we do intend to support that capability (but you much call PetscFinalize() each time also). Barry > On Jun 26, 2020, at 3:25 PM, Sam Guo wrote: > > Hi Barry, > Thanks for the quick response. > I will call PetscInitialize once and skip the PetscFinalize for now to avoid the crash. The crash is actually in PetscInitialize, not PetscFinalize. > > Thanks, > Sam > > On Fri, Jun 26, 2020 at 1:21 PM Barry Smith > wrote: > > Sam, > > You can skip PetscFinalize() so long as you only call PetscInitialize() once. It is not desirable in general to skip the finalize because PETSc can't free all its data structures and you cannot see the PETSc logging information with -log_view but in terms of the code running correctly you do not need to call PetscFinalize. > > If your code crashes in PetscFinalize() please send the full error output and we can try to help you debug it. > > > Barry > >> On Jun 26, 2020, at 3:14 PM, Sam Guo > wrote: >> >> To clarify, we have a mpi wrapper (so we can switch to different mpi at runtime). I compile petsc using our mpi wrapper. >> If I just call PETSc initialize once without calling finallize, it is ok. My question to you is that: can I skip finalize? >> Our program calls mpi_finalize at end anyway. >> >> On Fri, Jun 26, 2020 at 1:09 PM Sam Guo > wrote: >> Hi Junchao, >> Attached please find the configure.log. >> I also attach the pinit.c which contains your patch (I am currently using 3.11.3. I've applied your patch to 3.11.3). Your patch fixes the serial version. The error now is about the parallel. >> Here is the error log: >> >> [1]PETSC ERROR: #1 PetscInitialize() line 969 in ../../../petsc/src/sys/objects/pinit.c >> [1]PETSC ERROR: #2 checkError() line 56 in ../../../physics/src/eigensolver/SLEPc.cpp >> [1]PETSC ERROR: #3 PetscInitialize() line 966 in ../../../petsc/src/sys/objects/pinit.c >> [1]PETSC ERROR: #4 SlepcInitialize() line 262 in ../../../slepc/src/sys/slepcinit.c >> [0]PETSC ERROR: #1 PetscInitialize() line 969 in ../../../petsc/src/sys/objects/pinit.c >> [0]PETSC ERROR: #2 checkError() line 56 in ../../../physics/src/eigensolver/SLEPc.cpp >> [0]PETSC ERROR: #3 PetscInitialize() line 966 in ../../../petsc/src/sys/objects/pinit.c >> [0]PETSC ERROR: #4 SlepcInitialize() line 262 in ../../../slepc/src/sys/slepcinit.c >> PETSC ERROR: Logging has not been enabled. >> You might have forgotten to call PetscInitialize(). >> PETSC ERROR: Logging has not been enabled. >> You might have forgotten to call PetscInitialize(). >> -------------------------------------------------------------------------- >> MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD >> with errorcode 56. >> >> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. >> You may or may not see output from other processes, depending on >> exactly when Open MPI kills them. >> >> Thanks, >> Sam >> >> On Thu, Jun 25, 2020 at 7:37 PM Junchao Zhang > wrote: >> Sam, >> The MPI_Comm_create_keyval() error was fixed in maint/master. From the error message, it seems you need to configure --with-log=1 >> Otherwise, please send your full error stack trace and configure.log. >> Thanks. >> --Junchao Zhang >> >> >> On Thu, Jun 25, 2020 at 2:18 PM Sam Guo > wrote: >> Hi Junchao, >> I now encountered the same error with parallel. I am wondering if there is a need for parallel fix as well. >> [1]PETSC ERROR: #1 PetscInitialize() line 969 in ../../../petsc/src/sys/objects/pinit.c >> PETSC ERROR: Logging has not been enabled. >> You might have forgotten to call PetscInitialize(). >> PETSC ERROR: Logging has not been enabled. >> You might have forgotten to call PetscInitialize(). >> >> On Sat, Jun 20, 2020 at 7:35 PM Sam Guo > wrote: >> Hi Junchao, >> Your patch works. >> >> Thanks, >> Sam >> >> On Sat, Jun 20, 2020 at 4:23 PM Junchao Zhang > wrote: >> >> >> On Sat, Jun 20, 2020 at 12:24 PM Barry Smith > wrote: >> >> Junchao, >> >> This is a good bug fix. It solves the problem when PETSc initialize is called many times. >> >> There is another fix you can do to limit PETSc mpiuni running out of attributes inside a single PETSc run: >> >> >> int MPI_Comm_create_keyval(MPI_Copy_function *copy_fn,MPI_Delete_function *delete_fn,int *keyval,void *extra_state) >> { >> >> if (num_attr >= MAX_ATTR){ >> for (i=0; i> if (!attr_keyval[i].extra_state) { >> attr_keyval[i].extra_state is provided by user (could be NULL). We can not rely on it. >> /* reuse this slot */ >> attr_keyval[i].extra_state = extra_state; >> attr_keyval[i.]del = delete_fn; >> *keyval = i; >> return MPI_SUCCESS; >> } >> } >> return MPIUni_Abort(MPI_COMM_WORLD,1); >> } >> return MPIUni_Abort(MPI_COMM_WORLD,1); >> attr_keyval[num_attr].extra_state = extra_state; >> attr_keyval[num_attr].del = delete_fn; >> *keyval = num_attr++; >> return MPI_SUCCESS; >> } >> >> This will work if the user creates tons of attributes but is constantly deleting some as they new ones. So long as the number outstanding at one time is < MAX_ATTR) >> >> Barry >> >> >> >> >> >>> On Jun 20, 2020, at 10:54 AM, Junchao Zhang > wrote: >>> >>> I don't understand what your session means. Let's try this patch >>> >>> diff --git a/src/sys/mpiuni/mpi.c b/src/sys/mpiuni/mpi.c >>> index d559a513..c058265d 100644 >>> --- a/src/sys/mpiuni/mpi.c >>> +++ b/src/sys/mpiuni/mpi.c >>> @@ -283,6 +283,7 @@ int MPI_Finalize(void) >>> MPI_Comm_free(&comm); >>> comm = MPI_COMM_SELF; >>> MPI_Comm_free(&comm); >>> + num_attr = 1; /* reset the counter */ >>> MPI_was_finalized = 1; >>> return MPI_SUCCESS; >>> } >>> >>> >>> --Junchao Zhang >>> >>> >>> On Sat, Jun 20, 2020 at 10:48 AM Sam Guo > wrote: >>> Typo: I mean ?Assuming initializer is only needed once for entire session? >>> >>> On Saturday, June 20, 2020, Sam Guo > wrote: >>> Assuming finalizer is only needed once for entire session(?), I can put initializer into the static block to call it once but where do I call finalizer? >>> >>> >>> On Saturday, June 20, 2020, Junchao Zhang > wrote: >>> The counter num_attr should be recycled. But first try to call PETSc initialize/Finalize only once to see it fixes the error. >>> --Junchao Zhang >>> >>> >>> On Sat, Jun 20, 2020 at 12:48 AM Sam Guo > wrote: >>> To clarify, I call PETSc initialize and PETSc finalize everytime I call SLEPc: >>> >>> PetscInitializeNoPointers(argc,args,nullptr,nullptr); >>> SlepcInitialize(&argc,&args,static_cast(nullptr),help); >>> //calling slepc >>> SlepcFinalize(); >>> PetscFinalize(); >>> >>> >>> On Fri, Jun 19, 2020 at 10:32 PM Sam Guo > wrote: >>> Dear PETSc team, >>> When I called SLEPc multiple time, I eventually got following error: >>> >>> MPI operation not supported by PETSc's sequential MPI wrappers >>> [0]PETSC ERROR: #1 PetscInitialize() line 967 in ../../../petsc/src/sys/objects/pinit.c >>> [0]PETSC ERROR: #2 SlepcInitialize() line 262 in ../../../slepc/src/sys/slepcinit.c >>> [0]PETSC ERROR: #3 SlepcInitializeNoPointers() line 359 in ../../../slepc/src/sys/slepcinit.c >>> PETSC ERROR: Logging has not been enabled. >>> You might have forgotten to call PetscInitialize(). >>> >>> I debugged: it is because of following in petsc/src/sys/mpiuni/mpi.c >>> >>> if (num_attr >= MAX_ATTR) >>> >>> in function int MPI_Comm_create_keyval(MPI_Copy_function *copy_fn,MPI_Delete_function *delete_fn,int *keyval,void *extra_state) >>> >>> num_attr is declared static and keeps increasing every time MPI_Comm_create_keyval is called. >>> >>> I am using petsc 3.11.3 but found 3.13.2 has the same logic. >>> >>> Is this a bug or I didn't use it correctly? >>> >>> Thanks, >>> Sam >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sam.guo at cd-adapco.com Fri Jun 26 15:33:14 2020 From: sam.guo at cd-adapco.com (Sam Guo) Date: Fri, 26 Jun 2020 13:33:14 -0700 Subject: [petsc-users] PETSc initialization error In-Reply-To: References: <004EA4EB-815F-4EEE-8988-CA2179304157@petsc.dev> <9C60917A-5771-434D-BB65-D46750C49966@petsc.dev> Message-ID: I get the crash after calling Initialize/Finalize multiple times. Junchao fixed the bug for serial but parallel still crashes. On Fri, Jun 26, 2020 at 1:28 PM Barry Smith wrote: > > Ah, so you get the crash the second time you call PetscInitialize()? > That is a problem because we do intend to support that capability (but you > much call PetscFinalize() each time also). > > Barry > > > On Jun 26, 2020, at 3:25 PM, Sam Guo wrote: > > Hi Barry, > Thanks for the quick response. > I will call PetscInitialize once and skip the PetscFinalize for now to > avoid the crash. The crash is actually in PetscInitialize, not > PetscFinalize. > > Thanks, > Sam > > On Fri, Jun 26, 2020 at 1:21 PM Barry Smith wrote: > >> >> Sam, >> >> You can skip PetscFinalize() so long as you only call PetscInitialize() >> once. It is not desirable in general to skip the finalize because PETSc >> can't free all its data structures and you cannot see the PETSc logging >> information with -log_view but in terms of the code running correctly you >> do not need to call PetscFinalize. >> >> If your code crashes in PetscFinalize() please send the full error >> output and we can try to help you debug it. >> >> >> Barry >> >> On Jun 26, 2020, at 3:14 PM, Sam Guo wrote: >> >> To clarify, we have a mpi wrapper (so we can switch to different mpi at >> runtime). I compile petsc using our mpi wrapper. >> If I just call PETSc initialize once without calling finallize, it is ok. >> My question to you is that: can I skip finalize? >> Our program calls mpi_finalize at end anyway. >> >> On Fri, Jun 26, 2020 at 1:09 PM Sam Guo wrote: >> >>> Hi Junchao, >>> Attached please find the configure.log. >>> I also attach the pinit.c which contains your patch (I am currently >>> using 3.11.3. I've applied your patch to 3.11.3). Your patch fixes the >>> serial version. The error now is about the parallel. >>> Here is the error log: >>> >>> [1]PETSC ERROR: #1 PetscInitialize() line 969 in >>> ../../../petsc/src/sys/objects/pinit.c >>> [1]PETSC ERROR: #2 checkError() line 56 in >>> ../../../physics/src/eigensolver/SLEPc.cpp >>> [1]PETSC ERROR: #3 PetscInitialize() line 966 in >>> ../../../petsc/src/sys/objects/pinit.c >>> [1]PETSC ERROR: #4 SlepcInitialize() line 262 in >>> ../../../slepc/src/sys/slepcinit.c >>> [0]PETSC ERROR: #1 PetscInitialize() line 969 in >>> ../../../petsc/src/sys/objects/pinit.c >>> [0]PETSC ERROR: #2 checkError() line 56 in >>> ../../../physics/src/eigensolver/SLEPc.cpp >>> [0]PETSC ERROR: #3 PetscInitialize() line 966 in >>> ../../../petsc/src/sys/objects/pinit.c >>> [0]PETSC ERROR: #4 SlepcInitialize() line 262 in >>> ../../../slepc/src/sys/slepcinit.c >>> PETSC ERROR: Logging has not been enabled. >>> You might have forgotten to call PetscInitialize(). >>> PETSC ERROR: Logging has not been enabled. >>> You might have forgotten to call PetscInitialize(). >>> >>> -------------------------------------------------------------------------- >>> MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD >>> with errorcode 56. >>> >>> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. >>> You may or may not see output from other processes, depending on >>> exactly when Open MPI kills them. >>> >>> Thanks, >>> Sam >>> >>> On Thu, Jun 25, 2020 at 7:37 PM Junchao Zhang >>> wrote: >>> >>>> Sam, >>>> The MPI_Comm_create_keyval() error was fixed in maint/master. From >>>> the error message, it seems you need to configure --with-log=1 >>>> Otherwise, please send your full error stack trace and configure.log. >>>> Thanks. >>>> --Junchao Zhang >>>> >>>> >>>> On Thu, Jun 25, 2020 at 2:18 PM Sam Guo wrote: >>>> >>>>> Hi Junchao, >>>>> I now encountered the same error with parallel. I am wondering if >>>>> there is a need for parallel fix as well. >>>>> [1]PETSC ERROR: #1 PetscInitialize() line 969 in >>>>> ../../../petsc/src/sys/objects/pinit.c >>>>> PETSC ERROR: Logging has not been enabled. >>>>> You might have forgotten to call PetscInitialize(). >>>>> PETSC ERROR: Logging has not been enabled. >>>>> You might have forgotten to call PetscInitialize(). >>>>> >>>>> On Sat, Jun 20, 2020 at 7:35 PM Sam Guo wrote: >>>>> >>>>>> Hi Junchao, >>>>>> Your patch works. >>>>>> >>>>>> Thanks, >>>>>> Sam >>>>>> >>>>>> On Sat, Jun 20, 2020 at 4:23 PM Junchao Zhang < >>>>>> junchao.zhang at gmail.com> wrote: >>>>>> >>>>>>> >>>>>>> >>>>>>> On Sat, Jun 20, 2020 at 12:24 PM Barry Smith >>>>>>> wrote: >>>>>>> >>>>>>>> >>>>>>>> Junchao, >>>>>>>> >>>>>>>> This is a good bug fix. It solves the problem when PETSc >>>>>>>> initialize is called many times. >>>>>>>> >>>>>>>> There is another fix you can do to limit PETSc mpiuni running >>>>>>>> out of attributes inside a single PETSc run: >>>>>>>> >>>>>>>> >>>>>>>> int MPI_Comm_create_keyval(MPI_Copy_function >>>>>>>> *copy_fn,MPI_Delete_function *delete_fn,int *keyval,void *extra_state) >>>>>>>> { >>>>>>>> >>>>>>>> if (num_attr >= MAX_ATTR){ >>>>>>>> for (i=0; i>>>>>>> if (!attr_keyval[i].extra_state) { >>>>>>>> >>>>>>> attr_keyval[i].extra_state is provided by user (could be NULL). We >>>>>>> can not rely on it. >>>>>>> >>>>>>>> /* reuse this slot */ >>>>>>>> attr_keyval[i].extra_state = extra_state; >>>>>>>> attr_keyval[i.]del = delete_fn; >>>>>>>> *keyval = i; >>>>>>>> return MPI_SUCCESS; >>>>>>>> } >>>>>>>> } >>>>>>>> return MPIUni_Abort(MPI_COMM_WORLD,1); >>>>>>>> } >>>>>>>> return MPIUni_Abort(MPI_COMM_WORLD,1); >>>>>>>> attr_keyval[num_attr].extra_state = extra_state; >>>>>>>> attr_keyval[num_attr].del = delete_fn; >>>>>>>> *keyval = num_attr++; >>>>>>>> return MPI_SUCCESS; >>>>>>>> } >>>>>>>> >>>>>>>> This will work if the user creates tons of attributes but is >>>>>>>> constantly deleting some as they new ones. So long as the number >>>>>>>> outstanding at one time is < MAX_ATTR) >>>>>>>> >>>>>>>> Barry >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Jun 20, 2020, at 10:54 AM, Junchao Zhang < >>>>>>>> junchao.zhang at gmail.com> wrote: >>>>>>>> >>>>>>>> I don't understand what your session means. Let's try this patch >>>>>>>> >>>>>>>> diff --git a/src/sys/mpiuni/mpi.c b/src/sys/mpiuni/mpi.c >>>>>>>> index d559a513..c058265d 100644 >>>>>>>> --- a/src/sys/mpiuni/mpi.c >>>>>>>> +++ b/src/sys/mpiuni/mpi.c >>>>>>>> @@ -283,6 +283,7 @@ int MPI_Finalize(void) >>>>>>>> MPI_Comm_free(&comm); >>>>>>>> comm = MPI_COMM_SELF; >>>>>>>> MPI_Comm_free(&comm); >>>>>>>> + num_attr = 1; /* reset the counter */ >>>>>>>> MPI_was_finalized = 1; >>>>>>>> return MPI_SUCCESS; >>>>>>>> } >>>>>>>> >>>>>>>> >>>>>>>> --Junchao Zhang >>>>>>>> >>>>>>>> >>>>>>>> On Sat, Jun 20, 2020 at 10:48 AM Sam Guo >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Typo: I mean ?Assuming initializer is only needed once for entire >>>>>>>>> session? >>>>>>>>> >>>>>>>>> On Saturday, June 20, 2020, Sam Guo wrote: >>>>>>>>> >>>>>>>>>> Assuming finalizer is only needed once for entire session(?), I >>>>>>>>>> can put initializer into the static block to call it once but where do I >>>>>>>>>> call finalizer? >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Saturday, June 20, 2020, Junchao Zhang < >>>>>>>>>> junchao.zhang at gmail.com> wrote: >>>>>>>>>> >>>>>>>>>>> The counter num_attr should be recycled. But first try to call >>>>>>>>>>> PETSc initialize/Finalize only once to see it fixes the error. >>>>>>>>>>> --Junchao Zhang >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Sat, Jun 20, 2020 at 12:48 AM Sam Guo >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> To clarify, I call PETSc initialize and PETSc finalize >>>>>>>>>>>> everytime I call SLEPc: >>>>>>>>>>>> >>>>>>>>>>>> PetscInitializeNoPointers(argc,args,nullptr,nullptr); >>>>>>>>>>>> >>>>>>>>>>>> SlepcInitialize(&argc,&args,static_cast(nullptr),help); >>>>>>>>>>>> >>>>>>>>>>>> //calling slepc >>>>>>>>>>>> >>>>>>>>>>>> SlepcFinalize(); >>>>>>>>>>>> >>>>>>>>>>>> PetscFinalize(); >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Fri, Jun 19, 2020 at 10:32 PM Sam Guo >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Dear PETSc team, >>>>>>>>>>>>> When I called SLEPc multiple time, I eventually got >>>>>>>>>>>>> following error: >>>>>>>>>>>>> >>>>>>>>>>>>> MPI operation not supported by PETSc's sequential MPI wrappers >>>>>>>>>>>>> [0]PETSC ERROR: #1 PetscInitialize() line 967 in >>>>>>>>>>>>> ../../../petsc/src/sys/objects/pinit.c >>>>>>>>>>>>> [0]PETSC ERROR: #2 SlepcInitialize() line 262 in >>>>>>>>>>>>> ../../../slepc/src/sys/slepcinit.c >>>>>>>>>>>>> [0]PETSC ERROR: #3 SlepcInitializeNoPointers() line 359 in >>>>>>>>>>>>> ../../../slepc/src/sys/slepcinit.c >>>>>>>>>>>>> PETSC ERROR: Logging has not been enabled. >>>>>>>>>>>>> You might have forgotten to call PetscInitialize(). >>>>>>>>>>>>> >>>>>>>>>>>>> I debugged: it is because of following in >>>>>>>>>>>>> petsc/src/sys/mpiuni/mpi.c >>>>>>>>>>>>> >>>>>>>>>>>>> if (num_attr >= MAX_ATTR) >>>>>>>>>>>>> >>>>>>>>>>>>> in function int MPI_Comm_create_keyval(MPI_Copy_function >>>>>>>>>>>>> *copy_fn,MPI_Delete_function *delete_fn,int *keyval,void *extra_state) >>>>>>>>>>>>> >>>>>>>>>>>>> num_attr is declared static and keeps increasing every >>>>>>>>>>>>> time MPI_Comm_create_keyval is called. >>>>>>>>>>>>> >>>>>>>>>>>>> I am using petsc 3.11.3 but found 3.13.2 has the same logic. >>>>>>>>>>>>> >>>>>>>>>>>>> Is this a bug or I didn't use it correctly? >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks, >>>>>>>>>>>>> Sam >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sam.guo at cd-adapco.com Fri Jun 26 15:36:07 2020 From: sam.guo at cd-adapco.com (Sam Guo) Date: Fri, 26 Jun 2020 13:36:07 -0700 Subject: [petsc-users] PETSc initialization error In-Reply-To: References: <004EA4EB-815F-4EEE-8988-CA2179304157@petsc.dev> <9C60917A-5771-434D-BB65-D46750C49966@petsc.dev> Message-ID: One work around for me is to call PetscInitialize once for my entire program and skip PetscFinalize (since I don't have a good place to call PetscFinalize before ending the program). On Fri, Jun 26, 2020 at 1:33 PM Sam Guo wrote: > I get the crash after calling Initialize/Finalize multiple times. Junchao > fixed the bug for serial but parallel still crashes. > > On Fri, Jun 26, 2020 at 1:28 PM Barry Smith wrote: > >> >> Ah, so you get the crash the second time you call PetscInitialize()? >> That is a problem because we do intend to support that capability (but you >> much call PetscFinalize() each time also). >> >> Barry >> >> >> On Jun 26, 2020, at 3:25 PM, Sam Guo wrote: >> >> Hi Barry, >> Thanks for the quick response. >> I will call PetscInitialize once and skip the PetscFinalize for now to >> avoid the crash. The crash is actually in PetscInitialize, not >> PetscFinalize. >> >> Thanks, >> Sam >> >> On Fri, Jun 26, 2020 at 1:21 PM Barry Smith wrote: >> >>> >>> Sam, >>> >>> You can skip PetscFinalize() so long as you only call >>> PetscInitialize() once. It is not desirable in general to skip the finalize >>> because PETSc can't free all its data structures and you cannot see the >>> PETSc logging information with -log_view but in terms of the code running >>> correctly you do not need to call PetscFinalize. >>> >>> If your code crashes in PetscFinalize() please send the full error >>> output and we can try to help you debug it. >>> >>> >>> Barry >>> >>> On Jun 26, 2020, at 3:14 PM, Sam Guo wrote: >>> >>> To clarify, we have a mpi wrapper (so we can switch to different mpi at >>> runtime). I compile petsc using our mpi wrapper. >>> If I just call PETSc initialize once without calling finallize, it is >>> ok. My question to you is that: can I skip finalize? >>> Our program calls mpi_finalize at end anyway. >>> >>> On Fri, Jun 26, 2020 at 1:09 PM Sam Guo wrote: >>> >>>> Hi Junchao, >>>> Attached please find the configure.log. >>>> I also attach the pinit.c which contains your patch (I am currently >>>> using 3.11.3. I've applied your patch to 3.11.3). Your patch fixes the >>>> serial version. The error now is about the parallel. >>>> Here is the error log: >>>> >>>> [1]PETSC ERROR: #1 PetscInitialize() line 969 in >>>> ../../../petsc/src/sys/objects/pinit.c >>>> [1]PETSC ERROR: #2 checkError() line 56 in >>>> ../../../physics/src/eigensolver/SLEPc.cpp >>>> [1]PETSC ERROR: #3 PetscInitialize() line 966 in >>>> ../../../petsc/src/sys/objects/pinit.c >>>> [1]PETSC ERROR: #4 SlepcInitialize() line 262 in >>>> ../../../slepc/src/sys/slepcinit.c >>>> [0]PETSC ERROR: #1 PetscInitialize() line 969 in >>>> ../../../petsc/src/sys/objects/pinit.c >>>> [0]PETSC ERROR: #2 checkError() line 56 in >>>> ../../../physics/src/eigensolver/SLEPc.cpp >>>> [0]PETSC ERROR: #3 PetscInitialize() line 966 in >>>> ../../../petsc/src/sys/objects/pinit.c >>>> [0]PETSC ERROR: #4 SlepcInitialize() line 262 in >>>> ../../../slepc/src/sys/slepcinit.c >>>> PETSC ERROR: Logging has not been enabled. >>>> You might have forgotten to call PetscInitialize(). >>>> PETSC ERROR: Logging has not been enabled. >>>> You might have forgotten to call PetscInitialize(). >>>> >>>> -------------------------------------------------------------------------- >>>> MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD >>>> with errorcode 56. >>>> >>>> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. >>>> You may or may not see output from other processes, depending on >>>> exactly when Open MPI kills them. >>>> >>>> Thanks, >>>> Sam >>>> >>>> On Thu, Jun 25, 2020 at 7:37 PM Junchao Zhang >>>> wrote: >>>> >>>>> Sam, >>>>> The MPI_Comm_create_keyval() error was fixed in maint/master. From >>>>> the error message, it seems you need to configure --with-log=1 >>>>> Otherwise, please send your full error stack trace and >>>>> configure.log. >>>>> Thanks. >>>>> --Junchao Zhang >>>>> >>>>> >>>>> On Thu, Jun 25, 2020 at 2:18 PM Sam Guo wrote: >>>>> >>>>>> Hi Junchao, >>>>>> I now encountered the same error with parallel. I am wondering if >>>>>> there is a need for parallel fix as well. >>>>>> [1]PETSC ERROR: #1 PetscInitialize() line 969 in >>>>>> ../../../petsc/src/sys/objects/pinit.c >>>>>> PETSC ERROR: Logging has not been enabled. >>>>>> You might have forgotten to call PetscInitialize(). >>>>>> PETSC ERROR: Logging has not been enabled. >>>>>> You might have forgotten to call PetscInitialize(). >>>>>> >>>>>> On Sat, Jun 20, 2020 at 7:35 PM Sam Guo >>>>>> wrote: >>>>>> >>>>>>> Hi Junchao, >>>>>>> Your patch works. >>>>>>> >>>>>>> Thanks, >>>>>>> Sam >>>>>>> >>>>>>> On Sat, Jun 20, 2020 at 4:23 PM Junchao Zhang < >>>>>>> junchao.zhang at gmail.com> wrote: >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Sat, Jun 20, 2020 at 12:24 PM Barry Smith >>>>>>>> wrote: >>>>>>>> >>>>>>>>> >>>>>>>>> Junchao, >>>>>>>>> >>>>>>>>> This is a good bug fix. It solves the problem when PETSc >>>>>>>>> initialize is called many times. >>>>>>>>> >>>>>>>>> There is another fix you can do to limit PETSc mpiuni running >>>>>>>>> out of attributes inside a single PETSc run: >>>>>>>>> >>>>>>>>> >>>>>>>>> int MPI_Comm_create_keyval(MPI_Copy_function >>>>>>>>> *copy_fn,MPI_Delete_function *delete_fn,int *keyval,void *extra_state) >>>>>>>>> { >>>>>>>>> >>>>>>>>> if (num_attr >= MAX_ATTR){ >>>>>>>>> for (i=0; i>>>>>>>> if (!attr_keyval[i].extra_state) { >>>>>>>>> >>>>>>>> attr_keyval[i].extra_state is provided by user (could be NULL). We >>>>>>>> can not rely on it. >>>>>>>> >>>>>>>>> /* reuse this slot */ >>>>>>>>> attr_keyval[i].extra_state = extra_state; >>>>>>>>> attr_keyval[i.]del = delete_fn; >>>>>>>>> *keyval = i; >>>>>>>>> return MPI_SUCCESS; >>>>>>>>> } >>>>>>>>> } >>>>>>>>> return MPIUni_Abort(MPI_COMM_WORLD,1); >>>>>>>>> } >>>>>>>>> return MPIUni_Abort(MPI_COMM_WORLD,1); >>>>>>>>> attr_keyval[num_attr].extra_state = extra_state; >>>>>>>>> attr_keyval[num_attr].del = delete_fn; >>>>>>>>> *keyval = num_attr++; >>>>>>>>> return MPI_SUCCESS; >>>>>>>>> } >>>>>>>>> >>>>>>>>> This will work if the user creates tons of attributes but is >>>>>>>>> constantly deleting some as they new ones. So long as the number >>>>>>>>> outstanding at one time is < MAX_ATTR) >>>>>>>>> >>>>>>>>> Barry >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Jun 20, 2020, at 10:54 AM, Junchao Zhang < >>>>>>>>> junchao.zhang at gmail.com> wrote: >>>>>>>>> >>>>>>>>> I don't understand what your session means. Let's try this patch >>>>>>>>> >>>>>>>>> diff --git a/src/sys/mpiuni/mpi.c b/src/sys/mpiuni/mpi.c >>>>>>>>> index d559a513..c058265d 100644 >>>>>>>>> --- a/src/sys/mpiuni/mpi.c >>>>>>>>> +++ b/src/sys/mpiuni/mpi.c >>>>>>>>> @@ -283,6 +283,7 @@ int MPI_Finalize(void) >>>>>>>>> MPI_Comm_free(&comm); >>>>>>>>> comm = MPI_COMM_SELF; >>>>>>>>> MPI_Comm_free(&comm); >>>>>>>>> + num_attr = 1; /* reset the counter */ >>>>>>>>> MPI_was_finalized = 1; >>>>>>>>> return MPI_SUCCESS; >>>>>>>>> } >>>>>>>>> >>>>>>>>> >>>>>>>>> --Junchao Zhang >>>>>>>>> >>>>>>>>> >>>>>>>>> On Sat, Jun 20, 2020 at 10:48 AM Sam Guo >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Typo: I mean ?Assuming initializer is only needed once for entire >>>>>>>>>> session? >>>>>>>>>> >>>>>>>>>> On Saturday, June 20, 2020, Sam Guo >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Assuming finalizer is only needed once for entire session(?), I >>>>>>>>>>> can put initializer into the static block to call it once but where do I >>>>>>>>>>> call finalizer? >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Saturday, June 20, 2020, Junchao Zhang < >>>>>>>>>>> junchao.zhang at gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> The counter num_attr should be recycled. But first try to call >>>>>>>>>>>> PETSc initialize/Finalize only once to see it fixes the error. >>>>>>>>>>>> --Junchao Zhang >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Sat, Jun 20, 2020 at 12:48 AM Sam Guo >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> To clarify, I call PETSc initialize and PETSc finalize >>>>>>>>>>>>> everytime I call SLEPc: >>>>>>>>>>>>> >>>>>>>>>>>>> PetscInitializeNoPointers(argc,args,nullptr,nullptr); >>>>>>>>>>>>> >>>>>>>>>>>>> SlepcInitialize(&argc,&args,static_cast(nullptr),help); >>>>>>>>>>>>> >>>>>>>>>>>>> //calling slepc >>>>>>>>>>>>> >>>>>>>>>>>>> SlepcFinalize(); >>>>>>>>>>>>> >>>>>>>>>>>>> PetscFinalize(); >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Fri, Jun 19, 2020 at 10:32 PM Sam Guo < >>>>>>>>>>>>> sam.guo at cd-adapco.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Dear PETSc team, >>>>>>>>>>>>>> When I called SLEPc multiple time, I eventually got >>>>>>>>>>>>>> following error: >>>>>>>>>>>>>> >>>>>>>>>>>>>> MPI operation not supported by PETSc's sequential MPI wrappers >>>>>>>>>>>>>> [0]PETSC ERROR: #1 PetscInitialize() line 967 in >>>>>>>>>>>>>> ../../../petsc/src/sys/objects/pinit.c >>>>>>>>>>>>>> [0]PETSC ERROR: #2 SlepcInitialize() line 262 in >>>>>>>>>>>>>> ../../../slepc/src/sys/slepcinit.c >>>>>>>>>>>>>> [0]PETSC ERROR: #3 SlepcInitializeNoPointers() line 359 in >>>>>>>>>>>>>> ../../../slepc/src/sys/slepcinit.c >>>>>>>>>>>>>> PETSC ERROR: Logging has not been enabled. >>>>>>>>>>>>>> You might have forgotten to call PetscInitialize(). >>>>>>>>>>>>>> >>>>>>>>>>>>>> I debugged: it is because of following in >>>>>>>>>>>>>> petsc/src/sys/mpiuni/mpi.c >>>>>>>>>>>>>> >>>>>>>>>>>>>> if (num_attr >= MAX_ATTR) >>>>>>>>>>>>>> >>>>>>>>>>>>>> in function int MPI_Comm_create_keyval(MPI_Copy_function >>>>>>>>>>>>>> *copy_fn,MPI_Delete_function *delete_fn,int *keyval,void *extra_state) >>>>>>>>>>>>>> >>>>>>>>>>>>>> num_attr is declared static and keeps increasing every >>>>>>>>>>>>>> time MPI_Comm_create_keyval is called. >>>>>>>>>>>>>> >>>>>>>>>>>>>> I am using petsc 3.11.3 but found 3.13.2 has the same logic. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Is this a bug or I didn't use it correctly? >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>> Sam >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>> >>> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From junchao.zhang at gmail.com Fri Jun 26 17:33:32 2020 From: junchao.zhang at gmail.com (Junchao Zhang) Date: Fri, 26 Jun 2020 17:33:32 -0500 Subject: [petsc-users] PETSc initialization error In-Reply-To: References: <004EA4EB-815F-4EEE-8988-CA2179304157@petsc.dev> <9C60917A-5771-434D-BB65-D46750C49966@petsc.dev> Message-ID: Sam, Please discard the origin patch I sent you. A better fix is already in maint/master. An test is at src/sys/tests/ex53.c I modified that test at the end with for (i=0; i<500; i++) { ierr = PetscInitializeNoPointers(argc,argv,NULL,help);if (ierr) return ierr; ierr = SlepcInitialize(&argc,&argv,NULL,help);if (ierr) return ierr; ierr = SlepcFinalize();if (ierr) return ierr; ierr = PetscFinalize();if (ierr) return ierr; } then I ran it with multiple mpi ranks and it ran correctly. So try your program with petsc master first. If not work, see if you can come up with a test example for us. Thanks. --Junchao Zhang On Fri, Jun 26, 2020 at 3:37 PM Sam Guo wrote: > One work around for me is to call PetscInitialize once for my entire > program and skip PetscFinalize (since I don't have a good place to call > PetscFinalize before ending the program). > > On Fri, Jun 26, 2020 at 1:33 PM Sam Guo wrote: > >> I get the crash after calling Initialize/Finalize multiple times. Junchao >> fixed the bug for serial but parallel still crashes. >> >> On Fri, Jun 26, 2020 at 1:28 PM Barry Smith wrote: >> >>> >>> Ah, so you get the crash the second time you call PetscInitialize()? >>> That is a problem because we do intend to support that capability (but you >>> much call PetscFinalize() each time also). >>> >>> Barry >>> >>> >>> On Jun 26, 2020, at 3:25 PM, Sam Guo wrote: >>> >>> Hi Barry, >>> Thanks for the quick response. >>> I will call PetscInitialize once and skip the PetscFinalize for now >>> to avoid the crash. The crash is actually in PetscInitialize, not >>> PetscFinalize. >>> >>> Thanks, >>> Sam >>> >>> On Fri, Jun 26, 2020 at 1:21 PM Barry Smith wrote: >>> >>>> >>>> Sam, >>>> >>>> You can skip PetscFinalize() so long as you only call >>>> PetscInitialize() once. It is not desirable in general to skip the finalize >>>> because PETSc can't free all its data structures and you cannot see the >>>> PETSc logging information with -log_view but in terms of the code running >>>> correctly you do not need to call PetscFinalize. >>>> >>>> If your code crashes in PetscFinalize() please send the full error >>>> output and we can try to help you debug it. >>>> >>>> >>>> Barry >>>> >>>> On Jun 26, 2020, at 3:14 PM, Sam Guo wrote: >>>> >>>> To clarify, we have a mpi wrapper (so we can switch to different mpi at >>>> runtime). I compile petsc using our mpi wrapper. >>>> If I just call PETSc initialize once without calling finallize, it is >>>> ok. My question to you is that: can I skip finalize? >>>> Our program calls mpi_finalize at end anyway. >>>> >>>> On Fri, Jun 26, 2020 at 1:09 PM Sam Guo wrote: >>>> >>>>> Hi Junchao, >>>>> Attached please find the configure.log. >>>>> I also attach the pinit.c which contains your patch (I am currently >>>>> using 3.11.3. I've applied your patch to 3.11.3). Your patch fixes the >>>>> serial version. The error now is about the parallel. >>>>> Here is the error log: >>>>> >>>>> [1]PETSC ERROR: #1 PetscInitialize() line 969 in >>>>> ../../../petsc/src/sys/objects/pinit.c >>>>> [1]PETSC ERROR: #2 checkError() line 56 in >>>>> ../../../physics/src/eigensolver/SLEPc.cpp >>>>> [1]PETSC ERROR: #3 PetscInitialize() line 966 in >>>>> ../../../petsc/src/sys/objects/pinit.c >>>>> [1]PETSC ERROR: #4 SlepcInitialize() line 262 in >>>>> ../../../slepc/src/sys/slepcinit.c >>>>> [0]PETSC ERROR: #1 PetscInitialize() line 969 in >>>>> ../../../petsc/src/sys/objects/pinit.c >>>>> [0]PETSC ERROR: #2 checkError() line 56 in >>>>> ../../../physics/src/eigensolver/SLEPc.cpp >>>>> [0]PETSC ERROR: #3 PetscInitialize() line 966 in >>>>> ../../../petsc/src/sys/objects/pinit.c >>>>> [0]PETSC ERROR: #4 SlepcInitialize() line 262 in >>>>> ../../../slepc/src/sys/slepcinit.c >>>>> PETSC ERROR: Logging has not been enabled. >>>>> You might have forgotten to call PetscInitialize(). >>>>> PETSC ERROR: Logging has not been enabled. >>>>> You might have forgotten to call PetscInitialize(). >>>>> >>>>> -------------------------------------------------------------------------- >>>>> MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD >>>>> with errorcode 56. >>>>> >>>>> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. >>>>> You may or may not see output from other processes, depending on >>>>> exactly when Open MPI kills them. >>>>> >>>>> Thanks, >>>>> Sam >>>>> >>>>> On Thu, Jun 25, 2020 at 7:37 PM Junchao Zhang >>>>> wrote: >>>>> >>>>>> Sam, >>>>>> The MPI_Comm_create_keyval() error was fixed in maint/master. From >>>>>> the error message, it seems you need to configure --with-log=1 >>>>>> Otherwise, please send your full error stack trace and >>>>>> configure.log. >>>>>> Thanks. >>>>>> --Junchao Zhang >>>>>> >>>>>> >>>>>> On Thu, Jun 25, 2020 at 2:18 PM Sam Guo >>>>>> wrote: >>>>>> >>>>>>> Hi Junchao, >>>>>>> I now encountered the same error with parallel. I am wondering if >>>>>>> there is a need for parallel fix as well. >>>>>>> [1]PETSC ERROR: #1 PetscInitialize() line 969 in >>>>>>> ../../../petsc/src/sys/objects/pinit.c >>>>>>> PETSC ERROR: Logging has not been enabled. >>>>>>> You might have forgotten to call PetscInitialize(). >>>>>>> PETSC ERROR: Logging has not been enabled. >>>>>>> You might have forgotten to call PetscInitialize(). >>>>>>> >>>>>>> On Sat, Jun 20, 2020 at 7:35 PM Sam Guo >>>>>>> wrote: >>>>>>> >>>>>>>> Hi Junchao, >>>>>>>> Your patch works. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Sam >>>>>>>> >>>>>>>> On Sat, Jun 20, 2020 at 4:23 PM Junchao Zhang < >>>>>>>> junchao.zhang at gmail.com> wrote: >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Sat, Jun 20, 2020 at 12:24 PM Barry Smith >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> >>>>>>>>>> Junchao, >>>>>>>>>> >>>>>>>>>> This is a good bug fix. It solves the problem when PETSc >>>>>>>>>> initialize is called many times. >>>>>>>>>> >>>>>>>>>> There is another fix you can do to limit PETSc mpiuni >>>>>>>>>> running out of attributes inside a single PETSc run: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> int MPI_Comm_create_keyval(MPI_Copy_function >>>>>>>>>> *copy_fn,MPI_Delete_function *delete_fn,int *keyval,void *extra_state) >>>>>>>>>> { >>>>>>>>>> >>>>>>>>>> if (num_attr >= MAX_ATTR){ >>>>>>>>>> for (i=0; i>>>>>>>>> if (!attr_keyval[i].extra_state) { >>>>>>>>>> >>>>>>>>> attr_keyval[i].extra_state is provided by user (could be NULL). We >>>>>>>>> can not rely on it. >>>>>>>>> >>>>>>>>>> /* reuse this slot */ >>>>>>>>>> attr_keyval[i].extra_state = extra_state; >>>>>>>>>> attr_keyval[i.]del = delete_fn; >>>>>>>>>> *keyval = i; >>>>>>>>>> return MPI_SUCCESS; >>>>>>>>>> } >>>>>>>>>> } >>>>>>>>>> return MPIUni_Abort(MPI_COMM_WORLD,1); >>>>>>>>>> } >>>>>>>>>> return MPIUni_Abort(MPI_COMM_WORLD,1); >>>>>>>>>> attr_keyval[num_attr].extra_state = extra_state; >>>>>>>>>> attr_keyval[num_attr].del = delete_fn; >>>>>>>>>> *keyval = num_attr++; >>>>>>>>>> return MPI_SUCCESS; >>>>>>>>>> } >>>>>>>>>> >>>>>>>>>> This will work if the user creates tons of attributes but is >>>>>>>>>> constantly deleting some as they new ones. So long as the number >>>>>>>>>> outstanding at one time is < MAX_ATTR) >>>>>>>>>> >>>>>>>>>> Barry >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Jun 20, 2020, at 10:54 AM, Junchao Zhang < >>>>>>>>>> junchao.zhang at gmail.com> wrote: >>>>>>>>>> >>>>>>>>>> I don't understand what your session means. Let's try this patch >>>>>>>>>> >>>>>>>>>> diff --git a/src/sys/mpiuni/mpi.c b/src/sys/mpiuni/mpi.c >>>>>>>>>> index d559a513..c058265d 100644 >>>>>>>>>> --- a/src/sys/mpiuni/mpi.c >>>>>>>>>> +++ b/src/sys/mpiuni/mpi.c >>>>>>>>>> @@ -283,6 +283,7 @@ int MPI_Finalize(void) >>>>>>>>>> MPI_Comm_free(&comm); >>>>>>>>>> comm = MPI_COMM_SELF; >>>>>>>>>> MPI_Comm_free(&comm); >>>>>>>>>> + num_attr = 1; /* reset the counter */ >>>>>>>>>> MPI_was_finalized = 1; >>>>>>>>>> return MPI_SUCCESS; >>>>>>>>>> } >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> --Junchao Zhang >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Sat, Jun 20, 2020 at 10:48 AM Sam Guo >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Typo: I mean ?Assuming initializer is only needed once for >>>>>>>>>>> entire session? >>>>>>>>>>> >>>>>>>>>>> On Saturday, June 20, 2020, Sam Guo >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> Assuming finalizer is only needed once for entire session(?), I >>>>>>>>>>>> can put initializer into the static block to call it once but where do I >>>>>>>>>>>> call finalizer? >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Saturday, June 20, 2020, Junchao Zhang < >>>>>>>>>>>> junchao.zhang at gmail.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> The counter num_attr should be recycled. But first try to call >>>>>>>>>>>>> PETSc initialize/Finalize only once to see it fixes the error. >>>>>>>>>>>>> --Junchao Zhang >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Sat, Jun 20, 2020 at 12:48 AM Sam Guo < >>>>>>>>>>>>> sam.guo at cd-adapco.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> To clarify, I call PETSc initialize and PETSc finalize >>>>>>>>>>>>>> everytime I call SLEPc: >>>>>>>>>>>>>> >>>>>>>>>>>>>> PetscInitializeNoPointers(argc,args,nullptr,nullptr); >>>>>>>>>>>>>> >>>>>>>>>>>>>> SlepcInitialize(&argc,&args,static_cast(nullptr),help); >>>>>>>>>>>>>> >>>>>>>>>>>>>> //calling slepc >>>>>>>>>>>>>> >>>>>>>>>>>>>> SlepcFinalize(); >>>>>>>>>>>>>> >>>>>>>>>>>>>> PetscFinalize(); >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Fri, Jun 19, 2020 at 10:32 PM Sam Guo < >>>>>>>>>>>>>> sam.guo at cd-adapco.com> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Dear PETSc team, >>>>>>>>>>>>>>> When I called SLEPc multiple time, I eventually got >>>>>>>>>>>>>>> following error: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> MPI operation not supported by PETSc's sequential MPI >>>>>>>>>>>>>>> wrappers >>>>>>>>>>>>>>> [0]PETSC ERROR: #1 PetscInitialize() line 967 in >>>>>>>>>>>>>>> ../../../petsc/src/sys/objects/pinit.c >>>>>>>>>>>>>>> [0]PETSC ERROR: #2 SlepcInitialize() line 262 in >>>>>>>>>>>>>>> ../../../slepc/src/sys/slepcinit.c >>>>>>>>>>>>>>> [0]PETSC ERROR: #3 SlepcInitializeNoPointers() line 359 in >>>>>>>>>>>>>>> ../../../slepc/src/sys/slepcinit.c >>>>>>>>>>>>>>> PETSC ERROR: Logging has not been enabled. >>>>>>>>>>>>>>> You might have forgotten to call PetscInitialize(). >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I debugged: it is because of following in >>>>>>>>>>>>>>> petsc/src/sys/mpiuni/mpi.c >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> if (num_attr >= MAX_ATTR) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> in function int MPI_Comm_create_keyval(MPI_Copy_function >>>>>>>>>>>>>>> *copy_fn,MPI_Delete_function *delete_fn,int *keyval,void *extra_state) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> num_attr is declared static and keeps increasing every >>>>>>>>>>>>>>> time MPI_Comm_create_keyval is called. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I am using petsc 3.11.3 but found 3.13.2 has the same logic. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Is this a bug or I didn't use it correctly? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>> Sam >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>> >>>> >>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From sam.guo at cd-adapco.com Fri Jun 26 17:43:42 2020 From: sam.guo at cd-adapco.com (Sam Guo) Date: Fri, 26 Jun 2020 15:43:42 -0700 Subject: [petsc-users] PETSc initialization error In-Reply-To: References: <004EA4EB-815F-4EEE-8988-CA2179304157@petsc.dev> <9C60917A-5771-434D-BB65-D46750C49966@petsc.dev> Message-ID: Hi Junchao, I am not ready to upgrade petsc yet(due to the lengthy technical and legal approval process of our internal policy). Can you send me the diff file so I can apply it to petsc 3.11.3)? Thanks, Sam On Fri, Jun 26, 2020 at 3:33 PM Junchao Zhang wrote: > Sam, > Please discard the origin patch I sent you. A better fix is already in > maint/master. An test is at src/sys/tests/ex53.c > I modified that test at the end with > > for (i=0; i<500; i++) { > ierr = PetscInitializeNoPointers(argc,argv,NULL,help);if (ierr) return > ierr; > ierr = SlepcInitialize(&argc,&argv,NULL,help);if (ierr) return ierr; > ierr = SlepcFinalize();if (ierr) return ierr; > ierr = PetscFinalize();if (ierr) return ierr; > } > > > then I ran it with multiple mpi ranks and it ran correctly. So try your > program with petsc master first. If not work, see if you can come up with a > test example for us. > > Thanks. > --Junchao Zhang > > > On Fri, Jun 26, 2020 at 3:37 PM Sam Guo wrote: > >> One work around for me is to call PetscInitialize once for my entire >> program and skip PetscFinalize (since I don't have a good place to call >> PetscFinalize before ending the program). >> >> On Fri, Jun 26, 2020 at 1:33 PM Sam Guo wrote: >> >>> I get the crash after calling Initialize/Finalize multiple times. >>> Junchao fixed the bug for serial but parallel still crashes. >>> >>> On Fri, Jun 26, 2020 at 1:28 PM Barry Smith wrote: >>> >>>> >>>> Ah, so you get the crash the second time you call PetscInitialize()? >>>> That is a problem because we do intend to support that capability (but you >>>> much call PetscFinalize() each time also). >>>> >>>> Barry >>>> >>>> >>>> On Jun 26, 2020, at 3:25 PM, Sam Guo wrote: >>>> >>>> Hi Barry, >>>> Thanks for the quick response. >>>> I will call PetscInitialize once and skip the PetscFinalize for now >>>> to avoid the crash. The crash is actually in PetscInitialize, not >>>> PetscFinalize. >>>> >>>> Thanks, >>>> Sam >>>> >>>> On Fri, Jun 26, 2020 at 1:21 PM Barry Smith wrote: >>>> >>>>> >>>>> Sam, >>>>> >>>>> You can skip PetscFinalize() so long as you only call >>>>> PetscInitialize() once. It is not desirable in general to skip the finalize >>>>> because PETSc can't free all its data structures and you cannot see the >>>>> PETSc logging information with -log_view but in terms of the code running >>>>> correctly you do not need to call PetscFinalize. >>>>> >>>>> If your code crashes in PetscFinalize() please send the full error >>>>> output and we can try to help you debug it. >>>>> >>>>> >>>>> Barry >>>>> >>>>> On Jun 26, 2020, at 3:14 PM, Sam Guo wrote: >>>>> >>>>> To clarify, we have a mpi wrapper (so we can switch to different mpi >>>>> at runtime). I compile petsc using our mpi wrapper. >>>>> If I just call PETSc initialize once without calling finallize, it is >>>>> ok. My question to you is that: can I skip finalize? >>>>> Our program calls mpi_finalize at end anyway. >>>>> >>>>> On Fri, Jun 26, 2020 at 1:09 PM Sam Guo wrote: >>>>> >>>>>> Hi Junchao, >>>>>> Attached please find the configure.log. >>>>>> I also attach the pinit.c which contains your patch (I am >>>>>> currently using 3.11.3. I've applied your patch to 3.11.3). Your patch >>>>>> fixes the serial version. The error now is about the parallel. >>>>>> Here is the error log: >>>>>> >>>>>> [1]PETSC ERROR: #1 PetscInitialize() line 969 in >>>>>> ../../../petsc/src/sys/objects/pinit.c >>>>>> [1]PETSC ERROR: #2 checkError() line 56 in >>>>>> ../../../physics/src/eigensolver/SLEPc.cpp >>>>>> [1]PETSC ERROR: #3 PetscInitialize() line 966 in >>>>>> ../../../petsc/src/sys/objects/pinit.c >>>>>> [1]PETSC ERROR: #4 SlepcInitialize() line 262 in >>>>>> ../../../slepc/src/sys/slepcinit.c >>>>>> [0]PETSC ERROR: #1 PetscInitialize() line 969 in >>>>>> ../../../petsc/src/sys/objects/pinit.c >>>>>> [0]PETSC ERROR: #2 checkError() line 56 in >>>>>> ../../../physics/src/eigensolver/SLEPc.cpp >>>>>> [0]PETSC ERROR: #3 PetscInitialize() line 966 in >>>>>> ../../../petsc/src/sys/objects/pinit.c >>>>>> [0]PETSC ERROR: #4 SlepcInitialize() line 262 in >>>>>> ../../../slepc/src/sys/slepcinit.c >>>>>> PETSC ERROR: Logging has not been enabled. >>>>>> You might have forgotten to call PetscInitialize(). >>>>>> PETSC ERROR: Logging has not been enabled. >>>>>> You might have forgotten to call PetscInitialize(). >>>>>> >>>>>> -------------------------------------------------------------------------- >>>>>> MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD >>>>>> with errorcode 56. >>>>>> >>>>>> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. >>>>>> You may or may not see output from other processes, depending on >>>>>> exactly when Open MPI kills them. >>>>>> >>>>>> Thanks, >>>>>> Sam >>>>>> >>>>>> On Thu, Jun 25, 2020 at 7:37 PM Junchao Zhang < >>>>>> junchao.zhang at gmail.com> wrote: >>>>>> >>>>>>> Sam, >>>>>>> The MPI_Comm_create_keyval() error was fixed in maint/master. >>>>>>> From the error message, it seems you need to configure --with-log=1 >>>>>>> Otherwise, please send your full error stack trace and >>>>>>> configure.log. >>>>>>> Thanks. >>>>>>> --Junchao Zhang >>>>>>> >>>>>>> >>>>>>> On Thu, Jun 25, 2020 at 2:18 PM Sam Guo >>>>>>> wrote: >>>>>>> >>>>>>>> Hi Junchao, >>>>>>>> I now encountered the same error with parallel. I am wondering >>>>>>>> if there is a need for parallel fix as well. >>>>>>>> [1]PETSC ERROR: #1 PetscInitialize() line 969 in >>>>>>>> ../../../petsc/src/sys/objects/pinit.c >>>>>>>> PETSC ERROR: Logging has not been enabled. >>>>>>>> You might have forgotten to call PetscInitialize(). >>>>>>>> PETSC ERROR: Logging has not been enabled. >>>>>>>> You might have forgotten to call PetscInitialize(). >>>>>>>> >>>>>>>> On Sat, Jun 20, 2020 at 7:35 PM Sam Guo >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi Junchao, >>>>>>>>> Your patch works. >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Sam >>>>>>>>> >>>>>>>>> On Sat, Jun 20, 2020 at 4:23 PM Junchao Zhang < >>>>>>>>> junchao.zhang at gmail.com> wrote: >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Sat, Jun 20, 2020 at 12:24 PM Barry Smith >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Junchao, >>>>>>>>>>> >>>>>>>>>>> This is a good bug fix. It solves the problem when PETSc >>>>>>>>>>> initialize is called many times. >>>>>>>>>>> >>>>>>>>>>> There is another fix you can do to limit PETSc mpiuni >>>>>>>>>>> running out of attributes inside a single PETSc run: >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> int MPI_Comm_create_keyval(MPI_Copy_function >>>>>>>>>>> *copy_fn,MPI_Delete_function *delete_fn,int *keyval,void *extra_state) >>>>>>>>>>> { >>>>>>>>>>> >>>>>>>>>>> if (num_attr >= MAX_ATTR){ >>>>>>>>>>> for (i=0; i>>>>>>>>>> if (!attr_keyval[i].extra_state) { >>>>>>>>>>> >>>>>>>>>> attr_keyval[i].extra_state is provided by user (could be NULL). >>>>>>>>>> We can not rely on it. >>>>>>>>>> >>>>>>>>>>> /* reuse this slot */ >>>>>>>>>>> attr_keyval[i].extra_state = extra_state; >>>>>>>>>>> attr_keyval[i.]del = delete_fn; >>>>>>>>>>> *keyval = i; >>>>>>>>>>> return MPI_SUCCESS; >>>>>>>>>>> } >>>>>>>>>>> } >>>>>>>>>>> return MPIUni_Abort(MPI_COMM_WORLD,1); >>>>>>>>>>> } >>>>>>>>>>> return MPIUni_Abort(MPI_COMM_WORLD,1); >>>>>>>>>>> attr_keyval[num_attr].extra_state = extra_state; >>>>>>>>>>> attr_keyval[num_attr].del = delete_fn; >>>>>>>>>>> *keyval = num_attr++; >>>>>>>>>>> return MPI_SUCCESS; >>>>>>>>>>> } >>>>>>>>>>> >>>>>>>>>>> This will work if the user creates tons of attributes but is >>>>>>>>>>> constantly deleting some as they new ones. So long as the number >>>>>>>>>>> outstanding at one time is < MAX_ATTR) >>>>>>>>>>> >>>>>>>>>>> Barry >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Jun 20, 2020, at 10:54 AM, Junchao Zhang < >>>>>>>>>>> junchao.zhang at gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>>> I don't understand what your session means. Let's try this patch >>>>>>>>>>> >>>>>>>>>>> diff --git a/src/sys/mpiuni/mpi.c b/src/sys/mpiuni/mpi.c >>>>>>>>>>> index d559a513..c058265d 100644 >>>>>>>>>>> --- a/src/sys/mpiuni/mpi.c >>>>>>>>>>> +++ b/src/sys/mpiuni/mpi.c >>>>>>>>>>> @@ -283,6 +283,7 @@ int MPI_Finalize(void) >>>>>>>>>>> MPI_Comm_free(&comm); >>>>>>>>>>> comm = MPI_COMM_SELF; >>>>>>>>>>> MPI_Comm_free(&comm); >>>>>>>>>>> + num_attr = 1; /* reset the counter */ >>>>>>>>>>> MPI_was_finalized = 1; >>>>>>>>>>> return MPI_SUCCESS; >>>>>>>>>>> } >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> --Junchao Zhang >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Sat, Jun 20, 2020 at 10:48 AM Sam Guo >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> Typo: I mean ?Assuming initializer is only needed once for >>>>>>>>>>>> entire session? >>>>>>>>>>>> >>>>>>>>>>>> On Saturday, June 20, 2020, Sam Guo >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Assuming finalizer is only needed once for entire session(?), >>>>>>>>>>>>> I can put initializer into the static block to call it once but where do I >>>>>>>>>>>>> call finalizer? >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Saturday, June 20, 2020, Junchao Zhang < >>>>>>>>>>>>> junchao.zhang at gmail.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> The counter num_attr should be recycled. But first try to >>>>>>>>>>>>>> call PETSc initialize/Finalize only once to see it fixes the error. >>>>>>>>>>>>>> --Junchao Zhang >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Sat, Jun 20, 2020 at 12:48 AM Sam Guo < >>>>>>>>>>>>>> sam.guo at cd-adapco.com> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> To clarify, I call PETSc initialize and PETSc finalize >>>>>>>>>>>>>>> everytime I call SLEPc: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> PetscInitializeNoPointers(argc,args,nullptr,nullptr); >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> SlepcInitialize(&argc,&args,static_cast(nullptr),help); >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> //calling slepc >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> SlepcFinalize(); >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> PetscFinalize(); >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Fri, Jun 19, 2020 at 10:32 PM Sam Guo < >>>>>>>>>>>>>>> sam.guo at cd-adapco.com> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Dear PETSc team, >>>>>>>>>>>>>>>> When I called SLEPc multiple time, I eventually got >>>>>>>>>>>>>>>> following error: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> MPI operation not supported by PETSc's sequential MPI >>>>>>>>>>>>>>>> wrappers >>>>>>>>>>>>>>>> [0]PETSC ERROR: #1 PetscInitialize() line 967 in >>>>>>>>>>>>>>>> ../../../petsc/src/sys/objects/pinit.c >>>>>>>>>>>>>>>> [0]PETSC ERROR: #2 SlepcInitialize() line 262 in >>>>>>>>>>>>>>>> ../../../slepc/src/sys/slepcinit.c >>>>>>>>>>>>>>>> [0]PETSC ERROR: #3 SlepcInitializeNoPointers() line 359 in >>>>>>>>>>>>>>>> ../../../slepc/src/sys/slepcinit.c >>>>>>>>>>>>>>>> PETSC ERROR: Logging has not been enabled. >>>>>>>>>>>>>>>> You might have forgotten to call PetscInitialize(). >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I debugged: it is because of following in >>>>>>>>>>>>>>>> petsc/src/sys/mpiuni/mpi.c >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> if (num_attr >= MAX_ATTR) >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> in function int MPI_Comm_create_keyval(MPI_Copy_function >>>>>>>>>>>>>>>> *copy_fn,MPI_Delete_function *delete_fn,int *keyval,void *extra_state) >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> num_attr is declared static and keeps increasing every >>>>>>>>>>>>>>>> time MPI_Comm_create_keyval is called. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I am using petsc 3.11.3 but found 3.13.2 has the same logic. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Is this a bug or I didn't use it correctly? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>> Sam >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>> >>>>> >>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From sam.guo at cd-adapco.com Fri Jun 26 17:53:52 2020 From: sam.guo at cd-adapco.com (Sam Guo) Date: Fri, 26 Jun 2020 15:53:52 -0700 Subject: [petsc-users] PETSc initialization error In-Reply-To: References: <004EA4EB-815F-4EEE-8988-CA2179304157@petsc.dev> <9C60917A-5771-434D-BB65-D46750C49966@petsc.dev> Message-ID: Hi Junchao, If you are talking about this commit of yours https://gitlab.com/petsc/petsc/-/commit/f0463fa09df52ce43e7c5bf47a1c87df0c9e5cbb Recycle keyvals and fix bugs in MPI_Comm creation I think I got it. It fixes the serial one but parallel one is still crashing. Thanks, Sam On Fri, Jun 26, 2020 at 3:43 PM Sam Guo wrote: > Hi Junchao, > I am not ready to upgrade petsc yet(due to the lengthy technical and > legal approval process of our internal policy). Can you send me the diff > file so I can apply it to petsc 3.11.3)? > > Thanks, > Sam > > On Fri, Jun 26, 2020 at 3:33 PM Junchao Zhang > wrote: > >> Sam, >> Please discard the origin patch I sent you. A better fix is already in >> maint/master. An test is at src/sys/tests/ex53.c >> I modified that test at the end with >> >> for (i=0; i<500; i++) { >> ierr = PetscInitializeNoPointers(argc,argv,NULL,help);if (ierr) >> return ierr; >> ierr = SlepcInitialize(&argc,&argv,NULL,help);if (ierr) return ierr; >> ierr = SlepcFinalize();if (ierr) return ierr; >> ierr = PetscFinalize();if (ierr) return ierr; >> } >> >> >> then I ran it with multiple mpi ranks and it ran correctly. So try your >> program with petsc master first. If not work, see if you can come up with a >> test example for us. >> >> Thanks. >> --Junchao Zhang >> >> >> On Fri, Jun 26, 2020 at 3:37 PM Sam Guo wrote: >> >>> One work around for me is to call PetscInitialize once for my entire >>> program and skip PetscFinalize (since I don't have a good place to call >>> PetscFinalize before ending the program). >>> >>> On Fri, Jun 26, 2020 at 1:33 PM Sam Guo wrote: >>> >>>> I get the crash after calling Initialize/Finalize multiple times. >>>> Junchao fixed the bug for serial but parallel still crashes. >>>> >>>> On Fri, Jun 26, 2020 at 1:28 PM Barry Smith wrote: >>>> >>>>> >>>>> Ah, so you get the crash the second time you call >>>>> PetscInitialize()? That is a problem because we do intend to support that >>>>> capability (but you much call PetscFinalize() each time also). >>>>> >>>>> Barry >>>>> >>>>> >>>>> On Jun 26, 2020, at 3:25 PM, Sam Guo wrote: >>>>> >>>>> Hi Barry, >>>>> Thanks for the quick response. >>>>> I will call PetscInitialize once and skip the PetscFinalize for now >>>>> to avoid the crash. The crash is actually in PetscInitialize, not >>>>> PetscFinalize. >>>>> >>>>> Thanks, >>>>> Sam >>>>> >>>>> On Fri, Jun 26, 2020 at 1:21 PM Barry Smith wrote: >>>>> >>>>>> >>>>>> Sam, >>>>>> >>>>>> You can skip PetscFinalize() so long as you only call >>>>>> PetscInitialize() once. It is not desirable in general to skip the finalize >>>>>> because PETSc can't free all its data structures and you cannot see the >>>>>> PETSc logging information with -log_view but in terms of the code running >>>>>> correctly you do not need to call PetscFinalize. >>>>>> >>>>>> If your code crashes in PetscFinalize() please send the full error >>>>>> output and we can try to help you debug it. >>>>>> >>>>>> >>>>>> Barry >>>>>> >>>>>> On Jun 26, 2020, at 3:14 PM, Sam Guo wrote: >>>>>> >>>>>> To clarify, we have a mpi wrapper (so we can switch to different mpi >>>>>> at runtime). I compile petsc using our mpi wrapper. >>>>>> If I just call PETSc initialize once without calling finallize, it is >>>>>> ok. My question to you is that: can I skip finalize? >>>>>> Our program calls mpi_finalize at end anyway. >>>>>> >>>>>> On Fri, Jun 26, 2020 at 1:09 PM Sam Guo >>>>>> wrote: >>>>>> >>>>>>> Hi Junchao, >>>>>>> Attached please find the configure.log. >>>>>>> I also attach the pinit.c which contains your patch (I am >>>>>>> currently using 3.11.3. I've applied your patch to 3.11.3). Your patch >>>>>>> fixes the serial version. The error now is about the parallel. >>>>>>> Here is the error log: >>>>>>> >>>>>>> [1]PETSC ERROR: #1 PetscInitialize() line 969 in >>>>>>> ../../../petsc/src/sys/objects/pinit.c >>>>>>> [1]PETSC ERROR: #2 checkError() line 56 in >>>>>>> ../../../physics/src/eigensolver/SLEPc.cpp >>>>>>> [1]PETSC ERROR: #3 PetscInitialize() line 966 in >>>>>>> ../../../petsc/src/sys/objects/pinit.c >>>>>>> [1]PETSC ERROR: #4 SlepcInitialize() line 262 in >>>>>>> ../../../slepc/src/sys/slepcinit.c >>>>>>> [0]PETSC ERROR: #1 PetscInitialize() line 969 in >>>>>>> ../../../petsc/src/sys/objects/pinit.c >>>>>>> [0]PETSC ERROR: #2 checkError() line 56 in >>>>>>> ../../../physics/src/eigensolver/SLEPc.cpp >>>>>>> [0]PETSC ERROR: #3 PetscInitialize() line 966 in >>>>>>> ../../../petsc/src/sys/objects/pinit.c >>>>>>> [0]PETSC ERROR: #4 SlepcInitialize() line 262 in >>>>>>> ../../../slepc/src/sys/slepcinit.c >>>>>>> PETSC ERROR: Logging has not been enabled. >>>>>>> You might have forgotten to call PetscInitialize(). >>>>>>> PETSC ERROR: Logging has not been enabled. >>>>>>> You might have forgotten to call PetscInitialize(). >>>>>>> >>>>>>> -------------------------------------------------------------------------- >>>>>>> MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD >>>>>>> with errorcode 56. >>>>>>> >>>>>>> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. >>>>>>> You may or may not see output from other processes, depending on >>>>>>> exactly when Open MPI kills them. >>>>>>> >>>>>>> Thanks, >>>>>>> Sam >>>>>>> >>>>>>> On Thu, Jun 25, 2020 at 7:37 PM Junchao Zhang < >>>>>>> junchao.zhang at gmail.com> wrote: >>>>>>> >>>>>>>> Sam, >>>>>>>> The MPI_Comm_create_keyval() error was fixed in maint/master. >>>>>>>> From the error message, it seems you need to configure --with-log=1 >>>>>>>> Otherwise, please send your full error stack trace and >>>>>>>> configure.log. >>>>>>>> Thanks. >>>>>>>> --Junchao Zhang >>>>>>>> >>>>>>>> >>>>>>>> On Thu, Jun 25, 2020 at 2:18 PM Sam Guo >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi Junchao, >>>>>>>>> I now encountered the same error with parallel. I am wondering >>>>>>>>> if there is a need for parallel fix as well. >>>>>>>>> [1]PETSC ERROR: #1 PetscInitialize() line 969 in >>>>>>>>> ../../../petsc/src/sys/objects/pinit.c >>>>>>>>> PETSC ERROR: Logging has not been enabled. >>>>>>>>> You might have forgotten to call PetscInitialize(). >>>>>>>>> PETSC ERROR: Logging has not been enabled. >>>>>>>>> You might have forgotten to call PetscInitialize(). >>>>>>>>> >>>>>>>>> On Sat, Jun 20, 2020 at 7:35 PM Sam Guo >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hi Junchao, >>>>>>>>>> Your patch works. >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> Sam >>>>>>>>>> >>>>>>>>>> On Sat, Jun 20, 2020 at 4:23 PM Junchao Zhang < >>>>>>>>>> junchao.zhang at gmail.com> wrote: >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Sat, Jun 20, 2020 at 12:24 PM Barry Smith >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Junchao, >>>>>>>>>>>> >>>>>>>>>>>> This is a good bug fix. It solves the problem when PETSc >>>>>>>>>>>> initialize is called many times. >>>>>>>>>>>> >>>>>>>>>>>> There is another fix you can do to limit PETSc mpiuni >>>>>>>>>>>> running out of attributes inside a single PETSc run: >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> int MPI_Comm_create_keyval(MPI_Copy_function >>>>>>>>>>>> *copy_fn,MPI_Delete_function *delete_fn,int *keyval,void *extra_state) >>>>>>>>>>>> { >>>>>>>>>>>> >>>>>>>>>>>> if (num_attr >= MAX_ATTR){ >>>>>>>>>>>> for (i=0; i>>>>>>>>>>> if (!attr_keyval[i].extra_state) { >>>>>>>>>>>> >>>>>>>>>>> attr_keyval[i].extra_state is provided by user (could be NULL). >>>>>>>>>>> We can not rely on it. >>>>>>>>>>> >>>>>>>>>>>> /* reuse this slot */ >>>>>>>>>>>> attr_keyval[i].extra_state = extra_state; >>>>>>>>>>>> attr_keyval[i.]del = delete_fn; >>>>>>>>>>>> *keyval = i; >>>>>>>>>>>> return MPI_SUCCESS; >>>>>>>>>>>> } >>>>>>>>>>>> } >>>>>>>>>>>> return MPIUni_Abort(MPI_COMM_WORLD,1); >>>>>>>>>>>> } >>>>>>>>>>>> return MPIUni_Abort(MPI_COMM_WORLD,1); >>>>>>>>>>>> attr_keyval[num_attr].extra_state = extra_state; >>>>>>>>>>>> attr_keyval[num_attr].del = delete_fn; >>>>>>>>>>>> *keyval = num_attr++; >>>>>>>>>>>> return MPI_SUCCESS; >>>>>>>>>>>> } >>>>>>>>>>>> >>>>>>>>>>>> This will work if the user creates tons of attributes but is >>>>>>>>>>>> constantly deleting some as they new ones. So long as the number >>>>>>>>>>>> outstanding at one time is < MAX_ATTR) >>>>>>>>>>>> >>>>>>>>>>>> Barry >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Jun 20, 2020, at 10:54 AM, Junchao Zhang < >>>>>>>>>>>> junchao.zhang at gmail.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>> I don't understand what your session means. Let's try this patch >>>>>>>>>>>> >>>>>>>>>>>> diff --git a/src/sys/mpiuni/mpi.c b/src/sys/mpiuni/mpi.c >>>>>>>>>>>> index d559a513..c058265d 100644 >>>>>>>>>>>> --- a/src/sys/mpiuni/mpi.c >>>>>>>>>>>> +++ b/src/sys/mpiuni/mpi.c >>>>>>>>>>>> @@ -283,6 +283,7 @@ int MPI_Finalize(void) >>>>>>>>>>>> MPI_Comm_free(&comm); >>>>>>>>>>>> comm = MPI_COMM_SELF; >>>>>>>>>>>> MPI_Comm_free(&comm); >>>>>>>>>>>> + num_attr = 1; /* reset the counter */ >>>>>>>>>>>> MPI_was_finalized = 1; >>>>>>>>>>>> return MPI_SUCCESS; >>>>>>>>>>>> } >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> --Junchao Zhang >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Sat, Jun 20, 2020 at 10:48 AM Sam Guo >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Typo: I mean ?Assuming initializer is only needed once for >>>>>>>>>>>>> entire session? >>>>>>>>>>>>> >>>>>>>>>>>>> On Saturday, June 20, 2020, Sam Guo >>>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Assuming finalizer is only needed once for entire session(?), >>>>>>>>>>>>>> I can put initializer into the static block to call it once but where do I >>>>>>>>>>>>>> call finalizer? >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Saturday, June 20, 2020, Junchao Zhang < >>>>>>>>>>>>>> junchao.zhang at gmail.com> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> The counter num_attr should be recycled. But first try to >>>>>>>>>>>>>>> call PETSc initialize/Finalize only once to see it fixes the error. >>>>>>>>>>>>>>> --Junchao Zhang >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Sat, Jun 20, 2020 at 12:48 AM Sam Guo < >>>>>>>>>>>>>>> sam.guo at cd-adapco.com> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> To clarify, I call PETSc initialize and PETSc finalize >>>>>>>>>>>>>>>> everytime I call SLEPc: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> PetscInitializeNoPointers(argc,args,nullptr,nullptr); >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> SlepcInitialize(&argc,&args,static_cast(nullptr),help); >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> //calling slepc >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> SlepcFinalize(); >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> PetscFinalize(); >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Fri, Jun 19, 2020 at 10:32 PM Sam Guo < >>>>>>>>>>>>>>>> sam.guo at cd-adapco.com> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Dear PETSc team, >>>>>>>>>>>>>>>>> When I called SLEPc multiple time, I eventually got >>>>>>>>>>>>>>>>> following error: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> MPI operation not supported by PETSc's sequential MPI >>>>>>>>>>>>>>>>> wrappers >>>>>>>>>>>>>>>>> [0]PETSC ERROR: #1 PetscInitialize() line 967 in >>>>>>>>>>>>>>>>> ../../../petsc/src/sys/objects/pinit.c >>>>>>>>>>>>>>>>> [0]PETSC ERROR: #2 SlepcInitialize() line 262 in >>>>>>>>>>>>>>>>> ../../../slepc/src/sys/slepcinit.c >>>>>>>>>>>>>>>>> [0]PETSC ERROR: #3 SlepcInitializeNoPointers() line 359 in >>>>>>>>>>>>>>>>> ../../../slepc/src/sys/slepcinit.c >>>>>>>>>>>>>>>>> PETSC ERROR: Logging has not been enabled. >>>>>>>>>>>>>>>>> You might have forgotten to call PetscInitialize(). >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I debugged: it is because of following in >>>>>>>>>>>>>>>>> petsc/src/sys/mpiuni/mpi.c >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> if (num_attr >= MAX_ATTR) >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> in function int MPI_Comm_create_keyval(MPI_Copy_function >>>>>>>>>>>>>>>>> *copy_fn,MPI_Delete_function *delete_fn,int *keyval,void *extra_state) >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> num_attr is declared static and keeps increasing every >>>>>>>>>>>>>>>>> time MPI_Comm_create_keyval is called. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I am using petsc 3.11.3 but found 3.13.2 has the >>>>>>>>>>>>>>>>> same logic. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Is this a bug or I didn't use it correctly? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>> Sam >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>> >>>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From junchao.zhang at gmail.com Fri Jun 26 20:57:49 2020 From: junchao.zhang at gmail.com (Junchao Zhang) Date: Fri, 26 Jun 2020 20:57:49 -0500 Subject: [petsc-users] PETSc initialization error In-Reply-To: References: <004EA4EB-815F-4EEE-8988-CA2179304157@petsc.dev> <9C60917A-5771-434D-BB65-D46750C49966@petsc.dev> Message-ID: Did the test included in that commit fail in your environment? You can also change the test by adding calls to SlepcInitialize/SlepcFinalize between PetscInitializeNoPointers/PetscFinalize as in my previous email. --Junchao Zhang On Fri, Jun 26, 2020 at 5:54 PM Sam Guo wrote: > Hi Junchao, > If you are talking about this commit of yours > https://gitlab.com/petsc/petsc/-/commit/f0463fa09df52ce43e7c5bf47a1c87df0c9e5cbb > > Recycle keyvals and fix bugs in MPI_Comm creation > I think I got it. It fixes the serial one but parallel one is still > crashing. > > Thanks, > Sam > > On Fri, Jun 26, 2020 at 3:43 PM Sam Guo wrote: > >> Hi Junchao, >> I am not ready to upgrade petsc yet(due to the lengthy technical and >> legal approval process of our internal policy). Can you send me the diff >> file so I can apply it to petsc 3.11.3)? >> >> Thanks, >> Sam >> >> On Fri, Jun 26, 2020 at 3:33 PM Junchao Zhang >> wrote: >> >>> Sam, >>> Please discard the origin patch I sent you. A better fix is already in >>> maint/master. An test is at src/sys/tests/ex53.c >>> I modified that test at the end with >>> >>> for (i=0; i<500; i++) { >>> ierr = PetscInitializeNoPointers(argc,argv,NULL,help);if (ierr) >>> return ierr; >>> ierr = SlepcInitialize(&argc,&argv,NULL,help);if (ierr) return ierr; >>> ierr = SlepcFinalize();if (ierr) return ierr; >>> ierr = PetscFinalize();if (ierr) return ierr; >>> } >>> >>> >>> then I ran it with multiple mpi ranks and it ran correctly. So try your >>> program with petsc master first. If not work, see if you can come up with a >>> test example for us. >>> >>> Thanks. >>> --Junchao Zhang >>> >>> >>> On Fri, Jun 26, 2020 at 3:37 PM Sam Guo wrote: >>> >>>> One work around for me is to call PetscInitialize once for my entire >>>> program and skip PetscFinalize (since I don't have a good place to call >>>> PetscFinalize before ending the program). >>>> >>>> On Fri, Jun 26, 2020 at 1:33 PM Sam Guo wrote: >>>> >>>>> I get the crash after calling Initialize/Finalize multiple times. >>>>> Junchao fixed the bug for serial but parallel still crashes. >>>>> >>>>> On Fri, Jun 26, 2020 at 1:28 PM Barry Smith wrote: >>>>> >>>>>> >>>>>> Ah, so you get the crash the second time you call >>>>>> PetscInitialize()? That is a problem because we do intend to support that >>>>>> capability (but you much call PetscFinalize() each time also). >>>>>> >>>>>> Barry >>>>>> >>>>>> >>>>>> On Jun 26, 2020, at 3:25 PM, Sam Guo wrote: >>>>>> >>>>>> Hi Barry, >>>>>> Thanks for the quick response. >>>>>> I will call PetscInitialize once and skip the PetscFinalize for >>>>>> now to avoid the crash. The crash is actually in PetscInitialize, not >>>>>> PetscFinalize. >>>>>> >>>>>> Thanks, >>>>>> Sam >>>>>> >>>>>> On Fri, Jun 26, 2020 at 1:21 PM Barry Smith wrote: >>>>>> >>>>>>> >>>>>>> Sam, >>>>>>> >>>>>>> You can skip PetscFinalize() so long as you only call >>>>>>> PetscInitialize() once. It is not desirable in general to skip the finalize >>>>>>> because PETSc can't free all its data structures and you cannot see the >>>>>>> PETSc logging information with -log_view but in terms of the code running >>>>>>> correctly you do not need to call PetscFinalize. >>>>>>> >>>>>>> If your code crashes in PetscFinalize() please send the full >>>>>>> error output and we can try to help you debug it. >>>>>>> >>>>>>> >>>>>>> Barry >>>>>>> >>>>>>> On Jun 26, 2020, at 3:14 PM, Sam Guo wrote: >>>>>>> >>>>>>> To clarify, we have a mpi wrapper (so we can switch to different mpi >>>>>>> at runtime). I compile petsc using our mpi wrapper. >>>>>>> If I just call PETSc initialize once without calling finallize, it >>>>>>> is ok. My question to you is that: can I skip finalize? >>>>>>> Our program calls mpi_finalize at end anyway. >>>>>>> >>>>>>> On Fri, Jun 26, 2020 at 1:09 PM Sam Guo >>>>>>> wrote: >>>>>>> >>>>>>>> Hi Junchao, >>>>>>>> Attached please find the configure.log. >>>>>>>> I also attach the pinit.c which contains your patch (I am >>>>>>>> currently using 3.11.3. I've applied your patch to 3.11.3). Your patch >>>>>>>> fixes the serial version. The error now is about the parallel. >>>>>>>> Here is the error log: >>>>>>>> >>>>>>>> [1]PETSC ERROR: #1 PetscInitialize() line 969 in >>>>>>>> ../../../petsc/src/sys/objects/pinit.c >>>>>>>> [1]PETSC ERROR: #2 checkError() line 56 in >>>>>>>> ../../../physics/src/eigensolver/SLEPc.cpp >>>>>>>> [1]PETSC ERROR: #3 PetscInitialize() line 966 in >>>>>>>> ../../../petsc/src/sys/objects/pinit.c >>>>>>>> [1]PETSC ERROR: #4 SlepcInitialize() line 262 in >>>>>>>> ../../../slepc/src/sys/slepcinit.c >>>>>>>> [0]PETSC ERROR: #1 PetscInitialize() line 969 in >>>>>>>> ../../../petsc/src/sys/objects/pinit.c >>>>>>>> [0]PETSC ERROR: #2 checkError() line 56 in >>>>>>>> ../../../physics/src/eigensolver/SLEPc.cpp >>>>>>>> [0]PETSC ERROR: #3 PetscInitialize() line 966 in >>>>>>>> ../../../petsc/src/sys/objects/pinit.c >>>>>>>> [0]PETSC ERROR: #4 SlepcInitialize() line 262 in >>>>>>>> ../../../slepc/src/sys/slepcinit.c >>>>>>>> PETSC ERROR: Logging has not been enabled. >>>>>>>> You might have forgotten to call PetscInitialize(). >>>>>>>> PETSC ERROR: Logging has not been enabled. >>>>>>>> You might have forgotten to call PetscInitialize(). >>>>>>>> >>>>>>>> -------------------------------------------------------------------------- >>>>>>>> MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD >>>>>>>> with errorcode 56. >>>>>>>> >>>>>>>> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. >>>>>>>> You may or may not see output from other processes, depending on >>>>>>>> exactly when Open MPI kills them. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Sam >>>>>>>> >>>>>>>> On Thu, Jun 25, 2020 at 7:37 PM Junchao Zhang < >>>>>>>> junchao.zhang at gmail.com> wrote: >>>>>>>> >>>>>>>>> Sam, >>>>>>>>> The MPI_Comm_create_keyval() error was fixed in maint/master. >>>>>>>>> From the error message, it seems you need to configure --with-log=1 >>>>>>>>> Otherwise, please send your full error stack trace and >>>>>>>>> configure.log. >>>>>>>>> Thanks. >>>>>>>>> --Junchao Zhang >>>>>>>>> >>>>>>>>> >>>>>>>>> On Thu, Jun 25, 2020 at 2:18 PM Sam Guo >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hi Junchao, >>>>>>>>>> I now encountered the same error with parallel. I am wondering >>>>>>>>>> if there is a need for parallel fix as well. >>>>>>>>>> [1]PETSC ERROR: #1 PetscInitialize() line 969 in >>>>>>>>>> ../../../petsc/src/sys/objects/pinit.c >>>>>>>>>> PETSC ERROR: Logging has not been enabled. >>>>>>>>>> You might have forgotten to call PetscInitialize(). >>>>>>>>>> PETSC ERROR: Logging has not been enabled. >>>>>>>>>> You might have forgotten to call PetscInitialize(). >>>>>>>>>> >>>>>>>>>> On Sat, Jun 20, 2020 at 7:35 PM Sam Guo >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Hi Junchao, >>>>>>>>>>> Your patch works. >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> Sam >>>>>>>>>>> >>>>>>>>>>> On Sat, Jun 20, 2020 at 4:23 PM Junchao Zhang < >>>>>>>>>>> junchao.zhang at gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Sat, Jun 20, 2020 at 12:24 PM Barry Smith >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Junchao, >>>>>>>>>>>>> >>>>>>>>>>>>> This is a good bug fix. It solves the problem when PETSc >>>>>>>>>>>>> initialize is called many times. >>>>>>>>>>>>> >>>>>>>>>>>>> There is another fix you can do to limit PETSc mpiuni >>>>>>>>>>>>> running out of attributes inside a single PETSc run: >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> int MPI_Comm_create_keyval(MPI_Copy_function >>>>>>>>>>>>> *copy_fn,MPI_Delete_function *delete_fn,int *keyval,void *extra_state) >>>>>>>>>>>>> { >>>>>>>>>>>>> >>>>>>>>>>>>> if (num_attr >= MAX_ATTR){ >>>>>>>>>>>>> for (i=0; i>>>>>>>>>>>> if (!attr_keyval[i].extra_state) { >>>>>>>>>>>>> >>>>>>>>>>>> attr_keyval[i].extra_state is provided by user (could be NULL). >>>>>>>>>>>> We can not rely on it. >>>>>>>>>>>> >>>>>>>>>>>>> /* reuse this slot */ >>>>>>>>>>>>> attr_keyval[i].extra_state = extra_state; >>>>>>>>>>>>> attr_keyval[i.]del = delete_fn; >>>>>>>>>>>>> *keyval = i; >>>>>>>>>>>>> return MPI_SUCCESS; >>>>>>>>>>>>> } >>>>>>>>>>>>> } >>>>>>>>>>>>> return MPIUni_Abort(MPI_COMM_WORLD,1); >>>>>>>>>>>>> } >>>>>>>>>>>>> return MPIUni_Abort(MPI_COMM_WORLD,1); >>>>>>>>>>>>> attr_keyval[num_attr].extra_state = extra_state; >>>>>>>>>>>>> attr_keyval[num_attr].del = delete_fn; >>>>>>>>>>>>> *keyval = num_attr++; >>>>>>>>>>>>> return MPI_SUCCESS; >>>>>>>>>>>>> } >>>>>>>>>>>>> >>>>>>>>>>>>> This will work if the user creates tons of attributes but is >>>>>>>>>>>>> constantly deleting some as they new ones. So long as the number >>>>>>>>>>>>> outstanding at one time is < MAX_ATTR) >>>>>>>>>>>>> >>>>>>>>>>>>> Barry >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Jun 20, 2020, at 10:54 AM, Junchao Zhang < >>>>>>>>>>>>> junchao.zhang at gmail.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> I don't understand what your session means. Let's try this >>>>>>>>>>>>> patch >>>>>>>>>>>>> >>>>>>>>>>>>> diff --git a/src/sys/mpiuni/mpi.c b/src/sys/mpiuni/mpi.c >>>>>>>>>>>>> index d559a513..c058265d 100644 >>>>>>>>>>>>> --- a/src/sys/mpiuni/mpi.c >>>>>>>>>>>>> +++ b/src/sys/mpiuni/mpi.c >>>>>>>>>>>>> @@ -283,6 +283,7 @@ int MPI_Finalize(void) >>>>>>>>>>>>> MPI_Comm_free(&comm); >>>>>>>>>>>>> comm = MPI_COMM_SELF; >>>>>>>>>>>>> MPI_Comm_free(&comm); >>>>>>>>>>>>> + num_attr = 1; /* reset the counter */ >>>>>>>>>>>>> MPI_was_finalized = 1; >>>>>>>>>>>>> return MPI_SUCCESS; >>>>>>>>>>>>> } >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> --Junchao Zhang >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Sat, Jun 20, 2020 at 10:48 AM Sam Guo < >>>>>>>>>>>>> sam.guo at cd-adapco.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Typo: I mean ?Assuming initializer is only needed once for >>>>>>>>>>>>>> entire session? >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Saturday, June 20, 2020, Sam Guo >>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Assuming finalizer is only needed once for entire >>>>>>>>>>>>>>> session(?), I can put initializer into the static block to call it once but >>>>>>>>>>>>>>> where do I call finalizer? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Saturday, June 20, 2020, Junchao Zhang < >>>>>>>>>>>>>>> junchao.zhang at gmail.com> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> The counter num_attr should be recycled. But first try to >>>>>>>>>>>>>>>> call PETSc initialize/Finalize only once to see it fixes the error. >>>>>>>>>>>>>>>> --Junchao Zhang >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Sat, Jun 20, 2020 at 12:48 AM Sam Guo < >>>>>>>>>>>>>>>> sam.guo at cd-adapco.com> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> To clarify, I call PETSc initialize and PETSc finalize >>>>>>>>>>>>>>>>> everytime I call SLEPc: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> PetscInitializeNoPointers(argc,args,nullptr,nullptr); >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> SlepcInitialize(&argc,&args,static_cast(nullptr),help); >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> //calling slepc >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> SlepcFinalize(); >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> PetscFinalize(); >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Fri, Jun 19, 2020 at 10:32 PM Sam Guo < >>>>>>>>>>>>>>>>> sam.guo at cd-adapco.com> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Dear PETSc team, >>>>>>>>>>>>>>>>>> When I called SLEPc multiple time, I eventually got >>>>>>>>>>>>>>>>>> following error: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> MPI operation not supported by PETSc's sequential MPI >>>>>>>>>>>>>>>>>> wrappers >>>>>>>>>>>>>>>>>> [0]PETSC ERROR: #1 PetscInitialize() line 967 in >>>>>>>>>>>>>>>>>> ../../../petsc/src/sys/objects/pinit.c >>>>>>>>>>>>>>>>>> [0]PETSC ERROR: #2 SlepcInitialize() line 262 in >>>>>>>>>>>>>>>>>> ../../../slepc/src/sys/slepcinit.c >>>>>>>>>>>>>>>>>> [0]PETSC ERROR: #3 SlepcInitializeNoPointers() line 359 >>>>>>>>>>>>>>>>>> in ../../../slepc/src/sys/slepcinit.c >>>>>>>>>>>>>>>>>> PETSC ERROR: Logging has not been enabled. >>>>>>>>>>>>>>>>>> You might have forgotten to call PetscInitialize(). >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> I debugged: it is because of following in >>>>>>>>>>>>>>>>>> petsc/src/sys/mpiuni/mpi.c >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> if (num_attr >= MAX_ATTR) >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> in function int MPI_Comm_create_keyval(MPI_Copy_function >>>>>>>>>>>>>>>>>> *copy_fn,MPI_Delete_function *delete_fn,int *keyval,void *extra_state) >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> num_attr is declared static and keeps increasing every >>>>>>>>>>>>>>>>>> time MPI_Comm_create_keyval is called. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> I am using petsc 3.11.3 but found 3.13.2 has the >>>>>>>>>>>>>>>>>> same logic. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Is this a bug or I didn't use it correctly? >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>>> Sam >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>> >>>>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From paololampitella at hotmail.com Sun Jun 28 08:58:05 2020 From: paololampitella at hotmail.com (Paolo Lampitella) Date: Sun, 28 Jun 2020 13:58:05 +0000 Subject: [petsc-users] PETSc and Windows 10 Message-ID: Dear PETSc users, I?ve been an happy PETSc user since version 3.3, using it both under Ubuntu (from 14.04 up to 20.04) and CentOS (from 5 to 8). I use it as an optional component for a parallel Fortran code (that, BTW, also uses metis) and, wherever allowed, I used to install myself MPI (both MPICH and OpenMPI) and PETSc on top of it without any trouble ever (besides being, myself, as dumb as one can be in this). I did this on top of gnu compilers and, less extensively, intel compilers, both on a range of different systems (from virtual machines, to workstations to actual clusters). So far so good. Today I find myself in the need of deploying my application to Windows 10 users, which means giving them a folder with all the executables and libraries to make them run in it, including the mpi runtime. Unfortunately, I also have to rely on free tools (can?t afford Intel for the moment). To the best of my knowledge, considering also far from optimal solutions, my options would then be: Virtual machines and WSL1, Cygwin, MSYS2-MinGW64, Cross compiling with MinGW64 from within Linux, PGI + Visual Studio + Cygwin (not sure about this one) I know this is largely unsupported, but I was wondering if there is, nonetheless, some general (and more official) knowledge available on the matter. What I tried so far: 1. Virtual machines and WSL1: both work like a charm, just like in the native OS, but very far from ideal for the distribution purpose 1. Cygwin with gnu compilers (as opposed to using Intel and Visual Studio): I was unable to compile myself MPI as I am used to on Linux, so I just tried going all in and let PETSc do everything for me (using static linking): download and install MPICH, BLAS, LAPACK, METIS and HYPRE. Everything just worked (for now compiling and making trivial tests) and I am able to use everything from within a cygwin terminal (even with executables and dependencies outside cygwin). Still, even within cygwin, I can?t switch to use, say, the cygwin ompi mpirun/mpiexec for an mpi program compiled with PETSc mpich (things run but not as expected). Some troubles start when I try to use cmd.exe (which I pictured as the more natural way to launch in Windows). In particular, using (note that \ is in cmd.exe, / was used in cygwin terminal): .\mpiexec.hydra.exe -np 8 .\my.exe Nothing happens unless I push Enter a second time. Things seem to work then, but if I try to run a serial executable with the command above I get the following errors (which, instead, doesn?t happen using the cygwin terminal): [proxy:0:0 at Dell7540-Paolo] HYDU_sock_write (utils/sock/sock.c:286): write error (No such process) [proxy:0:0 at Dell7540-Paolo] HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:935): unable to write to downstream stdin [proxy:0:0 at Dell7540-Paolo] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status [proxy:0:0 at Dell7540-Paolo] main (pm/pmiserv/pmip.c:206): demux engine error waiting for event [mpiexec at Dell7540-Paolo] control_cb (pm/pmiserv/pmiserv_cb.c:200): assert (!closed) failed [mpiexec at Dell7540-Paolo] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status [mpiexec at Dell7540-Paolo] HYD_pmci_wait_for_completion (pm/pmiserv/pmiserv_pmci.c:198): error waiting for event [mpiexec at Dell7540-Paolo] main (ui/mpich/mpiexec.c:336): process manager error waiting for completion Just for the sake of completeness, I also tried using the Intel and Microsoft MPI redistributables, which might be more natural candidates, instead of the petsc compiled version of the MPI runtime (and they are MPICH derivatives, after all). But, running with: mpiexec -np 1 my.exe I get the following error with Intel: [cli_0]: write_line error; fd=440 buf=:cmd=init pmi_version=1 pmi_subversion=1 : system msg for write_line failure : Bad file descriptor [cli_0]: Unable to write to PMI_fd [cli_0]: write_line error; fd=440 buf=:cmd=get_appnum : system msg for write_line failure : Bad file descriptor Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(467): MPID_Init(140).......: channel initialization failed MPID_Init(421).......: PMI_Get_appnum returned -1 [cli_0]: aborting job: Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(467): MPID_Init(140).......: channel initialization failed MPID_Init(421).......: PMI_Get_appnum returned -1 And the following error with MS-MPI: [unset]: unable to decode hostport from 44e5747b-d19e-4ea8-ac7a-ec2102cabb21 Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(467): MPID_Init(140).......: channel initialization failed MPID_Init(403).......: PMI_Init returned -1 [unset]: aborting job: Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(467): MPID_Init(140).......: channel initialization failed MPID_Init(403).......: PMI_Init returned -1 independently from the number of processes, but more processes produce more copies of this. However, both Intel and MS-MPI are able to run a serial fortran executable built with cygwin. I think I made everything correctly and adding -localhost didn?t help (actually, it caused more problems to the interpretation of the cmd line arguments for mpiexec) 1. Cygwin with MinGW64 compilers. Never managed to compile MPI, not even trough PETSc. 1. MSYS2+MinGW64 compilers. I understood that MinGW is not well supported, probably because of how it handles paths, but I wanted to give it a try, because it should be more ?native? and there seems to be relevant examples out there that managed to do it. I first tried with the msys2 mpi distribution, produced the .mod file out of the mpi.f90 file in the distribution (I tried my best with different hacks from known limitations of this file as also present in the official MS-MPI distribution) and tried with my code without petsc, but it failed in compiling the code with some strange MPI related error (argument mismatch between two unrelated MPI calls in the code, which is non sense to me). In contrast, simple mpi tests (hello world like) worked as expected. Then I decided to follow this: https://doc.freefem.org/introduction/installation.html#compilation-on-windows but the exact same type of error came up (MPI calls in my code were different, but the error was the same). Trying again from scratch (i.e., without all the things I did in the beginning to compile my code) the same error came up in compiling some of the freefem dependencies (this time not even mpi calls). As a side note, there seems to be an official effort in porting petsc to msys2 (https://github.com/okhlybov/MINGW-packages/tree/whpc/mingw-w64-petsc), but it didn?t get into the official packages yet, which I interpret as a warning 1. Didn?t give a try to cross compiling with MinGw from Linux, as I tought it couldn?t be any better than doing it from MSYS2 2. Didn?t try PGI as I actually didn?t know if I would then been able to make PETSc work. So, here there are some questions I have with respect to where I stand now and the points above: * I haven?t seen the MSYS2-MinGw64 toolchain mentioned at all in official documentation/discussions. Should I definitely abandon it (despite someone mentioning it as working) because of known issues? * What about the PGI route? I don?t see it mentioned as well. I guess it would require some work on win32fe * For my Cygwin-GNU route (basically what is mentioned in PFLOTRAN documentation), am I expected to then run from the cygwin terminal or should the windows prompt work as well? Is the fact that I require a second Enter hit and the mismanagement of serial executables the sign of something wrong with the Windows prompt? * More generally, is there some known working, albeit non official, route given my constraints (free+fortran+windows+mpi+petsc)? Thanks for your attention and your great work on PETSc Best regards Paolo Lampitella -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Sun Jun 28 09:19:27 2020 From: balay at mcs.anl.gov (Satish Balay) Date: Sun, 28 Jun 2020 09:19:27 -0500 (CDT) Subject: [petsc-users] PETSc and Windows 10 In-Reply-To: References: Message-ID: On Sun, 28 Jun 2020, Paolo Lampitella wrote: > Dear PETSc users, > > I?ve been an happy PETSc user since version 3.3, using it both under Ubuntu (from 14.04 up to 20.04) and CentOS (from 5 to 8). > > I use it as an optional component for a parallel Fortran code (that, BTW, also uses metis) and, wherever allowed, I used to install myself MPI (both MPICH and OpenMPI) and PETSc on top of it without any trouble ever (besides being, myself, as dumb as one can be in this). > > I did this on top of gnu compilers and, less extensively, intel compilers, both on a range of different systems (from virtual machines, to workstations to actual clusters). > > So far so good. > > Today I find myself in the need of deploying my application to Windows 10 users, which means giving them a folder with all the executables and libraries to make them run in it, including the mpi runtime. Unfortunately, I also have to rely on free tools (can?t afford Intel for the moment). > > To the best of my knowledge, considering also far from optimal solutions, my options would then be: Virtual machines and WSL1, Cygwin, MSYS2-MinGW64, Cross compiling with MinGW64 from within Linux, PGI + Visual Studio + Cygwin (not sure about this one) > > I know this is largely unsupported, but I was wondering if there is, nonetheless, some general (and more official) knowledge available on the matter. What I tried so far: > > > 1. Virtual machines and WSL1: both work like a charm, just like in the native OS, but very far from ideal for the distribution purpose > > > 1. Cygwin with gnu compilers (as opposed to using Intel and Visual Studio): I was unable to compile myself MPI as I am used to on Linux, so I just tried going all in and let PETSc do everything for me (using static linking): download and install MPICH, BLAS, LAPACK, METIS and HYPRE. Everything just worked (for now compiling and making trivial tests) and I am able to use everything from within a cygwin terminal (even with executables and dependencies outside cygwin). Still, even within cygwin, I can?t switch to use, say, the cygwin ompi mpirun/mpiexec for an mpi program compiled with PETSc mpich (things run but not as expected). Some troubles start when I try to use cmd.exe (which I pictured as the more natural way to launch in Windows). In particular, using (note that \ is in cmd.exe, / was used in cygwin terminal): I don't understand. Why build with MPICH - but use mpiexec from OpenMPI? If it is because you can easily redistribute OpenMPI - why not build PETSc with OpenMPI? You can't use Intel/MS-MPI from cygwin/gcc/gfortran Also - even-though --download-mpich works with cygwin/gcc - its no loner supported on windows [by MPICH group]. > > .\mpiexec.hydra.exe -np 8 .\my.exe > > Nothing happens unless I push Enter a second time. Things seem to work then, but if I try to run a serial executable with the command above I get the following errors (which, instead, doesn?t happen using the cygwin terminal): > > [proxy:0:0 at Dell7540-Paolo] HYDU_sock_write (utils/sock/sock.c:286): write error (No such process) > [proxy:0:0 at Dell7540-Paolo] HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:935): unable to write to downstream stdin > [proxy:0:0 at Dell7540-Paolo] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status > [proxy:0:0 at Dell7540-Paolo] main (pm/pmiserv/pmip.c:206): demux engine error waiting for event > [mpiexec at Dell7540-Paolo] control_cb (pm/pmiserv/pmiserv_cb.c:200): assert (!closed) failed > [mpiexec at Dell7540-Paolo] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status > [mpiexec at Dell7540-Paolo] HYD_pmci_wait_for_completion (pm/pmiserv/pmiserv_pmci.c:198): error waiting for event > [mpiexec at Dell7540-Paolo] main (ui/mpich/mpiexec.c:336): process manager error waiting for completion > > Just for the sake of completeness, I also tried using the Intel and Microsoft MPI redistributables, which might be more natural candidates, instead of the petsc compiled version of the MPI runtime (and they are MPICH derivatives, after all). But, running with: > > mpiexec -np 1 my.exe > > I get the following error with Intel: > > [cli_0]: write_line error; fd=440 buf=:cmd=init pmi_version=1 pmi_subversion=1 > : > system msg for write_line failure : Bad file descriptor > [cli_0]: Unable to write to PMI_fd > [cli_0]: write_line error; fd=440 buf=:cmd=get_appnum > : > system msg for write_line failure : Bad file descriptor > Fatal error in MPI_Init: Other MPI error, error stack: > MPIR_Init_thread(467): > MPID_Init(140).......: channel initialization failed > MPID_Init(421).......: PMI_Get_appnum returned -1 > [cli_0]: aborting job: > Fatal error in MPI_Init: Other MPI error, error stack: > MPIR_Init_thread(467): > MPID_Init(140).......: channel initialization failed > MPID_Init(421).......: PMI_Get_appnum returned -1 > > And the following error with MS-MPI: > > [unset]: unable to decode hostport from 44e5747b-d19e-4ea8-ac7a-ec2102cabb21 > Fatal error in MPI_Init: Other MPI error, error stack: > MPIR_Init_thread(467): > MPID_Init(140).......: channel initialization failed > MPID_Init(403).......: PMI_Init returned -1 > [unset]: aborting job: > Fatal error in MPI_Init: Other MPI error, error stack: > MPIR_Init_thread(467): > MPID_Init(140).......: channel initialization failed > MPID_Init(403).......: PMI_Init returned -1 > > independently from the number of processes, but more processes produce more copies of this. However, both Intel and MS-MPI are able to run a serial fortran executable built with cygwin. I think I made everything correctly and adding -localhost didn?t help (actually, it caused more problems to the interpretation of the cmd line arguments for mpiexec) > > > 1. Cygwin with MinGW64 compilers. Never managed to compile MPI, not even trough PETSc. > > > > 1. MSYS2+MinGW64 compilers. I understood that MinGW is not well supported, probably because of how it handles paths, but I wanted to give it a try, because it should be more ?native? and there seems to be relevant examples out there that managed to do it. I first tried with the msys2 mpi distribution, produced the .mod file out of the mpi.f90 file in the distribution (I tried my best with different hacks from known limitations of this file as also present in the official MS-MPI distribution) and tried with my code without petsc, but it failed in compiling the code with some strange MPI related error (argument mismatch between two unrelated MPI calls in the code, which is non sense to me). In contrast, simple mpi tests (hello world like) worked as expected. Then I decided to follow this: > > > > https://doc.freefem.org/introduction/installation.html#compilation-on-windows > > > > but the exact same type of error came up (MPI calls in my code were different, but the error was the same). Trying again from scratch (i.e., without all the things I did in the beginning to compile my code) the same error came up in compiling some of the freefem dependencies (this time not even mpi calls). > > > > As a side note, there seems to be an official effort in porting petsc to msys2 (https://github.com/okhlybov/MINGW-packages/tree/whpc/mingw-w64-petsc), but it didn?t get into the official packages yet, which I interpret as a warning > > > > 1. Didn?t give a try to cross compiling with MinGw from Linux, as I tought it couldn?t be any better than doing it from MSYS2 > 2. Didn?t try PGI as I actually didn?t know if I would then been able to make PETSc work. > > So, here there are some questions I have with respect to where I stand now and the points above: > > > * I haven?t seen the MSYS2-MinGw64 toolchain mentioned at all in official documentation/discussions. Should I definitely abandon it (despite someone mentioning it as working) because of known issues? I don't have experience with MSYS2-MinGw64, However Pierre does - and perhaps can comment on this. I don't know how things work on the fortran side. > * What about the PGI route? I don?t see it mentioned as well. I guess it would require some work on win32fe Again - no experience here. > * For my Cygwin-GNU route (basically what is mentioned in PFLOTRAN documentation), am I expected to then run from the cygwin terminal or should the windows prompt work as well? Is the fact that I require a second Enter hit and the mismanagement of serial executables the sign of something wrong with the Windows prompt? I would think Cygwin-GNU route should work. I'll have to see if I can reproduce the issues you have. Satish > * More generally, is there some known working, albeit non official, route given my constraints (free+fortran+windows+mpi+petsc)? > > Thanks for your attention and your great work on PETSc > > Best regards > > Paolo Lampitella > From balay at mcs.anl.gov Sun Jun 28 09:30:57 2020 From: balay at mcs.anl.gov (Satish Balay) Date: Sun, 28 Jun 2020 09:30:57 -0500 (CDT) Subject: [petsc-users] PETSc and Windows 10 In-Reply-To: References: Message-ID: BTW: How does redistributing MPI/runtime work with all the choices you have? For ex: with MS-MPI, Intel-MPI - wouldn't the user have to install these packages? [i.e you can't just copy them over to a folder and have mpiexec work - from what I can tell] And how did you plan on installing MPICH - but make mpiexec from OpenMPI redistributable? Did you use OpeMPI from cygwin - or install it manually? And presumably you don't want users installing cygwin. Satish On Sun, 28 Jun 2020, Satish Balay via petsc-users wrote: > On Sun, 28 Jun 2020, Paolo Lampitella wrote: > > > Dear PETSc users, > > > > I?ve been an happy PETSc user since version 3.3, using it both under Ubuntu (from 14.04 up to 20.04) and CentOS (from 5 to 8). > > > > I use it as an optional component for a parallel Fortran code (that, BTW, also uses metis) and, wherever allowed, I used to install myself MPI (both MPICH and OpenMPI) and PETSc on top of it without any trouble ever (besides being, myself, as dumb as one can be in this). > > > > I did this on top of gnu compilers and, less extensively, intel compilers, both on a range of different systems (from virtual machines, to workstations to actual clusters). > > > > So far so good. > > > > Today I find myself in the need of deploying my application to Windows 10 users, which means giving them a folder with all the executables and libraries to make them run in it, including the mpi runtime. Unfortunately, I also have to rely on free tools (can?t afford Intel for the moment). > > > > To the best of my knowledge, considering also far from optimal solutions, my options would then be: Virtual machines and WSL1, Cygwin, MSYS2-MinGW64, Cross compiling with MinGW64 from within Linux, PGI + Visual Studio + Cygwin (not sure about this one) > > > > I know this is largely unsupported, but I was wondering if there is, nonetheless, some general (and more official) knowledge available on the matter. What I tried so far: > > > > > > 1. Virtual machines and WSL1: both work like a charm, just like in the native OS, but very far from ideal for the distribution purpose > > > > > > 1. Cygwin with gnu compilers (as opposed to using Intel and Visual Studio): I was unable to compile myself MPI as I am used to on Linux, so I just tried going all in and let PETSc do everything for me (using static linking): download and install MPICH, BLAS, LAPACK, METIS and HYPRE. Everything just worked (for now compiling and making trivial tests) and I am able to use everything from within a cygwin terminal (even with executables and dependencies outside cygwin). Still, even within cygwin, I can?t switch to use, say, the cygwin ompi mpirun/mpiexec for an mpi program compiled with PETSc mpich (things run but not as expected). Some troubles start when I try to use cmd.exe (which I pictured as the more natural way to launch in Windows). In particular, using (note that \ is in cmd.exe, / was used in cygwin terminal): > > I don't understand. Why build with MPICH - but use mpiexec from OpenMPI? > > If it is because you can easily redistribute OpenMPI - why not build PETSc with OpenMPI? > > You can't use Intel/MS-MPI from cygwin/gcc/gfortran > > Also - even-though --download-mpich works with cygwin/gcc - its no loner supported on windows [by MPICH group]. > > > > > .\mpiexec.hydra.exe -np 8 .\my.exe > > > > Nothing happens unless I push Enter a second time. Things seem to work then, but if I try to run a serial executable with the command above I get the following errors (which, instead, doesn?t happen using the cygwin terminal): > > > > [proxy:0:0 at Dell7540-Paolo] HYDU_sock_write (utils/sock/sock.c:286): write error (No such process) > > [proxy:0:0 at Dell7540-Paolo] HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:935): unable to write to downstream stdin > > [proxy:0:0 at Dell7540-Paolo] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status > > [proxy:0:0 at Dell7540-Paolo] main (pm/pmiserv/pmip.c:206): demux engine error waiting for event > > [mpiexec at Dell7540-Paolo] control_cb (pm/pmiserv/pmiserv_cb.c:200): assert (!closed) failed > > [mpiexec at Dell7540-Paolo] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status > > [mpiexec at Dell7540-Paolo] HYD_pmci_wait_for_completion (pm/pmiserv/pmiserv_pmci.c:198): error waiting for event > > [mpiexec at Dell7540-Paolo] main (ui/mpich/mpiexec.c:336): process manager error waiting for completion > > > > Just for the sake of completeness, I also tried using the Intel and Microsoft MPI redistributables, which might be more natural candidates, instead of the petsc compiled version of the MPI runtime (and they are MPICH derivatives, after all). But, running with: > > > > mpiexec -np 1 my.exe > > > > I get the following error with Intel: > > > > [cli_0]: write_line error; fd=440 buf=:cmd=init pmi_version=1 pmi_subversion=1 > > : > > system msg for write_line failure : Bad file descriptor > > [cli_0]: Unable to write to PMI_fd > > [cli_0]: write_line error; fd=440 buf=:cmd=get_appnum > > : > > system msg for write_line failure : Bad file descriptor > > Fatal error in MPI_Init: Other MPI error, error stack: > > MPIR_Init_thread(467): > > MPID_Init(140).......: channel initialization failed > > MPID_Init(421).......: PMI_Get_appnum returned -1 > > [cli_0]: aborting job: > > Fatal error in MPI_Init: Other MPI error, error stack: > > MPIR_Init_thread(467): > > MPID_Init(140).......: channel initialization failed > > MPID_Init(421).......: PMI_Get_appnum returned -1 > > > > And the following error with MS-MPI: > > > > [unset]: unable to decode hostport from 44e5747b-d19e-4ea8-ac7a-ec2102cabb21 > > Fatal error in MPI_Init: Other MPI error, error stack: > > MPIR_Init_thread(467): > > MPID_Init(140).......: channel initialization failed > > MPID_Init(403).......: PMI_Init returned -1 > > [unset]: aborting job: > > Fatal error in MPI_Init: Other MPI error, error stack: > > MPIR_Init_thread(467): > > MPID_Init(140).......: channel initialization failed > > MPID_Init(403).......: PMI_Init returned -1 > > > > independently from the number of processes, but more processes produce more copies of this. However, both Intel and MS-MPI are able to run a serial fortran executable built with cygwin. I think I made everything correctly and adding -localhost didn?t help (actually, it caused more problems to the interpretation of the cmd line arguments for mpiexec) > > > > > > 1. Cygwin with MinGW64 compilers. Never managed to compile MPI, not even trough PETSc. > > > > > > > > 1. MSYS2+MinGW64 compilers. I understood that MinGW is not well supported, probably because of how it handles paths, but I wanted to give it a try, because it should be more ?native? and there seems to be relevant examples out there that managed to do it. I first tried with the msys2 mpi distribution, produced the .mod file out of the mpi.f90 file in the distribution (I tried my best with different hacks from known limitations of this file as also present in the official MS-MPI distribution) and tried with my code without petsc, but it failed in compiling the code with some strange MPI related error (argument mismatch between two unrelated MPI calls in the code, which is non sense to me). In contrast, simple mpi tests (hello world like) worked as expected. Then I decided to follow this: > > > > > > > > https://doc.freefem.org/introduction/installation.html#compilation-on-windows > > > > > > > > but the exact same type of error came up (MPI calls in my code were different, but the error was the same). Trying again from scratch (i.e., without all the things I did in the beginning to compile my code) the same error came up in compiling some of the freefem dependencies (this time not even mpi calls). > > > > > > > > As a side note, there seems to be an official effort in porting petsc to msys2 (https://github.com/okhlybov/MINGW-packages/tree/whpc/mingw-w64-petsc), but it didn?t get into the official packages yet, which I interpret as a warning > > > > > > > > 1. Didn?t give a try to cross compiling with MinGw from Linux, as I tought it couldn?t be any better than doing it from MSYS2 > > 2. Didn?t try PGI as I actually didn?t know if I would then been able to make PETSc work. > > > > So, here there are some questions I have with respect to where I stand now and the points above: > > > > > > * I haven?t seen the MSYS2-MinGw64 toolchain mentioned at all in official documentation/discussions. Should I definitely abandon it (despite someone mentioning it as working) because of known issues? > > I don't have experience with MSYS2-MinGw64, However Pierre does - and perhaps can comment on this. I don't know how things work on the fortran side. > > > * What about the PGI route? I don?t see it mentioned as well. I guess it would require some work on win32fe > > Again - no experience here. > > > * For my Cygwin-GNU route (basically what is mentioned in PFLOTRAN documentation), am I expected to then run from the cygwin terminal or should the windows prompt work as well? Is the fact that I require a second Enter hit and the mismanagement of serial executables the sign of something wrong with the Windows prompt? > > I would think Cygwin-GNU route should work. I'll have to see if I can reproduce the issues you have. > > Satish > > > * More generally, is there some known working, albeit non official, route given my constraints (free+fortran+windows+mpi+petsc)? > > > > Thanks for your attention and your great work on PETSc > > > > Best regards > > > > Paolo Lampitella > > From paololampitella at hotmail.com Sun Jun 28 09:46:47 2020 From: paololampitella at hotmail.com (Paolo Lampitella) Date: Sun, 28 Jun 2020 14:46:47 +0000 Subject: [petsc-users] R: PETSc and Windows 10 In-Reply-To: References: , Message-ID: Dear Satish, let me first mention that using the OpenMPI runtime for running the executable built on top of the PETSc-mpich toolchain just came as an act of despair and was just a command away (despite knowing the ABI initiative is based on mpich). I already had OpenMPI in Cygwin because I was planning to move there in case of PETSc failing to install MPICH. But as PETSc managed to handle that, that route is on hold for now (and I have not even tought about a plan for destributing it). Also, I didn?t mess up myself with any PATH variable nor I installed anything in system paths, and I Always checked in both systems (cygwin terminal and Windows prompt) that the mpiexec that was used was the correct one. For what I can tell, just copying all the mpi executables made by PETSc in the same folder of my executable (say, a folder on the desktop), going there with the cygwin terminal or the Windows prompt and running from there, actually worked (yet, with the differences I specified in the first mail). So, in this very case, the idea would be that the user just installs cygwin and runs my executable from within the folder I send to him. If MS-MPI or Intel MPI had worked, it wouldn?t have been a problem (in my view) to let them install one of them, as long as a trivial install would have worked. Thanks Inviato da Posta per Windows 10 Da: Satish Balay Inviato: domenica 28 giugno 2020 16:30 A: Satish Balay via petsc-users Cc: Paolo Lampitella; Pierre Jolivet Oggetto: Re: [petsc-users] PETSc and Windows 10 BTW: How does redistributing MPI/runtime work with all the choices you have? For ex: with MS-MPI, Intel-MPI - wouldn't the user have to install these packages? [i.e you can't just copy them over to a folder and have mpiexec work - from what I can tell] And how did you plan on installing MPICH - but make mpiexec from OpenMPI redistributable? Did you use OpeMPI from cygwin - or install it manually? And presumably you don't want users installing cygwin. Satish On Sun, 28 Jun 2020, Satish Balay via petsc-users wrote: > On Sun, 28 Jun 2020, Paolo Lampitella wrote: > > > Dear PETSc users, > > > > I?ve been an happy PETSc user since version 3.3, using it both under Ubuntu (from 14.04 up to 20.04) and CentOS (from 5 to 8). > > > > I use it as an optional component for a parallel Fortran code (that, BTW, also uses metis) and, wherever allowed, I used to install myself MPI (both MPICH and OpenMPI) and PETSc on top of it without any trouble ever (besides being, myself, as dumb as one can be in this). > > > > I did this on top of gnu compilers and, less extensively, intel compilers, both on a range of different systems (from virtual machines, to workstations to actual clusters). > > > > So far so good. > > > > Today I find myself in the need of deploying my application to Windows 10 users, which means giving them a folder with all the executables and libraries to make them run in it, including the mpi runtime. Unfortunately, I also have to rely on free tools (can?t afford Intel for the moment). > > > > To the best of my knowledge, considering also far from optimal solutions, my options would then be: Virtual machines and WSL1, Cygwin, MSYS2-MinGW64, Cross compiling with MinGW64 from within Linux, PGI + Visual Studio + Cygwin (not sure about this one) > > > > I know this is largely unsupported, but I was wondering if there is, nonetheless, some general (and more official) knowledge available on the matter. What I tried so far: > > > > > > 1. Virtual machines and WSL1: both work like a charm, just like in the native OS, but very far from ideal for the distribution purpose > > > > > > 1. Cygwin with gnu compilers (as opposed to using Intel and Visual Studio): I was unable to compile myself MPI as I am used to on Linux, so I just tried going all in and let PETSc do everything for me (using static linking): download and install MPICH, BLAS, LAPACK, METIS and HYPRE. Everything just worked (for now compiling and making trivial tests) and I am able to use everything from within a cygwin terminal (even with executables and dependencies outside cygwin). Still, even within cygwin, I can?t switch to use, say, the cygwin ompi mpirun/mpiexec for an mpi program compiled with PETSc mpich (things run but not as expected). Some troubles start when I try to use cmd.exe (which I pictured as the more natural way to launch in Windows). In particular, using (note that \ is in cmd.exe, / was used in cygwin terminal): > > I don't understand. Why build with MPICH - but use mpiexec from OpenMPI? > > If it is because you can easily redistribute OpenMPI - why not build PETSc with OpenMPI? > > You can't use Intel/MS-MPI from cygwin/gcc/gfortran > > Also - even-though --download-mpich works with cygwin/gcc - its no loner supported on windows [by MPICH group]. > > > > > .\mpiexec.hydra.exe -np 8 .\my.exe > > > > Nothing happens unless I push Enter a second time. Things seem to work then, but if I try to run a serial executable with the command above I get the following errors (which, instead, doesn?t happen using the cygwin terminal): > > > > [proxy:0:0 at Dell7540-Paolo] HYDU_sock_write (utils/sock/sock.c:286): write error (No such process) > > [proxy:0:0 at Dell7540-Paolo] HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:935): unable to write to downstream stdin > > [proxy:0:0 at Dell7540-Paolo] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status > > [proxy:0:0 at Dell7540-Paolo] main (pm/pmiserv/pmip.c:206): demux engine error waiting for event > > [mpiexec at Dell7540-Paolo] control_cb (pm/pmiserv/pmiserv_cb.c:200): assert (!closed) failed > > [mpiexec at Dell7540-Paolo] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status > > [mpiexec at Dell7540-Paolo] HYD_pmci_wait_for_completion (pm/pmiserv/pmiserv_pmci.c:198): error waiting for event > > [mpiexec at Dell7540-Paolo] main (ui/mpich/mpiexec.c:336): process manager error waiting for completion > > > > Just for the sake of completeness, I also tried using the Intel and Microsoft MPI redistributables, which might be more natural candidates, instead of the petsc compiled version of the MPI runtime (and they are MPICH derivatives, after all). But, running with: > > > > mpiexec -np 1 my.exe > > > > I get the following error with Intel: > > > > [cli_0]: write_line error; fd=440 buf=:cmd=init pmi_version=1 pmi_subversion=1 > > : > > system msg for write_line failure : Bad file descriptor > > [cli_0]: Unable to write to PMI_fd > > [cli_0]: write_line error; fd=440 buf=:cmd=get_appnum > > : > > system msg for write_line failure : Bad file descriptor > > Fatal error in MPI_Init: Other MPI error, error stack: > > MPIR_Init_thread(467): > > MPID_Init(140).......: channel initialization failed > > MPID_Init(421).......: PMI_Get_appnum returned -1 > > [cli_0]: aborting job: > > Fatal error in MPI_Init: Other MPI error, error stack: > > MPIR_Init_thread(467): > > MPID_Init(140).......: channel initialization failed > > MPID_Init(421).......: PMI_Get_appnum returned -1 > > > > And the following error with MS-MPI: > > > > [unset]: unable to decode hostport from 44e5747b-d19e-4ea8-ac7a-ec2102cabb21 > > Fatal error in MPI_Init: Other MPI error, error stack: > > MPIR_Init_thread(467): > > MPID_Init(140).......: channel initialization failed > > MPID_Init(403).......: PMI_Init returned -1 > > [unset]: aborting job: > > Fatal error in MPI_Init: Other MPI error, error stack: > > MPIR_Init_thread(467): > > MPID_Init(140).......: channel initialization failed > > MPID_Init(403).......: PMI_Init returned -1 > > > > independently from the number of processes, but more processes produce more copies of this. However, both Intel and MS-MPI are able to run a serial fortran executable built with cygwin. I think I made everything correctly and adding -localhost didn?t help (actually, it caused more problems to the interpretation of the cmd line arguments for mpiexec) > > > > > > 1. Cygwin with MinGW64 compilers. Never managed to compile MPI, not even trough PETSc. > > > > > > > > 1. MSYS2+MinGW64 compilers. I understood that MinGW is not well supported, probably because of how it handles paths, but I wanted to give it a try, because it should be more ?native? and there seems to be relevant examples out there that managed to do it. I first tried with the msys2 mpi distribution, produced the .mod file out of the mpi.f90 file in the distribution (I tried my best with different hacks from known limitations of this file as also present in the official MS-MPI distribution) and tried with my code without petsc, but it failed in compiling the code with some strange MPI related error (argument mismatch between two unrelated MPI calls in the code, which is non sense to me). In contrast, simple mpi tests (hello world like) worked as expected. Then I decided to follow this: > > > > > > > > https://doc.freefem.org/introduction/installation.html#compilation-on-windows > > > > > > > > but the exact same type of error came up (MPI calls in my code were different, but the error was the same). Trying again from scratch (i.e., without all the things I did in the beginning to compile my code) the same error came up in compiling some of the freefem dependencies (this time not even mpi calls). > > > > > > > > As a side note, there seems to be an official effort in porting petsc to msys2 (https://github.com/okhlybov/MINGW-packages/tree/whpc/mingw-w64-petsc), but it didn?t get into the official packages yet, which I interpret as a warning > > > > > > > > 1. Didn?t give a try to cross compiling with MinGw from Linux, as I tought it couldn?t be any better than doing it from MSYS2 > > 2. Didn?t try PGI as I actually didn?t know if I would then been able to make PETSc work. > > > > So, here there are some questions I have with respect to where I stand now and the points above: > > > > > > * I haven?t seen the MSYS2-MinGw64 toolchain mentioned at all in official documentation/discussions. Should I definitely abandon it (despite someone mentioning it as working) because of known issues? > > I don't have experience with MSYS2-MinGw64, However Pierre does - and perhaps can comment on this. I don't know how things work on the fortran side. > > > * What about the PGI route? I don?t see it mentioned as well. I guess it would require some work on win32fe > > Again - no experience here. > > > * For my Cygwin-GNU route (basically what is mentioned in PFLOTRAN documentation), am I expected to then run from the cygwin terminal or should the windows prompt work as well? Is the fact that I require a second Enter hit and the mismanagement of serial executables the sign of something wrong with the Windows prompt? > > I would think Cygwin-GNU route should work. I'll have to see if I can reproduce the issues you have. > > Satish > > > * More generally, is there some known working, albeit non official, route given my constraints (free+fortran+windows+mpi+petsc)? > > > > Thanks for your attention and your great work on PETSc > > > > Best regards > > > > Paolo Lampitella > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pierre.jolivet at enseeiht.fr Sun Jun 28 09:42:06 2020 From: pierre.jolivet at enseeiht.fr (Pierre Jolivet) Date: Sun, 28 Jun 2020 16:42:06 +0200 Subject: [petsc-users] PETSc and Windows 10 In-Reply-To: References: Message-ID: <32B37EFD-DDFA-41AE-ACC7-CB5E6AE57996@enseeiht.fr> Hello Paolo, > On 28 Jun 2020, at 4:19 PM, Satish Balay wrote: > > On Sun, 28 Jun 2020, Paolo Lampitella wrote: > >> 1. MSYS2+MinGW64 compilers. I understood that MinGW is not well supported, probably because of how it handles paths, but I wanted to give it a try, because it should be more ?native? and there seems to be relevant examples out there that managed to do it. I first tried with the msys2 mpi distribution, produced the .mod file out of the mpi.f90 file in the distribution (I tried my best with different hacks from known limitations of this file as also present in the official MS-MPI distribution) and tried with my code without petsc, but it failed in compiling the code with some strange MPI related error (argument mismatch between two unrelated MPI calls in the code, which is non sense to me). In contrast, simple mpi tests (hello world like) worked as expected. Then I decided to follow this: >> >> >> >> https://doc.freefem.org/introduction/installation.html#compilation-on-windows >> Sorry, our (FreeFEM) documentation is not the best? MSYS2+MinGW64 is a fantastic tool to deploy .exe with PETSc. For example, in this .exe https://github.com/FreeFem/FreeFem-sources/releases/download/v4.6/FreeFEM-4.6-win7-64.exe , we ship PETSc + SLEPc (in real + complex) with MS-MPI, hypre, MUMPS, ScaLAPACK, SuperLU, SuiteSparse, ParMETIS, METIS, SCOTCH, TetGen, HPDDM, all compiled by PETSc, needless to say :) There are some tricks, that you can copy/paste from https://github.com/FreeFem/FreeFem-sources/blob/master/3rdparty/ff-petsc/Makefile#L99-L120 Basically, hypre + MinGW64 does not work if you don?t supply '--download-hypre-configure-arguments=--build=x86_64-linux-gnu --host=x86_64-linux-gnu' and all CMake packages need an additional flag as well: '--download-metis-cmake-arguments=-G "MSYS Makefiles"' '--download-parmetis-cmake-arguments=-G "MSYS Makefiles"' '--download-superlu-cmake-arguments=-G "MSYS Makefiles"' This is tested on a daily basis on Windows 7 and Windows 10, so I?m a little puzzled by your MPI problems. I?d suggest you stick to MS-MPI (that?s what we use and it?s trivial to install on MSYS2 https://packages.msys2.org/package/mingw-w64-x86_64-msmpi ). I?m not sure this is specific to PETSc, so feel free to have a chat in private. But I guess we can continue on the mailing list as well, it?s just that there is not much love for MSYS2 over here, sadly. Thanks, Pierre >> >> but the exact same type of error came up (MPI calls in my code were different, but the error was the same). Trying again from scratch (i.e., without all the things I did in the beginning to compile my code) the same error came up in compiling some of the freefem dependencies (this time not even mpi calls). >> >> >> >> As a side note, there seems to be an official effort in porting petsc to msys2 (https://github.com/okhlybov/MINGW-packages/tree/whpc/mingw-w64-petsc), but it didn?t get into the official packages yet, which I interpret as a warning >> >> >> >> 1. Didn?t give a try to cross compiling with MinGw from Linux, as I tought it couldn?t be any better than doing it from MSYS2 >> 2. Didn?t try PGI as I actually didn?t know if I would then been able to make PETSc work. >> >> So, here there are some questions I have with respect to where I stand now and the points above: >> >> >> * I haven?t seen the MSYS2-MinGw64 toolchain mentioned at all in official documentation/discussions. Should I definitely abandon it (despite someone mentioning it as working) because of known issues? > > I don't have experience with MSYS2-MinGw64, However Pierre does - and perhaps can comment on this. I don't know how things work on the fortran side. > >> * What about the PGI route? I don?t see it mentioned as well. I guess it would require some work on win32fe > > Again - no experience here. > >> * For my Cygwin-GNU route (basically what is mentioned in PFLOTRAN documentation), am I expected to then run from the cygwin terminal or should the windows prompt work as well? Is the fact that I require a second Enter hit and the mismanagement of serial executables the sign of something wrong with the Windows prompt? > > I would think Cygwin-GNU route should work. I'll have to see if I can reproduce the issues you have. > > Satish > >> * More generally, is there some known working, albeit non official, route given my constraints (free+fortran+windows+mpi+petsc)? >> >> Thanks for your attention and your great work on PETSc >> >> Best regards >> >> Paolo Lampitella -------------- next part -------------- An HTML attachment was scrubbed... URL: From paololampitella at hotmail.com Sun Jun 28 10:13:50 2020 From: paololampitella at hotmail.com (Paolo Lampitella) Date: Sun, 28 Jun 2020 15:13:50 +0000 Subject: [petsc-users] R: PETSc and Windows 10 In-Reply-To: <32B37EFD-DDFA-41AE-ACC7-CB5E6AE57996@enseeiht.fr> References: , <32B37EFD-DDFA-41AE-ACC7-CB5E6AE57996@enseeiht.fr> Message-ID: Hello Pierre, thank you very much. Knowing that you actually test it on a daily basis is already enough for me to then focus on the MSYS2-MinGw64 toolchain, which would be more straighforward to deploy (instead of having someone install cygwin) and valuable to reuse. I already had the impression that your work on it was recent but, knowing that your actual code is C++, and seeing some recent issues with MS-MPI and gfortran, say https://github.com/microsoft/Microsoft-MPI/issues/33 gave me the impression that the overall toolchain was poorly mantained/tested by Microsoft on the fortran side and maybe this could go undetected in non fortran projects. I can also confirm that my problems with MSYS2 and MinGW64 already started at the MPI level and had nothing to do with PETSc? yet. At this point, I guess, we can either go off radar (if there really isn?t much love for MSYS2 here ?) or keep it going. I will try to rework everything from scratch with MSYS2 and first make extensive MPI tests again. Maybe expect to be bothered again when I try to reuse your Makefile ? Thanks again Paolo Inviato da Posta per Windows 10 Da: Pierre Jolivet Inviato: domenica 28 giugno 2020 16:42 A: Paolo Lampitella Cc: Satish Balay; petsc-users Oggetto: Re: [petsc-users] PETSc and Windows 10 Hello Paolo, On 28 Jun 2020, at 4:19 PM, Satish Balay > wrote: On Sun, 28 Jun 2020, Paolo Lampitella wrote: 1. MSYS2+MinGW64 compilers. I understood that MinGW is not well supported, probably because of how it handles paths, but I wanted to give it a try, because it should be more ?native? and there seems to be relevant examples out there that managed to do it. I first tried with the msys2 mpi distribution, produced the .mod file out of the mpi.f90 file in the distribution (I tried my best with different hacks from known limitations of this file as also present in the official MS-MPI distribution) and tried with my code without petsc, but it failed in compiling the code with some strange MPI related error (argument mismatch between two unrelated MPI calls in the code, which is non sense to me). In contrast, simple mpi tests (hello world like) worked as expected. Then I decided to follow this: https://doc.freefem.org/introduction/installation.html#compilation-on-windows Sorry, our (FreeFEM) documentation is not the best? MSYS2+MinGW64 is a fantastic tool to deploy .exe with PETSc. For example, in this .exe https://github.com/FreeFem/FreeFem-sources/releases/download/v4.6/FreeFEM-4.6-win7-64.exe, we ship PETSc + SLEPc (in real + complex) with MS-MPI, hypre, MUMPS, ScaLAPACK, SuperLU, SuiteSparse, ParMETIS, METIS, SCOTCH, TetGen, HPDDM, all compiled by PETSc, needless to say :) There are some tricks, that you can copy/paste from https://github.com/FreeFem/FreeFem-sources/blob/master/3rdparty/ff-petsc/Makefile#L99-L120 Basically, hypre + MinGW64 does not work if you don?t supply '--download-hypre-configure-arguments=--build=x86_64-linux-gnu --host=x86_64-linux-gnu' and all CMake packages need an additional flag as well: '--download-metis-cmake-arguments=-G "MSYS Makefiles"' '--download-parmetis-cmake-arguments=-G "MSYS Makefiles"' '--download-superlu-cmake-arguments=-G "MSYS Makefiles"' This is tested on a daily basis on Windows 7 and Windows 10, so I?m a little puzzled by your MPI problems. I?d suggest you stick to MS-MPI (that?s what we use and it?s trivial to install on MSYS2 https://packages.msys2.org/package/mingw-w64-x86_64-msmpi). I?m not sure this is specific to PETSc, so feel free to have a chat in private. But I guess we can continue on the mailing list as well, it?s just that there is not much love for MSYS2 over here, sadly. Thanks, Pierre but the exact same type of error came up (MPI calls in my code were different, but the error was the same). Trying again from scratch (i.e., without all the things I did in the beginning to compile my code) the same error came up in compiling some of the freefem dependencies (this time not even mpi calls). As a side note, there seems to be an official effort in porting petsc to msys2 (https://github.com/okhlybov/MINGW-packages/tree/whpc/mingw-w64-petsc), but it didn?t get into the official packages yet, which I interpret as a warning 1. Didn?t give a try to cross compiling with MinGw from Linux, as I tought it couldn?t be any better than doing it from MSYS2 2. Didn?t try PGI as I actually didn?t know if I would then been able to make PETSc work. So, here there are some questions I have with respect to where I stand now and the points above: * I haven?t seen the MSYS2-MinGw64 toolchain mentioned at all in official documentation/discussions. Should I definitely abandon it (despite someone mentioning it as working) because of known issues? I don't have experience with MSYS2-MinGw64, However Pierre does - and perhaps can comment on this. I don't know how things work on the fortran side. * What about the PGI route? I don?t see it mentioned as well. I guess it would require some work on win32fe Again - no experience here. * For my Cygwin-GNU route (basically what is mentioned in PFLOTRAN documentation), am I expected to then run from the cygwin terminal or should the windows prompt work as well? Is the fact that I require a second Enter hit and the mismanagement of serial executables the sign of something wrong with the Windows prompt? I would think Cygwin-GNU route should work. I'll have to see if I can reproduce the issues you have. Satish * More generally, is there some known working, albeit non official, route given my constraints (free+fortran+windows+mpi+petsc)? Thanks for your attention and your great work on PETSc Best regards Paolo Lampitella -------------- next part -------------- An HTML attachment was scrubbed... URL: From pierre.jolivet at enseeiht.fr Sun Jun 28 10:32:17 2020 From: pierre.jolivet at enseeiht.fr (Pierre Jolivet) Date: Sun, 28 Jun 2020 17:32:17 +0200 Subject: [petsc-users] PETSc and Windows 10 In-Reply-To: References: <32B37EFD-DDFA-41AE-ACC7-CB5E6AE57996@enseeiht.fr> Message-ID: <0558CCF7-8203-4520-8673-3B0945C43233@enseeiht.fr> > On 28 Jun 2020, at 5:13 PM, Paolo Lampitella wrote: > > Hello Pierre, > > thank you very much. Knowing that you actually test it on a daily basis is already enough for me to then focus on the MSYS2-MinGw64 toolchain, which would be more straighforward to deploy (instead of having someone install cygwin) and valuable to reuse. > > I already had the impression that your work on it was recent but, knowing that your actual code is C++, and seeing some recent issues with MS-MPI and gfortran, say > > https://github.com/microsoft/Microsoft-MPI/issues/33 > > gave me the impression that the overall toolchain was poorly mantained/tested by Microsoft on the fortran side and maybe this could go undetected in non fortran projects. It may be poorly tested, but it?s working, at least with MUMPS, see attached screenshot. What I?d suggest is: 1) start from scratch with MSYS2 MS-MPI + mingw-w64-x86_64-toolchain, mingw-w64-x86_64-openblas, flex, gawk, gcc, gcc-fortran, gcc-libs, stuff like that (all installed by pacman, cf. https://github.com/FreeFem/FreeFem-sources/blob/master/etc/jenkins/deployRelease/README.md#windows-system ) 2) you need to explicitly _remove_ mingw-w64-x86_64-python because PETSc ./configure cannot work with it (there has been recent changes w.r.t. Python 3 so maybe it?s better now): pacman -R mingw-w64-x86_64-python mingw-w64-x86_64-gdb 3) then use the same makefile as we use for PETSc Do not hesitate if you run into issues. Thanks, Pierre > > I can also confirm that my problems with MSYS2 and MinGW64 already started at the MPI level and had nothing to do with PETSc? yet. > > At this point, I guess, we can either go off radar (if there really isn?t much love for MSYS2 here ?) or keep it going. > > I will try to rework everything from scratch with MSYS2 and first make extensive MPI tests again. Maybe expect to be bothered again when I try to reuse your Makefile ? > > Thanks again > > Paolo > > Inviato da Posta per Windows 10 > > Da: Pierre Jolivet > Inviato: domenica 28 giugno 2020 16:42 > A: Paolo Lampitella > Cc: Satish Balay ; petsc-users > Oggetto: Re: [petsc-users] PETSc and Windows 10 > > Hello Paolo, > > > On 28 Jun 2020, at 4:19 PM, Satish Balay > wrote: > > On Sun, 28 Jun 2020, Paolo Lampitella wrote: > > > 1. MSYS2+MinGW64 compilers. I understood that MinGW is not well supported, probably because of how it handles paths, but I wanted to give it a try, because it should be more ?native? and there seems to be relevant examples out there that managed to do it. I first tried with the msys2 mpi distribution, produced the .mod file out of the mpi.f90 file in the distribution (I tried my best with different hacks from known limitations of this file as also present in the official MS-MPI distribution) and tried with my code without petsc, but it failed in compiling the code with some strange MPI related error (argument mismatch between two unrelated MPI calls in the code, which is non sense to me). In contrast, simple mpi tests (hello world like) worked as expected. Then I decided to follow this: > > > > https://doc.freefem.org/introduction/installation.html#compilation-on-windows > > Sorry, our (FreeFEM) documentation is not the best? > > MSYS2+MinGW64 is a fantastic tool to deploy .exe with PETSc. > For example, in this .exe https://github.com/FreeFem/FreeFem-sources/releases/download/v4.6/FreeFEM-4.6-win7-64.exe , we ship PETSc + SLEPc (in real + complex) with MS-MPI, hypre, MUMPS, ScaLAPACK, SuperLU, SuiteSparse, ParMETIS, METIS, SCOTCH, TetGen, HPDDM, all compiled by PETSc, needless to say :) > There are some tricks, that you can copy/paste from https://github.com/FreeFem/FreeFem-sources/blob/master/3rdparty/ff-petsc/Makefile#L99-L120 > Basically, hypre + MinGW64 does not work if you don?t supply '--download-hypre-configure-arguments=--build=x86_64-linux-gnu --host=x86_64-linux-gnu' and all CMake packages need an additional flag as well: > '--download-metis-cmake-arguments=-G "MSYS Makefiles"' '--download-parmetis-cmake-arguments=-G "MSYS Makefiles"' '--download-superlu-cmake-arguments=-G "MSYS Makefiles"' > > This is tested on a daily basis on Windows 7 and Windows 10, so I?m a little puzzled by your MPI problems. > I?d suggest you stick to MS-MPI (that?s what we use and it?s trivial to install on MSYS2 https://packages.msys2.org/package/mingw-w64-x86_64-msmpi ). > > I?m not sure this is specific to PETSc, so feel free to have a chat in private. > But I guess we can continue on the mailing list as well, it?s just that there is not much love for MSYS2 over here, sadly. > > Thanks, > Pierre > > > > but the exact same type of error came up (MPI calls in my code were different, but the error was the same). Trying again from scratch (i.e., without all the things I did in the beginning to compile my code) the same error came up in compiling some of the freefem dependencies (this time not even mpi calls). > > > > As a side note, there seems to be an official effort in porting petsc to msys2 (https://github.com/okhlybov/MINGW-packages/tree/whpc/mingw-w64-petsc ), but it didn?t get into the official packages yet, which I interpret as a warning > > > > 1. Didn?t give a try to cross compiling with MinGw from Linux, as I tought it couldn?t be any better than doing it from MSYS2 > 2. Didn?t try PGI as I actually didn?t know if I would then been able to make PETSc work. > > So, here there are some questions I have with respect to where I stand now and the points above: > > > * I haven?t seen the MSYS2-MinGw64 toolchain mentioned at all in official documentation/discussions. Should I definitely abandon it (despite someone mentioning it as working) because of known issues? > > I don't have experience with MSYS2-MinGw64, However Pierre does - and perhaps can comment on this. I don't know how things work on the fortran side. > > > * What about the PGI route? I don?t see it mentioned as well. I guess it would require some work on win32fe > > Again - no experience here. > > > * For my Cygwin-GNU route (basically what is mentioned in PFLOTRAN documentation), am I expected to then run from the cygwin terminal or should the windows prompt work as well? Is the fact that I require a second Enter hit and the mismanagement of serial executables the sign of something wrong with the Windows prompt? > > I would think Cygwin-GNU route should work. I'll have to see if I can reproduce the issues you have. > > Satish > > > * More generally, is there some known working, albeit non official, route given my constraints (free+fortran+windows+mpi+petsc)? > > Thanks for your attention and your great work on PETSc > > Best regards > > Paolo Lampitella -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Screenshot 2020-06-28 at 5.17.13 PM.png Type: image/png Size: 462039 bytes Desc: not available URL: From balay at mcs.anl.gov Sun Jun 28 10:39:08 2020 From: balay at mcs.anl.gov (Satish Balay) Date: Sun, 28 Jun 2020 10:39:08 -0500 (CDT) Subject: [petsc-users] PETSc and Windows 10 In-Reply-To: <32B37EFD-DDFA-41AE-ACC7-CB5E6AE57996@enseeiht.fr> References: <32B37EFD-DDFA-41AE-ACC7-CB5E6AE57996@enseeiht.fr> Message-ID: To clarify - I would like to have a petsc test with MSYS2 working. Previously I was lost with MSYS2 - so I stick with what I'm familiar with. However with your instructions - I'm hoping to make progress. What I object to are suggestions (that come up regularly) of replacing what we currently have with: - WSL2 - MSYS2 Satish On Sun, 28 Jun 2020, Pierre Jolivet wrote: > Hello Paolo, > > > On 28 Jun 2020, at 4:19 PM, Satish Balay wrote: > > > > On Sun, 28 Jun 2020, Paolo Lampitella wrote: > > > >> 1. MSYS2+MinGW64 compilers. I understood that MinGW is not well supported, probably because of how it handles paths, but I wanted to give it a try, because it should be more ?native? and there seems to be relevant examples out there that managed to do it. I first tried with the msys2 mpi distribution, produced the .mod file out of the mpi.f90 file in the distribution (I tried my best with different hacks from known limitations of this file as also present in the official MS-MPI distribution) and tried with my code without petsc, but it failed in compiling the code with some strange MPI related error (argument mismatch between two unrelated MPI calls in the code, which is non sense to me). In contrast, simple mpi tests (hello world like) worked as expected. Then I decided to follow this: > >> > >> > >> > >> https://doc.freefem.org/introduction/installation.html#compilation-on-windows > >> > > Sorry, our (FreeFEM) documentation is not the best? > > MSYS2+MinGW64 is a fantastic tool to deploy .exe with PETSc. > For example, in this .exe https://github.com/FreeFem/FreeFem-sources/releases/download/v4.6/FreeFEM-4.6-win7-64.exe , we ship PETSc + SLEPc (in real + complex) with MS-MPI, hypre, MUMPS, ScaLAPACK, SuperLU, SuiteSparse, ParMETIS, METIS, SCOTCH, TetGen, HPDDM, all compiled by PETSc, needless to say :) > There are some tricks, that you can copy/paste from https://github.com/FreeFem/FreeFem-sources/blob/master/3rdparty/ff-petsc/Makefile#L99-L120 > Basically, hypre + MinGW64 does not work if you don?t supply '--download-hypre-configure-arguments=--build=x86_64-linux-gnu --host=x86_64-linux-gnu' and all CMake packages need an additional flag as well: > '--download-metis-cmake-arguments=-G "MSYS Makefiles"' '--download-parmetis-cmake-arguments=-G "MSYS Makefiles"' '--download-superlu-cmake-arguments=-G "MSYS Makefiles"' > > This is tested on a daily basis on Windows 7 and Windows 10, so I?m a little puzzled by your MPI problems. > I?d suggest you stick to MS-MPI (that?s what we use and it?s trivial to install on MSYS2 https://packages.msys2.org/package/mingw-w64-x86_64-msmpi ). > > I?m not sure this is specific to PETSc, so feel free to have a chat in private. > But I guess we can continue on the mailing list as well, it?s just that there is not much love for MSYS2 over here, sadly. > > Thanks, > Pierre > > >> > >> but the exact same type of error came up (MPI calls in my code were different, but the error was the same). Trying again from scratch (i.e., without all the things I did in the beginning to compile my code) the same error came up in compiling some of the freefem dependencies (this time not even mpi calls). > >> > >> > >> > >> As a side note, there seems to be an official effort in porting petsc to msys2 (https://github.com/okhlybov/MINGW-packages/tree/whpc/mingw-w64-petsc), but it didn?t get into the official packages yet, which I interpret as a warning > >> > >> > >> > >> 1. Didn?t give a try to cross compiling with MinGw from Linux, as I tought it couldn?t be any better than doing it from MSYS2 > >> 2. Didn?t try PGI as I actually didn?t know if I would then been able to make PETSc work. > >> > >> So, here there are some questions I have with respect to where I stand now and the points above: > >> > >> > >> * I haven?t seen the MSYS2-MinGw64 toolchain mentioned at all in official documentation/discussions. Should I definitely abandon it (despite someone mentioning it as working) because of known issues? > > > > I don't have experience with MSYS2-MinGw64, However Pierre does - and perhaps can comment on this. I don't know how things work on the fortran side. > > > >> * What about the PGI route? I don?t see it mentioned as well. I guess it would require some work on win32fe > > > > Again - no experience here. > > > >> * For my Cygwin-GNU route (basically what is mentioned in PFLOTRAN documentation), am I expected to then run from the cygwin terminal or should the windows prompt work as well? Is the fact that I require a second Enter hit and the mismanagement of serial executables the sign of something wrong with the Windows prompt? > > > > I would think Cygwin-GNU route should work. I'll have to see if I can reproduce the issues you have. > > > > Satish > > > >> * More generally, is there some known working, albeit non official, route given my constraints (free+fortran+windows+mpi+petsc)? > >> > >> Thanks for your attention and your great work on PETSc > >> > >> Best regards > >> > >> Paolo Lampitella > > From balay at mcs.anl.gov Sun Jun 28 11:17:43 2020 From: balay at mcs.anl.gov (Satish Balay) Date: Sun, 28 Jun 2020 11:17:43 -0500 (CDT) Subject: [petsc-users] PETSc and Windows 10 In-Reply-To: References: Message-ID: On Sun, 28 Jun 2020, Satish Balay via petsc-users wrote: > On Sun, 28 Jun 2020, Paolo Lampitella wrote: > > * For my Cygwin-GNU route (basically what is mentioned in PFLOTRAN documentation), am I expected to then run from the cygwin terminal or should the windows prompt work as well? Is the fact that I require a second Enter hit and the mismanagement of serial executables the sign of something wrong with the Windows prompt? > > I would think Cygwin-GNU route should work. I'll have to see if I can reproduce the issues you have. I attempted a couple of builds - one with mpich and the other with cygwin-openmpi mpich compiled petsc example works sequentially - however mpiexec appears to require cygwin env. >>>>>>>> C:\petsc-install\bin>ex5f Number of SNES iterations = 4 C:\petsc-install\bin>mpiexec -n 1 ex5f [cli_0]: write_line error; fd=448 buf=:cmd=init pmi_version=1 pmi_subversion=1 : system msg for write_line failure : Bad file descriptor [cli_0]: Unable to write to PMI_fd [cli_0]: write_line error; fd=448 buf=:cmd=get_appnum : system msg for write_line failure : Bad file descriptor Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(467): MPID_Init(140).......: channel initialization failed MPID_Init(421).......: PMI_Get_appnum returned -1 [cli_0]: aborting job: Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(467): MPID_Init(140).......: channel initialization failed MPID_Init(421).......: PMI_Get_appnum returned -1 C:\petsc-install\bin> <<<<<< cygwin-openmpi compiled petsc example binary gives errors even for sequential run >>>>>>>> C:\Users\balay\test>ex5f Warning: '/dev/shm' does not exists or is not a directory. POSIX shared memory objects require the existance of this directory. Create the directory '/dev/shm' and set the permissions to 01777. For instance on the command line: mkdir -m 01777 /dev/shm [ps5:00560] [[INVALID],INVALID] ORTE_ERROR_LOG: A system-required executable either could not be found or was not executable by this user in file /cygdrive/d/cyg_pub/devel/openmpi/v3.1/prova/openmpi-3.1.6-1.x86_64/src/openmpi-3.1.6/orte/mca/ess/singleton/ess_singleton_module.c at line 388 [ps5:00560] [[INVALID],INVALID] ORTE_ERROR_LOG: A system-required executable either could not be found or was not executable by this user in file /cygdrive/d/cyg_pub/devel/openmpi/v3.1/prova/openmpi-3.1.6-1.x86_64/src/openmpi-3.1.6/orte/mca/ess/singleton/ess_singleton_module.c at line 166 -------------------------------------------------------------------------- Sorry! You were supposed to get help about: orte_init:startup:internal-failure But I couldn't open the help file: /usr/share/openmpi/help-orte-runtime: No such file or directory. Sorry! -------------------------------------------------------------------------- <<<<<<< So looks like you would need cygwin installed to run Cygwin-MPI binaries.. Also I don't know how cygwin/windows interaction overhead will affect parallel performance. Satish From paololampitella at hotmail.com Sun Jun 28 16:45:45 2020 From: paololampitella at hotmail.com (Paolo Lampitella) Date: Sun, 28 Jun 2020 21:45:45 +0000 Subject: [petsc-users] PETSc and Windows 10 In-Reply-To: References: , Message-ID: Not sure if I did the same. I first made the petsc install in a folder in my cygwin home and then copied the mpich executables, their .dll dependencies, my executable and its dependencies in a folder on my desktop. Then I went there with both terminals (cygwin and window) and launched using mpiexec.hydra.exe in the folder (noting the difference between using ./ and .\ prepended to both executables in the two terminals). With cygwin terminal things worked as expected. It is kind of premarure now for testing performances, but I feel that some compromise here can be admitted, considering the different constraints. I didn't pay too much attention in this phase but I haven't seen nothing suspiciously slow as well (the point is that I don't have a native linux install now to make a meaningful comparison). However, running from the Windows terminal, things worked differently for me. It seems that it worked, but I had to give a second Enter hit... maybe I'm missing something behind the lines. I still have to recompile with OpenMPI to have a meaningful comparison Thanks Paolo Inviato da smartphone Samsung Galaxy. -------- Messaggio originale -------- Da: Satish Balay Data: 28/06/20 18:17 (GMT+01:00) A: Satish Balay via petsc-users Cc: Paolo Lampitella , Pierre Jolivet Oggetto: Re: [petsc-users] PETSc and Windows 10 On Sun, 28 Jun 2020, Satish Balay via petsc-users wrote: > On Sun, 28 Jun 2020, Paolo Lampitella wrote: > > * For my Cygwin-GNU route (basically what is mentioned in PFLOTRAN documentation), am I expected to then run from the cygwin terminal or should the windows prompt work as well? Is the fact that I require a second Enter hit and the mismanagement of serial executables the sign of something wrong with the Windows prompt? > > I would think Cygwin-GNU route should work. I'll have to see if I can reproduce the issues you have. I attempted a couple of builds - one with mpich and the other with cygwin-openmpi mpich compiled petsc example works sequentially - however mpiexec appears to require cygwin env. >>>>>>>> C:\petsc-install\bin>ex5f Number of SNES iterations = 4 C:\petsc-install\bin>mpiexec -n 1 ex5f [cli_0]: write_line error; fd=448 buf=:cmd=init pmi_version=1 pmi_subversion=1 : system msg for write_line failure : Bad file descriptor [cli_0]: Unable to write to PMI_fd [cli_0]: write_line error; fd=448 buf=:cmd=get_appnum : system msg for write_line failure : Bad file descriptor Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(467): MPID_Init(140).......: channel initialization failed MPID_Init(421).......: PMI_Get_appnum returned -1 [cli_0]: aborting job: Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(467): MPID_Init(140).......: channel initialization failed MPID_Init(421).......: PMI_Get_appnum returned -1 C:\petsc-install\bin> <<<<<< cygwin-openmpi compiled petsc example binary gives errors even for sequential run >>>>>>>> C:\Users\balay\test>ex5f Warning: '/dev/shm' does not exists or is not a directory. POSIX shared memory objects require the existance of this directory. Create the directory '/dev/shm' and set the permissions to 01777. For instance on the command line: mkdir -m 01777 /dev/shm [ps5:00560] [[INVALID],INVALID] ORTE_ERROR_LOG: A system-required executable either could not be found or was not executable by this user in file /cygdrive/d/cyg_pub/devel/openmpi/v3.1/prova/openmpi-3.1.6-1.x86_64/src/openmpi-3.1.6/orte/mca/ess/singleton/ess_singleton_module.c at line 388 [ps5:00560] [[INVALID],INVALID] ORTE_ERROR_LOG: A system-required executable either could not be found or was not executable by this user in file /cygdrive/d/cyg_pub/devel/openmpi/v3.1/prova/openmpi-3.1.6-1.x86_64/src/openmpi-3.1.6/orte/mca/ess/singleton/ess_singleton_module.c at line 166 -------------------------------------------------------------------------- Sorry! You were supposed to get help about: orte_init:startup:internal-failure But I couldn't open the help file: /usr/share/openmpi/help-orte-runtime: No such file or directory. Sorry! -------------------------------------------------------------------------- <<<<<<< So looks like you would need cygwin installed to run Cygwin-MPI binaries.. Also I don't know how cygwin/windows interaction overhead will affect parallel performance. Satish -------------- next part -------------- An HTML attachment was scrubbed... URL: From paololampitella at hotmail.com Mon Jun 29 03:32:44 2020 From: paololampitella at hotmail.com (Paolo Lampitella) Date: Mon, 29 Jun 2020 08:32:44 +0000 Subject: [petsc-users] R: PETSc and Windows 10 In-Reply-To: <0558CCF7-8203-4520-8673-3B0945C43233@enseeiht.fr> References: <32B37EFD-DDFA-41AE-ACC7-CB5E6AE57996@enseeiht.fr> , <0558CCF7-8203-4520-8673-3B0945C43233@enseeiht.fr> Message-ID: Thanks Pierre, I?ll give a shot at it as soon as I have everything in place again Paolo Inviato da Posta per Windows 10 Da: Pierre Jolivet Inviato: domenica 28 giugno 2020 17:32 A: Paolo Lampitella Cc: Satish Balay; petsc-users Oggetto: Re: [petsc-users] PETSc and Windows 10 On 28 Jun 2020, at 5:13 PM, Paolo Lampitella > wrote: Hello Pierre, thank you very much. Knowing that you actually test it on a daily basis is already enough for me to then focus on the MSYS2-MinGw64 toolchain, which would be more straighforward to deploy (instead of having someone install cygwin) and valuable to reuse. I already had the impression that your work on it was recent but, knowing that your actual code is C++, and seeing some recent issues with MS-MPI and gfortran, say https://github.com/microsoft/Microsoft-MPI/issues/33 gave me the impression that the overall toolchain was poorly mantained/tested by Microsoft on the fortran side and maybe this could go undetected in non fortran projects. It may be poorly tested, but it?s working, at least with MUMPS, see attached screenshot. What I?d suggest is: 1) start from scratch with MSYS2 MS-MPI + mingw-w64-x86_64-toolchain, mingw-w64-x86_64-openblas, flex, gawk, gcc, gcc-fortran, gcc-libs, stuff like that (all installed by pacman, cf. https://github.com/FreeFem/FreeFem-sources/blob/master/etc/jenkins/deployRelease/README.md#windows-system) 2) you need to explicitly _remove_ mingw-w64-x86_64-python because PETSc ./configure cannot work with it (there has been recent changes w.r.t. Python 3 so maybe it?s better now): pacman -R mingw-w64-x86_64-python mingw-w64-x86_64-gdb 3) then use the same makefile as we use for PETSc Do not hesitate if you run into issues. Thanks, Pierre [cid:image001.png at 01D64D76.54C49F60] I can also confirm that my problems with MSYS2 and MinGW64 already started at the MPI level and had nothing to do with PETSc? yet. At this point, I guess, we can either go off radar (if there really isn?t much love for MSYS2 here ?) or keep it going. I will try to rework everything from scratch with MSYS2 and first make extensive MPI tests again. Maybe expect to be bothered again when I try to reuse your Makefile ? Thanks again Paolo Inviato da Posta per Windows 10 Da: Pierre Jolivet Inviato: domenica 28 giugno 2020 16:42 A: Paolo Lampitella Cc: Satish Balay; petsc-users Oggetto: Re: [petsc-users] PETSc and Windows 10 Hello Paolo, On 28 Jun 2020, at 4:19 PM, Satish Balay > wrote: On Sun, 28 Jun 2020, Paolo Lampitella wrote: 1. MSYS2+MinGW64 compilers. I understood that MinGW is not well supported, probably because of how it handles paths, but I wanted to give it a try, because it should be more ?native? and there seems to be relevant examples out there that managed to do it. I first tried with the msys2 mpi distribution, produced the .mod file out of the mpi.f90 file in the distribution (I tried my best with different hacks from known limitations of this file as also present in the official MS-MPI distribution) and tried with my code without petsc, but it failed in compiling the code with some strange MPI related error (argument mismatch between two unrelated MPI calls in the code, which is non sense to me). In contrast, simple mpi tests (hello world like) worked as expected. Then I decided to follow this: https://doc.freefem.org/introduction/installation.html#compilation-on-windows Sorry, our (FreeFEM) documentation is not the best? MSYS2+MinGW64 is a fantastic tool to deploy .exe with PETSc. For example, in this .exe https://github.com/FreeFem/FreeFem-sources/releases/download/v4.6/FreeFEM-4.6-win7-64.exe, we ship PETSc + SLEPc (in real + complex) with MS-MPI, hypre, MUMPS, ScaLAPACK, SuperLU, SuiteSparse, ParMETIS, METIS, SCOTCH, TetGen, HPDDM, all compiled by PETSc, needless to say :) There are some tricks, that you can copy/paste from https://github.com/FreeFem/FreeFem-sources/blob/master/3rdparty/ff-petsc/Makefile#L99-L120 Basically, hypre + MinGW64 does not work if you don?t supply '--download-hypre-configure-arguments=--build=x86_64-linux-gnu --host=x86_64-linux-gnu' and all CMake packages need an additional flag as well: '--download-metis-cmake-arguments=-G "MSYS Makefiles"' '--download-parmetis-cmake-arguments=-G "MSYS Makefiles"' '--download-superlu-cmake-arguments=-G "MSYS Makefiles"' This is tested on a daily basis on Windows 7 and Windows 10, so I?m a little puzzled by your MPI problems. I?d suggest you stick to MS-MPI (that?s what we use and it?s trivial to install on MSYS2 https://packages.msys2.org/package/mingw-w64-x86_64-msmpi). I?m not sure this is specific to PETSc, so feel free to have a chat in private. But I guess we can continue on the mailing list as well, it?s just that there is not much love for MSYS2 over here, sadly. Thanks, Pierre but the exact same type of error came up (MPI calls in my code were different, but the error was the same). Trying again from scratch (i.e., without all the things I did in the beginning to compile my code) the same error came up in compiling some of the freefem dependencies (this time not even mpi calls). As a side note, there seems to be an official effort in porting petsc to msys2 (https://github.com/okhlybov/MINGW-packages/tree/whpc/mingw-w64-petsc), but it didn?t get into the official packages yet, which I interpret as a warning 1. Didn?t give a try to cross compiling with MinGw from Linux, as I tought it couldn?t be any better than doing it from MSYS2 2. Didn?t try PGI as I actually didn?t know if I would then been able to make PETSc work. So, here there are some questions I have with respect to where I stand now and the points above: * I haven?t seen the MSYS2-MinGw64 toolchain mentioned at all in official documentation/discussions. Should I definitely abandon it (despite someone mentioning it as working) because of known issues? I don't have experience with MSYS2-MinGw64, However Pierre does - and perhaps can comment on this. I don't know how things work on the fortran side. * What about the PGI route? I don?t see it mentioned as well. I guess it would require some work on win32fe Again - no experience here. * For my Cygwin-GNU route (basically what is mentioned in PFLOTRAN documentation), am I expected to then run from the cygwin terminal or should the windows prompt work as well? Is the fact that I require a second Enter hit and the mismanagement of serial executables the sign of something wrong with the Windows prompt? I would think Cygwin-GNU route should work. I'll have to see if I can reproduce the issues you have. Satish * More generally, is there some known working, albeit non official, route given my constraints (free+fortran+windows+mpi+petsc)? Thanks for your attention and your great work on PETSc Best regards Paolo Lampitella -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: DFA1D24AE6EB426B872BCEF219F6A1AD.png Type: image/png Size: 668154 bytes Desc: DFA1D24AE6EB426B872BCEF219F6A1AD.png URL: From paololampitella at hotmail.com Mon Jun 29 09:27:04 2020 From: paololampitella at hotmail.com (Paolo Lampitella) Date: Mon, 29 Jun 2020 14:27:04 +0000 Subject: [petsc-users] R: PETSc and Windows 10 In-Reply-To: References: , Message-ID: As a follow up on the OpenMPI matter in Cygwin, I wasn?t actually able to use the cygwin version at all, not even compiling a simple mpi test. And PETSc fails in using it as well, as it seems unable to find MPI_Init. I might try with having PETSc install it as it did with MPICH, but just for future reference if anyone is interested Paolo Inviato da Posta per Windows 10 Da: Satish Balay Inviato: domenica 28 giugno 2020 18:17 A: Satish Balay via petsc-users Cc: Paolo Lampitella; Pierre Jolivet Oggetto: Re: [petsc-users] PETSc and Windows 10 On Sun, 28 Jun 2020, Satish Balay via petsc-users wrote: > On Sun, 28 Jun 2020, Paolo Lampitella wrote: > > * For my Cygwin-GNU route (basically what is mentioned in PFLOTRAN documentation), am I expected to then run from the cygwin terminal or should the windows prompt work as well? Is the fact that I require a second Enter hit and the mismanagement of serial executables the sign of something wrong with the Windows prompt? > > I would think Cygwin-GNU route should work. I'll have to see if I can reproduce the issues you have. I attempted a couple of builds - one with mpich and the other with cygwin-openmpi mpich compiled petsc example works sequentially - however mpiexec appears to require cygwin env. >>>>>>>> C:\petsc-install\bin>ex5f Number of SNES iterations = 4 C:\petsc-install\bin>mpiexec -n 1 ex5f [cli_0]: write_line error; fd=448 buf=:cmd=init pmi_version=1 pmi_subversion=1 : system msg for write_line failure : Bad file descriptor [cli_0]: Unable to write to PMI_fd [cli_0]: write_line error; fd=448 buf=:cmd=get_appnum : system msg for write_line failure : Bad file descriptor Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(467): MPID_Init(140).......: channel initialization failed MPID_Init(421).......: PMI_Get_appnum returned -1 [cli_0]: aborting job: Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(467): MPID_Init(140).......: channel initialization failed MPID_Init(421).......: PMI_Get_appnum returned -1 C:\petsc-install\bin> <<<<<< cygwin-openmpi compiled petsc example binary gives errors even for sequential run >>>>>>>> C:\Users\balay\test>ex5f Warning: '/dev/shm' does not exists or is not a directory. POSIX shared memory objects require the existance of this directory. Create the directory '/dev/shm' and set the permissions to 01777. For instance on the command line: mkdir -m 01777 /dev/shm [ps5:00560] [[INVALID],INVALID] ORTE_ERROR_LOG: A system-required executable either could not be found or was not executable by this user in file /cygdrive/d/cyg_pub/devel/openmpi/v3.1/prova/openmpi-3.1.6-1.x86_64/src/openmpi-3.1.6/orte/mca/ess/singleton/ess_singleton_module.c at line 388 [ps5:00560] [[INVALID],INVALID] ORTE_ERROR_LOG: A system-required executable either could not be found or was not executable by this user in file /cygdrive/d/cyg_pub/devel/openmpi/v3.1/prova/openmpi-3.1.6-1.x86_64/src/openmpi-3.1.6/orte/mca/ess/singleton/ess_singleton_module.c at line 166 -------------------------------------------------------------------------- Sorry! You were supposed to get help about: orte_init:startup:internal-failure But I couldn't open the help file: /usr/share/openmpi/help-orte-runtime: No such file or directory. Sorry! -------------------------------------------------------------------------- <<<<<<< So looks like you would need cygwin installed to run Cygwin-MPI binaries.. Also I don't know how cygwin/windows interaction overhead will affect parallel performance. Satish -------------- next part -------------- An HTML attachment was scrubbed... URL: From paololampitella at hotmail.com Mon Jun 29 11:27:54 2020 From: paololampitella at hotmail.com (Paolo Lampitella) Date: Mon, 29 Jun 2020 16:27:54 +0000 Subject: [petsc-users] R: PETSc and Windows 10 In-Reply-To: References: <32B37EFD-DDFA-41AE-ACC7-CB5E6AE57996@enseeiht.fr> , <0558CCF7-8203-4520-8673-3B0945C43233@enseeiht.fr>, Message-ID: I think I made the first step of having mingw64 from msys2 working with ms-mpi. I found that the issue I was having was related to: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=91556 and, probably (but impossible to check now), I was using an msys2 and/or mingw mpi package before this fix: https://github.com/msys2/MINGW-packages/commit/11b4cff3d2ec7411037b692b0ad5a9f3e9b9978d#diff-eac59989e3096be97d940c8f47b50fba Admittedly, I never used gcc 10 before on any machine. Still, I feel that reporting that sort of error in that way is, at least, misleading (I would have preferred the initial implementation as mentioned in the gcc bug track). A second thing that I was not used to, and made me more uncertain of the procedure I was following, is having to compile myself the mpi module. There are several version of this out there, but I decided to stick with this one: https://www.scivision.dev/windows-mpi-msys2/ even if there seems to be no need to include -fno-range-check and the current mpi.f90 version is different from the mpif.h as reported here: https://github.com/microsoft/Microsoft-MPI/issues/33 which, to me, are both signs of lack of attention on the fortran side by those that maintain this thing. In summary, this is the procedure I followed so far (on a 64 bit machine with Windows 10): * Install MSYS2 from https://www.msys2.org/ and just follow the install wizard * Open the MSYS2 terminal and execute: pacman -Syuu * Close the terminal when asked and reopen it * Keep executing ?pacman -Syuu? until nothing else needs to be updated * Close the MSYS2 terminal and reopen it (I guess because was in paranoid mode), then install packages with: pacman -S base-devel git gcc gcc-fortran bsdcpio lndir pax-git unzip pacman -S mingw-w64-x86_64-toolchain pacman -S mingw-w64-x86_64-msmpi pacman -S mingw-w64-x86_64-cmake pacman -S mingw-w64-x86_64-freeglut pacman -S mingw-w64-x86_64-gsl pacman -S mingw-w64-x86_64-libmicroutils pacman -S mingw-w64-x86_64-hdf5 pacman -S mingw-w64-x86_64-openblas pacman -S mingw-w64-x86_64-arpack pacman -S mingw-w64-x86_64-jq This set should include all the libraries mentioned by Pierre and/or used by his Jenkins, as the final scope here is to have PETSc and dependencies working. But I think that for pure MPI one could stop to msmpi (even, maybe, just install msmpi and have the dependencies figured out by pacman). Honestly, I don?t remember the exact order I used to install the packages, but this should not affect things. Also, as I was still in paranoid mode, I kept executing ?pacman -Syuu? after each package was installed. After this, close the MSYS2 terminal. * Open the MINGW64 terminal and create the .mod file out of the mpi.f90 file, as mentioned here https://www.scivision.dev/windows-mpi-msys2/, with: cd /mingw64/include gfortran mpif90 -c -fno-range-check -fallow-invalid-boz This is needed to ?USE mpi? (as opposed to INCLUDE ?mpif.h?) * Install the latest MS-MPI (both sdk and setup) from https://www.microsoft.com/en-us/download/details.aspx?id=100593 At this point I?ve been able to compile (using the MINGW64 terminal) different mpi test programs and they run as expected in the classical Windows prompt. I added this function to my .bashrc in MSYS2 in order to easily copy the required dependencies out of MSYS: function copydep() { ldd $1 | grep "=> /$2" | awk '{print $3}' | xargs -I '{}' cp -v '{}' .; } which can be used, with the MINGW64 terminal, by navigating to the folder where the final executable, say, my.exe, resides (even if under a Windows path) and executing: copydep my.exe mingw64 This, of course, must be done before actually trying to execute the .exe in the windows cmd prompt. Hopefully, I should now be able to follow Pierre?s instructions for PETSc (but first I wanna give a try to the system python before removing it) Thanks Paolo -------------- next part -------------- An HTML attachment was scrubbed... URL: From pierre.jolivet at enseeiht.fr Mon Jun 29 11:34:27 2020 From: pierre.jolivet at enseeiht.fr (Pierre Jolivet) Date: Mon, 29 Jun 2020 18:34:27 +0200 Subject: [petsc-users] PETSc and Windows 10 In-Reply-To: References: <32B37EFD-DDFA-41AE-ACC7-CB5E6AE57996@enseeiht.fr> <0558CCF7-8203-4520-8673-3B0945C43233@enseeiht.fr> Message-ID: <20AB9F7A-4471-498B-9A78-AA48163900CB@enseeiht.fr> > On 29 Jun 2020, at 6:27 PM, Paolo Lampitella wrote: > > I think I made the first step of having mingw64 from msys2 working with ms-mpi. > > I found that the issue I was having was related to: > > https://gcc.gnu.org/bugzilla/show_bug.cgi?id=91556 > > and, probably (but impossible to check now), I was using an msys2 and/or mingw mpi package before this fix: > > https://github.com/msys2/MINGW-packages/commit/11b4cff3d2ec7411037b692b0ad5a9f3e9b9978d#diff-eac59989e3096be97d940c8f47b50fba > > Admittedly, I never used gcc 10 before on any machine. Still, I feel that reporting that sort of error in that way is, > at least, misleading (I would have preferred the initial implementation as mentioned in the gcc bug track). > > A second thing that I was not used to, and made me more uncertain of the procedure I was following, is having to compile myself the mpi module. There are several version of this out there, but I decided to stick with this one: > > https://www.scivision.dev/windows-mpi-msys2/ > > even if there seems to be no need to include -fno-range-check and the current mpi.f90 version is different from the mpif.h as reported here: > > https://github.com/microsoft/Microsoft-MPI/issues/33 > > which, to me, are both signs of lack of attention on the fortran side by those that maintain this thing. > > In summary, this is the procedure I followed so far (on a 64 bit machine with Windows 10): > > Install MSYS2 from https://www.msys2.org/ and just follow the install wizard > Open the MSYS2 terminal and execute: pacman -Syuu > Close the terminal when asked and reopen it > Keep executing ?pacman -Syuu? until nothing else needs to be updated > Close the MSYS2 terminal and reopen it (I guess because was in paranoid mode), then install packages with: > > pacman -S base-devel git gcc gcc-fortran bsdcpio lndir pax-git unzip > pacman -S mingw-w64-x86_64-toolchain > pacman -S mingw-w64-x86_64-msmpi > pacman -S mingw-w64-x86_64-cmake > pacman -S mingw-w64-x86_64-freeglut > pacman -S mingw-w64-x86_64-gsl > pacman -S mingw-w64-x86_64-libmicroutils > pacman -S mingw-w64-x86_64-hdf5 > pacman -S mingw-w64-x86_64-openblas > pacman -S mingw-w64-x86_64-arpack > pacman -S mingw-w64-x86_64-jq > > This set should include all the libraries mentioned by Pierre and/or used by his Jenkins, as the final scope here is to have PETSc and dependencies working. But I think that for pure MPI one could stop to msmpi (even, maybe, just install msmpi and have the dependencies figured out by pacman). Honestly, I don?t remember the exact order I used to install the packages, but this should not affect things. Also, as I was still in paranoid mode, I kept executing ?pacman -Syuu? after each package was installed. After this, close the MSYS2 terminal. > > Open the MINGW64 terminal and create the .mod file out of the mpi.f90 file, as mentioned here https://www.scivision.dev/windows-mpi-msys2/ , with: > > cd /mingw64/include > gfortran mpif90 -c -fno-range-check -fallow-invalid-boz Ah, yes, that?s new to gfortran 10 (we use gfortran 9 on our workers), which is now what?s ship with MSYS2 (we haven?t updated yet). Sorry that I forgot about that. > This is needed to ?USE mpi? (as opposed to INCLUDE ?mpif.h?) > > Install the latest MS-MPI (both sdk and setup) from https://www.microsoft.com/en-us/download/details.aspx?id=100593 > > At this point I?ve been able to compile (using the MINGW64 terminal) different mpi test programs and they run as expected in the classical Windows prompt. I added this function to my .bashrc in MSYS2 in order to easily copy the required dependencies out of MSYS: > > function copydep() { ldd $1 | grep "=> /$2" | awk '{print $3}' | xargs -I '{}' cp -v '{}' .; } > > which can be used, with the MINGW64 terminal, by navigating to the folder where the final executable, say, my.exe, resides (even if under a Windows path) and executing: > > copydep my.exe mingw64 > > This, of course, must be done before actually trying to execute the .exe in the windows cmd prompt. > > Hopefully, I should now be able to follow Pierre?s instructions for PETSc (but first I wanna give a try to the system python before removing it) Looks like the hard part is over. It?s usually easier to deal with ./configure issues. If you have weird errors like ?incomplete Cygwin install? or whatever, this is the kind of issues I was referring to earlier. In that case, what I?d suggest is just, as before: pacman -R mingw-w64-x86_64-python mingw-w64-x86_64-gdb pacman -S python Thanks, Pierre > Thanks > > Paolo -------------- next part -------------- An HTML attachment was scrubbed... URL: From paololampitella at hotmail.com Mon Jun 29 12:34:31 2020 From: paololampitella at hotmail.com (Paolo Lampitella) Date: Mon, 29 Jun 2020 17:34:31 +0000 Subject: [petsc-users] R: PETSc and Windows 10 In-Reply-To: <20AB9F7A-4471-498B-9A78-AA48163900CB@enseeiht.fr> References: <32B37EFD-DDFA-41AE-ACC7-CB5E6AE57996@enseeiht.fr> <0558CCF7-8203-4520-8673-3B0945C43233@enseeiht.fr> , <20AB9F7A-4471-498B-9A78-AA48163900CB@enseeiht.fr> Message-ID: Dear Pierre, sorry to bother you, but I already have some issues. What I did: * pacman -R mingw-w64-x86_64-python mingw-w64-x86_64-gdb (is gdb also troublesome?) * Followed points 6 and 7 at https://doc.freefem.org/introduction/installation.html#compilation-on-windows I first got a warning on the configure at point 6, as ?disable-hips is not recognized. Then, on make ?petsc-slepc? of point 7 (no SUDO=sudo flag was necessary) I got to this point: tar xzf ../pkg/petsc-lite-3.13.0.tar.gz patch -p1 < petsc-suitesparse.patch patching file petsc-3.13.0/config/BuildSystem/config/packages/SuiteSparse.py touch petsc-3.13.0/tag-tar cd petsc-3.13.0 && ./configure MAKEFLAGS='' \ --prefix=/home/paolo/freefem/ff-petsc//r \ --with-debugging=0 COPTFLAGS='-O3 -mtune=generic' CXXOPTFLAGS='-O3 -mtune=generic' FOPTFLAGS='-O3 -mtune=generic' --with-cxx-dialect=C++11 --with-ssl=0 --with-x=0 --with-fortran-bindings=0 --with-shared-libraries=0 --with-cc='gcc' --with-cxx='g++' --with-fc='gfortran' CXXFLAGS='-fno-stack-protector' CFLAGS='-fno-stack-protector' --with-scalar-type=real --with-mpi-lib='/c/Windows/System32/msmpi.dll' --with-mpi-include='/home/paolo/FreeFem-sources/3rdparty/include/msmpi' --with-mpiexec='/C/Program\ Files/Microsoft\ MPI/Bin/mpiexec' --with-blaslapack-include='' --with-blaslapack-lib='/mingw64/bin/libopenblas.dll' --download-scalapack --download-metis --download-ptscotch --download-mumps --download-hypre --download-parmetis --download-superlu --download-suitesparse --download-tetgen --download-slepc '--download-metis-cmake-arguments=-G "MSYS Makefiles"' '--download-parmetis-cmake-arguments=-G "MSYS Makefiles"' '--download-superlu-cmake-arguments=-G "MSYS Makefiles"' '--download-hypre-configure-arguments=--build=x86_64-linux-gnu --host=x86_64-linux-gnu' PETSC_ARCH=fr =============================================================================== Configuring PETSc to compile on your system =============================================================================== TESTING: FortranMPICheck from config.packages.MPI(config/BuildSystem/config/pack******************************************************************************* UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): ------------------------------------------------------------------------------- Fortran error! mpi_init() could not be located! ******************************************************************************* make: *** [Makefile:210: petsc-3.13.0/tag-conf-real] Errore 1 Note that I didn?t add anything to any PATH variable, because this is not mentioned in your documentation. On a side note, this is the same error I got when trying to build PETSc in Cygwin with the default OpenMPI available in Cygwin. I am attaching the configure.log? it seems to me that the error comes from the configure trying to include the mpif.h in your folder and not using the -fallow-invalid-boz flag that I had to use, for example, to compile mpi.f90 into mpi.mod But I?m not sure why this is happening Thanks Paolo Inviato da Posta per Windows 10 Da: Pierre Jolivet Inviato: luned? 29 giugno 2020 18:34 A: Paolo Lampitella Cc: Satish Balay; petsc-users Oggetto: Re: [petsc-users] PETSc and Windows 10 On 29 Jun 2020, at 6:27 PM, Paolo Lampitella > wrote: I think I made the first step of having mingw64 from msys2 working with ms-mpi. I found that the issue I was having was related to: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=91556 and, probably (but impossible to check now), I was using an msys2 and/or mingw mpi package before this fix: https://github.com/msys2/MINGW-packages/commit/11b4cff3d2ec7411037b692b0ad5a9f3e9b9978d#diff-eac59989e3096be97d940c8f47b50fba Admittedly, I never used gcc 10 before on any machine. Still, I feel that reporting that sort of error in that way is, at least, misleading (I would have preferred the initial implementation as mentioned in the gcc bug track). A second thing that I was not used to, and made me more uncertain of the procedure I was following, is having to compile myself the mpi module. There are several version of this out there, but I decided to stick with this one: https://www.scivision.dev/windows-mpi-msys2/ even if there seems to be no need to include -fno-range-check and the current mpi.f90 version is different from the mpif.h as reported here: https://github.com/microsoft/Microsoft-MPI/issues/33 which, to me, are both signs of lack of attention on the fortran side by those that maintain this thing. In summary, this is the procedure I followed so far (on a 64 bit machine with Windows 10): * Install MSYS2 from https://www.msys2.org/ and just follow the install wizard * Open the MSYS2 terminal and execute: pacman -Syuu * Close the terminal when asked and reopen it * Keep executing ?pacman -Syuu? until nothing else needs to be updated * Close the MSYS2 terminal and reopen it (I guess because was in paranoid mode), then install packages with: pacman -S base-devel git gcc gcc-fortran bsdcpio lndir pax-git unzip pacman -S mingw-w64-x86_64-toolchain pacman -S mingw-w64-x86_64-msmpi pacman -S mingw-w64-x86_64-cmake pacman -S mingw-w64-x86_64-freeglut pacman -S mingw-w64-x86_64-gsl pacman -S mingw-w64-x86_64-libmicroutils pacman -S mingw-w64-x86_64-hdf5 pacman -S mingw-w64-x86_64-openblas pacman -S mingw-w64-x86_64-arpack pacman -S mingw-w64-x86_64-jq This set should include all the libraries mentioned by Pierre and/or used by his Jenkins, as the final scope here is to have PETSc and dependencies working. But I think that for pure MPI one could stop to msmpi (even, maybe, just install msmpi and have the dependencies figured out by pacman). Honestly, I don?t remember the exact order I used to install the packages, but this should not affect things. Also, as I was still in paranoid mode, I kept executing ?pacman -Syuu? after each package was installed. After this, close the MSYS2 terminal. * Open the MINGW64 terminal and create the .mod file out of the mpi.f90 file, as mentioned here https://www.scivision.dev/windows-mpi-msys2/, with: cd /mingw64/include gfortran mpif90 -c -fno-range-check -fallow-invalid-boz Ah, yes, that?s new to gfortran 10 (we use gfortran 9 on our workers), which is now what?s ship with MSYS2 (we haven?t updated yet). Sorry that I forgot about that. This is needed to ?USE mpi? (as opposed to INCLUDE ?mpif.h?) * Install the latest MS-MPI (both sdk and setup) from https://www.microsoft.com/en-us/download/details.aspx?id=100593 At this point I?ve been able to compile (using the MINGW64 terminal) different mpi test programs and they run as expected in the classical Windows prompt. I added this function to my .bashrc in MSYS2 in order to easily copy the required dependencies out of MSYS: function copydep() { ldd $1 | grep "=> /$2" | awk '{print $3}' | xargs -I '{}' cp -v '{}' .; } which can be used, with the MINGW64 terminal, by navigating to the folder where the final executable, say, my.exe, resides (even if under a Windows path) and executing: copydep my.exe mingw64 This, of course, must be done before actually trying to execute the .exe in the windows cmd prompt. Hopefully, I should now be able to follow Pierre?s instructions for PETSc (but first I wanna give a try to the system python before removing it) Looks like the hard part is over. It?s usually easier to deal with ./configure issues. If you have weird errors like ?incomplete Cygwin install? or whatever, this is the kind of issues I was referring to earlier. In that case, what I?d suggest is just, as before: pacman -R mingw-w64-x86_64-python mingw-w64-x86_64-gdb pacman -S python Thanks, Pierre Thanks Paolo -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: configure.log Type: application/octet-stream Size: 662202 bytes Desc: configure.log URL: From knepley at gmail.com Mon Jun 29 12:47:15 2020 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 29 Jun 2020 13:47:15 -0400 Subject: [petsc-users] R: PETSc and Windows 10 In-Reply-To: References: <32B37EFD-DDFA-41AE-ACC7-CB5E6AE57996@enseeiht.fr> <0558CCF7-8203-4520-8673-3B0945C43233@enseeiht.fr> <20AB9F7A-4471-498B-9A78-AA48163900CB@enseeiht.fr> Message-ID: On Mon, Jun 29, 2020 at 1:35 PM Paolo Lampitella < paololampitella at hotmail.com> wrote: > Dear Pierre, sorry to bother you, but I already have some issues. What I > did: > > > > - pacman -R mingw-w64-x86_64-python mingw-w64-x86_64-gdb (is gdb also > troublesome?) > - Followed points 6 and 7 at > https://doc.freefem.org/introduction/installation.html#compilation-on-windows > > I first got a warning on the configure at point 6, as ?disable-hips is not > recognized. Then, on make ?petsc-slepc? of point 7 (no SUDO=sudo flag was > necessary) I got to this point: > > > > tar xzf ../pkg/petsc-lite-3.13.0.tar.gz > > patch -p1 < petsc-suitesparse.patch > > patching file > petsc-3.13.0/config/BuildSystem/config/packages/SuiteSparse.py > > touch petsc-3.13.0/tag-tar > > cd petsc-3.13.0 && ./configure MAKEFLAGS='' \ > > --prefix=/home/paolo/freefem/ff-petsc//r \ > > --with-debugging=0 COPTFLAGS='-O3 -mtune=generic' CXXOPTFLAGS='-O3 > -mtune=generic' FOPTFLAGS='-O3 -mtune=generic' --with-cxx-dialect=C++11 > --with-ssl=0 --with-x=0 --with-fortran-bindings=0 --with-shared-libraries=0 > --with-cc='gcc' --with-cxx='g++' --with-fc='gfortran' > CXXFLAGS='-fno-stack-protector' CFLAGS='-fno-stack-protector' > --with-scalar-type=real --with-mpi-lib='/c/Windows/System32/msmpi.dll' > --with-mpi-include='/home/paolo/FreeFem-sources/3rdparty/include/msmpi' > --with-mpiexec='/C/Program\ Files/Microsoft\ MPI/Bin/mpiexec' > --with-blaslapack-include='' > --with-blaslapack-lib='/mingw64/bin/libopenblas.dll' --download-scalapack > --download-metis --download-ptscotch --download-mumps --download-hypre > --download-parmetis --download-superlu --download-suitesparse > --download-tetgen --download-slepc '--download-metis-cmake-arguments=-G > "MSYS Makefiles"' '--download-parmetis-cmake-arguments=-G "MSYS Makefiles"' > '--download-superlu-cmake-arguments=-G "MSYS Makefiles"' > '--download-hypre-configure-arguments=--build=x86_64-linux-gnu > --host=x86_64-linux-gnu' PETSC_ARCH=fr > > > =============================================================================== > > Configuring PETSc to compile on your system > > > =============================================================================== > > TESTING: FortranMPICheck from > config.packages.MPI(config/BuildSystem/config/pack******************************************************************************* > > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for > details): > > > ------------------------------------------------------------------------------- > > Fortran error! mpi_init() could not be located! > > > ******************************************************************************* > > > > make: *** [Makefile:210: petsc-3.13.0/tag-conf-real] Errore 1 > > > > Note that I didn?t add anything to any PATH variable, because this is not > mentioned in your documentation. > > > > On a side note, this is the same error I got when trying to build PETSc in > Cygwin with the default OpenMPI available in Cygwin. > > > > I am attaching the configure.log? it seems to me that the error comes from > the configure trying to include the mpif.h in your folder and not using the > -fallow-invalid-boz flag that I had to use, for example, to compile mpi.f90 > into mpi.mod > > > > But I?m not sure why this is happening > Pierre, Could this be due to gcc 10? Executing: gfortran -c -o /tmp/petsc-ur0cff6a/config.libraries/conftest.o -I/tmp/petsc-ur0cff6a/config.compilers -I/tmp/petsc-ur0cff6a/config.setCompilers -I/tmp/petsc-ur0cff6a/config.compilersFortran -I/tmp/petsc-ur0cff6a/config.libraries -Wall -ffree-line-length-0 -Wno-unused-dummy-argument -O3 -mtune=generic -I/home/paolo/FreeFem-sources/3rdparty/include/msmpi /tmp/petsc-ur0cff6a/config.libraries/conftest.F90 Possible ERROR while running compiler: exit code 1 stderr: C:/msys64/home/paolo/FreeFem-sources/3rdparty/include/msmpi/mpif.h:227:36: 227 | PARAMETER (MPI_DATATYPE_NULL=z'0c000000') | 1 Error: BOZ literal constant at (1) is neither a data-stmt-constant nor an actual argument to INT, REAL, DBLE, or CMPLX intrinsic function [see '-fno-allow-invalid-boz'] C:/msys64/home/paolo/FreeFem-sources/3rdparty/include/msmpi/mpif.h:303:27: 303 | PARAMETER (MPI_CHAR=z'4c000101') | 1 Error: BOZ literal constant at (1) is neither a data-stmt-constant nor an actual argument to INT, REAL, DBLE, or CMPLX intrinsic function [see '-fno-allow-invalid-boz'] C:/msys64/home/paolo/FreeFem-sources/3rdparty/include/msmpi/mpif.h:305:36: 305 | PARAMETER (MPI_UNSIGNED_CHAR=z'4c000102') | 1 Thanks, Matt > Thanks > > > > Paolo > > > > Inviato da Posta per > Windows 10 > > > > *Da: *Pierre Jolivet > *Inviato: *luned? 29 giugno 2020 18:34 > *A: *Paolo Lampitella > *Cc: *Satish Balay ; petsc-users > > *Oggetto: *Re: [petsc-users] PETSc and Windows 10 > > > > > > > > On 29 Jun 2020, at 6:27 PM, Paolo Lampitella > wrote: > > > > I think I made the first step of having mingw64 from msys2 working with > ms-mpi. > > > > I found that the issue I was having was related to: > > > > https://gcc.gnu.org/bugzilla/show_bug.cgi?id=91556 > > > > and, probably (but impossible to check now), I was using an msys2 and/or > mingw mpi package before this fix: > > > > > https://github.com/msys2/MINGW-packages/commit/11b4cff3d2ec7411037b692b0ad5a9f3e9b9978d#diff-eac59989e3096be97d940c8f47b50fba > > > > Admittedly, I never used gcc 10 before on any machine. Still, I feel that > reporting that sort of error in that way is, > > at least, misleading (I would have preferred the initial implementation as > mentioned in the gcc bug track). > > > > A second thing that I was not used to, and made me more uncertain of the > procedure I was following, is having to compile myself the mpi module. > There are several version of this out there, but I decided to stick with > this one: > > > > https://www.scivision.dev/windows-mpi-msys2/ > > > > even if there seems to be no need to include -fno-range-check and the > current mpi.f90 version is different from the mpif.h as reported here: > > > > https://github.com/microsoft/Microsoft-MPI/issues/33 > > > > which, to me, are both signs of lack of attention on the fortran side by > those that maintain this thing. > > > > In summary, this is the procedure I followed so far (on a 64 bit machine > with Windows 10): > > > > - Install MSYS2 from https://www.msys2.org/ and just follow the > install wizard > - Open the MSYS2 terminal and execute: pacman -Syuu > - Close the terminal when asked and reopen it > - Keep executing ?pacman -Syuu? until nothing else needs to be updated > - Close the MSYS2 terminal and reopen it (I guess because was in > paranoid mode), then install packages with: > > > > pacman -S base-devel git gcc gcc-fortran bsdcpio lndir pax-git unzip > > pacman -S mingw-w64-x86_64-toolchain > > pacman -S mingw-w64-x86_64-msmpi > > pacman -S mingw-w64-x86_64-cmake > > pacman -S mingw-w64-x86_64-freeglut > > pacman -S mingw-w64-x86_64-gsl > > pacman -S mingw-w64-x86_64-libmicroutils > > pacman -S mingw-w64-x86_64-hdf5 > > pacman -S mingw-w64-x86_64-openblas > > pacman -S mingw-w64-x86_64-arpack > > pacman -S mingw-w64-x86_64-jq > > > > This set should include all the libraries mentioned by Pierre and/or used > by his Jenkins, as the final scope here is to have PETSc and dependencies > working. But I think that for pure MPI one could stop to msmpi (even, > maybe, just install msmpi and have the dependencies figured out by pacman). > Honestly, I don?t remember the exact order I used to install the packages, > but this should not affect things. Also, as I was still in paranoid mode, I > kept executing ?pacman -Syuu? after each package was installed. After this, > close the MSYS2 terminal. > > > > - Open the MINGW64 terminal and create the .mod file out of the > mpi.f90 file, as mentioned here > https://www.scivision.dev/windows-mpi-msys2/, with: > > > > cd /mingw64/include > > gfortran mpif90 -c -fno-range-check -fallow-invalid-boz > > > > Ah, yes, that?s new to gfortran 10 (we use gfortran 9 on our workers), > which is now what?s ship with MSYS2 (we haven?t updated yet). Sorry that I > forgot about that. > > > > This is needed to ?USE mpi? (as opposed to INCLUDE ?mpif.h?) > > > > - Install the latest MS-MPI (both sdk and setup) from > https://www.microsoft.com/en-us/download/details.aspx?id=100593 > > > > At this point I?ve been able to compile (using the MINGW64 terminal) > different mpi test programs and they run as expected in the classical > Windows prompt. I added this function to my .bashrc in MSYS2 in order to > easily copy the required dependencies out of MSYS: > > > > function copydep() { ldd $1 | grep "=> /$2" | awk '{print $3}' | xargs -I > '{}' cp -v '{}' .; } > > > > which can be used, with the MINGW64 terminal, by navigating to the folder > where the final executable, say, my.exe, resides (even if under a Windows > path) and executing: > > > > copydep my.exe mingw64 > > > > This, of course, must be done before actually trying to execute the .exe > in the windows cmd prompt. > > > > Hopefully, I should now be able to follow Pierre?s instructions for PETSc > (but first I wanna give a try to the system python before removing it) > > > > Looks like the hard part is over. It?s usually easier to deal with > ./configure issues. > > If you have weird errors like ?incomplete Cygwin install? or whatever, > this is the kind of issues I was referring to earlier. > > In that case, what I?d suggest is just, as before: > > pacman -R mingw-w64-x86_64-python mingw-w64-x86_64-gdb > > pacman -S python > > > > Thanks, > > Pierre > > > > Thanks > > > > Paolo > > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From pierre.jolivet at enseeiht.fr Mon Jun 29 13:01:12 2020 From: pierre.jolivet at enseeiht.fr (Pierre Jolivet) Date: Mon, 29 Jun 2020 20:01:12 +0200 Subject: [petsc-users] PETSc and Windows 10 In-Reply-To: References: <32B37EFD-DDFA-41AE-ACC7-CB5E6AE57996@enseeiht.fr> <0558CCF7-8203-4520-8673-3B0945C43233@enseeiht.fr> <20AB9F7A-4471-498B-9A78-AA48163900CB@enseeiht.fr> Message-ID: <94A8E366-2361-409E-84CE-B90EA54D9AF9@enseeiht.fr> > On 29 Jun 2020, at 7:47 PM, Matthew Knepley wrote: > > On Mon, Jun 29, 2020 at 1:35 PM Paolo Lampitella > wrote: > Dear Pierre, sorry to bother you, but I already have some issues. What I did: > > > > pacman -R mingw-w64-x86_64-python mingw-w64-x86_64-gdb (is gdb also troublesome?) > Followed points 6 and 7 at https://doc.freefem.org/introduction/installation.html#compilation-on-windows > I first got a warning on the configure at point 6, as ?disable-hips is not recognized. Then, on make ?petsc-slepc? of point 7 (no SUDO=sudo flag was necessary) I got to this point: > > > > tar xzf ../pkg/petsc-lite-3.13.0.tar.gz > > patch -p1 < petsc-suitesparse.patch > > patching file petsc-3.13.0/config/BuildSystem/config/packages/SuiteSparse.py > > touch petsc-3.13.0/tag-tar > > cd petsc-3.13.0 && ./configure MAKEFLAGS='' \ > > --prefix=/home/paolo/freefem/ff-petsc//r \ > > --with-debugging=0 COPTFLAGS='-O3 -mtune=generic' CXXOPTFLAGS='-O3 -mtune=generic' FOPTFLAGS='-O3 -mtune=generic' --with-cxx-dialect=C++11 --with-ssl=0 --with-x=0 --with-fortran-bindings=0 --with-shared-libraries=0 --with-cc='gcc' --with-cxx='g++' --with-fc='gfortran' CXXFLAGS='-fno-stack-protector' CFLAGS='-fno-stack-protector' --with-scalar-type=real --with-mpi-lib='/c/Windows/System32/msmpi.dll' --with-mpi-include='/home/paolo/FreeFem-sources/3rdparty/include/msmpi' --with-mpiexec='/C/Program\ Files/Microsoft\ MPI/Bin/mpiexec' --with-blaslapack-include='' --with-blaslapack-lib='/mingw64/bin/libopenblas.dll' --download-scalapack --download-metis --download-ptscotch --download-mumps --download-hypre --download-parmetis --download-superlu --download-suitesparse --download-tetgen --download-slepc '--download-metis-cmake-arguments=-G "MSYS Makefiles"' '--download-parmetis-cmake-arguments=-G "MSYS Makefiles"' '--download-superlu-cmake-arguments=-G "MSYS Makefiles"' '--download-hypre-configure-arguments=--build=x86_64-linux-gnu --host=x86_64-linux-gnu' PETSC_ARCH=fr > > =============================================================================== > > Configuring PETSc to compile on your system > > =============================================================================== > > TESTING: FortranMPICheck from config.packages.MPI(config/BuildSystem/config/pack******************************************************************************* > > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): > > ------------------------------------------------------------------------------- > > Fortran error! mpi_init() could not be located! > > ******************************************************************************* > > > > make: *** [Makefile:210: petsc-3.13.0/tag-conf-real] Errore 1 > > > > Note that I didn?t add anything to any PATH variable, because this is not mentioned in your documentation. > > > > On a side note, this is the same error I got when trying to build PETSc in Cygwin with the default OpenMPI available in Cygwin. > > > > I am attaching the configure.log? it seems to me that the error comes from the configure trying to include the mpif.h in your folder and not using the -fallow-invalid-boz flag that I had to use, for example, to compile mpi.f90 into mpi.mod > > > > But I?m not sure why this is happening > > > Pierre, > > Could this be due to gcc 10? Sorry, I?m slow. You are right. Our workers use gcc 9, everything is fine, but I see on my VM which I updated that I use gcc 10 and had to disable Fortran, I guess the MUMPS run I showcased was with a prior PETSc build. I?ll try to resolve this and will keep you posted. They really caught a lot of people off guard with gfortran 10? Thanks, Pierre > Executing: gfortran -c -o /tmp/petsc-ur0cff6a/config.libraries/conftest.o -I/tmp/petsc-ur0cff6a/config.compilers -I/tmp/petsc-ur0cff6a/config.setCompilers -I/tmp/petsc-ur0cff6a/config.compilersFortran -I/tmp/petsc-ur0cff6a/config.libraries -Wall -ffree-line-length-0 -Wno-unused-dummy-argument -O3 -mtune=generic -I/home/paolo/FreeFem-sources/3rdparty/include/msmpi /tmp/petsc-ur0cff6a/config.libraries/conftest.F90 > Possible ERROR while running compiler: exit code 1 > stderr: > C:/msys64/home/paolo/FreeFem-sources/3rdparty/include/msmpi/mpif.h:227:36: > > 227 | PARAMETER (MPI_DATATYPE_NULL=z'0c000000') > | 1 > Error: BOZ literal constant at (1) is neither a data-stmt-constant nor an actual argument to INT, REAL, DBLE, or CMPLX intrinsic function [see '-fno-allow-invalid-boz'] > C:/msys64/home/paolo/FreeFem-sources/3rdparty/include/msmpi/mpif.h:303:27: > > 303 | PARAMETER (MPI_CHAR=z'4c000101') > | 1 > Error: BOZ literal constant at (1) is neither a data-stmt-constant nor an actual argument to INT, REAL, DBLE, or CMPLX intrinsic function [see '-fno-allow-invalid-boz'] > C:/msys64/home/paolo/FreeFem-sources/3rdparty/include/msmpi/mpif.h:305:36: > > 305 | PARAMETER (MPI_UNSIGNED_CHAR=z'4c000102') > | 1 > > Thanks, > > Matt > > Thanks > > > > Paolo > > > > Inviato da Posta per Windows 10 > > > > Da: Pierre Jolivet > Inviato: luned? 29 giugno 2020 18:34 > A: Paolo Lampitella > Cc: Satish Balay ; petsc-users > Oggetto: Re: [petsc-users] PETSc and Windows 10 > > > > > > > > > On 29 Jun 2020, at 6:27 PM, Paolo Lampitella > wrote: > > > > I think I made the first step of having mingw64 from msys2 working with ms-mpi. > > > > I found that the issue I was having was related to: > > > > https://gcc.gnu.org/bugzilla/show_bug.cgi?id=91556 > > > and, probably (but impossible to check now), I was using an msys2 and/or mingw mpi package before this fix: > > > > https://github.com/msys2/MINGW-packages/commit/11b4cff3d2ec7411037b692b0ad5a9f3e9b9978d#diff-eac59989e3096be97d940c8f47b50fba > > > Admittedly, I never used gcc 10 before on any machine. Still, I feel that reporting that sort of error in that way is, > > at least, misleading (I would have preferred the initial implementation as mentioned in the gcc bug track). > > > > A second thing that I was not used to, and made me more uncertain of the procedure I was following, is having to compile myself the mpi module. There are several version of this out there, but I decided to stick with this one: > > > > https://www.scivision.dev/windows-mpi-msys2/ > > > even if there seems to be no need to include -fno-range-check and the current mpi.f90 version is different from the mpif.h as reported here: > > > > https://github.com/microsoft/Microsoft-MPI/issues/33 > > > which, to me, are both signs of lack of attention on the fortran side by those that maintain this thing. > > > > In summary, this is the procedure I followed so far (on a 64 bit machine with Windows 10): > > > > Install MSYS2 from https://www.msys2.org/ and just follow the install wizard > Open the MSYS2 terminal and execute: pacman -Syuu > Close the terminal when asked and reopen it > Keep executing ?pacman -Syuu? until nothing else needs to be updated > Close the MSYS2 terminal and reopen it (I guess because was in paranoid mode), then install packages with: > > > pacman -S base-devel git gcc gcc-fortran bsdcpio lndir pax-git unzip > > pacman -S mingw-w64-x86_64-toolchain > > pacman -S mingw-w64-x86_64-msmpi > > pacman -S mingw-w64-x86_64-cmake > > pacman -S mingw-w64-x86_64-freeglut > > pacman -S mingw-w64-x86_64-gsl > > pacman -S mingw-w64-x86_64-libmicroutils > > pacman -S mingw-w64-x86_64-hdf5 > > pacman -S mingw-w64-x86_64-openblas > > pacman -S mingw-w64-x86_64-arpack > > pacman -S mingw-w64-x86_64-jq > > > > This set should include all the libraries mentioned by Pierre and/or used by his Jenkins, as the final scope here is to have PETSc and dependencies working. But I think that for pure MPI one could stop to msmpi (even, maybe, just install msmpi and have the dependencies figured out by pacman). Honestly, I don?t remember the exact order I used to install the packages, but this should not affect things. Also, as I was still in paranoid mode, I kept executing ?pacman -Syuu? after each package was installed. After this, close the MSYS2 terminal. > > > > Open the MINGW64 terminal and create the .mod file out of the mpi.f90 file, as mentioned here https://www.scivision.dev/windows-mpi-msys2/ , with: > > > cd /mingw64/include > > gfortran mpif90 -c -fno-range-check -fallow-invalid-boz > > > > Ah, yes, that?s new to gfortran 10 (we use gfortran 9 on our workers), which is now what?s ship with MSYS2 (we haven?t updated yet). Sorry that I forgot about that. > > > > > This is needed to ?USE mpi? (as opposed to INCLUDE ?mpif.h?) > > > > Install the latest MS-MPI (both sdk and setup) from https://www.microsoft.com/en-us/download/details.aspx?id=100593 > > > At this point I?ve been able to compile (using the MINGW64 terminal) different mpi test programs and they run as expected in the classical Windows prompt. I added this function to my .bashrc in MSYS2 in order to easily copy the required dependencies out of MSYS: > > > > function copydep() { ldd $1 | grep "=> /$2" | awk '{print $3}' | xargs -I '{}' cp -v '{}' .; } > > > > which can be used, with the MINGW64 terminal, by navigating to the folder where the final executable, say, my.exe, resides (even if under a Windows path) and executing: > > > > copydep my.exe mingw64 > > > > This, of course, must be done before actually trying to execute the .exe in the windows cmd prompt. > > > > Hopefully, I should now be able to follow Pierre?s instructions for PETSc (but first I wanna give a try to the system python before removing it) > > > > Looks like the hard part is over. It?s usually easier to deal with ./configure issues. > > If you have weird errors like ?incomplete Cygwin install? or whatever, this is the kind of issues I was referring to earlier. > > In that case, what I?d suggest is just, as before: > > pacman -R mingw-w64-x86_64-python mingw-w64-x86_64-gdb > > pacman -S python > > > > Thanks, > > Pierre > > > > > Thanks > > > > Paolo > > > > > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From sam.guo at cd-adapco.com Mon Jun 29 13:00:39 2020 From: sam.guo at cd-adapco.com (Sam Guo) Date: Mon, 29 Jun 2020 11:00:39 -0700 Subject: [petsc-users] PETSc initialization error In-Reply-To: References: <004EA4EB-815F-4EEE-8988-CA2179304157@petsc.dev> <9C60917A-5771-434D-BB65-D46750C49966@petsc.dev> Message-ID: Hi Junchao, I'll test the ex53. At the meantime, I use the following work around: my program call MPI initialize once for entire program PetscInitialize once for entire program SlecpInitialize once for entire program (I think I can skip PetscInitialize above) calling slepc multiple times my program call MPI finalize before ending program You can see I stkip PetscFinalize/SlepcFinalize. I am uneasy for skipping them since I am not sure what is the consequence. Can you comment on it? Thanks, Sam On Fri, Jun 26, 2020 at 6:58 PM Junchao Zhang wrote: > Did the test included in that commit fail in your environment? You can > also change the test by adding calls to SlepcInitialize/SlepcFinalize > between PetscInitializeNoPointers/PetscFinalize as in my previous email. > > --Junchao Zhang > > > On Fri, Jun 26, 2020 at 5:54 PM Sam Guo wrote: > >> Hi Junchao, >> If you are talking about this commit of yours >> https://gitlab.com/petsc/petsc/-/commit/f0463fa09df52ce43e7c5bf47a1c87df0c9e5cbb >> >> Recycle keyvals and fix bugs in MPI_Comm creation >> I think I got it. It fixes the serial one but parallel one is still >> crashing. >> >> Thanks, >> Sam >> >> On Fri, Jun 26, 2020 at 3:43 PM Sam Guo wrote: >> >>> Hi Junchao, >>> I am not ready to upgrade petsc yet(due to the lengthy technical and >>> legal approval process of our internal policy). Can you send me the diff >>> file so I can apply it to petsc 3.11.3)? >>> >>> Thanks, >>> Sam >>> >>> On Fri, Jun 26, 2020 at 3:33 PM Junchao Zhang >>> wrote: >>> >>>> Sam, >>>> Please discard the origin patch I sent you. A better fix is already >>>> in maint/master. An test is at src/sys/tests/ex53.c >>>> I modified that test at the end with >>>> >>>> for (i=0; i<500; i++) { >>>> ierr = PetscInitializeNoPointers(argc,argv,NULL,help);if (ierr) >>>> return ierr; >>>> ierr = SlepcInitialize(&argc,&argv,NULL,help);if (ierr) return ierr; >>>> ierr = SlepcFinalize();if (ierr) return ierr; >>>> ierr = PetscFinalize();if (ierr) return ierr; >>>> } >>>> >>>> >>>> then I ran it with multiple mpi ranks and it ran correctly. So try >>>> your program with petsc master first. If not work, see if you can come up >>>> with a test example for us. >>>> >>>> Thanks. >>>> --Junchao Zhang >>>> >>>> >>>> On Fri, Jun 26, 2020 at 3:37 PM Sam Guo wrote: >>>> >>>>> One work around for me is to call PetscInitialize once for my entire >>>>> program and skip PetscFinalize (since I don't have a good place to call >>>>> PetscFinalize before ending the program). >>>>> >>>>> On Fri, Jun 26, 2020 at 1:33 PM Sam Guo wrote: >>>>> >>>>>> I get the crash after calling Initialize/Finalize multiple times. >>>>>> Junchao fixed the bug for serial but parallel still crashes. >>>>>> >>>>>> On Fri, Jun 26, 2020 at 1:28 PM Barry Smith wrote: >>>>>> >>>>>>> >>>>>>> Ah, so you get the crash the second time you call >>>>>>> PetscInitialize()? That is a problem because we do intend to support that >>>>>>> capability (but you much call PetscFinalize() each time also). >>>>>>> >>>>>>> Barry >>>>>>> >>>>>>> >>>>>>> On Jun 26, 2020, at 3:25 PM, Sam Guo wrote: >>>>>>> >>>>>>> Hi Barry, >>>>>>> Thanks for the quick response. >>>>>>> I will call PetscInitialize once and skip the PetscFinalize for >>>>>>> now to avoid the crash. The crash is actually in PetscInitialize, not >>>>>>> PetscFinalize. >>>>>>> >>>>>>> Thanks, >>>>>>> Sam >>>>>>> >>>>>>> On Fri, Jun 26, 2020 at 1:21 PM Barry Smith >>>>>>> wrote: >>>>>>> >>>>>>>> >>>>>>>> Sam, >>>>>>>> >>>>>>>> You can skip PetscFinalize() so long as you only call >>>>>>>> PetscInitialize() once. It is not desirable in general to skip the finalize >>>>>>>> because PETSc can't free all its data structures and you cannot see the >>>>>>>> PETSc logging information with -log_view but in terms of the code running >>>>>>>> correctly you do not need to call PetscFinalize. >>>>>>>> >>>>>>>> If your code crashes in PetscFinalize() please send the full >>>>>>>> error output and we can try to help you debug it. >>>>>>>> >>>>>>>> >>>>>>>> Barry >>>>>>>> >>>>>>>> On Jun 26, 2020, at 3:14 PM, Sam Guo wrote: >>>>>>>> >>>>>>>> To clarify, we have a mpi wrapper (so we can switch to different >>>>>>>> mpi at runtime). I compile petsc using our mpi wrapper. >>>>>>>> If I just call PETSc initialize once without calling finallize, it >>>>>>>> is ok. My question to you is that: can I skip finalize? >>>>>>>> Our program calls mpi_finalize at end anyway. >>>>>>>> >>>>>>>> On Fri, Jun 26, 2020 at 1:09 PM Sam Guo >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi Junchao, >>>>>>>>> Attached please find the configure.log. >>>>>>>>> I also attach the pinit.c which contains your patch (I am >>>>>>>>> currently using 3.11.3. I've applied your patch to 3.11.3). Your patch >>>>>>>>> fixes the serial version. The error now is about the parallel. >>>>>>>>> Here is the error log: >>>>>>>>> >>>>>>>>> [1]PETSC ERROR: #1 PetscInitialize() line 969 in >>>>>>>>> ../../../petsc/src/sys/objects/pinit.c >>>>>>>>> [1]PETSC ERROR: #2 checkError() line 56 in >>>>>>>>> ../../../physics/src/eigensolver/SLEPc.cpp >>>>>>>>> [1]PETSC ERROR: #3 PetscInitialize() line 966 in >>>>>>>>> ../../../petsc/src/sys/objects/pinit.c >>>>>>>>> [1]PETSC ERROR: #4 SlepcInitialize() line 262 in >>>>>>>>> ../../../slepc/src/sys/slepcinit.c >>>>>>>>> [0]PETSC ERROR: #1 PetscInitialize() line 969 in >>>>>>>>> ../../../petsc/src/sys/objects/pinit.c >>>>>>>>> [0]PETSC ERROR: #2 checkError() line 56 in >>>>>>>>> ../../../physics/src/eigensolver/SLEPc.cpp >>>>>>>>> [0]PETSC ERROR: #3 PetscInitialize() line 966 in >>>>>>>>> ../../../petsc/src/sys/objects/pinit.c >>>>>>>>> [0]PETSC ERROR: #4 SlepcInitialize() line 262 in >>>>>>>>> ../../../slepc/src/sys/slepcinit.c >>>>>>>>> PETSC ERROR: Logging has not been enabled. >>>>>>>>> You might have forgotten to call PetscInitialize(). >>>>>>>>> PETSC ERROR: Logging has not been enabled. >>>>>>>>> You might have forgotten to call PetscInitialize(). >>>>>>>>> >>>>>>>>> -------------------------------------------------------------------------- >>>>>>>>> MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD >>>>>>>>> with errorcode 56. >>>>>>>>> >>>>>>>>> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. >>>>>>>>> You may or may not see output from other processes, depending on >>>>>>>>> exactly when Open MPI kills them. >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Sam >>>>>>>>> >>>>>>>>> On Thu, Jun 25, 2020 at 7:37 PM Junchao Zhang < >>>>>>>>> junchao.zhang at gmail.com> wrote: >>>>>>>>> >>>>>>>>>> Sam, >>>>>>>>>> The MPI_Comm_create_keyval() error was fixed in maint/master. >>>>>>>>>> From the error message, it seems you need to configure --with-log=1 >>>>>>>>>> Otherwise, please send your full error stack trace and >>>>>>>>>> configure.log. >>>>>>>>>> Thanks. >>>>>>>>>> --Junchao Zhang >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Thu, Jun 25, 2020 at 2:18 PM Sam Guo >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Hi Junchao, >>>>>>>>>>> I now encountered the same error with parallel. I am >>>>>>>>>>> wondering if there is a need for parallel fix as well. >>>>>>>>>>> [1]PETSC ERROR: #1 PetscInitialize() line 969 in >>>>>>>>>>> ../../../petsc/src/sys/objects/pinit.c >>>>>>>>>>> PETSC ERROR: Logging has not been enabled. >>>>>>>>>>> You might have forgotten to call PetscInitialize(). >>>>>>>>>>> PETSC ERROR: Logging has not been enabled. >>>>>>>>>>> You might have forgotten to call PetscInitialize(). >>>>>>>>>>> >>>>>>>>>>> On Sat, Jun 20, 2020 at 7:35 PM Sam Guo >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi Junchao, >>>>>>>>>>>> Your patch works. >>>>>>>>>>>> >>>>>>>>>>>> Thanks, >>>>>>>>>>>> Sam >>>>>>>>>>>> >>>>>>>>>>>> On Sat, Jun 20, 2020 at 4:23 PM Junchao Zhang < >>>>>>>>>>>> junchao.zhang at gmail.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Sat, Jun 20, 2020 at 12:24 PM Barry Smith >>>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Junchao, >>>>>>>>>>>>>> >>>>>>>>>>>>>> This is a good bug fix. It solves the problem when PETSc >>>>>>>>>>>>>> initialize is called many times. >>>>>>>>>>>>>> >>>>>>>>>>>>>> There is another fix you can do to limit PETSc mpiuni >>>>>>>>>>>>>> running out of attributes inside a single PETSc run: >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> int MPI_Comm_create_keyval(MPI_Copy_function >>>>>>>>>>>>>> *copy_fn,MPI_Delete_function *delete_fn,int *keyval,void *extra_state) >>>>>>>>>>>>>> { >>>>>>>>>>>>>> >>>>>>>>>>>>>> if (num_attr >= MAX_ATTR){ >>>>>>>>>>>>>> for (i=0; i>>>>>>>>>>>>> if (!attr_keyval[i].extra_state) { >>>>>>>>>>>>>> >>>>>>>>>>>>> attr_keyval[i].extra_state is provided by user (could be >>>>>>>>>>>>> NULL). We can not rely on it. >>>>>>>>>>>>> >>>>>>>>>>>>>> /* reuse this slot */ >>>>>>>>>>>>>> attr_keyval[i].extra_state = extra_state; >>>>>>>>>>>>>> attr_keyval[i.]del = delete_fn; >>>>>>>>>>>>>> *keyval = i; >>>>>>>>>>>>>> return MPI_SUCCESS; >>>>>>>>>>>>>> } >>>>>>>>>>>>>> } >>>>>>>>>>>>>> return MPIUni_Abort(MPI_COMM_WORLD,1); >>>>>>>>>>>>>> } >>>>>>>>>>>>>> return MPIUni_Abort(MPI_COMM_WORLD,1); >>>>>>>>>>>>>> attr_keyval[num_attr].extra_state = extra_state; >>>>>>>>>>>>>> attr_keyval[num_attr].del = delete_fn; >>>>>>>>>>>>>> *keyval = num_attr++; >>>>>>>>>>>>>> return MPI_SUCCESS; >>>>>>>>>>>>>> } >>>>>>>>>>>>>> >>>>>>>>>>>>>> This will work if the user creates tons of attributes but >>>>>>>>>>>>>> is constantly deleting some as they new ones. So long as the number >>>>>>>>>>>>>> outstanding at one time is < MAX_ATTR) >>>>>>>>>>>>>> >>>>>>>>>>>>>> Barry >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Jun 20, 2020, at 10:54 AM, Junchao Zhang < >>>>>>>>>>>>>> junchao.zhang at gmail.com> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> I don't understand what your session means. Let's try this >>>>>>>>>>>>>> patch >>>>>>>>>>>>>> >>>>>>>>>>>>>> diff --git a/src/sys/mpiuni/mpi.c b/src/sys/mpiuni/mpi.c >>>>>>>>>>>>>> index d559a513..c058265d 100644 >>>>>>>>>>>>>> --- a/src/sys/mpiuni/mpi.c >>>>>>>>>>>>>> +++ b/src/sys/mpiuni/mpi.c >>>>>>>>>>>>>> @@ -283,6 +283,7 @@ int MPI_Finalize(void) >>>>>>>>>>>>>> MPI_Comm_free(&comm); >>>>>>>>>>>>>> comm = MPI_COMM_SELF; >>>>>>>>>>>>>> MPI_Comm_free(&comm); >>>>>>>>>>>>>> + num_attr = 1; /* reset the counter */ >>>>>>>>>>>>>> MPI_was_finalized = 1; >>>>>>>>>>>>>> return MPI_SUCCESS; >>>>>>>>>>>>>> } >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> --Junchao Zhang >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Sat, Jun 20, 2020 at 10:48 AM Sam Guo < >>>>>>>>>>>>>> sam.guo at cd-adapco.com> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Typo: I mean ?Assuming initializer is only needed once for >>>>>>>>>>>>>>> entire session? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Saturday, June 20, 2020, Sam Guo >>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Assuming finalizer is only needed once for entire >>>>>>>>>>>>>>>> session(?), I can put initializer into the static block to call it once but >>>>>>>>>>>>>>>> where do I call finalizer? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Saturday, June 20, 2020, Junchao Zhang < >>>>>>>>>>>>>>>> junchao.zhang at gmail.com> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> The counter num_attr should be recycled. But first try to >>>>>>>>>>>>>>>>> call PETSc initialize/Finalize only once to see it fixes the error. >>>>>>>>>>>>>>>>> --Junchao Zhang >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Sat, Jun 20, 2020 at 12:48 AM Sam Guo < >>>>>>>>>>>>>>>>> sam.guo at cd-adapco.com> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> To clarify, I call PETSc initialize and PETSc finalize >>>>>>>>>>>>>>>>>> everytime I call SLEPc: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> PetscInitializeNoPointers(argc,args,nullptr,nullptr); >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> SlepcInitialize(&argc,&args,static_cast(nullptr),help); >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> //calling slepc >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> SlepcFinalize(); >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> PetscFinalize(); >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On Fri, Jun 19, 2020 at 10:32 PM Sam Guo < >>>>>>>>>>>>>>>>>> sam.guo at cd-adapco.com> wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Dear PETSc team, >>>>>>>>>>>>>>>>>>> When I called SLEPc multiple time, I eventually got >>>>>>>>>>>>>>>>>>> following error: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> MPI operation not supported by PETSc's sequential MPI >>>>>>>>>>>>>>>>>>> wrappers >>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: #1 PetscInitialize() line 967 in >>>>>>>>>>>>>>>>>>> ../../../petsc/src/sys/objects/pinit.c >>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: #2 SlepcInitialize() line 262 in >>>>>>>>>>>>>>>>>>> ../../../slepc/src/sys/slepcinit.c >>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: #3 SlepcInitializeNoPointers() line 359 >>>>>>>>>>>>>>>>>>> in ../../../slepc/src/sys/slepcinit.c >>>>>>>>>>>>>>>>>>> PETSC ERROR: Logging has not been enabled. >>>>>>>>>>>>>>>>>>> You might have forgotten to call PetscInitialize(). >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> I debugged: it is because of following in >>>>>>>>>>>>>>>>>>> petsc/src/sys/mpiuni/mpi.c >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> if (num_attr >= MAX_ATTR) >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> in function int MPI_Comm_create_keyval(MPI_Copy_function >>>>>>>>>>>>>>>>>>> *copy_fn,MPI_Delete_function *delete_fn,int *keyval,void *extra_state) >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> num_attr is declared static and keeps increasing every >>>>>>>>>>>>>>>>>>> time MPI_Comm_create_keyval is called. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> I am using petsc 3.11.3 but found 3.13.2 has the >>>>>>>>>>>>>>>>>>> same logic. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Is this a bug or I didn't use it correctly? >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>>>> Sam >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>> >>>>>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From paololampitella at hotmail.com Mon Jun 29 13:09:41 2020 From: paololampitella at hotmail.com (Paolo Lampitella) Date: Mon, 29 Jun 2020 18:09:41 +0000 Subject: [petsc-users] R: PETSc and Windows 10 In-Reply-To: <94A8E366-2361-409E-84CE-B90EA54D9AF9@enseeiht.fr> References: <32B37EFD-DDFA-41AE-ACC7-CB5E6AE57996@enseeiht.fr> <0558CCF7-8203-4520-8673-3B0945C43233@enseeiht.fr> <20AB9F7A-4471-498B-9A78-AA48163900CB@enseeiht.fr> , <94A8E366-2361-409E-84CE-B90EA54D9AF9@enseeiht.fr> Message-ID: Dear Pierre, thanks again for your time I guess there is no way for me to use the toolchain you are using (I don?t remember having any choice on which version of MSYS or GCC I could install) Paolo Inviato da Posta per Windows 10 Da: Pierre Jolivet Inviato: luned? 29 giugno 2020 20:01 A: Matthew Knepley Cc: Paolo Lampitella; petsc-users Oggetto: Re: [petsc-users] PETSc and Windows 10 On 29 Jun 2020, at 7:47 PM, Matthew Knepley > wrote: On Mon, Jun 29, 2020 at 1:35 PM Paolo Lampitella > wrote: Dear Pierre, sorry to bother you, but I already have some issues. What I did: * pacman -R mingw-w64-x86_64-python mingw-w64-x86_64-gdb (is gdb also troublesome?) * Followed points 6 and 7 at https://doc.freefem.org/introduction/installation.html#compilation-on-windows I first got a warning on the configure at point 6, as ?disable-hips is not recognized. Then, on make ?petsc-slepc? of point 7 (no SUDO=sudo flag was necessary) I got to this point: tar xzf ../pkg/petsc-lite-3.13.0.tar.gz patch -p1 < petsc-suitesparse.patch patching file petsc-3.13.0/config/BuildSystem/config/packages/SuiteSparse.py touch petsc-3.13.0/tag-tar cd petsc-3.13.0 && ./configure MAKEFLAGS='' \ --prefix=/home/paolo/freefem/ff-petsc//r \ --with-debugging=0 COPTFLAGS='-O3 -mtune=generic' CXXOPTFLAGS='-O3 -mtune=generic' FOPTFLAGS='-O3 -mtune=generic' --with-cxx-dialect=C++11 --with-ssl=0 --with-x=0 --with-fortran-bindings=0 --with-shared-libraries=0 --with-cc='gcc' --with-cxx='g++' --with-fc='gfortran' CXXFLAGS='-fno-stack-protector' CFLAGS='-fno-stack-protector' --with-scalar-type=real --with-mpi-lib='/c/Windows/System32/msmpi.dll' --with-mpi-include='/home/paolo/FreeFem-sources/3rdparty/include/msmpi' --with-mpiexec='/C/Program\ Files/Microsoft\ MPI/Bin/mpiexec' --with-blaslapack-include='' --with-blaslapack-lib='/mingw64/bin/libopenblas.dll' --download-scalapack --download-metis --download-ptscotch --download-mumps --download-hypre --download-parmetis --download-superlu --download-suitesparse --download-tetgen --download-slepc '--download-metis-cmake-arguments=-G "MSYS Makefiles"' '--download-parmetis-cmake-arguments=-G "MSYS Makefiles"' '--download-superlu-cmake-arguments=-G "MSYS Makefiles"' '--download-hypre-configure-arguments=--build=x86_64-linux-gnu --host=x86_64-linux-gnu' PETSC_ARCH=fr =============================================================================== Configuring PETSc to compile on your system =============================================================================== TESTING: FortranMPICheck from config.packages.MPI(config/BuildSystem/config/pack******************************************************************************* UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): ------------------------------------------------------------------------------- Fortran error! mpi_init() could not be located! ******************************************************************************* make: *** [Makefile:210: petsc-3.13.0/tag-conf-real] Errore 1 Note that I didn?t add anything to any PATH variable, because this is not mentioned in your documentation. On a side note, this is the same error I got when trying to build PETSc in Cygwin with the default OpenMPI available in Cygwin. I am attaching the configure.log? it seems to me that the error comes from the configure trying to include the mpif.h in your folder and not using the -fallow-invalid-boz flag that I had to use, for example, to compile mpi.f90 into mpi.mod But I?m not sure why this is happening Pierre, Could this be due to gcc 10? Sorry, I?m slow. You are right. Our workers use gcc 9, everything is fine, but I see on my VM which I updated that I use gcc 10 and had to disable Fortran, I guess the MUMPS run I showcased was with a prior PETSc build. I?ll try to resolve this and will keep you posted. They really caught a lot of people off guard with gfortran 10? Thanks, Pierre Executing: gfortran -c -o /tmp/petsc-ur0cff6a/config.libraries/conftest.o -I/tmp/petsc-ur0cff6a/config.compilers -I/tmp/petsc-ur0cff6a/config.setCompilers -I/tmp/petsc-ur0cff6a/config.compilersFortran -I/tmp/petsc-ur0cff6a/config.libraries -Wall -ffree-line-length-0 -Wno-unused-dummy-argument -O3 -mtune=generic -I/home/paolo/FreeFem-sources/3rdparty/include/msmpi /tmp/petsc-ur0cff6a/config.libraries/conftest.F90 Possible ERROR while running compiler: exit code 1 stderr: C:/msys64/home/paolo/FreeFem-sources/3rdparty/include/msmpi/mpif.h:227:36: 227 | PARAMETER (MPI_DATATYPE_NULL=z'0c000000') | 1 Error: BOZ literal constant at (1) is neither a data-stmt-constant nor an actual argument to INT, REAL, DBLE, or CMPLX intrinsic function [see '-fno-allow-invalid-boz'] C:/msys64/home/paolo/FreeFem-sources/3rdparty/include/msmpi/mpif.h:303:27: 303 | PARAMETER (MPI_CHAR=z'4c000101') | 1 Error: BOZ literal constant at (1) is neither a data-stmt-constant nor an actual argument to INT, REAL, DBLE, or CMPLX intrinsic function [see '-fno-allow-invalid-boz'] C:/msys64/home/paolo/FreeFem-sources/3rdparty/include/msmpi/mpif.h:305:36: 305 | PARAMETER (MPI_UNSIGNED_CHAR=z'4c000102') | 1 Thanks, Matt Thanks Paolo Inviato da Posta per Windows 10 Da: Pierre Jolivet Inviato: luned? 29 giugno 2020 18:34 A: Paolo Lampitella Cc: Satish Balay; petsc-users Oggetto: Re: [petsc-users] PETSc and Windows 10 On 29 Jun 2020, at 6:27 PM, Paolo Lampitella > wrote: I think I made the first step of having mingw64 from msys2 working with ms-mpi. I found that the issue I was having was related to: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=91556 and, probably (but impossible to check now), I was using an msys2 and/or mingw mpi package before this fix: https://github.com/msys2/MINGW-packages/commit/11b4cff3d2ec7411037b692b0ad5a9f3e9b9978d#diff-eac59989e3096be97d940c8f47b50fba Admittedly, I never used gcc 10 before on any machine. Still, I feel that reporting that sort of error in that way is, at least, misleading (I would have preferred the initial implementation as mentioned in the gcc bug track). A second thing that I was not used to, and made me more uncertain of the procedure I was following, is having to compile myself the mpi module. There are several version of this out there, but I decided to stick with this one: https://www.scivision.dev/windows-mpi-msys2/ even if there seems to be no need to include -fno-range-check and the current mpi.f90 version is different from the mpif.h as reported here: https://github.com/microsoft/Microsoft-MPI/issues/33 which, to me, are both signs of lack of attention on the fortran side by those that maintain this thing. In summary, this is the procedure I followed so far (on a 64 bit machine with Windows 10): * Install MSYS2 from https://www.msys2.org/ and just follow the install wizard * Open the MSYS2 terminal and execute: pacman -Syuu * Close the terminal when asked and reopen it * Keep executing ?pacman -Syuu? until nothing else needs to be updated * Close the MSYS2 terminal and reopen it (I guess because was in paranoid mode), then install packages with: pacman -S base-devel git gcc gcc-fortran bsdcpio lndir pax-git unzip pacman -S mingw-w64-x86_64-toolchain pacman -S mingw-w64-x86_64-msmpi pacman -S mingw-w64-x86_64-cmake pacman -S mingw-w64-x86_64-freeglut pacman -S mingw-w64-x86_64-gsl pacman -S mingw-w64-x86_64-libmicroutils pacman -S mingw-w64-x86_64-hdf5 pacman -S mingw-w64-x86_64-openblas pacman -S mingw-w64-x86_64-arpack pacman -S mingw-w64-x86_64-jq This set should include all the libraries mentioned by Pierre and/or used by his Jenkins, as the final scope here is to have PETSc and dependencies working. But I think that for pure MPI one could stop to msmpi (even, maybe, just install msmpi and have the dependencies figured out by pacman). Honestly, I don?t remember the exact order I used to install the packages, but this should not affect things. Also, as I was still in paranoid mode, I kept executing ?pacman -Syuu? after each package was installed. After this, close the MSYS2 terminal. * Open the MINGW64 terminal and create the .mod file out of the mpi.f90 file, as mentioned here https://www.scivision.dev/windows-mpi-msys2/, with: cd /mingw64/include gfortran mpif90 -c -fno-range-check -fallow-invalid-boz Ah, yes, that?s new to gfortran 10 (we use gfortran 9 on our workers), which is now what?s ship with MSYS2 (we haven?t updated yet). Sorry that I forgot about that. This is needed to ?USE mpi? (as opposed to INCLUDE ?mpif.h?) * Install the latest MS-MPI (both sdk and setup) from https://www.microsoft.com/en-us/download/details.aspx?id=100593 At this point I?ve been able to compile (using the MINGW64 terminal) different mpi test programs and they run as expected in the classical Windows prompt. I added this function to my .bashrc in MSYS2 in order to easily copy the required dependencies out of MSYS: function copydep() { ldd $1 | grep "=> /$2" | awk '{print $3}' | xargs -I '{}' cp -v '{}' .; } which can be used, with the MINGW64 terminal, by navigating to the folder where the final executable, say, my.exe, resides (even if under a Windows path) and executing: copydep my.exe mingw64 This, of course, must be done before actually trying to execute the .exe in the windows cmd prompt. Hopefully, I should now be able to follow Pierre?s instructions for PETSc (but first I wanna give a try to the system python before removing it) Looks like the hard part is over. It?s usually easier to deal with ./configure issues. If you have weird errors like ?incomplete Cygwin install? or whatever, this is the kind of issues I was referring to earlier. In that case, what I?d suggest is just, as before: pacman -R mingw-w64-x86_64-python mingw-w64-x86_64-gdb pacman -S python Thanks, Pierre Thanks Paolo -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From pierre.jolivet at enseeiht.fr Mon Jun 29 14:37:13 2020 From: pierre.jolivet at enseeiht.fr (Pierre Jolivet) Date: Mon, 29 Jun 2020 21:37:13 +0200 Subject: [petsc-users] PETSc and Windows 10 In-Reply-To: References: <32B37EFD-DDFA-41AE-ACC7-CB5E6AE57996@enseeiht.fr> <0558CCF7-8203-4520-8673-3B0945C43233@enseeiht.fr> <20AB9F7A-4471-498B-9A78-AA48163900CB@enseeiht.fr> <94A8E366-2361-409E-84CE-B90EA54D9AF9@enseeiht.fr> Message-ID: <849E7E3B-8B54-4B40-A2B7-8DD5B55428B3@enseeiht.fr> I do not give up easily on Windows problems: 1) that?s around 50% of our (FreeFEM) user-base (and I want them to use PETSc and SLEPc, ofc?) 2) most people I work with from corporations just have Windows laptops/desktops and I always recommend MSYS because it?s very lightweight and you can pass .exe around 3) I?ve bothered enough Satish, Jed, and Matt on GitLab to take (at least partially) the blame now when it doesn?t work on MSYS That being said, the magic keyword is the added flag FFLAGS="-fallow-invalid-boz" (see, I told you ./configure issues were easier to deal with than the others). Here you?ll see that everything goes through just fine (sorry, it took me a long time to post this because everything is slow on my VM): 1) http://jolivet.perso.enseeiht.fr/win10/configure.log 2) http://jolivet.perso.enseeiht.fr/win10/make.log (both steps #1 and #2 in MSYS terminal, gcc/gfortran 10, MS-MPI see screenshot) 3) http://jolivet.perso.enseeiht.fr/win10/ex2.txt (Command Prompt, 4 processes + MUMPS, I can send you the .exe if you want to try on your machine) I just realize that I didn?t generate the Fortran bindings, but you can see I compiled MUMPS and ScaLAPACK, so that shouldn?t be a problem. Or if there is a problem, we will need to fix this in PETSc. I?ll push this added flag to the FreeFEM repo, thanks for reminding me of the brokenness of gcc/gfortran 10 + MS-MPI. Here is to hoping this won?t affect PETSc ./configure with previous gcc/gfortran version (unlikely, this option is apparently 13-year old https://gcc.gnu.org/bugzilla/show_bug.cgi?id=29471 ) Let me know of the next hiccup, if any. Thanks, Pierre > On 29 Jun 2020, at 8:09 PM, Paolo Lampitella wrote: > > Dear Pierre, > > thanks again for your time > > I guess there is no way for me to use the toolchain you are using (I don?t remember having any choice on which version of MSYS or GCC I could install) > > Paolo > > Inviato da Posta per Windows 10 > > Da: Pierre Jolivet > Inviato: luned? 29 giugno 2020 20:01 > A: Matthew Knepley > Cc: Paolo Lampitella ; petsc-users > Oggetto: Re: [petsc-users] PETSc and Windows 10 > > > > > On 29 Jun 2020, at 7:47 PM, Matthew Knepley > wrote: > > On Mon, Jun 29, 2020 at 1:35 PM Paolo Lampitella > wrote: > Dear Pierre, sorry to bother you, but I already have some issues. What I did: > > pacman -R mingw-w64-x86_64-python mingw-w64-x86_64-gdb (is gdb also troublesome?) > Followed points 6 and 7 at https://doc.freefem.org/introduction/installation.html#compilation-on-windows > I first got a warning on the configure at point 6, as ?disable-hips is not recognized. Then, on make ?petsc-slepc? of point 7 (no SUDO=sudo flag was necessary) I got to this point: > > tar xzf ../pkg/petsc-lite-3.13.0.tar.gz > patch -p1 < petsc-suitesparse.patch > patching file petsc-3.13.0/config/BuildSystem/config/packages/SuiteSparse.py > touch petsc-3.13.0/tag-tar > cd petsc-3.13.0 && ./configure MAKEFLAGS='' \ > --prefix=/home/paolo/freefem/ff-petsc//r \ > --with-debugging=0 COPTFLAGS='-O3 -mtune=generic' CXXOPTFLAGS='-O3 -mtune=generic' FOPTFLAGS='-O3 -mtune=generic' --with-cxx-dialect=C++11 --with-ssl=0 --with-x=0 --with-fortran-bindings=0 --with-shared-libraries=0 --with-cc='gcc' --with-cxx='g++' --with-fc='gfortran' CXXFLAGS='-fno-stack-protector' CFLAGS='-fno-stack-protector' --with-scalar-type=real --with-mpi-lib='/c/Windows/System32/msmpi.dll' --with-mpi-include='/home/paolo/FreeFem-sources/3rdparty/include/msmpi' --with-mpiexec='/C/Program\ Files/Microsoft\ MPI/Bin/mpiexec' --with-blaslapack-include='' --with-blaslapack-lib='/mingw64/bin/libopenblas.dll' --download-scalapack --download-metis --download-ptscotch --download-mumps --download-hypre --download-parmetis --download-superlu --download-suitesparse --download-tetgen --download-slepc '--download-metis-cmake-arguments=-G "MSYS Makefiles"' '--download-parmetis-cmake-arguments=-G "MSYS Makefiles"' '--download-superlu-cmake-arguments=-G "MSYS Makefiles"' '--download-hypre-configure-arguments=--build=x86_64-linux-gnu --host=x86_64-linux-gnu' PETSC_ARCH=fr > =============================================================================== > Configuring PETSc to compile on your system > =============================================================================== > TESTING: FortranMPICheck from config.packages.MPI(config/BuildSystem/config/pack******************************************************************************* > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): > ------------------------------------------------------------------------------- > Fortran error! mpi_init() could not be located! > ******************************************************************************* > > make: *** [Makefile:210: petsc-3.13.0/tag-conf-real] Errore 1 > > Note that I didn?t add anything to any PATH variable, because this is not mentioned in your documentation. > > On a side note, this is the same error I got when trying to build PETSc in Cygwin with the default OpenMPI available in Cygwin. > > I am attaching the configure.log? it seems to me that the error comes from the configure trying to include the mpif.h in your folder and not using the -fallow-invalid-boz flag that I had to use, for example, to compile mpi.f90 into mpi.mod > > But I?m not sure why this is happening > > Pierre, > > Could this be due to gcc 10? > > Sorry, I?m slow. You are right. Our workers use gcc 9, everything is fine, but I see on my VM which I updated that I use gcc 10 and had to disable Fortran, I guess the MUMPS run I showcased was with a prior PETSc build. > I?ll try to resolve this and will keep you posted. > They really caught a lot of people off guard with gfortran 10? > > Thanks, > Pierre > > > Executing: gfortran -c -o /tmp/petsc-ur0cff6a/config.libraries/conftest.o -I/tmp/petsc-ur0cff6a/config.compilers -I/tmp/petsc-ur0cff6a/config.setCompilers -I/tmp/petsc-ur0cff6a/config.compilersFortran -I/tmp/petsc-ur0cff6a/config.libraries -Wall -ffree-line-length-0 -Wno-unused-dummy-argument -O3 -mtune=generic -I/home/paolo/FreeFem-sources/3rdparty/include/msmpi /tmp/petsc-ur0cff6a/config.libraries/conftest.F90 > Possible ERROR while running compiler: exit code 1 > stderr: > C:/msys64/home/paolo/FreeFem-sources/3rdparty/include/msmpi/mpif.h:227:36: > > 227 | PARAMETER (MPI_DATATYPE_NULL=z'0c000000') > | 1 > Error: BOZ literal constant at (1) is neither a data-stmt-constant nor an actual argument to INT, REAL, DBLE, or CMPLX intrinsic function [see '-fno-allow-invalid-boz'] > C:/msys64/home/paolo/FreeFem-sources/3rdparty/include/msmpi/mpif.h:303:27: > > 303 | PARAMETER (MPI_CHAR=z'4c000101') > | 1 > Error: BOZ literal constant at (1) is neither a data-stmt-constant nor an actual argument to INT, REAL, DBLE, or CMPLX intrinsic function [see '-fno-allow-invalid-boz'] > C:/msys64/home/paolo/FreeFem-sources/3rdparty/include/msmpi/mpif.h:305:36: > > 305 | PARAMETER (MPI_UNSIGNED_CHAR=z'4c000102') > | 1 > > Thanks, > > Matt > > Thanks > > Paolo > > Inviato da Posta per Windows 10 > > Da: Pierre Jolivet > Inviato: luned? 29 giugno 2020 18:34 > A: Paolo Lampitella > Cc: Satish Balay ; petsc-users > Oggetto: Re: [petsc-users] PETSc and Windows 10 > > > > > On 29 Jun 2020, at 6:27 PM, Paolo Lampitella > wrote: > > I think I made the first step of having mingw64 from msys2 working with ms-mpi. > > I found that the issue I was having was related to: > > https://gcc.gnu.org/bugzilla/show_bug.cgi?id=91556 > > and, probably (but impossible to check now), I was using an msys2 and/or mingw mpi package before this fix: > > https://github.com/msys2/MINGW-packages/commit/11b4cff3d2ec7411037b692b0ad5a9f3e9b9978d#diff-eac59989e3096be97d940c8f47b50fba > > Admittedly, I never used gcc 10 before on any machine. Still, I feel that reporting that sort of error in that way is, > at least, misleading (I would have preferred the initial implementation as mentioned in the gcc bug track). > > A second thing that I was not used to, and made me more uncertain of the procedure I was following, is having to compile myself the mpi module. There are several version of this out there, but I decided to stick with this one: > > https://www.scivision.dev/windows-mpi-msys2/ > > even if there seems to be no need to include -fno-range-check and the current mpi.f90 version is different from the mpif.h as reported here: > > https://github.com/microsoft/Microsoft-MPI/issues/33 > > which, to me, are both signs of lack of attention on the fortran side by those that maintain this thing. > > In summary, this is the procedure I followed so far (on a 64 bit machine with Windows 10): > > Install MSYS2 from https://www.msys2.org/ and just follow the install wizard > Open the MSYS2 terminal and execute: pacman -Syuu > Close the terminal when asked and reopen it > Keep executing ?pacman -Syuu? until nothing else needs to be updated > Close the MSYS2 terminal and reopen it (I guess because was in paranoid mode), then install packages with: > > pacman -S base-devel git gcc gcc-fortran bsdcpio lndir pax-git unzip > pacman -S mingw-w64-x86_64-toolchain > pacman -S mingw-w64-x86_64-msmpi > pacman -S mingw-w64-x86_64-cmake > pacman -S mingw-w64-x86_64-freeglut > pacman -S mingw-w64-x86_64-gsl > pacman -S mingw-w64-x86_64-libmicroutils > pacman -S mingw-w64-x86_64-hdf5 > pacman -S mingw-w64-x86_64-openblas > pacman -S mingw-w64-x86_64-arpack > pacman -S mingw-w64-x86_64-jq > > This set should include all the libraries mentioned by Pierre and/or used by his Jenkins, as the final scope here is to have PETSc and dependencies working. But I think that for pure MPI one could stop to msmpi (even, maybe, just install msmpi and have the dependencies figured out by pacman). Honestly, I don?t remember the exact order I used to install the packages, but this should not affect things. Also, as I was still in paranoid mode, I kept executing ?pacman -Syuu? after each package was installed. After this, close the MSYS2 terminal. > > Open the MINGW64 terminal and create the .mod file out of the mpi.f90 file, as mentioned here https://www.scivision.dev/windows-mpi-msys2/ , with: > > cd /mingw64/include > gfortran mpif90 -c -fno-range-check -fallow-invalid-boz > > Ah, yes, that?s new to gfortran 10 (we use gfortran 9 on our workers), which is now what?s ship with MSYS2 (we haven?t updated yet). Sorry that I forgot about that. > > > This is needed to ?USE mpi? (as opposed to INCLUDE ?mpif.h?) > > Install the latest MS-MPI (both sdk and setup) from https://www.microsoft.com/en-us/download/details.aspx?id=100593 > > At this point I?ve been able to compile (using the MINGW64 terminal) different mpi test programs and they run as expected in the classical Windows prompt. I added this function to my .bashrc in MSYS2 in order to easily copy the required dependencies out of MSYS: > > function copydep() { ldd $1 | grep "=> /$2" | awk '{print $3}' | xargs -I '{}' cp -v '{}' .; } > > which can be used, with the MINGW64 terminal, by navigating to the folder where the final executable, say, my.exe, resides (even if under a Windows path) and executing: > > copydep my.exe mingw64 > > This, of course, must be done before actually trying to execute the .exe in the windows cmd prompt. > > Hopefully, I should now be able to follow Pierre?s instructions for PETSc (but first I wanna give a try to the system python before removing it) > > Looks like the hard part is over. It?s usually easier to deal with ./configure issues. > If you have weird errors like ?incomplete Cygwin install? or whatever, this is the kind of issues I was referring to earlier. > In that case, what I?d suggest is just, as before: > pacman -R mingw-w64-x86_64-python mingw-w64-x86_64-gdb > pacman -S python > > Thanks, > Pierre > > > Thanks > > Paolo > > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From pierre.jolivet at enseeiht.fr Mon Jun 29 15:03:54 2020 From: pierre.jolivet at enseeiht.fr (Pierre Jolivet) Date: Mon, 29 Jun 2020 22:03:54 +0200 Subject: [petsc-users] PETSc and Windows 10 In-Reply-To: <849E7E3B-8B54-4B40-A2B7-8DD5B55428B3@enseeiht.fr> References: <32B37EFD-DDFA-41AE-ACC7-CB5E6AE57996@enseeiht.fr> <0558CCF7-8203-4520-8673-3B0945C43233@enseeiht.fr> <20AB9F7A-4471-498B-9A78-AA48163900CB@enseeiht.fr> <94A8E366-2361-409E-84CE-B90EA54D9AF9@enseeiht.fr> <849E7E3B-8B54-4B40-A2B7-8DD5B55428B3@enseeiht.fr> Message-ID: <7D5BDEB9-CDAF-4591-BABD-5874354BCE5A@enseeiht.fr> > On 29 Jun 2020, at 9:37 PM, Pierre Jolivet wrote: > > I do not give up easily on Windows problems: > 1) that?s around 50% of our (FreeFEM) user-base (and I want them to use PETSc and SLEPc, ofc?) > 2) most people I work with from corporations just have Windows laptops/desktops and I always recommend MSYS because it?s very lightweight and you can pass .exe around > 3) I?ve bothered enough Satish, Jed, and Matt on GitLab to take (at least partially) the blame now when it doesn?t work on MSYS > > That being said, the magic keyword is the added flag FFLAGS="-fallow-invalid-boz" (see, I told you ./configure issues were easier to deal with than the others). > Here you?ll see that everything goes through just fine (sorry, it took me a long time to post this because everything is slow on my VM): > 1) http://jolivet.perso.enseeiht.fr/win10/configure.log > 2) http://jolivet.perso.enseeiht.fr/win10/make.log (both steps #1 and #2 in MSYS terminal, gcc/gfortran 10, MS-MPI see screenshot) > 3) http://jolivet.perso.enseeiht.fr/win10/ex2.txt (Command Prompt, 4 processes + MUMPS, I can send you the .exe if you want to try on your machine) > I just realize that I didn?t generate the Fortran bindings, but you can see I compiled MUMPS and ScaLAPACK, so that shouldn?t be a problem. > Or if there is a problem, we will need to fix this in PETSc. > > I?ll push this added flag to the FreeFEM repo Sorry for the noise, but maybe it?s better to put this in PETSc ./configure, like you did here Satish https://gitlab.com/petsc/petsc/-/commit/2cd8068296b34e127f055bb32f556e3599f17523 ? If Gfortran100 && MS-MPI, then FFLAGS += "-fallow-invalid-boz" WDY(PETSc-)GT? Thanks, Pierre > thanks for reminding me of the brokenness of gcc/gfortran 10 + MS-MPI. > Here is to hoping this won?t affect PETSc ./configure with previous gcc/gfortran version (unlikely, this option is apparently 13-year old https://gcc.gnu.org/bugzilla/show_bug.cgi?id=29471 ) > > Let me know of the next hiccup, if any. > Thanks, > Pierre > >> On 29 Jun 2020, at 8:09 PM, Paolo Lampitella > wrote: >> >> Dear Pierre, >> >> thanks again for your time >> >> I guess there is no way for me to use the toolchain you are using (I don?t remember having any choice on which version of MSYS or GCC I could install) >> >> Paolo >> >> Inviato da Posta per Windows 10 >> >> Da: Pierre Jolivet >> Inviato: luned? 29 giugno 2020 20:01 >> A: Matthew Knepley >> Cc: Paolo Lampitella ; petsc-users >> Oggetto: Re: [petsc-users] PETSc and Windows 10 >> >> >> >> >> On 29 Jun 2020, at 7:47 PM, Matthew Knepley > wrote: >> >> On Mon, Jun 29, 2020 at 1:35 PM Paolo Lampitella > wrote: >> Dear Pierre, sorry to bother you, but I already have some issues. What I did: >> >> pacman -R mingw-w64-x86_64-python mingw-w64-x86_64-gdb (is gdb also troublesome?) >> Followed points 6 and 7 at https://doc.freefem.org/introduction/installation.html#compilation-on-windows >> I first got a warning on the configure at point 6, as ?disable-hips is not recognized. Then, on make ?petsc-slepc? of point 7 (no SUDO=sudo flag was necessary) I got to this point: >> >> tar xzf ../pkg/petsc-lite-3.13.0.tar.gz >> patch -p1 < petsc-suitesparse.patch >> patching file petsc-3.13.0/config/BuildSystem/config/packages/SuiteSparse.py >> touch petsc-3.13.0/tag-tar >> cd petsc-3.13.0 && ./configure MAKEFLAGS='' \ >> --prefix=/home/paolo/freefem/ff-petsc//r \ >> --with-debugging=0 COPTFLAGS='-O3 -mtune=generic' CXXOPTFLAGS='-O3 -mtune=generic' FOPTFLAGS='-O3 -mtune=generic' --with-cxx-dialect=C++11 --with-ssl=0 --with-x=0 --with-fortran-bindings=0 --with-shared-libraries=0 --with-cc='gcc' --with-cxx='g++' --with-fc='gfortran' CXXFLAGS='-fno-stack-protector' CFLAGS='-fno-stack-protector' --with-scalar-type=real --with-mpi-lib='/c/Windows/System32/msmpi.dll' --with-mpi-include='/home/paolo/FreeFem-sources/3rdparty/include/msmpi' --with-mpiexec='/C/Program\ Files/Microsoft\ MPI/Bin/mpiexec' --with-blaslapack-include='' --with-blaslapack-lib='/mingw64/bin/libopenblas.dll' --download-scalapack --download-metis --download-ptscotch --download-mumps --download-hypre --download-parmetis --download-superlu --download-suitesparse --download-tetgen --download-slepc '--download-metis-cmake-arguments=-G "MSYS Makefiles"' '--download-parmetis-cmake-arguments=-G "MSYS Makefiles"' '--download-superlu-cmake-arguments=-G "MSYS Makefiles"' '--download-hypre-configure-arguments=--build=x86_64-linux-gnu --host=x86_64-linux-gnu' PETSC_ARCH=fr >> =============================================================================== >> Configuring PETSc to compile on your system >> =============================================================================== >> TESTING: FortranMPICheck from config.packages.MPI(config/BuildSystem/config/pack******************************************************************************* >> UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): >> ------------------------------------------------------------------------------- >> Fortran error! mpi_init() could not be located! >> ******************************************************************************* >> >> make: *** [Makefile:210: petsc-3.13.0/tag-conf-real] Errore 1 >> >> Note that I didn?t add anything to any PATH variable, because this is not mentioned in your documentation. >> >> On a side note, this is the same error I got when trying to build PETSc in Cygwin with the default OpenMPI available in Cygwin. >> >> I am attaching the configure.log? it seems to me that the error comes from the configure trying to include the mpif.h in your folder and not using the -fallow-invalid-boz flag that I had to use, for example, to compile mpi.f90 into mpi.mod >> >> But I?m not sure why this is happening >> >> Pierre, >> >> Could this be due to gcc 10? >> >> Sorry, I?m slow. You are right. Our workers use gcc 9, everything is fine, but I see on my VM which I updated that I use gcc 10 and had to disable Fortran, I guess the MUMPS run I showcased was with a prior PETSc build. >> I?ll try to resolve this and will keep you posted. >> They really caught a lot of people off guard with gfortran 10? >> >> Thanks, >> Pierre >> >> >> Executing: gfortran -c -o /tmp/petsc-ur0cff6a/config.libraries/conftest.o -I/tmp/petsc-ur0cff6a/config.compilers -I/tmp/petsc-ur0cff6a/config.setCompilers -I/tmp/petsc-ur0cff6a/config.compilersFortran -I/tmp/petsc-ur0cff6a/config.libraries -Wall -ffree-line-length-0 -Wno-unused-dummy-argument -O3 -mtune=generic -I/home/paolo/FreeFem-sources/3rdparty/include/msmpi /tmp/petsc-ur0cff6a/config.libraries/conftest.F90 >> Possible ERROR while running compiler: exit code 1 >> stderr: >> C:/msys64/home/paolo/FreeFem-sources/3rdparty/include/msmpi/mpif.h:227:36: >> >> 227 | PARAMETER (MPI_DATATYPE_NULL=z'0c000000') >> | 1 >> Error: BOZ literal constant at (1) is neither a data-stmt-constant nor an actual argument to INT, REAL, DBLE, or CMPLX intrinsic function [see '-fno-allow-invalid-boz'] >> C:/msys64/home/paolo/FreeFem-sources/3rdparty/include/msmpi/mpif.h:303:27: >> >> 303 | PARAMETER (MPI_CHAR=z'4c000101') >> | 1 >> Error: BOZ literal constant at (1) is neither a data-stmt-constant nor an actual argument to INT, REAL, DBLE, or CMPLX intrinsic function [see '-fno-allow-invalid-boz'] >> C:/msys64/home/paolo/FreeFem-sources/3rdparty/include/msmpi/mpif.h:305:36: >> >> 305 | PARAMETER (MPI_UNSIGNED_CHAR=z'4c000102') >> | 1 >> >> Thanks, >> >> Matt >> >> Thanks >> >> Paolo >> >> Inviato da Posta per Windows 10 >> >> Da: Pierre Jolivet >> Inviato: luned? 29 giugno 2020 18:34 >> A: Paolo Lampitella >> Cc: Satish Balay ; petsc-users >> Oggetto: Re: [petsc-users] PETSc and Windows 10 >> >> >> >> >> On 29 Jun 2020, at 6:27 PM, Paolo Lampitella > wrote: >> >> I think I made the first step of having mingw64 from msys2 working with ms-mpi. >> >> I found that the issue I was having was related to: >> >> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=91556 >> >> and, probably (but impossible to check now), I was using an msys2 and/or mingw mpi package before this fix: >> >> https://github.com/msys2/MINGW-packages/commit/11b4cff3d2ec7411037b692b0ad5a9f3e9b9978d#diff-eac59989e3096be97d940c8f47b50fba >> >> Admittedly, I never used gcc 10 before on any machine. Still, I feel that reporting that sort of error in that way is, >> at least, misleading (I would have preferred the initial implementation as mentioned in the gcc bug track). >> >> A second thing that I was not used to, and made me more uncertain of the procedure I was following, is having to compile myself the mpi module. There are several version of this out there, but I decided to stick with this one: >> >> https://www.scivision.dev/windows-mpi-msys2/ >> >> even if there seems to be no need to include -fno-range-check and the current mpi.f90 version is different from the mpif.h as reported here: >> >> https://github.com/microsoft/Microsoft-MPI/issues/33 >> >> which, to me, are both signs of lack of attention on the fortran side by those that maintain this thing. >> >> In summary, this is the procedure I followed so far (on a 64 bit machine with Windows 10): >> >> Install MSYS2 from https://www.msys2.org/ and just follow the install wizard >> Open the MSYS2 terminal and execute: pacman -Syuu >> Close the terminal when asked and reopen it >> Keep executing ?pacman -Syuu? until nothing else needs to be updated >> Close the MSYS2 terminal and reopen it (I guess because was in paranoid mode), then install packages with: >> >> pacman -S base-devel git gcc gcc-fortran bsdcpio lndir pax-git unzip >> pacman -S mingw-w64-x86_64-toolchain >> pacman -S mingw-w64-x86_64-msmpi >> pacman -S mingw-w64-x86_64-cmake >> pacman -S mingw-w64-x86_64-freeglut >> pacman -S mingw-w64-x86_64-gsl >> pacman -S mingw-w64-x86_64-libmicroutils >> pacman -S mingw-w64-x86_64-hdf5 >> pacman -S mingw-w64-x86_64-openblas >> pacman -S mingw-w64-x86_64-arpack >> pacman -S mingw-w64-x86_64-jq >> >> This set should include all the libraries mentioned by Pierre and/or used by his Jenkins, as the final scope here is to have PETSc and dependencies working. But I think that for pure MPI one could stop to msmpi (even, maybe, just install msmpi and have the dependencies figured out by pacman). Honestly, I don?t remember the exact order I used to install the packages, but this should not affect things. Also, as I was still in paranoid mode, I kept executing ?pacman -Syuu? after each package was installed. After this, close the MSYS2 terminal. >> >> Open the MINGW64 terminal and create the .mod file out of the mpi.f90 file, as mentioned here https://www.scivision.dev/windows-mpi-msys2/ , with: >> >> cd /mingw64/include >> gfortran mpif90 -c -fno-range-check -fallow-invalid-boz >> >> Ah, yes, that?s new to gfortran 10 (we use gfortran 9 on our workers), which is now what?s ship with MSYS2 (we haven?t updated yet). Sorry that I forgot about that. >> >> >> This is needed to ?USE mpi? (as opposed to INCLUDE ?mpif.h?) >> >> Install the latest MS-MPI (both sdk and setup) from https://www.microsoft.com/en-us/download/details.aspx?id=100593 >> >> At this point I?ve been able to compile (using the MINGW64 terminal) different mpi test programs and they run as expected in the classical Windows prompt. I added this function to my .bashrc in MSYS2 in order to easily copy the required dependencies out of MSYS: >> >> function copydep() { ldd $1 | grep "=> /$2" | awk '{print $3}' | xargs -I '{}' cp -v '{}' .; } >> >> which can be used, with the MINGW64 terminal, by navigating to the folder where the final executable, say, my.exe, resides (even if under a Windows path) and executing: >> >> copydep my.exe mingw64 >> >> This, of course, must be done before actually trying to execute the .exe in the windows cmd prompt. >> >> Hopefully, I should now be able to follow Pierre?s instructions for PETSc (but first I wanna give a try to the system python before removing it) >> >> Looks like the hard part is over. It?s usually easier to deal with ./configure issues. >> If you have weird errors like ?incomplete Cygwin install? or whatever, this is the kind of issues I was referring to earlier. >> In that case, what I?d suggest is just, as before: >> pacman -R mingw-w64-x86_64-python mingw-w64-x86_64-gdb >> pacman -S python >> >> Thanks, >> Pierre >> >> >> Thanks >> >> Paolo >> >> >> >> >> -- >> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From junchao.zhang at gmail.com Mon Jun 29 23:10:42 2020 From: junchao.zhang at gmail.com (Junchao Zhang) Date: Mon, 29 Jun 2020 23:10:42 -0500 Subject: [petsc-users] PETSc initialization error In-Reply-To: References: <004EA4EB-815F-4EEE-8988-CA2179304157@petsc.dev> <9C60917A-5771-434D-BB65-D46750C49966@petsc.dev> Message-ID: On Mon, Jun 29, 2020 at 1:00 PM Sam Guo wrote: > Hi Junchao, > I'll test the ex53. At the meantime, I use the following work around: > my program call MPI initialize once for entire program > PetscInitialize once for entire program > SlecpInitialize once for entire program (I think I can skip > PetscInitialize above) > calling slepc multiple times > my program call MPI finalize before ending program > > You can see I stkip PetscFinalize/SlepcFinalize. I am uneasy for > skipping them since I am not sure what is the consequence. Can you comment > on it? > It should be fine. MPI_Finalize does not free objects created by MPI. But since you end your program after MPI_Finalize, there should be no memory leaks. In general, one needs to call PetscFinalize/SlepcFinalize. Try to get a minimal working example and then we can have a look. > > Thanks, > Sam > > > > On Fri, Jun 26, 2020 at 6:58 PM Junchao Zhang > wrote: > >> Did the test included in that commit fail in your environment? You can >> also change the test by adding calls to SlepcInitialize/SlepcFinalize >> between PetscInitializeNoPointers/PetscFinalize as in my previous email. >> >> --Junchao Zhang >> >> >> On Fri, Jun 26, 2020 at 5:54 PM Sam Guo wrote: >> >>> Hi Junchao, >>> If you are talking about this commit of yours >>> https://gitlab.com/petsc/petsc/-/commit/f0463fa09df52ce43e7c5bf47a1c87df0c9e5cbb >>> >>> Recycle keyvals and fix bugs in MPI_Comm creation >>> I think I got it. It fixes the serial one but parallel one is still >>> crashing. >>> >>> Thanks, >>> Sam >>> >>> On Fri, Jun 26, 2020 at 3:43 PM Sam Guo wrote: >>> >>>> Hi Junchao, >>>> I am not ready to upgrade petsc yet(due to the lengthy technical and >>>> legal approval process of our internal policy). Can you send me the diff >>>> file so I can apply it to petsc 3.11.3)? >>>> >>>> Thanks, >>>> Sam >>>> >>>> On Fri, Jun 26, 2020 at 3:33 PM Junchao Zhang >>>> wrote: >>>> >>>>> Sam, >>>>> Please discard the origin patch I sent you. A better fix is already >>>>> in maint/master. An test is at src/sys/tests/ex53.c >>>>> I modified that test at the end with >>>>> >>>>> for (i=0; i<500; i++) { >>>>> ierr = PetscInitializeNoPointers(argc,argv,NULL,help);if (ierr) >>>>> return ierr; >>>>> ierr = SlepcInitialize(&argc,&argv,NULL,help);if (ierr) return >>>>> ierr; >>>>> ierr = SlepcFinalize();if (ierr) return ierr; >>>>> ierr = PetscFinalize();if (ierr) return ierr; >>>>> } >>>>> >>>>> >>>>> then I ran it with multiple mpi ranks and it ran correctly. So try >>>>> your program with petsc master first. If not work, see if you can come up >>>>> with a test example for us. >>>>> >>>>> Thanks. >>>>> --Junchao Zhang >>>>> >>>>> >>>>> On Fri, Jun 26, 2020 at 3:37 PM Sam Guo wrote: >>>>> >>>>>> One work around for me is to call PetscInitialize once for my entire >>>>>> program and skip PetscFinalize (since I don't have a good place to call >>>>>> PetscFinalize before ending the program). >>>>>> >>>>>> On Fri, Jun 26, 2020 at 1:33 PM Sam Guo >>>>>> wrote: >>>>>> >>>>>>> I get the crash after calling Initialize/Finalize multiple times. >>>>>>> Junchao fixed the bug for serial but parallel still crashes. >>>>>>> >>>>>>> On Fri, Jun 26, 2020 at 1:28 PM Barry Smith >>>>>>> wrote: >>>>>>> >>>>>>>> >>>>>>>> Ah, so you get the crash the second time you call >>>>>>>> PetscInitialize()? That is a problem because we do intend to support that >>>>>>>> capability (but you much call PetscFinalize() each time also). >>>>>>>> >>>>>>>> Barry >>>>>>>> >>>>>>>> >>>>>>>> On Jun 26, 2020, at 3:25 PM, Sam Guo wrote: >>>>>>>> >>>>>>>> Hi Barry, >>>>>>>> Thanks for the quick response. >>>>>>>> I will call PetscInitialize once and skip the PetscFinalize for >>>>>>>> now to avoid the crash. The crash is actually in PetscInitialize, not >>>>>>>> PetscFinalize. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Sam >>>>>>>> >>>>>>>> On Fri, Jun 26, 2020 at 1:21 PM Barry Smith >>>>>>>> wrote: >>>>>>>> >>>>>>>>> >>>>>>>>> Sam, >>>>>>>>> >>>>>>>>> You can skip PetscFinalize() so long as you only call >>>>>>>>> PetscInitialize() once. It is not desirable in general to skip the finalize >>>>>>>>> because PETSc can't free all its data structures and you cannot see the >>>>>>>>> PETSc logging information with -log_view but in terms of the code running >>>>>>>>> correctly you do not need to call PetscFinalize. >>>>>>>>> >>>>>>>>> If your code crashes in PetscFinalize() please send the full >>>>>>>>> error output and we can try to help you debug it. >>>>>>>>> >>>>>>>>> >>>>>>>>> Barry >>>>>>>>> >>>>>>>>> On Jun 26, 2020, at 3:14 PM, Sam Guo >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>> To clarify, we have a mpi wrapper (so we can switch to different >>>>>>>>> mpi at runtime). I compile petsc using our mpi wrapper. >>>>>>>>> If I just call PETSc initialize once without calling finallize, it >>>>>>>>> is ok. My question to you is that: can I skip finalize? >>>>>>>>> Our program calls mpi_finalize at end anyway. >>>>>>>>> >>>>>>>>> On Fri, Jun 26, 2020 at 1:09 PM Sam Guo >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hi Junchao, >>>>>>>>>> Attached please find the configure.log. >>>>>>>>>> I also attach the pinit.c which contains your patch (I am >>>>>>>>>> currently using 3.11.3. I've applied your patch to 3.11.3). Your patch >>>>>>>>>> fixes the serial version. The error now is about the parallel. >>>>>>>>>> Here is the error log: >>>>>>>>>> >>>>>>>>>> [1]PETSC ERROR: #1 PetscInitialize() line 969 in >>>>>>>>>> ../../../petsc/src/sys/objects/pinit.c >>>>>>>>>> [1]PETSC ERROR: #2 checkError() line 56 in >>>>>>>>>> ../../../physics/src/eigensolver/SLEPc.cpp >>>>>>>>>> [1]PETSC ERROR: #3 PetscInitialize() line 966 in >>>>>>>>>> ../../../petsc/src/sys/objects/pinit.c >>>>>>>>>> [1]PETSC ERROR: #4 SlepcInitialize() line 262 in >>>>>>>>>> ../../../slepc/src/sys/slepcinit.c >>>>>>>>>> [0]PETSC ERROR: #1 PetscInitialize() line 969 in >>>>>>>>>> ../../../petsc/src/sys/objects/pinit.c >>>>>>>>>> [0]PETSC ERROR: #2 checkError() line 56 in >>>>>>>>>> ../../../physics/src/eigensolver/SLEPc.cpp >>>>>>>>>> [0]PETSC ERROR: #3 PetscInitialize() line 966 in >>>>>>>>>> ../../../petsc/src/sys/objects/pinit.c >>>>>>>>>> [0]PETSC ERROR: #4 SlepcInitialize() line 262 in >>>>>>>>>> ../../../slepc/src/sys/slepcinit.c >>>>>>>>>> PETSC ERROR: Logging has not been enabled. >>>>>>>>>> You might have forgotten to call PetscInitialize(). >>>>>>>>>> PETSC ERROR: Logging has not been enabled. >>>>>>>>>> You might have forgotten to call PetscInitialize(). >>>>>>>>>> >>>>>>>>>> -------------------------------------------------------------------------- >>>>>>>>>> MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD >>>>>>>>>> with errorcode 56. >>>>>>>>>> >>>>>>>>>> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI >>>>>>>>>> processes. >>>>>>>>>> You may or may not see output from other processes, depending on >>>>>>>>>> exactly when Open MPI kills them. >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> Sam >>>>>>>>>> >>>>>>>>>> On Thu, Jun 25, 2020 at 7:37 PM Junchao Zhang < >>>>>>>>>> junchao.zhang at gmail.com> wrote: >>>>>>>>>> >>>>>>>>>>> Sam, >>>>>>>>>>> The MPI_Comm_create_keyval() error was fixed in maint/master. >>>>>>>>>>> From the error message, it seems you need to configure --with-log=1 >>>>>>>>>>> Otherwise, please send your full error stack trace and >>>>>>>>>>> configure.log. >>>>>>>>>>> Thanks. >>>>>>>>>>> --Junchao Zhang >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Thu, Jun 25, 2020 at 2:18 PM Sam Guo >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi Junchao, >>>>>>>>>>>> I now encountered the same error with parallel. I am >>>>>>>>>>>> wondering if there is a need for parallel fix as well. >>>>>>>>>>>> [1]PETSC ERROR: #1 PetscInitialize() line 969 in >>>>>>>>>>>> ../../../petsc/src/sys/objects/pinit.c >>>>>>>>>>>> PETSC ERROR: Logging has not been enabled. >>>>>>>>>>>> You might have forgotten to call PetscInitialize(). >>>>>>>>>>>> PETSC ERROR: Logging has not been enabled. >>>>>>>>>>>> You might have forgotten to call PetscInitialize(). >>>>>>>>>>>> >>>>>>>>>>>> On Sat, Jun 20, 2020 at 7:35 PM Sam Guo >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi Junchao, >>>>>>>>>>>>> Your patch works. >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks, >>>>>>>>>>>>> Sam >>>>>>>>>>>>> >>>>>>>>>>>>> On Sat, Jun 20, 2020 at 4:23 PM Junchao Zhang < >>>>>>>>>>>>> junchao.zhang at gmail.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Sat, Jun 20, 2020 at 12:24 PM Barry Smith < >>>>>>>>>>>>>> bsmith at petsc.dev> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Junchao, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> This is a good bug fix. It solves the problem when >>>>>>>>>>>>>>> PETSc initialize is called many times. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> There is another fix you can do to limit PETSc mpiuni >>>>>>>>>>>>>>> running out of attributes inside a single PETSc run: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> int MPI_Comm_create_keyval(MPI_Copy_function >>>>>>>>>>>>>>> *copy_fn,MPI_Delete_function *delete_fn,int *keyval,void *extra_state) >>>>>>>>>>>>>>> { >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> if (num_attr >= MAX_ATTR){ >>>>>>>>>>>>>>> for (i=0; i>>>>>>>>>>>>>> if (!attr_keyval[i].extra_state) { >>>>>>>>>>>>>>> >>>>>>>>>>>>>> attr_keyval[i].extra_state is provided by user (could be >>>>>>>>>>>>>> NULL). We can not rely on it. >>>>>>>>>>>>>> >>>>>>>>>>>>>>> /* reuse this slot */ >>>>>>>>>>>>>>> attr_keyval[i].extra_state = extra_state; >>>>>>>>>>>>>>> attr_keyval[i.]del = delete_fn; >>>>>>>>>>>>>>> *keyval = i; >>>>>>>>>>>>>>> return MPI_SUCCESS; >>>>>>>>>>>>>>> } >>>>>>>>>>>>>>> } >>>>>>>>>>>>>>> return MPIUni_Abort(MPI_COMM_WORLD,1); >>>>>>>>>>>>>>> } >>>>>>>>>>>>>>> return MPIUni_Abort(MPI_COMM_WORLD,1); >>>>>>>>>>>>>>> attr_keyval[num_attr].extra_state = extra_state; >>>>>>>>>>>>>>> attr_keyval[num_attr].del = delete_fn; >>>>>>>>>>>>>>> *keyval = num_attr++; >>>>>>>>>>>>>>> return MPI_SUCCESS; >>>>>>>>>>>>>>> } >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> This will work if the user creates tons of attributes but >>>>>>>>>>>>>>> is constantly deleting some as they new ones. So long as the number >>>>>>>>>>>>>>> outstanding at one time is < MAX_ATTR) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Barry >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Jun 20, 2020, at 10:54 AM, Junchao Zhang < >>>>>>>>>>>>>>> junchao.zhang at gmail.com> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I don't understand what your session means. Let's try this >>>>>>>>>>>>>>> patch >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> diff --git a/src/sys/mpiuni/mpi.c b/src/sys/mpiuni/mpi.c >>>>>>>>>>>>>>> index d559a513..c058265d 100644 >>>>>>>>>>>>>>> --- a/src/sys/mpiuni/mpi.c >>>>>>>>>>>>>>> +++ b/src/sys/mpiuni/mpi.c >>>>>>>>>>>>>>> @@ -283,6 +283,7 @@ int MPI_Finalize(void) >>>>>>>>>>>>>>> MPI_Comm_free(&comm); >>>>>>>>>>>>>>> comm = MPI_COMM_SELF; >>>>>>>>>>>>>>> MPI_Comm_free(&comm); >>>>>>>>>>>>>>> + num_attr = 1; /* reset the counter */ >>>>>>>>>>>>>>> MPI_was_finalized = 1; >>>>>>>>>>>>>>> return MPI_SUCCESS; >>>>>>>>>>>>>>> } >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> --Junchao Zhang >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Sat, Jun 20, 2020 at 10:48 AM Sam Guo < >>>>>>>>>>>>>>> sam.guo at cd-adapco.com> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Typo: I mean ?Assuming initializer is only needed once for >>>>>>>>>>>>>>>> entire session? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Saturday, June 20, 2020, Sam Guo >>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Assuming finalizer is only needed once for entire >>>>>>>>>>>>>>>>> session(?), I can put initializer into the static block to call it once but >>>>>>>>>>>>>>>>> where do I call finalizer? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Saturday, June 20, 2020, Junchao Zhang < >>>>>>>>>>>>>>>>> junchao.zhang at gmail.com> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> The counter num_attr should be recycled. But first try to >>>>>>>>>>>>>>>>>> call PETSc initialize/Finalize only once to see it fixes the error. >>>>>>>>>>>>>>>>>> --Junchao Zhang >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On Sat, Jun 20, 2020 at 12:48 AM Sam Guo < >>>>>>>>>>>>>>>>>> sam.guo at cd-adapco.com> wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> To clarify, I call PETSc initialize and PETSc finalize >>>>>>>>>>>>>>>>>>> everytime I call SLEPc: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> PetscInitializeNoPointers(argc,args,nullptr,nullptr); >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> SlepcInitialize(&argc,&args,static_cast(nullptr),help); >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> //calling slepc >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> SlepcFinalize(); >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> PetscFinalize(); >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> On Fri, Jun 19, 2020 at 10:32 PM Sam Guo < >>>>>>>>>>>>>>>>>>> sam.guo at cd-adapco.com> wrote: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Dear PETSc team, >>>>>>>>>>>>>>>>>>>> When I called SLEPc multiple time, I eventually got >>>>>>>>>>>>>>>>>>>> following error: >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> MPI operation not supported by PETSc's sequential MPI >>>>>>>>>>>>>>>>>>>> wrappers >>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: #1 PetscInitialize() line 967 in >>>>>>>>>>>>>>>>>>>> ../../../petsc/src/sys/objects/pinit.c >>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: #2 SlepcInitialize() line 262 in >>>>>>>>>>>>>>>>>>>> ../../../slepc/src/sys/slepcinit.c >>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: #3 SlepcInitializeNoPointers() line 359 >>>>>>>>>>>>>>>>>>>> in ../../../slepc/src/sys/slepcinit.c >>>>>>>>>>>>>>>>>>>> PETSC ERROR: Logging has not been enabled. >>>>>>>>>>>>>>>>>>>> You might have forgotten to call PetscInitialize(). >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> I debugged: it is because of following in >>>>>>>>>>>>>>>>>>>> petsc/src/sys/mpiuni/mpi.c >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> if (num_attr >= MAX_ATTR) >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> in function int >>>>>>>>>>>>>>>>>>>> MPI_Comm_create_keyval(MPI_Copy_function *copy_fn,MPI_Delete_function >>>>>>>>>>>>>>>>>>>> *delete_fn,int *keyval,void *extra_state) >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> num_attr is declared static and keeps increasing every >>>>>>>>>>>>>>>>>>>> time MPI_Comm_create_keyval is called. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> I am using petsc 3.11.3 but found 3.13.2 has the >>>>>>>>>>>>>>>>>>>> same logic. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Is this a bug or I didn't use it correctly? >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>>>>> Sam >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>> >>>>>>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From andrea_iob at hotmail.com Tue Jun 30 06:59:13 2020 From: andrea_iob at hotmail.com (Andrea Iob) Date: Tue, 30 Jun 2020 11:59:13 +0000 Subject: [petsc-users] DIVERGED_NANORINF when using HYPRE/BoomerAMG Message-ID: Hi, I'm trying to solve a linear system using HYPRE/BoomerAMG as preconditioner. The system comes from a two-dimensional adjoint Navier-Stokes problem. The mesh is structured (6400 cells) and there are 24 DoFs for each cell (the matrix has a total of 153600 rows). The condition number of the matrix should be in the order of 1e9. Using ILU preconditioner and FGMRES, the system is correctly solved (it takes 800 iterations to reach a residual of 1e-12). However, replacing ILU with HYPRE/BoomerAMG, the solver stops right after the first iteration (DIVERGED_NANORINF). I've tried different BoomerAMG options and different solvers (including Richardson to use BoomerAMG without a Krylov method), but I always get the same DIVERGED_NANORINF error. Using the same HYPRE/BoomerAMG + FGMERS setup on another problem (Laplacian on a highly stretched grid), the solver reaches convergence very quickly without any problems. At the bottom of this mail, you'll find the log of a run that stops with DIVERGED_NANORINF. Do you spot something in my setup that may explain why the solver is not converging? Thanks. Best regards, Andrea ----------------------------------------- Assembly started... Reading matrix file... - Number of rows (N) = 153600 - Number of non-zero entries (NNZ) = 18247680 Assembly completed. Time elapsed 83.3231s Set up started... Set up completed. Time elapsed 6.62441s Solution started... 0 KSP unpreconditioned resid norm 7.736650641501e-01 true resid norm 7.736650641501e-01 ||r(i)||/||b|| 1.000000000000e+00 0 KSP Residual norm 7.736650641501e-01 % max 1.000000000000e+00 min 1.000000000000e+00 max/min 1.000000000000e+00 [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: [0]PETSC ERROR: KSPSolve has not converged due to Nan or Inf inner product [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.10.3, unknown [0]PETSC ERROR: bitpit_system_solver on a named xxx by xxx Tue Jun 30 09:42:09 2020 [0]PETSC ERROR: Configure options PETSC_ARCH=arch-linux2-c-opt-gcc7-hypre --with-blaslapack-dir=/opt/lapack/3.8.0-gcc.7.4.0/lib64 --with-debugging=0 COPTFLAGS=-O3 CXXOPTFLAGS=-O3 FOPTFLAGS=-O3 --with-valgrind=1 --with-valgrind-dir=/opt/valgrind/3.14.0/ --prefix=/opt/petsc/3.10.3_gcc-7.4.0 --with-mpi=1 --download-hypre [0]PETSC ERROR: #1 KSPGMRESClassicalGramSchmidtOrthogonalization() line 67 in /root/InstallSources/petsc/src/ksp/ksp/impls/gmres/borthog2.c [0]PETSC ERROR: #2 KSPFGMRESCycle() line 175 in /root/InstallSources/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c [0]PETSC ERROR: #3 KSPSolve_FGMRES() line 291 in /root/InstallSources/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c [0]PETSC ERROR: #4 KSPSolve() line 780 in /root/InstallSources/petsc/src/ksp/ksp/interface/itfunc.c KSP Object: 1 MPI processes type: fgmres restart=45, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement happy breakdown tolerance 1e-30 maximum iterations=10000, nonzero initial guess tolerances: relative=1e-12, absolute=1e-50, divergence=10000. right preconditioning using UNPRECONDITIONED norm type for convergence test PC Object: 1 MPI processes type: hypre HYPRE BoomerAMG preconditioning Cycle type V Maximum number of levels 25 Maximum number of iterations PER hypre call 1 Convergence tolerance PER hypre call 0. Threshold for strong coupling 0.7 Interpolation truncation factor 0.3 Interpolation: max elements per row 2 Number of levels of aggressive coarsening 4 Number of paths for aggressive coarsening 5 Maximum row sums 0.9 Sweeps down 1 Sweeps up 1 Sweeps on coarse 1 Relax down sequential-Gauss-Seidel Relax up sequential-Gauss-Seidel Relax on coarse Gaussian-elimination Relax weight (all) 1. Outer relax weight (all) 1. Using CF-relaxation Smooth type Euclid Smooth num levels 25 Euclid ILU(k) levels 0 Euclid ILU(k) drop tolerance 0. Euclid ILU use Block-Jacobi? 1 Measure type local Coarsen type HMIS Interpolation type ext+i linear system matrix = precond matrix: Mat Object: Mat_0x1e88040_0 1 MPI processes type: seqaij rows=153600, cols=153600, bs=24 total: nonzeros=18247680, allocated nonzeros=18247680 total number of mallocs used during MatSetValues calls =0 using I-node routines: found 32000 nodes, limit used is 5 Solution completed. Time elapsed 6.93336s Solution information... Error (L2) : 32.8359 Error (Linf) : 1.93278 ************************************************************************************************************************ *** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript -r -fCourier9' to print this document *** ************************************************************************************************************************ ---------------------------------------------- PETSc Performance Summary: ---------------------------------------------- bitpit_system_solver on a named xxx with 1 processor, by xxx Tue Jun 30 09:43:46 2020 Using Petsc Release Version 3.10.3, unknown Max Max/Min Avg Total Time (sec): 9.718e+01 1.000 9.718e+01 Objects: 5.300e+01 1.000 5.300e+01 Flop: 1.107e+08 1.000 1.107e+08 1.107e+08 Flop/sec: 1.139e+06 1.000 1.139e+06 1.139e+06 MPI Messages: 0.000e+00 0.000 0.000e+00 0.000e+00 MPI Message Lengths: 0.000e+00 0.000 0.000e+00 0.000e+00 MPI Reductions: 0.000e+00 0.000 Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract) e.g., VecAXPY() for real vectors of length N --> 2N flop and VecAXPY() for complex vectors of length N --> 8N flop Summary of Stages: ----- Time ------ ----- Flop ------ --- Messages --- -- Message Lengths -- -- Reductions -- Avg %Total Avg %Total Count %Total Avg %Total Count %Total 0: Main Stage: 9.7178e+01 100.0% 1.1071e+08 100.0% 0.000e+00 0.0% 0.000e+00 0.0% 0.000e+00 0.0% ------------------------------------------------------------------------------------------------------------------------ See the 'Profiling' chapter of the users' manual for details on interpreting output. Phase summary info: Count: number of times phase was executed Time and Flop: Max - maximum over all processors Ratio - ratio of maximum to minimum over all processors Mess: number of messages sent AvgLen: average message length (bytes) Reduct: number of global reductions Global: entire computation Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop(). %T - percent time in this phase %F - percent flop in this phase %M - percent messages in this phase %L - percent message lengths in this phase %R - percent reductions in this phase Total Mflop/s: 10e-6 * (sum of flop over all processors)/(max time over all processors) ------------------------------------------------------------------------------------------------------------------------ Event Count Time (sec) Flop --- Global --- --- Stage ---- Total Max Ratio Max Ratio Max Ratio Mess AvgLen Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s ------------------------------------------------------------------------------------------------------------------------ --- Event Stage 0: Main Stage BuildTwoSidedF 2 1.0 1.8471e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatMult 3 1.0 6.8688e-02 1.0 1.09e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 98 0 0 0 0 98 0 0 0 1587 MatConvert 2 1.0 3.8534e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatAssemblyBegin 3 1.0 2.8112e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatAssemblyEnd 3 1.0 7.8563e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetRowIJ 2 1.0 3.6100e-06 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatView 2 1.0 5.4618e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 56 0 0 0 0 56 0 0 0 0 0 VecView 2 1.0 7.6981e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0 VecMDot 1 1.0 1.7635e-04 1.0 3.07e+05 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 1742 VecNorm 3 1.0 5.4832e-04 1.0 9.22e+05 1.0 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 1681 VecScale 1 1.0 1.5216e-04 1.0 1.54e+05 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 1009 VecCopy 1 1.0 1.5406e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecSet 21 1.0 7.1106e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAYPX 1 1.0 1.9940e-04 1.0 1.54e+05 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 770 VecWAXPY 1 1.0 7.8802e-04 1.0 1.54e+05 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 195 KSPSetUp 2 1.0 6.8995e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 PCSetUp 2 1.0 1.3190e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 14 0 0 0 0 14 0 0 0 0 0 PCApply 1 1.0 2.2728e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 ------------------------------------------------------------------------------------------------------------------------ Memory usage is given in bytes: Object Type Creations Destructions Memory Descendants' Mem. Reports information only for process 0. --- Event Stage 0: Main Stage Matrix 3 1 2872 0. Vector 32 20 17234800 0. Index Set 4 4 3200 0. IS L to G Mapping 2 0 0 0. Vec Scatter 2 0 0 0. Viewer 6 4 3392 0. Krylov Solver 2 1 61676 0. Preconditioner 2 1 1432 0. ======================================================================================================================== Average time to get PetscTime(): 3.36e-08 #PETSc Option Table entries: -ksp_converged_reason -ksp_error_if_not_converged -ksp_monitor_singular_value -ksp_monitor_true_residual -log_view -pc_hypre_boomeramg_agg_nl 4 -pc_hypre_boomeramg_agg_num_paths 5 -pc_hypre_boomeramg_coarsen_type HMIS -pc_hypre_boomeramg_eu_bj true -pc_hypre_boomeramg_interp_type ext+i -pc_hypre_boomeramg_P_max 2 -pc_hypre_boomeramg_relax_type_all sequential-Gauss-Seidel -pc_hypre_boomeramg_smooth_type Euclid -pc_hypre_boomeramg_strong_threshold 0.7 -pc_hypre_boomeramg_truncfactor 0.3 #End of PETSc Option Table entries Compiled without FORTRAN kernels Compiled with full precision matrices (default) sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4 Configure options: PETSC_ARCH=arch-linux2-c-opt-gcc7-hypre --with-blaslapack-dir=/opt/lapack/3.8.0-gcc.7.4.0/lib64 --with-debugging=0 COPTFLAGS=-O3 CXXOPTFLAGS=-O3 FOPTFLAGS=-O3 --with-valgrind=1 --with-valgrind-dir=/opt/valgrind/3.14.0/ --prefix=/opt/petsc/3.10.3_gcc-7.4.0 --with-mpi=1 --download-hypre ----------------------------------------- Libraries compiled on 2020-06-26 12:35:46 on xxx Machine characteristics: Linux-3.10.0-1127.8.2.el7.x86_64-x86_64-with-centos-7.6.1810-Core Using PETSc directory: /opt/petsc/3.10.3_gcc-7.4.0 Using PETSc arch: ----------------------------------------- Using C compiler: mpicc -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -fstack-protector -fvisibility=hidden -O3 Using Fortran compiler: mpif90 -fPIC -Wall -ffree-line-length-0 -Wno-unused-dummy-argument -O3 ----------------------------------------- Using include paths: -I/opt/petsc/3.10.3_gcc-7.4.0/include -I/opt/valgrind/3.14.0/include ----------------------------------------- Using C linker: mpicc Using Fortran linker: mpif90 Using libraries: -Wl,-rpath,/opt/petsc/3.10.3_gcc-7.4.0/lib -L/opt/petsc/3.10.3_gcc-7.4.0/lib -lpetsc -Wl,-rpath,/opt/petsc/3.10.3_gcc-7.4.0/lib -L/opt/petsc/3.10.3_gcc-7.4.0/lib -Wl,-rpath,/opt/lapack/3.8.0-gcc.7.4.0/lib64 -L/opt/lapack/3.8.0-gcc.7.4.0/lib64 -Wl,-rpath,/opt/openmpi4.0.0-gcc7.4.0_slurm/lib -L/opt/openmpi4.0.0-gcc7.4.0_slurm/lib -Wl,-rpath,/opt/gnu/7.4/lib/gcc/x86_64-pc-linux-gnu/7.4.0 -L/opt/gnu/7.4/lib/gcc/x86_64-pc-linux-gnu/7.4.0 -Wl,-rpath,/opt/gnu/7.4/lib/gcc -L/opt/gnu/7.4/lib/gcc -Wl,-rpath,/opt/gnu/7.4/lib64 -L/opt/gnu/7.4/lib64 -Wl,-rpath,/opt/gnu/7.4/lib -L/opt/gnu/7.4/lib -lHYPRE -llapack -lblas -lm -lX11 -lstdc++ -ldl -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi -lgfortran -lm -lgfortran -lm -lgcc_s -lquadmath -lpthread -lstdc++ -ldl ----------------------------------------- -------------- next part -------------- An HTML attachment was scrubbed... URL: From paololampitella at hotmail.com Tue Jun 30 08:09:59 2020 From: paololampitella at hotmail.com (Paolo Lampitella) Date: Tue, 30 Jun 2020 13:09:59 +0000 Subject: [petsc-users] R: PETSc and Windows 10 In-Reply-To: <849E7E3B-8B54-4B40-A2B7-8DD5B55428B3@enseeiht.fr> References: <32B37EFD-DDFA-41AE-ACC7-CB5E6AE57996@enseeiht.fr> <0558CCF7-8203-4520-8673-3B0945C43233@enseeiht.fr> <20AB9F7A-4471-498B-9A78-AA48163900CB@enseeiht.fr> <94A8E366-2361-409E-84CE-B90EA54D9AF9@enseeiht.fr> , <849E7E3B-8B54-4B40-A2B7-8DD5B55428B3@enseeiht.fr> Message-ID: Dear Pierre, thanks for the fast response. Unfortunately it still fails, but now in the configure of ScaLAPACK (which means that it went ok for slepc, tetgen, metis, parmetis, ptscotch, superlu and suitesparse). The way I applied the modification is by manually editing the Makefile in the 3rdparty/ff-petsc folder, adding -fallow-invalid-boz to both CFLAGS and FFLAGS (this entry added by me). Then executed make petsc-slepc. As my project is much less ambitious, I have a good feeling that I will be able to use your Makefile successfully, but as I am kind of slow I tought that it would have been useful for you to know. The configure.log is attached. This time the error is: Rank mismatch between actual argument at (1) and actual argument at (2) (scalar and rank-1) in subroutine pclarf.f of ScaLAPACK. However, before attampting with my project, I have few questions about your Makefile, in particular this piece: --with-mpi-lib=/c/Windows/System32/msmpi.dll --with-mpi-include=/home/paolo/FreeFem-sources/3rdparty/include/msmpi --with-mpiexec="/C/Program\ Files/Microsoft\ MPI/Bin/mpiexec" I see from MPI.py that I should not use ?--with-mpi-lib/include? if I want to use my now working mpi wrappers. Is this correct? Paolo Inviato da Posta per Windows 10 Da: Pierre Jolivet Inviato: luned? 29 giugno 2020 21:37 A: Paolo Lampitella Cc: Matthew Knepley; petsc-users Oggetto: Re: [petsc-users] PETSc and Windows 10 I do not give up easily on Windows problems: 1) that?s around 50% of our (FreeFEM) user-base (and I want them to use PETSc and SLEPc, ofc?) 2) most people I work with from corporations just have Windows laptops/desktops and I always recommend MSYS because it?s very lightweight and you can pass .exe around 3) I?ve bothered enough Satish, Jed, and Matt on GitLab to take (at least partially) the blame now when it doesn?t work on MSYS That being said, the magic keyword is the added flag FFLAGS="-fallow-invalid-boz" (see, I told you ./configure issues were easier to deal with than the others). Here you?ll see that everything goes through just fine (sorry, it took me a long time to post this because everything is slow on my VM): 1) http://jolivet.perso.enseeiht.fr/win10/configure.log 2) http://jolivet.perso.enseeiht.fr/win10/make.log (both steps #1 and #2 in MSYS terminal, gcc/gfortran 10, MS-MPI see screenshot) 3) http://jolivet.perso.enseeiht.fr/win10/ex2.txt (Command Prompt, 4 processes + MUMPS, I can send you the .exe if you want to try on your machine) I just realize that I didn?t generate the Fortran bindings, but you can see I compiled MUMPS and ScaLAPACK, so that shouldn?t be a problem. Or if there is a problem, we will need to fix this in PETSc. I?ll push this added flag to the FreeFEM repo, thanks for reminding me of the brokenness of gcc/gfortran 10 + MS-MPI. Here is to hoping this won?t affect PETSc ./configure with previous gcc/gfortran version (unlikely, this option is apparently 13-year old https://gcc.gnu.org/bugzilla/show_bug.cgi?id=29471) Let me know of the next hiccup, if any. Thanks, Pierre On 29 Jun 2020, at 8:09 PM, Paolo Lampitella > wrote: Dear Pierre, thanks again for your time I guess there is no way for me to use the toolchain you are using (I don?t remember having any choice on which version of MSYS or GCC I could install) Paolo Inviato da Posta per Windows 10 Da: Pierre Jolivet Inviato: luned? 29 giugno 2020 20:01 A: Matthew Knepley Cc: Paolo Lampitella; petsc-users Oggetto: Re: [petsc-users] PETSc and Windows 10 On 29 Jun 2020, at 7:47 PM, Matthew Knepley > wrote: On Mon, Jun 29, 2020 at 1:35 PM Paolo Lampitella > wrote: Dear Pierre, sorry to bother you, but I already have some issues. What I did: * pacman -R mingw-w64-x86_64-python mingw-w64-x86_64-gdb (is gdb also troublesome?) * Followed points 6 and 7 at https://doc.freefem.org/introduction/installation.html#compilation-on-windows I first got a warning on the configure at point 6, as ?disable-hips is not recognized. Then, on make ?petsc-slepc? of point 7 (no SUDO=sudo flag was necessary) I got to this point: tar xzf ../pkg/petsc-lite-3.13.0.tar.gz patch -p1 < petsc-suitesparse.patch patching file petsc-3.13.0/config/BuildSystem/config/packages/SuiteSparse.py touch petsc-3.13.0/tag-tar cd petsc-3.13.0 && ./configure MAKEFLAGS='' \ --prefix=/home/paolo/freefem/ff-petsc//r \ --with-debugging=0 COPTFLAGS='-O3 -mtune=generic' CXXOPTFLAGS='-O3 -mtune=generic' FOPTFLAGS='-O3 -mtune=generic' --with-cxx-dialect=C++11 --with-ssl=0 --with-x=0 --with-fortran-bindings=0 --with-shared-libraries=0 --with-cc='gcc' --with-cxx='g++' --with-fc='gfortran' CXXFLAGS='-fno-stack-protector' CFLAGS='-fno-stack-protector' --with-scalar-type=real --with-mpi-lib='/c/Windows/System32/msmpi.dll' --with-mpi-include='/home/paolo/FreeFem-sources/3rdparty/include/msmpi' --with-mpiexec='/C/Program\ Files/Microsoft\ MPI/Bin/mpiexec' --with-blaslapack-include='' --with-blaslapack-lib='/mingw64/bin/libopenblas.dll' --download-scalapack --download-metis --download-ptscotch --download-mumps --download-hypre --download-parmetis --download-superlu --download-suitesparse --download-tetgen --download-slepc '--download-metis-cmake-arguments=-G "MSYS Makefiles"' '--download-parmetis-cmake-arguments=-G "MSYS Makefiles"' '--download-superlu-cmake-arguments=-G "MSYS Makefiles"' '--download-hypre-configure-arguments=--build=x86_64-linux-gnu --host=x86_64-linux-gnu' PETSC_ARCH=fr =============================================================================== Configuring PETSc to compile on your system =============================================================================== TESTING: FortranMPICheck from config.packages.MPI(config/BuildSystem/config/pack******************************************************************************* UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): ------------------------------------------------------------------------------- Fortran error! mpi_init() could not be located! ******************************************************************************* make: *** [Makefile:210: petsc-3.13.0/tag-conf-real] Errore 1 Note that I didn?t add anything to any PATH variable, because this is not mentioned in your documentation. On a side note, this is the same error I got when trying to build PETSc in Cygwin with the default OpenMPI available in Cygwin. I am attaching the configure.log? it seems to me that the error comes from the configure trying to include the mpif.h in your folder and not using the -fallow-invalid-boz flag that I had to use, for example, to compile mpi.f90 into mpi.mod But I?m not sure why this is happening Pierre, Could this be due to gcc 10? Sorry, I?m slow. You are right. Our workers use gcc 9, everything is fine, but I see on my VM which I updated that I use gcc 10 and had to disable Fortran, I guess the MUMPS run I showcased was with a prior PETSc build. I?ll try to resolve this and will keep you posted. They really caught a lot of people off guard with gfortran 10? Thanks, Pierre Executing: gfortran -c -o /tmp/petsc-ur0cff6a/config.libraries/conftest.o -I/tmp/petsc-ur0cff6a/config.compilers -I/tmp/petsc-ur0cff6a/config.setCompilers -I/tmp/petsc-ur0cff6a/config.compilersFortran -I/tmp/petsc-ur0cff6a/config.libraries -Wall -ffree-line-length-0 -Wno-unused-dummy-argument -O3 -mtune=generic -I/home/paolo/FreeFem-sources/3rdparty/include/msmpi /tmp/petsc-ur0cff6a/config.libraries/conftest.F90 Possible ERROR while running compiler: exit code 1 stderr: C:/msys64/home/paolo/FreeFem-sources/3rdparty/include/msmpi/mpif.h:227:36: 227 | PARAMETER (MPI_DATATYPE_NULL=z'0c000000') | 1 Error: BOZ literal constant at (1) is neither a data-stmt-constant nor an actual argument to INT, REAL, DBLE, or CMPLX intrinsic function [see '-fno-allow-invalid-boz'] C:/msys64/home/paolo/FreeFem-sources/3rdparty/include/msmpi/mpif.h:303:27: 303 | PARAMETER (MPI_CHAR=z'4c000101') | 1 Error: BOZ literal constant at (1) is neither a data-stmt-constant nor an actual argument to INT, REAL, DBLE, or CMPLX intrinsic function [see '-fno-allow-invalid-boz'] C:/msys64/home/paolo/FreeFem-sources/3rdparty/include/msmpi/mpif.h:305:36: 305 | PARAMETER (MPI_UNSIGNED_CHAR=z'4c000102') | 1 Thanks, Matt Thanks Paolo Inviato da Posta per Windows 10 Da: Pierre Jolivet Inviato: luned? 29 giugno 2020 18:34 A: Paolo Lampitella Cc: Satish Balay; petsc-users Oggetto: Re: [petsc-users] PETSc and Windows 10 On 29 Jun 2020, at 6:27 PM, Paolo Lampitella > wrote: I think I made the first step of having mingw64 from msys2 working with ms-mpi. I found that the issue I was having was related to: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=91556 and, probably (but impossible to check now), I was using an msys2 and/or mingw mpi package before this fix: https://github.com/msys2/MINGW-packages/commit/11b4cff3d2ec7411037b692b0ad5a9f3e9b9978d#diff-eac59989e3096be97d940c8f47b50fba Admittedly, I never used gcc 10 before on any machine. Still, I feel that reporting that sort of error in that way is, at least, misleading (I would have preferred the initial implementation as mentioned in the gcc bug track). A second thing that I was not used to, and made me more uncertain of the procedure I was following, is having to compile myself the mpi module. There are several version of this out there, but I decided to stick with this one: https://www.scivision.dev/windows-mpi-msys2/ even if there seems to be no need to include -fno-range-check and the current mpi.f90 version is different from the mpif.h as reported here: https://github.com/microsoft/Microsoft-MPI/issues/33 which, to me, are both signs of lack of attention on the fortran side by those that maintain this thing. In summary, this is the procedure I followed so far (on a 64 bit machine with Windows 10): * Install MSYS2 from https://www.msys2.org/ and just follow the install wizard * Open the MSYS2 terminal and execute: pacman -Syuu * Close the terminal when asked and reopen it * Keep executing ?pacman -Syuu? until nothing else needs to be updated * Close the MSYS2 terminal and reopen it (I guess because was in paranoid mode), then install packages with: pacman -S base-devel git gcc gcc-fortran bsdcpio lndir pax-git unzip pacman -S mingw-w64-x86_64-toolchain pacman -S mingw-w64-x86_64-msmpi pacman -S mingw-w64-x86_64-cmake pacman -S mingw-w64-x86_64-freeglut pacman -S mingw-w64-x86_64-gsl pacman -S mingw-w64-x86_64-libmicroutils pacman -S mingw-w64-x86_64-hdf5 pacman -S mingw-w64-x86_64-openblas pacman -S mingw-w64-x86_64-arpack pacman -S mingw-w64-x86_64-jq This set should include all the libraries mentioned by Pierre and/or used by his Jenkins, as the final scope here is to have PETSc and dependencies working. But I think that for pure MPI one could stop to msmpi (even, maybe, just install msmpi and have the dependencies figured out by pacman). Honestly, I don?t remember the exact order I used to install the packages, but this should not affect things. Also, as I was still in paranoid mode, I kept executing ?pacman -Syuu? after each package was installed. After this, close the MSYS2 terminal. * Open the MINGW64 terminal and create the .mod file out of the mpi.f90 file, as mentioned here https://www.scivision.dev/windows-mpi-msys2/, with: cd /mingw64/include gfortran mpif90 -c -fno-range-check -fallow-invalid-boz Ah, yes, that?s new to gfortran 10 (we use gfortran 9 on our workers), which is now what?s ship with MSYS2 (we haven?t updated yet). Sorry that I forgot about that. This is needed to ?USE mpi? (as opposed to INCLUDE ?mpif.h?) * Install the latest MS-MPI (both sdk and setup) from https://www.microsoft.com/en-us/download/details.aspx?id=100593 At this point I?ve been able to compile (using the MINGW64 terminal) different mpi test programs and they run as expected in the classical Windows prompt. I added this function to my .bashrc in MSYS2 in order to easily copy the required dependencies out of MSYS: function copydep() { ldd $1 | grep "=> /$2" | awk '{print $3}' | xargs -I '{}' cp -v '{}' .; } which can be used, with the MINGW64 terminal, by navigating to the folder where the final executable, say, my.exe, resides (even if under a Windows path) and executing: copydep my.exe mingw64 This, of course, must be done before actually trying to execute the .exe in the windows cmd prompt. Hopefully, I should now be able to follow Pierre?s instructions for PETSc (but first I wanna give a try to the system python before removing it) Looks like the hard part is over. It?s usually easier to deal with ./configure issues. If you have weird errors like ?incomplete Cygwin install? or whatever, this is the kind of issues I was referring to earlier. In that case, what I?d suggest is just, as before: pacman -R mingw-w64-x86_64-python mingw-w64-x86_64-gdb pacman -S python Thanks, Pierre Thanks Paolo -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: configure.log Type: application/octet-stream Size: 2150504 bytes Desc: configure.log URL: From knepley at gmail.com Tue Jun 30 08:14:53 2020 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 30 Jun 2020 09:14:53 -0400 Subject: [petsc-users] DIVERGED_NANORINF when using HYPRE/BoomerAMG In-Reply-To: References: Message-ID: On Tue, Jun 30, 2020 at 7:59 AM Andrea Iob wrote: > Hi, > > I'm trying to solve a linear system using HYPRE/BoomerAMG as > preconditioner. The system comes from a two-dimensional adjoint > Navier-Stokes problem. > Do you expect the system you are solving to be elliptic? BoomerAMG is designed for elliptic systems, and can easily fail if applied to a more general system, say one with zeros on the diagonal. Thanks, Matt > The mesh is structured (6400 cells) and there are 24 DoFs for each cell > (the matrix has a total of 153600 rows). The condition number of the matrix > should be in the order of 1e9. Using ILU preconditioner and FGMRES, the > system is correctly solved (it takes 800 iterations to reach a residual of > 1e-12). However, replacing ILU with HYPRE/BoomerAMG, the solver stops right > after the first iteration (DIVERGED_NANORINF). I've tried different > BoomerAMG options and different solvers (including Richardson to use > BoomerAMG without a Krylov method), but I always get the same > DIVERGED_NANORINF error. > > Using the same HYPRE/BoomerAMG + FGMERS setup on another problem > (Laplacian on a highly stretched grid), the solver reaches convergence very > quickly without any problems. > > At the bottom of this mail, you'll find the log of a run that stops with > DIVERGED_NANORINF. Do you spot something in my setup that may explain why > the solver is not converging? > > Thanks. Best regards, > Andrea > > ----------------------------------------- > Assembly started... > Reading matrix file... > - Number of rows (N) = 153600 > - Number of non-zero entries (NNZ) = 18247680 > Assembly completed. > Time elapsed 83.3231s > Set up started... > Set up completed. > Time elapsed 6.62441s > Solution started... > 0 KSP unpreconditioned resid norm 7.736650641501e-01 true resid norm > 7.736650641501e-01 ||r(i)||/||b|| 1.000000000000e+00 > 0 KSP Residual norm 7.736650641501e-01 % max 1.000000000000e+00 min > 1.000000000000e+00 max/min 1.000000000000e+00 > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: > [0]PETSC ERROR: KSPSolve has not converged due to Nan or Inf inner product > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.10.3, unknown > [0]PETSC ERROR: bitpit_system_solver on a named xxx by xxx Tue Jun 30 > 09:42:09 2020 > [0]PETSC ERROR: Configure options PETSC_ARCH=arch-linux2-c-opt-gcc7-hypre > --with-blaslapack-dir=/opt/lapack/3.8.0-gcc.7.4.0/lib64 --with-debugging=0 > COPTFLAGS=-O3 CXXOPTFLAGS=-O3 FOPTFLAGS=-O3 --with-valgrind=1 > --with-valgrind-dir=/opt/valgrind/3.14.0/ > --prefix=/opt/petsc/3.10.3_gcc-7.4.0 --with-mpi=1 --download-hypre > [0]PETSC ERROR: #1 KSPGMRESClassicalGramSchmidtOrthogonalization() line 67 > in /root/InstallSources/petsc/src/ksp/ksp/impls/gmres/borthog2.c > [0]PETSC ERROR: #2 KSPFGMRESCycle() line 175 in > /root/InstallSources/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c > [0]PETSC ERROR: #3 KSPSolve_FGMRES() line 291 in > /root/InstallSources/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c > [0]PETSC ERROR: #4 KSPSolve() line 780 in > /root/InstallSources/petsc/src/ksp/ksp/interface/itfunc.c > KSP Object: 1 MPI processes > type: fgmres > restart=45, using Classical (unmodified) Gram-Schmidt > Orthogonalization with no iterative refinement > happy breakdown tolerance 1e-30 > maximum iterations=10000, nonzero initial guess > tolerances: relative=1e-12, absolute=1e-50, divergence=10000. > right preconditioning > using UNPRECONDITIONED norm type for convergence test > PC Object: 1 MPI processes > type: hypre > HYPRE BoomerAMG preconditioning > Cycle type V > Maximum number of levels 25 > Maximum number of iterations PER hypre call 1 > Convergence tolerance PER hypre call 0. > Threshold for strong coupling 0.7 > Interpolation truncation factor 0.3 > Interpolation: max elements per row 2 > Number of levels of aggressive coarsening 4 > Number of paths for aggressive coarsening 5 > Maximum row sums 0.9 > Sweeps down 1 > Sweeps up 1 > Sweeps on coarse 1 > Relax down sequential-Gauss-Seidel > Relax up sequential-Gauss-Seidel > Relax on coarse Gaussian-elimination > Relax weight (all) 1. > Outer relax weight (all) 1. > Using CF-relaxation > Smooth type Euclid > Smooth num levels 25 > Euclid ILU(k) levels 0 > Euclid ILU(k) drop tolerance 0. > Euclid ILU use Block-Jacobi? 1 > Measure type local > Coarsen type HMIS > Interpolation type ext+i > linear system matrix = precond matrix: > Mat Object: Mat_0x1e88040_0 1 MPI processes > type: seqaij > rows=153600, cols=153600, bs=24 > total: nonzeros=18247680, allocated nonzeros=18247680 > total number of mallocs used during MatSetValues calls =0 > using I-node routines: found 32000 nodes, limit used is 5 > Solution completed. > Time elapsed 6.93336s > Solution information... > Error (L2) : 32.8359 > Error (Linf) : 1.93278 > > ************************************************************************************************************************ > *** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript -r > -fCourier9' to print this document *** > > ************************************************************************************************************************ > > ---------------------------------------------- PETSc Performance Summary: > ---------------------------------------------- > > bitpit_system_solver on a named xxx with 1 processor, by xxx Tue Jun 30 > 09:43:46 2020 > Using Petsc Release Version 3.10.3, unknown > > Max Max/Min Avg Total > Time (sec): 9.718e+01 1.000 9.718e+01 > Objects: 5.300e+01 1.000 5.300e+01 > Flop: 1.107e+08 1.000 1.107e+08 1.107e+08 > Flop/sec: 1.139e+06 1.000 1.139e+06 1.139e+06 > MPI Messages: 0.000e+00 0.000 0.000e+00 0.000e+00 > MPI Message Lengths: 0.000e+00 0.000 0.000e+00 0.000e+00 > MPI Reductions: 0.000e+00 0.000 > > Flop counting convention: 1 flop = 1 real number operation of type > (multiply/divide/add/subtract) > e.g., VecAXPY() for real vectors of length N > --> 2N flop > and VecAXPY() for complex vectors of length N > --> 8N flop > > Summary of Stages: ----- Time ------ ----- Flop ------ --- Messages > --- -- Message Lengths -- -- Reductions -- > Avg %Total Avg %Total Count > %Total Avg %Total Count %Total > 0: Main Stage: 9.7178e+01 100.0% 1.1071e+08 100.0% 0.000e+00 > 0.0% 0.000e+00 0.0% 0.000e+00 0.0% > > > ------------------------------------------------------------------------------------------------------------------------ > See the 'Profiling' chapter of the users' manual for details on > interpreting output. > Phase summary info: > Count: number of times phase was executed > Time and Flop: Max - maximum over all processors > Ratio - ratio of maximum to minimum over all processors > Mess: number of messages sent > AvgLen: average message length (bytes) > Reduct: number of global reductions > Global: entire computation > Stage: stages of a computation. Set stages with PetscLogStagePush() and > PetscLogStagePop(). > %T - percent time in this phase %F - percent flop in this > phase > %M - percent messages in this phase %L - percent message lengths > in this phase > %R - percent reductions in this phase > Total Mflop/s: 10e-6 * (sum of flop over all processors)/(max time over > all processors) > > ------------------------------------------------------------------------------------------------------------------------ > Event Count Time (sec) > Flop --- Global --- --- Stage ---- Total > Max Ratio Max Ratio Max Ratio Mess AvgLen > Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s > > ------------------------------------------------------------------------------------------------------------------------ > > --- Event Stage 0: Main Stage > > BuildTwoSidedF 2 1.0 1.8471e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > MatMult 3 1.0 6.8688e-02 1.0 1.09e+08 1.0 0.0e+00 0.0e+00 > 0.0e+00 0 98 0 0 0 0 98 0 0 0 1587 > MatConvert 2 1.0 3.8534e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > MatAssemblyBegin 3 1.0 2.8112e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > MatAssemblyEnd 3 1.0 7.8563e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > MatGetRowIJ 2 1.0 3.6100e-06 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > MatView 2 1.0 5.4618e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 56 0 0 0 0 56 0 0 0 0 0 > VecView 2 1.0 7.6981e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 1 0 0 0 0 1 0 0 0 0 0 > VecMDot 1 1.0 1.7635e-04 1.0 3.07e+05 1.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 1742 > VecNorm 3 1.0 5.4832e-04 1.0 9.22e+05 1.0 0.0e+00 0.0e+00 > 0.0e+00 0 1 0 0 0 0 1 0 0 0 1681 > VecScale 1 1.0 1.5216e-04 1.0 1.54e+05 1.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 1009 > VecCopy 1 1.0 1.5406e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > VecSet 21 1.0 7.1106e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > VecAYPX 1 1.0 1.9940e-04 1.0 1.54e+05 1.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 770 > VecWAXPY 1 1.0 7.8802e-04 1.0 1.54e+05 1.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 195 > KSPSetUp 2 1.0 6.8995e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > PCSetUp 2 1.0 1.3190e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 14 0 0 0 0 14 0 0 0 0 0 > PCApply 1 1.0 2.2728e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > > ------------------------------------------------------------------------------------------------------------------------ > > Memory usage is given in bytes: > > Object Type Creations Destructions Memory Descendants' Mem. > Reports information only for process 0. > > --- Event Stage 0: Main Stage > > Matrix 3 1 2872 0. > Vector 32 20 17234800 0. > Index Set 4 4 3200 0. > IS L to G Mapping 2 0 0 0. > Vec Scatter 2 0 0 0. > Viewer 6 4 3392 0. > Krylov Solver 2 1 61676 0. > Preconditioner 2 1 1432 0. > > ======================================================================================================================== > Average time to get PetscTime(): 3.36e-08 > #PETSc Option Table entries: > -ksp_converged_reason > -ksp_error_if_not_converged > -ksp_monitor_singular_value > -ksp_monitor_true_residual > -log_view > -pc_hypre_boomeramg_agg_nl 4 > -pc_hypre_boomeramg_agg_num_paths 5 > -pc_hypre_boomeramg_coarsen_type HMIS > -pc_hypre_boomeramg_eu_bj true > -pc_hypre_boomeramg_interp_type ext+i > -pc_hypre_boomeramg_P_max 2 > -pc_hypre_boomeramg_relax_type_all sequential-Gauss-Seidel > -pc_hypre_boomeramg_smooth_type Euclid > -pc_hypre_boomeramg_strong_threshold 0.7 > -pc_hypre_boomeramg_truncfactor 0.3 > #End of PETSc Option Table entries > Compiled without FORTRAN kernels > Compiled with full precision matrices (default) > sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 > sizeof(PetscScalar) 8 sizeof(PetscInt) 4 > Configure options: PETSC_ARCH=arch-linux2-c-opt-gcc7-hypre > --with-blaslapack-dir=/opt/lapack/3.8.0-gcc.7.4.0/lib64 --with-debugging=0 > COPTFLAGS=-O3 CXXOPTFLAGS=-O3 FOPTFLAGS=-O3 --with-valgrind=1 > --with-valgrind-dir=/opt/valgrind/3.14.0/ > --prefix=/opt/petsc/3.10.3_gcc-7.4.0 --with-mpi=1 --download-hypre > ----------------------------------------- > Libraries compiled on 2020-06-26 12:35:46 on xxx > Machine characteristics: > Linux-3.10.0-1127.8.2.el7.x86_64-x86_64-with-centos-7.6.1810-Core > Using PETSc directory: /opt/petsc/3.10.3_gcc-7.4.0 > Using PETSc arch: > ----------------------------------------- > > Using C compiler: mpicc -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing > -Wno-unknown-pragmas -fstack-protector -fvisibility=hidden -O3 > Using Fortran compiler: mpif90 -fPIC -Wall -ffree-line-length-0 > -Wno-unused-dummy-argument -O3 > ----------------------------------------- > > Using include paths: -I/opt/petsc/3.10.3_gcc-7.4.0/include > -I/opt/valgrind/3.14.0/include > ----------------------------------------- > > Using C linker: mpicc > Using Fortran linker: mpif90 > Using libraries: -Wl,-rpath,/opt/petsc/3.10.3_gcc-7.4.0/lib > -L/opt/petsc/3.10.3_gcc-7.4.0/lib -lpetsc > -Wl,-rpath,/opt/petsc/3.10.3_gcc-7.4.0/lib > -L/opt/petsc/3.10.3_gcc-7.4.0/lib > -Wl,-rpath,/opt/lapack/3.8.0-gcc.7.4.0/lib64 > -L/opt/lapack/3.8.0-gcc.7.4.0/lib64 > -Wl,-rpath,/opt/openmpi4.0.0-gcc7.4.0_slurm/lib > -L/opt/openmpi4.0.0-gcc7.4.0_slurm/lib > -Wl,-rpath,/opt/gnu/7.4/lib/gcc/x86_64-pc-linux-gnu/7.4.0 > -L/opt/gnu/7.4/lib/gcc/x86_64-pc-linux-gnu/7.4.0 > -Wl,-rpath,/opt/gnu/7.4/lib/gcc -L/opt/gnu/7.4/lib/gcc > -Wl,-rpath,/opt/gnu/7.4/lib64 -L/opt/gnu/7.4/lib64 > -Wl,-rpath,/opt/gnu/7.4/lib -L/opt/gnu/7.4/lib -lHYPRE -llapack -lblas -lm > -lX11 -lstdc++ -ldl -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh > -lmpi -lgfortran -lm -lgfortran -lm -lgcc_s -lquadmath -lpthread -lstdc++ > -ldl > ----------------------------------------- > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From pierre.jolivet at enseeiht.fr Tue Jun 30 08:22:46 2020 From: pierre.jolivet at enseeiht.fr (Pierre Jolivet) Date: Tue, 30 Jun 2020 15:22:46 +0200 Subject: [petsc-users] PETSc and Windows 10 In-Reply-To: References: <32B37EFD-DDFA-41AE-ACC7-CB5E6AE57996@enseeiht.fr> <0558CCF7-8203-4520-8673-3B0945C43233@enseeiht.fr> <20AB9F7A-4471-498B-9A78-AA48163900CB@enseeiht.fr> <94A8E366-2361-409E-84CE-B90EA54D9AF9@enseeiht.fr> <849E7E3B-8B54-4B40-A2B7-8DD5B55428B3@enseeiht.fr> Message-ID: Please use the 3.13.2 tarball, this was fixed by Satish in the previous commit I already linked (https://gitlab.com/petsc/petsc/-/commit/2cd8068296b34e127f055bb32f556e3599f17523 ). (If you want FreeFEM to do the dirty work for you, just switch to the develop branch, and redo ?make petsc-slepc?) But I think you?ve got everything you need now for a smooth compilation :) Thanks, Pierre > On 30 Jun 2020, at 3:09 PM, Paolo Lampitella wrote: > > Dear Pierre, > > thanks for the fast response. Unfortunately it still fails, but now in the configure of ScaLAPACK > (which means that it went ok for slepc, tetgen, metis, parmetis, ptscotch, superlu and suitesparse). > > The way I applied the modification is by manually editing the Makefile in the 3rdparty/ff-petsc folder, adding -fallow-invalid-boz to both CFLAGS and FFLAGS (this entry added by me). Then executed make petsc-slepc. > > As my project is much less ambitious, I have a good feeling that I will be able to use your Makefile successfully, but as I am kind of slow I tought that it would have been useful for you to know. The configure.log is attached. This time the error is: > > Rank mismatch between actual argument at (1) and actual argument at (2) (scalar and rank-1) > > in subroutine pclarf.f of ScaLAPACK. > > However, before attampting with my project, I have few questions about your Makefile, in particular this piece: > > --with-mpi-lib=/c/Windows/System32/msmpi.dll --with-mpi-include=/home/paolo/FreeFem-sources/3rdparty/include/msmpi --with-mpiexec="/C/Program\ Files/Microsoft\ MPI/Bin/mpiexec" > > I see from MPI.py that I should not use ?--with-mpi-lib/include? if I want to use my now working mpi wrappers. Is this correct? > > Paolo > > Inviato da Posta per Windows 10 > > Da: Pierre Jolivet > Inviato: luned? 29 giugno 2020 21:37 > A: Paolo Lampitella > Cc: Matthew Knepley ; petsc-users > Oggetto: Re: [petsc-users] PETSc and Windows 10 > > I do not give up easily on Windows problems: > 1) that?s around 50% of our (FreeFEM) user-base (and I want them to use PETSc and SLEPc, ofc?) > 2) most people I work with from corporations just have Windows laptops/desktops and I always recommend MSYS because it?s very lightweight and you can pass .exe around > 3) I?ve bothered enough Satish, Jed, and Matt on GitLab to take (at least partially) the blame now when it doesn?t work on MSYS > > That being said, the magic keyword is the added flag FFLAGS="-fallow-invalid-boz" (see, I told you ./configure issues were easier to deal with than the others). > Here you?ll see that everything goes through just fine (sorry, it took me a long time to post this because everything is slow on my VM): > 1) http://jolivet.perso.enseeiht.fr/win10/configure.log > 2) http://jolivet.perso.enseeiht.fr/win10/make.log (both steps #1 and #2 in MSYS terminal, gcc/gfortran 10, MS-MPI see screenshot) > 3) http://jolivet.perso.enseeiht.fr/win10/ex2.txt (Command Prompt, 4 processes + MUMPS, I can send you the .exe if you want to try on your machine) > I just realize that I didn?t generate the Fortran bindings, but you can see I compiled MUMPS and ScaLAPACK, so that shouldn?t be a problem. > Or if there is a problem, we will need to fix this in PETSc. > > I?ll push this added flag to the FreeFEM repo, thanks for reminding me of the brokenness of gcc/gfortran 10 + MS-MPI. > Here is to hoping this won?t affect PETSc ./configure with previous gcc/gfortran version (unlikely, this option is apparently 13-year old https://gcc.gnu.org/bugzilla/show_bug.cgi?id=29471 ) > > Let me know of the next hiccup, if any. > Thanks, > Pierre > > > On 29 Jun 2020, at 8:09 PM, Paolo Lampitella > wrote: > > Dear Pierre, > > thanks again for your time > > I guess there is no way for me to use the toolchain you are using (I don?t remember having any choice on which version of MSYS or GCC I could install) > > Paolo > > Inviato da Posta per Windows 10 > > Da: Pierre Jolivet > Inviato: luned? 29 giugno 2020 20:01 > A: Matthew Knepley > Cc: Paolo Lampitella ; petsc-users > Oggetto: Re: [petsc-users] PETSc and Windows 10 > > > > > > On 29 Jun 2020, at 7:47 PM, Matthew Knepley > wrote: > > On Mon, Jun 29, 2020 at 1:35 PM Paolo Lampitella > wrote: > Dear Pierre, sorry to bother you, but I already have some issues. What I did: > > pacman -R mingw-w64-x86_64-python mingw-w64-x86_64-gdb (is gdb also troublesome?) > Followed points 6 and 7 at https://doc.freefem.org/introduction/installation.html#compilation-on-windows > I first got a warning on the configure at point 6, as ?disable-hips is not recognized. Then, on make ?petsc-slepc? of point 7 (no SUDO=sudo flag was necessary) I got to this point: > > tar xzf ../pkg/petsc-lite-3.13.0.tar.gz > patch -p1 < petsc-suitesparse.patch > patching file petsc-3.13.0/config/BuildSystem/config/packages/SuiteSparse.py > touch petsc-3.13.0/tag-tar > cd petsc-3.13.0 && ./configure MAKEFLAGS='' \ > --prefix=/home/paolo/freefem/ff-petsc//r \ > --with-debugging=0 COPTFLAGS='-O3 -mtune=generic' CXXOPTFLAGS='-O3 -mtune=generic' FOPTFLAGS='-O3 -mtune=generic' --with-cxx-dialect=C++11 --with-ssl=0 --with-x=0 --with-fortran-bindings=0 --with-shared-libraries=0 --with-cc='gcc' --with-cxx='g++' --with-fc='gfortran' CXXFLAGS='-fno-stack-protector' CFLAGS='-fno-stack-protector' --with-scalar-type=real --with-mpi-lib='/c/Windows/System32/msmpi.dll' --with-mpi-include='/home/paolo/FreeFem-sources/3rdparty/include/msmpi' --with-mpiexec='/C/Program\ Files/Microsoft\ MPI/Bin/mpiexec' --with-blaslapack-include='' --with-blaslapack-lib='/mingw64/bin/libopenblas.dll' --download-scalapack --download-metis --download-ptscotch --download-mumps --download-hypre --download-parmetis --download-superlu --download-suitesparse --download-tetgen --download-slepc '--download-metis-cmake-arguments=-G "MSYS Makefiles"' '--download-parmetis-cmake-arguments=-G "MSYS Makefiles"' '--download-superlu-cmake-arguments=-G "MSYS Makefiles"' '--download-hypre-configure-arguments=--build=x86_64-linux-gnu --host=x86_64-linux-gnu' PETSC_ARCH=fr > =============================================================================== > Configuring PETSc to compile on your system > =============================================================================== > TESTING: FortranMPICheck from config.packages.MPI(config/BuildSystem/config/pack******************************************************************************* > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): > ------------------------------------------------------------------------------- > Fortran error! mpi_init() could not be located! > ******************************************************************************* > > make: *** [Makefile:210: petsc-3.13.0/tag-conf-real] Errore 1 > > Note that I didn?t add anything to any PATH variable, because this is not mentioned in your documentation. > > On a side note, this is the same error I got when trying to build PETSc in Cygwin with the default OpenMPI available in Cygwin. > > I am attaching the configure.log? it seems to me that the error comes from the configure trying to include the mpif.h in your folder and not using the -fallow-invalid-boz flag that I had to use, for example, to compile mpi.f90 into mpi.mod > > But I?m not sure why this is happening > > Pierre, > > Could this be due to gcc 10? > > Sorry, I?m slow. You are right. Our workers use gcc 9, everything is fine, but I see on my VM which I updated that I use gcc 10 and had to disable Fortran, I guess the MUMPS run I showcased was with a prior PETSc build. > I?ll try to resolve this and will keep you posted. > They really caught a lot of people off guard with gfortran 10? > > Thanks, > Pierre > > > > Executing: gfortran -c -o /tmp/petsc-ur0cff6a/config.libraries/conftest.o -I/tmp/petsc-ur0cff6a/config.compilers -I/tmp/petsc-ur0cff6a/config.setCompilers -I/tmp/petsc-ur0cff6a/config.compilersFortran -I/tmp/petsc-ur0cff6a/config.libraries -Wall -ffree-line-length-0 -Wno-unused-dummy-argument -O3 -mtune=generic -I/home/paolo/FreeFem-sources/3rdparty/include/msmpi /tmp/petsc-ur0cff6a/config.libraries/conftest.F90 > Possible ERROR while running compiler: exit code 1 > stderr: > C:/msys64/home/paolo/FreeFem-sources/3rdparty/include/msmpi/mpif.h:227:36: > > 227 | PARAMETER (MPI_DATATYPE_NULL=z'0c000000') > | 1 > Error: BOZ literal constant at (1) is neither a data-stmt-constant nor an actual argument to INT, REAL, DBLE, or CMPLX intrinsic function [see '-fno-allow-invalid-boz'] > C:/msys64/home/paolo/FreeFem-sources/3rdparty/include/msmpi/mpif.h:303:27: > > 303 | PARAMETER (MPI_CHAR=z'4c000101') > | 1 > Error: BOZ literal constant at (1) is neither a data-stmt-constant nor an actual argument to INT, REAL, DBLE, or CMPLX intrinsic function [see '-fno-allow-invalid-boz'] > C:/msys64/home/paolo/FreeFem-sources/3rdparty/include/msmpi/mpif.h:305:36: > > 305 | PARAMETER (MPI_UNSIGNED_CHAR=z'4c000102') > | 1 > > Thanks, > > Matt > > Thanks > > Paolo > > Inviato da Posta per Windows 10 > > Da: Pierre Jolivet > Inviato: luned? 29 giugno 2020 18:34 > A: Paolo Lampitella > Cc: Satish Balay ; petsc-users > Oggetto: Re: [petsc-users] PETSc and Windows 10 > > > > On 29 Jun 2020, at 6:27 PM, Paolo Lampitella > wrote: > > I think I made the first step of having mingw64 from msys2 working with ms-mpi. > > I found that the issue I was having was related to: > > https://gcc.gnu.org/bugzilla/show_bug.cgi?id=91556 > > and, probably (but impossible to check now), I was using an msys2 and/or mingw mpi package before this fix: > > https://github.com/msys2/MINGW-packages/commit/11b4cff3d2ec7411037b692b0ad5a9f3e9b9978d#diff-eac59989e3096be97d940c8f47b50fba > > Admittedly, I never used gcc 10 before on any machine. Still, I feel that reporting that sort of error in that way is, > at least, misleading (I would have preferred the initial implementation as mentioned in the gcc bug track). > > A second thing that I was not used to, and made me more uncertain of the procedure I was following, is having to compile myself the mpi module. There are several version of this out there, but I decided to stick with this one: > > https://www.scivision.dev/windows-mpi-msys2/ > > even if there seems to be no need to include -fno-range-check and the current mpi.f90 version is different from the mpif.h as reported here: > > https://github.com/microsoft/Microsoft-MPI/issues/33 > > which, to me, are both signs of lack of attention on the fortran side by those that maintain this thing. > > In summary, this is the procedure I followed so far (on a 64 bit machine with Windows 10): > > Install MSYS2 from https://www.msys2.org/ and just follow the install wizard > Open the MSYS2 terminal and execute: pacman -Syuu > Close the terminal when asked and reopen it > Keep executing ?pacman -Syuu? until nothing else needs to be updated > Close the MSYS2 terminal and reopen it (I guess because was in paranoid mode), then install packages with: > > pacman -S base-devel git gcc gcc-fortran bsdcpio lndir pax-git unzip > pacman -S mingw-w64-x86_64-toolchain > pacman -S mingw-w64-x86_64-msmpi > pacman -S mingw-w64-x86_64-cmake > pacman -S mingw-w64-x86_64-freeglut > pacman -S mingw-w64-x86_64-gsl > pacman -S mingw-w64-x86_64-libmicroutils > pacman -S mingw-w64-x86_64-hdf5 > pacman -S mingw-w64-x86_64-openblas > pacman -S mingw-w64-x86_64-arpack > pacman -S mingw-w64-x86_64-jq > > This set should include all the libraries mentioned by Pierre and/or used by his Jenkins, as the final scope here is to have PETSc and dependencies working. But I think that for pure MPI one could stop to msmpi (even, maybe, just install msmpi and have the dependencies figured out by pacman). Honestly, I don?t remember the exact order I used to install the packages, but this should not affect things. Also, as I was still in paranoid mode, I kept executing ?pacman -Syuu? after each package was installed. After this, close the MSYS2 terminal. > > Open the MINGW64 terminal and create the .mod file out of the mpi.f90 file, as mentioned here https://www.scivision.dev/windows-mpi-msys2/ , with: > > cd /mingw64/include > gfortran mpif90 -c -fno-range-check -fallow-invalid-boz > > Ah, yes, that?s new to gfortran 10 (we use gfortran 9 on our workers), which is now what?s ship with MSYS2 (we haven?t updated yet). Sorry that I forgot about that. > > This is needed to ?USE mpi? (as opposed to INCLUDE ?mpif.h?) > > Install the latest MS-MPI (both sdk and setup) from https://www.microsoft.com/en-us/download/details.aspx?id=100593 > > At this point I?ve been able to compile (using the MINGW64 terminal) different mpi test programs and they run as expected in the classical Windows prompt. I added this function to my .bashrc in MSYS2 in order to easily copy the required dependencies out of MSYS: > > function copydep() { ldd $1 | grep "=> /$2" | awk '{print $3}' | xargs -I '{}' cp -v '{}' .; } > > which can be used, with the MINGW64 terminal, by navigating to the folder where the final executable, say, my.exe, resides (even if under a Windows path) and executing: > > copydep my.exe mingw64 > > This, of course, must be done before actually trying to execute the .exe in the windows cmd prompt. > > Hopefully, I should now be able to follow Pierre?s instructions for PETSc (but first I wanna give a try to the system python before removing it) > > Looks like the hard part is over. It?s usually easier to deal with ./configure issues. > If you have weird errors like ?incomplete Cygwin install? or whatever, this is the kind of issues I was referring to earlier. > In that case, what I?d suggest is just, as before: > pacman -R mingw-w64-x86_64-python mingw-w64-x86_64-gdb > pacman -S python > > Thanks, > Pierre > > Thanks > > Paolo > > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mukkundsunjii at gmail.com Tue Jun 30 11:55:18 2020 From: mukkundsunjii at gmail.com (MUKKUND SUNJII) Date: Tue, 30 Jun 2020 18:55:18 +0200 Subject: [petsc-users] Regarding Adding to ex11.c Message-ID: <3F4CEA70-F9B3-4497-B931-BBB534F3A27B@gmail.com> Greetings, I am a master?s student in TU Delft, The Netherlands, and I am carrying out my thesis using the adaptive mesh refinement solver provided in ./ts/tutorials/ex11.c. In the process of my work, I have added a HLL Riemann Solver for the shallow water equation. I had already tested and validated the Riemann Solver using several cases (I am happy to provide a copy of my thesis report). It would indeed be nice to see my work reflected in the current version of ex11.c as an addition to its capability. Furthermore, I had also added a Riemann Solver that contains topography terms and is well-balanced*. The caveat is that I see that there some variations in the well-balancedness when choosing between p4est and Plex DM?s. Perhaps, this could be looked at and fixed in the future. The purpose of this email is to propose the addition of the HLL Riemann Solver for the shallow water equations to ex11.c and perhaps later explore the possibility of adding the Riemann Solver with the bathymetry terms. I have already gone through the CI process instituted by PETSc (previously integrated ./ts/tutorials/ex54.c into the repository with the valuable assistance of Mr. Satish Balay). Hence, I would be more or less familiar with the process. I kindly ask you provide any feedback on this proposition. Regards, Mukkund Sunjii From knepley at gmail.com Tue Jun 30 12:16:59 2020 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 30 Jun 2020 13:16:59 -0400 Subject: [petsc-users] Regarding Adding to ex11.c In-Reply-To: <3F4CEA70-F9B3-4497-B931-BBB534F3A27B@gmail.com> References: <3F4CEA70-F9B3-4497-B931-BBB534F3A27B@gmail.com> Message-ID: On Tue, Jun 30, 2020 at 12:56 PM MUKKUND SUNJII wrote: > Greetings, > > I am a master?s student in TU Delft, The Netherlands, and I am carrying > out my thesis using the adaptive mesh refinement solver provided in > ./ts/tutorials/ex11.c. > > In the process of my work, I have added a HLL Riemann Solver for the > shallow water equation. I had already tested and validated the Riemann > Solver using several cases (I am happy to provide a copy of my thesis > report). It would indeed be nice to see my work reflected in the current > version of ex11.c as an addition to its capability. > > Furthermore, I had also added a Riemann Solver that contains topography > terms and is well-balanced*. The caveat is that I see that there some > variations in the well-balancedness when choosing between p4est and Plex > DM?s. Perhaps, this could be looked at and fixed in the future. > > The purpose of this email is to propose the addition of the HLL Riemann > Solver for the shallow water equations to ex11.c and perhaps later explore > the possibility of adding the Riemann Solver with the bathymetry terms. > > I have already gone through the CI process instituted by PETSc (previously > integrated ./ts/tutorials/ex54.c into the repository with the valuable > assistance of Mr. Satish Balay). Hence, I would be more or less familiar > with the process. > > I kindly ask you provide any feedback on this proposition. > Yes, this sounds great to me. Thanks, Matt > Regards, > > Mukkund Sunjii -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From mukkundsunjii at gmail.com Tue Jun 30 16:10:10 2020 From: mukkundsunjii at gmail.com (MUKKUND SUNJII) Date: Tue, 30 Jun 2020 23:10:10 +0200 Subject: [petsc-users] Regarding Adding to ex11.c In-Reply-To: References: <3F4CEA70-F9B3-4497-B931-BBB534F3A27B@gmail.com> Message-ID: <9537D2A9-3DE8-4BF5-BCCA-45185554B971@gmail.com> Greetings, Thank you for your message. Excellent! Does it suffice to ask for the merge request once I made the necessary additions to ex11.c? Regards, Mukkund > On 30 Jun 2020, at 19:16, Matthew Knepley wrote: > > On Tue, Jun 30, 2020 at 12:56 PM MUKKUND SUNJII > wrote: > Greetings, > > I am a master?s student in TU Delft, The Netherlands, and I am carrying out my thesis using the adaptive mesh refinement solver provided in ./ts/tutorials/ex11.c. > > In the process of my work, I have added a HLL Riemann Solver for the shallow water equation. I had already tested and validated the Riemann Solver using several cases (I am happy to provide a copy of my thesis report). It would indeed be nice to see my work reflected in the current version of ex11.c as an addition to its capability. > > Furthermore, I had also added a Riemann Solver that contains topography terms and is well-balanced*. The caveat is that I see that there some variations in the well-balancedness when choosing between p4est and Plex DM?s. Perhaps, this could be looked at and fixed in the future. > > The purpose of this email is to propose the addition of the HLL Riemann Solver for the shallow water equations to ex11.c and perhaps later explore the possibility of adding the Riemann Solver with the bathymetry terms. > > I have already gone through the CI process instituted by PETSc (previously integrated ./ts/tutorials/ex54.c into the repository with the valuable assistance of Mr. Satish Balay). Hence, I would be more or less familiar with the process. > > I kindly ask you provide any feedback on this proposition. > > Yes, this sounds great to me. > > Thanks, > > Matt > > Regards, > > Mukkund Sunjii > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Tue Jun 30 16:55:02 2020 From: mfadams at lbl.gov (Mark Adams) Date: Tue, 30 Jun 2020 17:55:02 -0400 Subject: [petsc-users] Regarding Adding to ex11.c In-Reply-To: <9537D2A9-3DE8-4BF5-BCCA-45185554B971@gmail.com> References: <3F4CEA70-F9B3-4497-B931-BBB534F3A27B@gmail.com> <9537D2A9-3DE8-4BF5-BCCA-45185554B971@gmail.com> Message-ID: You want to make an MR (check to delete branch after merge and squash commits), you can mark as WIP, run the pipeline and after you get it to run clean and merge any comments, umark WIP. If it does not get merged in a few days you could ask Satish. Mark On Tue, Jun 30, 2020 at 5:12 PM MUKKUND SUNJII wrote: > Greetings, > > Thank you for your message. > > Excellent! Does it suffice to ask for the merge request once I made the > necessary additions to ex11.c? > > Regards, > > Mukkund > > On 30 Jun 2020, at 19:16, Matthew Knepley wrote: > > On Tue, Jun 30, 2020 at 12:56 PM MUKKUND SUNJII > wrote: > >> Greetings, >> >> I am a master?s student in TU Delft, The Netherlands, and I am carrying >> out my thesis using the adaptive mesh refinement solver provided in >> ./ts/tutorials/ex11.c. >> >> In the process of my work, I have added a HLL Riemann Solver for the >> shallow water equation. I had already tested and validated the Riemann >> Solver using several cases (I am happy to provide a copy of my thesis >> report). It would indeed be nice to see my work reflected in the current >> version of ex11.c as an addition to its capability. >> >> Furthermore, I had also added a Riemann Solver that contains topography >> terms and is well-balanced*. The caveat is that I see that there some >> variations in the well-balancedness when choosing between p4est and Plex >> DM?s. Perhaps, this could be looked at and fixed in the future. >> >> The purpose of this email is to propose the addition of the HLL Riemann >> Solver for the shallow water equations to ex11.c and perhaps later explore >> the possibility of adding the Riemann Solver with the bathymetry terms. >> >> I have already gone through the CI process instituted by PETSc >> (previously integrated ./ts/tutorials/ex54.c into the repository with the >> valuable assistance of Mr. Satish Balay). Hence, I would be more or less >> familiar with the process. >> >> I kindly ask you provide any feedback on this proposition. >> > > Yes, this sounds great to me. > > Thanks, > > Matt > > >> Regards, >> >> Mukkund Sunjii > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: