From daniel.stone at opengosim.com Wed May 1 03:37:26 2024 From: daniel.stone at opengosim.com (Daniel Stone) Date: Wed, 1 May 2024 09:37:26 +0100 Subject: [petsc-users] Spack build and ptscotch In-Reply-To: References: Message-ID: Hi Satish, Many thanks for the interesting response. I'll see if I can figure out something useful to do for our goals (a reliable spack package for our simulator, which is dependent on bringing in the exact right petsc build as dependency), but as you say, maintaining these things and accounting for everything is a lot harder than it looks. Time constraints from other projects might limit how much more tinkering I can do, but we have a better idea of the issues we face now. Daniel On Wed, Apr 24, 2024 at 4:25?PM Satish Balay wrote: > This is the complexity with maintaining dependencies (and dependencies > of dependencies), and different build systems > > - Its not easy to keep the "defaults" in both builds exactly the same. > - And its not easy to expose all "variants" or keep the same variants in > both builds. > - And each pkg has its own issues that prevents some combinations to > work or not [or tested combinations vs untested]. > > This e-mail query has multiple things: > > - understand "why" the current impl of [spack, petsc] build tools are the > way they are. > - if they can be improved > - and build use cases that you need working > - [and subsequently your code working] > > Addressing them all is not easy - so lets stick with what you need to make > progress. > > For one - we recommend using latest petsc version [i.e 3.21 - not 3.19] - > any fixes we have will address the current release. > > > - spack: ptscotch will always be built without parmetis wrappers, can't > turn on > > diff --git a/var/spack/repos/builtin/packages/petsc/package.py > b/var/spack/repos/builtin/packages/petsc/package.py > index b7b1d86b15..ae27ba4c4e 100644 > --- a/var/spack/repos/builtin/packages/petsc/package.py > +++ b/var/spack/repos/builtin/packages/petsc/package.py > @@ -268,9 +268,7 @@ def check_fortran_compiler(self): > depends_on("metis at 5:~int64", when="@3.8:+metis~int64") > depends_on("metis at 5:+int64", when="@3.8:+metis+int64") > > - # PTScotch: Currently disable Parmetis wrapper, this means > - # nested disection won't be available thought PTScotch > - depends_on("scotch+esmumps~metis+mpi", when="+ptscotch") > + depends_on("scotch+esmumps+mpi", when="+ptscotch") > depends_on("scotch+int64", when="+ptscotch+int64") > > depends_on("hdf5@:1.10+mpi", when="@:3.12+hdf5+mpi") > > Now you can try: > > spack install petsc~metis+ptscotch ^scotch+metis > vs > spack install petsc~metis+ptscotch ^scotch~metis [~metis is the default > for scotch] > > Note the following comment in > spack/var/spack/repos/builtin/packages/scotch/package.py > > >>>> > # Vendored dependency of METIS/ParMETIS conflicts with standard > # installations > conflicts("metis", when="+metis") > conflicts("parmetis", when="+metis") > <<<<< > > > - classical: ptscotch will always be built with parmetis wrappers, can't > seem to turn off > > Looks like spack uses cmake build of ptscotch. PETSc uses Makefile > interface. It likely doesn't support turning off metis wrappers [without > hacks]. > > So you might either need to hack scotch build via petsc - or just install > it separately - and use it with petsc. > > I see an effort to migrate scotch build in petsc to cmake > > https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/merge_requests/7242/__;!!G_uCfscf7eWS!axLWsOWdCnVQSjIurDkWvFmG4riOizNNPbVM78TQoVScHx7ERMUENiQ-VW2Lh5e83QHhKcA7-HO0nDJ_hTez8JffqQz8h1I$ > https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/merge_requests/7495/__;!!G_uCfscf7eWS!axLWsOWdCnVQSjIurDkWvFmG4riOizNNPbVM78TQoVScHx7ERMUENiQ-VW2Lh5e83QHhKcA7-HO0nDJ_hTez8JffgEb5IwI$ > > Satish > > On Wed, 24 Apr 2024, Daniel Stone wrote: > > > Hi PETSc community, > > > > I've been looking at using Spack to build PETSc, in particular I need to > > disable the default metis/parmetis dependencies and use PTScotch instead, > > for our software. > > I've had quite a bit of trouble with this - it seems like something in > the > > resulting build of our simulator ends up badly optimised and an mpi > > bottleneck, when I build against > > PETSc built with Spack. > > > > I've been trying to track this down, and noticed this in the PETSc Spack > > build recipe: > > > > # PTScotch: Currently disable Parmetis wrapper, this means > > # nested disection won't be available thought PTScotch > > depends_on("scotch+esmumps~metis+mpi", when="+ptscotch") > > depends_on("scotch+int64", when="+ptscotch+int64") > > > > > > Sure enough - when I compare the build with Spack and a traditional build > > with ./configure etc, I see that, in the traditional build, Scotch is > > always built with the parmetis wrapper, > > but not in the Scotch build. In fact, I'm not sure how to turn off the > > parmetis wrapper option for scotch, in the case of a traditional build > > (i.e. there doesn't seem to be a flag in the > > configure script for it) - which would be a very useful test for me (I > can > > of course do similar experiments by doing a classical build of petsc > > against ptscotch built separately without the > > wrappers, etc - will try that). > > > > Does anyone know why the parmetis wrapper is always disabled in the spack > > build options? Is there something about Spack that would prevent it from > > working? It's notable - but I might > > be missing it - that there's no warning that there's a difference in the > > way ptscotch is built between the spack and classical builds: > > - classical: ptscotch will always be built with parmetis wrappers, can't > > seem to turn off > > - spack: ptscotch will always be built without parmetis wrappers, can't > > turn on > > > > Any insight at all would be great, I'm new to Spack and am not super > > familiar with the logic that goes into setting up builds for the system. > > > > Here is the kind of command I give to Spack for PETSc builds, which may > > well be less than ideal: > > > > spack install petsc at 3.19.1 ~metis +ptscotch ^hdf5 +fortran +hl > > > > Separate tiny note: when building with hdf5, I have to ensure that the > > fortran flag is set for it, as above. There's a fortran flag for the > petsc > > module, default true, and a fortran flag for the hdf5 > > module, default false. A naive user (i.e. me), will see the fortran flag > > for the petsc module, and assume that all dependencies will > correspondingly > > be built with fortran capability - then see that > > hdf5.mod is missing when trying to build their software against petsc. > It's > > the old "did you forget --with-hdf5-fortran-bindings?" issue, resurrected > > for a new build system. > > > > Thanks, > > > > Daniel > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From FERRANJ2 at my.erau.edu Wed May 1 18:33:57 2024 From: FERRANJ2 at my.erau.edu (Ferrand, Jesus A.) Date: Wed, 1 May 2024 23:33:57 +0000 Subject: [petsc-users] Is there anything like a "DMPlexSetCones()" ? Message-ID: Dear PETSc team: For a project that I'm working on, I need to manually build a DMPlex. >From studying the source code of the various APIs in which the plex is built from some supported file format, I get that the workflow is this: 1. DMPlexSetChart() <-- Input nCells + nVerts 2. DMPlexSetConeSize() <-- Input ConeSize for each point in [0,nCells) 3. DMSetUp() ? Allocates memory internally. 4. DMPlexGetCones() --> Gives you the memory onto which to write the cell connectivity. 5. *Write connectivity* 6. DMPlexReorderCell() <-- For each point in [0,nCells) I'm in a situation where the memory given by step (4) is available a-priori. I was hoping to skip steps 2, 3 , and 4 with something like a "DMPlexSetCones()", but such an API does not exist. My current workaround is to implement steps 2 through 4 as always and have double the memory allocated in the interim (my instance + DM's internal instance). I was thinking of looking for the name of the struct member and = it to my memory, but I can't overcome the flag check in DMSetUp() during later calls to DMPlexGetCones() or DMPlexGetTransitiveClosure(). Sincerely: J.A. Ferrand Embry-Riddle Aeronautical University - Daytona Beach - FL Ph.D. Candidate, Aerospace Engineering M.Sc. Aerospace Engineering B.Sc. Aerospace Engineering B.Sc. Computational Mathematics Phone: (386)-843-1829 Email(s): ferranj2 at my.erau.edu jesus.ferrand at gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed May 1 19:07:07 2024 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 1 May 2024 16:07:07 -0800 Subject: [petsc-users] Is there anything like a "DMPlexSetCones()" ? In-Reply-To: References: Message-ID: On Wed, May 1, 2024 at 3:34?PM Ferrand, Jesus A. wrote: > Dear PETSc team: For a project that I'm working on, I need to manually > build a DMPlex. From studying the source code of the various APIs in which > the plex is built from some supported file format, I get that the workflow > is this: DMPlexSetChart() > ZjQcmQRYFpfptBannerStart > This Message Is From an External Sender > This message came from outside your organization. > > ZjQcmQRYFpfptBannerEnd > Dear PETSc team: > > For a project that I'm working on, I need to manually build a DMPlex. > From studying the source code of the various APIs in which the plex is > built from some supported file format, I get that the workflow is this: > > > 1. DMPlexSetChart() <-- Input nCells + nVerts > 2. DMPlexSetConeSize() <-- Input ConeSize for each point in [0,nCells) > 3. DMSetUp() ? Allocates memory internally. > 4. DMPlexGetCones() --> Gives you the memory onto which to write the > cell connectivity. > 5. *Write connectivity* > 6. DMPlexReorderCell() <-- For each point in [0,nCells) > > > I'm in a situation where the memory given by step (4) is available > a-priori. > I was hoping to skip steps 2, 3 , and 4 with something like a > "DMPlexSetCones()", but such an API does not exist. > My current workaround is to implement steps 2 through 4 as always and have > double the memory allocated in the interim (my instance + DM's internal > instance). > I was thinking of looking for the name of the struct member and = it to my > memory, but I can't overcome the flag check in DMSetUp() during later calls > to DMPlexGetCones() or DMPlexGetTransitiveClosure(). > I do not understand the "flag check". What is that? We could make a DMPlexSetCones(), but as you point out, the workflow for DMSetUp() would have to change. Where does the memory come from for your connectivity? Thanks, Matt > > Sincerely: > > *J.A. Ferrand* > > Embry-Riddle Aeronautical University - Daytona Beach - FL > Ph.D. Candidate, Aerospace Engineering > > M.Sc. Aerospace Engineering > > B.Sc. Aerospace Engineering > > B.Sc. Computational Mathematics > > > *Phone:* (386)-843-1829 > > *Email(s):* ferranj2 at my.erau.edu > > jesus.ferrand at gmail.com > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!ZR1_fNBzyO9hgjbXCkFEbX3CMsLXVmoeah8FCeMOb9AMhbszC-gLvEU6mbq67TLMx9kDRY2b--nt1pUHr_Np$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From FERRANJ2 at my.erau.edu Wed May 1 19:17:15 2024 From: FERRANJ2 at my.erau.edu (Ferrand, Jesus A.) Date: Thu, 2 May 2024 00:17:15 +0000 Subject: [petsc-users] [EXTERNAL] Re: Is there anything like a "DMPlexSetCones()" ? In-Reply-To: References: Message-ID: Matt: "I do not understand the "flag check". What is that?" ??????My bad, I should have referred to the "dm->setupcalled". ??????I believe this PetscBool is checked by the other DM (not just DMPlex) APIs. ??????The subsequent checks for dm->setupcalled == PETSC_TRUE is what I meant to say. ??????Here's a copy of DMSetUp(). PetscErrorCode DMSetUp(DM dm) 817: { 818: PetscFunctionBegin; 820: if (dm->setupcalled) PetscFunctionReturn(PETSC_SUCCESS); 821: PetscTryTypeMethod(dm, setup); 822: dm->setupcalled = PETSC_TRUE; 823: PetscFunctionReturn(PETSC_SUCCESS); 824: } "We could make a DMPlexSetCones(), but as you point out, the workflow for DMSetUp() would have to change. Where does the memory come from for your connectivity?" ??????It's a memory that I myself allocate based on a file's contents. ??????The nature of the I/O makes it so that I need to read the connectivity before I get a semblance of buffer sizes. ??????Otherwise, I would stick to the tried and tested way. Also, when replying to the PETSc developers, users must reply to petsc-users at mcs.anl.gov and not just to the individual email accounts of the developers, right? ________________________________ From: Matthew Knepley Sent: Wednesday, May 1, 2024 8:07 PM To: Ferrand, Jesus A. Cc: petsc-users at mcs.anl.gov Subject: [EXTERNAL] Re: [petsc-users] Is there anything like a "DMPlexSetCones()" ? CAUTION: This email originated outside of Embry-Riddle Aeronautical University. Do not click links or open attachments unless you recognize the sender and know the content is safe. On Wed, May 1, 2024 at 3:34?PM Ferrand, Jesus A. > wrote: This Message Is From an External Sender This message came from outside your organization. Dear PETSc team: For a project that I'm working on, I need to manually build a DMPlex. >From studying the source code of the various APIs in which the plex is built from some supported file format, I get that the workflow is this: 1. DMPlexSetChart() <-- Input nCells + nVerts 2. DMPlexSetConeSize() <-- Input ConeSize for each point in [0,nCells) 3. DMSetUp() ? Allocates memory internally. 4. DMPlexGetCones() --> Gives you the memory onto which to write the cell connectivity. 5. *Write connectivity* 6. DMPlexReorderCell() <-- For each point in [0,nCells) I'm in a situation where the memory given by step (4) is available a-priori. I was hoping to skip steps 2, 3 , and 4 with something like a "DMPlexSetCones()", but such an API does not exist. My current workaround is to implement steps 2 through 4 as always and have double the memory allocated in the interim (my instance + DM's internal instance). I was thinking of looking for the name of the struct member and = it to my memory, but I can't overcome the flag check in DMSetUp() during later calls to DMPlexGetCones() or DMPlexGetTransitiveClosure(). I do not understand the "flag check". What is that? We could make a DMPlexSetCones(), but as you point out, the workflow for DMSetUp() would have to change. Where does the memory come from for your connectivity? Thanks, Matt Sincerely: J.A. Ferrand Embry-Riddle Aeronautical University - Daytona Beach - FL Ph.D. Candidate, Aerospace Engineering M.Sc. Aerospace Engineering B.Sc. Aerospace Engineering B.Sc. Computational Mathematics Phone: (386)-843-1829 Email(s): ferranj2 at my.erau.edu jesus.ferrand at gmail.com -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!YFXAyokfQwnPYORwZQ6qcxVJHF6uyeDin1NgKfpvqUZafCX_TGDCu_5NYVR63aFhpdQwl9zaJ_woQpUfaXoUdNOo-OU$ ________________________________ From: Matthew Knepley Sent: Wednesday, May 1, 2024 8:07 PM To: Ferrand, Jesus A. Cc: petsc-users at mcs.anl.gov Subject: [EXTERNAL] Re: [petsc-users] Is there anything like a "DMPlexSetCones()" ? CAUTION: This email originated outside of Embry-Riddle Aeronautical University. Do not click links or open attachments unless you recognize the sender and know the content is safe. On Wed, May 1, 2024 at 3:34?PM Ferrand, Jesus A. > wrote: This Message Is From an External Sender This message came from outside your organization. Dear PETSc team: For a project that I'm working on, I need to manually build a DMPlex. >From studying the source code of the various APIs in which the plex is built from some supported file format, I get that the workflow is this: 1. DMPlexSetChart() <-- Input nCells + nVerts 2. DMPlexSetConeSize() <-- Input ConeSize for each point in [0,nCells) 3. DMSetUp() ? Allocates memory internally. 4. DMPlexGetCones() --> Gives you the memory onto which to write the cell connectivity. 5. *Write connectivity* 6. DMPlexReorderCell() <-- For each point in [0,nCells) I'm in a situation where the memory given by step (4) is available a-priori. I was hoping to skip steps 2, 3 , and 4 with something like a "DMPlexSetCones()", but such an API does not exist. My current workaround is to implement steps 2 through 4 as always and have double the memory allocated in the interim (my instance + DM's internal instance). I was thinking of looking for the name of the struct member and = it to my memory, but I can't overcome the flag check in DMSetUp() during later calls to DMPlexGetCones() or DMPlexGetTransitiveClosure(). I do not understand the "flag check". What is that? We could make a DMPlexSetCones(), but as you point out, the workflow for DMSetUp() would have to change. Where does the memory come from for your connectivity? Thanks, Matt Sincerely: J.A. Ferrand Embry-Riddle Aeronautical University - Daytona Beach - FL Ph.D. Candidate, Aerospace Engineering M.Sc. Aerospace Engineering B.Sc. Aerospace Engineering B.Sc. Computational Mathematics Phone: (386)-843-1829 Email(s): ferranj2 at my.erau.edu jesus.ferrand at gmail.com -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!YFXAyokfQwnPYORwZQ6qcxVJHF6uyeDin1NgKfpvqUZafCX_TGDCu_5NYVR63aFhpdQwl9zaJ_woQpUfaXoUdNOo-OU$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From FERRANJ2 at my.erau.edu Wed May 1 19:23:33 2024 From: FERRANJ2 at my.erau.edu (Ferrand, Jesus A.) Date: Thu, 2 May 2024 00:23:33 +0000 Subject: [petsc-users] [EXTERNAL] Re: Is there anything like a "DMPlexSetCones()" ? In-Reply-To: References: Message-ID: Matt: My bad again, I need to clarify something that I just realized doesn't make sense. I said: "The nature of the I/O makes it so that I need to read the connectivity before I get a semblance of buffer sizes." Scratch that, obviously, if I can read the connectivity, I know the buffer sizes. What I should have said is that I cannot know the "chart" that one sets in DMPlexSetChart() a priori like in most cases. I need to determine the chart from the connectivity lists. ________________________________ From: Ferrand, Jesus A. Sent: Wednesday, May 1, 2024 8:17 PM To: Matthew Knepley Cc: petsc-users at mcs.anl.gov Subject: Re: [EXTERNAL] Re: [petsc-users] Is there anything like a "DMPlexSetCones()" ? Matt: "I do not understand the "flag check". What is that?" ??????My bad, I should have referred to the "dm->setupcalled". ??????I believe this PetscBool is checked by the other DM (not just DMPlex) APIs. ??????The subsequent checks for dm->setupcalled == PETSC_TRUE is what I meant to say. ??????Here's a copy of DMSetUp(). PetscErrorCode DMSetUp(DM dm) 817: { 818: PetscFunctionBegin; 820: if (dm->setupcalled) PetscFunctionReturn(PETSC_SUCCESS); 821: PetscTryTypeMethod(dm, setup); 822: dm->setupcalled = PETSC_TRUE; 823: PetscFunctionReturn(PETSC_SUCCESS); 824: } "We could make a DMPlexSetCones(), but as you point out, the workflow for DMSetUp() would have to change. Where does the memory come from for your connectivity?" ??????It's a memory that I myself allocate based on a file's contents. ??????The nature of the I/O makes it so that I need to read the connectivity before I get a semblance of buffer sizes. ??????Otherwise, I would stick to the tried and tested way. Also, when replying to the PETSc developers, users must reply to petsc-users at mcs.anl.gov and not just to the individual email accounts of the developers, right? ________________________________ From: Matthew Knepley Sent: Wednesday, May 1, 2024 8:07 PM To: Ferrand, Jesus A. Cc: petsc-users at mcs.anl.gov Subject: [EXTERNAL] Re: [petsc-users] Is there anything like a "DMPlexSetCones()" ? CAUTION: This email originated outside of Embry-Riddle Aeronautical University. Do not click links or open attachments unless you recognize the sender and know the content is safe. On Wed, May 1, 2024 at 3:34?PM Ferrand, Jesus A. > wrote: This Message Is From an External Sender This message came from outside your organization. Dear PETSc team: For a project that I'm working on, I need to manually build a DMPlex. >From studying the source code of the various APIs in which the plex is built from some supported file format, I get that the workflow is this: 1. DMPlexSetChart() <-- Input nCells + nVerts 2. DMPlexSetConeSize() <-- Input ConeSize for each point in [0,nCells) 3. DMSetUp() ? Allocates memory internally. 4. DMPlexGetCones() --> Gives you the memory onto which to write the cell connectivity. 5. *Write connectivity* 6. DMPlexReorderCell() <-- For each point in [0,nCells) I'm in a situation where the memory given by step (4) is available a-priori. I was hoping to skip steps 2, 3 , and 4 with something like a "DMPlexSetCones()", but such an API does not exist. My current workaround is to implement steps 2 through 4 as always and have double the memory allocated in the interim (my instance + DM's internal instance). I was thinking of looking for the name of the struct member and = it to my memory, but I can't overcome the flag check in DMSetUp() during later calls to DMPlexGetCones() or DMPlexGetTransitiveClosure(). I do not understand the "flag check". What is that? We could make a DMPlexSetCones(), but as you point out, the workflow for DMSetUp() would have to change. Where does the memory come from for your connectivity? Thanks, Matt Sincerely: J.A. Ferrand Embry-Riddle Aeronautical University - Daytona Beach - FL Ph.D. Candidate, Aerospace Engineering M.Sc. Aerospace Engineering B.Sc. Aerospace Engineering B.Sc. Computational Mathematics Phone: (386)-843-1829 Email(s): ferranj2 at my.erau.edu jesus.ferrand at gmail.com -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!Y59qV_AfccorvDZ2Ux5zlODn2plUGK9DDK2MadlAOvbh7uPhFCvfhWkUgJ_4ncaI6RxBeDbsKAPuyB-H6hOIlMqTtdQ$ ________________________________ From: Matthew Knepley Sent: Wednesday, May 1, 2024 8:07 PM To: Ferrand, Jesus A. Cc: petsc-users at mcs.anl.gov Subject: [EXTERNAL] Re: [petsc-users] Is there anything like a "DMPlexSetCones()" ? CAUTION: This email originated outside of Embry-Riddle Aeronautical University. Do not click links or open attachments unless you recognize the sender and know the content is safe. On Wed, May 1, 2024 at 3:34?PM Ferrand, Jesus A. > wrote: This Message Is From an External Sender This message came from outside your organization. Dear PETSc team: For a project that I'm working on, I need to manually build a DMPlex. >From studying the source code of the various APIs in which the plex is built from some supported file format, I get that the workflow is this: 1. DMPlexSetChart() <-- Input nCells + nVerts 2. DMPlexSetConeSize() <-- Input ConeSize for each point in [0,nCells) 3. DMSetUp() ? Allocates memory internally. 4. DMPlexGetCones() --> Gives you the memory onto which to write the cell connectivity. 5. *Write connectivity* 6. DMPlexReorderCell() <-- For each point in [0,nCells) I'm in a situation where the memory given by step (4) is available a-priori. I was hoping to skip steps 2, 3 , and 4 with something like a "DMPlexSetCones()", but such an API does not exist. My current workaround is to implement steps 2 through 4 as always and have double the memory allocated in the interim (my instance + DM's internal instance). I was thinking of looking for the name of the struct member and = it to my memory, but I can't overcome the flag check in DMSetUp() during later calls to DMPlexGetCones() or DMPlexGetTransitiveClosure(). I do not understand the "flag check". What is that? We could make a DMPlexSetCones(), but as you point out, the workflow for DMSetUp() would have to change. Where does the memory come from for your connectivity? Thanks, Matt Sincerely: J.A. Ferrand Embry-Riddle Aeronautical University - Daytona Beach - FL Ph.D. Candidate, Aerospace Engineering M.Sc. Aerospace Engineering B.Sc. Aerospace Engineering B.Sc. Computational Mathematics Phone: (386)-843-1829 Email(s): ferranj2 at my.erau.edu jesus.ferrand at gmail.com -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!Y59qV_AfccorvDZ2Ux5zlODn2plUGK9DDK2MadlAOvbh7uPhFCvfhWkUgJ_4ncaI6RxBeDbsKAPuyB-H6hOIlMqTtdQ$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed May 1 19:52:32 2024 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 1 May 2024 16:52:32 -0800 Subject: [petsc-users] [EXTERNAL] Re: Is there anything like a "DMPlexSetCones()" ? In-Reply-To: References: Message-ID: On Wed, May 1, 2024 at 4:23?PM Ferrand, Jesus A. wrote: > Matt: > > My bad again, I need to clarify something that I just realized doesn't > make sense. > I said: "The nature of the I/O makes it so that I need to read the > connectivity before I get a semblance of buffer sizes." > Scratch that, obviously, if I can read the connectivity, I know the buffer > sizes. > > What I should have said is that I cannot know the "chart" that one sets in > DMPlexSetChart() a priori like in most cases. > I need to determine the chart from the connectivity lists. > You know the number of cells if you can read connectivity. Do you mean that the format does not tell you the number of vertices? Thanks, Matt > ------------------------------ > *From:* Ferrand, Jesus A. > *Sent:* Wednesday, May 1, 2024 8:17 PM > *To:* Matthew Knepley > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [EXTERNAL] Re: [petsc-users] Is there anything like a > "DMPlexSetCones()" ? > > Matt: > > "I do not understand the "flag check". What is that?" > > My bad, I should have referred to the "dm->setupcalled". > I believe this PetscBool is checked by the other DM (not just DMPlex) APIs. > The subsequent checks for dm->setupcalled == PETSC_TRUE is what I meant to > say. > Here's a copy of DMSetUp(). > > *PetscErrorCode DMSetUp (DM dm)* > 817: { > 818: PetscFunctionBegin ; > 820: if (dm->setupcalled) PetscFunctionReturn (PETSC_SUCCESS ); > 821: PetscTryTypeMethod (dm, setup); > 822: dm->setupcalled = PETSC_TRUE ; > 823: PetscFunctionReturn (PETSC_SUCCESS ); > 824: } > > "We could make a DMPlexSetCones(), but as you point out, the workflow for > DMSetUp() would have to change. Where does the memory come from for your > connectivity?" > It's a memory that I myself allocate based on a file's contents. > The nature of the I/O makes it so that I need to read the connectivity > before I get a semblance of buffer sizes. > Otherwise, I would stick to the tried and tested way. > > Also, when replying to the PETSc developers, users must reply to > petsc-users at mcs.anl.gov and not just to the individual email accounts of > the developers, right? > > > > > ------------------------------ > *From:* Matthew Knepley > *Sent:* Wednesday, May 1, 2024 8:07 PM > *To:* Ferrand, Jesus A. > *Cc:* petsc-users at mcs.anl.gov > *Subject:* [EXTERNAL] Re: [petsc-users] Is there anything like a > "DMPlexSetCones()" ? > > *CAUTION:* This email originated outside of Embry-Riddle Aeronautical > University. Do not click links or open attachments unless you recognize the > sender and know the content is safe. > > On Wed, May 1, 2024 at 3:34?PM Ferrand, Jesus A. > wrote: > > This Message Is From an External Sender > This message came from outside your organization. > > Dear PETSc team: > > For a project that I'm working on, I need to manually build a DMPlex. > From studying the source code of the various APIs in which the plex is > built from some supported file format, I get that the workflow is this: > > > 1. DMPlexSetChart() <-- Input nCells + nVerts > 2. DMPlexSetConeSize() <-- Input ConeSize for each point in [0,nCells) > 3. DMSetUp() ? Allocates memory internally. > 4. DMPlexGetCones() --> Gives you the memory onto which to write the > cell connectivity. > 5. *Write connectivity* > 6. DMPlexReorderCell() <-- For each point in [0,nCells) > > > I'm in a situation where the memory given by step (4) is available > a-priori. > I was hoping to skip steps 2, 3 , and 4 with something like a > "DMPlexSetCones()", but such an API does not exist. > My current workaround is to implement steps 2 through 4 as always and have > double the memory allocated in the interim (my instance + DM's internal > instance). > I was thinking of looking for the name of the struct member and = it to my > memory, but I can't overcome the flag check in DMSetUp() during later calls > to DMPlexGetCones() or DMPlexGetTransitiveClosure(). > > > I do not understand the "flag check". What is that? > > We could make a DMPlexSetCones(), but as you point out, the workflow for > DMSetUp() would have to change. Where does the memory come from for your > connectivity? > > Thanks, > > Matt > > > > Sincerely: > > *J.A. Ferrand* > > Embry-Riddle Aeronautical University - Daytona Beach - FL > Ph.D. Candidate, Aerospace Engineering > > M.Sc. Aerospace Engineering > > B.Sc. Aerospace Engineering > > B.Sc. Computational Mathematics > > > *Phone:* (386)-843-1829 > > *Email(s):* ferranj2 at my.erau.edu > > jesus.ferrand at gmail.com > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!dYZsj6bNyjrW3n2ZUHSK0N1eCUS5ylIPHxjjS0SQT-DSlMCOeipzRVmuuasQCL9GHi7tXPQbRftLbI8UMqBN$ > > > ------------------------------ > *From:* Matthew Knepley > *Sent:* Wednesday, May 1, 2024 8:07 PM > *To:* Ferrand, Jesus A. > *Cc:* petsc-users at mcs.anl.gov > *Subject:* [EXTERNAL] Re: [petsc-users] Is there anything like a > "DMPlexSetCones()" ? > > *CAUTION:* This email originated outside of Embry-Riddle Aeronautical > University. Do not click links or open attachments unless you recognize the > sender and know the content is safe. > > On Wed, May 1, 2024 at 3:34?PM Ferrand, Jesus A. > wrote: > > This Message Is From an External Sender > This message came from outside your organization. > > Dear PETSc team: > > For a project that I'm working on, I need to manually build a DMPlex. > From studying the source code of the various APIs in which the plex is > built from some supported file format, I get that the workflow is this: > > > 1. DMPlexSetChart() <-- Input nCells + nVerts > 2. DMPlexSetConeSize() <-- Input ConeSize for each point in [0,nCells) > 3. DMSetUp() ? Allocates memory internally. > 4. DMPlexGetCones() --> Gives you the memory onto which to write the > cell connectivity. > 5. *Write connectivity* > 6. DMPlexReorderCell() <-- For each point in [0,nCells) > > > I'm in a situation where the memory given by step (4) is available > a-priori. > I was hoping to skip steps 2, 3 , and 4 with something like a > "DMPlexSetCones()", but such an API does not exist. > My current workaround is to implement steps 2 through 4 as always and have > double the memory allocated in the interim (my instance + DM's internal > instance). > I was thinking of looking for the name of the struct member and = it to my > memory, but I can't overcome the flag check in DMSetUp() during later calls > to DMPlexGetCones() or DMPlexGetTransitiveClosure(). > > > I do not understand the "flag check". What is that? > > We could make a DMPlexSetCones(), but as you point out, the workflow for > DMSetUp() would have to change. Where does the memory come from for your > connectivity? > > Thanks, > > Matt > > > > Sincerely: > > *J.A. Ferrand* > > Embry-Riddle Aeronautical University - Daytona Beach - FL > Ph.D. Candidate, Aerospace Engineering > > M.Sc. Aerospace Engineering > > B.Sc. Aerospace Engineering > > B.Sc. Computational Mathematics > > > *Phone:* (386)-843-1829 > > *Email(s):* ferranj2 at my.erau.edu > > jesus.ferrand at gmail.com > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!dYZsj6bNyjrW3n2ZUHSK0N1eCUS5ylIPHxjjS0SQT-DSlMCOeipzRVmuuasQCL9GHi7tXPQbRftLbI8UMqBN$ > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!dYZsj6bNyjrW3n2ZUHSK0N1eCUS5ylIPHxjjS0SQT-DSlMCOeipzRVmuuasQCL9GHi7tXPQbRftLbI8UMqBN$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From FERRANJ2 at my.erau.edu Wed May 1 21:30:14 2024 From: FERRANJ2 at my.erau.edu (Ferrand, Jesus A.) Date: Thu, 2 May 2024 02:30:14 +0000 Subject: [petsc-users] [EXTERNAL] Re: Is there anything like a "DMPlexSetCones()" ? In-Reply-To: References: Message-ID: "You know the number of cells if you can read connectivity. Do you mean that the format does not tell you the number of vertices?" Sort of... So, the format does provide the number of vertices, however, due to the way I read the data in parallel I don't know immediately how many local vertices there will be. Which prevents me from knowing the chart a priori. I figured out the number of vertices using PetscHashSetI to determine the number of unique entries in the connectivity list. ________________________________ From: Matthew Knepley Sent: Wednesday, May 1, 2024 8:52 PM To: Ferrand, Jesus A. Cc: petsc-users at mcs.anl.gov Subject: Re: [EXTERNAL] Re: [petsc-users] Is there anything like a "DMPlexSetCones()" ? On Wed, May 1, 2024 at 4:23?PM Ferrand, Jesus A. > wrote: Matt: My bad again, I need to clarify something that I just realized doesn't make sense. I said: "The nature of the I/O makes it so that I need to read the connectivity before I get a semblance of buffer sizes." Scratch that, obviously, if I can read the connectivity, I know the buffer sizes. What I should have said is that I cannot know the "chart" that one sets in DMPlexSetChart() a priori like in most cases. I need to determine the chart from the connectivity lists. You know the number of cells if you can read connectivity. Do you mean that the format does not tell you the number of vertices? Thanks, Matt ________________________________ From: Ferrand, Jesus A. > Sent: Wednesday, May 1, 2024 8:17 PM To: Matthew Knepley > Cc: petsc-users at mcs.anl.gov > Subject: Re: [EXTERNAL] Re: [petsc-users] Is there anything like a "DMPlexSetCones()" ? Matt: "I do not understand the "flag check". What is that?" ??????My bad, I should have referred to the "dm->setupcalled". ??????I believe this PetscBool is checked by the other DM (not just DMPlex) APIs. ??????The subsequent checks for dm->setupcalled == PETSC_TRUE is what I meant to say. ??????Here's a copy of DMSetUp(). PetscErrorCode DMSetUp(DM dm) 817: { 818: PetscFunctionBegin; 820: if (dm->setupcalled) PetscFunctionReturn(PETSC_SUCCESS); 821: PetscTryTypeMethod(dm, setup); 822: dm->setupcalled = PETSC_TRUE; 823: PetscFunctionReturn(PETSC_SUCCESS); 824: } "We could make a DMPlexSetCones(), but as you point out, the workflow for DMSetUp() would have to change. Where does the memory come from for your connectivity?" ??????It's a memory that I myself allocate based on a file's contents. ??????The nature of the I/O makes it so that I need to read the connectivity before I get a semblance of buffer sizes. ??????Otherwise, I would stick to the tried and tested way. Also, when replying to the PETSc developers, users must reply to petsc-users at mcs.anl.gov and not just to the individual email accounts of the developers, right? ________________________________ From: Matthew Knepley > Sent: Wednesday, May 1, 2024 8:07 PM To: Ferrand, Jesus A. > Cc: petsc-users at mcs.anl.gov > Subject: [EXTERNAL] Re: [petsc-users] Is there anything like a "DMPlexSetCones()" ? CAUTION: This email originated outside of Embry-Riddle Aeronautical University. Do not click links or open attachments unless you recognize the sender and know the content is safe. On Wed, May 1, 2024 at 3:34?PM Ferrand, Jesus A. > wrote: This Message Is From an External Sender This message came from outside your organization. Dear PETSc team: For a project that I'm working on, I need to manually build a DMPlex. >From studying the source code of the various APIs in which the plex is built from some supported file format, I get that the workflow is this: 1. DMPlexSetChart() <-- Input nCells + nVerts 2. DMPlexSetConeSize() <-- Input ConeSize for each point in [0,nCells) 3. DMSetUp() ? Allocates memory internally. 4. DMPlexGetCones() --> Gives you the memory onto which to write the cell connectivity. 5. *Write connectivity* 6. DMPlexReorderCell() <-- For each point in [0,nCells) I'm in a situation where the memory given by step (4) is available a-priori. I was hoping to skip steps 2, 3 , and 4 with something like a "DMPlexSetCones()", but such an API does not exist. My current workaround is to implement steps 2 through 4 as always and have double the memory allocated in the interim (my instance + DM's internal instance). I was thinking of looking for the name of the struct member and = it to my memory, but I can't overcome the flag check in DMSetUp() during later calls to DMPlexGetCones() or DMPlexGetTransitiveClosure(). I do not understand the "flag check". What is that? We could make a DMPlexSetCones(), but as you point out, the workflow for DMSetUp() would have to change. Where does the memory come from for your connectivity? Thanks, Matt Sincerely: J.A. Ferrand Embry-Riddle Aeronautical University - Daytona Beach - FL Ph.D. Candidate, Aerospace Engineering M.Sc. Aerospace Engineering B.Sc. Aerospace Engineering B.Sc. Computational Mathematics Phone: (386)-843-1829 Email(s): ferranj2 at my.erau.edu jesus.ferrand at gmail.com -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!bg9QrJhAkKOSNFt840OnuOpmX_IlSabDICy99_9tMp9uABL-2lhCm_LG2Fj8rwdySPMiK7rnfMLSIPxs3jDVQRKJmPA$ ________________________________ From: Matthew Knepley > Sent: Wednesday, May 1, 2024 8:07 PM To: Ferrand, Jesus A. > Cc: petsc-users at mcs.anl.gov > Subject: [EXTERNAL] Re: [petsc-users] Is there anything like a "DMPlexSetCones()" ? CAUTION: This email originated outside of Embry-Riddle Aeronautical University. Do not click links or open attachments unless you recognize the sender and know the content is safe. On Wed, May 1, 2024 at 3:34?PM Ferrand, Jesus A. > wrote: This Message Is From an External Sender This message came from outside your organization. Dear PETSc team: For a project that I'm working on, I need to manually build a DMPlex. >From studying the source code of the various APIs in which the plex is built from some supported file format, I get that the workflow is this: 1. DMPlexSetChart() <-- Input nCells + nVerts 2. DMPlexSetConeSize() <-- Input ConeSize for each point in [0,nCells) 3. DMSetUp() ? Allocates memory internally. 4. DMPlexGetCones() --> Gives you the memory onto which to write the cell connectivity. 5. *Write connectivity* 6. DMPlexReorderCell() <-- For each point in [0,nCells) I'm in a situation where the memory given by step (4) is available a-priori. I was hoping to skip steps 2, 3 , and 4 with something like a "DMPlexSetCones()", but such an API does not exist. My current workaround is to implement steps 2 through 4 as always and have double the memory allocated in the interim (my instance + DM's internal instance). I was thinking of looking for the name of the struct member and = it to my memory, but I can't overcome the flag check in DMSetUp() during later calls to DMPlexGetCones() or DMPlexGetTransitiveClosure(). I do not understand the "flag check". What is that? We could make a DMPlexSetCones(), but as you point out, the workflow for DMSetUp() would have to change. Where does the memory come from for your connectivity? Thanks, Matt Sincerely: J.A. Ferrand Embry-Riddle Aeronautical University - Daytona Beach - FL Ph.D. Candidate, Aerospace Engineering M.Sc. Aerospace Engineering B.Sc. Aerospace Engineering B.Sc. Computational Mathematics Phone: (386)-843-1829 Email(s): ferranj2 at my.erau.edu jesus.ferrand at gmail.com -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!bg9QrJhAkKOSNFt840OnuOpmX_IlSabDICy99_9tMp9uABL-2lhCm_LG2Fj8rwdySPMiK7rnfMLSIPxs3jDVQRKJmPA$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!bg9QrJhAkKOSNFt840OnuOpmX_IlSabDICy99_9tMp9uABL-2lhCm_LG2Fj8rwdySPMiK7rnfMLSIPxs3jDVQRKJmPA$ ________________________________ From: Matthew Knepley Sent: Wednesday, May 1, 2024 8:52 PM To: Ferrand, Jesus A. Cc: petsc-users at mcs.anl.gov Subject: Re: [EXTERNAL] Re: [petsc-users] Is there anything like a "DMPlexSetCones()" ? On Wed, May 1, 2024 at 4:23?PM Ferrand, Jesus A. > wrote: Matt: My bad again, I need to clarify something that I just realized doesn't make sense. I said: "The nature of the I/O makes it so that I need to read the connectivity before I get a semblance of buffer sizes." Scratch that, obviously, if I can read the connectivity, I know the buffer sizes. What I should have said is that I cannot know the "chart" that one sets in DMPlexSetChart() a priori like in most cases. I need to determine the chart from the connectivity lists. You know the number of cells if you can read connectivity. Do you mean that the format does not tell you the number of vertices? Thanks, Matt ________________________________ From: Ferrand, Jesus A. > Sent: Wednesday, May 1, 2024 8:17 PM To: Matthew Knepley > Cc: petsc-users at mcs.anl.gov > Subject: Re: [EXTERNAL] Re: [petsc-users] Is there anything like a "DMPlexSetCones()" ? Matt: "I do not understand the "flag check". What is that?" ??????My bad, I should have referred to the "dm->setupcalled". ??????I believe this PetscBool is checked by the other DM (not just DMPlex) APIs. ??????The subsequent checks for dm->setupcalled == PETSC_TRUE is what I meant to say. ??????Here's a copy of DMSetUp(). PetscErrorCode DMSetUp(DM dm) 817: { 818: PetscFunctionBegin; 820: if (dm->setupcalled) PetscFunctionReturn(PETSC_SUCCESS); 821: PetscTryTypeMethod(dm, setup); 822: dm->setupcalled = PETSC_TRUE; 823: PetscFunctionReturn(PETSC_SUCCESS); 824: } "We could make a DMPlexSetCones(), but as you point out, the workflow for DMSetUp() would have to change. Where does the memory come from for your connectivity?" ??????It's a memory that I myself allocate based on a file's contents. ??????The nature of the I/O makes it so that I need to read the connectivity before I get a semblance of buffer sizes. ??????Otherwise, I would stick to the tried and tested way. Also, when replying to the PETSc developers, users must reply to petsc-users at mcs.anl.gov and not just to the individual email accounts of the developers, right? ________________________________ From: Matthew Knepley > Sent: Wednesday, May 1, 2024 8:07 PM To: Ferrand, Jesus A. > Cc: petsc-users at mcs.anl.gov > Subject: [EXTERNAL] Re: [petsc-users] Is there anything like a "DMPlexSetCones()" ? CAUTION: This email originated outside of Embry-Riddle Aeronautical University. Do not click links or open attachments unless you recognize the sender and know the content is safe. On Wed, May 1, 2024 at 3:34?PM Ferrand, Jesus A. > wrote: This Message Is From an External Sender This message came from outside your organization. Dear PETSc team: For a project that I'm working on, I need to manually build a DMPlex. >From studying the source code of the various APIs in which the plex is built from some supported file format, I get that the workflow is this: 1. DMPlexSetChart() <-- Input nCells + nVerts 2. DMPlexSetConeSize() <-- Input ConeSize for each point in [0,nCells) 3. DMSetUp() ? Allocates memory internally. 4. DMPlexGetCones() --> Gives you the memory onto which to write the cell connectivity. 5. *Write connectivity* 6. DMPlexReorderCell() <-- For each point in [0,nCells) I'm in a situation where the memory given by step (4) is available a-priori. I was hoping to skip steps 2, 3 , and 4 with something like a "DMPlexSetCones()", but such an API does not exist. My current workaround is to implement steps 2 through 4 as always and have double the memory allocated in the interim (my instance + DM's internal instance). I was thinking of looking for the name of the struct member and = it to my memory, but I can't overcome the flag check in DMSetUp() during later calls to DMPlexGetCones() or DMPlexGetTransitiveClosure(). I do not understand the "flag check". What is that? We could make a DMPlexSetCones(), but as you point out, the workflow for DMSetUp() would have to change. Where does the memory come from for your connectivity? Thanks, Matt Sincerely: J.A. Ferrand Embry-Riddle Aeronautical University - Daytona Beach - FL Ph.D. Candidate, Aerospace Engineering M.Sc. Aerospace Engineering B.Sc. Aerospace Engineering B.Sc. Computational Mathematics Phone: (386)-843-1829 Email(s): ferranj2 at my.erau.edu jesus.ferrand at gmail.com -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!bg9QrJhAkKOSNFt840OnuOpmX_IlSabDICy99_9tMp9uABL-2lhCm_LG2Fj8rwdySPMiK7rnfMLSIPxs3jDVQRKJmPA$ ________________________________ From: Matthew Knepley > Sent: Wednesday, May 1, 2024 8:07 PM To: Ferrand, Jesus A. > Cc: petsc-users at mcs.anl.gov > Subject: [EXTERNAL] Re: [petsc-users] Is there anything like a "DMPlexSetCones()" ? CAUTION: This email originated outside of Embry-Riddle Aeronautical University. Do not click links or open attachments unless you recognize the sender and know the content is safe. On Wed, May 1, 2024 at 3:34?PM Ferrand, Jesus A. > wrote: This Message Is From an External Sender This message came from outside your organization. Dear PETSc team: For a project that I'm working on, I need to manually build a DMPlex. >From studying the source code of the various APIs in which the plex is built from some supported file format, I get that the workflow is this: 1. DMPlexSetChart() <-- Input nCells + nVerts 2. DMPlexSetConeSize() <-- Input ConeSize for each point in [0,nCells) 3. DMSetUp() ? Allocates memory internally. 4. DMPlexGetCones() --> Gives you the memory onto which to write the cell connectivity. 5. *Write connectivity* 6. DMPlexReorderCell() <-- For each point in [0,nCells) I'm in a situation where the memory given by step (4) is available a-priori. I was hoping to skip steps 2, 3 , and 4 with something like a "DMPlexSetCones()", but such an API does not exist. My current workaround is to implement steps 2 through 4 as always and have double the memory allocated in the interim (my instance + DM's internal instance). I was thinking of looking for the name of the struct member and = it to my memory, but I can't overcome the flag check in DMSetUp() during later calls to DMPlexGetCones() or DMPlexGetTransitiveClosure(). I do not understand the "flag check". What is that? We could make a DMPlexSetCones(), but as you point out, the workflow for DMSetUp() would have to change. Where does the memory come from for your connectivity? Thanks, Matt Sincerely: J.A. Ferrand Embry-Riddle Aeronautical University - Daytona Beach - FL Ph.D. Candidate, Aerospace Engineering M.Sc. Aerospace Engineering B.Sc. Aerospace Engineering B.Sc. Computational Mathematics Phone: (386)-843-1829 Email(s): ferranj2 at my.erau.edu jesus.ferrand at gmail.com -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!bg9QrJhAkKOSNFt840OnuOpmX_IlSabDICy99_9tMp9uABL-2lhCm_LG2Fj8rwdySPMiK7rnfMLSIPxs3jDVQRKJmPA$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!bg9QrJhAkKOSNFt840OnuOpmX_IlSabDICy99_9tMp9uABL-2lhCm_LG2Fj8rwdySPMiK7rnfMLSIPxs3jDVQRKJmPA$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From hellyj at ucsd.edu Wed May 1 21:32:02 2024 From: hellyj at ucsd.edu (John Helly) Date: Wed, 1 May 2024 16:32:02 -1000 Subject: [petsc-users] unsubscribe Message-ID: <06d90121-f462-46ec-bfc5-311d86c1e2b4@ucsd.edu> An HTML attachment was scrubbed... URL: From a.croucher at auckland.ac.nz Wed May 1 22:03:08 2024 From: a.croucher at auckland.ac.nz (Adrian Croucher) Date: Thu, 2 May 2024 15:03:08 +1200 Subject: [petsc-users] HDF5 time step count In-Reply-To: <220daca5-21ea-00df-bc51-ba6124d020ad@auckland.ac.nz> References: <425d849a-eba3-6739-5e71-251e0f29b5fb@auckland.ac.nz> <8735pcd12k.fsf@jedbrown.org> <7639e2f6-d326-e04b-bd8e-53cecb8d794e@auckland.ac.nz> <220daca5-21ea-00df-bc51-ba6124d020ad@auckland.ac.nz> Message-ID: <6aa6644b-c36b-4f4d-b843-d1a1a4adf7bd@auckland.ac.nz> hi Matt & all, I just had a query from one of my users which prompted me to see if any progress had been made on the issue below - using PETSc to get the number of time steps in an HDF5 file. I can't see anything new in PETSc on this - I did try using PetscViewerHDF5ReadSizes() to see if that would do it, but it seems it doesn't. If I use that on the "time" dataset it just returns 1. Regards, Adrian On 11/10/21 2:08 pm, Adrian Croucher wrote: > On 10/11/21 11:59 AM, Matthew Knepley wrote: >> On Sun, Oct 10, 2021 at 6:51 PM Adrian Croucher >> wrote: >> >> hi >> >> Is there any way to query the PETSc HDF5 viewer to find the >> number of >> time steps in the file? >> >> A common use case I have is that an HDF5 file from a previous >> simulation >> is used to get initial conditions for a subsequent run. The most >> common >> thing you want to do is restart from the last set of results in the >> previous output. To do that you need to know how many time steps >> there >> are, so you can set the output index to be the last one. >> >> I thought maybe I could just query the size of the "time" >> dataset, but I >> can't even see any obvious way to do that using the viewer functions. >> >> >> There is nothing in there that does it right now. Do you know how to >> do it in HDF5? >> If so, I can put it in. Otherwise, I will have to learn more HDF5 :) > > I haven't actually tried this myself but it looks like what you do is: > > 1) get the dataspace for the dataset (in our case the "time" dataset): > > hid_t dspace = H5Dget_space(dset); > > 2) Get the dimensions of the dataspace: > > const int ndims = 1; > > hsize_t dims[ndims]; > H5Sget_simple_extent_dims(dspace, dims, NULL); > > The first element of dims should be the number of time steps. Here > I've assumed the number of dimensions of the time dataset is 1. In > general you can instead query the rank of the dataspace using > H5Sget_simple_extent_ndims() to get the rank ndims. > > Regards, Adrian > -- Dr Adrian Croucher Senior Research Fellow Department of Engineering Science Waipapa Taumata Rau / University of Auckland, New Zealand email:a.croucher at auckland.ac.nz tel: +64 (0)9 923 4611 -------------- next part -------------- An HTML attachment was scrubbed... URL: From marcos.vanella at nist.gov Thu May 2 10:15:36 2024 From: marcos.vanella at nist.gov (Vanella, Marcos (Fed)) Date: Thu, 2 May 2024 15:15:36 +0000 Subject: [petsc-users] Compiling PETSc in Polaris with gnu Message-ID: Hi all, it seems the modules in Polaris have changed (can't find cudatoolkit-standalone anymore). Does anyone have recent experience compiling the library with gnu and cuda in the machine? Thank you! Marcos -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Thu May 2 10:23:26 2024 From: balay at mcs.anl.gov (Satish Balay) Date: Thu, 2 May 2024 10:23:26 -0500 (CDT) Subject: [petsc-users] Compiling PETSc in Polaris with gnu In-Reply-To: References: Message-ID: <4e64e212-5c94-20ed-0dd1-418706d8993b@mcs.anl.gov> Try: module use /soft/modulefiles Satish On Thu, 2 May 2024, Vanella, Marcos (Fed) via petsc-users wrote: > Hi all, it seems the modules in Polaris have changed (can't find cudatoolkit-standalone anymore). > Does anyone have recent experience compiling the library with gnu and cuda in the machine? > Thank you! > Marcos > From junchao.zhang at gmail.com Thu May 2 10:32:04 2024 From: junchao.zhang at gmail.com (Junchao Zhang) Date: Thu, 2 May 2024 10:32:04 -0500 Subject: [petsc-users] Compiling PETSc in Polaris with gnu In-Reply-To: <4e64e212-5c94-20ed-0dd1-418706d8993b@mcs.anl.gov> References: <4e64e212-5c94-20ed-0dd1-418706d8993b@mcs.anl.gov> Message-ID: I used cudatoolkit-standalone/12.4.1 and gcc-12.3. Be sure to use the latest petsc/main or petsc/release, which contains fixes for Polaris. --Junchao Zhang On Thu, May 2, 2024 at 10:23?AM Satish Balay via petsc-users < petsc-users at mcs.anl.gov> wrote: > Try: > > module use /soft/modulefiles > > Satish > > On Thu, 2 May 2024, Vanella, Marcos (Fed) via petsc-users wrote: > > > Hi all, it seems the modules in Polaris have changed (can't find > cudatoolkit-standalone anymore). > > Does anyone have recent experience compiling the library with gnu and > cuda in the machine? > > Thank you! > > Marcos > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Thu May 2 10:58:21 2024 From: balay at mcs.anl.gov (Satish Balay) Date: Thu, 2 May 2024 10:58:21 -0500 (CDT) Subject: [petsc-users] Compiling PETSc in Polaris with gnu In-Reply-To: References: <4e64e212-5c94-20ed-0dd1-418706d8993b@mcs.anl.gov> Message-ID: <8e34a17f-ac52-cb64-f46b-a8eb418c93e2@mcs.anl.gov> I just tried a build (used default versions) - and the following builds for me [on the login node]. module use /soft/modulefiles module load PrgEnv-gnu module load cudatoolkit-standalone module load cray-libsci ./configure --with-cc=cc --with-fc=ftn --with-cxx=CC --with-make-np=4 --with-cuda=1 --with-cudac=nvcc --with-cuda-arch=80 \ --with-debugging=0 COPTFLAGS=-O2 CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 CUDAOPTFLAGS=-O2 --download-kokkos --download-kokkos-kernels make Satish --- balay at polaris-login-01:~> module list Currently Loaded Modules: 1) libfabric/1.15.2.0 4) darshan/3.4.4 7) cray-dsmml/0.2.2 10) cray-pals/1.3.4 13) PrgEnv-gnu/8.5.0 2) craype-network-ofi 5) gcc-native/12.3 8) cray-mpich/8.1.28 11) cray-libpals/1.3.4 14) cudatoolkit-standalone/12.2.2 3) perftools-base/23.12.0 6) craype/2.7.30 9) cray-pmi/6.1.13 12) craype-x86-milan 15) cray-libsci/23.12.5 On Thu, 2 May 2024, Junchao Zhang wrote: > I used cudatoolkit-standalone/12.4.1 and gcc-12.3. > > Be sure to use the latest petsc/main or petsc/release, which contains fixes > for Polaris. > > --Junchao Zhang > > > On Thu, May 2, 2024 at 10:23?AM Satish Balay via petsc-users < > petsc-users at mcs.anl.gov> wrote: > > > Try: > > > > module use /soft/modulefiles > > > > Satish > > > > On Thu, 2 May 2024, Vanella, Marcos (Fed) via petsc-users wrote: > > > > > Hi all, it seems the modules in Polaris have changed (can't find > > cudatoolkit-standalone anymore). > > > Does anyone have recent experience compiling the library with gnu and > > cuda in the machine? > > > Thank you! > > > Marcos > > > > > > > > From samar.khatiwala at earth.ox.ac.uk Thu May 2 15:53:28 2024 From: samar.khatiwala at earth.ox.ac.uk (Samar Khatiwala) Date: Thu, 2 May 2024 20:53:28 +0000 Subject: [petsc-users] Question about petsc4py createWithArray function Message-ID: An HTML attachment was scrubbed... URL: From marcos.vanella at nist.gov Thu May 2 16:02:41 2024 From: marcos.vanella at nist.gov (Vanella, Marcos (Fed)) Date: Thu, 2 May 2024 21:02:41 +0000 Subject: [petsc-users] Compiling PETSc in Polaris with gnu In-Reply-To: <8e34a17f-ac52-cb64-f46b-a8eb418c93e2@mcs.anl.gov> References: <4e64e212-5c94-20ed-0dd1-418706d8993b@mcs.anl.gov> <8e34a17f-ac52-cb64-f46b-a8eb418c93e2@mcs.anl.gov> Message-ID: Thank you Satish and Junchao! I was able to compile PETSc with your configure options + suitesparse and hypre, and then compile my fortran code linking to PETSc. But when I try to run my test run I'm picking up an error at the very beginning: MPICH ERROR [Rank 0] [job id 01eb3c4a-28a7-4178-aced-512b4fb704c6] [Thu May 2 20:44:26 2024] [x3006c0s19b1n0] - Abort(-1) (rank 0 in comm 0): MPIDI_CRAY_init: GPU_SUPPORT_ENABLED is requested, but GTL library is not linked (Other MPI error) aborting job: MPIDI_CRAY_init: GPU_SUPPORT_ENABLED is requested, but GTL library is not linked It says in the Polaris user guide that: The environment variable MPICH_GPU_SUPPORT_ENABLED=1 needs to be set if your application requires MPI-GPU support whereby the MPI library sends and receives data directly from GPU buffers. In this case, it will be important to have the craype-accel-nvidia80 module loaded both when compiling your application and during runtime to correctly link against a GPU Transport Layer (GTL) MPI library. Otherwise, you'll likely see GPU_SUPPORT_ENABLED is requested, but GTL library is not linked errors during runtime. I tried adding loading this module (also needed to add nvhpc-mixed) in my submission script but I get the same result. I'll get in touch with alcf help on this. ________________________________ From: Satish Balay Sent: Thursday, May 2, 2024 11:58 AM To: Junchao Zhang Cc: petsc-users ; Vanella, Marcos (Fed) ; Mueller, Eric V. (Fed) Subject: Re: [petsc-users] Compiling PETSc in Polaris with gnu I just tried a build (used default versions) - and the following builds for me [on the login node]. module use /soft/modulefiles module load PrgEnv-gnu module load cudatoolkit-standalone module load cray-libsci ./configure --with-cc=cc --with-fc=ftn --with-cxx=CC --with-make-np=4 --with-cuda=1 --with-cudac=nvcc --with-cuda-arch=80 \ --with-debugging=0 COPTFLAGS=-O2 CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 CUDAOPTFLAGS=-O2 --download-kokkos --download-kokkos-kernels make Satish --- balay at polaris-login-01:~> module list Currently Loaded Modules: 1) libfabric/1.15.2.0 4) darshan/3.4.4 7) cray-dsmml/0.2.2 10) cray-pals/1.3.4 13) PrgEnv-gnu/8.5.0 2) craype-network-ofi 5) gcc-native/12.3 8) cray-mpich/8.1.28 11) cray-libpals/1.3.4 14) cudatoolkit-standalone/12.2.2 3) perftools-base/23.12.0 6) craype/2.7.30 9) cray-pmi/6.1.13 12) craype-x86-milan 15) cray-libsci/23.12.5 On Thu, 2 May 2024, Junchao Zhang wrote: > I used cudatoolkit-standalone/12.4.1 and gcc-12.3. > > Be sure to use the latest petsc/main or petsc/release, which contains fixes > for Polaris. > > --Junchao Zhang > > > On Thu, May 2, 2024 at 10:23?AM Satish Balay via petsc-users < > petsc-users at mcs.anl.gov> wrote: > > > Try: > > > > module use /soft/modulefiles > > > > Satish > > > > On Thu, 2 May 2024, Vanella, Marcos (Fed) via petsc-users wrote: > > > > > Hi all, it seems the modules in Polaris have changed (can't find > > cudatoolkit-standalone anymore). > > > Does anyone have recent experience compiling the library with gnu and > > cuda in the machine? > > > Thank you! > > > Marcos > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Thu May 2 16:51:47 2024 From: balay at mcs.anl.gov (Satish Balay) Date: Thu, 2 May 2024 16:51:47 -0500 (CDT) Subject: [petsc-users] Compiling PETSc in Polaris with gnu In-Reply-To: References: <4e64e212-5c94-20ed-0dd1-418706d8993b@mcs.anl.gov> <8e34a17f-ac52-cb64-f46b-a8eb418c93e2@mcs.anl.gov> Message-ID: <3ef2c8cd-e045-9efc-d0b7-16e3fb3b99fa@mcs.anl.gov> Perhaps you need to: module load craype-accel-nvidia80 And then rebuild PETSc, your application And have the same list of modules loaded at runtime. Satish On Thu, 2 May 2024, Vanella, Marcos (Fed) via petsc-users wrote: > Thank you Satish and Junchao! I was able to compile PETSc with your configure options + suitesparse and hypre, and then compile my fortran code linking to PETSc. > But when I try to run my test run I'm picking up an error at the very beginning: > > MPICH ERROR [Rank 0] [job id 01eb3c4a-28a7-4178-aced-512b4fb704c6] [Thu May 2 20:44:26 2024] [x3006c0s19b1n0] - Abort(-1) (rank 0 in comm 0): MPIDI_CRAY_init: GPU_SUPPORT_ENABLED is requested, but GTL library is not linked > (Other MPI error) > > aborting job: > MPIDI_CRAY_init: GPU_SUPPORT_ENABLED is requested, but GTL library is not linked > > It says in the Polaris user guide that: > > The environment variable MPICH_GPU_SUPPORT_ENABLED=1 needs to be set if your application requires MPI-GPU support whereby the MPI library sends and receives data directly from GPU buffers. In this case, it will be important to have the craype-accel-nvidia80 module loaded both when compiling your application and during runtime to correctly link against a GPU Transport Layer (GTL) MPI library. Otherwise, you'll likely see GPU_SUPPORT_ENABLED is requested, but GTL library is not linked errors during runtime. > > I tried adding loading this module (also needed to add nvhpc-mixed) in my submission script but I get the same result. > I'll get in touch with alcf help on this. > > > > ________________________________ > From: Satish Balay > Sent: Thursday, May 2, 2024 11:58 AM > To: Junchao Zhang > Cc: petsc-users ; Vanella, Marcos (Fed) ; Mueller, Eric V. (Fed) > Subject: Re: [petsc-users] Compiling PETSc in Polaris with gnu > > I just tried a build (used default versions) - and the following builds for me [on the login node]. > > > module use /soft/modulefiles > module load PrgEnv-gnu > module load cudatoolkit-standalone > module load cray-libsci > ./configure --with-cc=cc --with-fc=ftn --with-cxx=CC --with-make-np=4 --with-cuda=1 --with-cudac=nvcc --with-cuda-arch=80 \ > --with-debugging=0 COPTFLAGS=-O2 CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 CUDAOPTFLAGS=-O2 --download-kokkos --download-kokkos-kernels > make > > Satish > > --- > > balay at polaris-login-01:~> module list > > Currently Loaded Modules: > 1) libfabric/1.15.2.0 4) darshan/3.4.4 7) cray-dsmml/0.2.2 10) cray-pals/1.3.4 13) PrgEnv-gnu/8.5.0 > 2) craype-network-ofi 5) gcc-native/12.3 8) cray-mpich/8.1.28 11) cray-libpals/1.3.4 14) cudatoolkit-standalone/12.2.2 > 3) perftools-base/23.12.0 6) craype/2.7.30 9) cray-pmi/6.1.13 12) craype-x86-milan 15) cray-libsci/23.12.5 > > > On Thu, 2 May 2024, Junchao Zhang wrote: > > > I used cudatoolkit-standalone/12.4.1 and gcc-12.3. > > > > Be sure to use the latest petsc/main or petsc/release, which contains fixes > > for Polaris. > > > > --Junchao Zhang > > > > > > On Thu, May 2, 2024 at 10:23?AM Satish Balay via petsc-users < > > petsc-users at mcs.anl.gov> wrote: > > > > > Try: > > > > > > module use /soft/modulefiles > > > > > > Satish > > > > > > On Thu, 2 May 2024, Vanella, Marcos (Fed) via petsc-users wrote: > > > > > > > Hi all, it seems the modules in Polaris have changed (can't find > > > cudatoolkit-standalone anymore). > > > > Does anyone have recent experience compiling the library with gnu and > > > cuda in the machine? > > > > Thank you! > > > > Marcos > > > > > > > > > > > > > From knepley at gmail.com Thu May 2 17:56:10 2024 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 2 May 2024 14:56:10 -0800 Subject: [petsc-users] Question about petsc4py createWithArray function In-Reply-To: References: Message-ID: On Thu, May 2, 2024 at 12:53?PM Samar Khatiwala < samar.khatiwala at earth.ox.ac.uk> wrote: > Hello, I have a couple of questions about createWithArray in petsc4py: 1) > What is the correct usage for creating a standard MPI Vec with it? > Something like this seems to work but is it right?: On each rank do: a = > np. zeros(localSize) v = PETSc. Vec(). createWithArray(a, > ZjQcmQRYFpfptBannerStart > This Message Is From an External Sender > This message came from outside your organization. > > ZjQcmQRYFpfptBannerEnd > > Hello, > > I have a couple of questions about createWithArray in petsc4py: > > 1) What is the correct usage for creating a standard MPI Vec with it? Something like this seems to work but is it right?: > > On each rank do: > a = np.zeros(localSize) > v = PETSc.Vec().createWithArray(a, comm=PETSc.COMM_WORLD) > > Is that all it takes? > > That looks right to me. > 2) Who ?owns? the underlying memory for a Vec created with the createWithArray method, i.e., who is responsible for managing it and doing garbage collection? In my problem, the numpy array is created in a Cython module where memory is allocated, and a pointer to it is associated with a numpy ndarray via PyArray_SimpleNewFromData and PyArray_SetBaseObject. I have a deallocator method of my own that is called when the numpy array is deleted/goes out of scope/whenever python does garbage collection. All of that works fine. But if I use this array to create a Vec with createWithArray what happens when the Vec is, e.g., destroyed? Will my deallocator be called? > > No. The PETSc struct will be deallocated, but the storage will not be touched. Thanks, Matt > Or does petsc4py know that it doesn?t own the memory and won?t attempt to free it? I can?t quite figure out from the petsc4py code what is going on. And help would be appreciated. > > Thanks very much. > > Samar > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!anRKQn_8v127_CSkzk22FfTFRodKT0G2BLwgi_kPAUQt_eqCiySYWSg3ctwewCceXLeJY4FU01ONUG3xfSeD$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From samar.khatiwala at earth.ox.ac.uk Fri May 3 03:41:59 2024 From: samar.khatiwala at earth.ox.ac.uk (Samar Khatiwala) Date: Fri, 3 May 2024 08:41:59 +0000 Subject: [petsc-users] Question about petsc4py createWithArray function In-Reply-To: References: Message-ID: <6ED2F015-541E-485B-AACB-EAE329A4A307@earth.ox.ac.uk> Hi Matt, Thanks so much for the quick reply! Regarding #2, I put some debug statement in my code and what I find is that when I use createWithArray on my Cython-allocated numpy array, the destructor I set for it is no longer called when I delete the array. (If I don?t use createWithArray then the destructor is triggered.) I interpret that to suggest that the petsc4py Vec is somehow ?taking over? management of the numpy array. But I don?t understand where that could be happening. (I don?t think it has to do with the actual freeing of memory by PETSc's VecDestroy.) createWithArray calls iarray_s which in turn calls PyArray_FROM_OTF. Could it be there?s something going on there? The numpy documentation is unclear. Lisandro: do you have any thoughts on this? Thanks, Samar On May 2, 2024, at 11:56 PM, Matthew Knepley wrote: On Thu, May 2, 2024 at 12:53?PM Samar Khatiwala > wrote: This Message Is From an External Sender This message came from outside your organization. Hello, I have a couple of questions about createWithArray in petsc4py: 1) What is the correct usage for creating a standard MPI Vec with it? Something like this seems to work but is it right?: On each rank do: a = np.zeros(localSize) v = PETSc.Vec().createWithArray(a, comm=PETSc.COMM_WORLD) Is that all it takes? That looks right to me. 2) Who ?owns? the underlying memory for a Vec created with the createWithArray method, i.e., who is responsible for managing it and doing garbage collection? In my problem, the numpy array is created in a Cython module where memory is allocated, and a pointer to it is associated with a numpy ndarray via PyArray_SimpleNewFromData and PyArray_SetBaseObject. I have a deallocator method of my own that is called when the numpy array is deleted/goes out of scope/whenever python does garbage collection. All of that works fine. But if I use this array to create a Vec with createWithArray what happens when the Vec is, e.g., destroyed? Will my deallocator be called? No. The PETSc struct will be deallocated, but the storage will not be touched. Thanks, Matt Or does petsc4py know that it doesn?t own the memory and won?t attempt to free it? I can?t quite figure out from the petsc4py code what is going on. And help would be appreciated. Thanks very much. Samar -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!dXZuKR18IO2hZ4uPEqHaG0Jb0ALPgw1T0dm4HAWfyBfsCvkYTmq7aG7yw3yc-5CVhcFMaUL3WzGM8YHlBTTA_JMs_dYUCTX8wgHdY_Y$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefano.zampini at gmail.com Fri May 3 06:45:35 2024 From: stefano.zampini at gmail.com (Stefano Zampini) Date: Fri, 3 May 2024 14:45:35 +0300 Subject: [petsc-users] Question about petsc4py createWithArray function In-Reply-To: <6ED2F015-541E-485B-AACB-EAE329A4A307@earth.ox.ac.uk> References: <6ED2F015-541E-485B-AACB-EAE329A4A307@earth.ox.ac.uk> Message-ID: While waiting for our Python wizard to shed light on this, I note that, from the documentation of PyArray_FROM_OTF https://urldefense.us/v3/__https://numpy.org/devdocs/user/c-info.how-to-extend.html*converting-an-arbitrary-sequence-object__;Iw!!G_uCfscf7eWS!cPqvd3brsxbc9HOSop19DutpF2hSdn6iPY382KBUd45kJQBA2AyAU5Neifq0WTDf49xH3CybbVSfg7gg6NOaKMOEYdkmd8Y$ , we have The object can be any Python object convertible to an ndarray. If the object is already (a subclass of) the ndarray that satisfies the requirements then a new reference is returned. I guess we should call "del" on the ndarray returned by iarray_s after having called self.set_attr('__array__', array) in this case, but let's wait for Lisandro to confirm Il giorno ven 3 mag 2024 alle ore 11:42 Samar Khatiwala < samar.khatiwala at earth.ox.ac.uk> ha scritto: > Hi Matt, Thanks so much for the quick reply! Regarding #2, I put some > debug statement in my code and what I find is that when I use > createWithArray on my Cython-allocated numpy array, the destructor I set > for it is no longer called when I delete > ZjQcmQRYFpfptBannerStart > This Message Is From an External Sender > This message came from outside your organization. > > ZjQcmQRYFpfptBannerEnd > Hi Matt, > > Thanks so much for the quick reply! > > Regarding #2, I put some debug statement in my code and what I find is > that when I use createWithArray on my Cython-allocated numpy array, the > destructor I set for it is no longer called when I delete the array. (If I > don?t use createWithArray then the destructor is triggered.) I interpret > that to suggest that the petsc4py Vec is somehow ?taking over? management > of the numpy array. But I don?t understand where that could be > happening. (I don?t think it has to do with the actual freeing of memory by > PETSc's VecDestroy.) > > createWithArray calls iarray_s which in turn calls PyArray_FROM_OTF. > Could it be there?s something going on there? The numpy documentation is > unclear. > > Lisandro: do you have any thoughts on this? > > Thanks, > > Samar > > On May 2, 2024, at 11:56 PM, Matthew Knepley wrote: > > On Thu, May 2, 2024 at 12:53?PM Samar Khatiwala < > samar.khatiwala at earth.ox.ac.uk> wrote: > >> This Message Is From an External Sender >> This message came from outside your organization. >> >> >> Hello, >> >> I have a couple of questions about createWithArray in petsc4py: >> >> 1) What is the correct usage for creating a standard MPI Vec with it? Something like this seems to work but is it right?: >> >> On each rank do: >> a = np.zeros(localSize) >> v = PETSc.Vec().createWithArray(a, comm=PETSc.COMM_WORLD) >> >> Is that all it takes? >> >> > That looks right to me. > >> 2) Who ?owns? the underlying memory for a Vec created with the createWithArray method, i.e., who is responsible for managing it and doing garbage collection? In my problem, the numpy array is created in a Cython module where memory is allocated, and a pointer to it is associated with a numpy ndarray via PyArray_SimpleNewFromData and PyArray_SetBaseObject. I have a deallocator method of my own that is called when the numpy array is deleted/goes out of scope/whenever python does garbage collection. All of that works fine. But if I use this array to create a Vec with createWithArray what happens when the Vec is, e.g., destroyed? Will my deallocator be called? >> >> No. The PETSc struct will be deallocated, but the storage will not be > touched. > > Thanks, > > Matt > >> Or does petsc4py know that it doesn?t own the memory and won?t attempt to free it? I can?t quite figure out from the petsc4py code what is going on. And help would be appreciated. >> >> Thanks very much. >> >> Samar >> >> >> > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!cPqvd3brsxbc9HOSop19DutpF2hSdn6iPY382KBUd45kJQBA2AyAU5Neifq0WTDf49xH3CybbVSfg7gg6NOaKMOEOy05fZU$ > > > > -- Stefano -------------- next part -------------- An HTML attachment was scrubbed... URL: From marcos.vanella at nist.gov Fri May 3 08:55:23 2024 From: marcos.vanella at nist.gov (Vanella, Marcos (Fed)) Date: Fri, 3 May 2024 13:55:23 +0000 Subject: [petsc-users] Compiling PETSc in Polaris with gnu In-Reply-To: <3ef2c8cd-e045-9efc-d0b7-16e3fb3b99fa@mcs.anl.gov> References: <4e64e212-5c94-20ed-0dd1-418706d8993b@mcs.anl.gov> <8e34a17f-ac52-cb64-f46b-a8eb418c93e2@mcs.anl.gov> <3ef2c8cd-e045-9efc-d0b7-16e3fb3b99fa@mcs.anl.gov> Message-ID: Thank you Satish, I'll give it a try. ________________________________ From: Satish Balay Sent: Thursday, May 2, 2024 5:51 PM To: Vanella, Marcos (Fed) Cc: Junchao Zhang ; petsc-users ; Mueller, Eric V. (Fed) Subject: Re: [petsc-users] Compiling PETSc in Polaris with gnu Perhaps you need to: module load craype-accel-nvidia80 And then rebuild PETSc, your application And have the same list of modules loaded at runtime. Satish On Thu, 2 May 2024, Vanella, Marcos (Fed) via petsc-users wrote: > Thank you Satish and Junchao! I was able to compile PETSc with your configure options + suitesparse and hypre, and then compile my fortran code linking to PETSc. > But when I try to run my test run I'm picking up an error at the very beginning: > > MPICH ERROR [Rank 0] [job id 01eb3c4a-28a7-4178-aced-512b4fb704c6] [Thu May 2 20:44:26 2024] [x3006c0s19b1n0] - Abort(-1) (rank 0 in comm 0): MPIDI_CRAY_init: GPU_SUPPORT_ENABLED is requested, but GTL library is not linked > (Other MPI error) > > aborting job: > MPIDI_CRAY_init: GPU_SUPPORT_ENABLED is requested, but GTL library is not linked > > It says in the Polaris user guide that: > > The environment variable MPICH_GPU_SUPPORT_ENABLED=1 needs to be set if your application requires MPI-GPU support whereby the MPI library sends and receives data directly from GPU buffers. In this case, it will be important to have the craype-accel-nvidia80 module loaded both when compiling your application and during runtime to correctly link against a GPU Transport Layer (GTL) MPI library. Otherwise, you'll likely see GPU_SUPPORT_ENABLED is requested, but GTL library is not linked errors during runtime. > > I tried adding loading this module (also needed to add nvhpc-mixed) in my submission script but I get the same result. > I'll get in touch with alcf help on this. > > > > ________________________________ > From: Satish Balay > Sent: Thursday, May 2, 2024 11:58 AM > To: Junchao Zhang > Cc: petsc-users ; Vanella, Marcos (Fed) ; Mueller, Eric V. (Fed) > Subject: Re: [petsc-users] Compiling PETSc in Polaris with gnu > > I just tried a build (used default versions) - and the following builds for me [on the login node]. > > > module use /soft/modulefiles > module load PrgEnv-gnu > module load cudatoolkit-standalone > module load cray-libsci > ./configure --with-cc=cc --with-fc=ftn --with-cxx=CC --with-make-np=4 --with-cuda=1 --with-cudac=nvcc --with-cuda-arch=80 \ > --with-debugging=0 COPTFLAGS=-O2 CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 CUDAOPTFLAGS=-O2 --download-kokkos --download-kokkos-kernels > make > > Satish > > --- > > balay at polaris-login-01:~> module list > > Currently Loaded Modules: > 1) libfabric/1.15.2.0 4) darshan/3.4.4 7) cray-dsmml/0.2.2 10) cray-pals/1.3.4 13) PrgEnv-gnu/8.5.0 > 2) craype-network-ofi 5) gcc-native/12.3 8) cray-mpich/8.1.28 11) cray-libpals/1.3.4 14) cudatoolkit-standalone/12.2.2 > 3) perftools-base/23.12.0 6) craype/2.7.30 9) cray-pmi/6.1.13 12) craype-x86-milan 15) cray-libsci/23.12.5 > > > On Thu, 2 May 2024, Junchao Zhang wrote: > > > I used cudatoolkit-standalone/12.4.1 and gcc-12.3. > > > > Be sure to use the latest petsc/main or petsc/release, which contains fixes > > for Polaris. > > > > --Junchao Zhang > > > > > > On Thu, May 2, 2024 at 10:23?AM Satish Balay via petsc-users < > > petsc-users at mcs.anl.gov> wrote: > > > > > Try: > > > > > > module use /soft/modulefiles > > > > > > Satish > > > > > > On Thu, 2 May 2024, Vanella, Marcos (Fed) via petsc-users wrote: > > > > > > > Hi all, it seems the modules in Polaris have changed (can't find > > > cudatoolkit-standalone anymore). > > > > Does anyone have recent experience compiling the library with gnu and > > > cuda in the machine? > > > > Thank you! > > > > Marcos > > > > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri May 3 17:45:16 2024 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 3 May 2024 14:45:16 -0800 Subject: [petsc-users] [EXTERNAL] Re: Is there anything like a "DMPlexSetCones()" ? In-Reply-To: References: Message-ID: On Wed, May 1, 2024 at 6:30?PM Ferrand, Jesus A. wrote: > "You know the number of cells if you can read connectivity. Do you mean > that the format does not tell you > the number of vertices?" > > Sort of... > So, the format does provide the number of vertices, however, due to the > way I read the data in parallel I don't know immediately how many local > vertices there will be. > Which prevents me from knowing the chart a priori. > I figured out the number of vertices using PetscHashSetI to determine the > number of unique entries in the connectivity list. > Right, but that means that the connectivity you read in is not in local numbering. You will have to renumber it, so making a copy is probably necessary anyway. This is the same sort of processing I do for the parallel read. Thanks, Matt > ------------------------------ > *From:* Matthew Knepley > *Sent:* Wednesday, May 1, 2024 8:52 PM > *To:* Ferrand, Jesus A. > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [EXTERNAL] Re: [petsc-users] Is there anything like a > "DMPlexSetCones()" ? > > On Wed, May 1, 2024 at 4:23?PM Ferrand, Jesus A. > wrote: > > Matt: > > My bad again, I need to clarify something that I just realized doesn't > make sense. > I said: "The nature of the I/O makes it so that I need to read the > connectivity before I get a semblance of buffer sizes." > Scratch that, obviously, if I can read the connectivity, I know the buffer > sizes. > > What I should have said is that I cannot know the "chart" that one sets in > DMPlexSetChart() a priori like in most cases. > I need to determine the chart from the connectivity lists. > > > You know the number of cells if you can read connectivity. Do you mean > that the format does not tell you > the number of vertices? > > Thanks, > > Matt > > > ------------------------------ > *From:* Ferrand, Jesus A. > *Sent:* Wednesday, May 1, 2024 8:17 PM > *To:* Matthew Knepley > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [EXTERNAL] Re: [petsc-users] Is there anything like a > "DMPlexSetCones()" ? > > Matt: > > "I do not understand the "flag check". What is that?" > > My bad, I should have referred to the "dm->setupcalled". > I believe this PetscBool is checked by the other DM (not just DMPlex) APIs. > The subsequent checks for dm->setupcalled == PETSC_TRUE is what I meant to > say. > Here's a copy of DMSetUp(). > > *PetscErrorCode DMSetUp (DM dm)* > 817: { > 818: PetscFunctionBegin ; > 820: if (dm->setupcalled) PetscFunctionReturn (PETSC_SUCCESS ); > 821: PetscTryTypeMethod (dm, setup); > 822: dm->setupcalled = PETSC_TRUE ; > 823: PetscFunctionReturn (PETSC_SUCCESS ); > 824: } > > "We could make a DMPlexSetCones(), but as you point out, the workflow for > DMSetUp() would have to change. Where does the memory come from for your > connectivity?" > It's a memory that I myself allocate based on a file's contents. > The nature of the I/O makes it so that I need to read the connectivity > before I get a semblance of buffer sizes. > Otherwise, I would stick to the tried and tested way. > > Also, when replying to the PETSc developers, users must reply to > petsc-users at mcs.anl.gov and not just to the individual email accounts of > the developers, right? > > > > > ------------------------------ > *From:* Matthew Knepley > *Sent:* Wednesday, May 1, 2024 8:07 PM > *To:* Ferrand, Jesus A. > *Cc:* petsc-users at mcs.anl.gov > *Subject:* [EXTERNAL] Re: [petsc-users] Is there anything like a > "DMPlexSetCones()" ? > > *CAUTION:* This email originated outside of Embry-Riddle Aeronautical > University. Do not click links or open attachments unless you recognize the > sender and know the content is safe. > > On Wed, May 1, 2024 at 3:34?PM Ferrand, Jesus A. > wrote: > > This Message Is From an External Sender > This message came from outside your organization. > > Dear PETSc team: > > For a project that I'm working on, I need to manually build a DMPlex. > From studying the source code of the various APIs in which the plex is > built from some supported file format, I get that the workflow is this: > > > 1. DMPlexSetChart() <-- Input nCells + nVerts > 2. DMPlexSetConeSize() <-- Input ConeSize for each point in [0,nCells) > 3. DMSetUp() ? Allocates memory internally. > 4. DMPlexGetCones() --> Gives you the memory onto which to write the > cell connectivity. > 5. *Write connectivity* > 6. DMPlexReorderCell() <-- For each point in [0,nCells) > > > I'm in a situation where the memory given by step (4) is available > a-priori. > I was hoping to skip steps 2, 3 , and 4 with something like a > "DMPlexSetCones()", but such an API does not exist. > My current workaround is to implement steps 2 through 4 as always and have > double the memory allocated in the interim (my instance + DM's internal > instance). > I was thinking of looking for the name of the struct member and = it to my > memory, but I can't overcome the flag check in DMSetUp() during later calls > to DMPlexGetCones() or DMPlexGetTransitiveClosure(). > > > I do not understand the "flag check". What is that? > > We could make a DMPlexSetCones(), but as you point out, the workflow for > DMSetUp() would have to change. Where does the memory come from for your > connectivity? > > Thanks, > > Matt > > > > Sincerely: > > *J.A. Ferrand* > > Embry-Riddle Aeronautical University - Daytona Beach - FL > Ph.D. Candidate, Aerospace Engineering > > M.Sc. Aerospace Engineering > > B.Sc. Aerospace Engineering > > B.Sc. Computational Mathematics > > > *Phone:* (386)-843-1829 > > *Email(s):* ferranj2 at my.erau.edu > > jesus.ferrand at gmail.com > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!e603Zn0jXSCVvzNnzzRiius-0-C5jw6GGzCOFJ7EW5nKTGdX8aI6qchOlBW23rM9TlwMTiidfBsowhNf6eH7$ > > > ------------------------------ > *From:* Matthew Knepley > *Sent:* Wednesday, May 1, 2024 8:07 PM > *To:* Ferrand, Jesus A. > *Cc:* petsc-users at mcs.anl.gov > *Subject:* [EXTERNAL] Re: [petsc-users] Is there anything like a > "DMPlexSetCones()" ? > > *CAUTION:* This email originated outside of Embry-Riddle Aeronautical > University. Do not click links or open attachments unless you recognize the > sender and know the content is safe. > > On Wed, May 1, 2024 at 3:34?PM Ferrand, Jesus A. > wrote: > > This Message Is From an External Sender > This message came from outside your organization. > > Dear PETSc team: > > For a project that I'm working on, I need to manually build a DMPlex. > From studying the source code of the various APIs in which the plex is > built from some supported file format, I get that the workflow is this: > > > 1. DMPlexSetChart() <-- Input nCells + nVerts > 2. DMPlexSetConeSize() <-- Input ConeSize for each point in [0,nCells) > 3. DMSetUp() ? Allocates memory internally. > 4. DMPlexGetCones() --> Gives you the memory onto which to write the > cell connectivity. > 5. *Write connectivity* > 6. DMPlexReorderCell() <-- For each point in [0,nCells) > > > I'm in a situation where the memory given by step (4) is available > a-priori. > I was hoping to skip steps 2, 3 , and 4 with something like a > "DMPlexSetCones()", but such an API does not exist. > My current workaround is to implement steps 2 through 4 as always and have > double the memory allocated in the interim (my instance + DM's internal > instance). > I was thinking of looking for the name of the struct member and = it to my > memory, but I can't overcome the flag check in DMSetUp() during later calls > to DMPlexGetCones() or DMPlexGetTransitiveClosure(). > > > I do not understand the "flag check". What is that? > > We could make a DMPlexSetCones(), but as you point out, the workflow for > DMSetUp() would have to change. Where does the memory come from for your > connectivity? > > Thanks, > > Matt > > > > Sincerely: > > *J.A. Ferrand* > > Embry-Riddle Aeronautical University - Daytona Beach - FL > Ph.D. Candidate, Aerospace Engineering > > M.Sc. Aerospace Engineering > > B.Sc. Aerospace Engineering > > B.Sc. Computational Mathematics > > > *Phone:* (386)-843-1829 > > *Email(s):* ferranj2 at my.erau.edu > > jesus.ferrand at gmail.com > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!e603Zn0jXSCVvzNnzzRiius-0-C5jw6GGzCOFJ7EW5nKTGdX8aI6qchOlBW23rM9TlwMTiidfBsowhNf6eH7$ > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!e603Zn0jXSCVvzNnzzRiius-0-C5jw6GGzCOFJ7EW5nKTGdX8aI6qchOlBW23rM9TlwMTiidfBsowhNf6eH7$ > > > ------------------------------ > *From:* Matthew Knepley > *Sent:* Wednesday, May 1, 2024 8:52 PM > *To:* Ferrand, Jesus A. > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [EXTERNAL] Re: [petsc-users] Is there anything like a > "DMPlexSetCones()" ? > > On Wed, May 1, 2024 at 4:23?PM Ferrand, Jesus A. > wrote: > > Matt: > > My bad again, I need to clarify something that I just realized doesn't > make sense. > I said: "The nature of the I/O makes it so that I need to read the > connectivity before I get a semblance of buffer sizes." > Scratch that, obviously, if I can read the connectivity, I know the buffer > sizes. > > What I should have said is that I cannot know the "chart" that one sets in > DMPlexSetChart() a priori like in most cases. > I need to determine the chart from the connectivity lists. > > > You know the number of cells if you can read connectivity. Do you mean > that the format does not tell you > the number of vertices? > > Thanks, > > Matt > > > ------------------------------ > *From:* Ferrand, Jesus A. > *Sent:* Wednesday, May 1, 2024 8:17 PM > *To:* Matthew Knepley > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [EXTERNAL] Re: [petsc-users] Is there anything like a > "DMPlexSetCones()" ? > > Matt: > > "I do not understand the "flag check". What is that?" > > My bad, I should have referred to the "dm->setupcalled". > I believe this PetscBool is checked by the other DM (not just DMPlex) APIs. > The subsequent checks for dm->setupcalled == PETSC_TRUE is what I meant to > say. > Here's a copy of DMSetUp(). > > *PetscErrorCode DMSetUp (DM dm)* > 817: { > 818: PetscFunctionBegin ; > 820: if (dm->setupcalled) PetscFunctionReturn (PETSC_SUCCESS ); > 821: PetscTryTypeMethod (dm, setup); > 822: dm->setupcalled = PETSC_TRUE ; > 823: PetscFunctionReturn (PETSC_SUCCESS ); > 824: } > > "We could make a DMPlexSetCones(), but as you point out, the workflow for > DMSetUp() would have to change. Where does the memory come from for your > connectivity?" > It's a memory that I myself allocate based on a file's contents. > The nature of the I/O makes it so that I need to read the connectivity > before I get a semblance of buffer sizes. > Otherwise, I would stick to the tried and tested way. > > Also, when replying to the PETSc developers, users must reply to > petsc-users at mcs.anl.gov and not just to the individual email accounts of > the developers, right? > > > > > ------------------------------ > *From:* Matthew Knepley > *Sent:* Wednesday, May 1, 2024 8:07 PM > *To:* Ferrand, Jesus A. > *Cc:* petsc-users at mcs.anl.gov > *Subject:* [EXTERNAL] Re: [petsc-users] Is there anything like a > "DMPlexSetCones()" ? > > *CAUTION:* This email originated outside of Embry-Riddle Aeronautical > University. Do not click links or open attachments unless you recognize the > sender and know the content is safe. > > On Wed, May 1, 2024 at 3:34?PM Ferrand, Jesus A. > wrote: > > This Message Is From an External Sender > This message came from outside your organization. > > Dear PETSc team: > > For a project that I'm working on, I need to manually build a DMPlex. > From studying the source code of the various APIs in which the plex is > built from some supported file format, I get that the workflow is this: > > > 1. DMPlexSetChart() <-- Input nCells + nVerts > 2. DMPlexSetConeSize() <-- Input ConeSize for each point in [0,nCells) > 3. DMSetUp() ? Allocates memory internally. > 4. DMPlexGetCones() --> Gives you the memory onto which to write the > cell connectivity. > 5. *Write connectivity* > 6. DMPlexReorderCell() <-- For each point in [0,nCells) > > > I'm in a situation where the memory given by step (4) is available > a-priori. > I was hoping to skip steps 2, 3 , and 4 with something like a > "DMPlexSetCones()", but such an API does not exist. > My current workaround is to implement steps 2 through 4 as always and have > double the memory allocated in the interim (my instance + DM's internal > instance). > I was thinking of looking for the name of the struct member and = it to my > memory, but I can't overcome the flag check in DMSetUp() during later calls > to DMPlexGetCones() or DMPlexGetTransitiveClosure(). > > > I do not understand the "flag check". What is that? > > We could make a DMPlexSetCones(), but as you point out, the workflow for > DMSetUp() would have to change. Where does the memory come from for your > connectivity? > > Thanks, > > Matt > > > > Sincerely: > > *J.A. Ferrand* > > Embry-Riddle Aeronautical University - Daytona Beach - FL > Ph.D. Candidate, Aerospace Engineering > > M.Sc. Aerospace Engineering > > B.Sc. Aerospace Engineering > > B.Sc. Computational Mathematics > > > *Phone:* (386)-843-1829 > > *Email(s):* ferranj2 at my.erau.edu > > jesus.ferrand at gmail.com > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!e603Zn0jXSCVvzNnzzRiius-0-C5jw6GGzCOFJ7EW5nKTGdX8aI6qchOlBW23rM9TlwMTiidfBsowhNf6eH7$ > > > ------------------------------ > *From:* Matthew Knepley > *Sent:* Wednesday, May 1, 2024 8:07 PM > *To:* Ferrand, Jesus A. > *Cc:* petsc-users at mcs.anl.gov > *Subject:* [EXTERNAL] Re: [petsc-users] Is there anything like a > "DMPlexSetCones()" ? > > *CAUTION:* This email originated outside of Embry-Riddle Aeronautical > University. Do not click links or open attachments unless you recognize the > sender and know the content is safe. > > On Wed, May 1, 2024 at 3:34?PM Ferrand, Jesus A. > wrote: > > This Message Is From an External Sender > This message came from outside your organization. > > Dear PETSc team: > > For a project that I'm working on, I need to manually build a DMPlex. > From studying the source code of the various APIs in which the plex is > built from some supported file format, I get that the workflow is this: > > > 1. DMPlexSetChart() <-- Input nCells + nVerts > 2. DMPlexSetConeSize() <-- Input ConeSize for each point in [0,nCells) > 3. DMSetUp() ? Allocates memory internally. > 4. DMPlexGetCones() --> Gives you the memory onto which to write the > cell connectivity. > 5. *Write connectivity* > 6. DMPlexReorderCell() <-- For each point in [0,nCells) > > > I'm in a situation where the memory given by step (4) is available > a-priori. > I was hoping to skip steps 2, 3 , and 4 with something like a > "DMPlexSetCones()", but such an API does not exist. > My current workaround is to implement steps 2 through 4 as always and have > double the memory allocated in the interim (my instance + DM's internal > instance). > I was thinking of looking for the name of the struct member and = it to my > memory, but I can't overcome the flag check in DMSetUp() during later calls > to DMPlexGetCones() or DMPlexGetTransitiveClosure(). > > > I do not understand the "flag check". What is that? > > We could make a DMPlexSetCones(), but as you point out, the workflow for > DMSetUp() would have to change. Where does the memory come from for your > connectivity? > > Thanks, > > Matt > > > > Sincerely: > > *J.A. Ferrand* > > Embry-Riddle Aeronautical University - Daytona Beach - FL > Ph.D. Candidate, Aerospace Engineering > > M.Sc. Aerospace Engineering > > B.Sc. Aerospace Engineering > > B.Sc. Computational Mathematics > > > *Phone:* (386)-843-1829 > > *Email(s):* ferranj2 at my.erau.edu > > jesus.ferrand at gmail.com > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!e603Zn0jXSCVvzNnzzRiius-0-C5jw6GGzCOFJ7EW5nKTGdX8aI6qchOlBW23rM9TlwMTiidfBsowhNf6eH7$ > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!e603Zn0jXSCVvzNnzzRiius-0-C5jw6GGzCOFJ7EW5nKTGdX8aI6qchOlBW23rM9TlwMTiidfBsowhNf6eH7$ > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!e603Zn0jXSCVvzNnzzRiius-0-C5jw6GGzCOFJ7EW5nKTGdX8aI6qchOlBW23rM9TlwMTiidfBsowhNf6eH7$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From FERRANJ2 at my.erau.edu Fri May 3 18:32:28 2024 From: FERRANJ2 at my.erau.edu (Ferrand, Jesus A.) Date: Fri, 3 May 2024 23:32:28 +0000 Subject: [petsc-users] [EXTERNAL] Re: Is there anything like a "DMPlexSetCones()" ? In-Reply-To: References: Message-ID: "Right, but that means that the connectivity you read in is not in local numbering. You will have to renumber it, so making a copy is probably necessary anyway. This is the same sort of processing I do for the parallel read." ??????True, the connectivity is read in global numbering. ??????Renumbering is facilitated by the table from PetscHashSetI. ??????The connectivity list I am able to renumber in place though, I don't see the need to have another interim copy of the numbers. ??????My concern is that I have two memory chunks that both contain the same information. ??????The only reason It has to be this way is because I can't manually set the Cones first and the Chart/ConeSizes second. ________________________________ From: Matthew Knepley Sent: Friday, May 3, 2024 6:45 PM To: Ferrand, Jesus A. Cc: petsc-users at mcs.anl.gov Subject: Re: [EXTERNAL] Re: [petsc-users] Is there anything like a "DMPlexSetCones()" ? On Wed, May 1, 2024 at 6:30?PM Ferrand, Jesus A. > wrote: "You know the number of cells if you can read connectivity. Do you mean that the format does not tell you the number of vertices?" Sort of... So, the format does provide the number of vertices, however, due to the way I read the data in parallel I don't know immediately how many local vertices there will be. Which prevents me from knowing the chart a priori. I figured out the number of vertices using PetscHashSetI to determine the number of unique entries in the connectivity list. Right, but that means that the connectivity you read in is not in local numbering. You will have to renumber it, so making a copy is probably necessary anyway. This is the same sort of processing I do for the parallel read. Thanks, Matt ________________________________ From: Matthew Knepley > Sent: Wednesday, May 1, 2024 8:52 PM To: Ferrand, Jesus A. > Cc: petsc-users at mcs.anl.gov > Subject: Re: [EXTERNAL] Re: [petsc-users] Is there anything like a "DMPlexSetCones()" ? On Wed, May 1, 2024 at 4:23?PM Ferrand, Jesus A. > wrote: Matt: My bad again, I need to clarify something that I just realized doesn't make sense. I said: "The nature of the I/O makes it so that I need to read the connectivity before I get a semblance of buffer sizes." Scratch that, obviously, if I can read the connectivity, I know the buffer sizes. What I should have said is that I cannot know the "chart" that one sets in DMPlexSetChart() a priori like in most cases. I need to determine the chart from the connectivity lists. You know the number of cells if you can read connectivity. Do you mean that the format does not tell you the number of vertices? Thanks, Matt ________________________________ From: Ferrand, Jesus A. > Sent: Wednesday, May 1, 2024 8:17 PM To: Matthew Knepley > Cc: petsc-users at mcs.anl.gov > Subject: Re: [EXTERNAL] Re: [petsc-users] Is there anything like a "DMPlexSetCones()" ? Matt: "I do not understand the "flag check". What is that?" ??????My bad, I should have referred to the "dm->setupcalled". ??????I believe this PetscBool is checked by the other DM (not just DMPlex) APIs. ??????The subsequent checks for dm->setupcalled == PETSC_TRUE is what I meant to say. ??????Here's a copy of DMSetUp(). PetscErrorCode DMSetUp(DM dm) 817: { 818: PetscFunctionBegin; 820: if (dm->setupcalled) PetscFunctionReturn(PETSC_SUCCESS); 821: PetscTryTypeMethod(dm, setup); 822: dm->setupcalled = PETSC_TRUE; 823: PetscFunctionReturn(PETSC_SUCCESS); 824: } "We could make a DMPlexSetCones(), but as you point out, the workflow for DMSetUp() would have to change. Where does the memory come from for your connectivity?" ??????It's a memory that I myself allocate based on a file's contents. ??????The nature of the I/O makes it so that I need to read the connectivity before I get a semblance of buffer sizes. ??????Otherwise, I would stick to the tried and tested way. Also, when replying to the PETSc developers, users must reply to petsc-users at mcs.anl.gov and not just to the individual email accounts of the developers, right? ________________________________ From: Matthew Knepley > Sent: Wednesday, May 1, 2024 8:07 PM To: Ferrand, Jesus A. > Cc: petsc-users at mcs.anl.gov > Subject: [EXTERNAL] Re: [petsc-users] Is there anything like a "DMPlexSetCones()" ? CAUTION: This email originated outside of Embry-Riddle Aeronautical University. Do not click links or open attachments unless you recognize the sender and know the content is safe. On Wed, May 1, 2024 at 3:34?PM Ferrand, Jesus A. > wrote: This Message Is From an External Sender This message came from outside your organization. Dear PETSc team: For a project that I'm working on, I need to manually build a DMPlex. >From studying the source code of the various APIs in which the plex is built from some supported file format, I get that the workflow is this: 1. DMPlexSetChart() <-- Input nCells + nVerts 2. DMPlexSetConeSize() <-- Input ConeSize for each point in [0,nCells) 3. DMSetUp() ? Allocates memory internally. 4. DMPlexGetCones() --> Gives you the memory onto which to write the cell connectivity. 5. *Write connectivity* 6. DMPlexReorderCell() <-- For each point in [0,nCells) I'm in a situation where the memory given by step (4) is available a-priori. I was hoping to skip steps 2, 3 , and 4 with something like a "DMPlexSetCones()", but such an API does not exist. My current workaround is to implement steps 2 through 4 as always and have double the memory allocated in the interim (my instance + DM's internal instance). I was thinking of looking for the name of the struct member and = it to my memory, but I can't overcome the flag check in DMSetUp() during later calls to DMPlexGetCones() or DMPlexGetTransitiveClosure(). I do not understand the "flag check". What is that? We could make a DMPlexSetCones(), but as you point out, the workflow for DMSetUp() would have to change. Where does the memory come from for your connectivity? Thanks, Matt Sincerely: J.A. Ferrand Embry-Riddle Aeronautical University - Daytona Beach - FL Ph.D. Candidate, Aerospace Engineering M.Sc. Aerospace Engineering B.Sc. Aerospace Engineering B.Sc. Computational Mathematics Phone: (386)-843-1829 Email(s): ferranj2 at my.erau.edu jesus.ferrand at gmail.com -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!c7KVr76YJDliUdUPWmCJIBMhcOTynQ-nax52ZuOkqzWsuI1Lfvv-6S8PGQIjKS_Fru4T84MO2sYpAORILP_FzJbeQPY$ ________________________________ From: Matthew Knepley > Sent: Wednesday, May 1, 2024 8:07 PM To: Ferrand, Jesus A. > Cc: petsc-users at mcs.anl.gov > Subject: [EXTERNAL] Re: [petsc-users] Is there anything like a "DMPlexSetCones()" ? CAUTION: This email originated outside of Embry-Riddle Aeronautical University. Do not click links or open attachments unless you recognize the sender and know the content is safe. On Wed, May 1, 2024 at 3:34?PM Ferrand, Jesus A. > wrote: This Message Is From an External Sender This message came from outside your organization. Dear PETSc team: For a project that I'm working on, I need to manually build a DMPlex. >From studying the source code of the various APIs in which the plex is built from some supported file format, I get that the workflow is this: 1. DMPlexSetChart() <-- Input nCells + nVerts 2. DMPlexSetConeSize() <-- Input ConeSize for each point in [0,nCells) 3. DMSetUp() ? Allocates memory internally. 4. DMPlexGetCones() --> Gives you the memory onto which to write the cell connectivity. 5. *Write connectivity* 6. DMPlexReorderCell() <-- For each point in [0,nCells) I'm in a situation where the memory given by step (4) is available a-priori. I was hoping to skip steps 2, 3 , and 4 with something like a "DMPlexSetCones()", but such an API does not exist. My current workaround is to implement steps 2 through 4 as always and have double the memory allocated in the interim (my instance + DM's internal instance). I was thinking of looking for the name of the struct member and = it to my memory, but I can't overcome the flag check in DMSetUp() during later calls to DMPlexGetCones() or DMPlexGetTransitiveClosure(). I do not understand the "flag check". What is that? We could make a DMPlexSetCones(), but as you point out, the workflow for DMSetUp() would have to change. Where does the memory come from for your connectivity? Thanks, Matt Sincerely: J.A. Ferrand Embry-Riddle Aeronautical University - Daytona Beach - FL Ph.D. Candidate, Aerospace Engineering M.Sc. Aerospace Engineering B.Sc. Aerospace Engineering B.Sc. Computational Mathematics Phone: (386)-843-1829 Email(s): ferranj2 at my.erau.edu jesus.ferrand at gmail.com -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!c7KVr76YJDliUdUPWmCJIBMhcOTynQ-nax52ZuOkqzWsuI1Lfvv-6S8PGQIjKS_Fru4T84MO2sYpAORILP_FzJbeQPY$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!c7KVr76YJDliUdUPWmCJIBMhcOTynQ-nax52ZuOkqzWsuI1Lfvv-6S8PGQIjKS_Fru4T84MO2sYpAORILP_FzJbeQPY$ ________________________________ From: Matthew Knepley > Sent: Wednesday, May 1, 2024 8:52 PM To: Ferrand, Jesus A. > Cc: petsc-users at mcs.anl.gov > Subject: Re: [EXTERNAL] Re: [petsc-users] Is there anything like a "DMPlexSetCones()" ? On Wed, May 1, 2024 at 4:23?PM Ferrand, Jesus A. > wrote: Matt: My bad again, I need to clarify something that I just realized doesn't make sense. I said: "The nature of the I/O makes it so that I need to read the connectivity before I get a semblance of buffer sizes." Scratch that, obviously, if I can read the connectivity, I know the buffer sizes. What I should have said is that I cannot know the "chart" that one sets in DMPlexSetChart() a priori like in most cases. I need to determine the chart from the connectivity lists. You know the number of cells if you can read connectivity. Do you mean that the format does not tell you the number of vertices? Thanks, Matt ________________________________ From: Ferrand, Jesus A. > Sent: Wednesday, May 1, 2024 8:17 PM To: Matthew Knepley > Cc: petsc-users at mcs.anl.gov > Subject: Re: [EXTERNAL] Re: [petsc-users] Is there anything like a "DMPlexSetCones()" ? Matt: "I do not understand the "flag check". What is that?" ??????My bad, I should have referred to the "dm->setupcalled". ??????I believe this PetscBool is checked by the other DM (not just DMPlex) APIs. ??????The subsequent checks for dm->setupcalled == PETSC_TRUE is what I meant to say. ??????Here's a copy of DMSetUp(). PetscErrorCode DMSetUp(DM dm) 817: { 818: PetscFunctionBegin; 820: if (dm->setupcalled) PetscFunctionReturn(PETSC_SUCCESS); 821: PetscTryTypeMethod(dm, setup); 822: dm->setupcalled = PETSC_TRUE; 823: PetscFunctionReturn(PETSC_SUCCESS); 824: } "We could make a DMPlexSetCones(), but as you point out, the workflow for DMSetUp() would have to change. Where does the memory come from for your connectivity?" ??????It's a memory that I myself allocate based on a file's contents. ??????The nature of the I/O makes it so that I need to read the connectivity before I get a semblance of buffer sizes. ??????Otherwise, I would stick to the tried and tested way. Also, when replying to the PETSc developers, users must reply to petsc-users at mcs.anl.gov and not just to the individual email accounts of the developers, right? ________________________________ From: Matthew Knepley > Sent: Wednesday, May 1, 2024 8:07 PM To: Ferrand, Jesus A. > Cc: petsc-users at mcs.anl.gov > Subject: [EXTERNAL] Re: [petsc-users] Is there anything like a "DMPlexSetCones()" ? CAUTION: This email originated outside of Embry-Riddle Aeronautical University. Do not click links or open attachments unless you recognize the sender and know the content is safe. On Wed, May 1, 2024 at 3:34?PM Ferrand, Jesus A. > wrote: This Message Is From an External Sender This message came from outside your organization. Dear PETSc team: For a project that I'm working on, I need to manually build a DMPlex. >From studying the source code of the various APIs in which the plex is built from some supported file format, I get that the workflow is this: 1. DMPlexSetChart() <-- Input nCells + nVerts 2. DMPlexSetConeSize() <-- Input ConeSize for each point in [0,nCells) 3. DMSetUp() ? Allocates memory internally. 4. DMPlexGetCones() --> Gives you the memory onto which to write the cell connectivity. 5. *Write connectivity* 6. DMPlexReorderCell() <-- For each point in [0,nCells) I'm in a situation where the memory given by step (4) is available a-priori. I was hoping to skip steps 2, 3 , and 4 with something like a "DMPlexSetCones()", but such an API does not exist. My current workaround is to implement steps 2 through 4 as always and have double the memory allocated in the interim (my instance + DM's internal instance). I was thinking of looking for the name of the struct member and = it to my memory, but I can't overcome the flag check in DMSetUp() during later calls to DMPlexGetCones() or DMPlexGetTransitiveClosure(). I do not understand the "flag check". What is that? We could make a DMPlexSetCones(), but as you point out, the workflow for DMSetUp() would have to change. Where does the memory come from for your connectivity? Thanks, Matt Sincerely: J.A. Ferrand Embry-Riddle Aeronautical University - Daytona Beach - FL Ph.D. Candidate, Aerospace Engineering M.Sc. Aerospace Engineering B.Sc. Aerospace Engineering B.Sc. Computational Mathematics Phone: (386)-843-1829 Email(s): ferranj2 at my.erau.edu jesus.ferrand at gmail.com -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!c7KVr76YJDliUdUPWmCJIBMhcOTynQ-nax52ZuOkqzWsuI1Lfvv-6S8PGQIjKS_Fru4T84MO2sYpAORILP_FzJbeQPY$ ________________________________ From: Matthew Knepley > Sent: Wednesday, May 1, 2024 8:07 PM To: Ferrand, Jesus A. > Cc: petsc-users at mcs.anl.gov > Subject: [EXTERNAL] Re: [petsc-users] Is there anything like a "DMPlexSetCones()" ? CAUTION: This email originated outside of Embry-Riddle Aeronautical University. Do not click links or open attachments unless you recognize the sender and know the content is safe. On Wed, May 1, 2024 at 3:34?PM Ferrand, Jesus A. > wrote: This Message Is From an External Sender This message came from outside your organization. Dear PETSc team: For a project that I'm working on, I need to manually build a DMPlex. >From studying the source code of the various APIs in which the plex is built from some supported file format, I get that the workflow is this: 1. DMPlexSetChart() <-- Input nCells + nVerts 2. DMPlexSetConeSize() <-- Input ConeSize for each point in [0,nCells) 3. DMSetUp() ? Allocates memory internally. 4. DMPlexGetCones() --> Gives you the memory onto which to write the cell connectivity. 5. *Write connectivity* 6. DMPlexReorderCell() <-- For each point in [0,nCells) I'm in a situation where the memory given by step (4) is available a-priori. I was hoping to skip steps 2, 3 , and 4 with something like a "DMPlexSetCones()", but such an API does not exist. My current workaround is to implement steps 2 through 4 as always and have double the memory allocated in the interim (my instance + DM's internal instance). I was thinking of looking for the name of the struct member and = it to my memory, but I can't overcome the flag check in DMSetUp() during later calls to DMPlexGetCones() or DMPlexGetTransitiveClosure(). I do not understand the "flag check". What is that? We could make a DMPlexSetCones(), but as you point out, the workflow for DMSetUp() would have to change. Where does the memory come from for your connectivity? Thanks, Matt Sincerely: J.A. Ferrand Embry-Riddle Aeronautical University - Daytona Beach - FL Ph.D. Candidate, Aerospace Engineering M.Sc. Aerospace Engineering B.Sc. Aerospace Engineering B.Sc. Computational Mathematics Phone: (386)-843-1829 Email(s): ferranj2 at my.erau.edu jesus.ferrand at gmail.com -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!c7KVr76YJDliUdUPWmCJIBMhcOTynQ-nax52ZuOkqzWsuI1Lfvv-6S8PGQIjKS_Fru4T84MO2sYpAORILP_FzJbeQPY$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!c7KVr76YJDliUdUPWmCJIBMhcOTynQ-nax52ZuOkqzWsuI1Lfvv-6S8PGQIjKS_Fru4T84MO2sYpAORILP_FzJbeQPY$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!c7KVr76YJDliUdUPWmCJIBMhcOTynQ-nax52ZuOkqzWsuI1Lfvv-6S8PGQIjKS_Fru4T84MO2sYpAORILP_FzJbeQPY$ ________________________________ From: Matthew Knepley Sent: Friday, May 3, 2024 6:45 PM To: Ferrand, Jesus A. Cc: petsc-users at mcs.anl.gov Subject: Re: [EXTERNAL] Re: [petsc-users] Is there anything like a "DMPlexSetCones()" ? On Wed, May 1, 2024 at 6:30?PM Ferrand, Jesus A. > wrote: "You know the number of cells if you can read connectivity. Do you mean that the format does not tell you the number of vertices?" Sort of... So, the format does provide the number of vertices, however, due to the way I read the data in parallel I don't know immediately how many local vertices there will be. Which prevents me from knowing the chart a priori. I figured out the number of vertices using PetscHashSetI to determine the number of unique entries in the connectivity list. Right, but that means that the connectivity you read in is not in local numbering. You will have to renumber it, so making a copy is probably necessary anyway. This is the same sort of processing I do for the parallel read. Thanks, Matt ________________________________ From: Matthew Knepley > Sent: Wednesday, May 1, 2024 8:52 PM To: Ferrand, Jesus A. > Cc: petsc-users at mcs.anl.gov > Subject: Re: [EXTERNAL] Re: [petsc-users] Is there anything like a "DMPlexSetCones()" ? On Wed, May 1, 2024 at 4:23?PM Ferrand, Jesus A. > wrote: Matt: My bad again, I need to clarify something that I just realized doesn't make sense. I said: "The nature of the I/O makes it so that I need to read the connectivity before I get a semblance of buffer sizes." Scratch that, obviously, if I can read the connectivity, I know the buffer sizes. What I should have said is that I cannot know the "chart" that one sets in DMPlexSetChart() a priori like in most cases. I need to determine the chart from the connectivity lists. You know the number of cells if you can read connectivity. Do you mean that the format does not tell you the number of vertices? Thanks, Matt ________________________________ From: Ferrand, Jesus A. > Sent: Wednesday, May 1, 2024 8:17 PM To: Matthew Knepley > Cc: petsc-users at mcs.anl.gov > Subject: Re: [EXTERNAL] Re: [petsc-users] Is there anything like a "DMPlexSetCones()" ? Matt: "I do not understand the "flag check". What is that?" ??????My bad, I should have referred to the "dm->setupcalled". ??????I believe this PetscBool is checked by the other DM (not just DMPlex) APIs. ??????The subsequent checks for dm->setupcalled == PETSC_TRUE is what I meant to say. ??????Here's a copy of DMSetUp(). PetscErrorCode DMSetUp(DM dm) 817: { 818: PetscFunctionBegin; 820: if (dm->setupcalled) PetscFunctionReturn(PETSC_SUCCESS); 821: PetscTryTypeMethod(dm, setup); 822: dm->setupcalled = PETSC_TRUE; 823: PetscFunctionReturn(PETSC_SUCCESS); 824: } "We could make a DMPlexSetCones(), but as you point out, the workflow for DMSetUp() would have to change. Where does the memory come from for your connectivity?" ??????It's a memory that I myself allocate based on a file's contents. ??????The nature of the I/O makes it so that I need to read the connectivity before I get a semblance of buffer sizes. ??????Otherwise, I would stick to the tried and tested way. Also, when replying to the PETSc developers, users must reply to petsc-users at mcs.anl.gov and not just to the individual email accounts of the developers, right? ________________________________ From: Matthew Knepley > Sent: Wednesday, May 1, 2024 8:07 PM To: Ferrand, Jesus A. > Cc: petsc-users at mcs.anl.gov > Subject: [EXTERNAL] Re: [petsc-users] Is there anything like a "DMPlexSetCones()" ? CAUTION: This email originated outside of Embry-Riddle Aeronautical University. Do not click links or open attachments unless you recognize the sender and know the content is safe. On Wed, May 1, 2024 at 3:34?PM Ferrand, Jesus A. > wrote: This Message Is From an External Sender This message came from outside your organization. Dear PETSc team: For a project that I'm working on, I need to manually build a DMPlex. >From studying the source code of the various APIs in which the plex is built from some supported file format, I get that the workflow is this: 1. DMPlexSetChart() <-- Input nCells + nVerts 2. DMPlexSetConeSize() <-- Input ConeSize for each point in [0,nCells) 3. DMSetUp() ? Allocates memory internally. 4. DMPlexGetCones() --> Gives you the memory onto which to write the cell connectivity. 5. *Write connectivity* 6. DMPlexReorderCell() <-- For each point in [0,nCells) I'm in a situation where the memory given by step (4) is available a-priori. I was hoping to skip steps 2, 3 , and 4 with something like a "DMPlexSetCones()", but such an API does not exist. My current workaround is to implement steps 2 through 4 as always and have double the memory allocated in the interim (my instance + DM's internal instance). I was thinking of looking for the name of the struct member and = it to my memory, but I can't overcome the flag check in DMSetUp() during later calls to DMPlexGetCones() or DMPlexGetTransitiveClosure(). I do not understand the "flag check". What is that? We could make a DMPlexSetCones(), but as you point out, the workflow for DMSetUp() would have to change. Where does the memory come from for your connectivity? Thanks, Matt Sincerely: J.A. Ferrand Embry-Riddle Aeronautical University - Daytona Beach - FL Ph.D. Candidate, Aerospace Engineering M.Sc. Aerospace Engineering B.Sc. Aerospace Engineering B.Sc. Computational Mathematics Phone: (386)-843-1829 Email(s): ferranj2 at my.erau.edu jesus.ferrand at gmail.com -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!c7KVr76YJDliUdUPWmCJIBMhcOTynQ-nax52ZuOkqzWsuI1Lfvv-6S8PGQIjKS_Fru4T84MO2sYpAORILP_FzJbeQPY$ ________________________________ From: Matthew Knepley > Sent: Wednesday, May 1, 2024 8:07 PM To: Ferrand, Jesus A. > Cc: petsc-users at mcs.anl.gov > Subject: [EXTERNAL] Re: [petsc-users] Is there anything like a "DMPlexSetCones()" ? CAUTION: This email originated outside of Embry-Riddle Aeronautical University. Do not click links or open attachments unless you recognize the sender and know the content is safe. On Wed, May 1, 2024 at 3:34?PM Ferrand, Jesus A. > wrote: This Message Is From an External Sender This message came from outside your organization. Dear PETSc team: For a project that I'm working on, I need to manually build a DMPlex. >From studying the source code of the various APIs in which the plex is built from some supported file format, I get that the workflow is this: 1. DMPlexSetChart() <-- Input nCells + nVerts 2. DMPlexSetConeSize() <-- Input ConeSize for each point in [0,nCells) 3. DMSetUp() ? Allocates memory internally. 4. DMPlexGetCones() --> Gives you the memory onto which to write the cell connectivity. 5. *Write connectivity* 6. DMPlexReorderCell() <-- For each point in [0,nCells) I'm in a situation where the memory given by step (4) is available a-priori. I was hoping to skip steps 2, 3 , and 4 with something like a "DMPlexSetCones()", but such an API does not exist. My current workaround is to implement steps 2 through 4 as always and have double the memory allocated in the interim (my instance + DM's internal instance). I was thinking of looking for the name of the struct member and = it to my memory, but I can't overcome the flag check in DMSetUp() during later calls to DMPlexGetCones() or DMPlexGetTransitiveClosure(). I do not understand the "flag check". What is that? We could make a DMPlexSetCones(), but as you point out, the workflow for DMSetUp() would have to change. Where does the memory come from for your connectivity? Thanks, Matt Sincerely: J.A. Ferrand Embry-Riddle Aeronautical University - Daytona Beach - FL Ph.D. Candidate, Aerospace Engineering M.Sc. Aerospace Engineering B.Sc. Aerospace Engineering B.Sc. Computational Mathematics Phone: (386)-843-1829 Email(s): ferranj2 at my.erau.edu jesus.ferrand at gmail.com -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!c7KVr76YJDliUdUPWmCJIBMhcOTynQ-nax52ZuOkqzWsuI1Lfvv-6S8PGQIjKS_Fru4T84MO2sYpAORILP_FzJbeQPY$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!c7KVr76YJDliUdUPWmCJIBMhcOTynQ-nax52ZuOkqzWsuI1Lfvv-6S8PGQIjKS_Fru4T84MO2sYpAORILP_FzJbeQPY$ ________________________________ From: Matthew Knepley > Sent: Wednesday, May 1, 2024 8:52 PM To: Ferrand, Jesus A. > Cc: petsc-users at mcs.anl.gov > Subject: Re: [EXTERNAL] Re: [petsc-users] Is there anything like a "DMPlexSetCones()" ? On Wed, May 1, 2024 at 4:23?PM Ferrand, Jesus A. > wrote: Matt: My bad again, I need to clarify something that I just realized doesn't make sense. I said: "The nature of the I/O makes it so that I need to read the connectivity before I get a semblance of buffer sizes." Scratch that, obviously, if I can read the connectivity, I know the buffer sizes. What I should have said is that I cannot know the "chart" that one sets in DMPlexSetChart() a priori like in most cases. I need to determine the chart from the connectivity lists. You know the number of cells if you can read connectivity. Do you mean that the format does not tell you the number of vertices? Thanks, Matt ________________________________ From: Ferrand, Jesus A. > Sent: Wednesday, May 1, 2024 8:17 PM To: Matthew Knepley > Cc: petsc-users at mcs.anl.gov > Subject: Re: [EXTERNAL] Re: [petsc-users] Is there anything like a "DMPlexSetCones()" ? Matt: "I do not understand the "flag check". What is that?" ??????My bad, I should have referred to the "dm->setupcalled". ??????I believe this PetscBool is checked by the other DM (not just DMPlex) APIs. ??????The subsequent checks for dm->setupcalled == PETSC_TRUE is what I meant to say. ??????Here's a copy of DMSetUp(). PetscErrorCode DMSetUp(DM dm) 817: { 818: PetscFunctionBegin; 820: if (dm->setupcalled) PetscFunctionReturn(PETSC_SUCCESS); 821: PetscTryTypeMethod(dm, setup); 822: dm->setupcalled = PETSC_TRUE; 823: PetscFunctionReturn(PETSC_SUCCESS); 824: } "We could make a DMPlexSetCones(), but as you point out, the workflow for DMSetUp() would have to change. Where does the memory come from for your connectivity?" ??????It's a memory that I myself allocate based on a file's contents. ??????The nature of the I/O makes it so that I need to read the connectivity before I get a semblance of buffer sizes. ??????Otherwise, I would stick to the tried and tested way. Also, when replying to the PETSc developers, users must reply to petsc-users at mcs.anl.gov and not just to the individual email accounts of the developers, right? ________________________________ From: Matthew Knepley > Sent: Wednesday, May 1, 2024 8:07 PM To: Ferrand, Jesus A. > Cc: petsc-users at mcs.anl.gov > Subject: [EXTERNAL] Re: [petsc-users] Is there anything like a "DMPlexSetCones()" ? CAUTION: This email originated outside of Embry-Riddle Aeronautical University. Do not click links or open attachments unless you recognize the sender and know the content is safe. On Wed, May 1, 2024 at 3:34?PM Ferrand, Jesus A. > wrote: This Message Is From an External Sender This message came from outside your organization. Dear PETSc team: For a project that I'm working on, I need to manually build a DMPlex. >From studying the source code of the various APIs in which the plex is built from some supported file format, I get that the workflow is this: 1. DMPlexSetChart() <-- Input nCells + nVerts 2. DMPlexSetConeSize() <-- Input ConeSize for each point in [0,nCells) 3. DMSetUp() ? Allocates memory internally. 4. DMPlexGetCones() --> Gives you the memory onto which to write the cell connectivity. 5. *Write connectivity* 6. DMPlexReorderCell() <-- For each point in [0,nCells) I'm in a situation where the memory given by step (4) is available a-priori. I was hoping to skip steps 2, 3 , and 4 with something like a "DMPlexSetCones()", but such an API does not exist. My current workaround is to implement steps 2 through 4 as always and have double the memory allocated in the interim (my instance + DM's internal instance). I was thinking of looking for the name of the struct member and = it to my memory, but I can't overcome the flag check in DMSetUp() during later calls to DMPlexGetCones() or DMPlexGetTransitiveClosure(). I do not understand the "flag check". What is that? We could make a DMPlexSetCones(), but as you point out, the workflow for DMSetUp() would have to change. Where does the memory come from for your connectivity? Thanks, Matt Sincerely: J.A. Ferrand Embry-Riddle Aeronautical University - Daytona Beach - FL Ph.D. Candidate, Aerospace Engineering M.Sc. Aerospace Engineering B.Sc. Aerospace Engineering B.Sc. Computational Mathematics Phone: (386)-843-1829 Email(s): ferranj2 at my.erau.edu jesus.ferrand at gmail.com -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!c7KVr76YJDliUdUPWmCJIBMhcOTynQ-nax52ZuOkqzWsuI1Lfvv-6S8PGQIjKS_Fru4T84MO2sYpAORILP_FzJbeQPY$ ________________________________ From: Matthew Knepley > Sent: Wednesday, May 1, 2024 8:07 PM To: Ferrand, Jesus A. > Cc: petsc-users at mcs.anl.gov > Subject: [EXTERNAL] Re: [petsc-users] Is there anything like a "DMPlexSetCones()" ? CAUTION: This email originated outside of Embry-Riddle Aeronautical University. Do not click links or open attachments unless you recognize the sender and know the content is safe. On Wed, May 1, 2024 at 3:34?PM Ferrand, Jesus A. > wrote: This Message Is From an External Sender This message came from outside your organization. Dear PETSc team: For a project that I'm working on, I need to manually build a DMPlex. >From studying the source code of the various APIs in which the plex is built from some supported file format, I get that the workflow is this: 1. DMPlexSetChart() <-- Input nCells + nVerts 2. DMPlexSetConeSize() <-- Input ConeSize for each point in [0,nCells) 3. DMSetUp() ? Allocates memory internally. 4. DMPlexGetCones() --> Gives you the memory onto which to write the cell connectivity. 5. *Write connectivity* 6. DMPlexReorderCell() <-- For each point in [0,nCells) I'm in a situation where the memory given by step (4) is available a-priori. I was hoping to skip steps 2, 3 , and 4 with something like a "DMPlexSetCones()", but such an API does not exist. My current workaround is to implement steps 2 through 4 as always and have double the memory allocated in the interim (my instance + DM's internal instance). I was thinking of looking for the name of the struct member and = it to my memory, but I can't overcome the flag check in DMSetUp() during later calls to DMPlexGetCones() or DMPlexGetTransitiveClosure(). I do not understand the "flag check". What is that? We could make a DMPlexSetCones(), but as you point out, the workflow for DMSetUp() would have to change. Where does the memory come from for your connectivity? Thanks, Matt Sincerely: J.A. Ferrand Embry-Riddle Aeronautical University - Daytona Beach - FL Ph.D. Candidate, Aerospace Engineering M.Sc. Aerospace Engineering B.Sc. Aerospace Engineering B.Sc. Computational Mathematics Phone: (386)-843-1829 Email(s): ferranj2 at my.erau.edu jesus.ferrand at gmail.com -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!c7KVr76YJDliUdUPWmCJIBMhcOTynQ-nax52ZuOkqzWsuI1Lfvv-6S8PGQIjKS_Fru4T84MO2sYpAORILP_FzJbeQPY$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!c7KVr76YJDliUdUPWmCJIBMhcOTynQ-nax52ZuOkqzWsuI1Lfvv-6S8PGQIjKS_Fru4T84MO2sYpAORILP_FzJbeQPY$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!c7KVr76YJDliUdUPWmCJIBMhcOTynQ-nax52ZuOkqzWsuI1Lfvv-6S8PGQIjKS_Fru4T84MO2sYpAORILP_FzJbeQPY$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From techvalueinc at protonmail.com Sun May 5 23:38:19 2024 From: techvalueinc at protonmail.com (techvalueinc) Date: Mon, 06 May 2024 04:38:19 +0000 Subject: [petsc-users] ATELIER DU METAL MAROC Message-ID: ATELIER DU METAL MAROC EST LEADER DANS LE DOMAINE DE LA CONSTRUCTION ET D?CORATION EN INOX OU ALUMINIUM, Travaux Menuiserie inox, Garde corps, 304L , 316L , Rampe escaliers , Passe ? main , Porte inox, Verre ? profil? , Grille , Muret , Terrasse, Cuisine en acier inoxydable pour restaurants, caf?s, Patisserie, Boulangerie Chaudronnerie et maisons etc. Tables , Comptoir inox , les Hottes inox, Double Paroi, Cuve, R?servoir. MOBILIER ET AGENCEMENTS POUR PHARMACIES D?H?PITAUX Le secteur pharmaceutique est soumis ? des exigences s?v?res en mati?re d?hygi?ne afin de lutter contre la prolif?ration des micro-organismes ind?sirables (bact?ries et moisissures). L?acier inoxydable est justement pl?biscit? par ces industries pharmaceutiques et cosm?tiques, pour des raisons d?hygi?ne, de s?curit? et de nettoyage. Cette prolif?ration s?organise notamment dans les microfissures et autres d?fauts, sur des mat?riaux rugueux. Ce qui n?est pas le cas de l?inox. Ainsi que, rugosit? surfacique et capacit? de nettoyage ?tant directement li?es, l?inox est un mat?riau qui se nettoie facilement. De plus, l?inox ne produit aucune substance pouvant s?introduire dans le produit final. Cette surface ?tant inerte, cela ?vite que des substances entrent dans le produit fini. ATELIER DU M?TAL MAROC vous offre des solutions innovantes et s?curisantes pour vous permettre de mener ? bien vos objectifs. Fabrication des mobiliers m?dical en inox. Tables d?examen, Fauteuils gyn?cologiques, Escabeaux, Paravents, Tabourets, Gu?ridons, N?gatoscopes et autresCatalogue de Tables de Travail Inox | Plonges | Chariots | Etagers Inox [https://urldefense.us/v3/__https://atelierdumetal.ma/fabrication-inox-au-maroc-pour-professionnel*(https:/*atelierdumetal.ma/fabrication-inox-au-maroc-pour-professionnel/)__;XS8!!G_uCfscf7eWS!arL4lDrmCOqVyUmyHdX3mZ396swEM1wvibhyjq-LZfpGu68XF13uNT-le5zwPoEs4u_r3Fx3LVqJCsmu54HI2EmOH_rSuYk$ ATELIER DU M?TAL MAROC est votre partenaire de confiance pour fabriquer des mobilier INOX pour cuisine conformes aux normes sanitaires et de s?curit?. Nos produits en acier inoxydable sont con?us pour r?pondre aux exigences de nos clients en mati?re d?hygi?ne et de facilit? de nettoyage, tout en offrant une durabilit? pour r?sister aux rigueurs d?une utilisation quotidienne : Des tables Inox avec Dosseret ou Sans , Tables de chef, Tables de pr?paration, Lave Main P?dale, Lave Main ? genou, Tables d?enter / sortie?.. etc. ATELIER DU M?TAL MAROC propose ?galement des services d?installation professionnels pour garantir que votre ?quipement de cuisine est correctement assembl? et fonctionne de mani?re optimale dans votre restaurant. https://urldefense.us/v3/__https://atelierdumetal.ma/__;!!G_uCfscf7eWS!arL4lDrmCOqVyUmyHdX3mZ396swEM1wvibhyjq-LZfpGu68XF13uNT-le5zwPoEs4u_r3Fx3LVqJCsmu54HI2EmOwY8GrHs$ Plonge et Eviers en inox | Chariots en inox | Mobilier Inox snack et Pizzeria | Table de travail inox | hotte aspirante Inox professionnelle | Meuble ? Caf? Inox | Armoires en inox | Soubassements Inox Ouverts ou Ferm?s | Table entr?e et sortie | Table et Tiroirs placards mural Inox | Table de travail et pr?paration en inox | ?tag?res en inox | Rayonnages en inox | lave-mains inox | barbecue en inox | ?chelles p?tisserie Inox Mobilier Inox Sur mesure pour caf? ? Mobilier Inox Snack et Pizzeria ? Mobilier Inox Boucherie ? Mobilier Inox Boulangerie ? Chaises et tables caf? ? ?quipement caf? Maroc ? Mat?riel de Buanderie ? Mat?riel de Restaurant ? Mat?riel de Boucherie ? Mat?riel Snack et Pizzeria ? Mat?riel Boulangerie et P?tisserie -------------- next part -------------- An HTML attachment was scrubbed... URL: From techvalueinc at protonmail.com Sun May 5 23:42:01 2024 From: techvalueinc at protonmail.com (techvalueinc) Date: Mon, 06 May 2024 04:42:01 +0000 Subject: [petsc-users] ATELIER DU METAL MAROC In-Reply-To: References: Message-ID: ATELIER DU METAL MAROC EST LEADER DANS LE DOMAINE DE LA CONSTRUCTION ET D?CORATION EN INOX OU ALUMINIUM, Travaux Menuiserie inox, Garde corps, 304L , 316L , Rampe escaliers , Passe ? main , Porte inox, Verre ? profil? , Grille , Muret , Terrasse, Cuisine en acier inoxydable pour restaurants, caf?s, Patisserie, Boulangerie Chaudronnerie et maisons etc. Tables , Comptoir inox , les Hottes inox, Double Paroi, Cuve, R?servoir. MOBILIER ET AGENCEMENTS POUR PHARMACIES D?H?PITAUX Le secteur pharmaceutique est soumis ? des exigences s?v?res en mati?re d?hygi?ne afin de lutter contre la prolif?ration des micro-organismes ind?sirables (bact?ries et moisissures). L?acier inoxydable est justement pl?biscit? par ces industries pharmaceutiques et cosm?tiques, pour des raisons d?hygi?ne, de s?curit? et de nettoyage. Cette prolif?ration s?organise notamment dans les microfissures et autres d?fauts, sur des mat?riaux rugueux. Ce qui n?est pas le cas de l?inox. Ainsi que, rugosit? surfacique et capacit? de nettoyage ?tant directement li?es, l?inox est un mat?riau qui se nettoie facilement. De plus, l?inox ne produit aucune substance pouvant s?introduire dans le produit final. Cette surface ?tant inerte, cela ?vite que des substances entrent dans le produit fini. ATELIER DU M?TAL MAROC vous offre des solutions innovantes et s?curisantes pour vous permettre de mener ? bien vos objectifs. Fabrication des mobiliers m?dical en inox. Tables d?examen, Fauteuils gyn?cologiques, Escabeaux, Paravents, Tabourets, Gu?ridons, N?gatoscopes et autresCatalogue de Tables de Travail Inox | Plonges | Chariots | Etagers Inox [https://urldefense.us/v3/__https://atelierdumetal.ma/fabrication-inox-au-maroc-pour-professionnel*(https:/*urldefense.us/v3/__https:/*atelierdumetal.ma/fabrication-inox-au-maroc-pour-professionnel/__;!!G_uCfscf7eWS!arL4lDrmCOqVyUmyHdX3mZ396swEM1wvibhyjq-LZfpGu68XF13uNT-le5zwPoEs4u_r3Fx3LVqJCsmu54HI2EmO_I_xG4s$)__;XS8v!!G_uCfscf7eWS!bj0Rvn-ai6WLxNkfIV14x8hl69m1ch0-QBBtQu-a1po3izWNXIV7UpTKAuMFgGc9r2ILHPP9L-Bp3L1ywF6GSICbsc25gC0$ ATELIER DU M?TAL MAROC est votre partenaire de confiance pour fabriquer des mobilier INOX pour cuisine conformes aux normes sanitaires et de s?curit?. Nos produits en acier inoxydable sont con?us pour r?pondre aux exigences de nos clients en mati?re d?hygi?ne et de facilit? de nettoyage, tout en offrant une durabilit? pour r?sister aux rigueurs d?une utilisation quotidienne : Des tables Inox avec Dosseret ou Sans , Tables de chef, Tables de pr?paration, Lave Main P?dale, Lave Main ? genou, Tables d?enter / sortie?.. etc. ATELIER DU M?TAL MAROC propose ?galement des services d?installation professionnels pour garantir que votre ?quipement de cuisine est correctement assembl? et fonctionne de mani?re optimale dans votre restaurant. [https://urldefense.us/v3/__https://atelierdumetal.ma/*(https:/*urldefense.us/v3/__https:/*atelierdumetal.ma/__;!!G_uCfscf7eWS!arL4lDrmCOqVyUmyHdX3mZ396swEM1wvibhyjq-LZfpGu68XF13uNT-le5zwPoEs4u_r3Fx3LVqJCsmu54HI2EmOwY8GrHs$)__;XS8v!!G_uCfscf7eWS!bj0Rvn-ai6WLxNkfIV14x8hl69m1ch0-QBBtQu-a1po3izWNXIV7UpTKAuMFgGc9r2ILHPP9L-Bp3L1ywF6GSICbLolvy9s$ Plonge et Eviers en inox | Chariots en inox | Mobilier Inox snack et Pizzeria | Table de travail inox | hotte aspirante Inox professionnelle | Meuble ? Caf? Inox | Armoires en inox | Soubassements Inox Ouverts ou Ferm?s | Table entr?e et sortie | Table et Tiroirs placards mural Inox | Table de travail et pr?paration en inox | ?tag?res en inox | Rayonnages en inox | lave-mains inox | barbecue en inox | ?chelles p?tisserie Inox Mobilier Inox Sur mesure pour caf? ? Mobilier Inox Snack et Pizzeria ? Mobilier Inox Boucherie ? Mobilier Inox Boulangerie ? Chaises et tables caf? ? ?quipement caf? Maroc ? Mat?riel de Buanderie ? Mat?riel de Restaurant ? Mat?riel de Boucherie ? Mat?riel Snack et Pizzeria ? Mat?riel Boulangerie et P?tisserie Le lundi 6 mai 2024 ? 05:38, techvalueinc via petsc-users a ?crit : > ATELIER DU METAL MAROC EST LEADER DANS LE DOMAINE DE LA CONSTRUCTION ET D?CORATION EN INOX OU ALUMINIUM, Travaux Menuiserie inox, Garde corps, 304L , 316L , Rampe escaliers , Passe ? main , Porte inox, Verre ? profil? , Grille , Muret , Terrasse, > ZjQcmQRYFpfptBannerStart > This Message Is From an External Sender > This message came from outside your organization. > > ZjQcmQRYFpfptBannerEnd > ATELIER DU METAL MAROC EST LEADER DANS LE DOMAINE DE LA CONSTRUCTION ET > > D?CORATION EN INOX OU ALUMINIUM, Travaux Menuiserie inox, Garde corps, 304L > , 316L , Rampe escaliers , Passe ? main , Porte inox, Verre ? profil? , > Grille , Muret , Terrasse, Cuisine en acier inoxydable pour restaurants, > caf?s, Patisserie, Boulangerie Chaudronnerie et maisons etc. Tables , > Comptoir inox , les Hottes inox, Double Paroi, Cuve, R?servoir. > MOBILIER ET AGENCEMENTS POUR PHARMACIES D?H?PITAUX > > Le secteur pharmaceutique est soumis ? des exigences s?v?res en mati?re > d?hygi?ne afin de lutter contre la prolif?ration des micro-organismes > ind?sirables (bact?ries et moisissures). L?acier inoxydable est justement > pl?biscit? par ces industries pharmaceutiques et cosm?tiques, pour des > raisons d?hygi?ne, de s?curit? et de nettoyage. Cette prolif?ration > s?organise notamment dans les microfissures et autres d?fauts, sur des > mat?riaux rugueux. Ce qui n?est pas le cas de l?inox. Ainsi que, rugosit? > surfacique et capacit? de nettoyage ?tant directement li?es, l?inox est un > mat?riau qui se nettoie facilement. De plus, l?inox ne produit aucune > substance pouvant s?introduire dans le produit final. Cette surface ?tant > inerte, cela ?vite que des substances entrent dans le produit fini. > > ATELIER DU M?TAL MAROC vous offre des solutions innovantes et s?curisantes > pour vous permettre de mener ? bien vos objectifs. > Fabrication des mobiliers m?dical en inox. Tables d?examen, Fauteuils > gyn?cologiques, Escabeaux, Paravents, Tabourets, Gu?ridons, N?gatoscopes et > autresCatalogue de Tables de Travail Inox | Plonges | Chariots | Etagers > Inox > [https://urldefense.us/v3/__https://atelierdumetal.ma/fabrication-inox-au-maroc-pour-professionnel*(https:/*urldefense.us/v3/__https:/*atelierdumetal.ma/fabrication-inox-au-maroc-pour-professionnel/__;!!G_uCfscf7eWS!arL4lDrmCOqVyUmyHdX3mZ396swEM1wvibhyjq-LZfpGu68XF13uNT-le5zwPoEs4u_r3Fx3LVqJCsmu54HI2EmO_I_xG4s$)__;XS8v!!G_uCfscf7eWS!bj0Rvn-ai6WLxNkfIV14x8hl69m1ch0-QBBtQu-a1po3izWNXIV7UpTKAuMFgGc9r2ILHPP9L-Bp3L1ywF6GSICbsc25gC0$ > ATELIER DU M?TAL MAROC est votre partenaire de confiance pour fabriquer > des mobilier INOX pour cuisine conformes aux normes sanitaires et de > s?curit?. Nos produits en acier inoxydable sont con?us pour r?pondre aux > exigences de nos clients en mati?re d?hygi?ne et de facilit? de nettoyage, > tout en offrant une durabilit? pour r?sister aux rigueurs d?une utilisation > quotidienne : Des tables Inox avec Dosseret ou Sans , Tables de chef, > Tables de pr?paration, Lave Main P?dale, Lave Main ? genou, Tables d?enter > / sortie?.. etc. ATELIER DU M?TAL MAROC propose ?galement des services > d?installation professionnels pour garantir que votre ?quipement de cuisine > est correctement assembl? et fonctionne de mani?re optimale dans votre > restaurant. > [https://urldefense.us/v3/__https://atelierdumetal.ma/*(https:/*urldefense.us/v3/__https:/*atelierdumetal.ma/__;!!G_uCfscf7eWS!arL4lDrmCOqVyUmyHdX3mZ396swEM1wvibhyjq-LZfpGu68XF13uNT-le5zwPoEs4u_r3Fx3LVqJCsmu54HI2EmOwY8GrHs$)__;XS8v!!G_uCfscf7eWS!bj0Rvn-ai6WLxNkfIV14x8hl69m1ch0-QBBtQu-a1po3izWNXIV7UpTKAuMFgGc9r2ILHPP9L-Bp3L1ywF6GSICbLolvy9s$ > Plonge et Eviers en inox | Chariots en inox | Mobilier Inox snack et > Pizzeria | Table de travail inox | hotte aspirante Inox professionnelle | > Meuble ? Caf? Inox | Armoires en inox | Soubassements Inox Ouverts ou > Ferm?s | Table entr?e et sortie | Table et Tiroirs placards mural Inox | > Table de travail et pr?paration en inox | ?tag?res en inox | Rayonnages en > inox | lave-mains inox | barbecue en inox | ?chelles p?tisserie Inox Mobilier > Inox Sur mesure pour caf? ? Mobilier Inox Snack et Pizzeria ? Mobilier Inox > Boucherie ? Mobilier Inox Boulangerie ? Chaises et tables > caf? ? ?quipement caf? Maroc ? Mat?riel de Buanderie ? Mat?riel de > Restaurant ? Mat?riel de Boucherie ? Mat?riel Snack et Pizzeria ? Mat?riel > Boulangerie et P?tisserie -------------- next part -------------- An HTML attachment was scrubbed... URL: From a.croucher at auckland.ac.nz Mon May 6 00:04:08 2024 From: a.croucher at auckland.ac.nz (Adrian Croucher) Date: Mon, 6 May 2024 17:04:08 +1200 Subject: [petsc-users] PETSc options Message-ID: <2d56c6ea-e9e0-4a7a-b565-90a1404e08b7@auckland.ac.nz> An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon May 6 06:52:04 2024 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 6 May 2024 07:52:04 -0400 Subject: [petsc-users] [EXTERNAL] Re: Is there anything like a "DMPlexSetCones()" ? In-Reply-To: References: Message-ID: On Fri, May 3, 2024 at 7:32?PM Ferrand, Jesus A. wrote: > "Right, but that means that the connectivity you read in is not in local > numbering. You will have to renumber it, so making a copy is probably > necessary anyway. This is the same sort of processing I do for the parallel > read." > > True, the connectivity is read in global numbering. > Renumbering is facilitated by the table from PetscHashSetI. > The connectivity list I am able to renumber in place though, I don't see > the need to have another interim copy of the numbers. > My concern is that I have two memory chunks that both contain the same > information. > The only reason It has to be this way is because I can't manually set the > Cones first and the Chart/ConeSizes second. > The cone sizes are used to allocate the array to hold the cones. How would you have an array holding the cone information without knowing the sizes? Thanks, Matt > ------------------------------ > *From:* Matthew Knepley > *Sent:* Friday, May 3, 2024 6:45 PM > *To:* Ferrand, Jesus A. > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [EXTERNAL] Re: [petsc-users] Is there anything like a > "DMPlexSetCones()" ? > > On Wed, May 1, 2024 at 6:30?PM Ferrand, Jesus A. > wrote: > > "You know the number of cells if you can read connectivity. Do you mean > that the format does not tell you > the number of vertices?" > > Sort of... > So, the format does provide the number of vertices, however, due to the > way I read the data in parallel I don't know immediately how many local > vertices there will be. > Which prevents me from knowing the chart a priori. > I figured out the number of vertices using PetscHashSetI to determine the > number of unique entries in the connectivity list. > > > Right, but that means that the connectivity you read in is not in local > numbering. You will have to renumber it, so making a copy is probably > necessary anyway. This is the same sort of processing I do for the parallel > read. > > Thanks, > > Matt > > > ------------------------------ > *From:* Matthew Knepley > *Sent:* Wednesday, May 1, 2024 8:52 PM > *To:* Ferrand, Jesus A. > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [EXTERNAL] Re: [petsc-users] Is there anything like a > "DMPlexSetCones()" ? > > On Wed, May 1, 2024 at 4:23?PM Ferrand, Jesus A. > wrote: > > Matt: > > My bad again, I need to clarify something that I just realized doesn't > make sense. > I said: "The nature of the I/O makes it so that I need to read the > connectivity before I get a semblance of buffer sizes." > Scratch that, obviously, if I can read the connectivity, I know the buffer > sizes. > > What I should have said is that I cannot know the "chart" that one sets in > DMPlexSetChart() a priori like in most cases. > I need to determine the chart from the connectivity lists. > > > You know the number of cells if you can read connectivity. Do you mean > that the format does not tell you > the number of vertices? > > Thanks, > > Matt > > > ------------------------------ > *From:* Ferrand, Jesus A. > *Sent:* Wednesday, May 1, 2024 8:17 PM > *To:* Matthew Knepley > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [EXTERNAL] Re: [petsc-users] Is there anything like a > "DMPlexSetCones()" ? > > Matt: > > "I do not understand the "flag check". What is that?" > > My bad, I should have referred to the "dm->setupcalled". > I believe this PetscBool is checked by the other DM (not just DMPlex) APIs. > The subsequent checks for dm->setupcalled == PETSC_TRUE is what I meant to > say. > Here's a copy of DMSetUp(). > > *PetscErrorCode DMSetUp (DM dm)* > 817: { > 818: PetscFunctionBegin ; > 820: if (dm->setupcalled) PetscFunctionReturn (PETSC_SUCCESS ); > 821: PetscTryTypeMethod (dm, setup); > 822: dm->setupcalled = PETSC_TRUE ; > 823: PetscFunctionReturn (PETSC_SUCCESS ); > 824: } > > "We could make a DMPlexSetCones(), but as you point out, the workflow for > DMSetUp() would have to change. Where does the memory come from for your > connectivity?" > It's a memory that I myself allocate based on a file's contents. > The nature of the I/O makes it so that I need to read the connectivity > before I get a semblance of buffer sizes. > Otherwise, I would stick to the tried and tested way. > > Also, when replying to the PETSc developers, users must reply to > petsc-users at mcs.anl.gov and not just to the individual email accounts of > the developers, right? > > > > > ------------------------------ > *From:* Matthew Knepley > *Sent:* Wednesday, May 1, 2024 8:07 PM > *To:* Ferrand, Jesus A. > *Cc:* petsc-users at mcs.anl.gov > *Subject:* [EXTERNAL] Re: [petsc-users] Is there anything like a > "DMPlexSetCones()" ? > > *CAUTION:* This email originated outside of Embry-Riddle Aeronautical > University. Do not click links or open attachments unless you recognize the > sender and know the content is safe. > > On Wed, May 1, 2024 at 3:34?PM Ferrand, Jesus A. > wrote: > > This Message Is From an External Sender > This message came from outside your organization. > > Dear PETSc team: > > For a project that I'm working on, I need to manually build a DMPlex. > From studying the source code of the various APIs in which the plex is > built from some supported file format, I get that the workflow is this: > > > 1. DMPlexSetChart() <-- Input nCells + nVerts > 2. DMPlexSetConeSize() <-- Input ConeSize for each point in [0,nCells) > 3. DMSetUp() ? Allocates memory internally. > 4. DMPlexGetCones() --> Gives you the memory onto which to write the > cell connectivity. > 5. *Write connectivity* > 6. DMPlexReorderCell() <-- For each point in [0,nCells) > > > I'm in a situation where the memory given by step (4) is available > a-priori. > I was hoping to skip steps 2, 3 , and 4 with something like a > "DMPlexSetCones()", but such an API does not exist. > My current workaround is to implement steps 2 through 4 as always and have > double the memory allocated in the interim (my instance + DM's internal > instance). > I was thinking of looking for the name of the struct member and = it to my > memory, but I can't overcome the flag check in DMSetUp() during later calls > to DMPlexGetCones() or DMPlexGetTransitiveClosure(). > > > I do not understand the "flag check". What is that? > > We could make a DMPlexSetCones(), but as you point out, the workflow for > DMSetUp() would have to change. Where does the memory come from for your > connectivity? > > Thanks, > > Matt > > > > Sincerely: > > *J.A. Ferrand* > > Embry-Riddle Aeronautical University - Daytona Beach - FL > Ph.D. Candidate, Aerospace Engineering > > M.Sc. Aerospace Engineering > > B.Sc. Aerospace Engineering > > B.Sc. Computational Mathematics > > > *Phone:* (386)-843-1829 > > *Email(s):* ferranj2 at my.erau.edu > > jesus.ferrand at gmail.com > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fO0pwSt99ISQDFM7Ioe1NhbBE_v7qVVrHxyaL60C6opn3c9PYf5hK7WjgrAQJUZh1UQDk3UhZUjZelBUE2lU$ > > > ------------------------------ > *From:* Matthew Knepley > *Sent:* Wednesday, May 1, 2024 8:07 PM > *To:* Ferrand, Jesus A. > *Cc:* petsc-users at mcs.anl.gov > *Subject:* [EXTERNAL] Re: [petsc-users] Is there anything like a > "DMPlexSetCones()" ? > > *CAUTION:* This email originated outside of Embry-Riddle Aeronautical > University. Do not click links or open attachments unless you recognize the > sender and know the content is safe. > > On Wed, May 1, 2024 at 3:34?PM Ferrand, Jesus A. > wrote: > > This Message Is From an External Sender > This message came from outside your organization. > > Dear PETSc team: > > For a project that I'm working on, I need to manually build a DMPlex. > From studying the source code of the various APIs in which the plex is > built from some supported file format, I get that the workflow is this: > > > 1. DMPlexSetChart() <-- Input nCells + nVerts > 2. DMPlexSetConeSize() <-- Input ConeSize for each point in [0,nCells) > 3. DMSetUp() ? Allocates memory internally. > 4. DMPlexGetCones() --> Gives you the memory onto which to write the > cell connectivity. > 5. *Write connectivity* > 6. DMPlexReorderCell() <-- For each point in [0,nCells) > > > I'm in a situation where the memory given by step (4) is available > a-priori. > I was hoping to skip steps 2, 3 , and 4 with something like a > "DMPlexSetCones()", but such an API does not exist. > My current workaround is to implement steps 2 through 4 as always and have > double the memory allocated in the interim (my instance + DM's internal > instance). > I was thinking of looking for the name of the struct member and = it to my > memory, but I can't overcome the flag check in DMSetUp() during later calls > to DMPlexGetCones() or DMPlexGetTransitiveClosure(). > > > I do not understand the "flag check". What is that? > > We could make a DMPlexSetCones(), but as you point out, the workflow for > DMSetUp() would have to change. Where does the memory come from for your > connectivity? > > Thanks, > > Matt > > > > Sincerely: > > *J.A. Ferrand* > > Embry-Riddle Aeronautical University - Daytona Beach - FL > Ph.D. Candidate, Aerospace Engineering > > M.Sc. Aerospace Engineering > > B.Sc. Aerospace Engineering > > B.Sc. Computational Mathematics > > > *Phone:* (386)-843-1829 > > *Email(s):* ferranj2 at my.erau.edu > > jesus.ferrand at gmail.com > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fO0pwSt99ISQDFM7Ioe1NhbBE_v7qVVrHxyaL60C6opn3c9PYf5hK7WjgrAQJUZh1UQDk3UhZUjZelBUE2lU$ > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fO0pwSt99ISQDFM7Ioe1NhbBE_v7qVVrHxyaL60C6opn3c9PYf5hK7WjgrAQJUZh1UQDk3UhZUjZelBUE2lU$ > > > ------------------------------ > *From:* Matthew Knepley > *Sent:* Wednesday, May 1, 2024 8:52 PM > *To:* Ferrand, Jesus A. > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [EXTERNAL] Re: [petsc-users] Is there anything like a > "DMPlexSetCones()" ? > > On Wed, May 1, 2024 at 4:23?PM Ferrand, Jesus A. > wrote: > > Matt: > > My bad again, I need to clarify something that I just realized doesn't > make sense. > I said: "The nature of the I/O makes it so that I need to read the > connectivity before I get a semblance of buffer sizes." > Scratch that, obviously, if I can read the connectivity, I know the buffer > sizes. > > What I should have said is that I cannot know the "chart" that one sets in > DMPlexSetChart() a priori like in most cases. > I need to determine the chart from the connectivity lists. > > > You know the number of cells if you can read connectivity. Do you mean > that the format does not tell you > the number of vertices? > > Thanks, > > Matt > > > ------------------------------ > *From:* Ferrand, Jesus A. > *Sent:* Wednesday, May 1, 2024 8:17 PM > *To:* Matthew Knepley > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [EXTERNAL] Re: [petsc-users] Is there anything like a > "DMPlexSetCones()" ? > > Matt: > > "I do not understand the "flag check". What is that?" > > My bad, I should have referred to the "dm->setupcalled". > I believe this PetscBool is checked by the other DM (not just DMPlex) APIs. > The subsequent checks for dm->setupcalled == PETSC_TRUE is what I meant to > say. > Here's a copy of DMSetUp(). > > *PetscErrorCode DMSetUp (DM dm)* > 817: { > 818: PetscFunctionBegin ; > 820: if (dm->setupcalled) PetscFunctionReturn (PETSC_SUCCESS ); > 821: PetscTryTypeMethod (dm, setup); > 822: dm->setupcalled = PETSC_TRUE ; > 823: PetscFunctionReturn (PETSC_SUCCESS ); > 824: } > > "We could make a DMPlexSetCones(), but as you point out, the workflow for > DMSetUp() would have to change. Where does the memory come from for your > connectivity?" > It's a memory that I myself allocate based on a file's contents. > The nature of the I/O makes it so that I need to read the connectivity > before I get a semblance of buffer sizes. > Otherwise, I would stick to the tried and tested way. > > Also, when replying to the PETSc developers, users must reply to > petsc-users at mcs.anl.gov and not just to the individual email accounts of > the developers, right? > > > > > ------------------------------ > *From:* Matthew Knepley > *Sent:* Wednesday, May 1, 2024 8:07 PM > *To:* Ferrand, Jesus A. > *Cc:* petsc-users at mcs.anl.gov > *Subject:* [EXTERNAL] Re: [petsc-users] Is there anything like a > "DMPlexSetCones()" ? > > *CAUTION:* This email originated outside of Embry-Riddle Aeronautical > University. Do not click links or open attachments unless you recognize the > sender and know the content is safe. > > On Wed, May 1, 2024 at 3:34?PM Ferrand, Jesus A. > wrote: > > This Message Is From an External Sender > This message came from outside your organization. > > Dear PETSc team: > > For a project that I'm working on, I need to manually build a DMPlex. > From studying the source code of the various APIs in which the plex is > built from some supported file format, I get that the workflow is this: > > > 1. DMPlexSetChart() <-- Input nCells + nVerts > 2. DMPlexSetConeSize() <-- Input ConeSize for each point in [0,nCells) > 3. DMSetUp() ? Allocates memory internally. > 4. DMPlexGetCones() --> Gives you the memory onto which to write the > cell connectivity. > 5. *Write connectivity* > 6. DMPlexReorderCell() <-- For each point in [0,nCells) > > > I'm in a situation where the memory given by step (4) is available > a-priori. > I was hoping to skip steps 2, 3 , and 4 with something like a > "DMPlexSetCones()", but such an API does not exist. > My current workaround is to implement steps 2 through 4 as always and have > double the memory allocated in the interim (my instance + DM's internal > instance). > I was thinking of looking for the name of the struct member and = it to my > memory, but I can't overcome the flag check in DMSetUp() during later calls > to DMPlexGetCones() or DMPlexGetTransitiveClosure(). > > > I do not understand the "flag check". What is that? > > We could make a DMPlexSetCones(), but as you point out, the workflow for > DMSetUp() would have to change. Where does the memory come from for your > connectivity? > > Thanks, > > Matt > > > > Sincerely: > > *J.A. Ferrand* > > Embry-Riddle Aeronautical University - Daytona Beach - FL > Ph.D. Candidate, Aerospace Engineering > > M.Sc. Aerospace Engineering > > B.Sc. Aerospace Engineering > > B.Sc. Computational Mathematics > > > *Phone:* (386)-843-1829 > > *Email(s):* ferranj2 at my.erau.edu > > jesus.ferrand at gmail.com > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fO0pwSt99ISQDFM7Ioe1NhbBE_v7qVVrHxyaL60C6opn3c9PYf5hK7WjgrAQJUZh1UQDk3UhZUjZelBUE2lU$ > > > ------------------------------ > *From:* Matthew Knepley > *Sent:* Wednesday, May 1, 2024 8:07 PM > *To:* Ferrand, Jesus A. > *Cc:* petsc-users at mcs.anl.gov > *Subject:* [EXTERNAL] Re: [petsc-users] Is there anything like a > "DMPlexSetCones()" ? > > *CAUTION:* This email originated outside of Embry-Riddle Aeronautical > University. Do not click links or open attachments unless you recognize the > sender and know the content is safe. > > On Wed, May 1, 2024 at 3:34?PM Ferrand, Jesus A. > wrote: > > This Message Is From an External Sender > This message came from outside your organization. > > Dear PETSc team: > > For a project that I'm working on, I need to manually build a DMPlex. > From studying the source code of the various APIs in which the plex is > built from some supported file format, I get that the workflow is this: > > > 1. DMPlexSetChart() <-- Input nCells + nVerts > 2. DMPlexSetConeSize() <-- Input ConeSize for each point in [0,nCells) > 3. DMSetUp() ? Allocates memory internally. > 4. DMPlexGetCones() --> Gives you the memory onto which to write the > cell connectivity. > 5. *Write connectivity* > 6. DMPlexReorderCell() <-- For each point in [0,nCells) > > > I'm in a situation where the memory given by step (4) is available > a-priori. > I was hoping to skip steps 2, 3 , and 4 with something like a > "DMPlexSetCones()", but such an API does not exist. > My current workaround is to implement steps 2 through 4 as always and have > double the memory allocated in the interim (my instance + DM's internal > instance). > I was thinking of looking for the name of the struct member and = it to my > memory, but I can't overcome the flag check in DMSetUp() during later calls > to DMPlexGetCones() or DMPlexGetTransitiveClosure(). > > > I do not understand the "flag check". What is that? > > We could make a DMPlexSetCones(), but as you point out, the workflow for > DMSetUp() would have to change. Where does the memory come from for your > connectivity? > > Thanks, > > Matt > > > > Sincerely: > > *J.A. Ferrand* > > Embry-Riddle Aeronautical University - Daytona Beach - FL > Ph.D. Candidate, Aerospace Engineering > > M.Sc. Aerospace Engineering > > B.Sc. Aerospace Engineering > > B.Sc. Computational Mathematics > > > *Phone:* (386)-843-1829 > > *Email(s):* ferranj2 at my.erau.edu > > jesus.ferrand at gmail.com > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fO0pwSt99ISQDFM7Ioe1NhbBE_v7qVVrHxyaL60C6opn3c9PYf5hK7WjgrAQJUZh1UQDk3UhZUjZelBUE2lU$ > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fO0pwSt99ISQDFM7Ioe1NhbBE_v7qVVrHxyaL60C6opn3c9PYf5hK7WjgrAQJUZh1UQDk3UhZUjZelBUE2lU$ > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fO0pwSt99ISQDFM7Ioe1NhbBE_v7qVVrHxyaL60C6opn3c9PYf5hK7WjgrAQJUZh1UQDk3UhZUjZelBUE2lU$ > > > ------------------------------ > *From:* Matthew Knepley > *Sent:* Friday, May 3, 2024 6:45 PM > *To:* Ferrand, Jesus A. > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [EXTERNAL] Re: [petsc-users] Is there anything like a > "DMPlexSetCones()" ? > > On Wed, May 1, 2024 at 6:30?PM Ferrand, Jesus A. > wrote: > > "You know the number of cells if you can read connectivity. Do you mean > that the format does not tell you > the number of vertices?" > > Sort of... > So, the format does provide the number of vertices, however, due to the > way I read the data in parallel I don't know immediately how many local > vertices there will be. > Which prevents me from knowing the chart a priori. > I figured out the number of vertices using PetscHashSetI to determine the > number of unique entries in the connectivity list. > > > Right, but that means that the connectivity you read in is not in local > numbering. You will have to renumber it, so making a copy is probably > necessary anyway. This is the same sort of processing I do for the parallel > read. > > Thanks, > > Matt > > > ------------------------------ > *From:* Matthew Knepley > *Sent:* Wednesday, May 1, 2024 8:52 PM > *To:* Ferrand, Jesus A. > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [EXTERNAL] Re: [petsc-users] Is there anything like a > "DMPlexSetCones()" ? > > On Wed, May 1, 2024 at 4:23?PM Ferrand, Jesus A. > wrote: > > Matt: > > My bad again, I need to clarify something that I just realized doesn't > make sense. > I said: "The nature of the I/O makes it so that I need to read the > connectivity before I get a semblance of buffer sizes." > Scratch that, obviously, if I can read the connectivity, I know the buffer > sizes. > > What I should have said is that I cannot know the "chart" that one sets in > DMPlexSetChart() a priori like in most cases. > I need to determine the chart from the connectivity lists. > > > You know the number of cells if you can read connectivity. Do you mean > that the format does not tell you > the number of vertices? > > Thanks, > > Matt > > > ------------------------------ > *From:* Ferrand, Jesus A. > *Sent:* Wednesday, May 1, 2024 8:17 PM > *To:* Matthew Knepley > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [EXTERNAL] Re: [petsc-users] Is there anything like a > "DMPlexSetCones()" ? > > Matt: > > "I do not understand the "flag check". What is that?" > > My bad, I should have referred to the "dm->setupcalled". > I believe this PetscBool is checked by the other DM (not just DMPlex) APIs. > The subsequent checks for dm->setupcalled == PETSC_TRUE is what I meant to > say. > Here's a copy of DMSetUp(). > > *PetscErrorCode DMSetUp (DM dm)* > 817: { > 818: PetscFunctionBegin ; > 820: if (dm->setupcalled) PetscFunctionReturn (PETSC_SUCCESS ); > 821: PetscTryTypeMethod (dm, setup); > 822: dm->setupcalled = PETSC_TRUE ; > 823: PetscFunctionReturn (PETSC_SUCCESS ); > 824: } > > "We could make a DMPlexSetCones(), but as you point out, the workflow for > DMSetUp() would have to change. Where does the memory come from for your > connectivity?" > It's a memory that I myself allocate based on a file's contents. > The nature of the I/O makes it so that I need to read the connectivity > before I get a semblance of buffer sizes. > Otherwise, I would stick to the tried and tested way. > > Also, when replying to the PETSc developers, users must reply to > petsc-users at mcs.anl.gov and not just to the individual email accounts of > the developers, right? > > > > > ------------------------------ > *From:* Matthew Knepley > *Sent:* Wednesday, May 1, 2024 8:07 PM > *To:* Ferrand, Jesus A. > *Cc:* petsc-users at mcs.anl.gov > *Subject:* [EXTERNAL] Re: [petsc-users] Is there anything like a > "DMPlexSetCones()" ? > > *CAUTION:* This email originated outside of Embry-Riddle Aeronautical > University. Do not click links or open attachments unless you recognize the > sender and know the content is safe. > > On Wed, May 1, 2024 at 3:34?PM Ferrand, Jesus A. > wrote: > > This Message Is From an External Sender > This message came from outside your organization. > > Dear PETSc team: > > For a project that I'm working on, I need to manually build a DMPlex. > From studying the source code of the various APIs in which the plex is > built from some supported file format, I get that the workflow is this: > > > 1. DMPlexSetChart() <-- Input nCells + nVerts > 2. DMPlexSetConeSize() <-- Input ConeSize for each point in [0,nCells) > 3. DMSetUp() ? Allocates memory internally. > 4. DMPlexGetCones() --> Gives you the memory onto which to write the > cell connectivity. > 5. *Write connectivity* > 6. DMPlexReorderCell() <-- For each point in [0,nCells) > > > I'm in a situation where the memory given by step (4) is available > a-priori. > I was hoping to skip steps 2, 3 , and 4 with something like a > "DMPlexSetCones()", but such an API does not exist. > My current workaround is to implement steps 2 through 4 as always and have > double the memory allocated in the interim (my instance + DM's internal > instance). > I was thinking of looking for the name of the struct member and = it to my > memory, but I can't overcome the flag check in DMSetUp() during later calls > to DMPlexGetCones() or DMPlexGetTransitiveClosure(). > > > I do not understand the "flag check". What is that? > > We could make a DMPlexSetCones(), but as you point out, the workflow for > DMSetUp() would have to change. Where does the memory come from for your > connectivity? > > Thanks, > > Matt > > > > Sincerely: > > *J.A. Ferrand* > > Embry-Riddle Aeronautical University - Daytona Beach - FL > Ph.D. Candidate, Aerospace Engineering > > M.Sc. Aerospace Engineering > > B.Sc. Aerospace Engineering > > B.Sc. Computational Mathematics > > > *Phone:* (386)-843-1829 > > *Email(s):* ferranj2 at my.erau.edu > > jesus.ferrand at gmail.com > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fO0pwSt99ISQDFM7Ioe1NhbBE_v7qVVrHxyaL60C6opn3c9PYf5hK7WjgrAQJUZh1UQDk3UhZUjZelBUE2lU$ > > > ------------------------------ > *From:* Matthew Knepley > *Sent:* Wednesday, May 1, 2024 8:07 PM > *To:* Ferrand, Jesus A. > *Cc:* petsc-users at mcs.anl.gov > *Subject:* [EXTERNAL] Re: [petsc-users] Is there anything like a > "DMPlexSetCones()" ? > > *CAUTION:* This email originated outside of Embry-Riddle Aeronautical > University. Do not click links or open attachments unless you recognize the > sender and know the content is safe. > > On Wed, May 1, 2024 at 3:34?PM Ferrand, Jesus A. > wrote: > > This Message Is From an External Sender > This message came from outside your organization. > > Dear PETSc team: > > For a project that I'm working on, I need to manually build a DMPlex. > From studying the source code of the various APIs in which the plex is > built from some supported file format, I get that the workflow is this: > > > 1. DMPlexSetChart() <-- Input nCells + nVerts > 2. DMPlexSetConeSize() <-- Input ConeSize for each point in [0,nCells) > 3. DMSetUp() ? Allocates memory internally. > 4. DMPlexGetCones() --> Gives you the memory onto which to write the > cell connectivity. > 5. *Write connectivity* > 6. DMPlexReorderCell() <-- For each point in [0,nCells) > > > I'm in a situation where the memory given by step (4) is available > a-priori. > I was hoping to skip steps 2, 3 , and 4 with something like a > "DMPlexSetCones()", but such an API does not exist. > My current workaround is to implement steps 2 through 4 as always and have > double the memory allocated in the interim (my instance + DM's internal > instance). > I was thinking of looking for the name of the struct member and = it to my > memory, but I can't overcome the flag check in DMSetUp() during later calls > to DMPlexGetCones() or DMPlexGetTransitiveClosure(). > > > I do not understand the "flag check". What is that? > > We could make a DMPlexSetCones(), but as you point out, the workflow for > DMSetUp() would have to change. Where does the memory come from for your > connectivity? > > Thanks, > > Matt > > > > Sincerely: > > *J.A. Ferrand* > > Embry-Riddle Aeronautical University - Daytona Beach - FL > Ph.D. Candidate, Aerospace Engineering > > M.Sc. Aerospace Engineering > > B.Sc. Aerospace Engineering > > B.Sc. Computational Mathematics > > > *Phone:* (386)-843-1829 > > *Email(s):* ferranj2 at my.erau.edu > > jesus.ferrand at gmail.com > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fO0pwSt99ISQDFM7Ioe1NhbBE_v7qVVrHxyaL60C6opn3c9PYf5hK7WjgrAQJUZh1UQDk3UhZUjZelBUE2lU$ > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fO0pwSt99ISQDFM7Ioe1NhbBE_v7qVVrHxyaL60C6opn3c9PYf5hK7WjgrAQJUZh1UQDk3UhZUjZelBUE2lU$ > > > ------------------------------ > *From:* Matthew Knepley > *Sent:* Wednesday, May 1, 2024 8:52 PM > *To:* Ferrand, Jesus A. > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [EXTERNAL] Re: [petsc-users] Is there anything like a > "DMPlexSetCones()" ? > > On Wed, May 1, 2024 at 4:23?PM Ferrand, Jesus A. > wrote: > > Matt: > > My bad again, I need to clarify something that I just realized doesn't > make sense. > I said: "The nature of the I/O makes it so that I need to read the > connectivity before I get a semblance of buffer sizes." > Scratch that, obviously, if I can read the connectivity, I know the buffer > sizes. > > What I should have said is that I cannot know the "chart" that one sets in > DMPlexSetChart() a priori like in most cases. > I need to determine the chart from the connectivity lists. > > > You know the number of cells if you can read connectivity. Do you mean > that the format does not tell you > the number of vertices? > > Thanks, > > Matt > > > ------------------------------ > *From:* Ferrand, Jesus A. > *Sent:* Wednesday, May 1, 2024 8:17 PM > *To:* Matthew Knepley > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [EXTERNAL] Re: [petsc-users] Is there anything like a > "DMPlexSetCones()" ? > > Matt: > > "I do not understand the "flag check". What is that?" > > My bad, I should have referred to the "dm->setupcalled". > I believe this PetscBool is checked by the other DM (not just DMPlex) APIs. > The subsequent checks for dm->setupcalled == PETSC_TRUE is what I meant to > say. > Here's a copy of DMSetUp(). > > *PetscErrorCode DMSetUp (DM dm)* > 817: { > 818: PetscFunctionBegin ; > 820: if (dm->setupcalled) PetscFunctionReturn (PETSC_SUCCESS ); > 821: PetscTryTypeMethod (dm, setup); > 822: dm->setupcalled = PETSC_TRUE ; > 823: PetscFunctionReturn (PETSC_SUCCESS ); > 824: } > > "We could make a DMPlexSetCones(), but as you point out, the workflow for > DMSetUp() would have to change. Where does the memory come from for your > connectivity?" > It's a memory that I myself allocate based on a file's contents. > The nature of the I/O makes it so that I need to read the connectivity > before I get a semblance of buffer sizes. > Otherwise, I would stick to the tried and tested way. > > Also, when replying to the PETSc developers, users must reply to > petsc-users at mcs.anl.gov and not just to the individual email accounts of > the developers, right? > > > > > ------------------------------ > *From:* Matthew Knepley > *Sent:* Wednesday, May 1, 2024 8:07 PM > *To:* Ferrand, Jesus A. > *Cc:* petsc-users at mcs.anl.gov > *Subject:* [EXTERNAL] Re: [petsc-users] Is there anything like a > "DMPlexSetCones()" ? > > *CAUTION:* This email originated outside of Embry-Riddle Aeronautical > University. Do not click links or open attachments unless you recognize the > sender and know the content is safe. > > On Wed, May 1, 2024 at 3:34?PM Ferrand, Jesus A. > wrote: > > This Message Is From an External Sender > This message came from outside your organization. > > Dear PETSc team: > > For a project that I'm working on, I need to manually build a DMPlex. > From studying the source code of the various APIs in which the plex is > built from some supported file format, I get that the workflow is this: > > > 1. DMPlexSetChart() <-- Input nCells + nVerts > 2. DMPlexSetConeSize() <-- Input ConeSize for each point in [0,nCells) > 3. DMSetUp() ? Allocates memory internally. > 4. DMPlexGetCones() --> Gives you the memory onto which to write the > cell connectivity. > 5. *Write connectivity* > 6. DMPlexReorderCell() <-- For each point in [0,nCells) > > > I'm in a situation where the memory given by step (4) is available > a-priori. > I was hoping to skip steps 2, 3 , and 4 with something like a > "DMPlexSetCones()", but such an API does not exist. > My current workaround is to implement steps 2 through 4 as always and have > double the memory allocated in the interim (my instance + DM's internal > instance). > I was thinking of looking for the name of the struct member and = it to my > memory, but I can't overcome the flag check in DMSetUp() during later calls > to DMPlexGetCones() or DMPlexGetTransitiveClosure(). > > > I do not understand the "flag check". What is that? > > We could make a DMPlexSetCones(), but as you point out, the workflow for > DMSetUp() would have to change. Where does the memory come from for your > connectivity? > > Thanks, > > Matt > > > > Sincerely: > > *J.A. Ferrand* > > Embry-Riddle Aeronautical University - Daytona Beach - FL > Ph.D. Candidate, Aerospace Engineering > > M.Sc. Aerospace Engineering > > B.Sc. Aerospace Engineering > > B.Sc. Computational Mathematics > > > *Phone:* (386)-843-1829 > > *Email(s):* ferranj2 at my.erau.edu > > jesus.ferrand at gmail.com > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fO0pwSt99ISQDFM7Ioe1NhbBE_v7qVVrHxyaL60C6opn3c9PYf5hK7WjgrAQJUZh1UQDk3UhZUjZelBUE2lU$ > > > ------------------------------ > *From:* Matthew Knepley > *Sent:* Wednesday, May 1, 2024 8:07 PM > *To:* Ferrand, Jesus A. > *Cc:* petsc-users at mcs.anl.gov > *Subject:* [EXTERNAL] Re: [petsc-users] Is there anything like a > "DMPlexSetCones()" ? > > *CAUTION:* This email originated outside of Embry-Riddle Aeronautical > University. Do not click links or open attachments unless you recognize the > sender and know the content is safe. > > On Wed, May 1, 2024 at 3:34?PM Ferrand, Jesus A. > wrote: > > This Message Is From an External Sender > This message came from outside your organization. > > Dear PETSc team: > > For a project that I'm working on, I need to manually build a DMPlex. > From studying the source code of the various APIs in which the plex is > built from some supported file format, I get that the workflow is this: > > > 1. DMPlexSetChart() <-- Input nCells + nVerts > 2. DMPlexSetConeSize() <-- Input ConeSize for each point in [0,nCells) > 3. DMSetUp() ? Allocates memory internally. > 4. DMPlexGetCones() --> Gives you the memory onto which to write the > cell connectivity. > 5. *Write connectivity* > 6. DMPlexReorderCell() <-- For each point in [0,nCells) > > > I'm in a situation where the memory given by step (4) is available > a-priori. > I was hoping to skip steps 2, 3 , and 4 with something like a > "DMPlexSetCones()", but such an API does not exist. > My current workaround is to implement steps 2 through 4 as always and have > double the memory allocated in the interim (my instance + DM's internal > instance). > I was thinking of looking for the name of the struct member and = it to my > memory, but I can't overcome the flag check in DMSetUp() during later calls > to DMPlexGetCones() or DMPlexGetTransitiveClosure(). > > > I do not understand the "flag check". What is that? > > We could make a DMPlexSetCones(), but as you point out, the workflow for > DMSetUp() would have to change. Where does the memory come from for your > connectivity? > > Thanks, > > Matt > > > > Sincerely: > > *J.A. Ferrand* > > Embry-Riddle Aeronautical University - Daytona Beach - FL > Ph.D. Candidate, Aerospace Engineering > > M.Sc. Aerospace Engineering > > B.Sc. Aerospace Engineering > > B.Sc. Computational Mathematics > > > *Phone:* (386)-843-1829 > > *Email(s):* ferranj2 at my.erau.edu > > jesus.ferrand at gmail.com > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fO0pwSt99ISQDFM7Ioe1NhbBE_v7qVVrHxyaL60C6opn3c9PYf5hK7WjgrAQJUZh1UQDk3UhZUjZelBUE2lU$ > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fO0pwSt99ISQDFM7Ioe1NhbBE_v7qVVrHxyaL60C6opn3c9PYf5hK7WjgrAQJUZh1UQDk3UhZUjZelBUE2lU$ > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fO0pwSt99ISQDFM7Ioe1NhbBE_v7qVVrHxyaL60C6opn3c9PYf5hK7WjgrAQJUZh1UQDk3UhZUjZelBUE2lU$ > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fO0pwSt99ISQDFM7Ioe1NhbBE_v7qVVrHxyaL60C6opn3c9PYf5hK7WjgrAQJUZh1UQDk3UhZUjZelBUE2lU$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Mon May 6 07:38:39 2024 From: mfadams at lbl.gov (Mark Adams) Date: Mon, 6 May 2024 08:38:39 -0400 Subject: [petsc-users] PETSc options In-Reply-To: <2d56c6ea-e9e0-4a7a-b565-90a1404e08b7@auckland.ac.nz> References: <2d56c6ea-e9e0-4a7a-b565-90a1404e08b7@auckland.ac.nz> Message-ID: I don't know why this should have changed, but you can either not feed -v to PETSc (a pain probably), use PETSc's getOptions methods instead of Fortran's, or make a dummy call PETSc's methods in addition to yours. Hope this helps, Mark On Mon, May 6, 2024 at 1:04?AM Adrian Croucher wrote: > hi, My code has some optional command line arguments -v and -h for output > of version number and usage help. These are processed using Fortran's > get_command_argument(). Since updating PETSc to version 3. 21, I get some > extra warnings after the > ZjQcmQRYFpfptBannerStart > This Message Is From an External Sender > This message came from outside your organization. > > ZjQcmQRYFpfptBannerEnd > > hi, > > My code has some optional command line arguments -v and -h for output of > version number and usage help. These are processed using Fortran's > get_command_argument(). > > Since updating PETSc to version 3.21, I get some extra warnings after > the output: > > acro018 at EN438880:~$ waiwera -v > 1.5.0b1 > WARNING! There are options you set that were not used! > WARNING! could be spelling mistake, etc! > There is one unused database option. It is: > Option left: name:-v (no value) source: command line > > That didn't used to happen. What should I do to make them go away? > > Regards, Adrian > > -- > Dr Adrian Croucher > Senior Research Fellow > Department of Engineering Science > Waipapa Taumata Rau / University of Auckland, New Zealand > email: a.croucher at auckland.ac.nz > tel: +64 (0)9 923 4611 > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From samar.khatiwala at earth.ox.ac.uk Mon May 6 07:49:11 2024 From: samar.khatiwala at earth.ox.ac.uk (Samar Khatiwala) Date: Mon, 6 May 2024 12:49:11 +0000 Subject: [petsc-users] Question about petsc4py createWithArray function In-Reply-To: References: <6ED2F015-541E-485B-AACB-EAE329A4A307@earth.ox.ac.uk> Message-ID: Hi Stefano, Thanks for looking into this. Since createWithArray calls VecCreateMPIWithArray which, as Matt noted and is documented (https://urldefense.us/v3/__https://petsc.org/main/manualpages/Vec/VecCreateMPIWithArray/__;!!G_uCfscf7eWS!YpH_OGif6W8Ql9KZDYBiRMKyALSRhMZOHRnkCkBHqBcVB4ef_UBHFbT_EuTgx8ivGmDNQU8bRVaOc-B_tqQ2heGjdvoE9tp8ZbPi3QQ$ ) doesn?t free the memory, then there?s a memory leak (and, furthermore, calling del on the original array will have no effect). Lisandro: would be great if you can provide some guidance. Thanks, Samar On May 3, 2024, at 12:45 PM, Stefano Zampini wrote: While waiting for our Python wizard to shed light on this, I note that, from the documentation of PyArray_FROM_OTF https://urldefense.us/v3/__https://numpy.org/devdocs/user/c-info.how-to-extend.html*converting-an-arbitrary-sequence-object__;Iw!!G_uCfscf7eWS!YpH_OGif6W8Ql9KZDYBiRMKyALSRhMZOHRnkCkBHqBcVB4ef_UBHFbT_EuTgx8ivGmDNQU8bRVaOc-B_tqQ2heGjdvoE9tp8_QyxtQk$ , we have The object can be any Python object convertible to an ndarray. If the object is already (a subclass of) the ndarray that satisfies the requirements then a new reference is returned. I guess we should call "del" on the ndarray returned by iarray_s after having called self.set_attr('__array__', array) in this case, but let's wait for Lisandro to confirm Il giorno ven 3 mag 2024 alle ore 11:42 Samar Khatiwala > ha scritto: Hi Matt, Thanks so much for the quick reply! Regarding #2, I put some debug statement in my code and what I find is that when I use createWithArray on my Cython-allocated numpy array, the destructor I set for it is no longer called when I delete ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hi Matt, Thanks so much for the quick reply! Regarding #2, I put some debug statement in my code and what I find is that when I use createWithArray on my Cython-allocated numpy array, the destructor I set for it is no longer called when I delete the array. (If I don?t use createWithArray then the destructor is triggered.) I interpret that to suggest that the petsc4py Vec is somehow ?taking over? management of the numpy array. But I don?t understand where that could be happening. (I don?t think it has to do with the actual freeing of memory by PETSc's VecDestroy.) createWithArray calls iarray_s which in turn calls PyArray_FROM_OTF. Could it be there?s something going on there? The numpy documentation is unclear. Lisandro: do you have any thoughts on this? Thanks, Samar On May 2, 2024, at 11:56 PM, Matthew Knepley > wrote: On Thu, May 2, 2024 at 12:53?PM Samar Khatiwala > wrote: This Message Is From an External Sender This message came from outside your organization. Hello, I have a couple of questions about createWithArray in petsc4py: 1) What is the correct usage for creating a standard MPI Vec with it? Something like this seems to work but is it right?: On each rank do: a = np.zeros(localSize) v = PETSc.Vec().createWithArray(a, comm=PETSc.COMM_WORLD) Is that all it takes? That looks right to me. 2) Who ?owns? the underlying memory for a Vec created with the createWithArray method, i.e., who is responsible for managing it and doing garbage collection? In my problem, the numpy array is created in a Cython module where memory is allocated, and a pointer to it is associated with a numpy ndarray via PyArray_SimpleNewFromData and PyArray_SetBaseObject. I have a deallocator method of my own that is called when the numpy array is deleted/goes out of scope/whenever python does garbage collection. All of that works fine. But if I use this array to create a Vec with createWithArray what happens when the Vec is, e.g., destroyed? Will my deallocator be called? No. The PETSc struct will be deallocated, but the storage will not be touched. Thanks, Matt Or does petsc4py know that it doesn?t own the memory and won?t attempt to free it? I can?t quite figure out from the petsc4py code what is going on. And help would be appreciated. Thanks very much. Samar -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!YpH_OGif6W8Ql9KZDYBiRMKyALSRhMZOHRnkCkBHqBcVB4ef_UBHFbT_EuTgx8ivGmDNQU8bRVaOc-B_tqQ2heGjdvoE9tp8YS3M6pE$ -- Stefano -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefano.zampini at gmail.com Mon May 6 08:09:36 2024 From: stefano.zampini at gmail.com (Stefano Zampini) Date: Mon, 6 May 2024 09:09:36 -0400 Subject: [petsc-users] Question about petsc4py createWithArray function In-Reply-To: References: <6ED2F015-541E-485B-AACB-EAE329A4A307@earth.ox.ac.uk> Message-ID: Samar After a second look, I believe the petsc4py code is correct. You can test it using the script below. After destroy is called (or del), the reference count of the numpy array is back to its initial state. Maybe you are not calling del or destroy? a MWE would help to understand your use case kl-18448:~ szampini$ cat t.py import sys import numpy as np from petsc4py import PETSc x = np.zeros(4, dtype=PETSc.ScalarType) print('initial ref count',sys.getrefcount(x)) v = PETSc.Vec().createWithArray(x) print('after create',sys.getrefcount(x)) # check if they share memory v.view() x[1] = 2 v.view() # free v.destroy() # you can also call del # del v print('after destroy',sys.getrefcount(x)) kl-18448:~ szampini$ python t.py initial ref count 2 after create 3 Vec Object: 1 MPI process type: seq 0. 0. 0. 0. Vec Object: 1 MPI process type: seq 0. 2. 0. 0. after destroy 2 Il giorno lun 6 mag 2024 alle ore 08:49 Samar Khatiwala < samar.khatiwala at earth.ox.ac.uk> ha scritto: > Hi Stefano, > > Thanks for looking into this. Since createWithArray calls > VecCreateMPIWithArray which, as Matt noted and is documented ( > https://urldefense.us/v3/__https://petsc.org/main/manualpages/Vec/VecCreateMPIWithArray/__;!!G_uCfscf7eWS!Z9RkH5ffTuJDtBUf8_Gk0BHuG__BKv4jPYeg89Rp6_GcS9VTcFs2J8uyLd5_wiqdDmy9ABXjczA3PYB1raRliVrs4DeDixY$ ) doesn?t > free the memory, then there?s a memory leak (and, furthermore, calling del > on the original array will have no effect). > > Lisandro: would be great if you can provide some guidance. > > Thanks, > > Samar > > On May 3, 2024, at 12:45 PM, Stefano Zampini > wrote: > > While waiting for our Python wizard to shed light on this, I note that, > from the documentation of PyArray_FROM_OTF > https://urldefense.us/v3/__https://numpy.org/devdocs/user/c-info.how-to-extend.html*converting-an-arbitrary-sequence-object__;Iw!!G_uCfscf7eWS!Z9RkH5ffTuJDtBUf8_Gk0BHuG__BKv4jPYeg89Rp6_GcS9VTcFs2J8uyLd5_wiqdDmy9ABXjczA3PYB1raRliVrsskf0ioU$ , > we have > > The object can be any Python object convertible to an ndarray. If the > object is already (a subclass of) the ndarray that satisfies the > requirements then a new reference is returned. > > I guess we should call "del" on the ndarray returned by iarray_s after > having called self.set_attr('__array__', array) in this case, but let's > wait for Lisandro to confirm > > > > > Il giorno ven 3 mag 2024 alle ore 11:42 Samar Khatiwala < > samar.khatiwala at earth.ox.ac.uk> ha scritto: > >> Hi Matt, Thanks so much for the quick reply! Regarding #2, I put some >> debug statement in my code and what I find is that when I use >> createWithArray on my Cython-allocated numpy array, the destructor I set >> for it is no longer called when I delete >> ZjQcmQRYFpfptBannerStart >> This Message Is From an External Sender >> This message came from outside your organization. >> >> ZjQcmQRYFpfptBannerEnd >> Hi Matt, >> >> Thanks so much for the quick reply! >> >> Regarding #2, I put some debug statement in my code and what I find is >> that when I use createWithArray on my Cython-allocated numpy array, the >> destructor I set for it is no longer called when I delete the array. (If I >> don?t use createWithArray then the destructor is triggered.) I interpret >> that to suggest that the petsc4py Vec is somehow ?taking over? management >> of the numpy array. But I don?t understand where that could be >> happening. (I don?t think it has to do with the actual freeing of memory by >> PETSc's VecDestroy.) >> >> createWithArray calls iarray_s which in turn calls PyArray_FROM_OTF. >> Could it be there?s something going on there? The numpy documentation is >> unclear. >> >> Lisandro: do you have any thoughts on this? >> >> Thanks, >> >> Samar >> >> On May 2, 2024, at 11:56 PM, Matthew Knepley wrote: >> >> On Thu, May 2, 2024 at 12:53?PM Samar Khatiwala < >> samar.khatiwala at earth.ox.ac.uk> wrote: >> >>> This Message Is From an External Sender >>> This message came from outside your organization. >>> >>> >>> Hello, >>> >>> I have a couple of questions about createWithArray in petsc4py: >>> >>> 1) What is the correct usage for creating a standard MPI Vec with it? Something like this seems to work but is it right?: >>> >>> On each rank do: >>> a = np.zeros(localSize) >>> v = PETSc.Vec().createWithArray(a, comm=PETSc.COMM_WORLD) >>> >>> Is that all it takes? >>> >>> >> That looks right to me. >> >>> 2) Who ?owns? the underlying memory for a Vec created with the createWithArray method, i.e., who is responsible for managing it and doing garbage collection? In my problem, the numpy array is created in a Cython module where memory is allocated, and a pointer to it is associated with a numpy ndarray via PyArray_SimpleNewFromData and PyArray_SetBaseObject. I have a deallocator method of my own that is called when the numpy array is deleted/goes out of scope/whenever python does garbage collection. All of that works fine. But if I use this array to create a Vec with createWithArray what happens when the Vec is, e.g., destroyed? Will my deallocator be called? >>> >>> No. The PETSc struct will be deallocated, but the storage will not be >> touched. >> >> Thanks, >> >> Matt >> >>> Or does petsc4py know that it doesn?t own the memory and won?t attempt to free it? I can?t quite figure out from the petsc4py code what is going on. And help would be appreciated. >>> >>> Thanks very much. >>> >>> Samar >>> >>> >>> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!Z9RkH5ffTuJDtBUf8_Gk0BHuG__BKv4jPYeg89Rp6_GcS9VTcFs2J8uyLd5_wiqdDmy9ABXjczA3PYB1raRliVrs-JUDe0A$ >> >> >> >> > > -- > Stefano > > > -- Stefano -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Mon May 6 08:11:51 2024 From: bsmith at petsc.dev (Barry Smith) Date: Mon, 6 May 2024 09:11:51 -0400 Subject: [petsc-users] PETSc options In-Reply-To: References: <2d56c6ea-e9e0-4a7a-b565-90a1404e08b7@auckland.ac.nz> Message-ID: <53C98BF3-786C-4981-8C44-F4D3F364972F@petsc.dev> > On May 6, 2024, at 8:38?AM, Mark Adams wrote: > > This Message Is From an External Sender > This message came from outside your organization. > I don't know why this should have changed, but you can either not feed -v to PETSc (a pain probably), use PETSc's getOptions methods instead of Fortran's, or make a dummy call PETSc's methods in addition to yours. Yes, just call PetscOptionsHasName() with -v and ignore the result. > > Hope this helps, > Mark > > On Mon, May 6, 2024 at 1:04?AM Adrian Croucher > wrote: >> This Message Is From an External Sender >> This message came from outside your organization. >> >> hi, >> >> My code has some optional command line arguments -v and -h for output of >> version number and usage help. These are processed using Fortran's >> get_command_argument(). >> >> Since updating PETSc to version 3.21, I get some extra warnings after >> the output: >> >> acro018 at EN438880:~$ waiwera -v >> 1.5.0b1 >> WARNING! There are options you set that were not used! >> WARNING! could be spelling mistake, etc! >> There is one unused database option. It is: >> Option left: name:-v (no value) source: command line >> >> That didn't used to happen. What should I do to make them go away? >> >> Regards, Adrian >> >> -- >> Dr Adrian Croucher >> Senior Research Fellow >> Department of Engineering Science >> Waipapa Taumata Rau / University of Auckland, New Zealand >> email: a.croucher at auckland.ac.nz >> tel: +64 (0)9 923 4611 >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon May 6 08:14:27 2024 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 6 May 2024 09:14:27 -0400 Subject: [petsc-users] PETSc options In-Reply-To: <2d56c6ea-e9e0-4a7a-b565-90a1404e08b7@auckland.ac.nz> References: <2d56c6ea-e9e0-4a7a-b565-90a1404e08b7@auckland.ac.nz> Message-ID: On Mon, May 6, 2024 at 1:04?AM Adrian Croucher wrote: > hi, My code has some optional command line arguments -v and -h for output > of version number and usage help. These are processed using Fortran's > get_command_argument(). Since updating PETSc to version 3. 21, I get some > extra warnings after the > ZjQcmQRYFpfptBannerStart > This Message Is From an External Sender > This message came from outside your organization. > > ZjQcmQRYFpfptBannerEnd > > hi, > > My code has some optional command line arguments -v and -h for output of > version number and usage help. These are processed using Fortran's > get_command_argument(). > > Since updating PETSc to version 3.21, I get some extra warnings after > the output: > > acro018 at EN438880:~$ waiwera -v > 1.5.0b1 > WARNING! There are options you set that were not used! > WARNING! could be spelling mistake, etc! > There is one unused database option. It is: > Option left: name:-v (no value) source: command line > > That didn't used to happen. What should I do to make them go away? > > Hi Adrian, Barry and Mark's suggestions will make this go away. However, it should not happen in the first place. We should try to figure this out. This warning is usually activated by the -options_left argument. Could that be in the PETSC_OPTIONS env variable, or in ~/.petscrc? Thanks, Matt > Regards, Adrian > > -- > Dr Adrian Croucher > Senior Research Fellow > Department of Engineering Science > Waipapa Taumata Rau / University of Auckland, New Zealand > email: a.croucher at auckland.ac.nz > tel: +64 (0)9 923 4611 > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fo7-rCLfudbdd6yHxfeF1pyKANLJjDLd2VvUtM7xc31s9AoJjSSu-vjiyaVVAhNAqV6c4s-9aJPkL_AI6ctk$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From FERRANJ2 at my.erau.edu Mon May 6 09:17:52 2024 From: FERRANJ2 at my.erau.edu (Ferrand, Jesus A.) Date: Mon, 6 May 2024 14:17:52 +0000 Subject: [petsc-users] [EXTERNAL] Re: Is there anything like a "DMPlexSetCones()" ? In-Reply-To: References: Message-ID: "The cone sizes are used to allocate the array to hold the cones. How would you have an array holding the cone information without knowing the sizes?" ??????You're right, I must first know the sizes (which I determine from the file). ??????It lists the number of cells and their types by sequential chunks. ??????The issue I am trying to describe, is that because I cannot know the number of local vertices a priori, the pipeline DMSetChart()->DMPlexSetConeSize()->DMSetUp() essentially produces a redundant array (because I already have the connectivity that I read from the file). ??????As far as I can tell, in my case, the end result of calling these APIs is already available, hence my wish for a "DMPlexSetCones()". ??????If such an API existed, I would then attempt to retroactively set the Chart and the ConeSizes. ??????Like you hinted in a reply, for this extra array to be worthwhile, I would have to keep it as a global numbering list (which I want to by the way). ??????I was hoping to at least keep them in the DM's global PetscSection as offsets. ??????Unfortunately, PetscSection has a similar workflow to DM: PetscSectionSetChart(), PetscSectionAddDof(), PetscSectionSetUp(), PetscSectionSetOffset(). ?????? ________________________________ From: Matthew Knepley Sent: Monday, May 6, 2024 7:52 AM To: Ferrand, Jesus A. Cc: petsc-users at mcs.anl.gov Subject: Re: [EXTERNAL] Re: [petsc-users] Is there anything like a "DMPlexSetCones()" ? On Fri, May 3, 2024 at 7:32?PM Ferrand, Jesus A. > wrote: "Right, but that means that the connectivity you read in is not in local numbering. You will have to renumber it, so making a copy is probably necessary anyway. This is the same sort of processing I do for the parallel read." ??????True, the connectivity is read in global numbering. ??????Renumbering is facilitated by the table from PetscHashSetI. ??????The connectivity list I am able to renumber in place though, I don't see the need to have another interim copy of the numbers. ??????My concern is that I have two memory chunks that both contain the same information. ??????The only reason It has to be this way is because I can't manually set the Cones first and the Chart/ConeSizes second. The cone sizes are used to allocate the array to hold the cones. How would you have an array holding the cone information without knowing the sizes? Thanks, Matt ________________________________ From: Matthew Knepley > Sent: Friday, May 3, 2024 6:45 PM To: Ferrand, Jesus A. > Cc: petsc-users at mcs.anl.gov > Subject: Re: [EXTERNAL] Re: [petsc-users] Is there anything like a "DMPlexSetCones()" ? On Wed, May 1, 2024 at 6:30?PM Ferrand, Jesus A. > wrote: "You know the number of cells if you can read connectivity. Do you mean that the format does not tell you the number of vertices?" Sort of... So, the format does provide the number of vertices, however, due to the way I read the data in parallel I don't know immediately how many local vertices there will be. Which prevents me from knowing the chart a priori. I figured out the number of vertices using PetscHashSetI to determine the number of unique entries in the connectivity list. Right, but that means that the connectivity you read in is not in local numbering. You will have to renumber it, so making a copy is probably necessary anyway. This is the same sort of processing I do for the parallel read. Thanks, Matt ________________________________ From: Matthew Knepley > Sent: Wednesday, May 1, 2024 8:52 PM To: Ferrand, Jesus A. > Cc: petsc-users at mcs.anl.gov > Subject: Re: [EXTERNAL] Re: [petsc-users] Is there anything like a "DMPlexSetCones()" ? On Wed, May 1, 2024 at 4:23?PM Ferrand, Jesus A. > wrote: Matt: My bad again, I need to clarify something that I just realized doesn't make sense. I said: "The nature of the I/O makes it so that I need to read the connectivity before I get a semblance of buffer sizes." Scratch that, obviously, if I can read the connectivity, I know the buffer sizes. What I should have said is that I cannot know the "chart" that one sets in DMPlexSetChart() a priori like in most cases. I need to determine the chart from the connectivity lists. You know the number of cells if you can read connectivity. Do you mean that the format does not tell you the number of vertices? Thanks, Matt ________________________________ From: Ferrand, Jesus A. > Sent: Wednesday, May 1, 2024 8:17 PM To: Matthew Knepley > Cc: petsc-users at mcs.anl.gov > Subject: Re: [EXTERNAL] Re: [petsc-users] Is there anything like a "DMPlexSetCones()" ? Matt: "I do not understand the "flag check". What is that?" ??????My bad, I should have referred to the "dm->setupcalled". ??????I believe this PetscBool is checked by the other DM (not just DMPlex) APIs. ??????The subsequent checks for dm->setupcalled == PETSC_TRUE is what I meant to say. ??????Here's a copy of DMSetUp(). PetscErrorCode DMSetUp(DM dm) 817: { 818: PetscFunctionBegin; 820: if (dm->setupcalled) PetscFunctionReturn(PETSC_SUCCESS); 821: PetscTryTypeMethod(dm, setup); 822: dm->setupcalled = PETSC_TRUE; 823: PetscFunctionReturn(PETSC_SUCCESS); 824: } "We could make a DMPlexSetCones(), but as you point out, the workflow for DMSetUp() would have to change. Where does the memory come from for your connectivity?" ??????It's a memory that I myself allocate based on a file's contents. ??????The nature of the I/O makes it so that I need to read the connectivity before I get a semblance of buffer sizes. ??????Otherwise, I would stick to the tried and tested way. Also, when replying to the PETSc developers, users must reply to petsc-users at mcs.anl.gov and not just to the individual email accounts of the developers, right? ________________________________ From: Matthew Knepley > Sent: Wednesday, May 1, 2024 8:07 PM To: Ferrand, Jesus A. > Cc: petsc-users at mcs.anl.gov > Subject: [EXTERNAL] Re: [petsc-users] Is there anything like a "DMPlexSetCones()" ? CAUTION: This email originated outside of Embry-Riddle Aeronautical University. Do not click links or open attachments unless you recognize the sender and know the content is safe. On Wed, May 1, 2024 at 3:34?PM Ferrand, Jesus A. > wrote: This Message Is From an External Sender This message came from outside your organization. Dear PETSc team: For a project that I'm working on, I need to manually build a DMPlex. >From studying the source code of the various APIs in which the plex is built from some supported file format, I get that the workflow is this: 1. DMPlexSetChart() <-- Input nCells + nVerts 2. DMPlexSetConeSize() <-- Input ConeSize for each point in [0,nCells) 3. DMSetUp() ? Allocates memory internally. 4. DMPlexGetCones() --> Gives you the memory onto which to write the cell connectivity. 5. *Write connectivity* 6. DMPlexReorderCell() <-- For each point in [0,nCells) I'm in a situation where the memory given by step (4) is available a-priori. I was hoping to skip steps 2, 3 , and 4 with something like a "DMPlexSetCones()", but such an API does not exist. My current workaround is to implement steps 2 through 4 as always and have double the memory allocated in the interim (my instance + DM's internal instance). I was thinking of looking for the name of the struct member and = it to my memory, but I can't overcome the flag check in DMSetUp() during later calls to DMPlexGetCones() or DMPlexGetTransitiveClosure(). I do not understand the "flag check". What is that? We could make a DMPlexSetCones(), but as you point out, the workflow for DMSetUp() would have to change. Where does the memory come from for your connectivity? Thanks, Matt Sincerely: J.A. Ferrand Embry-Riddle Aeronautical University - Daytona Beach - FL Ph.D. Candidate, Aerospace Engineering M.Sc. Aerospace Engineering B.Sc. Aerospace Engineering B.Sc. Computational Mathematics Phone: (386)-843-1829 Email(s): ferranj2 at my.erau.edu jesus.ferrand at gmail.com -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!YA9N-9_O6-4rSg848GHyIY2nSendFsTvYClTS9F7288syR3Qqj7dAAwx9vwfVzBedRxEAI057ebAdXwzNbD-x-OTHfg$ ________________________________ From: Matthew Knepley > Sent: Wednesday, May 1, 2024 8:07 PM To: Ferrand, Jesus A. > Cc: petsc-users at mcs.anl.gov > Subject: [EXTERNAL] Re: [petsc-users] Is there anything like a "DMPlexSetCones()" ? CAUTION: This email originated outside of Embry-Riddle Aeronautical University. Do not click links or open attachments unless you recognize the sender and know the content is safe. On Wed, May 1, 2024 at 3:34?PM Ferrand, Jesus A. > wrote: This Message Is From an External Sender This message came from outside your organization. Dear PETSc team: For a project that I'm working on, I need to manually build a DMPlex. >From studying the source code of the various APIs in which the plex is built from some supported file format, I get that the workflow is this: 1. DMPlexSetChart() <-- Input nCells + nVerts 2. DMPlexSetConeSize() <-- Input ConeSize for each point in [0,nCells) 3. DMSetUp() ? Allocates memory internally. 4. DMPlexGetCones() --> Gives you the memory onto which to write the cell connectivity. 5. *Write connectivity* 6. DMPlexReorderCell() <-- For each point in [0,nCells) I'm in a situation where the memory given by step (4) is available a-priori. I was hoping to skip steps 2, 3 , and 4 with something like a "DMPlexSetCones()", but such an API does not exist. My current workaround is to implement steps 2 through 4 as always and have double the memory allocated in the interim (my instance + DM's internal instance). I was thinking of looking for the name of the struct member and = it to my memory, but I can't overcome the flag check in DMSetUp() during later calls to DMPlexGetCones() or DMPlexGetTransitiveClosure(). I do not understand the "flag check". What is that? We could make a DMPlexSetCones(), but as you point out, the workflow for DMSetUp() would have to change. Where does the memory come from for your connectivity? Thanks, Matt Sincerely: J.A. Ferrand Embry-Riddle Aeronautical University - Daytona Beach - FL Ph.D. Candidate, Aerospace Engineering M.Sc. Aerospace Engineering B.Sc. Aerospace Engineering B.Sc. Computational Mathematics Phone: (386)-843-1829 Email(s): ferranj2 at my.erau.edu jesus.ferrand at gmail.com -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!YA9N-9_O6-4rSg848GHyIY2nSendFsTvYClTS9F7288syR3Qqj7dAAwx9vwfVzBedRxEAI057ebAdXwzNbD-x-OTHfg$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!YA9N-9_O6-4rSg848GHyIY2nSendFsTvYClTS9F7288syR3Qqj7dAAwx9vwfVzBedRxEAI057ebAdXwzNbD-x-OTHfg$ ________________________________ From: Matthew Knepley > Sent: Wednesday, May 1, 2024 8:52 PM To: Ferrand, Jesus A. > Cc: petsc-users at mcs.anl.gov > Subject: Re: [EXTERNAL] Re: [petsc-users] Is there anything like a "DMPlexSetCones()" ? On Wed, May 1, 2024 at 4:23?PM Ferrand, Jesus A. > wrote: Matt: My bad again, I need to clarify something that I just realized doesn't make sense. I said: "The nature of the I/O makes it so that I need to read the connectivity before I get a semblance of buffer sizes." Scratch that, obviously, if I can read the connectivity, I know the buffer sizes. What I should have said is that I cannot know the "chart" that one sets in DMPlexSetChart() a priori like in most cases. I need to determine the chart from the connectivity lists. You know the number of cells if you can read connectivity. Do you mean that the format does not tell you the number of vertices? Thanks, Matt ________________________________ From: Ferrand, Jesus A. > Sent: Wednesday, May 1, 2024 8:17 PM To: Matthew Knepley > Cc: petsc-users at mcs.anl.gov > Subject: Re: [EXTERNAL] Re: [petsc-users] Is there anything like a "DMPlexSetCones()" ? Matt: "I do not understand the "flag check". What is that?" ??????My bad, I should have referred to the "dm->setupcalled". ??????I believe this PetscBool is checked by the other DM (not just DMPlex) APIs. ??????The subsequent checks for dm->setupcalled == PETSC_TRUE is what I meant to say. ??????Here's a copy of DMSetUp(). PetscErrorCode DMSetUp(DM dm) 817: { 818: PetscFunctionBegin; 820: if (dm->setupcalled) PetscFunctionReturn(PETSC_SUCCESS); 821: PetscTryTypeMethod(dm, setup); 822: dm->setupcalled = PETSC_TRUE; 823: PetscFunctionReturn(PETSC_SUCCESS); 824: } "We could make a DMPlexSetCones(), but as you point out, the workflow for DMSetUp() would have to change. Where does the memory come from for your connectivity?" ??????It's a memory that I myself allocate based on a file's contents. ??????The nature of the I/O makes it so that I need to read the connectivity before I get a semblance of buffer sizes. ??????Otherwise, I would stick to the tried and tested way. Also, when replying to the PETSc developers, users must reply to petsc-users at mcs.anl.gov and not just to the individual email accounts of the developers, right? ________________________________ From: Matthew Knepley > Sent: Wednesday, May 1, 2024 8:07 PM To: Ferrand, Jesus A. > Cc: petsc-users at mcs.anl.gov > Subject: [EXTERNAL] Re: [petsc-users] Is there anything like a "DMPlexSetCones()" ? CAUTION: This email originated outside of Embry-Riddle Aeronautical University. Do not click links or open attachments unless you recognize the sender and know the content is safe. On Wed, May 1, 2024 at 3:34?PM Ferrand, Jesus A. > wrote: This Message Is From an External Sender This message came from outside your organization. Dear PETSc team: For a project that I'm working on, I need to manually build a DMPlex. >From studying the source code of the various APIs in which the plex is built from some supported file format, I get that the workflow is this: 1. DMPlexSetChart() <-- Input nCells + nVerts 2. DMPlexSetConeSize() <-- Input ConeSize for each point in [0,nCells) 3. DMSetUp() ? Allocates memory internally. 4. DMPlexGetCones() --> Gives you the memory onto which to write the cell connectivity. 5. *Write connectivity* 6. DMPlexReorderCell() <-- For each point in [0,nCells) I'm in a situation where the memory given by step (4) is available a-priori. I was hoping to skip steps 2, 3 , and 4 with something like a "DMPlexSetCones()", but such an API does not exist. My current workaround is to implement steps 2 through 4 as always and have double the memory allocated in the interim (my instance + DM's internal instance). I was thinking of looking for the name of the struct member and = it to my memory, but I can't overcome the flag check in DMSetUp() during later calls to DMPlexGetCones() or DMPlexGetTransitiveClosure(). I do not understand the "flag check". What is that? We could make a DMPlexSetCones(), but as you point out, the workflow for DMSetUp() would have to change. Where does the memory come from for your connectivity? Thanks, Matt Sincerely: J.A. Ferrand Embry-Riddle Aeronautical University - Daytona Beach - FL Ph.D. Candidate, Aerospace Engineering M.Sc. Aerospace Engineering B.Sc. Aerospace Engineering B.Sc. Computational Mathematics Phone: (386)-843-1829 Email(s): ferranj2 at my.erau.edu jesus.ferrand at gmail.com -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!YA9N-9_O6-4rSg848GHyIY2nSendFsTvYClTS9F7288syR3Qqj7dAAwx9vwfVzBedRxEAI057ebAdXwzNbD-x-OTHfg$ ________________________________ From: Matthew Knepley > Sent: Wednesday, May 1, 2024 8:07 PM To: Ferrand, Jesus A. > Cc: petsc-users at mcs.anl.gov > Subject: [EXTERNAL] Re: [petsc-users] Is there anything like a "DMPlexSetCones()" ? CAUTION: This email originated outside of Embry-Riddle Aeronautical University. Do not click links or open attachments unless you recognize the sender and know the content is safe. On Wed, May 1, 2024 at 3:34?PM Ferrand, Jesus A. > wrote: This Message Is From an External Sender This message came from outside your organization. Dear PETSc team: For a project that I'm working on, I need to manually build a DMPlex. >From studying the source code of the various APIs in which the plex is built from some supported file format, I get that the workflow is this: 1. DMPlexSetChart() <-- Input nCells + nVerts 2. DMPlexSetConeSize() <-- Input ConeSize for each point in [0,nCells) 3. DMSetUp() ? Allocates memory internally. 4. DMPlexGetCones() --> Gives you the memory onto which to write the cell connectivity. 5. *Write connectivity* 6. DMPlexReorderCell() <-- For each point in [0,nCells) I'm in a situation where the memory given by step (4) is available a-priori. I was hoping to skip steps 2, 3 , and 4 with something like a "DMPlexSetCones()", but such an API does not exist. My current workaround is to implement steps 2 through 4 as always and have double the memory allocated in the interim (my instance + DM's internal instance). I was thinking of looking for the name of the struct member and = it to my memory, but I can't overcome the flag check in DMSetUp() during later calls to DMPlexGetCones() or DMPlexGetTransitiveClosure(). I do not understand the "flag check". What is that? We could make a DMPlexSetCones(), but as you point out, the workflow for DMSetUp() would have to change. Where does the memory come from for your connectivity? Thanks, Matt Sincerely: J.A. Ferrand Embry-Riddle Aeronautical University - Daytona Beach - FL Ph.D. Candidate, Aerospace Engineering M.Sc. Aerospace Engineering B.Sc. Aerospace Engineering B.Sc. Computational Mathematics Phone: (386)-843-1829 Email(s): ferranj2 at my.erau.edu jesus.ferrand at gmail.com -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!YA9N-9_O6-4rSg848GHyIY2nSendFsTvYClTS9F7288syR3Qqj7dAAwx9vwfVzBedRxEAI057ebAdXwzNbD-x-OTHfg$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!YA9N-9_O6-4rSg848GHyIY2nSendFsTvYClTS9F7288syR3Qqj7dAAwx9vwfVzBedRxEAI057ebAdXwzNbD-x-OTHfg$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!YA9N-9_O6-4rSg848GHyIY2nSendFsTvYClTS9F7288syR3Qqj7dAAwx9vwfVzBedRxEAI057ebAdXwzNbD-x-OTHfg$ ________________________________ From: Matthew Knepley > Sent: Friday, May 3, 2024 6:45 PM To: Ferrand, Jesus A. > Cc: petsc-users at mcs.anl.gov > Subject: Re: [EXTERNAL] Re: [petsc-users] Is there anything like a "DMPlexSetCones()" ? On Wed, May 1, 2024 at 6:30?PM Ferrand, Jesus A. > wrote: "You know the number of cells if you can read connectivity. Do you mean that the format does not tell you the number of vertices?" Sort of... So, the format does provide the number of vertices, however, due to the way I read the data in parallel I don't know immediately how many local vertices there will be. Which prevents me from knowing the chart a priori. I figured out the number of vertices using PetscHashSetI to determine the number of unique entries in the connectivity list. Right, but that means that the connectivity you read in is not in local numbering. You will have to renumber it, so making a copy is probably necessary anyway. This is the same sort of processing I do for the parallel read. Thanks, Matt ________________________________ From: Matthew Knepley > Sent: Wednesday, May 1, 2024 8:52 PM To: Ferrand, Jesus A. > Cc: petsc-users at mcs.anl.gov > Subject: Re: [EXTERNAL] Re: [petsc-users] Is there anything like a "DMPlexSetCones()" ? On Wed, May 1, 2024 at 4:23?PM Ferrand, Jesus A. > wrote: Matt: My bad again, I need to clarify something that I just realized doesn't make sense. I said: "The nature of the I/O makes it so that I need to read the connectivity before I get a semblance of buffer sizes." Scratch that, obviously, if I can read the connectivity, I know the buffer sizes. What I should have said is that I cannot know the "chart" that one sets in DMPlexSetChart() a priori like in most cases. I need to determine the chart from the connectivity lists. You know the number of cells if you can read connectivity. Do you mean that the format does not tell you the number of vertices? Thanks, Matt ________________________________ From: Ferrand, Jesus A. > Sent: Wednesday, May 1, 2024 8:17 PM To: Matthew Knepley > Cc: petsc-users at mcs.anl.gov > Subject: Re: [EXTERNAL] Re: [petsc-users] Is there anything like a "DMPlexSetCones()" ? Matt: "I do not understand the "flag check". What is that?" ??????My bad, I should have referred to the "dm->setupcalled". ??????I believe this PetscBool is checked by the other DM (not just DMPlex) APIs. ??????The subsequent checks for dm->setupcalled == PETSC_TRUE is what I meant to say. ??????Here's a copy of DMSetUp(). PetscErrorCode DMSetUp(DM dm) 817: { 818: PetscFunctionBegin; 820: if (dm->setupcalled) PetscFunctionReturn(PETSC_SUCCESS); 821: PetscTryTypeMethod(dm, setup); 822: dm->setupcalled = PETSC_TRUE; 823: PetscFunctionReturn(PETSC_SUCCESS); 824: } "We could make a DMPlexSetCones(), but as you point out, the workflow for DMSetUp() would have to change. Where does the memory come from for your connectivity?" ??????It's a memory that I myself allocate based on a file's contents. ??????The nature of the I/O makes it so that I need to read the connectivity before I get a semblance of buffer sizes. ??????Otherwise, I would stick to the tried and tested way. Also, when replying to the PETSc developers, users must reply to petsc-users at mcs.anl.gov and not just to the individual email accounts of the developers, right? ________________________________ From: Matthew Knepley > Sent: Wednesday, May 1, 2024 8:07 PM To: Ferrand, Jesus A. > Cc: petsc-users at mcs.anl.gov > Subject: [EXTERNAL] Re: [petsc-users] Is there anything like a "DMPlexSetCones()" ? CAUTION: This email originated outside of Embry-Riddle Aeronautical University. Do not click links or open attachments unless you recognize the sender and know the content is safe. On Wed, May 1, 2024 at 3:34?PM Ferrand, Jesus A. > wrote: This Message Is From an External Sender This message came from outside your organization. Dear PETSc team: For a project that I'm working on, I need to manually build a DMPlex. >From studying the source code of the various APIs in which the plex is built from some supported file format, I get that the workflow is this: 1. DMPlexSetChart() <-- Input nCells + nVerts 2. DMPlexSetConeSize() <-- Input ConeSize for each point in [0,nCells) 3. DMSetUp() ? Allocates memory internally. 4. DMPlexGetCones() --> Gives you the memory onto which to write the cell connectivity. 5. *Write connectivity* 6. DMPlexReorderCell() <-- For each point in [0,nCells) I'm in a situation where the memory given by step (4) is available a-priori. I was hoping to skip steps 2, 3 , and 4 with something like a "DMPlexSetCones()", but such an API does not exist. My current workaround is to implement steps 2 through 4 as always and have double the memory allocated in the interim (my instance + DM's internal instance). I was thinking of looking for the name of the struct member and = it to my memory, but I can't overcome the flag check in DMSetUp() during later calls to DMPlexGetCones() or DMPlexGetTransitiveClosure(). I do not understand the "flag check". What is that? We could make a DMPlexSetCones(), but as you point out, the workflow for DMSetUp() would have to change. Where does the memory come from for your connectivity? Thanks, Matt Sincerely: J.A. Ferrand Embry-Riddle Aeronautical University - Daytona Beach - FL Ph.D. Candidate, Aerospace Engineering M.Sc. Aerospace Engineering B.Sc. Aerospace Engineering B.Sc. Computational Mathematics Phone: (386)-843-1829 Email(s): ferranj2 at my.erau.edu jesus.ferrand at gmail.com -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!YA9N-9_O6-4rSg848GHyIY2nSendFsTvYClTS9F7288syR3Qqj7dAAwx9vwfVzBedRxEAI057ebAdXwzNbD-x-OTHfg$ ________________________________ From: Matthew Knepley > Sent: Wednesday, May 1, 2024 8:07 PM To: Ferrand, Jesus A. > Cc: petsc-users at mcs.anl.gov > Subject: [EXTERNAL] Re: [petsc-users] Is there anything like a "DMPlexSetCones()" ? CAUTION: This email originated outside of Embry-Riddle Aeronautical University. Do not click links or open attachments unless you recognize the sender and know the content is safe. On Wed, May 1, 2024 at 3:34?PM Ferrand, Jesus A. > wrote: This Message Is From an External Sender This message came from outside your organization. Dear PETSc team: For a project that I'm working on, I need to manually build a DMPlex. >From studying the source code of the various APIs in which the plex is built from some supported file format, I get that the workflow is this: 1. DMPlexSetChart() <-- Input nCells + nVerts 2. DMPlexSetConeSize() <-- Input ConeSize for each point in [0,nCells) 3. DMSetUp() ? Allocates memory internally. 4. DMPlexGetCones() --> Gives you the memory onto which to write the cell connectivity. 5. *Write connectivity* 6. DMPlexReorderCell() <-- For each point in [0,nCells) I'm in a situation where the memory given by step (4) is available a-priori. I was hoping to skip steps 2, 3 , and 4 with something like a "DMPlexSetCones()", but such an API does not exist. My current workaround is to implement steps 2 through 4 as always and have double the memory allocated in the interim (my instance + DM's internal instance). I was thinking of looking for the name of the struct member and = it to my memory, but I can't overcome the flag check in DMSetUp() during later calls to DMPlexGetCones() or DMPlexGetTransitiveClosure(). I do not understand the "flag check". What is that? We could make a DMPlexSetCones(), but as you point out, the workflow for DMSetUp() would have to change. Where does the memory come from for your connectivity? Thanks, Matt Sincerely: J.A. Ferrand Embry-Riddle Aeronautical University - Daytona Beach - FL Ph.D. Candidate, Aerospace Engineering M.Sc. Aerospace Engineering B.Sc. Aerospace Engineering B.Sc. Computational Mathematics Phone: (386)-843-1829 Email(s): ferranj2 at my.erau.edu jesus.ferrand at gmail.com -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!YA9N-9_O6-4rSg848GHyIY2nSendFsTvYClTS9F7288syR3Qqj7dAAwx9vwfVzBedRxEAI057ebAdXwzNbD-x-OTHfg$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!YA9N-9_O6-4rSg848GHyIY2nSendFsTvYClTS9F7288syR3Qqj7dAAwx9vwfVzBedRxEAI057ebAdXwzNbD-x-OTHfg$ ________________________________ From: Matthew Knepley > Sent: Wednesday, May 1, 2024 8:52 PM To: Ferrand, Jesus A. > Cc: petsc-users at mcs.anl.gov > Subject: Re: [EXTERNAL] Re: [petsc-users] Is there anything like a "DMPlexSetCones()" ? On Wed, May 1, 2024 at 4:23?PM Ferrand, Jesus A. > wrote: Matt: My bad again, I need to clarify something that I just realized doesn't make sense. I said: "The nature of the I/O makes it so that I need to read the connectivity before I get a semblance of buffer sizes." Scratch that, obviously, if I can read the connectivity, I know the buffer sizes. What I should have said is that I cannot know the "chart" that one sets in DMPlexSetChart() a priori like in most cases. I need to determine the chart from the connectivity lists. You know the number of cells if you can read connectivity. Do you mean that the format does not tell you the number of vertices? Thanks, Matt ________________________________ From: Ferrand, Jesus A. > Sent: Wednesday, May 1, 2024 8:17 PM To: Matthew Knepley > Cc: petsc-users at mcs.anl.gov > Subject: Re: [EXTERNAL] Re: [petsc-users] Is there anything like a "DMPlexSetCones()" ? Matt: "I do not understand the "flag check". What is that?" ??????My bad, I should have referred to the "dm->setupcalled". ??????I believe this PetscBool is checked by the other DM (not just DMPlex) APIs. ??????The subsequent checks for dm->setupcalled == PETSC_TRUE is what I meant to say. ??????Here's a copy of DMSetUp(). PetscErrorCode DMSetUp(DM dm) 817: { 818: PetscFunctionBegin; 820: if (dm->setupcalled) PetscFunctionReturn(PETSC_SUCCESS); 821: PetscTryTypeMethod(dm, setup); 822: dm->setupcalled = PETSC_TRUE; 823: PetscFunctionReturn(PETSC_SUCCESS); 824: } "We could make a DMPlexSetCones(), but as you point out, the workflow for DMSetUp() would have to change. Where does the memory come from for your connectivity?" ??????It's a memory that I myself allocate based on a file's contents. ??????The nature of the I/O makes it so that I need to read the connectivity before I get a semblance of buffer sizes. ??????Otherwise, I would stick to the tried and tested way. Also, when replying to the PETSc developers, users must reply to petsc-users at mcs.anl.gov and not just to the individual email accounts of the developers, right? ________________________________ From: Matthew Knepley > Sent: Wednesday, May 1, 2024 8:07 PM To: Ferrand, Jesus A. > Cc: petsc-users at mcs.anl.gov > Subject: [EXTERNAL] Re: [petsc-users] Is there anything like a "DMPlexSetCones()" ? CAUTION: This email originated outside of Embry-Riddle Aeronautical University. Do not click links or open attachments unless you recognize the sender and know the content is safe. On Wed, May 1, 2024 at 3:34?PM Ferrand, Jesus A. > wrote: This Message Is From an External Sender This message came from outside your organization. Dear PETSc team: For a project that I'm working on, I need to manually build a DMPlex. >From studying the source code of the various APIs in which the plex is built from some supported file format, I get that the workflow is this: 1. DMPlexSetChart() <-- Input nCells + nVerts 2. DMPlexSetConeSize() <-- Input ConeSize for each point in [0,nCells) 3. DMSetUp() ? Allocates memory internally. 4. DMPlexGetCones() --> Gives you the memory onto which to write the cell connectivity. 5. *Write connectivity* 6. DMPlexReorderCell() <-- For each point in [0,nCells) I'm in a situation where the memory given by step (4) is available a-priori. I was hoping to skip steps 2, 3 , and 4 with something like a "DMPlexSetCones()", but such an API does not exist. My current workaround is to implement steps 2 through 4 as always and have double the memory allocated in the interim (my instance + DM's internal instance). I was thinking of looking for the name of the struct member and = it to my memory, but I can't overcome the flag check in DMSetUp() during later calls to DMPlexGetCones() or DMPlexGetTransitiveClosure(). I do not understand the "flag check". What is that? We could make a DMPlexSetCones(), but as you point out, the workflow for DMSetUp() would have to change. Where does the memory come from for your connectivity? Thanks, Matt Sincerely: J.A. Ferrand Embry-Riddle Aeronautical University - Daytona Beach - FL Ph.D. Candidate, Aerospace Engineering M.Sc. Aerospace Engineering B.Sc. Aerospace Engineering B.Sc. Computational Mathematics Phone: (386)-843-1829 Email(s): ferranj2 at my.erau.edu jesus.ferrand at gmail.com -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!YA9N-9_O6-4rSg848GHyIY2nSendFsTvYClTS9F7288syR3Qqj7dAAwx9vwfVzBedRxEAI057ebAdXwzNbD-x-OTHfg$ ________________________________ From: Matthew Knepley > Sent: Wednesday, May 1, 2024 8:07 PM To: Ferrand, Jesus A. > Cc: petsc-users at mcs.anl.gov > Subject: [EXTERNAL] Re: [petsc-users] Is there anything like a "DMPlexSetCones()" ? CAUTION: This email originated outside of Embry-Riddle Aeronautical University. Do not click links or open attachments unless you recognize the sender and know the content is safe. On Wed, May 1, 2024 at 3:34?PM Ferrand, Jesus A. > wrote: This Message Is From an External Sender This message came from outside your organization. Dear PETSc team: For a project that I'm working on, I need to manually build a DMPlex. >From studying the source code of the various APIs in which the plex is built from some supported file format, I get that the workflow is this: 1. DMPlexSetChart() <-- Input nCells + nVerts 2. DMPlexSetConeSize() <-- Input ConeSize for each point in [0,nCells) 3. DMSetUp() ? Allocates memory internally. 4. DMPlexGetCones() --> Gives you the memory onto which to write the cell connectivity. 5. *Write connectivity* 6. DMPlexReorderCell() <-- For each point in [0,nCells) I'm in a situation where the memory given by step (4) is available a-priori. I was hoping to skip steps 2, 3 , and 4 with something like a "DMPlexSetCones()", but such an API does not exist. My current workaround is to implement steps 2 through 4 as always and have double the memory allocated in the interim (my instance + DM's internal instance). I was thinking of looking for the name of the struct member and = it to my memory, but I can't overcome the flag check in DMSetUp() during later calls to DMPlexGetCones() or DMPlexGetTransitiveClosure(). I do not understand the "flag check". What is that? We could make a DMPlexSetCones(), but as you point out, the workflow for DMSetUp() would have to change. Where does the memory come from for your connectivity? Thanks, Matt Sincerely: J.A. Ferrand Embry-Riddle Aeronautical University - Daytona Beach - FL Ph.D. Candidate, Aerospace Engineering M.Sc. Aerospace Engineering B.Sc. Aerospace Engineering B.Sc. Computational Mathematics Phone: (386)-843-1829 Email(s): ferranj2 at my.erau.edu jesus.ferrand at gmail.com -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!YA9N-9_O6-4rSg848GHyIY2nSendFsTvYClTS9F7288syR3Qqj7dAAwx9vwfVzBedRxEAI057ebAdXwzNbD-x-OTHfg$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!YA9N-9_O6-4rSg848GHyIY2nSendFsTvYClTS9F7288syR3Qqj7dAAwx9vwfVzBedRxEAI057ebAdXwzNbD-x-OTHfg$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!YA9N-9_O6-4rSg848GHyIY2nSendFsTvYClTS9F7288syR3Qqj7dAAwx9vwfVzBedRxEAI057ebAdXwzNbD-x-OTHfg$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!YA9N-9_O6-4rSg848GHyIY2nSendFsTvYClTS9F7288syR3Qqj7dAAwx9vwfVzBedRxEAI057ebAdXwzNbD-x-OTHfg$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon May 6 09:55:31 2024 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 6 May 2024 10:55:31 -0400 Subject: [petsc-users] [EXTERNAL] Re: Is there anything like a "DMPlexSetCones()" ? In-Reply-To: References: Message-ID: On Mon, May 6, 2024 at 10:17?AM Ferrand, Jesus A. wrote: > "The cone sizes are used to allocate the array to hold the cones. How > would you have an array holding the cone information without knowing the > sizes?" > > You're right, I must first know the sizes (which I determine from the > file). > It lists the number of cells and their types by sequential chunks. > > The issue I am trying to describe, is that because I cannot know the > number of local vertices a priori, the pipeline > DMSetChart()->DMPlexSetConeSize()->DMSetUp() essentially produces a > redundant array (because I already have the connectivity that I read from > the file). > As far as I can tell, in my case, the end result of calling these APIs is > already available, hence my wish for a "DMPlexSetCones()". > If such an API existed, I would then attempt to retroactively set the > Chart and the ConeSizes. > > Like you hinted in a reply, for this extra array to be worthwhile, I would > have to keep it as a global numbering list (which I want to by the way). > I was hoping to at least keep them in the DM's global PetscSection as > offsets. > Unfortunately, PetscSection has a similar workflow to DM: > PetscSectionSetChart(), PetscSectionAddDof(), PetscSectionSetUp(), > PetscSectionSetOffset(). > I looked at the other parallel code. It does maintain two arrays, so that you can compute this stuff. I am reluctant to add this interface, since I think the optimization is very small and it is not a good pattern. You will save allocating the cell connectivity. However, if you interpolate the mesh, this is small. This is the same size as the cell orientations, which we will also have to allocate. We are likely to allocate vectors over the mesh. It does not seem likely that this allocation will increase the peak allocation, and it will be freed after creation, so total size will not be affected. What is the argument for this optimization? Thanks, Matt > ------------------------------ > *From:* Matthew Knepley > *Sent:* Monday, May 6, 2024 7:52 AM > *To:* Ferrand, Jesus A. > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [EXTERNAL] Re: [petsc-users] Is there anything like a > "DMPlexSetCones()" ? > > On Fri, May 3, 2024 at 7:32?PM Ferrand, Jesus A. > wrote: > > "Right, but that means that the connectivity you read in is not in local > numbering. You will have to renumber it, so making a copy is probably > necessary anyway. This is the same sort of processing I do for the parallel > read." > > True, the connectivity is read in global numbering. > Renumbering is facilitated by the table from PetscHashSetI. > The connectivity list I am able to renumber in place though, I don't see > the need to have another interim copy of the numbers. > My concern is that I have two memory chunks that both contain the same > information. > The only reason It has to be this way is because I can't manually set the > Cones first and the Chart/ConeSizes second. > > > The cone sizes are used to allocate the array to hold the cones. How would > you have an array holding the cone information without knowing the sizes? > > Thanks, > > Matt > > > ------------------------------ > *From:* Matthew Knepley > *Sent:* Friday, May 3, 2024 6:45 PM > *To:* Ferrand, Jesus A. > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [EXTERNAL] Re: [petsc-users] Is there anything like a > "DMPlexSetCones()" ? > > On Wed, May 1, 2024 at 6:30?PM Ferrand, Jesus A. > wrote: > > "You know the number of cells if you can read connectivity. Do you mean > that the format does not tell you > the number of vertices?" > > Sort of... > So, the format does provide the number of vertices, however, due to the > way I read the data in parallel I don't know immediately how many local > vertices there will be. > Which prevents me from knowing the chart a priori. > I figured out the number of vertices using PetscHashSetI to determine the > number of unique entries in the connectivity list. > > > Right, but that means that the connectivity you read in is not in local > numbering. You will have to renumber it, so making a copy is probably > necessary anyway. This is the same sort of processing I do for the parallel > read. > > Thanks, > > Matt > > > ------------------------------ > *From:* Matthew Knepley > *Sent:* Wednesday, May 1, 2024 8:52 PM > *To:* Ferrand, Jesus A. > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [EXTERNAL] Re: [petsc-users] Is there anything like a > "DMPlexSetCones()" ? > > On Wed, May 1, 2024 at 4:23?PM Ferrand, Jesus A. > wrote: > > Matt: > > My bad again, I need to clarify something that I just realized doesn't > make sense. > I said: "The nature of the I/O makes it so that I need to read the > connectivity before I get a semblance of buffer sizes." > Scratch that, obviously, if I can read the connectivity, I know the buffer > sizes. > > What I should have said is that I cannot know the "chart" that one sets in > DMPlexSetChart() a priori like in most cases. > I need to determine the chart from the connectivity lists. > > > You know the number of cells if you can read connectivity. Do you mean > that the format does not tell you > the number of vertices? > > Thanks, > > Matt > > > ------------------------------ > *From:* Ferrand, Jesus A. > *Sent:* Wednesday, May 1, 2024 8:17 PM > *To:* Matthew Knepley > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [EXTERNAL] Re: [petsc-users] Is there anything like a > "DMPlexSetCones()" ? > > Matt: > > "I do not understand the "flag check". What is that?" > > My bad, I should have referred to the "dm->setupcalled". > I believe this PetscBool is checked by the other DM (not just DMPlex) APIs. > The subsequent checks for dm->setupcalled == PETSC_TRUE is what I meant to > say. > Here's a copy of DMSetUp(). > > *PetscErrorCode DMSetUp (DM dm)* > 817: { > 818: PetscFunctionBegin ; > 820: if (dm->setupcalled) PetscFunctionReturn (PETSC_SUCCESS ); > 821: PetscTryTypeMethod (dm, setup); > 822: dm->setupcalled = PETSC_TRUE ; > 823: PetscFunctionReturn (PETSC_SUCCESS ); > 824: } > > "We could make a DMPlexSetCones(), but as you point out, the workflow for > DMSetUp() would have to change. Where does the memory come from for your > connectivity?" > It's a memory that I myself allocate based on a file's contents. > The nature of the I/O makes it so that I need to read the connectivity > before I get a semblance of buffer sizes. > Otherwise, I would stick to the tried and tested way. > > Also, when replying to the PETSc developers, users must reply to > petsc-users at mcs.anl.gov and not just to the individual email accounts of > the developers, right? > > > > > ------------------------------ > *From:* Matthew Knepley > *Sent:* Wednesday, May 1, 2024 8:07 PM > *To:* Ferrand, Jesus A. > *Cc:* petsc-users at mcs.anl.gov > *Subject:* [EXTERNAL] Re: [petsc-users] Is there anything like a > "DMPlexSetCones()" ? > > *CAUTION:* This email originated outside of Embry-Riddle Aeronautical > University. Do not click links or open attachments unless you recognize the > sender and know the content is safe. > > On Wed, May 1, 2024 at 3:34?PM Ferrand, Jesus A. > wrote: > > This Message Is From an External Sender > This message came from outside your organization. > > Dear PETSc team: > > For a project that I'm working on, I need to manually build a DMPlex. > From studying the source code of the various APIs in which the plex is > built from some supported file format, I get that the workflow is this: > > > 1. DMPlexSetChart() <-- Input nCells + nVerts > 2. DMPlexSetConeSize() <-- Input ConeSize for each point in [0,nCells) > 3. DMSetUp() ? Allocates memory internally. > 4. DMPlexGetCones() --> Gives you the memory onto which to write the > cell connectivity. > 5. *Write connectivity* > 6. DMPlexReorderCell() <-- For each point in [0,nCells) > > > I'm in a situation where the memory given by step (4) is available > a-priori. > I was hoping to skip steps 2, 3 , and 4 with something like a > "DMPlexSetCones()", but such an API does not exist. > My current workaround is to implement steps 2 through 4 as always and have > double the memory allocated in the interim (my instance + DM's internal > instance). > I was thinking of looking for the name of the struct member and = it to my > memory, but I can't overcome the flag check in DMSetUp() during later calls > to DMPlexGetCones() or DMPlexGetTransitiveClosure(). > > > I do not understand the "flag check". What is that? > > We could make a DMPlexSetCones(), but as you point out, the workflow for > DMSetUp() would have to change. Where does the memory come from for your > connectivity? > > Thanks, > > Matt > > > > Sincerely: > > *J.A. Ferrand* > > Embry-Riddle Aeronautical University - Daytona Beach - FL > Ph.D. Candidate, Aerospace Engineering > > M.Sc. Aerospace Engineering > > B.Sc. Aerospace Engineering > > B.Sc. Computational Mathematics > > > *Phone:* (386)-843-1829 > > *Email(s):* ferranj2 at my.erau.edu > > jesus.ferrand at gmail.com > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!cfVoObgiLc90zIrUaHVMSXqAJtv69Ufe3A1lF0j0zEtULlt1kmKQ7ywA6tGfxdvxMMGDSlHe66KyWihF3ryu$ > > > ------------------------------ > *From:* Matthew Knepley > *Sent:* Wednesday, May 1, 2024 8:07 PM > *To:* Ferrand, Jesus A. > *Cc:* petsc-users at mcs.anl.gov > *Subject:* [EXTERNAL] Re: [petsc-users] Is there anything like a > "DMPlexSetCones()" ? > > *CAUTION:* This email originated outside of Embry-Riddle Aeronautical > University. Do not click links or open attachments unless you recognize the > sender and know the content is safe. > > On Wed, May 1, 2024 at 3:34?PM Ferrand, Jesus A. > wrote: > > This Message Is From an External Sender > This message came from outside your organization. > > Dear PETSc team: > > For a project that I'm working on, I need to manually build a DMPlex. > From studying the source code of the various APIs in which the plex is > built from some supported file format, I get that the workflow is this: > > > 1. DMPlexSetChart() <-- Input nCells + nVerts > 2. DMPlexSetConeSize() <-- Input ConeSize for each point in [0,nCells) > 3. DMSetUp() ? Allocates memory internally. > 4. DMPlexGetCones() --> Gives you the memory onto which to write the > cell connectivity. > 5. *Write connectivity* > 6. DMPlexReorderCell() <-- For each point in [0,nCells) > > > I'm in a situation where the memory given by step (4) is available > a-priori. > I was hoping to skip steps 2, 3 , and 4 with something like a > "DMPlexSetCones()", but such an API does not exist. > My current workaround is to implement steps 2 through 4 as always and have > double the memory allocated in the interim (my instance + DM's internal > instance). > I was thinking of looking for the name of the struct member and = it to my > memory, but I can't overcome the flag check in DMSetUp() during later calls > to DMPlexGetCones() or DMPlexGetTransitiveClosure(). > > > I do not understand the "flag check". What is that? > > We could make a DMPlexSetCones(), but as you point out, the workflow for > DMSetUp() would have to change. Where does the memory come from for your > connectivity? > > Thanks, > > Matt > > > > Sincerely: > > *J.A. Ferrand* > > Embry-Riddle Aeronautical University - Daytona Beach - FL > Ph.D. Candidate, Aerospace Engineering > > M.Sc. Aerospace Engineering > > B.Sc. Aerospace Engineering > > B.Sc. Computational Mathematics > > > *Phone:* (386)-843-1829 > > *Email(s):* ferranj2 at my.erau.edu > > jesus.ferrand at gmail.com > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!cfVoObgiLc90zIrUaHVMSXqAJtv69Ufe3A1lF0j0zEtULlt1kmKQ7ywA6tGfxdvxMMGDSlHe66KyWihF3ryu$ > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!cfVoObgiLc90zIrUaHVMSXqAJtv69Ufe3A1lF0j0zEtULlt1kmKQ7ywA6tGfxdvxMMGDSlHe66KyWihF3ryu$ > > > ------------------------------ > *From:* Matthew Knepley > *Sent:* Wednesday, May 1, 2024 8:52 PM > *To:* Ferrand, Jesus A. > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [EXTERNAL] Re: [petsc-users] Is there anything like a > "DMPlexSetCones()" ? > > On Wed, May 1, 2024 at 4:23?PM Ferrand, Jesus A. > wrote: > > Matt: > > My bad again, I need to clarify something that I just realized doesn't > make sense. > I said: "The nature of the I/O makes it so that I need to read the > connectivity before I get a semblance of buffer sizes." > Scratch that, obviously, if I can read the connectivity, I know the buffer > sizes. > > What I should have said is that I cannot know the "chart" that one sets in > DMPlexSetChart() a priori like in most cases. > I need to determine the chart from the connectivity lists. > > > You know the number of cells if you can read connectivity. Do you mean > that the format does not tell you > the number of vertices? > > Thanks, > > Matt > > > ------------------------------ > *From:* Ferrand, Jesus A. > *Sent:* Wednesday, May 1, 2024 8:17 PM > *To:* Matthew Knepley > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [EXTERNAL] Re: [petsc-users] Is there anything like a > "DMPlexSetCones()" ? > > Matt: > > "I do not understand the "flag check". What is that?" > > My bad, I should have referred to the "dm->setupcalled". > I believe this PetscBool is checked by the other DM (not just DMPlex) APIs. > The subsequent checks for dm->setupcalled == PETSC_TRUE is what I meant to > say. > Here's a copy of DMSetUp(). > > *PetscErrorCode DMSetUp (DM dm)* > 817: { > 818: PetscFunctionBegin ; > 820: if (dm->setupcalled) PetscFunctionReturn (PETSC_SUCCESS ); > 821: PetscTryTypeMethod (dm, setup); > 822: dm->setupcalled = PETSC_TRUE ; > 823: PetscFunctionReturn (PETSC_SUCCESS ); > 824: } > > "We could make a DMPlexSetCones(), but as you point out, the workflow for > DMSetUp() would have to change. Where does the memory come from for your > connectivity?" > It's a memory that I myself allocate based on a file's contents. > The nature of the I/O makes it so that I need to read the connectivity > before I get a semblance of buffer sizes. > Otherwise, I would stick to the tried and tested way. > > Also, when replying to the PETSc developers, users must reply to > petsc-users at mcs.anl.gov and not just to the individual email accounts of > the developers, right? > > > > > ------------------------------ > *From:* Matthew Knepley > *Sent:* Wednesday, May 1, 2024 8:07 PM > *To:* Ferrand, Jesus A. > *Cc:* petsc-users at mcs.anl.gov > *Subject:* [EXTERNAL] Re: [petsc-users] Is there anything like a > "DMPlexSetCones()" ? > > *CAUTION:* This email originated outside of Embry-Riddle Aeronautical > University. Do not click links or open attachments unless you recognize the > sender and know the content is safe. > > On Wed, May 1, 2024 at 3:34?PM Ferrand, Jesus A. > wrote: > > This Message Is From an External Sender > This message came from outside your organization. > > Dear PETSc team: > > For a project that I'm working on, I need to manually build a DMPlex. > From studying the source code of the various APIs in which the plex is > built from some supported file format, I get that the workflow is this: > > > 1. DMPlexSetChart() <-- Input nCells + nVerts > 2. DMPlexSetConeSize() <-- Input ConeSize for each point in [0,nCells) > 3. DMSetUp() ? Allocates memory internally. > 4. DMPlexGetCones() --> Gives you the memory onto which to write the > cell connectivity. > 5. *Write connectivity* > 6. DMPlexReorderCell() <-- For each point in [0,nCells) > > > I'm in a situation where the memory given by step (4) is available > a-priori. > I was hoping to skip steps 2, 3 , and 4 with something like a > "DMPlexSetCones()", but such an API does not exist. > My current workaround is to implement steps 2 through 4 as always and have > double the memory allocated in the interim (my instance + DM's internal > instance). > I was thinking of looking for the name of the struct member and = it to my > memory, but I can't overcome the flag check in DMSetUp() during later calls > to DMPlexGetCones() or DMPlexGetTransitiveClosure(). > > > I do not understand the "flag check". What is that? > > We could make a DMPlexSetCones(), but as you point out, the workflow for > DMSetUp() would have to change. Where does the memory come from for your > connectivity? > > Thanks, > > Matt > > > > Sincerely: > > *J.A. Ferrand* > > Embry-Riddle Aeronautical University - Daytona Beach - FL > Ph.D. Candidate, Aerospace Engineering > > M.Sc. Aerospace Engineering > > B.Sc. Aerospace Engineering > > B.Sc. Computational Mathematics > > > *Phone:* (386)-843-1829 > > *Email(s):* ferranj2 at my.erau.edu > > jesus.ferrand at gmail.com > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!cfVoObgiLc90zIrUaHVMSXqAJtv69Ufe3A1lF0j0zEtULlt1kmKQ7ywA6tGfxdvxMMGDSlHe66KyWihF3ryu$ > > > ------------------------------ > *From:* Matthew Knepley > *Sent:* Wednesday, May 1, 2024 8:07 PM > *To:* Ferrand, Jesus A. > *Cc:* petsc-users at mcs.anl.gov > *Subject:* [EXTERNAL] Re: [petsc-users] Is there anything like a > "DMPlexSetCones()" ? > > *CAUTION:* This email originated outside of Embry-Riddle Aeronautical > University. Do not click links or open attachments unless you recognize the > sender and know the content is safe. > > On Wed, May 1, 2024 at 3:34?PM Ferrand, Jesus A. > wrote: > > This Message Is From an External Sender > This message came from outside your organization. > > Dear PETSc team: > > For a project that I'm working on, I need to manually build a DMPlex. > From studying the source code of the various APIs in which the plex is > built from some supported file format, I get that the workflow is this: > > > 1. DMPlexSetChart() <-- Input nCells + nVerts > 2. DMPlexSetConeSize() <-- Input ConeSize for each point in [0,nCells) > 3. DMSetUp() ? Allocates memory internally. > 4. DMPlexGetCones() --> Gives you the memory onto which to write the > cell connectivity. > 5. *Write connectivity* > 6. DMPlexReorderCell() <-- For each point in [0,nCells) > > > I'm in a situation where the memory given by step (4) is available > a-priori. > I was hoping to skip steps 2, 3 , and 4 with something like a > "DMPlexSetCones()", but such an API does not exist. > My current workaround is to implement steps 2 through 4 as always and have > double the memory allocated in the interim (my instance + DM's internal > instance). > I was thinking of looking for the name of the struct member and = it to my > memory, but I can't overcome the flag check in DMSetUp() during later calls > to DMPlexGetCones() or DMPlexGetTransitiveClosure(). > > > I do not understand the "flag check". What is that? > > We could make a DMPlexSetCones(), but as you point out, the workflow for > DMSetUp() would have to change. Where does the memory come from for your > connectivity? > > Thanks, > > Matt > > > > Sincerely: > > *J.A. Ferrand* > > Embry-Riddle Aeronautical University - Daytona Beach - FL > Ph.D. Candidate, Aerospace Engineering > > M.Sc. Aerospace Engineering > > B.Sc. Aerospace Engineering > > B.Sc. Computational Mathematics > > > *Phone:* (386)-843-1829 > > *Email(s):* ferranj2 at my.erau.edu > > jesus.ferrand at gmail.com > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!cfVoObgiLc90zIrUaHVMSXqAJtv69Ufe3A1lF0j0zEtULlt1kmKQ7ywA6tGfxdvxMMGDSlHe66KyWihF3ryu$ > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!cfVoObgiLc90zIrUaHVMSXqAJtv69Ufe3A1lF0j0zEtULlt1kmKQ7ywA6tGfxdvxMMGDSlHe66KyWihF3ryu$ > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!cfVoObgiLc90zIrUaHVMSXqAJtv69Ufe3A1lF0j0zEtULlt1kmKQ7ywA6tGfxdvxMMGDSlHe66KyWihF3ryu$ > > > ------------------------------ > *From:* Matthew Knepley > *Sent:* Friday, May 3, 2024 6:45 PM > *To:* Ferrand, Jesus A. > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [EXTERNAL] Re: [petsc-users] Is there anything like a > "DMPlexSetCones()" ? > > On Wed, May 1, 2024 at 6:30?PM Ferrand, Jesus A. > wrote: > > "You know the number of cells if you can read connectivity. Do you mean > that the format does not tell you > the number of vertices?" > > Sort of... > So, the format does provide the number of vertices, however, due to the > way I read the data in parallel I don't know immediately how many local > vertices there will be. > Which prevents me from knowing the chart a priori. > I figured out the number of vertices using PetscHashSetI to determine the > number of unique entries in the connectivity list. > > > Right, but that means that the connectivity you read in is not in local > numbering. You will have to renumber it, so making a copy is probably > necessary anyway. This is the same sort of processing I do for the parallel > read. > > Thanks, > > Matt > > > ------------------------------ > *From:* Matthew Knepley > *Sent:* Wednesday, May 1, 2024 8:52 PM > *To:* Ferrand, Jesus A. > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [EXTERNAL] Re: [petsc-users] Is there anything like a > "DMPlexSetCones()" ? > > On Wed, May 1, 2024 at 4:23?PM Ferrand, Jesus A. > wrote: > > Matt: > > My bad again, I need to clarify something that I just realized doesn't > make sense. > I said: "The nature of the I/O makes it so that I need to read the > connectivity before I get a semblance of buffer sizes." > Scratch that, obviously, if I can read the connectivity, I know the buffer > sizes. > > What I should have said is that I cannot know the "chart" that one sets in > DMPlexSetChart() a priori like in most cases. > I need to determine the chart from the connectivity lists. > > > You know the number of cells if you can read connectivity. Do you mean > that the format does not tell you > the number of vertices? > > Thanks, > > Matt > > > ------------------------------ > *From:* Ferrand, Jesus A. > *Sent:* Wednesday, May 1, 2024 8:17 PM > *To:* Matthew Knepley > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [EXTERNAL] Re: [petsc-users] Is there anything like a > "DMPlexSetCones()" ? > > Matt: > > "I do not understand the "flag check". What is that?" > > My bad, I should have referred to the "dm->setupcalled". > I believe this PetscBool is checked by the other DM (not just DMPlex) APIs. > The subsequent checks for dm->setupcalled == PETSC_TRUE is what I meant to > say. > Here's a copy of DMSetUp(). > > *PetscErrorCode DMSetUp (DM dm)* > 817: { > 818: PetscFunctionBegin ; > 820: if (dm->setupcalled) PetscFunctionReturn (PETSC_SUCCESS ); > 821: PetscTryTypeMethod (dm, setup); > 822: dm->setupcalled = PETSC_TRUE ; > 823: PetscFunctionReturn (PETSC_SUCCESS ); > 824: } > > "We could make a DMPlexSetCones(), but as you point out, the workflow for > DMSetUp() would have to change. Where does the memory come from for your > connectivity?" > It's a memory that I myself allocate based on a file's contents. > The nature of the I/O makes it so that I need to read the connectivity > before I get a semblance of buffer sizes. > Otherwise, I would stick to the tried and tested way. > > Also, when replying to the PETSc developers, users must reply to > petsc-users at mcs.anl.gov and not just to the individual email accounts of > the developers, right? > > > > > ------------------------------ > *From:* Matthew Knepley > *Sent:* Wednesday, May 1, 2024 8:07 PM > *To:* Ferrand, Jesus A. > *Cc:* petsc-users at mcs.anl.gov > *Subject:* [EXTERNAL] Re: [petsc-users] Is there anything like a > "DMPlexSetCones()" ? > > *CAUTION:* This email originated outside of Embry-Riddle Aeronautical > University. Do not click links or open attachments unless you recognize the > sender and know the content is safe. > > On Wed, May 1, 2024 at 3:34?PM Ferrand, Jesus A. > wrote: > > This Message Is From an External Sender > This message came from outside your organization. > > Dear PETSc team: > > For a project that I'm working on, I need to manually build a DMPlex. > From studying the source code of the various APIs in which the plex is > built from some supported file format, I get that the workflow is this: > > > 1. DMPlexSetChart() <-- Input nCells + nVerts > 2. DMPlexSetConeSize() <-- Input ConeSize for each point in [0,nCells) > 3. DMSetUp() ? Allocates memory internally. > 4. DMPlexGetCones() --> Gives you the memory onto which to write the > cell connectivity. > 5. *Write connectivity* > 6. DMPlexReorderCell() <-- For each point in [0,nCells) > > > I'm in a situation where the memory given by step (4) is available > a-priori. > I was hoping to skip steps 2, 3 , and 4 with something like a > "DMPlexSetCones()", but such an API does not exist. > My current workaround is to implement steps 2 through 4 as always and have > double the memory allocated in the interim (my instance + DM's internal > instance). > I was thinking of looking for the name of the struct member and = it to my > memory, but I can't overcome the flag check in DMSetUp() during later calls > to DMPlexGetCones() or DMPlexGetTransitiveClosure(). > > > I do not understand the "flag check". What is that? > > We could make a DMPlexSetCones(), but as you point out, the workflow for > DMSetUp() would have to change. Where does the memory come from for your > connectivity? > > Thanks, > > Matt > > > > Sincerely: > > *J.A. Ferrand* > > Embry-Riddle Aeronautical University - Daytona Beach - FL > Ph.D. Candidate, Aerospace Engineering > > M.Sc. Aerospace Engineering > > B.Sc. Aerospace Engineering > > B.Sc. Computational Mathematics > > > *Phone:* (386)-843-1829 > > *Email(s):* ferranj2 at my.erau.edu > > jesus.ferrand at gmail.com > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!cfVoObgiLc90zIrUaHVMSXqAJtv69Ufe3A1lF0j0zEtULlt1kmKQ7ywA6tGfxdvxMMGDSlHe66KyWihF3ryu$ > > > ------------------------------ > *From:* Matthew Knepley > *Sent:* Wednesday, May 1, 2024 8:07 PM > *To:* Ferrand, Jesus A. > *Cc:* petsc-users at mcs.anl.gov > *Subject:* [EXTERNAL] Re: [petsc-users] Is there anything like a > "DMPlexSetCones()" ? > > *CAUTION:* This email originated outside of Embry-Riddle Aeronautical > University. Do not click links or open attachments unless you recognize the > sender and know the content is safe. > > On Wed, May 1, 2024 at 3:34?PM Ferrand, Jesus A. > wrote: > > This Message Is From an External Sender > This message came from outside your organization. > > Dear PETSc team: > > For a project that I'm working on, I need to manually build a DMPlex. > From studying the source code of the various APIs in which the plex is > built from some supported file format, I get that the workflow is this: > > > 1. DMPlexSetChart() <-- Input nCells + nVerts > 2. DMPlexSetConeSize() <-- Input ConeSize for each point in [0,nCells) > 3. DMSetUp() ? Allocates memory internally. > 4. DMPlexGetCones() --> Gives you the memory onto which to write the > cell connectivity. > 5. *Write connectivity* > 6. DMPlexReorderCell() <-- For each point in [0,nCells) > > > I'm in a situation where the memory given by step (4) is available > a-priori. > I was hoping to skip steps 2, 3 , and 4 with something like a > "DMPlexSetCones()", but such an API does not exist. > My current workaround is to implement steps 2 through 4 as always and have > double the memory allocated in the interim (my instance + DM's internal > instance). > I was thinking of looking for the name of the struct member and = it to my > memory, but I can't overcome the flag check in DMSetUp() during later calls > to DMPlexGetCones() or DMPlexGetTransitiveClosure(). > > > I do not understand the "flag check". What is that? > > We could make a DMPlexSetCones(), but as you point out, the workflow for > DMSetUp() would have to change. Where does the memory come from for your > connectivity? > > Thanks, > > Matt > > > > Sincerely: > > *J.A. Ferrand* > > Embry-Riddle Aeronautical University - Daytona Beach - FL > Ph.D. Candidate, Aerospace Engineering > > M.Sc. Aerospace Engineering > > B.Sc. Aerospace Engineering > > B.Sc. Computational Mathematics > > > *Phone:* (386)-843-1829 > > *Email(s):* ferranj2 at my.erau.edu > > jesus.ferrand at gmail.com > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!cfVoObgiLc90zIrUaHVMSXqAJtv69Ufe3A1lF0j0zEtULlt1kmKQ7ywA6tGfxdvxMMGDSlHe66KyWihF3ryu$ > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!cfVoObgiLc90zIrUaHVMSXqAJtv69Ufe3A1lF0j0zEtULlt1kmKQ7ywA6tGfxdvxMMGDSlHe66KyWihF3ryu$ > > > ------------------------------ > *From:* Matthew Knepley > *Sent:* Wednesday, May 1, 2024 8:52 PM > *To:* Ferrand, Jesus A. > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [EXTERNAL] Re: [petsc-users] Is there anything like a > "DMPlexSetCones()" ? > > On Wed, May 1, 2024 at 4:23?PM Ferrand, Jesus A. > wrote: > > Matt: > > My bad again, I need to clarify something that I just realized doesn't > make sense. > I said: "The nature of the I/O makes it so that I need to read the > connectivity before I get a semblance of buffer sizes." > Scratch that, obviously, if I can read the connectivity, I know the buffer > sizes. > > What I should have said is that I cannot know the "chart" that one sets in > DMPlexSetChart() a priori like in most cases. > I need to determine the chart from the connectivity lists. > > > You know the number of cells if you can read connectivity. Do you mean > that the format does not tell you > the number of vertices? > > Thanks, > > Matt > > > ------------------------------ > *From:* Ferrand, Jesus A. > *Sent:* Wednesday, May 1, 2024 8:17 PM > *To:* Matthew Knepley > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [EXTERNAL] Re: [petsc-users] Is there anything like a > "DMPlexSetCones()" ? > > Matt: > > "I do not understand the "flag check". What is that?" > > My bad, I should have referred to the "dm->setupcalled". > I believe this PetscBool is checked by the other DM (not just DMPlex) APIs. > The subsequent checks for dm->setupcalled == PETSC_TRUE is what I meant to > say. > Here's a copy of DMSetUp(). > > *PetscErrorCode DMSetUp (DM dm)* > 817: { > 818: PetscFunctionBegin ; > 820: if (dm->setupcalled) PetscFunctionReturn (PETSC_SUCCESS ); > 821: PetscTryTypeMethod (dm, setup); > 822: dm->setupcalled = PETSC_TRUE ; > 823: PetscFunctionReturn (PETSC_SUCCESS ); > 824: } > > "We could make a DMPlexSetCones(), but as you point out, the workflow for > DMSetUp() would have to change. Where does the memory come from for your > connectivity?" > It's a memory that I myself allocate based on a file's contents. > The nature of the I/O makes it so that I need to read the connectivity > before I get a semblance of buffer sizes. > Otherwise, I would stick to the tried and tested way. > > Also, when replying to the PETSc developers, users must reply to > petsc-users at mcs.anl.gov and not just to the individual email accounts of > the developers, right? > > > > > ------------------------------ > *From:* Matthew Knepley > *Sent:* Wednesday, May 1, 2024 8:07 PM > *To:* Ferrand, Jesus A. > *Cc:* petsc-users at mcs.anl.gov > *Subject:* [EXTERNAL] Re: [petsc-users] Is there anything like a > "DMPlexSetCones()" ? > > *CAUTION:* This email originated outside of Embry-Riddle Aeronautical > University. Do not click links or open attachments unless you recognize the > sender and know the content is safe. > > On Wed, May 1, 2024 at 3:34?PM Ferrand, Jesus A. > wrote: > > This Message Is From an External Sender > This message came from outside your organization. > > Dear PETSc team: > > For a project that I'm working on, I need to manually build a DMPlex. > From studying the source code of the various APIs in which the plex is > built from some supported file format, I get that the workflow is this: > > > 1. DMPlexSetChart() <-- Input nCells + nVerts > 2. DMPlexSetConeSize() <-- Input ConeSize for each point in [0,nCells) > 3. DMSetUp() ? Allocates memory internally. > 4. DMPlexGetCones() --> Gives you the memory onto which to write the > cell connectivity. > 5. *Write connectivity* > 6. DMPlexReorderCell() <-- For each point in [0,nCells) > > > I'm in a situation where the memory given by step (4) is available > a-priori. > I was hoping to skip steps 2, 3 , and 4 with something like a > "DMPlexSetCones()", but such an API does not exist. > My current workaround is to implement steps 2 through 4 as always and have > double the memory allocated in the interim (my instance + DM's internal > instance). > I was thinking of looking for the name of the struct member and = it to my > memory, but I can't overcome the flag check in DMSetUp() during later calls > to DMPlexGetCones() or DMPlexGetTransitiveClosure(). > > > I do not understand the "flag check". What is that? > > We could make a DMPlexSetCones(), but as you point out, the workflow for > DMSetUp() would have to change. Where does the memory come from for your > connectivity? > > Thanks, > > Matt > > > > Sincerely: > > *J.A. Ferrand* > > Embry-Riddle Aeronautical University - Daytona Beach - FL > Ph.D. Candidate, Aerospace Engineering > > M.Sc. Aerospace Engineering > > B.Sc. Aerospace Engineering > > B.Sc. Computational Mathematics > > > *Phone:* (386)-843-1829 > > *Email(s):* ferranj2 at my.erau.edu > > jesus.ferrand at gmail.com > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!cfVoObgiLc90zIrUaHVMSXqAJtv69Ufe3A1lF0j0zEtULlt1kmKQ7ywA6tGfxdvxMMGDSlHe66KyWihF3ryu$ > > > ------------------------------ > *From:* Matthew Knepley > *Sent:* Wednesday, May 1, 2024 8:07 PM > *To:* Ferrand, Jesus A. > *Cc:* petsc-users at mcs.anl.gov > *Subject:* [EXTERNAL] Re: [petsc-users] Is there anything like a > "DMPlexSetCones()" ? > > *CAUTION:* This email originated outside of Embry-Riddle Aeronautical > University. Do not click links or open attachments unless you recognize the > sender and know the content is safe. > > On Wed, May 1, 2024 at 3:34?PM Ferrand, Jesus A. > wrote: > > This Message Is From an External Sender > This message came from outside your organization. > > Dear PETSc team: > > For a project that I'm working on, I need to manually build a DMPlex. > From studying the source code of the various APIs in which the plex is > built from some supported file format, I get that the workflow is this: > > > 1. DMPlexSetChart() <-- Input nCells + nVerts > 2. DMPlexSetConeSize() <-- Input ConeSize for each point in [0,nCells) > 3. DMSetUp() ? Allocates memory internally. > 4. DMPlexGetCones() --> Gives you the memory onto which to write the > cell connectivity. > 5. *Write connectivity* > 6. DMPlexReorderCell() <-- For each point in [0,nCells) > > > I'm in a situation where the memory given by step (4) is available > a-priori. > I was hoping to skip steps 2, 3 , and 4 with something like a > "DMPlexSetCones()", but such an API does not exist. > My current workaround is to implement steps 2 through 4 as always and have > double the memory allocated in the interim (my instance + DM's internal > instance). > I was thinking of looking for the name of the struct member and = it to my > memory, but I can't overcome the flag check in DMSetUp() during later calls > to DMPlexGetCones() or DMPlexGetTransitiveClosure(). > > > I do not understand the "flag check". What is that? > > We could make a DMPlexSetCones(), but as you point out, the workflow for > DMSetUp() would have to change. Where does the memory come from for your > connectivity? > > Thanks, > > Matt > > > > Sincerely: > > *J.A. Ferrand* > > Embry-Riddle Aeronautical University - Daytona Beach - FL > Ph.D. Candidate, Aerospace Engineering > > M.Sc. Aerospace Engineering > > B.Sc. Aerospace Engineering > > B.Sc. Computational Mathematics > > > *Phone:* (386)-843-1829 > > *Email(s):* ferranj2 at my.erau.edu > > jesus.ferrand at gmail.com > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!cfVoObgiLc90zIrUaHVMSXqAJtv69Ufe3A1lF0j0zEtULlt1kmKQ7ywA6tGfxdvxMMGDSlHe66KyWihF3ryu$ > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!cfVoObgiLc90zIrUaHVMSXqAJtv69Ufe3A1lF0j0zEtULlt1kmKQ7ywA6tGfxdvxMMGDSlHe66KyWihF3ryu$ > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!cfVoObgiLc90zIrUaHVMSXqAJtv69Ufe3A1lF0j0zEtULlt1kmKQ7ywA6tGfxdvxMMGDSlHe66KyWihF3ryu$ > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!cfVoObgiLc90zIrUaHVMSXqAJtv69Ufe3A1lF0j0zEtULlt1kmKQ7ywA6tGfxdvxMMGDSlHe66KyWihF3ryu$ > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!cfVoObgiLc90zIrUaHVMSXqAJtv69Ufe3A1lF0j0zEtULlt1kmKQ7ywA6tGfxdvxMMGDSlHe66KyWihF3ryu$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From pierre at joliv.et Mon May 6 10:15:35 2024 From: pierre at joliv.et (Pierre Jolivet) Date: Mon, 6 May 2024 17:15:35 +0200 Subject: [petsc-users] PETSc options In-Reply-To: References: <2d56c6ea-e9e0-4a7a-b565-90a1404e08b7@auckland.ac.nz> Message-ID: > On 6 May 2024, at 3:14?PM, Matthew Knepley wrote: > > This Message Is From an External Sender > This message came from outside your organization. > On Mon, May 6, 2024 at 1:04?AM Adrian Croucher > wrote: >> This Message Is From an External Sender >> This message came from outside your organization. >> >> hi, >> >> My code has some optional command line arguments -v and -h for output of >> version number and usage help. These are processed using Fortran's >> get_command_argument(). >> >> Since updating PETSc to version 3.21, I get some extra warnings after >> the output: >> >> acro018 at EN438880:~$ waiwera -v >> 1.5.0b1 >> WARNING! There are options you set that were not used! >> WARNING! could be spelling mistake, etc! >> There is one unused database option. It is: >> Option left: name:-v (no value) source: command line >> >> That didn't used to happen. What should I do to make them go away? > > Hi Adrian, > > Barry and Mark's suggestions will make this go away. However, it should not happen > in the first place. It should happen if Adrian was previously not using 3.19.X or below. See https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/merge_requests/6601__;!!G_uCfscf7eWS!ZTd_vYQ25gBqyomaus8Y3ZvnKjc8AdhWKmEE3UIP1XKXce16F5mXog_qr60phhd9LqIo6_dzVSLpfzONhaDkQA$ Thanks, Pierre > We should try to figure this out. > > This warning is usually activated by the -options_left argument. Could that be in the > PETSC_OPTIONS env variable, or in ~/.petscrc? > > Thanks, > > Matt > >> Regards, Adrian >> >> -- >> Dr Adrian Croucher >> Senior Research Fellow >> Department of Engineering Science >> Waipapa Taumata Rau / University of Auckland, New Zealand >> email: a.croucher at auckland.ac.nz >> tel: +64 (0)9 923 4611 >> > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!ZTd_vYQ25gBqyomaus8Y3ZvnKjc8AdhWKmEE3UIP1XKXce16F5mXog_qr60phhd9LqIo6_dzVSLpfzNkK3KD_g$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon May 6 10:30:10 2024 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 6 May 2024 11:30:10 -0400 Subject: [petsc-users] PETSc options In-Reply-To: References: <2d56c6ea-e9e0-4a7a-b565-90a1404e08b7@auckland.ac.nz> Message-ID: On Mon, May 6, 2024 at 11:15?AM Pierre Jolivet wrote: > On 6 May 2024, at 3:14?PM, Matthew Knepley wrote: > > This Message Is From an External Sender > This message came from outside your organization. > On Mon, May 6, 2024 at 1:04?AM Adrian Croucher > wrote: > >> This Message Is From an External Sender >> This message came from outside your organization. >> >> >> hi, >> >> My code has some optional command line arguments -v and -h for output of >> version number and usage help. These are processed using Fortran's >> get_command_argument(). >> >> Since updating PETSc to version 3.21, I get some extra warnings after >> the output: >> >> acro018 at EN438880:~$ waiwera -v >> 1.5.0b1 >> WARNING! There are options you set that were not used! >> WARNING! could be spelling mistake, etc! >> There is one unused database option. It is: >> Option left: name:-v (no value) source: command line >> >> That didn't used to happen. What should I do to make them go away? >> >> > Hi Adrian, > > Barry and Mark's suggestions will make this go away. However, it should > not happen > in the first place. > > > It should happen if Adrian was previously not using 3.19.X or below. > See https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/merge_requests/6601__;!!G_uCfscf7eWS!bVUciSLxU1Cw3mv04eO2KtlHR8v6ZR7CRE_CpMzmxmCn4x58fUdEJ4L-dknPq_yMbgnc9OkN_O8aOSu_aPK3$ > I forgot. Not my favorite change. Okay. You can shut this off using PetscCall(PetscOptionsSetValue(NULL, "-options_left", "0")) Thanks, Matt > Thanks, > Pierre > > We should try to figure this out. > > This warning is usually activated by the -options_left argument. Could > that be in the > PETSC_OPTIONS env variable, or in ~/.petscrc? > > Thanks, > > Matt > > >> Regards, Adrian >> >> -- >> Dr Adrian Croucher >> Senior Research Fellow >> Department of Engineering Science >> Waipapa Taumata Rau / University of Auckland, New Zealand >> email: a.croucher at auckland.ac.nz >> tel: +64 (0)9 923 4611 >> >> >> > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!bVUciSLxU1Cw3mv04eO2KtlHR8v6ZR7CRE_CpMzmxmCn4x58fUdEJ4L-dknPq_yMbgnc9OkN_O8aOQZGyQDX$ > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!bVUciSLxU1Cw3mv04eO2KtlHR8v6ZR7CRE_CpMzmxmCn4x58fUdEJ4L-dknPq_yMbgnc9OkN_O8aOQZGyQDX$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Mon May 6 12:14:31 2024 From: mfadams at lbl.gov (Mark Adams) Date: Mon, 6 May 2024 13:14:31 -0400 Subject: [petsc-users] PETSc options In-Reply-To: References: <2d56c6ea-e9e0-4a7a-b565-90a1404e08b7@auckland.ac.nz> Message-ID: But that will hardwire disabling -options_left. right? On Mon, May 6, 2024 at 11:30?AM Matthew Knepley wrote: > On Mon, May 6, 2024 at 11: 15 AM Pierre Jolivet > wrote: On 6 May 2024, at 3: 14 PM, Matthew Knepley > wrote: This Message Is From an External Sender This message came from > outside your organization. On > ZjQcmQRYFpfptBannerStart > This Message Is From an External Sender > This message came from outside your organization. > > ZjQcmQRYFpfptBannerEnd > On Mon, May 6, 2024 at 11:15?AM Pierre Jolivet wrote: > >> On 6 May 2024, at 3:14?PM, Matthew Knepley wrote: >> >> This Message Is From an External Sender >> This message came from outside your organization. >> On Mon, May 6, 2024 at 1:04?AM Adrian Croucher >> wrote: >> >>> This Message Is From an External Sender >>> This message came from outside your organization. >>> >>> >>> hi, >>> >>> My code has some optional command line arguments -v and -h for output of >>> version number and usage help. These are processed using Fortran's >>> get_command_argument(). >>> >>> Since updating PETSc to version 3.21, I get some extra warnings after >>> the output: >>> >>> acro018 at EN438880:~$ waiwera -v >>> 1.5.0b1 >>> WARNING! There are options you set that were not used! >>> WARNING! could be spelling mistake, etc! >>> There is one unused database option. It is: >>> Option left: name:-v (no value) source: command line >>> >>> That didn't used to happen. What should I do to make them go away? >>> >>> >> Hi Adrian, >> >> Barry and Mark's suggestions will make this go away. However, it should >> not happen >> in the first place. >> >> >> It should happen if Adrian was previously not using 3.19.X or below. >> See https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/merge_requests/6601__;!!G_uCfscf7eWS!cmPiNWMt3bGuHoTFucwv-1M4V_pfI817gtN7UnXj1iyfa0CKbwbgakWu87qbrcmtcNirNSVXQIg6jFSKSTzez_M$ >> >> > > I forgot. Not my favorite change. > > Okay. You can shut this off using > > PetscCall(PetscOptionsSetValue(NULL, "-options_left", "0")) > > Thanks, > > Matt > > >> Thanks, >> Pierre >> >> We should try to figure this out. >> >> This warning is usually activated by the -options_left argument. Could >> that be in the >> PETSC_OPTIONS env variable, or in ~/.petscrc? >> >> Thanks, >> >> Matt >> >> >>> Regards, Adrian >>> >>> -- >>> Dr Adrian Croucher >>> Senior Research Fellow >>> Department of Engineering Science >>> Waipapa Taumata Rau / University of Auckland, New Zealand >>> email: a.croucher at auckland.ac.nz >>> tel: +64 (0)9 923 4611 >>> >>> >>> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!cmPiNWMt3bGuHoTFucwv-1M4V_pfI817gtN7UnXj1iyfa0CKbwbgakWu87qbrcmtcNirNSVXQIg6jFSKdehPgHA$ >> >> >> >> > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!cmPiNWMt3bGuHoTFucwv-1M4V_pfI817gtN7UnXj1iyfa0CKbwbgakWu87qbrcmtcNirNSVXQIg6jFSKdehPgHA$ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon May 6 13:14:27 2024 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 6 May 2024 14:14:27 -0400 Subject: [petsc-users] PETSc options In-Reply-To: References: <2d56c6ea-e9e0-4a7a-b565-90a1404e08b7@auckland.ac.nz> Message-ID: On Mon, May 6, 2024 at 1:14?PM Mark Adams wrote: > But that will hardwire disabling -options_left. right? > True. Your suggestion would only filter out those two in particular. Thanks, Matt > On Mon, May 6, 2024 at 11:30?AM Matthew Knepley wrote: > >> On Mon, May 6, 2024 at 11: 15 AM Pierre Jolivet >> wrote: On 6 May 2024, at 3: 14 PM, Matthew Knepley >> wrote: This Message Is From an External Sender This message came from >> outside your organization. On >> ZjQcmQRYFpfptBannerStart >> This Message Is From an External Sender >> This message came from outside your organization. >> >> ZjQcmQRYFpfptBannerEnd >> On Mon, May 6, 2024 at 11:15?AM Pierre Jolivet wrote: >> >>> On 6 May 2024, at 3:14?PM, Matthew Knepley wrote: >>> >>> This Message Is From an External Sender >>> This message came from outside your organization. >>> On Mon, May 6, 2024 at 1:04?AM Adrian Croucher < >>> a.croucher at auckland.ac.nz> wrote: >>> >>>> This Message Is From an External Sender >>>> This message came from outside your organization. >>>> >>>> >>>> hi, >>>> >>>> My code has some optional command line arguments -v and -h for output of >>>> version number and usage help. These are processed using Fortran's >>>> get_command_argument(). >>>> >>>> Since updating PETSc to version 3.21, I get some extra warnings after >>>> the output: >>>> >>>> acro018 at EN438880:~$ waiwera -v >>>> 1.5.0b1 >>>> WARNING! There are options you set that were not used! >>>> WARNING! could be spelling mistake, etc! >>>> There is one unused database option. It is: >>>> Option left: name:-v (no value) source: command line >>>> >>>> That didn't used to happen. What should I do to make them go away? >>>> >>>> >>> Hi Adrian, >>> >>> Barry and Mark's suggestions will make this go away. However, it should >>> not happen >>> in the first place. >>> >>> >>> It should happen if Adrian was previously not using 3.19.X or below. >>> See https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/merge_requests/6601__;!!G_uCfscf7eWS!aNM60JUO9Vbf5saWU1g8lA5UztQH35oMhYJcuYQfRU3hKT9s_58lqzRoWbtXuf0G1azqCD_9R9cWfJxmWrGE$ >>> >>> >> >> I forgot. Not my favorite change. >> >> Okay. You can shut this off using >> >> PetscCall(PetscOptionsSetValue(NULL, "-options_left", "0")) >> >> Thanks, >> >> Matt >> >> >>> Thanks, >>> Pierre >>> >>> We should try to figure this out. >>> >>> This warning is usually activated by the -options_left argument. Could >>> that be in the >>> PETSC_OPTIONS env variable, or in ~/.petscrc? >>> >>> Thanks, >>> >>> Matt >>> >>> >>>> Regards, Adrian >>>> >>>> -- >>>> Dr Adrian Croucher >>>> Senior Research Fellow >>>> Department of Engineering Science >>>> Waipapa Taumata Rau / University of Auckland, New Zealand >>>> email: a.croucher at auckland.ac.nz >>>> tel: +64 (0)9 923 4611 >>>> >>>> >>>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!aNM60JUO9Vbf5saWU1g8lA5UztQH35oMhYJcuYQfRU3hKT9s_58lqzRoWbtXuf0G1azqCD_9R9cWfFIGaM_Q$ >>> >>> >>> >>> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!aNM60JUO9Vbf5saWU1g8lA5UztQH35oMhYJcuYQfRU3hKT9s_58lqzRoWbtXuf0G1azqCD_9R9cWfFIGaM_Q$ >> >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!aNM60JUO9Vbf5saWU1g8lA5UztQH35oMhYJcuYQfRU3hKT9s_58lqzRoWbtXuf0G1azqCD_9R9cWfFIGaM_Q$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From FERRANJ2 at my.erau.edu Mon May 6 15:33:11 2024 From: FERRANJ2 at my.erau.edu (Ferrand, Jesus A.) Date: Mon, 6 May 2024 20:33:11 +0000 Subject: [petsc-users] [EXTERNAL] Re: Is there anything like a "DMPlexSetCones()" ? In-Reply-To: References: Message-ID: "What is the argument for this optimization?" ??????Ah! ??????That's all I had really. ??????That you would skip this one malloc step but without regard to the grander picture. ??????I had a gut feeling that it is a sizeable double malloc because it is the cell connectivity. ??????However, once you bring into perspective all the other stuff (CellOrientation, Vecs, etc.) it does seem like a micro-optimization. ??????If in your estimation this isn't worth the development, then I'll continue to follow the guidance from the PETSc source code. ________________________________ From: Matthew Knepley Sent: Monday, May 6, 2024 10:55 AM To: Ferrand, Jesus A. Cc: petsc-users at mcs.anl.gov Subject: Re: [EXTERNAL] Re: [petsc-users] Is there anything like a "DMPlexSetCones()" ? On Mon, May 6, 2024 at 10:17?AM Ferrand, Jesus A. > wrote: "The cone sizes are used to allocate the array to hold the cones. How would you have an array holding the cone information without knowing the sizes?" ??????You're right, I must first know the sizes (which I determine from the file). ??????It lists the number of cells and their types by sequential chunks. ??????The issue I am trying to describe, is that because I cannot know the number of local vertices a priori, the pipeline DMSetChart()->DMPlexSetConeSize()->DMSetUp() essentially produces a redundant array (because I already have the connectivity that I read from the file). ??????As far as I can tell, in my case, the end result of calling these APIs is already available, hence my wish for a "DMPlexSetCones()". ??????If such an API existed, I would then attempt to retroactively set the Chart and the ConeSizes. ??????Like you hinted in a reply, for this extra array to be worthwhile, I would have to keep it as a global numbering list (which I want to by the way). ??????I was hoping to at least keep them in the DM's global PetscSection as offsets. ??????Unfortunately, PetscSection has a similar workflow to DM: PetscSectionSetChart(), PetscSectionAddDof(), PetscSectionSetUp(), PetscSectionSetOffset(). I looked at the other parallel code. It does maintain two arrays, so that you can compute this stuff. I am reluctant to add this interface, since I think the optimization is very small and it is not a good pattern. You will save allocating the cell connectivity. However, if you interpolate the mesh, this is small. This is the same size as the cell orientations, which we will also have to allocate. We are likely to allocate vectors over the mesh. It does not seem likely that this allocation will increase the peak allocation, and it will be freed after creation, so total size will not be affected. What is the argument for this optimization? Thanks, Matt ________________________________ From: Matthew Knepley > Sent: Monday, May 6, 2024 7:52 AM To: Ferrand, Jesus A. > Cc: petsc-users at mcs.anl.gov > Subject: Re: [EXTERNAL] Re: [petsc-users] Is there anything like a "DMPlexSetCones()" ? On Fri, May 3, 2024 at 7:32?PM Ferrand, Jesus A. > wrote: "Right, but that means that the connectivity you read in is not in local numbering. You will have to renumber it, so making a copy is probably necessary anyway. This is the same sort of processing I do for the parallel read." ??????True, the connectivity is read in global numbering. ??????Renumbering is facilitated by the table from PetscHashSetI. ??????The connectivity list I am able to renumber in place though, I don't see the need to have another interim copy of the numbers. ??????My concern is that I have two memory chunks that both contain the same information. ??????The only reason It has to be this way is because I can't manually set the Cones first and the Chart/ConeSizes second. The cone sizes are used to allocate the array to hold the cones. How would you have an array holding the cone information without knowing the sizes? Thanks, Matt ________________________________ From: Matthew Knepley > Sent: Friday, May 3, 2024 6:45 PM To: Ferrand, Jesus A. > Cc: petsc-users at mcs.anl.gov > Subject: Re: [EXTERNAL] Re: [petsc-users] Is there anything like a "DMPlexSetCones()" ? On Wed, May 1, 2024 at 6:30?PM Ferrand, Jesus A. > wrote: "You know the number of cells if you can read connectivity. Do you mean that the format does not tell you the number of vertices?" Sort of... So, the format does provide the number of vertices, however, due to the way I read the data in parallel I don't know immediately how many local vertices there will be. Which prevents me from knowing the chart a priori. I figured out the number of vertices using PetscHashSetI to determine the number of unique entries in the connectivity list. Right, but that means that the connectivity you read in is not in local numbering. You will have to renumber it, so making a copy is probably necessary anyway. This is the same sort of processing I do for the parallel read. Thanks, Matt ________________________________ From: Matthew Knepley > Sent: Wednesday, May 1, 2024 8:52 PM To: Ferrand, Jesus A. > Cc: petsc-users at mcs.anl.gov > Subject: Re: [EXTERNAL] Re: [petsc-users] Is there anything like a "DMPlexSetCones()" ? On Wed, May 1, 2024 at 4:23?PM Ferrand, Jesus A. > wrote: Matt: My bad again, I need to clarify something that I just realized doesn't make sense. I said: "The nature of the I/O makes it so that I need to read the connectivity before I get a semblance of buffer sizes." Scratch that, obviously, if I can read the connectivity, I know the buffer sizes. What I should have said is that I cannot know the "chart" that one sets in DMPlexSetChart() a priori like in most cases. I need to determine the chart from the connectivity lists. You know the number of cells if you can read connectivity. Do you mean that the format does not tell you the number of vertices? Thanks, Matt ________________________________ From: Ferrand, Jesus A. > Sent: Wednesday, May 1, 2024 8:17 PM To: Matthew Knepley > Cc: petsc-users at mcs.anl.gov > Subject: Re: [EXTERNAL] Re: [petsc-users] Is there anything like a "DMPlexSetCones()" ? Matt: "I do not understand the "flag check". What is that?" ??????My bad, I should have referred to the "dm->setupcalled". ??????I believe this PetscBool is checked by the other DM (not just DMPlex) APIs. ??????The subsequent checks for dm->setupcalled == PETSC_TRUE is what I meant to say. ??????Here's a copy of DMSetUp(). PetscErrorCode DMSetUp(DM dm) 817: { 818: PetscFunctionBegin; 820: if (dm->setupcalled) PetscFunctionReturn(PETSC_SUCCESS); 821: PetscTryTypeMethod(dm, setup); 822: dm->setupcalled = PETSC_TRUE; 823: PetscFunctionReturn(PETSC_SUCCESS); 824: } "We could make a DMPlexSetCones(), but as you point out, the workflow for DMSetUp() would have to change. Where does the memory come from for your connectivity?" ??????It's a memory that I myself allocate based on a file's contents. ??????The nature of the I/O makes it so that I need to read the connectivity before I get a semblance of buffer sizes. ??????Otherwise, I would stick to the tried and tested way. Also, when replying to the PETSc developers, users must reply to petsc-users at mcs.anl.gov and not just to the individual email accounts of the developers, right? ________________________________ From: Matthew Knepley > Sent: Wednesday, May 1, 2024 8:07 PM To: Ferrand, Jesus A. > Cc: petsc-users at mcs.anl.gov > Subject: [EXTERNAL] Re: [petsc-users] Is there anything like a "DMPlexSetCones()" ? CAUTION: This email originated outside of Embry-Riddle Aeronautical University. Do not click links or open attachments unless you recognize the sender and know the content is safe. On Wed, May 1, 2024 at 3:34?PM Ferrand, Jesus A. > wrote: This Message Is From an External Sender This message came from outside your organization. Dear PETSc team: For a project that I'm working on, I need to manually build a DMPlex. >From studying the source code of the various APIs in which the plex is built from some supported file format, I get that the workflow is this: 1. DMPlexSetChart() <-- Input nCells + nVerts 2. DMPlexSetConeSize() <-- Input ConeSize for each point in [0,nCells) 3. DMSetUp() ? Allocates memory internally. 4. DMPlexGetCones() --> Gives you the memory onto which to write the cell connectivity. 5. *Write connectivity* 6. DMPlexReorderCell() <-- For each point in [0,nCells) I'm in a situation where the memory given by step (4) is available a-priori. I was hoping to skip steps 2, 3 , and 4 with something like a "DMPlexSetCones()", but such an API does not exist. My current workaround is to implement steps 2 through 4 as always and have double the memory allocated in the interim (my instance + DM's internal instance). I was thinking of looking for the name of the struct member and = it to my memory, but I can't overcome the flag check in DMSetUp() during later calls to DMPlexGetCones() or DMPlexGetTransitiveClosure(). I do not understand the "flag check". What is that? We could make a DMPlexSetCones(), but as you point out, the workflow for DMSetUp() would have to change. Where does the memory come from for your connectivity? Thanks, Matt Sincerely: J.A. Ferrand Embry-Riddle Aeronautical University - Daytona Beach - FL Ph.D. Candidate, Aerospace Engineering M.Sc. Aerospace Engineering B.Sc. Aerospace Engineering B.Sc. Computational Mathematics Phone: (386)-843-1829 Email(s): ferranj2 at my.erau.edu jesus.ferrand at gmail.com -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!eJZaDdTGDKv1PKAoyrj5dR-1ZLRLekHn46Kq1QN8YsQTCX5IaaoSZ2ownEaefDvc2AOhkc8pqTms9d9lCHYRtccNFDE$ ________________________________ From: Matthew Knepley > Sent: Wednesday, May 1, 2024 8:07 PM To: Ferrand, Jesus A. > Cc: petsc-users at mcs.anl.gov > Subject: [EXTERNAL] Re: [petsc-users] Is there anything like a "DMPlexSetCones()" ? CAUTION: This email originated outside of Embry-Riddle Aeronautical University. Do not click links or open attachments unless you recognize the sender and know the content is safe. On Wed, May 1, 2024 at 3:34?PM Ferrand, Jesus A. > wrote: This Message Is From an External Sender This message came from outside your organization. Dear PETSc team: For a project that I'm working on, I need to manually build a DMPlex. >From studying the source code of the various APIs in which the plex is built from some supported file format, I get that the workflow is this: 1. DMPlexSetChart() <-- Input nCells + nVerts 2. DMPlexSetConeSize() <-- Input ConeSize for each point in [0,nCells) 3. DMSetUp() ? Allocates memory internally. 4. DMPlexGetCones() --> Gives you the memory onto which to write the cell connectivity. 5. *Write connectivity* 6. DMPlexReorderCell() <-- For each point in [0,nCells) I'm in a situation where the memory given by step (4) is available a-priori. I was hoping to skip steps 2, 3 , and 4 with something like a "DMPlexSetCones()", but such an API does not exist. My current workaround is to implement steps 2 through 4 as always and have double the memory allocated in the interim (my instance + DM's internal instance). I was thinking of looking for the name of the struct member and = it to my memory, but I can't overcome the flag check in DMSetUp() during later calls to DMPlexGetCones() or DMPlexGetTransitiveClosure(). I do not understand the "flag check". What is that? We could make a DMPlexSetCones(), but as you point out, the workflow for DMSetUp() would have to change. Where does the memory come from for your connectivity? Thanks, Matt Sincerely: J.A. Ferrand Embry-Riddle Aeronautical University - Daytona Beach - FL Ph.D. Candidate, Aerospace Engineering M.Sc. Aerospace Engineering B.Sc. Aerospace Engineering B.Sc. Computational Mathematics Phone: (386)-843-1829 Email(s): ferranj2 at my.erau.edu jesus.ferrand at gmail.com -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!eJZaDdTGDKv1PKAoyrj5dR-1ZLRLekHn46Kq1QN8YsQTCX5IaaoSZ2ownEaefDvc2AOhkc8pqTms9d9lCHYRtccNFDE$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!eJZaDdTGDKv1PKAoyrj5dR-1ZLRLekHn46Kq1QN8YsQTCX5IaaoSZ2ownEaefDvc2AOhkc8pqTms9d9lCHYRtccNFDE$ ________________________________ From: Matthew Knepley > Sent: Wednesday, May 1, 2024 8:52 PM To: Ferrand, Jesus A. > Cc: petsc-users at mcs.anl.gov > Subject: Re: [EXTERNAL] Re: [petsc-users] Is there anything like a "DMPlexSetCones()" ? On Wed, May 1, 2024 at 4:23?PM Ferrand, Jesus A. > wrote: Matt: My bad again, I need to clarify something that I just realized doesn't make sense. I said: "The nature of the I/O makes it so that I need to read the connectivity before I get a semblance of buffer sizes." Scratch that, obviously, if I can read the connectivity, I know the buffer sizes. What I should have said is that I cannot know the "chart" that one sets in DMPlexSetChart() a priori like in most cases. I need to determine the chart from the connectivity lists. You know the number of cells if you can read connectivity. Do you mean that the format does not tell you the number of vertices? Thanks, Matt ________________________________ From: Ferrand, Jesus A. > Sent: Wednesday, May 1, 2024 8:17 PM To: Matthew Knepley > Cc: petsc-users at mcs.anl.gov > Subject: Re: [EXTERNAL] Re: [petsc-users] Is there anything like a "DMPlexSetCones()" ? Matt: "I do not understand the "flag check". What is that?" ??????My bad, I should have referred to the "dm->setupcalled". ??????I believe this PetscBool is checked by the other DM (not just DMPlex) APIs. ??????The subsequent checks for dm->setupcalled == PETSC_TRUE is what I meant to say. ??????Here's a copy of DMSetUp(). PetscErrorCode DMSetUp(DM dm) 817: { 818: PetscFunctionBegin; 820: if (dm->setupcalled) PetscFunctionReturn(PETSC_SUCCESS); 821: PetscTryTypeMethod(dm, setup); 822: dm->setupcalled = PETSC_TRUE; 823: PetscFunctionReturn(PETSC_SUCCESS); 824: } "We could make a DMPlexSetCones(), but as you point out, the workflow for DMSetUp() would have to change. Where does the memory come from for your connectivity?" ??????It's a memory that I myself allocate based on a file's contents. ??????The nature of the I/O makes it so that I need to read the connectivity before I get a semblance of buffer sizes. ??????Otherwise, I would stick to the tried and tested way. Also, when replying to the PETSc developers, users must reply to petsc-users at mcs.anl.gov and not just to the individual email accounts of the developers, right? ________________________________ From: Matthew Knepley > Sent: Wednesday, May 1, 2024 8:07 PM To: Ferrand, Jesus A. > Cc: petsc-users at mcs.anl.gov > Subject: [EXTERNAL] Re: [petsc-users] Is there anything like a "DMPlexSetCones()" ? CAUTION: This email originated outside of Embry-Riddle Aeronautical University. Do not click links or open attachments unless you recognize the sender and know the content is safe. On Wed, May 1, 2024 at 3:34?PM Ferrand, Jesus A. > wrote: This Message Is From an External Sender This message came from outside your organization. Dear PETSc team: For a project that I'm working on, I need to manually build a DMPlex. >From studying the source code of the various APIs in which the plex is built from some supported file format, I get that the workflow is this: 1. DMPlexSetChart() <-- Input nCells + nVerts 2. DMPlexSetConeSize() <-- Input ConeSize for each point in [0,nCells) 3. DMSetUp() ? Allocates memory internally. 4. DMPlexGetCones() --> Gives you the memory onto which to write the cell connectivity. 5. *Write connectivity* 6. DMPlexReorderCell() <-- For each point in [0,nCells) I'm in a situation where the memory given by step (4) is available a-priori. I was hoping to skip steps 2, 3 , and 4 with something like a "DMPlexSetCones()", but such an API does not exist. My current workaround is to implement steps 2 through 4 as always and have double the memory allocated in the interim (my instance + DM's internal instance). I was thinking of looking for the name of the struct member and = it to my memory, but I can't overcome the flag check in DMSetUp() during later calls to DMPlexGetCones() or DMPlexGetTransitiveClosure(). I do not understand the "flag check". What is that? We could make a DMPlexSetCones(), but as you point out, the workflow for DMSetUp() would have to change. Where does the memory come from for your connectivity? Thanks, Matt Sincerely: J.A. Ferrand Embry-Riddle Aeronautical University - Daytona Beach - FL Ph.D. Candidate, Aerospace Engineering M.Sc. Aerospace Engineering B.Sc. Aerospace Engineering B.Sc. Computational Mathematics Phone: (386)-843-1829 Email(s): ferranj2 at my.erau.edu jesus.ferrand at gmail.com -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!eJZaDdTGDKv1PKAoyrj5dR-1ZLRLekHn46Kq1QN8YsQTCX5IaaoSZ2ownEaefDvc2AOhkc8pqTms9d9lCHYRtccNFDE$ ________________________________ From: Matthew Knepley > Sent: Wednesday, May 1, 2024 8:07 PM To: Ferrand, Jesus A. > Cc: petsc-users at mcs.anl.gov > Subject: [EXTERNAL] Re: [petsc-users] Is there anything like a "DMPlexSetCones()" ? CAUTION: This email originated outside of Embry-Riddle Aeronautical University. Do not click links or open attachments unless you recognize the sender and know the content is safe. On Wed, May 1, 2024 at 3:34?PM Ferrand, Jesus A. > wrote: This Message Is From an External Sender This message came from outside your organization. Dear PETSc team: For a project that I'm working on, I need to manually build a DMPlex. >From studying the source code of the various APIs in which the plex is built from some supported file format, I get that the workflow is this: 1. DMPlexSetChart() <-- Input nCells + nVerts 2. DMPlexSetConeSize() <-- Input ConeSize for each point in [0,nCells) 3. DMSetUp() ? Allocates memory internally. 4. DMPlexGetCones() --> Gives you the memory onto which to write the cell connectivity. 5. *Write connectivity* 6. DMPlexReorderCell() <-- For each point in [0,nCells) I'm in a situation where the memory given by step (4) is available a-priori. I was hoping to skip steps 2, 3 , and 4 with something like a "DMPlexSetCones()", but such an API does not exist. My current workaround is to implement steps 2 through 4 as always and have double the memory allocated in the interim (my instance + DM's internal instance). I was thinking of looking for the name of the struct member and = it to my memory, but I can't overcome the flag check in DMSetUp() during later calls to DMPlexGetCones() or DMPlexGetTransitiveClosure(). I do not understand the "flag check". What is that? We could make a DMPlexSetCones(), but as you point out, the workflow for DMSetUp() would have to change. Where does the memory come from for your connectivity? Thanks, Matt Sincerely: J.A. Ferrand Embry-Riddle Aeronautical University - Daytona Beach - FL Ph.D. Candidate, Aerospace Engineering M.Sc. Aerospace Engineering B.Sc. Aerospace Engineering B.Sc. Computational Mathematics Phone: (386)-843-1829 Email(s): ferranj2 at my.erau.edu jesus.ferrand at gmail.com -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!eJZaDdTGDKv1PKAoyrj5dR-1ZLRLekHn46Kq1QN8YsQTCX5IaaoSZ2ownEaefDvc2AOhkc8pqTms9d9lCHYRtccNFDE$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!eJZaDdTGDKv1PKAoyrj5dR-1ZLRLekHn46Kq1QN8YsQTCX5IaaoSZ2ownEaefDvc2AOhkc8pqTms9d9lCHYRtccNFDE$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!eJZaDdTGDKv1PKAoyrj5dR-1ZLRLekHn46Kq1QN8YsQTCX5IaaoSZ2ownEaefDvc2AOhkc8pqTms9d9lCHYRtccNFDE$ ________________________________ From: Matthew Knepley > Sent: Friday, May 3, 2024 6:45 PM To: Ferrand, Jesus A. > Cc: petsc-users at mcs.anl.gov > Subject: Re: [EXTERNAL] Re: [petsc-users] Is there anything like a "DMPlexSetCones()" ? On Wed, May 1, 2024 at 6:30?PM Ferrand, Jesus A. > wrote: "You know the number of cells if you can read connectivity. Do you mean that the format does not tell you the number of vertices?" Sort of... So, the format does provide the number of vertices, however, due to the way I read the data in parallel I don't know immediately how many local vertices there will be. Which prevents me from knowing the chart a priori. I figured out the number of vertices using PetscHashSetI to determine the number of unique entries in the connectivity list. Right, but that means that the connectivity you read in is not in local numbering. You will have to renumber it, so making a copy is probably necessary anyway. This is the same sort of processing I do for the parallel read. Thanks, Matt ________________________________ From: Matthew Knepley > Sent: Wednesday, May 1, 2024 8:52 PM To: Ferrand, Jesus A. > Cc: petsc-users at mcs.anl.gov > Subject: Re: [EXTERNAL] Re: [petsc-users] Is there anything like a "DMPlexSetCones()" ? On Wed, May 1, 2024 at 4:23?PM Ferrand, Jesus A. > wrote: Matt: My bad again, I need to clarify something that I just realized doesn't make sense. I said: "The nature of the I/O makes it so that I need to read the connectivity before I get a semblance of buffer sizes." Scratch that, obviously, if I can read the connectivity, I know the buffer sizes. What I should have said is that I cannot know the "chart" that one sets in DMPlexSetChart() a priori like in most cases. I need to determine the chart from the connectivity lists. You know the number of cells if you can read connectivity. Do you mean that the format does not tell you the number of vertices? Thanks, Matt ________________________________ From: Ferrand, Jesus A. > Sent: Wednesday, May 1, 2024 8:17 PM To: Matthew Knepley > Cc: petsc-users at mcs.anl.gov > Subject: Re: [EXTERNAL] Re: [petsc-users] Is there anything like a "DMPlexSetCones()" ? Matt: "I do not understand the "flag check". What is that?" ??????My bad, I should have referred to the "dm->setupcalled". ??????I believe this PetscBool is checked by the other DM (not just DMPlex) APIs. ??????The subsequent checks for dm->setupcalled == PETSC_TRUE is what I meant to say. ??????Here's a copy of DMSetUp(). PetscErrorCode DMSetUp(DM dm) 817: { 818: PetscFunctionBegin; 820: if (dm->setupcalled) PetscFunctionReturn(PETSC_SUCCESS); 821: PetscTryTypeMethod(dm, setup); 822: dm->setupcalled = PETSC_TRUE; 823: PetscFunctionReturn(PETSC_SUCCESS); 824: } "We could make a DMPlexSetCones(), but as you point out, the workflow for DMSetUp() would have to change. Where does the memory come from for your connectivity?" ??????It's a memory that I myself allocate based on a file's contents. ??????The nature of the I/O makes it so that I need to read the connectivity before I get a semblance of buffer sizes. ??????Otherwise, I would stick to the tried and tested way. Also, when replying to the PETSc developers, users must reply to petsc-users at mcs.anl.gov and not just to the individual email accounts of the developers, right? ________________________________ From: Matthew Knepley > Sent: Wednesday, May 1, 2024 8:07 PM To: Ferrand, Jesus A. > Cc: petsc-users at mcs.anl.gov > Subject: [EXTERNAL] Re: [petsc-users] Is there anything like a "DMPlexSetCones()" ? CAUTION: This email originated outside of Embry-Riddle Aeronautical University. Do not click links or open attachments unless you recognize the sender and know the content is safe. On Wed, May 1, 2024 at 3:34?PM Ferrand, Jesus A. > wrote: This Message Is From an External Sender This message came from outside your organization. Dear PETSc team: For a project that I'm working on, I need to manually build a DMPlex. >From studying the source code of the various APIs in which the plex is built from some supported file format, I get that the workflow is this: 1. DMPlexSetChart() <-- Input nCells + nVerts 2. DMPlexSetConeSize() <-- Input ConeSize for each point in [0,nCells) 3. DMSetUp() ? Allocates memory internally. 4. DMPlexGetCones() --> Gives you the memory onto which to write the cell connectivity. 5. *Write connectivity* 6. DMPlexReorderCell() <-- For each point in [0,nCells) I'm in a situation where the memory given by step (4) is available a-priori. I was hoping to skip steps 2, 3 , and 4 with something like a "DMPlexSetCones()", but such an API does not exist. My current workaround is to implement steps 2 through 4 as always and have double the memory allocated in the interim (my instance + DM's internal instance). I was thinking of looking for the name of the struct member and = it to my memory, but I can't overcome the flag check in DMSetUp() during later calls to DMPlexGetCones() or DMPlexGetTransitiveClosure(). I do not understand the "flag check". What is that? We could make a DMPlexSetCones(), but as you point out, the workflow for DMSetUp() would have to change. Where does the memory come from for your connectivity? Thanks, Matt Sincerely: J.A. Ferrand Embry-Riddle Aeronautical University - Daytona Beach - FL Ph.D. Candidate, Aerospace Engineering M.Sc. Aerospace Engineering B.Sc. Aerospace Engineering B.Sc. Computational Mathematics Phone: (386)-843-1829 Email(s): ferranj2 at my.erau.edu jesus.ferrand at gmail.com -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!eJZaDdTGDKv1PKAoyrj5dR-1ZLRLekHn46Kq1QN8YsQTCX5IaaoSZ2ownEaefDvc2AOhkc8pqTms9d9lCHYRtccNFDE$ ________________________________ From: Matthew Knepley > Sent: Wednesday, May 1, 2024 8:07 PM To: Ferrand, Jesus A. > Cc: petsc-users at mcs.anl.gov > Subject: [EXTERNAL] Re: [petsc-users] Is there anything like a "DMPlexSetCones()" ? CAUTION: This email originated outside of Embry-Riddle Aeronautical University. Do not click links or open attachments unless you recognize the sender and know the content is safe. On Wed, May 1, 2024 at 3:34?PM Ferrand, Jesus A. > wrote: This Message Is From an External Sender This message came from outside your organization. Dear PETSc team: For a project that I'm working on, I need to manually build a DMPlex. >From studying the source code of the various APIs in which the plex is built from some supported file format, I get that the workflow is this: 1. DMPlexSetChart() <-- Input nCells + nVerts 2. DMPlexSetConeSize() <-- Input ConeSize for each point in [0,nCells) 3. DMSetUp() ? Allocates memory internally. 4. DMPlexGetCones() --> Gives you the memory onto which to write the cell connectivity. 5. *Write connectivity* 6. DMPlexReorderCell() <-- For each point in [0,nCells) I'm in a situation where the memory given by step (4) is available a-priori. I was hoping to skip steps 2, 3 , and 4 with something like a "DMPlexSetCones()", but such an API does not exist. My current workaround is to implement steps 2 through 4 as always and have double the memory allocated in the interim (my instance + DM's internal instance). I was thinking of looking for the name of the struct member and = it to my memory, but I can't overcome the flag check in DMSetUp() during later calls to DMPlexGetCones() or DMPlexGetTransitiveClosure(). I do not understand the "flag check". What is that? We could make a DMPlexSetCones(), but as you point out, the workflow for DMSetUp() would have to change. Where does the memory come from for your connectivity? Thanks, Matt Sincerely: J.A. Ferrand Embry-Riddle Aeronautical University - Daytona Beach - FL Ph.D. Candidate, Aerospace Engineering M.Sc. Aerospace Engineering B.Sc. Aerospace Engineering B.Sc. Computational Mathematics Phone: (386)-843-1829 Email(s): ferranj2 at my.erau.edu jesus.ferrand at gmail.com -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!eJZaDdTGDKv1PKAoyrj5dR-1ZLRLekHn46Kq1QN8YsQTCX5IaaoSZ2ownEaefDvc2AOhkc8pqTms9d9lCHYRtccNFDE$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!eJZaDdTGDKv1PKAoyrj5dR-1ZLRLekHn46Kq1QN8YsQTCX5IaaoSZ2ownEaefDvc2AOhkc8pqTms9d9lCHYRtccNFDE$ ________________________________ From: Matthew Knepley > Sent: Wednesday, May 1, 2024 8:52 PM To: Ferrand, Jesus A. > Cc: petsc-users at mcs.anl.gov > Subject: Re: [EXTERNAL] Re: [petsc-users] Is there anything like a "DMPlexSetCones()" ? On Wed, May 1, 2024 at 4:23?PM Ferrand, Jesus A. > wrote: Matt: My bad again, I need to clarify something that I just realized doesn't make sense. I said: "The nature of the I/O makes it so that I need to read the connectivity before I get a semblance of buffer sizes." Scratch that, obviously, if I can read the connectivity, I know the buffer sizes. What I should have said is that I cannot know the "chart" that one sets in DMPlexSetChart() a priori like in most cases. I need to determine the chart from the connectivity lists. You know the number of cells if you can read connectivity. Do you mean that the format does not tell you the number of vertices? Thanks, Matt ________________________________ From: Ferrand, Jesus A. > Sent: Wednesday, May 1, 2024 8:17 PM To: Matthew Knepley > Cc: petsc-users at mcs.anl.gov > Subject: Re: [EXTERNAL] Re: [petsc-users] Is there anything like a "DMPlexSetCones()" ? Matt: "I do not understand the "flag check". What is that?" ??????My bad, I should have referred to the "dm->setupcalled". ??????I believe this PetscBool is checked by the other DM (not just DMPlex) APIs. ??????The subsequent checks for dm->setupcalled == PETSC_TRUE is what I meant to say. ??????Here's a copy of DMSetUp(). PetscErrorCode DMSetUp(DM dm) 817: { 818: PetscFunctionBegin; 820: if (dm->setupcalled) PetscFunctionReturn(PETSC_SUCCESS); 821: PetscTryTypeMethod(dm, setup); 822: dm->setupcalled = PETSC_TRUE; 823: PetscFunctionReturn(PETSC_SUCCESS); 824: } "We could make a DMPlexSetCones(), but as you point out, the workflow for DMSetUp() would have to change. Where does the memory come from for your connectivity?" ??????It's a memory that I myself allocate based on a file's contents. ??????The nature of the I/O makes it so that I need to read the connectivity before I get a semblance of buffer sizes. ??????Otherwise, I would stick to the tried and tested way. Also, when replying to the PETSc developers, users must reply to petsc-users at mcs.anl.gov and not just to the individual email accounts of the developers, right? ________________________________ From: Matthew Knepley > Sent: Wednesday, May 1, 2024 8:07 PM To: Ferrand, Jesus A. > Cc: petsc-users at mcs.anl.gov > Subject: [EXTERNAL] Re: [petsc-users] Is there anything like a "DMPlexSetCones()" ? CAUTION: This email originated outside of Embry-Riddle Aeronautical University. Do not click links or open attachments unless you recognize the sender and know the content is safe. On Wed, May 1, 2024 at 3:34?PM Ferrand, Jesus A. > wrote: This Message Is From an External Sender This message came from outside your organization. Dear PETSc team: For a project that I'm working on, I need to manually build a DMPlex. >From studying the source code of the various APIs in which the plex is built from some supported file format, I get that the workflow is this: 1. DMPlexSetChart() <-- Input nCells + nVerts 2. DMPlexSetConeSize() <-- Input ConeSize for each point in [0,nCells) 3. DMSetUp() ? Allocates memory internally. 4. DMPlexGetCones() --> Gives you the memory onto which to write the cell connectivity. 5. *Write connectivity* 6. DMPlexReorderCell() <-- For each point in [0,nCells) I'm in a situation where the memory given by step (4) is available a-priori. I was hoping to skip steps 2, 3 , and 4 with something like a "DMPlexSetCones()", but such an API does not exist. My current workaround is to implement steps 2 through 4 as always and have double the memory allocated in the interim (my instance + DM's internal instance). I was thinking of looking for the name of the struct member and = it to my memory, but I can't overcome the flag check in DMSetUp() during later calls to DMPlexGetCones() or DMPlexGetTransitiveClosure(). I do not understand the "flag check". What is that? We could make a DMPlexSetCones(), but as you point out, the workflow for DMSetUp() would have to change. Where does the memory come from for your connectivity? Thanks, Matt Sincerely: J.A. Ferrand Embry-Riddle Aeronautical University - Daytona Beach - FL Ph.D. Candidate, Aerospace Engineering M.Sc. Aerospace Engineering B.Sc. Aerospace Engineering B.Sc. Computational Mathematics Phone: (386)-843-1829 Email(s): ferranj2 at my.erau.edu jesus.ferrand at gmail.com -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!eJZaDdTGDKv1PKAoyrj5dR-1ZLRLekHn46Kq1QN8YsQTCX5IaaoSZ2ownEaefDvc2AOhkc8pqTms9d9lCHYRtccNFDE$ ________________________________ From: Matthew Knepley > Sent: Wednesday, May 1, 2024 8:07 PM To: Ferrand, Jesus A. > Cc: petsc-users at mcs.anl.gov > Subject: [EXTERNAL] Re: [petsc-users] Is there anything like a "DMPlexSetCones()" ? CAUTION: This email originated outside of Embry-Riddle Aeronautical University. Do not click links or open attachments unless you recognize the sender and know the content is safe. On Wed, May 1, 2024 at 3:34?PM Ferrand, Jesus A. > wrote: This Message Is From an External Sender This message came from outside your organization. Dear PETSc team: For a project that I'm working on, I need to manually build a DMPlex. >From studying the source code of the various APIs in which the plex is built from some supported file format, I get that the workflow is this: 1. DMPlexSetChart() <-- Input nCells + nVerts 2. DMPlexSetConeSize() <-- Input ConeSize for each point in [0,nCells) 3. DMSetUp() ? Allocates memory internally. 4. DMPlexGetCones() --> Gives you the memory onto which to write the cell connectivity. 5. *Write connectivity* 6. DMPlexReorderCell() <-- For each point in [0,nCells) I'm in a situation where the memory given by step (4) is available a-priori. I was hoping to skip steps 2, 3 , and 4 with something like a "DMPlexSetCones()", but such an API does not exist. My current workaround is to implement steps 2 through 4 as always and have double the memory allocated in the interim (my instance + DM's internal instance). I was thinking of looking for the name of the struct member and = it to my memory, but I can't overcome the flag check in DMSetUp() during later calls to DMPlexGetCones() or DMPlexGetTransitiveClosure(). I do not understand the "flag check". What is that? We could make a DMPlexSetCones(), but as you point out, the workflow for DMSetUp() would have to change. Where does the memory come from for your connectivity? Thanks, Matt Sincerely: J.A. Ferrand Embry-Riddle Aeronautical University - Daytona Beach - FL Ph.D. Candidate, Aerospace Engineering M.Sc. Aerospace Engineering B.Sc. Aerospace Engineering B.Sc. Computational Mathematics Phone: (386)-843-1829 Email(s): ferranj2 at my.erau.edu jesus.ferrand at gmail.com -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!eJZaDdTGDKv1PKAoyrj5dR-1ZLRLekHn46Kq1QN8YsQTCX5IaaoSZ2ownEaefDvc2AOhkc8pqTms9d9lCHYRtccNFDE$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!eJZaDdTGDKv1PKAoyrj5dR-1ZLRLekHn46Kq1QN8YsQTCX5IaaoSZ2ownEaefDvc2AOhkc8pqTms9d9lCHYRtccNFDE$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!eJZaDdTGDKv1PKAoyrj5dR-1ZLRLekHn46Kq1QN8YsQTCX5IaaoSZ2ownEaefDvc2AOhkc8pqTms9d9lCHYRtccNFDE$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!eJZaDdTGDKv1PKAoyrj5dR-1ZLRLekHn46Kq1QN8YsQTCX5IaaoSZ2ownEaefDvc2AOhkc8pqTms9d9lCHYRtccNFDE$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!eJZaDdTGDKv1PKAoyrj5dR-1ZLRLekHn46Kq1QN8YsQTCX5IaaoSZ2ownEaefDvc2AOhkc8pqTms9d9lCHYRtccNFDE$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon May 6 16:14:35 2024 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 6 May 2024 17:14:35 -0400 Subject: [petsc-users] [EXTERNAL] Re: Is there anything like a "DMPlexSetCones()" ? In-Reply-To: References: Message-ID: On Mon, May 6, 2024 at 4:33?PM Ferrand, Jesus A. wrote: > "What is the argument for this optimization?" > > Ah! > That's all I had really. > That you would skip this one malloc step but without regard to the grander > picture. > I had a gut feeling that it is a sizeable double malloc because it is the > cell connectivity. > However, once you bring into perspective all the other stuff > (CellOrientation, Vecs, etc.) it does seem like a micro-optimization. > If in your estimation this isn't worth the development, then I'll continue > to follow the guidance from the PETSc source code. > Once everything is coded, we can put in monitoring, and see if it would make a difference. Thanks, Matt > ------------------------------ > *From:* Matthew Knepley > *Sent:* Monday, May 6, 2024 10:55 AM > *To:* Ferrand, Jesus A. > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [EXTERNAL] Re: [petsc-users] Is there anything like a > "DMPlexSetCones()" ? > > On Mon, May 6, 2024 at 10:17?AM Ferrand, Jesus A. > wrote: > > "The cone sizes are used to allocate the array to hold the cones. How > would you have an array holding the cone information without knowing the > sizes?" > > You're right, I must first know the sizes (which I determine from the > file). > It lists the number of cells and their types by sequential chunks. > > The issue I am trying to describe, is that because I cannot know the > number of local vertices a priori, the pipeline > DMSetChart()->DMPlexSetConeSize()->DMSetUp() essentially produces a > redundant array (because I already have the connectivity that I read from > the file). > As far as I can tell, in my case, the end result of calling these APIs is > already available, hence my wish for a "DMPlexSetCones()". > If such an API existed, I would then attempt to retroactively set the > Chart and the ConeSizes. > > Like you hinted in a reply, for this extra array to be worthwhile, I would > have to keep it as a global numbering list (which I want to by the way). > I was hoping to at least keep them in the DM's global PetscSection as > offsets. > Unfortunately, PetscSection has a similar workflow to DM: > PetscSectionSetChart(), PetscSectionAddDof(), PetscSectionSetUp(), > PetscSectionSetOffset(). > > > I looked at the other parallel code. It does maintain two arrays, so that > you can compute this stuff. > > I am reluctant to add this interface, since I think the optimization is > very small and it is not a good pattern. > You will save allocating the cell connectivity. However, if you > interpolate the mesh, this is small. This is the > same size as the cell orientations, which we will also have to allocate. > We are likely to allocate vectors over the mesh. It does not seem likely > that this allocation will increase the peak allocation, and it will be > freed after creation, so total size will not be affected. What is the > argument for this optimization? > > Thanks, > > Matt > > > ------------------------------ > *From:* Matthew Knepley > *Sent:* Monday, May 6, 2024 7:52 AM > *To:* Ferrand, Jesus A. > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [EXTERNAL] Re: [petsc-users] Is there anything like a > "DMPlexSetCones()" ? > > On Fri, May 3, 2024 at 7:32?PM Ferrand, Jesus A. > wrote: > > "Right, but that means that the connectivity you read in is not in local > numbering. You will have to renumber it, so making a copy is probably > necessary anyway. This is the same sort of processing I do for the parallel > read." > > True, the connectivity is read in global numbering. > Renumbering is facilitated by the table from PetscHashSetI. > The connectivity list I am able to renumber in place though, I don't see > the need to have another interim copy of the numbers. > My concern is that I have two memory chunks that both contain the same > information. > The only reason It has to be this way is because I can't manually set the > Cones first and the Chart/ConeSizes second. > > > The cone sizes are used to allocate the array to hold the cones. How would > you have an array holding the cone information without knowing the sizes? > > Thanks, > > Matt > > > ------------------------------ > *From:* Matthew Knepley > *Sent:* Friday, May 3, 2024 6:45 PM > *To:* Ferrand, Jesus A. > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [EXTERNAL] Re: [petsc-users] Is there anything like a > "DMPlexSetCones()" ? > > On Wed, May 1, 2024 at 6:30?PM Ferrand, Jesus A. > wrote: > > "You know the number of cells if you can read connectivity. Do you mean > that the format does not tell you > the number of vertices?" > > Sort of... > So, the format does provide the number of vertices, however, due to the > way I read the data in parallel I don't know immediately how many local > vertices there will be. > Which prevents me from knowing the chart a priori. > I figured out the number of vertices using PetscHashSetI to determine the > number of unique entries in the connectivity list. > > > Right, but that means that the connectivity you read in is not in local > numbering. You will have to renumber it, so making a copy is probably > necessary anyway. This is the same sort of processing I do for the parallel > read. > > Thanks, > > Matt > > > ------------------------------ > *From:* Matthew Knepley > *Sent:* Wednesday, May 1, 2024 8:52 PM > *To:* Ferrand, Jesus A. > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [EXTERNAL] Re: [petsc-users] Is there anything like a > "DMPlexSetCones()" ? > > On Wed, May 1, 2024 at 4:23?PM Ferrand, Jesus A. > wrote: > > Matt: > > My bad again, I need to clarify something that I just realized doesn't > make sense. > I said: "The nature of the I/O makes it so that I need to read the > connectivity before I get a semblance of buffer sizes." > Scratch that, obviously, if I can read the connectivity, I know the buffer > sizes. > > What I should have said is that I cannot know the "chart" that one sets in > DMPlexSetChart() a priori like in most cases. > I need to determine the chart from the connectivity lists. > > > You know the number of cells if you can read connectivity. Do you mean > that the format does not tell you > the number of vertices? > > Thanks, > > Matt > > > ------------------------------ > *From:* Ferrand, Jesus A. > *Sent:* Wednesday, May 1, 2024 8:17 PM > *To:* Matthew Knepley > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [EXTERNAL] Re: [petsc-users] Is there anything like a > "DMPlexSetCones()" ? > > Matt: > > "I do not understand the "flag check". What is that?" > > My bad, I should have referred to the "dm->setupcalled". > I believe this PetscBool is checked by the other DM (not just DMPlex) APIs. > The subsequent checks for dm->setupcalled == PETSC_TRUE is what I meant to > say. > Here's a copy of DMSetUp(). > > *PetscErrorCode DMSetUp (DM dm)* > 817: { > 818: PetscFunctionBegin ; > 820: if (dm->setupcalled) PetscFunctionReturn (PETSC_SUCCESS ); > 821: PetscTryTypeMethod (dm, setup); > 822: dm->setupcalled = PETSC_TRUE ; > 823: PetscFunctionReturn (PETSC_SUCCESS ); > 824: } > > "We could make a DMPlexSetCones(), but as you point out, the workflow for > DMSetUp() would have to change. Where does the memory come from for your > connectivity?" > It's a memory that I myself allocate based on a file's contents. > The nature of the I/O makes it so that I need to read the connectivity > before I get a semblance of buffer sizes. > Otherwise, I would stick to the tried and tested way. > > Also, when replying to the PETSc developers, users must reply to > petsc-users at mcs.anl.gov and not just to the individual email accounts of > the developers, right? > > > > > ------------------------------ > *From:* Matthew Knepley > *Sent:* Wednesday, May 1, 2024 8:07 PM > *To:* Ferrand, Jesus A. > *Cc:* petsc-users at mcs.anl.gov > *Subject:* [EXTERNAL] Re: [petsc-users] Is there anything like a > "DMPlexSetCones()" ? > > *CAUTION:* This email originated outside of Embry-Riddle Aeronautical > University. Do not click links or open attachments unless you recognize the > sender and know the content is safe. > > On Wed, May 1, 2024 at 3:34?PM Ferrand, Jesus A. > wrote: > > This Message Is From an External Sender > This message came from outside your organization. > > Dear PETSc team: > > For a project that I'm working on, I need to manually build a DMPlex. > From studying the source code of the various APIs in which the plex is > built from some supported file format, I get that the workflow is this: > > > 1. DMPlexSetChart() <-- Input nCells + nVerts > 2. DMPlexSetConeSize() <-- Input ConeSize for each point in [0,nCells) > 3. DMSetUp() ? Allocates memory internally. > 4. DMPlexGetCones() --> Gives you the memory onto which to write the > cell connectivity. > 5. *Write connectivity* > 6. DMPlexReorderCell() <-- For each point in [0,nCells) > > > I'm in a situation where the memory given by step (4) is available > a-priori. > I was hoping to skip steps 2, 3 , and 4 with something like a > "DMPlexSetCones()", but such an API does not exist. > My current workaround is to implement steps 2 through 4 as always and have > double the memory allocated in the interim (my instance + DM's internal > instance). > I was thinking of looking for the name of the struct member and = it to my > memory, but I can't overcome the flag check in DMSetUp() during later calls > to DMPlexGetCones() or DMPlexGetTransitiveClosure(). > > > I do not understand the "flag check". What is that? > > We could make a DMPlexSetCones(), but as you point out, the workflow for > DMSetUp() would have to change. Where does the memory come from for your > connectivity? > > Thanks, > > Matt > > > > Sincerely: > > *J.A. Ferrand* > > Embry-Riddle Aeronautical University - Daytona Beach - FL > Ph.D. Candidate, Aerospace Engineering > > M.Sc. Aerospace Engineering > > B.Sc. Aerospace Engineering > > B.Sc. Computational Mathematics > > > *Phone:* (386)-843-1829 > > *Email(s):* ferranj2 at my.erau.edu > > jesus.ferrand at gmail.com > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!e9WPKXONc0pLYPYlljjUQFKHdYCzY9jNdMc8ArfhWOkp0rNjus9LqI5q44BmF41a_NqGuSVzGofYEOhAg41j$ > > > ------------------------------ > *From:* Matthew Knepley > *Sent:* Wednesday, May 1, 2024 8:07 PM > *To:* Ferrand, Jesus A. > *Cc:* petsc-users at mcs.anl.gov > *Subject:* [EXTERNAL] Re: [petsc-users] Is there anything like a > "DMPlexSetCones()" ? > > *CAUTION:* This email originated outside of Embry-Riddle Aeronautical > University. Do not click links or open attachments unless you recognize the > sender and know the content is safe. > > On Wed, May 1, 2024 at 3:34?PM Ferrand, Jesus A. > wrote: > > This Message Is From an External Sender > This message came from outside your organization. > > Dear PETSc team: > > For a project that I'm working on, I need to manually build a DMPlex. > From studying the source code of the various APIs in which the plex is > built from some supported file format, I get that the workflow is this: > > > 1. DMPlexSetChart() <-- Input nCells + nVerts > 2. DMPlexSetConeSize() <-- Input ConeSize for each point in [0,nCells) > 3. DMSetUp() ? Allocates memory internally. > 4. DMPlexGetCones() --> Gives you the memory onto which to write the > cell connectivity. > 5. *Write connectivity* > 6. DMPlexReorderCell() <-- For each point in [0,nCells) > > > I'm in a situation where the memory given by step (4) is available > a-priori. > I was hoping to skip steps 2, 3 , and 4 with something like a > "DMPlexSetCones()", but such an API does not exist. > My current workaround is to implement steps 2 through 4 as always and have > double the memory allocated in the interim (my instance + DM's internal > instance). > I was thinking of looking for the name of the struct member and = it to my > memory, but I can't overcome the flag check in DMSetUp() during later calls > to DMPlexGetCones() or DMPlexGetTransitiveClosure(). > > > I do not understand the "flag check". What is that? > > We could make a DMPlexSetCones(), but as you point out, the workflow for > DMSetUp() would have to change. Where does the memory come from for your > connectivity? > > Thanks, > > Matt > > > > Sincerely: > > *J.A. Ferrand* > > Embry-Riddle Aeronautical University - Daytona Beach - FL > Ph.D. Candidate, Aerospace Engineering > > M.Sc. Aerospace Engineering > > B.Sc. Aerospace Engineering > > B.Sc. Computational Mathematics > > > *Phone:* (386)-843-1829 > > *Email(s):* ferranj2 at my.erau.edu > > jesus.ferrand at gmail.com > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!e9WPKXONc0pLYPYlljjUQFKHdYCzY9jNdMc8ArfhWOkp0rNjus9LqI5q44BmF41a_NqGuSVzGofYEOhAg41j$ > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!e9WPKXONc0pLYPYlljjUQFKHdYCzY9jNdMc8ArfhWOkp0rNjus9LqI5q44BmF41a_NqGuSVzGofYEOhAg41j$ > > > ------------------------------ > *From:* Matthew Knepley > *Sent:* Wednesday, May 1, 2024 8:52 PM > *To:* Ferrand, Jesus A. > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [EXTERNAL] Re: [petsc-users] Is there anything like a > "DMPlexSetCones()" ? > > On Wed, May 1, 2024 at 4:23?PM Ferrand, Jesus A. > wrote: > > Matt: > > My bad again, I need to clarify something that I just realized doesn't > make sense. > I said: "The nature of the I/O makes it so that I need to read the > connectivity before I get a semblance of buffer sizes." > Scratch that, obviously, if I can read the connectivity, I know the buffer > sizes. > > What I should have said is that I cannot know the "chart" that one sets in > DMPlexSetChart() a priori like in most cases. > I need to determine the chart from the connectivity lists. > > > You know the number of cells if you can read connectivity. Do you mean > that the format does not tell you > the number of vertices? > > Thanks, > > Matt > > > ------------------------------ > *From:* Ferrand, Jesus A. > *Sent:* Wednesday, May 1, 2024 8:17 PM > *To:* Matthew Knepley > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [EXTERNAL] Re: [petsc-users] Is there anything like a > "DMPlexSetCones()" ? > > Matt: > > "I do not understand the "flag check". What is that?" > > My bad, I should have referred to the "dm->setupcalled". > I believe this PetscBool is checked by the other DM (not just DMPlex) APIs. > The subsequent checks for dm->setupcalled == PETSC_TRUE is what I meant to > say. > Here's a copy of DMSetUp(). > > *PetscErrorCode DMSetUp (DM dm)* > 817: { > 818: PetscFunctionBegin ; > 820: if (dm->setupcalled) PetscFunctionReturn (PETSC_SUCCESS ); > 821: PetscTryTypeMethod (dm, setup); > 822: dm->setupcalled = PETSC_TRUE ; > 823: PetscFunctionReturn (PETSC_SUCCESS ); > 824: } > > "We could make a DMPlexSetCones(), but as you point out, the workflow for > DMSetUp() would have to change. Where does the memory come from for your > connectivity?" > It's a memory that I myself allocate based on a file's contents. > The nature of the I/O makes it so that I need to read the connectivity > before I get a semblance of buffer sizes. > Otherwise, I would stick to the tried and tested way. > > Also, when replying to the PETSc developers, users must reply to > petsc-users at mcs.anl.gov and not just to the individual email accounts of > the developers, right? > > > > > ------------------------------ > *From:* Matthew Knepley > *Sent:* Wednesday, May 1, 2024 8:07 PM > *To:* Ferrand, Jesus A. > *Cc:* petsc-users at mcs.anl.gov > *Subject:* [EXTERNAL] Re: [petsc-users] Is there anything like a > "DMPlexSetCones()" ? > > *CAUTION:* This email originated outside of Embry-Riddle Aeronautical > University. Do not click links or open attachments unless you recognize the > sender and know the content is safe. > > On Wed, May 1, 2024 at 3:34?PM Ferrand, Jesus A. > wrote: > > This Message Is From an External Sender > This message came from outside your organization. > > Dear PETSc team: > > For a project that I'm working on, I need to manually build a DMPlex. > From studying the source code of the various APIs in which the plex is > built from some supported file format, I get that the workflow is this: > > > 1. DMPlexSetChart() <-- Input nCells + nVerts > 2. DMPlexSetConeSize() <-- Input ConeSize for each point in [0,nCells) > 3. DMSetUp() ? Allocates memory internally. > 4. DMPlexGetCones() --> Gives you the memory onto which to write the > cell connectivity. > 5. *Write connectivity* > 6. DMPlexReorderCell() <-- For each point in [0,nCells) > > > I'm in a situation where the memory given by step (4) is available > a-priori. > I was hoping to skip steps 2, 3 , and 4 with something like a > "DMPlexSetCones()", but such an API does not exist. > My current workaround is to implement steps 2 through 4 as always and have > double the memory allocated in the interim (my instance + DM's internal > instance). > I was thinking of looking for the name of the struct member and = it to my > memory, but I can't overcome the flag check in DMSetUp() during later calls > to DMPlexGetCones() or DMPlexGetTransitiveClosure(). > > > I do not understand the "flag check". What is that? > > We could make a DMPlexSetCones(), but as you point out, the workflow for > DMSetUp() would have to change. Where does the memory come from for your > connectivity? > > Thanks, > > Matt > > > > Sincerely: > > *J.A. Ferrand* > > Embry-Riddle Aeronautical University - Daytona Beach - FL > Ph.D. Candidate, Aerospace Engineering > > M.Sc. Aerospace Engineering > > B.Sc. Aerospace Engineering > > B.Sc. Computational Mathematics > > > *Phone:* (386)-843-1829 > > *Email(s):* ferranj2 at my.erau.edu > > jesus.ferrand at gmail.com > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!e9WPKXONc0pLYPYlljjUQFKHdYCzY9jNdMc8ArfhWOkp0rNjus9LqI5q44BmF41a_NqGuSVzGofYEOhAg41j$ > > > ------------------------------ > *From:* Matthew Knepley > *Sent:* Wednesday, May 1, 2024 8:07 PM > *To:* Ferrand, Jesus A. > *Cc:* petsc-users at mcs.anl.gov > *Subject:* [EXTERNAL] Re: [petsc-users] Is there anything like a > "DMPlexSetCones()" ? > > *CAUTION:* This email originated outside of Embry-Riddle Aeronautical > University. Do not click links or open attachments unless you recognize the > sender and know the content is safe. > > On Wed, May 1, 2024 at 3:34?PM Ferrand, Jesus A. > wrote: > > This Message Is From an External Sender > This message came from outside your organization. > > Dear PETSc team: > > For a project that I'm working on, I need to manually build a DMPlex. > From studying the source code of the various APIs in which the plex is > built from some supported file format, I get that the workflow is this: > > > 1. DMPlexSetChart() <-- Input nCells + nVerts > 2. DMPlexSetConeSize() <-- Input ConeSize for each point in [0,nCells) > 3. DMSetUp() ? Allocates memory internally. > 4. DMPlexGetCones() --> Gives you the memory onto which to write the > cell connectivity. > 5. *Write connectivity* > 6. DMPlexReorderCell() <-- For each point in [0,nCells) > > > I'm in a situation where the memory given by step (4) is available > a-priori. > I was hoping to skip steps 2, 3 , and 4 with something like a > "DMPlexSetCones()", but such an API does not exist. > My current workaround is to implement steps 2 through 4 as always and have > double the memory allocated in the interim (my instance + DM's internal > instance). > I was thinking of looking for the name of the struct member and = it to my > memory, but I can't overcome the flag check in DMSetUp() during later calls > to DMPlexGetCones() or DMPlexGetTransitiveClosure(). > > > I do not understand the "flag check". What is that? > > We could make a DMPlexSetCones(), but as you point out, the workflow for > DMSetUp() would have to change. Where does the memory come from for your > connectivity? > > Thanks, > > Matt > > > > Sincerely: > > *J.A. Ferrand* > > Embry-Riddle Aeronautical University - Daytona Beach - FL > Ph.D. Candidate, Aerospace Engineering > > M.Sc. Aerospace Engineering > > B.Sc. Aerospace Engineering > > B.Sc. Computational Mathematics > > > *Phone:* (386)-843-1829 > > *Email(s):* ferranj2 at my.erau.edu > > jesus.ferrand at gmail.com > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!e9WPKXONc0pLYPYlljjUQFKHdYCzY9jNdMc8ArfhWOkp0rNjus9LqI5q44BmF41a_NqGuSVzGofYEOhAg41j$ > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!e9WPKXONc0pLYPYlljjUQFKHdYCzY9jNdMc8ArfhWOkp0rNjus9LqI5q44BmF41a_NqGuSVzGofYEOhAg41j$ > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!e9WPKXONc0pLYPYlljjUQFKHdYCzY9jNdMc8ArfhWOkp0rNjus9LqI5q44BmF41a_NqGuSVzGofYEOhAg41j$ > > > ------------------------------ > *From:* Matthew Knepley > *Sent:* Friday, May 3, 2024 6:45 PM > *To:* Ferrand, Jesus A. > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [EXTERNAL] Re: [petsc-users] Is there anything like a > "DMPlexSetCones()" ? > > On Wed, May 1, 2024 at 6:30?PM Ferrand, Jesus A. > wrote: > > "You know the number of cells if you can read connectivity. Do you mean > that the format does not tell you > the number of vertices?" > > Sort of... > So, the format does provide the number of vertices, however, due to the > way I read the data in parallel I don't know immediately how many local > vertices there will be. > Which prevents me from knowing the chart a priori. > I figured out the number of vertices using PetscHashSetI to determine the > number of unique entries in the connectivity list. > > > Right, but that means that the connectivity you read in is not in local > numbering. You will have to renumber it, so making a copy is probably > necessary anyway. This is the same sort of processing I do for the parallel > read. > > Thanks, > > Matt > > > ------------------------------ > *From:* Matthew Knepley > *Sent:* Wednesday, May 1, 2024 8:52 PM > *To:* Ferrand, Jesus A. > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [EXTERNAL] Re: [petsc-users] Is there anything like a > "DMPlexSetCones()" ? > > On Wed, May 1, 2024 at 4:23?PM Ferrand, Jesus A. > wrote: > > Matt: > > My bad again, I need to clarify something that I just realized doesn't > make sense. > I said: "The nature of the I/O makes it so that I need to read the > connectivity before I get a semblance of buffer sizes." > Scratch that, obviously, if I can read the connectivity, I know the buffer > sizes. > > What I should have said is that I cannot know the "chart" that one sets in > DMPlexSetChart() a priori like in most cases. > I need to determine the chart from the connectivity lists. > > > You know the number of cells if you can read connectivity. Do you mean > that the format does not tell you > the number of vertices? > > Thanks, > > Matt > > > ------------------------------ > *From:* Ferrand, Jesus A. > *Sent:* Wednesday, May 1, 2024 8:17 PM > *To:* Matthew Knepley > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [EXTERNAL] Re: [petsc-users] Is there anything like a > "DMPlexSetCones()" ? > > Matt: > > "I do not understand the "flag check". What is that?" > > My bad, I should have referred to the "dm->setupcalled". > I believe this PetscBool is checked by the other DM (not just DMPlex) APIs. > The subsequent checks for dm->setupcalled == PETSC_TRUE is what I meant to > say. > Here's a copy of DMSetUp(). > > *PetscErrorCode DMSetUp (DM dm)* > 817: { > 818: PetscFunctionBegin ; > 820: if (dm->setupcalled) PetscFunctionReturn (PETSC_SUCCESS ); > 821: PetscTryTypeMethod (dm, setup); > 822: dm->setupcalled = PETSC_TRUE ; > 823: PetscFunctionReturn (PETSC_SUCCESS ); > 824: } > > "We could make a DMPlexSetCones(), but as you point out, the workflow for > DMSetUp() would have to change. Where does the memory come from for your > connectivity?" > It's a memory that I myself allocate based on a file's contents. > The nature of the I/O makes it so that I need to read the connectivity > before I get a semblance of buffer sizes. > Otherwise, I would stick to the tried and tested way. > > Also, when replying to the PETSc developers, users must reply to > petsc-users at mcs.anl.gov and not just to the individual email accounts of > the developers, right? > > > > > ------------------------------ > *From:* Matthew Knepley > *Sent:* Wednesday, May 1, 2024 8:07 PM > *To:* Ferrand, Jesus A. > *Cc:* petsc-users at mcs.anl.gov > *Subject:* [EXTERNAL] Re: [petsc-users] Is there anything like a > "DMPlexSetCones()" ? > > *CAUTION:* This email originated outside of Embry-Riddle Aeronautical > University. Do not click links or open attachments unless you recognize the > sender and know the content is safe. > > On Wed, May 1, 2024 at 3:34?PM Ferrand, Jesus A. > wrote: > > This Message Is From an External Sender > This message came from outside your organization. > > Dear PETSc team: > > For a project that I'm working on, I need to manually build a DMPlex. > From studying the source code of the various APIs in which the plex is > built from some supported file format, I get that the workflow is this: > > > 1. DMPlexSetChart() <-- Input nCells + nVerts > 2. DMPlexSetConeSize() <-- Input ConeSize for each point in [0,nCells) > 3. DMSetUp() ? Allocates memory internally. > 4. DMPlexGetCones() --> Gives you the memory onto which to write the > cell connectivity. > 5. *Write connectivity* > 6. DMPlexReorderCell() <-- For each point in [0,nCells) > > > I'm in a situation where the memory given by step (4) is available > a-priori. > I was hoping to skip steps 2, 3 , and 4 with something like a > "DMPlexSetCones()", but such an API does not exist. > My current workaround is to implement steps 2 through 4 as always and have > double the memory allocated in the interim (my instance + DM's internal > instance). > I was thinking of looking for the name of the struct member and = it to my > memory, but I can't overcome the flag check in DMSetUp() during later calls > to DMPlexGetCones() or DMPlexGetTransitiveClosure(). > > > I do not understand the "flag check". What is that? > > We could make a DMPlexSetCones(), but as you point out, the workflow for > DMSetUp() would have to change. Where does the memory come from for your > connectivity? > > Thanks, > > Matt > > > > Sincerely: > > *J.A. Ferrand* > > Embry-Riddle Aeronautical University - Daytona Beach - FL > Ph.D. Candidate, Aerospace Engineering > > M.Sc. Aerospace Engineering > > B.Sc. Aerospace Engineering > > B.Sc. Computational Mathematics > > > *Phone:* (386)-843-1829 > > *Email(s):* ferranj2 at my.erau.edu > > jesus.ferrand at gmail.com > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!e9WPKXONc0pLYPYlljjUQFKHdYCzY9jNdMc8ArfhWOkp0rNjus9LqI5q44BmF41a_NqGuSVzGofYEOhAg41j$ > > > ------------------------------ > *From:* Matthew Knepley > *Sent:* Wednesday, May 1, 2024 8:07 PM > *To:* Ferrand, Jesus A. > *Cc:* petsc-users at mcs.anl.gov > *Subject:* [EXTERNAL] Re: [petsc-users] Is there anything like a > "DMPlexSetCones()" ? > > *CAUTION:* This email originated outside of Embry-Riddle Aeronautical > University. Do not click links or open attachments unless you recognize the > sender and know the content is safe. > > On Wed, May 1, 2024 at 3:34?PM Ferrand, Jesus A. > wrote: > > This Message Is From an External Sender > This message came from outside your organization. > > Dear PETSc team: > > For a project that I'm working on, I need to manually build a DMPlex. > From studying the source code of the various APIs in which the plex is > built from some supported file format, I get that the workflow is this: > > > 1. DMPlexSetChart() <-- Input nCells + nVerts > 2. DMPlexSetConeSize() <-- Input ConeSize for each point in [0,nCells) > 3. DMSetUp() ? Allocates memory internally. > 4. DMPlexGetCones() --> Gives you the memory onto which to write the > cell connectivity. > 5. *Write connectivity* > 6. DMPlexReorderCell() <-- For each point in [0,nCells) > > > I'm in a situation where the memory given by step (4) is available > a-priori. > I was hoping to skip steps 2, 3 , and 4 with something like a > "DMPlexSetCones()", but such an API does not exist. > My current workaround is to implement steps 2 through 4 as always and have > double the memory allocated in the interim (my instance + DM's internal > instance). > I was thinking of looking for the name of the struct member and = it to my > memory, but I can't overcome the flag check in DMSetUp() during later calls > to DMPlexGetCones() or DMPlexGetTransitiveClosure(). > > > I do not understand the "flag check". What is that? > > We could make a DMPlexSetCones(), but as you point out, the workflow for > DMSetUp() would have to change. Where does the memory come from for your > connectivity? > > Thanks, > > Matt > > > > Sincerely: > > *J.A. Ferrand* > > Embry-Riddle Aeronautical University - Daytona Beach - FL > Ph.D. Candidate, Aerospace Engineering > > M.Sc. Aerospace Engineering > > B.Sc. Aerospace Engineering > > B.Sc. Computational Mathematics > > > *Phone:* (386)-843-1829 > > *Email(s):* ferranj2 at my.erau.edu > > jesus.ferrand at gmail.com > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!e9WPKXONc0pLYPYlljjUQFKHdYCzY9jNdMc8ArfhWOkp0rNjus9LqI5q44BmF41a_NqGuSVzGofYEOhAg41j$ > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!e9WPKXONc0pLYPYlljjUQFKHdYCzY9jNdMc8ArfhWOkp0rNjus9LqI5q44BmF41a_NqGuSVzGofYEOhAg41j$ > > > ------------------------------ > *From:* Matthew Knepley > *Sent:* Wednesday, May 1, 2024 8:52 PM > *To:* Ferrand, Jesus A. > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [EXTERNAL] Re: [petsc-users] Is there anything like a > "DMPlexSetCones()" ? > > On Wed, May 1, 2024 at 4:23?PM Ferrand, Jesus A. > wrote: > > Matt: > > My bad again, I need to clarify something that I just realized doesn't > make sense. > I said: "The nature of the I/O makes it so that I need to read the > connectivity before I get a semblance of buffer sizes." > Scratch that, obviously, if I can read the connectivity, I know the buffer > sizes. > > What I should have said is that I cannot know the "chart" that one sets in > DMPlexSetChart() a priori like in most cases. > I need to determine the chart from the connectivity lists. > > > You know the number of cells if you can read connectivity. Do you mean > that the format does not tell you > the number of vertices? > > Thanks, > > Matt > > > ------------------------------ > *From:* Ferrand, Jesus A. > *Sent:* Wednesday, May 1, 2024 8:17 PM > *To:* Matthew Knepley > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [EXTERNAL] Re: [petsc-users] Is there anything like a > "DMPlexSetCones()" ? > > Matt: > > "I do not understand the "flag check". What is that?" > > My bad, I should have referred to the "dm->setupcalled". > I believe this PetscBool is checked by the other DM (not just DMPlex) APIs. > The subsequent checks for dm->setupcalled == PETSC_TRUE is what I meant to > say. > Here's a copy of DMSetUp(). > > *PetscErrorCode DMSetUp (DM dm)* > 817: { > 818: PetscFunctionBegin ; > 820: if (dm->setupcalled) PetscFunctionReturn (PETSC_SUCCESS ); > 821: PetscTryTypeMethod (dm, setup); > 822: dm->setupcalled = PETSC_TRUE ; > 823: PetscFunctionReturn (PETSC_SUCCESS ); > 824: } > > "We could make a DMPlexSetCones(), but as you point out, the workflow for > DMSetUp() would have to change. Where does the memory come from for your > connectivity?" > It's a memory that I myself allocate based on a file's contents. > The nature of the I/O makes it so that I need to read the connectivity > before I get a semblance of buffer sizes. > Otherwise, I would stick to the tried and tested way. > > Also, when replying to the PETSc developers, users must reply to > petsc-users at mcs.anl.gov and not just to the individual email accounts of > the developers, right? > > > > > ------------------------------ > *From:* Matthew Knepley > *Sent:* Wednesday, May 1, 2024 8:07 PM > *To:* Ferrand, Jesus A. > *Cc:* petsc-users at mcs.anl.gov > *Subject:* [EXTERNAL] Re: [petsc-users] Is there anything like a > "DMPlexSetCones()" ? > > *CAUTION:* This email originated outside of Embry-Riddle Aeronautical > University. Do not click links or open attachments unless you recognize the > sender and know the content is safe. > > On Wed, May 1, 2024 at 3:34?PM Ferrand, Jesus A. > wrote: > > This Message Is From an External Sender > This message came from outside your organization. > > Dear PETSc team: > > For a project that I'm working on, I need to manually build a DMPlex. > From studying the source code of the various APIs in which the plex is > built from some supported file format, I get that the workflow is this: > > > 1. DMPlexSetChart() <-- Input nCells + nVerts > 2. DMPlexSetConeSize() <-- Input ConeSize for each point in [0,nCells) > 3. DMSetUp() ? Allocates memory internally. > 4. DMPlexGetCones() --> Gives you the memory onto which to write the > cell connectivity. > 5. *Write connectivity* > 6. DMPlexReorderCell() <-- For each point in [0,nCells) > > > I'm in a situation where the memory given by step (4) is available > a-priori. > I was hoping to skip steps 2, 3 , and 4 with something like a > "DMPlexSetCones()", but such an API does not exist. > My current workaround is to implement steps 2 through 4 as always and have > double the memory allocated in the interim (my instance + DM's internal > instance). > I was thinking of looking for the name of the struct member and = it to my > memory, but I can't overcome the flag check in DMSetUp() during later calls > to DMPlexGetCones() or DMPlexGetTransitiveClosure(). > > > I do not understand the "flag check". What is that? > > We could make a DMPlexSetCones(), but as you point out, the workflow for > DMSetUp() would have to change. Where does the memory come from for your > connectivity? > > Thanks, > > Matt > > > > Sincerely: > > *J.A. Ferrand* > > Embry-Riddle Aeronautical University - Daytona Beach - FL > Ph.D. Candidate, Aerospace Engineering > > M.Sc. Aerospace Engineering > > B.Sc. Aerospace Engineering > > B.Sc. Computational Mathematics > > > *Phone:* (386)-843-1829 > > *Email(s):* ferranj2 at my.erau.edu > > jesus.ferrand at gmail.com > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!e9WPKXONc0pLYPYlljjUQFKHdYCzY9jNdMc8ArfhWOkp0rNjus9LqI5q44BmF41a_NqGuSVzGofYEOhAg41j$ > > > ------------------------------ > *From:* Matthew Knepley > *Sent:* Wednesday, May 1, 2024 8:07 PM > *To:* Ferrand, Jesus A. > *Cc:* petsc-users at mcs.anl.gov > *Subject:* [EXTERNAL] Re: [petsc-users] Is there anything like a > "DMPlexSetCones()" ? > > *CAUTION:* This email originated outside of Embry-Riddle Aeronautical > University. Do not click links or open attachments unless you recognize the > sender and know the content is safe. > > On Wed, May 1, 2024 at 3:34?PM Ferrand, Jesus A. > wrote: > > This Message Is From an External Sender > This message came from outside your organization. > > Dear PETSc team: > > For a project that I'm working on, I need to manually build a DMPlex. > From studying the source code of the various APIs in which the plex is > built from some supported file format, I get that the workflow is this: > > > 1. DMPlexSetChart() <-- Input nCells + nVerts > 2. DMPlexSetConeSize() <-- Input ConeSize for each point in [0,nCells) > 3. DMSetUp() ? Allocates memory internally. > 4. DMPlexGetCones() --> Gives you the memory onto which to write the > cell connectivity. > 5. *Write connectivity* > 6. DMPlexReorderCell() <-- For each point in [0,nCells) > > > I'm in a situation where the memory given by step (4) is available > a-priori. > I was hoping to skip steps 2, 3 , and 4 with something like a > "DMPlexSetCones()", but such an API does not exist. > My current workaround is to implement steps 2 through 4 as always and have > double the memory allocated in the interim (my instance + DM's internal > instance). > I was thinking of looking for the name of the struct member and = it to my > memory, but I can't overcome the flag check in DMSetUp() during later calls > to DMPlexGetCones() or DMPlexGetTransitiveClosure(). > > > I do not understand the "flag check". What is that? > > We could make a DMPlexSetCones(), but as you point out, the workflow for > DMSetUp() would have to change. Where does the memory come from for your > connectivity? > > Thanks, > > Matt > > > > Sincerely: > > *J.A. Ferrand* > > Embry-Riddle Aeronautical University - Daytona Beach - FL > Ph.D. Candidate, Aerospace Engineering > > M.Sc. Aerospace Engineering > > B.Sc. Aerospace Engineering > > B.Sc. Computational Mathematics > > > *Phone:* (386)-843-1829 > > *Email(s):* ferranj2 at my.erau.edu > > jesus.ferrand at gmail.com > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!e9WPKXONc0pLYPYlljjUQFKHdYCzY9jNdMc8ArfhWOkp0rNjus9LqI5q44BmF41a_NqGuSVzGofYEOhAg41j$ > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!e9WPKXONc0pLYPYlljjUQFKHdYCzY9jNdMc8ArfhWOkp0rNjus9LqI5q44BmF41a_NqGuSVzGofYEOhAg41j$ > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!e9WPKXONc0pLYPYlljjUQFKHdYCzY9jNdMc8ArfhWOkp0rNjus9LqI5q44BmF41a_NqGuSVzGofYEOhAg41j$ > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!e9WPKXONc0pLYPYlljjUQFKHdYCzY9jNdMc8ArfhWOkp0rNjus9LqI5q44BmF41a_NqGuSVzGofYEOhAg41j$ > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!e9WPKXONc0pLYPYlljjUQFKHdYCzY9jNdMc8ArfhWOkp0rNjus9LqI5q44BmF41a_NqGuSVzGofYEOhAg41j$ > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!e9WPKXONc0pLYPYlljjUQFKHdYCzY9jNdMc8ArfhWOkp0rNjus9LqI5q44BmF41a_NqGuSVzGofYEOhAg41j$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From a.croucher at auckland.ac.nz Mon May 6 18:46:30 2024 From: a.croucher at auckland.ac.nz (Adrian Croucher) Date: Tue, 7 May 2024 11:46:30 +1200 Subject: [petsc-users] PETSc options In-Reply-To: <53C98BF3-786C-4981-8C44-F4D3F364972F@petsc.dev> References: <2d56c6ea-e9e0-4a7a-b565-90a1404e08b7@auckland.ac.nz> <53C98BF3-786C-4981-8C44-F4D3F364972F@petsc.dev> Message-ID: On 7/05/24 1:11 am, Barry Smith wrote: > > ?? Yes, just call PetscOptionsHasName() with -v and ignore the result. Thanks Barry, that does the trick. Yes, I was using PETSc <3.19 before which is why it didn't show up until now. - Adrian >> >> On Mon, May 6, 2024 at 1:04?AM Adrian Croucher >> wrote: >> >> hi, >> >> My code has some optional command line arguments -v and -h for output of >> version number and usage help. These are processed using Fortran's >> get_command_argument(). >> >> Since updating PETSc to version 3.21, I get some extra warnings after >> the output: >> >> acro018 at EN438880:~$ waiwera -v >> 1.5.0b1 >> WARNING! There are options you set that were not used! >> WARNING! could be spelling mistake, etc! >> There is one unused database option. It is: >> Option left: name:-v (no value) source: command line >> >> That didn't used to happen. What should I do to make them go away? >> >> Regards, Adrian >> >> > -- Dr Adrian Croucher Senior Research Fellow Department of Engineering Science Waipapa Taumata Rau / University of Auckland, New Zealand email:a.croucher at auckland.ac.nz tel: +64 (0)9 923 4611 -------------- next part -------------- An HTML attachment was scrubbed... URL: From wthacher at lbl.gov Mon May 6 21:02:44 2024 From: wthacher at lbl.gov (William Thacher) Date: Mon, 6 May 2024 19:02:44 -0700 Subject: [petsc-users] Fortran Linker Issue on Mac Sonoma Message-ID: Hello, I am encountering an issue when configuring/building PETSc on a Mac with Sonoma 14.4.1. Here are the exact steps I have performed so far: 1. Pulled the most recent release of PETSc 2. Ran "./configure --ignoreLinkOutput=1 LDFLAGS=-Wl,-ld_classic" In the configuration process (line 2518 of attached configure.log) it looks like there is a linker error when testing the fortran compiler. (I get the same error if I just run ./configure) If I run the make command given after running step 2. above, the C/C++ code builds fine, but I would also like to build the Fortran code. Have you all seen this error before on Mac Sonoma/have any ideas for a workaround? I installed open-mpi through Homebrew and upgraded before building PETSc. Appreciate the help, Will -- Will Thacher Graduate Student Researcher, Lawrence Berkeley National Lab PhD Student, Applied Science and Technology Group, UC Berkeley -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: configure.log Type: application/octet-stream Size: 5432547 bytes Desc: not available URL: From balay at mcs.anl.gov Mon May 6 21:28:44 2024 From: balay at mcs.anl.gov (Satish Balay) Date: Mon, 6 May 2024 21:28:44 -0500 (CDT) Subject: [petsc-users] Fortran Linker Issue on Mac Sonoma In-Reply-To: References: Message-ID: 1. Suggest reinstalling brew/gfortran - to make sure its compatible with latest xcode you have. https://urldefense.us/v3/__https://petsc.org/release/install/install/*installing-on-macos__;Iw!!G_uCfscf7eWS!Y7eNVqUTtBNMYgHPI9pgKFBFMZJwLjwjtp4gGnxFTyYpQMN-rfvF4wJiuZywnYU-Rof6DANF1xg1U0VLr7FgG54$ 2. Try: ./configure --download-openmpi=https://urldefense.us/v3/__https://web.cels.anl.gov/projects/petsc/download/externalpackages/openmpi-5.0.3-xcode15.tar.gz__;!!G_uCfscf7eWS!Y7eNVqUTtBNMYgHPI9pgKFBFMZJwLjwjtp4gGnxFTyYpQMN-rfvF4wJiuZywnYU-Rof6DANF1xg1U0VLmz41ilY$ Satish On Tue, 7 May 2024, William Thacher wrote: > Hello, > > I am encountering an issue when configuring/building PETSc on a Mac with > Sonoma 14.4.1. Here are the exact steps I have performed so far: > > 1. Pulled the most recent release of PETSc > 2. Ran "./configure --ignoreLinkOutput=1 LDFLAGS=-Wl,-ld_classic" > > In the configuration process (line 2518 of attached configure.log) it looks > like there is a linker error when testing the fortran compiler. (I get the > same error if I just run ./configure) > > If I run the make command given after running step 2. above, the C/C++ code > builds fine, but I would also like to build the Fortran code. > > Have you all seen this error before on Mac Sonoma/have any ideas for a > workaround? > > I installed open-mpi through Homebrew and upgraded before building PETSc. > > Appreciate the help, > > Will > From marco at kit.ac.jp Tue May 7 00:04:11 2024 From: marco at kit.ac.jp (Marco Seiz) Date: Tue, 7 May 2024 14:04:11 +0900 Subject: [petsc-users] Reasons for breakdown in preconditioned LSQR Message-ID: <30b07a76-9479-4a3a-9cf1-37251be79c07@kit.ac.jp> An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: lsdiv.zip Type: application/zip Size: 4834 bytes Desc: not available URL: From pierre at joliv.et Tue May 7 00:24:23 2024 From: pierre at joliv.et (Pierre Jolivet) Date: Tue, 7 May 2024 07:24:23 +0200 Subject: [petsc-users] Reasons for breakdown in preconditioned LSQR In-Reply-To: <30b07a76-9479-4a3a-9cf1-37251be79c07@kit.ac.jp> References: <30b07a76-9479-4a3a-9cf1-37251be79c07@kit.ac.jp> Message-ID: > On 7 May 2024, at 7:04?AM, Marco Seiz wrote: > > This Message Is From an External Sender > This message came from outside your organization. > Hello, > > something a bit different from my last question, since that didn't > progress so well: > I have a related model which generally produces a rectangular matrix A, > so I am using LSQR to solve the system. > The matrix A has two nonzeros (1, -1) per row, with A^T A being similar > to a finite difference Poisson matrix if the rows were permuted randomly. > The problem is singular in that the solution is only specified up to a > constant from the matrix, with my target solution being a weighted zero > average one, which I can handle by adding a nullspace to my matrix. > However, I'd also like to pin (potentially many) DOFs in the future so I > also tried pinning a single value, and afterwards subtracting the > average from the KSP solution. > This leads to the KSP *sometimes* diverging when I use a preconditioner; > the target size of the matrix will be something like ([1,20] N) x N, > with N ~ [2, 1e6] so for the higher end I will require a preconditioner > for reasonable execution time. > > For a smaller example system, I set up my application to dump the input > to the KSP when it breaks down and I've attached a simple python script > + data using petsc4py to demonstrate the divergence for those specific > systems. > With `python3 lsdiv.py -pc_type lu -ksp_converged_reason` that > particular system shows breakdown, but if I remove the pinned DOF and > add the nullspace (pass -usens) it converges. I did try different PCs > but they tend to break down at different steps, e.g. `python3 lsdiv.py > -usenormal -qrdiv -pc_type qr -ksp_converged_reason` shows the breakdown > for PCQR when I use MatCreateNormal for creating the PC mat, but > interestingly it doesn't break down when I explicitly form A^T A (don't > pass -usenormal). What version are you using? All those commands are returning Linear solve converged due to CONVERGED_RTOL_NORMAL iterations 1 So I cannot reproduce any breakdown, but there have been recent changes to KSPLSQR. > For the moment I can work by adding the nullspace but eventually the > need for pinning DOFs will resurface, so I'd like to ask where the > breakdown is coming from. What causes the breakdowns? Is that a generic > problem occurring when adding (dof_i = val) rows to least-squares > systems which prevents these preconditioners from being robust? If so, > what preconditioners could be robust? > I did a minimal sweep of the available PCs by going over the possible > inputs of -pc_type for my application while pinning one DOF. Excepting > unavailable PCs (not compiled for, other setup missing, ...) and those > that did break down, I am left with ( hmg jacobi mat none pbjacobi sor > svd ). It?s unlikely any of these preconditioners will scale (or even converge) for problems with up to 1E6 unknowns. I could help you setup https://urldefense.us/v3/__https://epubs.siam.org/doi/abs/10.1137/21M1434891__;!!G_uCfscf7eWS!cEzzrez47D-rTba54wHj2wMvjkRrPiIlyhhfvWp2H4HT4EO9-soMFVTQUt0bjRj08xqYWHXUBJusDpvvB-LqSQ$ if you are willing to share a larger example (the current Mat are extremely tiny). Thanks, Pierre > > > Best regards, > Marco > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From marco at kit.ac.jp Tue May 7 02:10:15 2024 From: marco at kit.ac.jp (Marco Seiz) Date: Tue, 7 May 2024 16:10:15 +0900 Subject: [petsc-users] Reasons for breakdown in preconditioned LSQR In-Reply-To: References: <30b07a76-9479-4a3a-9cf1-37251be79c07@kit.ac.jp> Message-ID: <31c6214d-c333-4624-ab74-a9ae626b50a7@kit.ac.jp> An HTML attachment was scrubbed... URL: From pierre at joliv.et Tue May 7 04:12:18 2024 From: pierre at joliv.et (Pierre Jolivet) Date: Tue, 7 May 2024 11:12:18 +0200 Subject: [petsc-users] Reasons for breakdown in preconditioned LSQR In-Reply-To: <31c6214d-c333-4624-ab74-a9ae626b50a7@kit.ac.jp> References: <30b07a76-9479-4a3a-9cf1-37251be79c07@kit.ac.jp> <31c6214d-c333-4624-ab74-a9ae626b50a7@kit.ac.jp> Message-ID: <7E39F820-0309-46C2-BCC5-C5F21ADCAB4D@joliv.et> > On 7 May 2024, at 9:10?AM, Marco Seiz wrote: > > Thanks for the quick response! > > On 07.05.24 14:24, Pierre Jolivet wrote: >> >> >>> On 7 May 2024, at 7:04?AM, Marco Seiz wrote: >>> >>> This Message Is From an External Sender >>> This message came from outside your organization. >>> Hello, >>> >>> something a bit different from my last question, since that didn't >>> progress so well: >>> I have a related model which generally produces a rectangular matrix A, >>> so I am using LSQR to solve the system. >>> The matrix A has two nonzeros (1, -1) per row, with A^T A being similar >>> to a finite difference Poisson matrix if the rows were permuted randomly. >>> The problem is singular in that the solution is only specified up to a >>> constant from the matrix, with my target solution being a weighted zero >>> average one, which I can handle by adding a nullspace to my matrix. >>> However, I'd also like to pin (potentially many) DOFs in the future so I >>> also tried pinning a single value, and afterwards subtracting the >>> average from the KSP solution. >>> This leads to the KSP *sometimes* diverging when I use a preconditioner; >>> the target size of the matrix will be something like ([1,20] N) x N, >>> with N ~ [2, 1e6] so for the higher end I will require a preconditioner >>> for reasonable execution time. >>> >>> For a smaller example system, I set up my application to dump the input >>> to the KSP when it breaks down and I've attached a simple python script >>> + data using petsc4py to demonstrate the divergence for those specific >>> systems. >>> With `python3 lsdiv.py -pc_type lu -ksp_converged_reason` that >>> particular system shows breakdown, but if I remove the pinned DOF and >>> add the nullspace (pass -usens) it converges. I did try different PCs >>> but they tend to break down at different steps, e.g. `python3 lsdiv.py >>> -usenormal -qrdiv -pc_type qr -ksp_converged_reason` shows the breakdown >>> for PCQR when I use MatCreateNormal for creating the PC mat, but >>> interestingly it doesn't break down when I explicitly form A^T A (don't >>> pass -usenormal). >> >> What version are you using? All those commands are returning >> Linear solve converged due to CONVERGED_RTOL_NORMAL iterations 1 >> So I cannot reproduce any breakdown, but there have been recent changes to KSPLSQR. > For those tests I've been using PETSc 3.20.5 (last githash was > 4b82c11ab5d ). > I pulled the latest version from gitlab ( 6b3135e3cbe ) and compiled it, > but I had to drop --download-suitesparse=1 from my earlier config due to > errors. > Should I write a separate mail about this? > > The LU example still behaves the same for me (`python3 lsdiv.py -pc_type > lu -ksp_converged_reason` gives DIVERGED_BREAKDOWN, `python3 lsdiv.py > -usens -pc_type lu -ksp_converged_reason` gives CONVERGED_RTOL_NORMAL) > but the QR example fails since I had to remove suitesparse. > petsc4py.__version__ reports 3.21.1 and if I rebuild my application, > then `ldd app` gives me `libpetsc.so .3.21 => > /opt/petsc/linux-c-opt/lib/libpetsc.so .3.21` so it should be using the > newly built one. > The application then still eventually yields a DIVERGED_BREAKDOWN. > I don't have a ~/.petscrc and PETSC_OPTIONS is unset, so if we are on > the same version and there's still a discrepancy it is quite weird. Quite weird indeed? $ python3 lsdiv.py -pc_type lu -ksp_converged_reason Linear solve converged due to CONVERGED_RTOL_NORMAL iterations 1 $ python3 lsdiv.py -usens -pc_type lu -ksp_converged_reason Linear solve converged due to CONVERGED_RTOL_NORMAL iterations 1 $ python3 lsdiv.py -pc_type qr -ksp_converged_reason Linear solve converged due to CONVERGED_RTOL_NORMAL iterations 1 $ python3 lsdiv.py -usens -pc_type qr -ksp_converged_reason Linear solve converged due to CONVERGED_RTOL_NORMAL iterations 1 >>> For the moment I can work by adding the nullspace but eventually the >>> need for pinning DOFs will resurface, so I'd like to ask where the >>> breakdown is coming from. What causes the breakdowns? Is that a generic >>> problem occurring when adding (dof_i = val) rows to least-squares >>> systems which prevents these preconditioners from being robust? If so, >>> what preconditioners could be robust? >>> I did a minimal sweep of the available PCs by going over the possible >>> inputs of -pc_type for my application while pinning one DOF. Excepting >>> unavailable PCs (not compiled for, other setup missing, ...) and those >>> that did break down, I am left with ( hmg jacobi mat none pbjacobi sor >>> svd ). >> It?s unlikely any of these preconditioners will scale (or even converge) for problems with up to 1E6 unknowns. >> I could help you setup https://urldefense.us/v3/__https://epubs.siam.org/doi/abs/10.1137/21M1434891__;!!G_uCfscf7eWS!auri5B6VaP-JYC4fuoLQd6QGnMRYi45UVg6GvK8V2FIlWo6HdPSPwjqjQnRiV2HkM5lAHgRRgpwXScvk6kPrWA$ if you are willing to share a larger example (the current Mat are extremely tiny). > Yes, that would be great. About how large of a matrix do you need? I can > probably quickly get something non-artificial up to O(N) ~ 1e3, That?s big enough. If you?re in luck, AMG on the normal equations won?t behave too badly, but I?ll try some more robust (in theory) methods nonetheless. Thanks, Pierre > bigger > matrices will take some time since I purposefully ignored MPI previously. > The matrix basically describes the contacts between particles which are > resolved on a uniform grid, so the main memory hog isn't the matrix but > rather resolving the particles. > I should mention that the matrix changes over the course of the > simulation but stays constant for many solves, i.e. hundreds to > thousands of solves with variable RHS between periods of contact > formation/loss. > >> >> Thanks, >> Pierre >>> >>> >>> Best regards, >>> Marco >>> >>> >> >> > Best regards, > Marco -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue May 7 07:02:27 2024 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 7 May 2024 08:02:27 -0400 Subject: [petsc-users] Reasons for breakdown in preconditioned LSQR In-Reply-To: <7E39F820-0309-46C2-BCC5-C5F21ADCAB4D@joliv.et> References: <30b07a76-9479-4a3a-9cf1-37251be79c07@kit.ac.jp> <31c6214d-c333-4624-ab74-a9ae626b50a7@kit.ac.jp> <7E39F820-0309-46C2-BCC5-C5F21ADCAB4D@joliv.et> Message-ID: On Tue, May 7, 2024 at 5:12?AM Pierre Jolivet wrote: > On 7 May 2024, at 9: 10 AM, Marco Seiz wrote: Thanks > for the quick response! On 07. 05. 24 14: 24, Pierre Jolivet wrote: On 7 > May 2024, at 7: 04 AM, Marco Seiz wrote: This > Message Is From an External > ZjQcmQRYFpfptBannerStart > This Message Is From an External Sender > This message came from outside your organization. > > ZjQcmQRYFpfptBannerEnd > > > On 7 May 2024, at 9:10?AM, Marco Seiz wrote: > > Thanks for the quick response! > > On 07.05.24 14:24, Pierre Jolivet wrote: > > > > On 7 May 2024, at 7:04?AM, Marco Seiz wrote: > > This Message Is From an External Sender > This message came from outside your organization. > Hello, > > something a bit different from my last question, since that didn't > progress so well: > I have a related model which generally produces a rectangular matrix A, > so I am using LSQR to solve the system. > The matrix A has two nonzeros (1, -1) per row, with A^T A being similar > to a finite difference Poisson matrix if the rows were permuted randomly. > The problem is singular in that the solution is only specified up to a > constant from the matrix, with my target solution being a weighted zero > average one, which I can handle by adding a nullspace to my matrix. > However, I'd also like to pin (potentially many) DOFs in the future so I > also tried pinning a single value, and afterwards subtracting the > average from the KSP solution. > This leads to the KSP *sometimes* diverging when I use a preconditioner; > the target size of the matrix will be something like ([1,20] N) x N, > with N ~ [2, 1e6] so for the higher end I will require a preconditioner > for reasonable execution time. > > For a smaller example system, I set up my application to dump the input > to the KSP when it breaks down and I've attached a simple python script > + data using petsc4py to demonstrate the divergence for those specific > systems. > With `python3 lsdiv.py -pc_type lu -ksp_converged_reason` that > particular system shows breakdown, but if I remove the pinned DOF and > add the nullspace (pass -usens) it converges. I did try different PCs > but they tend to break down at different steps, e.g. `python3 lsdiv.py > -usenormal -qrdiv -pc_type qr -ksp_converged_reason` shows the breakdown > for PCQR when I use MatCreateNormal for creating the PC mat, but > interestingly it doesn't break down when I explicitly form A^T A (don't > pass -usenormal). > > > What version are you using? All those commands are returning > Linear solve converged due to CONVERGED_RTOL_NORMAL iterations 1 > So I cannot reproduce any breakdown, but there have been recent changes to > KSPLSQR. > > For those tests I've been using PETSc 3.20.5 (last githash was > 4b82c11ab5d ). > I pulled the latest version from gitlab ( 6b3135e3cbe ) and compiled it, > but I had to drop --download-suitesparse=1 from my earlier config due to > errors. > Should I write a separate mail about this? > > The LU example still behaves the same for me (`python3 lsdiv.py -pc_type > lu -ksp_converged_reason` gives DIVERGED_BREAKDOWN, `python3 lsdiv.py > -usens -pc_type lu -ksp_converged_reason` gives CONVERGED_RTOL_NORMAL) > but the QR example fails since I had to remove suitesparse. > petsc4py.__version__ reports 3.21.1 and if I rebuild my application, > then `ldd app` gives me `libpetsc.so > .3.21 > => > /opt/petsc/linux-c-opt/lib/libpetsc.so > .3.21` > so it should be using the > newly built one. > The application then still eventually yields a DIVERGED_BREAKDOWN. > I don't have a ~/.petscrc and PETSC_OPTIONS is unset, so if we are on > the same version and there's still a discrepancy it is quite weird. > > > Quite weird indeed? > $ python3 lsdiv.py -pc_type lu -ksp_converged_reason > Linear solve converged due to CONVERGED_RTOL_NORMAL iterations 1 > $ python3 lsdiv.py -usens -pc_type lu -ksp_converged_reason > Linear solve converged due to CONVERGED_RTOL_NORMAL iterations 1 > $ python3 lsdiv.py -pc_type qr -ksp_converged_reason > Linear solve converged due to CONVERGED_RTOL_NORMAL iterations 1 > $ python3 lsdiv.py -usens -pc_type qr -ksp_converged_reason > Linear solve converged due to CONVERGED_RTOL_NORMAL iterations 1 > > For the moment I can work by adding the nullspace but eventually the > need for pinning DOFs will resurface, so I'd like to ask where the > breakdown is coming from. What causes the breakdowns? Is that a generic > problem occurring when adding (dof_i = val) rows to least-squares > systems which prevents these preconditioners from being robust? If so, > what preconditioners could be robust? > I did a minimal sweep of the available PCs by going over the possible > inputs of -pc_type for my application while pinning one DOF. Excepting > unavailable PCs (not compiled for, other setup missing, ...) and those > that did break down, I am left with ( hmg jacobi mat none pbjacobi sor > svd ). > > It?s unlikely any of these preconditioners will scale (or even converge) > for problems with up to 1E6 unknowns. > I could help you setup https://urldefense.us/v3/__https://epubs.siam.org/doi/abs/10.1137/21M1434891__;!!G_uCfscf7eWS!fB-MI7viuwcYPEUg4w1S4_woxMQH7Kg5wnygmQdQdtqlCY5hQY4bFmFI3dJtuNZX9R-0i_z0Hq4eR73CPbQx$ > > if you are willing to share a larger example (the current Mat are extremely > tiny). > > Yes, that would be great. About how large of a matrix do you need? I can > probably quickly get something non-artificial up to O(N) ~ 1e3, > > > That?s big enough. > If you?re in luck, AMG on the normal equations won?t behave too badly, but > I?ll try some more robust (in theory) methods nonetheless. > We have also had good luck forming and factoring a block diagonal approximation of the normal equations. It depends on your problem. What is this problem? Thanks, Matt > Thanks, > Pierre > > bigger > matrices will take some time since I purposefully ignored MPI previously. > The matrix basically describes the contacts between particles which are > resolved on a uniform grid, so the main memory hog isn't the matrix but > rather resolving the particles. > I should mention that the matrix changes over the course of the > simulation but stays constant for many solves, i.e. hundreds to > thousands of solves with variable RHS between periods of contact > formation/loss. > > > Thanks, > Pierre > > > > Best regards, > Marco > > > > > > Best regards, > Marco > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fB-MI7viuwcYPEUg4w1S4_woxMQH7Kg5wnygmQdQdtqlCY5hQY4bFmFI3dJtuNZX9R-0i_z0Hq4eRzQ_nPK6$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Tue May 7 07:27:50 2024 From: mfadams at lbl.gov (Mark Adams) Date: Tue, 7 May 2024 08:27:50 -0400 Subject: [petsc-users] Reasons for breakdown in preconditioned LSQR In-Reply-To: References: <30b07a76-9479-4a3a-9cf1-37251be79c07@kit.ac.jp> <31c6214d-c333-4624-ab74-a9ae626b50a7@kit.ac.jp> <7E39F820-0309-46C2-BCC5-C5F21ADCAB4D@joliv.et> Message-ID: "A^T A being similar to a finite difference Poisson matrix if the rows were permuted randomly." Normal eqs are a good option in general for rectangular systems and we have Poisson solvers. I'm not sure what you mean by "permuted randomly." A random permutation of the matrix can kill performance but not math. Mark On Tue, May 7, 2024 at 8:03?AM Matthew Knepley wrote: > On Tue, May 7, 2024 at 5: 12 AM Pierre Jolivet wrote: > On 7 May 2024, at 9: 10 AM, Marco Seiz wrote: Thanks > for the quick response! On 07. 05. 24 14: 24, Pierre Jolivet wrote: On 7 > May 2024, > ZjQcmQRYFpfptBannerStart > This Message Is From an External Sender > This message came from outside your organization. > > ZjQcmQRYFpfptBannerEnd > On Tue, May 7, 2024 at 5:12?AM Pierre Jolivet wrote: > >> On 7 May 2024, at 9: 10 AM, Marco Seiz wrote: >> Thanks for the quick response! On 07. 05. 24 14: 24, Pierre Jolivet wrote: >> On 7 May 2024, at 7: 04 AM, Marco Seiz wrote: This >> Message Is From an External >> ZjQcmQRYFpfptBannerStart >> This Message Is From an External Sender >> This message came from outside your organization. >> >> ZjQcmQRYFpfptBannerEnd >> >> >> On 7 May 2024, at 9:10?AM, Marco Seiz wrote: >> >> Thanks for the quick response! >> >> On 07.05.24 14:24, Pierre Jolivet wrote: >> >> >> >> On 7 May 2024, at 7:04?AM, Marco Seiz wrote: >> >> This Message Is From an External Sender >> This message came from outside your organization. >> Hello, >> >> something a bit different from my last question, since that didn't >> progress so well: >> I have a related model which generally produces a rectangular matrix A, >> so I am using LSQR to solve the system. >> The matrix A has two nonzeros (1, -1) per row, with A^T A being similar >> to a finite difference Poisson matrix if the rows were permuted randomly. >> The problem is singular in that the solution is only specified up to a >> constant from the matrix, with my target solution being a weighted zero >> average one, which I can handle by adding a nullspace to my matrix. >> However, I'd also like to pin (potentially many) DOFs in the future so I >> also tried pinning a single value, and afterwards subtracting the >> average from the KSP solution. >> This leads to the KSP *sometimes* diverging when I use a preconditioner; >> the target size of the matrix will be something like ([1,20] N) x N, >> with N ~ [2, 1e6] so for the higher end I will require a preconditioner >> for reasonable execution time. >> >> For a smaller example system, I set up my application to dump the input >> to the KSP when it breaks down and I've attached a simple python script >> + data using petsc4py to demonstrate the divergence for those specific >> systems. >> With `python3 lsdiv.py -pc_type lu -ksp_converged_reason` that >> particular system shows breakdown, but if I remove the pinned DOF and >> add the nullspace (pass -usens) it converges. I did try different PCs >> but they tend to break down at different steps, e.g. `python3 lsdiv.py >> -usenormal -qrdiv -pc_type qr -ksp_converged_reason` shows the breakdown >> for PCQR when I use MatCreateNormal for creating the PC mat, but >> interestingly it doesn't break down when I explicitly form A^T A (don't >> pass -usenormal). >> >> >> What version are you using? All those commands are returning >> Linear solve converged due to CONVERGED_RTOL_NORMAL iterations 1 >> So I cannot reproduce any breakdown, but there have been recent changes >> to KSPLSQR. >> >> For those tests I've been using PETSc 3.20.5 (last githash was >> 4b82c11ab5d ). >> I pulled the latest version from gitlab ( 6b3135e3cbe ) and compiled it, >> but I had to drop --download-suitesparse=1 from my earlier config due to >> errors. >> Should I write a separate mail about this? >> >> The LU example still behaves the same for me (`python3 lsdiv.py -pc_type >> lu -ksp_converged_reason` gives DIVERGED_BREAKDOWN, `python3 lsdiv.py >> -usens -pc_type lu -ksp_converged_reason` gives CONVERGED_RTOL_NORMAL) >> but the QR example fails since I had to remove suitesparse. >> petsc4py.__version__ reports 3.21.1 and if I rebuild my application, >> then `ldd app` gives me `libpetsc.so >> .3.21 >> => >> /opt/petsc/linux-c-opt/lib/libpetsc.so >> .3.21` >> so it should be using the >> newly built one. >> The application then still eventually yields a DIVERGED_BREAKDOWN. >> I don't have a ~/.petscrc and PETSC_OPTIONS is unset, so if we are on >> the same version and there's still a discrepancy it is quite weird. >> >> >> Quite weird indeed? >> $ python3 lsdiv.py -pc_type lu -ksp_converged_reason >> Linear solve converged due to CONVERGED_RTOL_NORMAL iterations 1 >> $ python3 lsdiv.py -usens -pc_type lu -ksp_converged_reason >> Linear solve converged due to CONVERGED_RTOL_NORMAL iterations 1 >> $ python3 lsdiv.py -pc_type qr -ksp_converged_reason >> Linear solve converged due to CONVERGED_RTOL_NORMAL iterations 1 >> $ python3 lsdiv.py -usens -pc_type qr -ksp_converged_reason >> Linear solve converged due to CONVERGED_RTOL_NORMAL iterations 1 >> >> For the moment I can work by adding the nullspace but eventually the >> need for pinning DOFs will resurface, so I'd like to ask where the >> breakdown is coming from. What causes the breakdowns? Is that a generic >> problem occurring when adding (dof_i = val) rows to least-squares >> systems which prevents these preconditioners from being robust? If so, >> what preconditioners could be robust? >> I did a minimal sweep of the available PCs by going over the possible >> inputs of -pc_type for my application while pinning one DOF. Excepting >> unavailable PCs (not compiled for, other setup missing, ...) and those >> that did break down, I am left with ( hmg jacobi mat none pbjacobi sor >> svd ). >> >> It?s unlikely any of these preconditioners will scale (or even converge) >> for problems with up to 1E6 unknowns. >> I could help you setup https://urldefense.us/v3/__https://epubs.siam.org/doi/abs/10.1137/21M1434891__;!!G_uCfscf7eWS!clIDFE_3D83l3xZqcVa-mF8uwEJjY2YOrDaWN1pqR-3xEwIaMN8TrzIpfjT2HT6x7gRyjPF-mnryOEVYXhmiKj8$ >> >> if you are willing to share a larger example (the current Mat are extremely >> tiny). >> >> Yes, that would be great. About how large of a matrix do you need? I can >> probably quickly get something non-artificial up to O(N) ~ 1e3, >> >> >> That?s big enough. >> If you?re in luck, AMG on the normal equations won?t behave too badly, >> but I?ll try some more robust (in theory) methods nonetheless. >> > > We have also had good luck forming and factoring a block diagonal > approximation of the normal equations. It depends on your problem. What is > this problem? > > Thanks, > > Matt > > >> Thanks, >> Pierre >> >> bigger >> matrices will take some time since I purposefully ignored MPI previously. >> The matrix basically describes the contacts between particles which are >> resolved on a uniform grid, so the main memory hog isn't the matrix but >> rather resolving the particles. >> I should mention that the matrix changes over the course of the >> simulation but stays constant for many solves, i.e. hundreds to >> thousands of solves with variable RHS between periods of contact >> formation/loss. >> >> >> Thanks, >> Pierre >> >> >> >> Best regards, >> Marco >> >> >> >> >> >> Best regards, >> Marco >> >> >> > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!clIDFE_3D83l3xZqcVa-mF8uwEJjY2YOrDaWN1pqR-3xEwIaMN8TrzIpfjT2HT6x7gRyjPF-mnryOEVY1OVU6_4$ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue May 7 07:32:55 2024 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 7 May 2024 08:32:55 -0400 Subject: [petsc-users] Reasons for breakdown in preconditioned LSQR In-Reply-To: References: <30b07a76-9479-4a3a-9cf1-37251be79c07@kit.ac.jp> <31c6214d-c333-4624-ab74-a9ae626b50a7@kit.ac.jp> <7E39F820-0309-46C2-BCC5-C5F21ADCAB4D@joliv.et> Message-ID: On Tue, May 7, 2024 at 8:28?AM Mark Adams wrote: > "A^T A being similar to a finite difference Poisson matrix if the rows > were permuted randomly." > Normal eqs are a good option in general for rectangular systems and we > have Poisson solvers. > I missed that. Why not first permute back to the Poisson matrix? Then it would be trivial. Thanks, Matt > I'm not sure what you mean by "permuted randomly." A random permutation > of the matrix can kill performance but not math. > > Mark > > > On Tue, May 7, 2024 at 8:03?AM Matthew Knepley wrote: > >> On Tue, May 7, 2024 at 5: 12 AM Pierre Jolivet >> wrote: On 7 May 2024, at 9: 10 AM, Marco Seiz >> wrote: Thanks for the quick response! On 07. 05. 24 14: 24, Pierre Jolivet >> wrote: On 7 May 2024, >> ZjQcmQRYFpfptBannerStart >> This Message Is From an External Sender >> This message came from outside your organization. >> >> ZjQcmQRYFpfptBannerEnd >> On Tue, May 7, 2024 at 5:12?AM Pierre Jolivet wrote: >> >>> On 7 May 2024, at 9: 10 AM, Marco Seiz wrote: >>> Thanks for the quick response! On 07. 05. 24 14: 24, Pierre Jolivet wrote: >>> On 7 May 2024, at 7: 04 AM, Marco Seiz wrote: This >>> Message Is From an External >>> ZjQcmQRYFpfptBannerStart >>> This Message Is From an External Sender >>> This message came from outside your organization. >>> >>> ZjQcmQRYFpfptBannerEnd >>> >>> >>> On 7 May 2024, at 9:10?AM, Marco Seiz wrote: >>> >>> Thanks for the quick response! >>> >>> On 07.05.24 14:24, Pierre Jolivet wrote: >>> >>> >>> >>> On 7 May 2024, at 7:04?AM, Marco Seiz wrote: >>> >>> This Message Is From an External Sender >>> This message came from outside your organization. >>> Hello, >>> >>> something a bit different from my last question, since that didn't >>> progress so well: >>> I have a related model which generally produces a rectangular matrix A, >>> so I am using LSQR to solve the system. >>> The matrix A has two nonzeros (1, -1) per row, with A^T A being similar >>> to a finite difference Poisson matrix if the rows were permuted randomly. >>> The problem is singular in that the solution is only specified up to a >>> constant from the matrix, with my target solution being a weighted zero >>> average one, which I can handle by adding a nullspace to my matrix. >>> However, I'd also like to pin (potentially many) DOFs in the future so I >>> also tried pinning a single value, and afterwards subtracting the >>> average from the KSP solution. >>> This leads to the KSP *sometimes* diverging when I use a preconditioner; >>> the target size of the matrix will be something like ([1,20] N) x N, >>> with N ~ [2, 1e6] so for the higher end I will require a preconditioner >>> for reasonable execution time. >>> >>> For a smaller example system, I set up my application to dump the input >>> to the KSP when it breaks down and I've attached a simple python script >>> + data using petsc4py to demonstrate the divergence for those specific >>> systems. >>> With `python3 lsdiv.py -pc_type lu -ksp_converged_reason` that >>> particular system shows breakdown, but if I remove the pinned DOF and >>> add the nullspace (pass -usens) it converges. I did try different PCs >>> but they tend to break down at different steps, e.g. `python3 lsdiv.py >>> -usenormal -qrdiv -pc_type qr -ksp_converged_reason` shows the breakdown >>> for PCQR when I use MatCreateNormal for creating the PC mat, but >>> interestingly it doesn't break down when I explicitly form A^T A (don't >>> pass -usenormal). >>> >>> >>> What version are you using? All those commands are returning >>> Linear solve converged due to CONVERGED_RTOL_NORMAL iterations 1 >>> So I cannot reproduce any breakdown, but there have been recent changes >>> to KSPLSQR. >>> >>> For those tests I've been using PETSc 3.20.5 (last githash was >>> 4b82c11ab5d ). >>> I pulled the latest version from gitlab ( 6b3135e3cbe ) and compiled it, >>> but I had to drop --download-suitesparse=1 from my earlier config due to >>> errors. >>> Should I write a separate mail about this? >>> >>> The LU example still behaves the same for me (`python3 lsdiv.py -pc_type >>> lu -ksp_converged_reason` gives DIVERGED_BREAKDOWN, `python3 lsdiv.py >>> -usens -pc_type lu -ksp_converged_reason` gives CONVERGED_RTOL_NORMAL) >>> but the QR example fails since I had to remove suitesparse. >>> petsc4py.__version__ reports 3.21.1 and if I rebuild my application, >>> then `ldd app` gives me `libpetsc.so >>> .3.21 >>> => >>> /opt/petsc/linux-c-opt/lib/libpetsc.so >>> .3.21` >>> so it should be using the >>> newly built one. >>> The application then still eventually yields a DIVERGED_BREAKDOWN. >>> I don't have a ~/.petscrc and PETSC_OPTIONS is unset, so if we are on >>> the same version and there's still a discrepancy it is quite weird. >>> >>> >>> Quite weird indeed? >>> $ python3 lsdiv.py -pc_type lu -ksp_converged_reason >>> Linear solve converged due to CONVERGED_RTOL_NORMAL iterations 1 >>> $ python3 lsdiv.py -usens -pc_type lu -ksp_converged_reason >>> Linear solve converged due to CONVERGED_RTOL_NORMAL iterations 1 >>> $ python3 lsdiv.py -pc_type qr -ksp_converged_reason >>> Linear solve converged due to CONVERGED_RTOL_NORMAL iterations 1 >>> $ python3 lsdiv.py -usens -pc_type qr -ksp_converged_reason >>> Linear solve converged due to CONVERGED_RTOL_NORMAL iterations 1 >>> >>> For the moment I can work by adding the nullspace but eventually the >>> need for pinning DOFs will resurface, so I'd like to ask where the >>> breakdown is coming from. What causes the breakdowns? Is that a generic >>> problem occurring when adding (dof_i = val) rows to least-squares >>> systems which prevents these preconditioners from being robust? If so, >>> what preconditioners could be robust? >>> I did a minimal sweep of the available PCs by going over the possible >>> inputs of -pc_type for my application while pinning one DOF. Excepting >>> unavailable PCs (not compiled for, other setup missing, ...) and those >>> that did break down, I am left with ( hmg jacobi mat none pbjacobi sor >>> svd ). >>> >>> It?s unlikely any of these preconditioners will scale (or even converge) >>> for problems with up to 1E6 unknowns. >>> I could help you setup https://urldefense.us/v3/__https://epubs.siam.org/doi/abs/10.1137/21M1434891__;!!G_uCfscf7eWS!Yw9Mv8kLlC8eKJnJjK2drMrR2BsrOWhEvxW7gJShAxCS1rsRjmRcgAPRoq5fq9fQtr8Xhz9nIKh-AdPZRchJ$ >>> >>> if you are willing to share a larger example (the current Mat are extremely >>> tiny). >>> >>> Yes, that would be great. About how large of a matrix do you need? I can >>> probably quickly get something non-artificial up to O(N) ~ 1e3, >>> >>> >>> That?s big enough. >>> If you?re in luck, AMG on the normal equations won?t behave too badly, >>> but I?ll try some more robust (in theory) methods nonetheless. >>> >> >> We have also had good luck forming and factoring a block diagonal >> approximation of the normal equations. It depends on your problem. What is >> this problem? >> >> Thanks, >> >> Matt >> >> >>> Thanks, >>> Pierre >>> >>> bigger >>> matrices will take some time since I purposefully ignored MPI previously. >>> The matrix basically describes the contacts between particles which are >>> resolved on a uniform grid, so the main memory hog isn't the matrix but >>> rather resolving the particles. >>> I should mention that the matrix changes over the course of the >>> simulation but stays constant for many solves, i.e. hundreds to >>> thousands of solves with variable RHS between periods of contact >>> formation/loss. >>> >>> >>> Thanks, >>> Pierre >>> >>> >>> >>> Best regards, >>> Marco >>> >>> >>> >>> >>> >>> Best regards, >>> Marco >>> >>> >>> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!Yw9Mv8kLlC8eKJnJjK2drMrR2BsrOWhEvxW7gJShAxCS1rsRjmRcgAPRoq5fq9fQtr8Xhz9nIKh-AQY4i0p0$ >> >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!Yw9Mv8kLlC8eKJnJjK2drMrR2BsrOWhEvxW7gJShAxCS1rsRjmRcgAPRoq5fq9fQtr8Xhz9nIKh-AQY4i0p0$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From marco at kit.ac.jp Tue May 7 07:50:38 2024 From: marco at kit.ac.jp (=?UTF-8?B?IlNlaXosTWFyY28i?=) Date: Tue, 7 May 2024 21:50:38 +0900 Subject: [petsc-users] Reasons for breakdown in preconditioned LSQR In-Reply-To: <7E39F820-0309-46C2-BCC5-C5F21ADCAB4D@joliv.et> References: <30b07a76-9479-4a3a-9cf1-37251be79c07@kit.ac.jp> <31c6214d-c333-4624-ab74-a9ae626b50a7@kit.ac.jp> <7E39F820-0309-46C2-BCC5-C5F21ADCAB4D@joliv.et> Message-ID: <171508623875175.1449297990@sys11mail03.cis.kit.ac.jp> An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: system.zip Type: application/zip Size: 38886 bytes Desc: not available URL: From marco at kit.ac.jp Tue May 7 08:51:25 2024 From: marco at kit.ac.jp (=?UTF-8?B?IlNlaXosTWFyY28i?=) Date: Tue, 7 May 2024 22:51:25 +0900 Subject: [petsc-users] Reasons for breakdown in preconditioned LSQR In-Reply-To: References: <30b07a76-9479-4a3a-9cf1-37251be79c07@kit.ac.jp> <31c6214d-c333-4624-ab74-a9ae626b50a7@kit.ac.jp> <7E39F820-0309-46C2-BCC5-C5F21ADCAB4D@joliv.et> Message-ID: <1715089885448499.1982548869@sys11mail03.cis.kit.ac.jp> An HTML attachment was scrubbed... URL: From pierre at joliv.et Tue May 7 09:29:23 2024 From: pierre at joliv.et (Pierre Jolivet) Date: Tue, 7 May 2024 16:29:23 +0200 Subject: [petsc-users] Reasons for breakdown in preconditioned LSQR In-Reply-To: <171508623875175.1449297990@sys11mail03.cis.kit.ac.jp> References: <30b07a76-9479-4a3a-9cf1-37251be79c07@kit.ac.jp> <31c6214d-c333-4624-ab74-a9ae626b50a7@kit.ac.jp> <7E39F820-0309-46C2-BCC5-C5F21ADCAB4D@joliv.et> <171508623875175.1449297990@sys11mail03.cis.kit.ac.jp> Message-ID: <7AF1F3F7-C5C6-4960-96B4-53122CEEF830@joliv.et> OK, it?s very trivial to solve with algebraic solvers. If you are willing to share larger test cases, maybe then some issue will arise (please do not attach them to your mail, send a URL, if you cannot and must attach it to your mail, switch to the following mailing list: petsc-maint at mcs.anl.gov) Thanks, Pierre $ mpirun -n 8 ./sparse_ls -mat_name /tmp/system/matdump_step00010000_dir0.bin -options_file gmres.rc -pc_type hypre -options_left 0 0 KSP Residual norm 3.453633143461e+01 1 KSP Residual norm 1.454005637026e+00 2 KSP Residual norm 7.254564223178e-02 3 KSP Residual norm 2.595967767025e-03 4 KSP Residual norm 8.616844003887e-05 5 KSP Residual norm 2.519788767995e-06 6 KSP Residual norm 6.830984907384e-08 Linear solve converged due to CONVERGED_RTOL iterations 6 ||A^T(Ax-b)|| / ||Ax-b|| = 0.000000 / 17.017700 = 0.000000 $ mpirun -n 8 ./sparse_ls -mat_name /tmp/system/matdump_step00010000_dir0.bin -options_file gmres.rc -pc_type gamg -options_left 0 0 KSP Residual norm 3.453633143461e+01 1 KSP Residual norm 9.410331701775e+00 2 KSP Residual norm 7.249670993944e-01 3 KSP Residual norm 8.415995591379e-02 4 KSP Residual norm 8.230048116783e-03 5 KSP Residual norm 8.200299281980e-04 6 KSP Residual norm 8.332302028270e-05 7 KSP Residual norm 9.103576883087e-06 8 KSP Residual norm 8.837914581158e-07 9 KSP Residual norm 8.295880033739e-08 Linear solve converged due to CONVERGED_RTOL iterations 9 ||A^T(Ax-b)|| / ||Ax-b|| = 0.000000 / 17.017700 = 0.000000 $ mpirun -n 8 ./sparse_ls -mat_name /tmp/system/matdump_step00010000_dir0.bin -options_file gmres.rc -pc_type hpddm -options_left 0 0 KSP Residual norm 3.453633143461e+01 1 KSP Residual norm 4.455259710220e+00 2 KSP Residual norm 1.190904699181e-01 3 KSP Residual norm 5.220970280301e-03 4 KSP Residual norm 6.214389182723e-05 5 KSP Residual norm 1.693927074022e-06 6 KSP Residual norm 3.700055659502e-08 Linear solve converged due to CONVERGED_RTOL iterations 6 ||A^T(Ax-b)|| / ||Ax-b|| = 0.000000 / 17.017700 = 0.000000 > On 7 May 2024, at 2:50?PM, Seiz,Marco wrote: > > This Message Is From an External Sender > This message came from outside your organization. > Pierre, > > I've attached the dumps of the matrix + RHS for something of about 3k x 1k. > > > Regarding the weird divergence behaviour, I tried again at home but I still get the same results. > > I am running a rolling release distribution on both machines, but that really shouldn't matter for divergence behavior I would think. > > Is there some kind of option in PETSc to get more information about the breakdown from my side? > > > Best regards, > > Marco > > ----- Original Message ----- > >> From: Pierre Jolivet > > >> To: Marco Seiz > > >> Cc: petsc-users at mcs.anl.gov > >> Date: 2024-05-07 18:12:18 > >> Subject: Re: [petsc-users] Reasons for breakdown in preconditioned LSQR > >> > > >> > >> > On 7 May 2024, at 9:10?AM, Marco Seiz > wrote: > >> > > >> > Thanks for the quick response! > >> > > >> > On 07.05.24 14:24, Pierre Jolivet wrote: > >> >> > >> >> > >> >>> On 7 May 2024, at 7:04?AM, Marco Seiz > wrote: > >> >>> > >> >>> This Message Is From an External Sender > >> >>> This message came from outside your organization. > >> >>> Hello, > >> >>> > >> >>> something a bit different from my last question, since that didn't > >> >>> progress so well: > >> >>> I have a related model which generally produces a rectangular matrix A, > >> >>> so I am using LSQR to solve the system. > >> >>> The matrix A has two nonzeros (1, -1) per row, with A^T A being similar > >> >>> to a finite difference Poisson matrix if the rows were permuted randomly. > >> >>> The problem is singular in that the solution is only specified up to a > >> >>> constant from the matrix, with my target solution being a weighted zero > >> >>> average one, which I can handle by adding a nullspace to my matrix. > >> >>> However, I'd also like to pin (potentially many) DOFs in the future so I > >> >>> also tried pinning a single value, and afterwards subtracting the > >> >>> average from the KSP solution. > >> >>> This leads to the KSP *sometimes* diverging when I use a preconditioner; > >> >>> the target size of the matrix will be something like ([1,20] N) x N, > >> >>> with N ~ [2, 1e6] so for the higher end I will require a preconditioner > >> >>> for reasonable execution time. > >> >>> > >> >>> For a smaller example system, I set up my application to dump the input > >> >>> to the KSP when it breaks down and I've attached a simple python script > >> >>> + data using petsc4py to demonstrate the divergence for those specific > >> >>> systems. > >> >>> With `python3 lsdiv.py -pc_type lu -ksp_converged_reason` that > >> >>> particular system shows breakdown, but if I remove the pinned DOF and > >> >>> add the nullspace (pass -usens) it converges. I did try different PCs > >> >>> but they tend to break down at different steps, e.g. `python3 lsdiv.py > >> >>> -usenormal -qrdiv -pc_type qr -ksp_converged_reason` shows the breakdown > >> >>> for PCQR when I use MatCreateNormal for creating the PC mat, but > >> >>> interestingly it doesn't break down when I explicitly form A^T A (don't > >> >>> pass -usenormal). > >> >> > >> >> What version are you using? All those commands are returning > >> >> Linear solve converged due to CONVERGED_RTOL_NORMAL iterations 1 > >> >> So I cannot reproduce any breakdown, but there have been recent changes to KSPLSQR. > >> > For those tests I've been using PETSc 3.20.5 (last githash was > >> > 4b82c11ab5d ). > >> > I pulled the latest version from gitlab ( 6b3135e3cbe ) and compiled it, > >> > but I had to drop --download-suitesparse=1 from my earlier config due to > >> > errors. > >> > Should I write a separate mail about this? > >> > > >> > The LU example still behaves the same for me (`python3 lsdiv.py -pc_type > >> > lu -ksp_converged_reason` gives DIVERGED_BREAKDOWN, `python3 lsdiv.py > >> > -usens -pc_type lu -ksp_converged_reason` gives CONVERGED_RTOL_NORMAL) > >> > but the QR example fails since I had to remove suitesparse. > >> > petsc4py.__version__ reports 3.21.1 and if I rebuild my application, > >> > then `ldd app` gives me `libpetsc.so .3.21 => > >> > /opt/petsc/linux-c-opt/lib/libpetsc.so .3.21` so it should be using the > >> > newly built one. > >> > The application then still eventually yields a DIVERGED_BREAKDOWN. > >> > I don't have a ~/.petscrc and PETSC_OPTIONS is unset, so if we are on > >> > the same version and there's still a discrepancy it is quite weird. > >> > >> Quite weird indeed? > >> $ python3 lsdiv.py -pc_type lu -ksp_converged_reason > >> Linear solve converged due to CONVERGED_RTOL_NORMAL iterations 1 > >> $ python3 lsdiv.py -usens -pc_type lu -ksp_converged_reason > >> Linear solve converged due to CONVERGED_RTOL_NORMAL iterations 1 > >> $ python3 lsdiv.py -pc_type qr -ksp_converged_reason > >> Linear solve converged due to CONVERGED_RTOL_NORMAL iterations 1 > >> $ python3 lsdiv.py -usens -pc_type qr -ksp_converged_reason > >> Linear solve converged due to CONVERGED_RTOL_NORMAL iterations 1 > >> > >> >>> For the moment I can work by adding the nullspace but eventually the > >> >>> need for pinning DOFs will resurface, so I'd like to ask where the > >> >>> breakdown is coming from. What causes the breakdowns? Is that a generic > >> >>> problem occurring when adding (dof_i = val) rows to least-squares > >> >>> systems which prevents these preconditioners from being robust? If so, > >> >>> what preconditioners could be robust? > >> >>> I did a minimal sweep of the available PCs by going over the possible > >> >>> inputs of -pc_type for my application while pinning one DOF. Excepting > >> >>> unavailable PCs (not compiled for, other setup missing, ...) and those > >> >>> that did break down, I am left with ( hmg jacobi mat none pbjacobi sor > >> >>> svd ). > >> >> It?s unlikely any of these preconditioners will scale (or even converge) for problems with up to 1E6 unknowns. > >> >> I could help you setup https://urldefense.us/v3/__https://epubs.siam.org/doi/abs/10.1137/21M1434891__;!!G_uCfscf7eWS!fW1baXZMAQIKi0VDUIDUUzpMi4xQf7jrWGCXPlpIllqKAXJBzDClVwrLKYuWuT7LYfZoDzK4g9I9g_z0Iwv7Sg$ if you are willing to share a larger example (the current Mat are extremely tiny). > >> > Yes, that would be great. About how large of a matrix do you need? I can > >> > probably quickly get something non-artificial up to O(N) ~ 1e3, > >> > >> That?s big enough. > >> If you?re in luck, AMG on the normal equations won?t behave too badly, but I?ll try some more robust (in theory) methods nonetheless. > >> > >> Thanks, > >> Pierre > >> > >> > bigger > >> > matrices will take some time since I purposefully ignored MPI previously. > >> > The matrix basically describes the contacts between particles which are > >> > resolved on a uniform grid, so the main memory hog isn't the matrix but > >> > rather resolving the particles. > >> > I should mention that the matrix changes over the course of the > >> > simulation but stays constant for many solves, i.e. hundreds to > >> > thousands of solves with variable RHS between periods of contact > >> > formation/loss. > >> > > >> >> > >> >> Thanks, > >> >> Pierre > >> >>> > >> >>> > >> >>> Best regards, > >> >>> Marco > >> >>> > >> >>> > >> >> > >> >> > >> > Best regards, > >> > Marco > >> > >> > >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sawsan.shatanawi at wsu.edu Tue May 7 13:22:43 2024 From: sawsan.shatanawi at wsu.edu (Shatanawi, Sawsan Muhammad) Date: Tue, 7 May 2024 18:22:43 +0000 Subject: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code In-Reply-To: References: <17B6144E-4F3A-4173-9818-4B03669736C4@petsc.dev> <9BD74F88-20FD-4021-B02A-195A58B72282@petsc.dev> Message-ID: Hello everyone, I hope this email finds you well. My Name is Sawsan Shatanawi, and I was developing a Fortran code for simulating groundwater flow in a 3D system with nonlinear behavior. I solved the nonlinear system using the PCG solver and Picard iteration, but I did not get good results although I checked my matrix and RHS and everything, I decided to change my solver to Newton Rapson method. I checked PETSc documents but I have a few questions: 1) My groundwater system is time-dependent, so should I use TS only instead of SNES? 2) My system has its deltaT, would using deltaT as dt affect my solver, or is it better to use TS-PETSc dt? Also, would using PETSc dt affect the simulation of the groundwater system 3) I want my Jacobian matrix to be calculated by PETSc automatically 4) Do I need to define and calculate the residual vector? My A-Matrix contains coefficients and external sources and my RHS vector includes the boundary conditions Please find the attached file contains a draft of my code Thank you in advance for your time and help. Best regards, Sawsan ________________________________ From: Shatanawi, Sawsan Muhammad Sent: Tuesday, January 16, 2024 10:43 AM To: Junchao Zhang Cc: Barry Smith ; Matthew Knepley ; Mark Adams ; petsc-users at mcs.anl.gov Subject: Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code Hello all, Thank you for your valuable help. I will do your recommendations and hope it will run without any issues. Bests, Sawsan ________________________________ From: Junchao Zhang Sent: Friday, January 12, 2024 8:46 AM To: Shatanawi, Sawsan Muhammad Cc: Barry Smith ; Matthew Knepley ; Mark Adams ; petsc-users at mcs.anl.gov Subject: Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code [EXTERNAL EMAIL] Hi, Sawsan, First in test_main.F90, you need to call VecGetArrayF90(temp_solution, H_vector, ierr) and VecRestoreArrayF90 (temp_solution, H_vector, ierr) as Barry mentioned. Secondly, in the loop of test_main.F90, it calls GW_solver(). Within it, it calls PetscInitialize()/PetscFinalize(). But without MPI being initialized, PetscInitialize()/PetscFinalize() can only be called once. do timestep =2 , NTSP call GW_boundary_conditions(timestep-1) !print *,HNEW(1,1,1) call GW_elevation() ! print *, GWTOP(2,2,2) call GW_conductance() ! print *, CC(2,2,2) call GW_recharge() ! print *, B_Rech(5,4) call GW_pumping(timestep-1) ! print *, B_pump(2,2,2) call GW_SW(timestep-1) print *,B_RIVER (2,2,2) call GW_solver(timestep-1,N) call GW_deallocate_loop() end do A solution is to delete PetscInitialize()/PetscFinalize() in GW_solver_try.F90 and move it to test_main.F90, outside the do loop. diff --git a/test_main.F90 b/test_main.F90 index b5997c55..107bd3ee 100644 --- a/test_main.F90 +++ b/test_main.F90 @@ -1,5 +1,6 @@ program test_GW +#include use petsc use GW_constants use GW_param_by_user @@ -8,6 +9,9 @@ program test_GW implicit none integer :: N integer :: timestep + PetscErrorCode ierr + + call PetscInitialize(ierr) call GW_domain(N) !print *, "N=",N !print *, DELTAT @@ -37,4 +41,5 @@ program test_GW end do print *, HNEW(NCOL,3,2) call GW_deallocate () + call PetscFinalize(ierr) end program test_GW With that, the MPI error will be fixed. The code could run to gw_deallocate () before abort. There are other memory errors. You can install/use valgrind to fix them. Run it with valgrind ./GW.exe and look through the output Thanks. --Junchao Zhang On Thu, Jan 11, 2024 at 10:49?PM Shatanawi, Sawsan Muhammad > wrote: Hello, Thank you all for your help. I have changed VecGetArray to VecGetArrayF90, and the location of destory call. but I want to make sure that VecGet ArrayF90 is to make a new array( vector) that I can use in the rest of my Fortran code? when I run it and debugged it, I got 5.2000000E-03 50.00000 10.00000 0.0000000E+00 PETSC: Attaching gdb to /weka/data/lab/richey/sawsan/GW_CODE/code2024/SS_GWM/./GW.exe of pid 33065 on display :0.0 on machine sn16 Unable to start debugger in xterm: No such file or directory 0.0000000E+00 Attempting to use an MPI routine after finalizing MPICH srun: error: sn16: task 0: Exited with exit code 1 [sawsan.shatanawi at login-p2n02 SS_GWM]$ gdb ./GW/exe GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-100.el7 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later > This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". For bug reporting instructions, please see: >... ./GW/exe: No such file or directory. (gdb) run Starting program: No executable file specified. Use the "file" or "exec-file" command. (gdb) bt No stack. (gdb) If the highlighted line is the error, I don't know why when I write gdb , it does not show me the location of error The code : sshatanawi/SS_GWM (github.com) I really appreciate your helps Sawsan ________________________________ From: Barry Smith > Sent: Wednesday, January 10, 2024 5:35 PM To: Junchao Zhang > Cc: Shatanawi, Sawsan Muhammad >; Mark Adams >; petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code [EXTERNAL EMAIL] On Jan 10, 2024, at 6:49?PM, Junchao Zhang > wrote: Hi, Sawsan, I could build your code and I also could gdb it. $ gdb ./GW.exe ... $ Thread 1 "GW.exe" received signal SIGSEGV, Segmentation fault. 0x00007ffff1e6d44f in vecgetarray_ (x=0x7fffffffa718, fa=0x0, ia=0x7fffffffa75c, ierr=0x0) at /scratch/jczhang/petsc/src/vec/vec/interface/ftn-custom/zvectorf.c:257 257 *ierr = VecGetArray(*x, &lx); (gdb) bt #0 0x00007ffff1e6d44f in vecgetarray_ (x=0x7fffffffa718, fa=0x0, ia=0x7fffffffa75c, ierr=0x0) at /scratch/jczhang/petsc/src/vec/vec/interface/ftn-custom/zvectorf.c:257 #1 0x000000000040b6e3 in gw_solver (t_s=1.40129846e-45, n=300) at GW_solver_try.F90:169 #2 0x000000000040c6a8 in test_gw () at test_main.F90:35 ierr=0x0 caused the segfault. See https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/VecGetArray/*vecgetarray__;Iw!!G_uCfscf7eWS!aDPVIaXxxRNvtadvU_F4xbshLyPmphwU9eHojKr-5XWWpGF7rqbuu-EKWjZxPfCL5B090Aolufvvd329KjQF3rYmFLWZhBG_dA$ , you should use VecGetArrayF90 instead. BTW, Barry, the code https://urldefense.us/v3/__https://github.com/sshatanawi/SS_GWM/blob/main/GW_solver_try.F90*L169__;Iw!!G_uCfscf7eWS!aDPVIaXxxRNvtadvU_F4xbshLyPmphwU9eHojKr-5XWWpGF7rqbuu-EKWjZxPfCL5B090Aolufvvd329KjQF3rYmFLWGbsJMyg$ has "call VecGetArray(temp_solution, H_vector, ierr)". I don't find petsc Fortran examples doing VecGetArray. Do we still support it? This is not the correct calling sequence for VecGetArray() from Fortran. Regardless, definitely should not be writing any new code that uses VecGetArray() from Fortran. Should use VecGetArrayF90(). --Junchao Zhang On Wed, Jan 10, 2024 at 2:38?PM Shatanawi, Sawsan Muhammad via petsc-users > wrote: Hello all, I hope you are doing well. Generally, I use gdb to debug the code. I got the attached error message. I have tried to add the flag -start_in_debugger in the make file, but it didn't work, so it seems I was doing it in the wrong way This is the link for the whole code: sshatanawi/SS_GWM (github.com) [https://urldefense.us/v3/__https://opengraph.githubassets.com/9eb6cd14baf12f04848ed209b6f502415eb531bdd7b3a5f9696af68663b870c0/sshatanawi/SS_GWM__;!!G_uCfscf7eWS!aDPVIaXxxRNvtadvU_F4xbshLyPmphwU9eHojKr-5XWWpGF7rqbuu-EKWjZxPfCL5B090Aolufvvd329KjQF3rYmFLVukOhM2Q$ ] GitHub - sshatanawi/SS_GWM Contribute to sshatanawi/SS_GWM development by creating an account on GitHub. github.com ? You can read the description of the code in " Model Desprciption.pdf" the compiling file is makefile_f90 where you can find the linked code files I really appreciate your help Bests, Sawsan ________________________________ From: Mark Adams > Sent: Friday, January 5, 2024 4:53 AM To: Shatanawi, Sawsan Muhammad > Cc: Matthew Knepley >; petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code [EXTERNAL EMAIL] This is a segv. As Matt said, you need to use a debugger for this or add print statements to narrow down the place where this happens. You will need to learn how to use debuggers to do your project so you might as well start now. If you have a machine with a GUI debugger that is easier but command line debuggers are good to learn anyway. I tend to run debuggers directly (eg, lldb ./a.out -- program-args ...) and use a GUI debugger (eg, Totalview or DDT) if available. Mark On Wed, Dec 20, 2023 at 10:02?PM Shatanawi, Sawsan Muhammad via petsc-users > wrote: Hello Matthew, Thank you for your help. I am sorry that I keep coming back with my error messages, but I reached a point that I don't know how to fix them, and I don't understand them easily. The list of errors is getting shorter, now I am getting the attached error messages Thank you again, Sawsan ________________________________ From: Matthew Knepley > Sent: Wednesday, December 20, 2023 6:54 PM To: Shatanawi, Sawsan Muhammad > Cc: Barry Smith >; petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code [EXTERNAL EMAIL] On Wed, Dec 20, 2023 at 9:49?PM Shatanawi, Sawsan Muhammad via petsc-users > wrote: Hello Barry, Thank you a lot for your help, Now I am getting the attached error message. Do not destroy the PC from KSPGetPC() THanks, Matt Bests, Sawsan ________________________________ From: Barry Smith > Sent: Wednesday, December 20, 2023 6:32 PM To: Shatanawi, Sawsan Muhammad > Cc: Mark Adams >; petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code [EXTERNAL EMAIL] Instead of call PCCreate(PETSC_COMM_WORLD, pc, ierr) call PCSetType(pc, PCILU,ierr) ! Choose a preconditioner type (ILU) call KSPSetPC(ksp, pc,ierr) ! Associate the preconditioner with the KSP solver do call KSPGetPC(ksp,pc,ierr) call PCSetType(pc, PCILU,ierr) Do not call KSPSetUp(). It will be taken care of automatically during the solve On Dec 20, 2023, at 8:52?PM, Shatanawi, Sawsan Muhammad via petsc-users > wrote: Hello, I don't think that I set preallocation values when I created the matrix, would you please have look at my code. It is just the petsc related part from my code. I was able to fix some of the error messages. Now I have a new set of error messages related to the KSP solver (attached) I appreciate your help Sawsan ________________________________ From: Mark Adams > Sent: Wednesday, December 20, 2023 6:44 AM To: Shatanawi, Sawsan Muhammad > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code [EXTERNAL EMAIL] Did you set preallocation values when you created the matrix? Don't do that. On Wed, Dec 20, 2023 at 9:36?AM Shatanawi, Sawsan Muhammad > wrote: Hello, I am trying to create a sparse matrix( which is as I believe a zero matrix) then adding some nonzero elements to it over a loop, then assembling it Get Outlook for iOS ________________________________ From: Mark Adams > Sent: Wednesday, December 20, 2023 2:48 AM To: Shatanawi, Sawsan Muhammad > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code [EXTERNAL EMAIL] I am guessing that you are creating a matrix, adding to it, finalizing it ("assembly"), and then adding to it again, which is fine, but you are adding new non-zeros to the sparsity pattern. If this is what you want then you can tell the matrix to let you do that. Otherwise you have a bug. Mark On Tue, Dec 19, 2023 at 9:50?PM Shatanawi, Sawsan Muhammad via petsc-users > wrote: Hello everyone, I hope this email finds you well. My Name is Sawsan Shatanawi, and I am currently working on developing a Fortran code for simulating groundwater flow in a 3D system. The code involves solving a nonlinear system, and I have created the matrix to be solved using the PCG solver and Picard iteration. However, when I tried to assign it as a PETSc matrix I started getting a lot of error messages. I am kindly asking if someone can help me, I would be happy to share my code with him/her. Please find the attached file contains a list of errors I have gotten Thank you in advance for your time and assistance. Best regards, Sawsan -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!aDPVIaXxxRNvtadvU_F4xbshLyPmphwU9eHojKr-5XWWpGF7rqbuu-EKWjZxPfCL5B090Aolufvvd329KjQF3rYmFLX0Bmayww$ -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: Newton_PETSc.txt URL: From bsmith at petsc.dev Tue May 7 13:28:20 2024 From: bsmith at petsc.dev (Barry Smith) Date: Tue, 7 May 2024 14:28:20 -0400 Subject: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code In-Reply-To: References: <17B6144E-4F3A-4173-9818-4B03669736C4@petsc.dev> <9BD74F88-20FD-4021-B02A-195A58B72282@petsc.dev> Message-ID: <01ABDCDE-D636-4859-A501-9C8528BBF981@petsc.dev> Have you considered https://urldefense.us/v3/__https://www.pflotran.org/documentation/user_guide/how_to/installation/installation.html__;!!G_uCfscf7eWS!eZkBlApyezAyEBaDxGemEyUmTe87omvkv59rrf0Mq8L4bOTqEL9wynuMJ1ci9kRDOqucZBiYdGT2tRTVuCSJdSs$ > On May 7, 2024, at 2:22?PM, Shatanawi, Sawsan Muhammad via petsc-users wrote: > > This Message Is From an External Sender > This message came from outside your organization. > Hello everyone, > > I hope this email finds you well. > > > My Name is Sawsan Shatanawi, and I was developing a Fortran code for simulating groundwater flow in a 3D system with nonlinear behavior. I solved the nonlinear system using the PCG solver and Picard iteration, but I did not get good results although I checked my matrix and RHS and everything, I decided to change my solver to Newton Rapson method. > I checked PETSc documents but I have a few questions: > 1) My groundwater system is time-dependent, so should I use TS only instead of SNES? > 2) My system has its deltaT, would using deltaT as dt affect my solver, or is it better to use TS-PETSc dt? Also, would using PETSc dt affect the simulation of the groundwater system > 3) I want my Jacobian matrix to be calculated by PETSc automatically > 4) Do I need to define and calculate the residual vector? > > My A-Matrix contains coefficients and external sources and my RHS vector includes the boundary conditions > > > Please find the attached file contains a draft of my code > > Thank you in advance for your time and help. > > Best regards, > > Sawsan > > > From: Shatanawi, Sawsan Muhammad > > Sent: Tuesday, January 16, 2024 10:43 AM > To: Junchao Zhang > > Cc: Barry Smith >; Matthew Knepley >; Mark Adams >; petsc-users at mcs.anl.gov > > Subject: Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code > > Hello all, > > Thank you for your valuable help. I will do your recommendations and hope it will run without any issues. > > Bests, > Sawsan > > From: Junchao Zhang > > Sent: Friday, January 12, 2024 8:46 AM > To: Shatanawi, Sawsan Muhammad > > Cc: Barry Smith >; Matthew Knepley >; Mark Adams >; petsc-users at mcs.anl.gov > > Subject: Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code > > [EXTERNAL EMAIL] > Hi, Sawsan, > First in test_main.F90, you need to call VecGetArrayF90(temp_solution, H_vector, ierr) and VecRestoreArrayF90 (temp_solution, H_vector, ierr) as Barry mentioned. > Secondly, in the loop of test_main.F90, it calls GW_solver(). Within it, it calls PetscInitialize()/PetscFinalize(). But without MPI being initialized, PetscInitialize()/PetscFinalize() can only be called once. > do timestep =2 , NTSP > call GW_boundary_conditions(timestep-1) > !print *,HNEW(1,1,1) > call GW_elevation() > ! print *, GWTOP(2,2,2) > call GW_conductance() > ! print *, CC(2,2,2) > call GW_recharge() > ! print *, B_Rech(5,4) > call GW_pumping(timestep-1) > ! print *, B_pump(2,2,2) > call GW_SW(timestep-1) > print *,B_RIVER (2,2,2) > call GW_solver(timestep-1,N) > call GW_deallocate_loop() > end do > > A solution is to delete PetscInitialize()/PetscFinalize() in GW_solver_try.F90 and move it to test_main.F90, outside the do loop. > > diff --git a/test_main.F90 b/test_main.F90 > index b5997c55..107bd3ee 100644 > --- a/test_main.F90 > +++ b/test_main.F90 > @@ -1,5 +1,6 @@ > program test_GW > > +#include > use petsc > use GW_constants > use GW_param_by_user > @@ -8,6 +9,9 @@ program test_GW > implicit none > integer :: N > integer :: timestep > + PetscErrorCode ierr > + > + call PetscInitialize(ierr) > call GW_domain(N) > !print *, "N=",N > !print *, DELTAT > @@ -37,4 +41,5 @@ program test_GW > end do > print *, HNEW(NCOL,3,2) > call GW_deallocate () > + call PetscFinalize(ierr) > end program test_GW > > With that, the MPI error will be fixed. The code could run to gw_deallocate () before abort. There are other memory errors. You can install/use valgrind to fix them. Run it with valgrind ./GW.exe and look through the output > > > Thanks. > --Junchao Zhang > > > On Thu, Jan 11, 2024 at 10:49?PM Shatanawi, Sawsan Muhammad > wrote: > Hello, > > Thank you all for your help. > > I have changed VecGetArray to VecGetArrayF90, and the location of destory call. but I want to make sure that VecGet ArrayF90 is to make a new array( vector) that I can use in the rest of my Fortran code? > > when I run it and debugged it, I got > > 5.2000000E-03 > 50.00000 > 10.00000 > 0.0000000E+00 > PETSC: Attaching gdb to /weka/data/lab/richey/sawsan/GW_CODE/code2024/SS_GWM/./GW.exe of pid 33065 on display :0.0 on machine sn16 > Unable to start debugger in xterm: No such file or directory > 0.0000000E+00 > Attempting to use an MPI routine after finalizing MPICH > srun: error: sn16: task 0: Exited with exit code 1 > [sawsan.shatanawi at login-p2n02 SS_GWM]$ gdb ./GW/exe > GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-100.el7 > Copyright (C) 2013 Free Software Foundation, Inc. > License GPLv3+: GNU GPL version 3 or later > > This is free software: you are free to change and redistribute it. > There is NO WARRANTY, to the extent permitted by law. Type "show copying" > and "show warranty" for details. > This GDB was configured as "x86_64-redhat-linux-gnu". > For bug reporting instructions, please see: > >... > ./GW/exe: No such file or directory. > (gdb) run > Starting program: > No executable file specified. > Use the "file" or "exec-file" command. > (gdb) bt > No stack. > (gdb) > > If the highlighted line is the error, I don't know why when I write gdb , it does not show me the location of error > The code : sshatanawi/SS_GWM (github.com) > > I really appreciate your helps > > Sawsan > From: Barry Smith > > Sent: Wednesday, January 10, 2024 5:35 PM > To: Junchao Zhang > > Cc: Shatanawi, Sawsan Muhammad >; Mark Adams >; petsc-users at mcs.anl.gov > > Subject: Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code > > [EXTERNAL EMAIL] > > >> On Jan 10, 2024, at 6:49?PM, Junchao Zhang > wrote: >> >> Hi, Sawsan, >> I could build your code and I also could gdb it. >> >> $ gdb ./GW.exe >> ... >> $ Thread 1 "GW.exe" received signal SIGSEGV, Segmentation fault. >> 0x00007ffff1e6d44f in vecgetarray_ (x=0x7fffffffa718, fa=0x0, ia=0x7fffffffa75c, ierr=0x0) at /scratch/jczhang/petsc/src/vec/vec/interface/ftn-custom/zvectorf.c:257 >> 257 *ierr = VecGetArray(*x, &lx); >> (gdb) bt >> #0 0x00007ffff1e6d44f in vecgetarray_ (x=0x7fffffffa718, fa=0x0, ia=0x7fffffffa75c, ierr=0x0) at /scratch/jczhang/petsc/src/vec/vec/interface/ftn-custom/zvectorf.c:257 >> #1 0x000000000040b6e3 in gw_solver (t_s=1.40129846e-45, n=300) at GW_solver_try.F90:169 >> #2 0x000000000040c6a8 in test_gw () at test_main.F90:35 >> >> ierr=0x0 caused the segfault. See https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/VecGetArray/*vecgetarray__;Iw!!G_uCfscf7eWS!eZkBlApyezAyEBaDxGemEyUmTe87omvkv59rrf0Mq8L4bOTqEL9wynuMJ1ci9kRDOqucZBiYdGT2tRTV25yWFSQ$ , you should use VecGetArrayF90 instead. >> >> BTW, Barry, the code https://urldefense.us/v3/__https://github.com/sshatanawi/SS_GWM/blob/main/GW_solver_try.F90*L169__;Iw!!G_uCfscf7eWS!eZkBlApyezAyEBaDxGemEyUmTe87omvkv59rrf0Mq8L4bOTqEL9wynuMJ1ci9kRDOqucZBiYdGT2tRTVocqNAbg$ has "call VecGetArray(temp_solution, H_vector, ierr)". I don't find petsc Fortran examples doing VecGetArray. Do we still support it? > > This is not the correct calling sequence for VecGetArray() from Fortran. > > Regardless, definitely should not be writing any new code that uses VecGetArray() from Fortran. Should use VecGetArrayF90(). > >> >> --Junchao Zhang >> >> >> On Wed, Jan 10, 2024 at 2:38?PM Shatanawi, Sawsan Muhammad via petsc-users > wrote: >> Hello all, >> >> I hope you are doing well. >> >> Generally, I use gdb to debug the code. >> I got the attached error message. >> >> I have tried to add the flag -start_in_debugger in the make file, but it didn't work, so it seems I was doing it in the wrong way >> >> This is the link for the whole code: sshatanawi/SS_GWM (github.com) >> >> GitHub - sshatanawi/SS_GWM >> Contribute to sshatanawi/SS_GWM development by creating an account on GitHub. >> github.com >> ? >> >> You can read the description of the code in " Model Desprciption.pdf" >> the compiling file is makefile_f90 where you can find the linked code files >> >> I really appreciate your help >> >> Bests, >> Sawsan >> From: Mark Adams > >> Sent: Friday, January 5, 2024 4:53 AM >> To: Shatanawi, Sawsan Muhammad > >> Cc: Matthew Knepley >; petsc-users at mcs.anl.gov > >> Subject: Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code >> >> [EXTERNAL EMAIL] >> This is a segv. As Matt said, you need to use a debugger for this or add print statements to narrow down the place where this happens. >> >> You will need to learn how to use debuggers to do your project so you might as well start now. >> >> If you have a machine with a GUI debugger that is easier but command line debuggers are good to learn anyway. >> >> I tend to run debuggers directly (eg, lldb ./a.out -- program-args ...) and use a GUI debugger (eg, Totalview or DDT) if available. >> >> Mark >> >> >> On Wed, Dec 20, 2023 at 10:02?PM Shatanawi, Sawsan Muhammad via petsc-users > wrote: >> Hello Matthew, >> >> Thank you for your help. I am sorry that I keep coming back with my error messages, but I reached a point that I don't know how to fix them, and I don't understand them easily. >> The list of errors is getting shorter, now I am getting the attached error messages >> >> Thank you again, >> >> Sawsan >> From: Matthew Knepley > >> Sent: Wednesday, December 20, 2023 6:54 PM >> To: Shatanawi, Sawsan Muhammad > >> Cc: Barry Smith >; petsc-users at mcs.anl.gov > >> Subject: Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code >> >> [EXTERNAL EMAIL] >> On Wed, Dec 20, 2023 at 9:49?PM Shatanawi, Sawsan Muhammad via petsc-users > wrote: >> Hello Barry, >> >> Thank you a lot for your help, Now I am getting the attached error message. >> >> Do not destroy the PC from KSPGetPC() >> >> THanks, >> >> Matt >> >> Bests, >> Sawsan >> From: Barry Smith > >> Sent: Wednesday, December 20, 2023 6:32 PM >> To: Shatanawi, Sawsan Muhammad > >> Cc: Mark Adams >; petsc-users at mcs.anl.gov > >> Subject: Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code >> >> [EXTERNAL EMAIL] >> >> Instead of >> >> call PCCreate(PETSC_COMM_WORLD, pc, ierr) >> call PCSetType(pc, PCILU,ierr) ! Choose a preconditioner type (ILU) >> call KSPSetPC(ksp, pc,ierr) ! Associate the preconditioner with the KSP solver >> >> do >> >> call KSPGetPC(ksp,pc,ierr) >> call PCSetType(pc, PCILU,ierr) >> >> Do not call KSPSetUp(). It will be taken care of automatically during the solve >> >> >> >>> On Dec 20, 2023, at 8:52?PM, Shatanawi, Sawsan Muhammad via petsc-users > wrote: >>> >>> Hello, >>> I don't think that I set preallocation values when I created the matrix, would you please have look at my code. It is just the petsc related part from my code. >>> I was able to fix some of the error messages. Now I have a new set of error messages related to the KSP solver (attached) >>> >>> I appreciate your help >>> >>> Sawsan >>> From: Mark Adams > >>> Sent: Wednesday, December 20, 2023 6:44 AM >>> To: Shatanawi, Sawsan Muhammad > >>> Cc: petsc-users at mcs.anl.gov > >>> Subject: Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code >>> >>> [EXTERNAL EMAIL] >>> Did you set preallocation values when you created the matrix? >>> Don't do that. >>> >>> On Wed, Dec 20, 2023 at 9:36?AM Shatanawi, Sawsan Muhammad > wrote: >>> Hello, >>> >>> I am trying to create a sparse matrix( which is as I believe a zero matrix) then adding some nonzero elements to it over a loop, then assembling it >>> >>> Get Outlook for iOS >>> From: Mark Adams > >>> Sent: Wednesday, December 20, 2023 2:48 AM >>> To: Shatanawi, Sawsan Muhammad > >>> Cc: petsc-users at mcs.anl.gov > >>> Subject: Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code >>> >>> [EXTERNAL EMAIL] >>> I am guessing that you are creating a matrix, adding to it, finalizing it ("assembly"), and then adding to it again, which is fine, but you are adding new non-zeros to the sparsity pattern. >>> If this is what you want then you can tell the matrix to let you do that. >>> Otherwise you have a bug. >>> >>> Mark >>> >>> On Tue, Dec 19, 2023 at 9:50?PM Shatanawi, Sawsan Muhammad via petsc-users > wrote: >>> Hello everyone, >>> >>> I hope this email finds you well. >>> >>> My Name is Sawsan Shatanawi, and I am currently working on developing a Fortran code for simulating groundwater flow in a 3D system. The code involves solving a nonlinear system, and I have created the matrix to be solved using the PCG solver and Picard iteration. However, when I tried to assign it as a PETSc matrix I started getting a lot of error messages. >>> >>> I am kindly asking if someone can help me, I would be happy to share my code with him/her. >>> >>> Please find the attached file contains a list of errors I have gotten >>> >>> Thank you in advance for your time and assistance. >>> Best regards, >>> >>> Sawsan >>> >>> >>> >> >> >> >> -- >> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >> -- Norbert Wiener >> >> https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!eZkBlApyezAyEBaDxGemEyUmTe87omvkv59rrf0Mq8L4bOTqEL9wynuMJ1ci9kRDOqucZBiYdGT2tRTVcuRVDQ4$ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Tue May 7 15:57:43 2024 From: mfadams at lbl.gov (Mark Adams) Date: Tue, 7 May 2024 16:57:43 -0400 Subject: [petsc-users] Reasons for breakdown in preconditioned LSQR In-Reply-To: <7AF1F3F7-C5C6-4960-96B4-53122CEEF830@joliv.et> References: <30b07a76-9479-4a3a-9cf1-37251be79c07@kit.ac.jp> <31c6214d-c333-4624-ab74-a9ae626b50a7@kit.ac.jp> <7E39F820-0309-46C2-BCC5-C5F21ADCAB4D@joliv.et> <171508623875175.1449297990@sys11mail03.cis.kit.ac.jp> <7AF1F3F7-C5C6-4960-96B4-53122CEEF830@joliv.et> Message-ID: Yes, easy to solve. It is a kind of Lapalcian that comes up in graph algorithms. You mentioned problems with pinned points, with zero displacement. Conceptually you want to create A'A without any modification for pinned points. Then remove the rows and columns associated with pinned points. A'A has a null space, but your method requires pinning and that will remove the null space, so I don't think you need to worry about that. Mark On Tue, May 7, 2024 at 10:30?AM Pierre Jolivet wrote: > OK, it?s very trivial to solve with algebraic solvers. If you are willing > to share larger test cases, maybe then some issue will arise (please do not > attach them to your mail, send a URL, if you cannot and must attach it to > your mail, switch > ZjQcmQRYFpfptBannerStart > This Message Is From an External Sender > This message came from outside your organization. > > ZjQcmQRYFpfptBannerEnd > OK, it?s very trivial to solve with algebraic solvers. > If you are willing to share larger test cases, maybe then some issue will > arise (please do not attach them to your mail, send a URL, if you cannot > and must attach it to your mail, switch to the following mailing list: > petsc-maint at mcs.anl.gov) > > Thanks, > Pierre > > $ mpirun -n 8 ./sparse_ls -mat_name > /tmp/system/matdump_step00010000_dir0.bin -options_file gmres.rc -pc_type > hypre -options_left 0 > 0 KSP Residual norm 3.453633143461e+01 > 1 KSP Residual norm 1.454005637026e+00 > 2 KSP Residual norm 7.254564223178e-02 > 3 KSP Residual norm 2.595967767025e-03 > 4 KSP Residual norm 8.616844003887e-05 > 5 KSP Residual norm 2.519788767995e-06 > 6 KSP Residual norm 6.830984907384e-08 > Linear solve converged due to CONVERGED_RTOL iterations 6 > ||A^T(Ax-b)|| / ||Ax-b|| = 0.000000 / 17.017700 = 0.000000 > $ mpirun -n 8 ./sparse_ls -mat_name > /tmp/system/matdump_step00010000_dir0.bin -options_file gmres.rc -pc_type > gamg -options_left 0 > 0 KSP Residual norm 3.453633143461e+01 > 1 KSP Residual norm 9.410331701775e+00 > 2 KSP Residual norm 7.249670993944e-01 > 3 KSP Residual norm 8.415995591379e-02 > 4 KSP Residual norm 8.230048116783e-03 > 5 KSP Residual norm 8.200299281980e-04 > 6 KSP Residual norm 8.332302028270e-05 > 7 KSP Residual norm 9.103576883087e-06 > 8 KSP Residual norm 8.837914581158e-07 > 9 KSP Residual norm 8.295880033739e-08 > Linear solve converged due to CONVERGED_RTOL iterations 9 > ||A^T(Ax-b)|| / ||Ax-b|| = 0.000000 / 17.017700 = 0.000000 > $ mpirun -n 8 ./sparse_ls -mat_name > /tmp/system/matdump_step00010000_dir0.bin -options_file gmres.rc -pc_type > hpddm -options_left 0 > 0 KSP Residual norm 3.453633143461e+01 > 1 KSP Residual norm 4.455259710220e+00 > 2 KSP Residual norm 1.190904699181e-01 > 3 KSP Residual norm 5.220970280301e-03 > 4 KSP Residual norm 6.214389182723e-05 > 5 KSP Residual norm 1.693927074022e-06 > 6 KSP Residual norm 3.700055659502e-08 > Linear solve converged due to CONVERGED_RTOL iterations 6 > ||A^T(Ax-b)|| / ||Ax-b|| = 0.000000 / 17.017700 = 0.000000 > > > On 7 May 2024, at 2:50?PM, Seiz,Marco wrote: > > This Message Is From an External Sender > This message came from outside your organization. > > Pierre, > > I've attached the dumps of the matrix + RHS for something of about 3k x 1k. > > > Regarding the weird divergence behaviour, I tried again at home but I still get the same results. > > I am running a rolling release distribution on both machines, but that really shouldn't matter for divergence behavior I would think. > > Is there some kind of option in PETSc to get more information about the breakdown from my side? > > > Best regards, > > Marco > > ----- Original Message ----- > >> From: Pierre Jolivet > >> To: Marco Seiz > >> Cc: petsc-users at mcs.anl.gov > >> Date: 2024-05-07 18:12:18 > >> Subject: Re: [petsc-users] Reasons for breakdown in preconditioned LSQR > >> > > >> > >> > On 7 May 2024, at 9:10?AM, Marco Seiz wrote: > >> > > >> > Thanks for the quick response! > >> > > >> > On 07.05.24 14:24, Pierre Jolivet wrote: > >> >> > >> >> > >> >>> On 7 May 2024, at 7:04?AM, Marco Seiz wrote: > >> >>> > >> >>> This Message Is From an External Sender > >> >>> This message came from outside your organization. > >> >>> Hello, > >> >>> > >> >>> something a bit different from my last question, since that didn't > >> >>> progress so well: > >> >>> I have a related model which generally produces a rectangular matrix A, > >> >>> so I am using LSQR to solve the system. > >> >>> The matrix A has two nonzeros (1, -1) per row, with A^T A being similar > >> >>> to a finite difference Poisson matrix if the rows were permuted randomly. > >> >>> The problem is singular in that the solution is only specified up to a > >> >>> constant from the matrix, with my target solution being a weighted zero > >> >>> average one, which I can handle by adding a nullspace to my matrix. > >> >>> However, I'd also like to pin (potentially many) DOFs in the future so I > >> >>> also tried pinning a single value, and afterwards subtracting the > >> >>> average from the KSP solution. > >> >>> This leads to the KSP *sometimes* diverging when I use a preconditioner; > >> >>> the target size of the matrix will be something like ([1,20] N) x N, > >> >>> with N ~ [2, 1e6] so for the higher end I will require a preconditioner > >> >>> for reasonable execution time. > >> >>> > >> >>> For a smaller example system, I set up my application to dump the input > >> >>> to the KSP when it breaks down and I've attached a simple python script > >> >>> + data using petsc4py to demonstrate the divergence for those specific > >> >>> systems. > >> >>> With `python3 lsdiv.py -pc_type lu -ksp_converged_reason` that > >> >>> particular system shows breakdown, but if I remove the pinned DOF and > >> >>> add the nullspace (pass -usens) it converges. I did try different PCs > >> >>> but they tend to break down at different steps, e.g. `python3 lsdiv.py > >> >>> -usenormal -qrdiv -pc_type qr -ksp_converged_reason` shows the breakdown > >> >>> for PCQR when I use MatCreateNormal for creating the PC mat, but > >> >>> interestingly it doesn't break down when I explicitly form A^T A (don't > >> >>> pass -usenormal). > >> >> > >> >> What version are you using? All those commands are returning > >> >> Linear solve converged due to CONVERGED_RTOL_NORMAL iterations 1 > >> >> So I cannot reproduce any breakdown, but there have been recent changes to KSPLSQR. > >> > For those tests I've been using PETSc 3.20.5 (last githash was > >> > 4b82c11ab5d ). > >> > I pulled the latest version from gitlab ( 6b3135e3cbe ) and compiled it, > >> > but I had to drop --download-suitesparse=1 from my earlier config due to > >> > errors. > >> > Should I write a separate mail about this? > >> > > >> > The LU example still behaves the same for me (`python3 lsdiv.py -pc_type > >> > lu -ksp_converged_reason` gives DIVERGED_BREAKDOWN, `python3 lsdiv.py > >> > -usens -pc_type lu -ksp_converged_reason` gives CONVERGED_RTOL_NORMAL) > >> > but the QR example fails since I had to remove suitesparse. > >> > petsc4py.__version__ reports 3.21.1 and if I rebuild my application, > >> > then `ldd app` gives me `libpetsc.so .3.21 => > >> > /opt/petsc/linux-c-opt/lib/libpetsc.so .3.21` so it should be using the > >> > newly built one. > >> > The application then still eventually yields a DIVERGED_BREAKDOWN. > >> > I don't have a ~/.petscrc and PETSC_OPTIONS is unset, so if we are on > >> > the same version and there's still a discrepancy it is quite weird. > >> > >> Quite weird indeed? > >> $ python3 lsdiv.py -pc_type lu -ksp_converged_reason > >> Linear solve converged due to CONVERGED_RTOL_NORMAL iterations 1 > >> $ python3 lsdiv.py -usens -pc_type lu -ksp_converged_reason > >> Linear solve converged due to CONVERGED_RTOL_NORMAL iterations 1 > >> $ python3 lsdiv.py -pc_type qr -ksp_converged_reason > >> Linear solve converged due to CONVERGED_RTOL_NORMAL iterations 1 > >> $ python3 lsdiv.py -usens -pc_type qr -ksp_converged_reason > >> Linear solve converged due to CONVERGED_RTOL_NORMAL iterations 1 > >> > >> >>> For the moment I can work by adding the nullspace but eventually the > >> >>> need for pinning DOFs will resurface, so I'd like to ask where the > >> >>> breakdown is coming from. What causes the breakdowns? Is that a generic > >> >>> problem occurring when adding (dof_i = val) rows to least-squares > >> >>> systems which prevents these preconditioners from being robust? If so, > >> >>> what preconditioners could be robust? > >> >>> I did a minimal sweep of the available PCs by going over the possible > >> >>> inputs of -pc_type for my application while pinning one DOF. Excepting > >> >>> unavailable PCs (not compiled for, other setup missing, ...) and those > >> >>> that did break down, I am left with ( hmg jacobi mat none pbjacobi sor > >> >>> svd ). > >> >> It?s unlikely any of these preconditioners will scale (or even converge) for problems with up to 1E6 unknowns. > >> >> I could help you setup https://urldefense.us/v3/__https://epubs.siam.org/doi/abs/10.1137/21M1434891__;!!G_uCfscf7eWS!fW1baXZMAQIKi0VDUIDUUzpMi4xQf7jrWGCXPlpIllqKAXJBzDClVwrLKYuWuT7LYfZoDzK4g9I9g_z0Iwv7Sg$ if you are willing to share a larger example (the current Mat are extremely tiny). > >> > Yes, that would be great. About how large of a matrix do you need? I can > >> > probably quickly get something non-artificial up to O(N) ~ 1e3, > >> > >> That?s big enough. > >> If you?re in luck, AMG on the normal equations won?t behave too badly, but I?ll try some more robust (in theory) methods nonetheless. > >> > >> Thanks, > >> Pierre > >> > >> > bigger > >> > matrices will take some time since I purposefully ignored MPI previously. > >> > The matrix basically describes the contacts between particles which are > >> > resolved on a uniform grid, so the main memory hog isn't the matrix but > >> > rather resolving the particles. > >> > I should mention that the matrix changes over the course of the > >> > simulation but stays constant for many solves, i.e. hundreds to > >> > thousands of solves with variable RHS between periods of contact > >> > formation/loss. > >> > > >> >> > >> >> Thanks, > >> >> Pierre > >> >>> > >> >>> > >> >>> Best regards, > >> >>> Marco > >> >>> > >> >>> > >> >> > >> >> > >> > Best regards, > >> > Marco > >> > >> > >> > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue May 7 19:17:42 2024 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 7 May 2024 20:17:42 -0400 Subject: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code In-Reply-To: References: <17B6144E-4F3A-4173-9818-4B03669736C4@petsc.dev> <9BD74F88-20FD-4021-B02A-195A58B72282@petsc.dev> Message-ID: On Tue, May 7, 2024 at 2:23?PM Shatanawi, Sawsan Muhammad via petsc-users < petsc-users at mcs.anl.gov> wrote: > Hello everyone, I hope this email finds you well. My Name is Sawsan > Shatanawi, and I was developing a Fortran code for simulating groundwater > flow in a 3D system with nonlinear behavior. I solved the nonlinear system > using the PCG solver and > ZjQcmQRYFpfptBannerStart > This Message Is From an External Sender > This message came from outside your organization. > > ZjQcmQRYFpfptBannerEnd > > Hello everyone, > > > > I hope this email finds you well. > > > > My Name is Sawsan Shatanawi, and I was developing a Fortran code for > simulating groundwater flow in a 3D system with nonlinear behavior. I > solved the nonlinear system using the PCG solver and Picard iteration, but > I did not get good results although I checked my matrix and RHS and > everything, I decided to change my solver to Newton Rapson method. > I checked PETSc documents but I have a few questions: > 1) My groundwater system is time-dependent, so should I use TS only > instead of SNES? > You could use TS, but it is not necessary. You could use SNES and your own timestepping. THe advantage of TS is that you can try many different timesteppers without recoding (just like you can try many different linear and nonlinear solvers). > 2) My system has its deltaT, would using deltaT as dt affect my solver, or > is it better to use TS-PETSc dt? Also, would using PETSc dt affect the > simulation of the groundwater system > It sounds like your dt comes from your timestepper. If you use TS, you would use the dt from that. > 3) I want my Jacobian matrix to be calculated by PETSc automatically > PETSc can calculate a full Jacobian for smaller problems, or a finite-difference Jacobian for any problem (but this impacts the solver). It should be straightfoward to code up the analytic Jacobian. Is there a reason it would be a problem? > 4) Do I need to define and calculate the residual vector? > Yes. > My A-Matrix contains coefficients and external sources and my RHS vector > includes the boundary conditions > It is strange that your matrix would contain source terms. Do they depend on the flow fields? Barry is right, you should consider PFlotran, and at least know why it would not work for your problem if you don't use it. Thanks, Matt > > Please find the attached file contains a draft of my code > > Thank you in advance for your time and help. > > > Best regards, > > Sawsan > > ------------------------------ > *From:* Shatanawi, Sawsan Muhammad > *Sent:* Tuesday, January 16, 2024 10:43 AM > *To:* Junchao Zhang > *Cc:* Barry Smith ; Matthew Knepley ; > Mark Adams ; petsc-users at mcs.anl.gov < > petsc-users at mcs.anl.gov> > *Subject:* Re: [petsc-users] Help with Integrating PETSc into Fortran > Groundwater Flow Simulation Code > > Hello all, > > Thank you for your valuable help. I will do your recommendations and hope > it will run without any issues. > > Bests, > Sawsan > > ------------------------------ > *From:* Junchao Zhang > *Sent:* Friday, January 12, 2024 8:46 AM > *To:* Shatanawi, Sawsan Muhammad > *Cc:* Barry Smith ; Matthew Knepley ; > Mark Adams ; petsc-users at mcs.anl.gov < > petsc-users at mcs.anl.gov> > *Subject:* Re: [petsc-users] Help with Integrating PETSc into Fortran > Groundwater Flow Simulation Code > > > *[EXTERNAL EMAIL]* > Hi, Sawsan, > First in test_main.F90, you need to call VecGetArrayF90(temp_solution, > H_vector, ierr) and VecRestoreArrayF90 (temp_solution, H_vector, ierr) as > Barry mentioned. > Secondly, in the loop of test_main.F90, it calls GW_solver(). Within > it, it calls PetscInitialize()/PetscFinalize(). But without MPI being > initialized, PetscInitialize()/PetscFinalize()* can only be called once. * > do timestep =2 , NTSP > call GW_boundary_conditions(timestep-1) > !print *,HNEW(1,1,1) > call GW_elevation() > ! print *, GWTOP(2,2,2) > call GW_conductance() > ! print *, CC(2,2,2) > call GW_recharge() > ! print *, B_Rech(5,4) > call GW_pumping(timestep-1) > ! print *, B_pump(2,2,2) > call GW_SW(timestep-1) > print *,B_RIVER (2,2,2) > call GW_solver(timestep-1,N) > call GW_deallocate_loop() > end do > > A solution is to delete PetscInitialize()/PetscFinalize() in > GW_solver_try.F90 and move it to test_main.F90, outside the do loop. > > diff --git a/test_main.F90 b/test_main.F90 > index b5997c55..107bd3ee 100644 > --- a/test_main.F90 > +++ b/test_main.F90 > @@ -1,5 +1,6 @@ > program test_GW > > +#include > use petsc > use GW_constants > use GW_param_by_user > @@ -8,6 +9,9 @@ program test_GW > implicit none > integer :: N > integer :: timestep > + PetscErrorCode ierr > + > + call PetscInitialize(ierr) > call GW_domain(N) > !print *, "N=",N > !print *, DELTAT > @@ -37,4 +41,5 @@ program test_GW > end do > print *, HNEW(NCOL,3,2) > call GW_deallocate () > + call PetscFinalize(ierr) > end program test_GW > > With that, the MPI error will be fixed. The code could run to > gw_deallocate () before abort. There are other memory errors. You can > install/use valgrind to fix them. Run it with valgrind ./GW.exe and look > through the output > > > Thanks. > --Junchao Zhang > > > On Thu, Jan 11, 2024 at 10:49?PM Shatanawi, Sawsan Muhammad < > sawsan.shatanawi at wsu.edu> wrote: > > Hello, > > Thank you all for your help. > > I have changed VecGetArray to VecGetArrayF90, and the location of destory > call. but I want to make sure that VecGet ArrayF90 is to make a new array( > vector) that I can use in the rest of my Fortran code? > > when I run it and debugged it, I got > > 5.2000000E-03 > 50.00000 > 10.00000 > 0.0000000E+00 > PETSC: Attaching gdb to > /weka/data/lab/richey/sawsan/GW_CODE/code2024/SS_GWM/./GW.exe of pid 33065 > on display :0.0 on machine sn16 > Unable to start debugger in xterm: No such file or directory > 0.0000000E+00 > Attempting to use an MPI routine after finalizing MPICH > srun: error: sn16: task 0: Exited with exit code 1 > [sawsan.shatanawi at login-p2n02 SS_GWM]$ gdb ./GW/exe > GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-100.el7 > Copyright (C) 2013 Free Software Foundation, Inc. > License GPLv3+: GNU GPL version 3 or later < > https://urldefense.us/v3/__http://gnu.org/licenses/gpl.html__;!!G_uCfscf7eWS!a5WSXOX8ewfPLoPMZrwcIgaFqi8MKa016aCtKSJ_a6Ic0bWJcexX38TUj1Whfm88oF2dghzS5-WjN-wliID1$ > > > > This is free software: you are free to change and redistribute it. > There is NO WARRANTY, to the extent permitted by law. Type "show copying" > and "show warranty" for details. > This GDB was configured as "x86_64-redhat-linux-gnu". > For bug reporting instructions, please see: > > >... > ./GW/exe: No such file or directory. > (gdb) run > Starting program: > No executable file specified. > Use the "file" or "exec-file" command. > (gdb) bt > No stack. > (gdb) > > If the highlighted line is the error, I don't know why when I write gdb , > it does not show me the location of error > The code : sshatanawi/SS_GWM (github.com) > > > > I really appreciate your helps > > Sawsan > ------------------------------ > *From:* Barry Smith > *Sent:* Wednesday, January 10, 2024 5:35 PM > *To:* Junchao Zhang > *Cc:* Shatanawi, Sawsan Muhammad ; Mark Adams < > mfadams at lbl.gov>; petsc-users at mcs.anl.gov > *Subject:* Re: [petsc-users] Help with Integrating PETSc into Fortran > Groundwater Flow Simulation Code > > > *[EXTERNAL EMAIL]* > > > On Jan 10, 2024, at 6:49?PM, Junchao Zhang > wrote: > > Hi, Sawsan, > I could build your code and I also could gdb it. > > $ gdb ./GW.exe > ... > $ Thread 1 "GW.exe" received signal SIGSEGV, Segmentation fault. > 0x00007ffff1e6d44f in vecgetarray_ (x=0x7fffffffa718, fa=0x0, > ia=0x7fffffffa75c, ierr=0x0) at > /scratch/jczhang/petsc/src/vec/vec/interface/ftn-custom/zvectorf.c:257 > 257 *ierr = VecGetArray(*x, &lx); > (gdb) bt > #0 0x00007ffff1e6d44f in vecgetarray_ (x=0x7fffffffa718, fa=0x0, > ia=0x7fffffffa75c, ierr=0x0) at > /scratch/jczhang/petsc/src/vec/vec/interface/ftn-custom/zvectorf.c:257 > #1 0x000000000040b6e3 in gw_solver (t_s=1.40129846e-45, n=300) at > GW_solver_try.F90:169 > #2 0x000000000040c6a8 in test_gw () at test_main.F90:35 > > ierr=0x0 caused the segfault. See > https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/VecGetArray/*vecgetarray__;Iw!!G_uCfscf7eWS!a5WSXOX8ewfPLoPMZrwcIgaFqi8MKa016aCtKSJ_a6Ic0bWJcexX38TUj1Whfm88oF2dghzS5-WjN1gHtGF9$ > , > you should use VecGetArrayF90 instead. > > BTW, Barry, the code > https://urldefense.us/v3/__https://github.com/sshatanawi/SS_GWM/blob/main/GW_solver_try.F90*L169__;Iw!!G_uCfscf7eWS!a5WSXOX8ewfPLoPMZrwcIgaFqi8MKa016aCtKSJ_a6Ic0bWJcexX38TUj1Whfm88oF2dghzS5-WjN38ZAbIr$ > > has "call VecGetArray(temp_solution, H_vector, ierr)". I don't find > petsc Fortran examples doing VecGetArray. Do we still support it? > > > This is not the correct calling sequence for VecGetArray() from > Fortran. > > Regardless, definitely should not be writing any new code that uses > VecGetArray() from Fortran. Should use VecGetArrayF90(). > > > --Junchao Zhang > > > On Wed, Jan 10, 2024 at 2:38?PM Shatanawi, Sawsan Muhammad via petsc-users > wrote: > > Hello all, > > I hope you are doing well. > > Generally, I use gdb to debug the code. > I got the attached error message. > > I have tried to add the flag -start_in_debugger in the make file, but it > didn't work, so it seems I was doing it in the wrong way > > This is the link for the whole code: sshatanawi/SS_GWM (github.com) > > > > GitHub - sshatanawi/SS_GWM > > Contribute to sshatanawi/SS_GWM development by creating an account on > GitHub. > github.com > > *?* > > You can read the description of the code in " Model Desprciption.pdf" > the compiling file is makefile_f90 where you can find the linked code > files > > I really appreciate your help > > Bests, > Sawsan > ------------------------------ > *From:* Mark Adams > *Sent:* Friday, January 5, 2024 4:53 AM > *To:* Shatanawi, Sawsan Muhammad > *Cc:* Matthew Knepley ; petsc-users at mcs.anl.gov < > petsc-users at mcs.anl.gov> > *Subject:* Re: [petsc-users] Help with Integrating PETSc into Fortran > Groundwater Flow Simulation Code > > > *[EXTERNAL EMAIL]* > This is a segv. As Matt said, you need to use a debugger for this or add > print statements to narrow down the place where this happens. > > You will need to learn how to use debuggers to do your project so you > might as well start now. > > If you have a machine with a GUI debugger that is easier but command line > debuggers are good to learn anyway. > > I tend to run debuggers directly (eg, lldb ./a.out -- program-args ...) > and use a GUI debugger (eg, Totalview or DDT) if available. > > Mark > > > On Wed, Dec 20, 2023 at 10:02?PM Shatanawi, Sawsan Muhammad via > petsc-users wrote: > > Hello Matthew, > > Thank you for your help. I am sorry that I keep coming back with my error > messages, but I reached a point that I don't know how to fix them, and I > don't understand them easily. > The list of errors is getting shorter, now I am getting the attached error > messages > > Thank you again, > > Sawsan > ------------------------------ > *From:* Matthew Knepley > *Sent:* Wednesday, December 20, 2023 6:54 PM > *To:* Shatanawi, Sawsan Muhammad > *Cc:* Barry Smith ; petsc-users at mcs.anl.gov < > petsc-users at mcs.anl.gov> > *Subject:* Re: [petsc-users] Help with Integrating PETSc into Fortran > Groundwater Flow Simulation Code > > > *[EXTERNAL EMAIL]* > On Wed, Dec 20, 2023 at 9:49?PM Shatanawi, Sawsan Muhammad via petsc-users > wrote: > > Hello Barry, > > Thank you a lot for your help, Now I am getting the attached error message. > > > Do not destroy the PC from KSPGetPC() > > THanks, > > Matt > > > Bests, > Sawsan > ------------------------------ > *From:* Barry Smith > *Sent:* Wednesday, December 20, 2023 6:32 PM > *To:* Shatanawi, Sawsan Muhammad > *Cc:* Mark Adams ; petsc-users at mcs.anl.gov < > petsc-users at mcs.anl.gov> > *Subject:* Re: [petsc-users] Help with Integrating PETSc into Fortran > Groundwater Flow Simulation Code > > > *[EXTERNAL EMAIL]* > > Instead of > > call PCCreate(PETSC_COMM_WORLD, pc, ierr) > call PCSetType(pc, PCILU,ierr) ! Choose a preconditioner type (ILU) > call KSPSetPC(ksp, pc,ierr) ! Associate the preconditioner with the > KSP solver > > do > > call KSPGetPC(ksp,pc,ierr) > call PCSetType(pc, PCILU,ierr) > > Do not call KSPSetUp(). It will be taken care of automatically during the > solve > > > > On Dec 20, 2023, at 8:52?PM, Shatanawi, Sawsan Muhammad via petsc-users < > petsc-users at mcs.anl.gov> wrote: > > Hello, > I don't think that I set preallocation values when I created the matrix, > would you please have look at my code. It is just the petsc related part > from my code. > I was able to fix some of the error messages. Now I have a new set of > error messages related to the KSP solver (attached) > > I appreciate your help > > Sawsan > ------------------------------ > *From:* Mark Adams > *Sent:* Wednesday, December 20, 2023 6:44 AM > *To:* Shatanawi, Sawsan Muhammad > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [petsc-users] Help with Integrating PETSc into Fortran > Groundwater Flow Simulation Code > > *[EXTERNAL EMAIL]* > Did you set preallocation values when you created the matrix? > Don't do that. > > On Wed, Dec 20, 2023 at 9:36?AM Shatanawi, Sawsan Muhammad < > sawsan.shatanawi at wsu.edu> wrote: > > Hello, > > I am trying to create a sparse matrix( which is as I believe a zero > matrix) then adding some nonzero elements to it over a loop, then > assembling it > > Get Outlook for iOS > > ------------------------------ > *From:* Mark Adams > *Sent:* Wednesday, December 20, 2023 2:48 AM > *To:* Shatanawi, Sawsan Muhammad > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [petsc-users] Help with Integrating PETSc into Fortran > Groundwater Flow Simulation Code > > *[EXTERNAL EMAIL]* > I am guessing that you are creating a matrix, adding to it, finalizing it > ("assembly"), and then adding to it again, which is fine, but you are > adding new non-zeros to the sparsity pattern. > If this is what you want then you can tell the matrix to let you do that. > Otherwise you have a bug. > > Mark > > On Tue, Dec 19, 2023 at 9:50?PM Shatanawi, Sawsan Muhammad via petsc-users > wrote: > > Hello everyone, > > I hope this email finds you well. > > My Name is Sawsan Shatanawi, and I am currently working on developing a > Fortran code for simulating groundwater flow in a 3D system. The code > involves solving a nonlinear system, and I have created the matrix to be > solved using the PCG solver and Picard iteration. However, when I tried > to assign it as a PETSc matrix I started getting a lot of error messages. > > I am kindly asking if someone can help me, I would be happy to share my > code with him/her. > > Please find the attached file contains a list of errors I have gotten > > Thank you in advance for your time and assistance. > > Best regards, > > Sawsan > > > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!a5WSXOX8ewfPLoPMZrwcIgaFqi8MKa016aCtKSJ_a6Ic0bWJcexX38TUj1Whfm88oF2dghzS5-WjN8rPXZEB$ > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!a5WSXOX8ewfPLoPMZrwcIgaFqi8MKa016aCtKSJ_a6Ic0bWJcexX38TUj1Whfm88oF2dghzS5-WjN8rPXZEB$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From samar.khatiwala at earth.ox.ac.uk Wed May 8 02:53:49 2024 From: samar.khatiwala at earth.ox.ac.uk (Samar Khatiwala) Date: Wed, 8 May 2024 07:53:49 +0000 Subject: [petsc-users] Question about petsc4py createWithArray function In-Reply-To: References: <6ED2F015-541E-485B-AACB-EAE329A4A307@earth.ox.ac.uk> Message-ID: <03579AC8-B167-4226-B964-1E3D5CBDCC83@earth.ox.ac.uk> Hi Stefano, Thank you for this. Your example was invaluable in helping me figure out what is going on. (I did not know about about sys.getrefcount - I?m only just starting to learn python.) It was indeed the refcount. Until the Vec is destroyed it can?t go to zero and my array destructor method won?t be triggered (which is not really a problem because the memory is freed elsewhere). I take back what I said about there being a memory leak in petsc4py. Sorry. Thanks again for your help! Best, Samar On May 6, 2024, at 2:09 PM, Stefano Zampini wrote: Samar After a second look, I believe the petsc4py code is correct. You can test it using the script below. After destroy is called (or del), the reference count of the numpy array is back to its initial state. Maybe you are not calling del or destroy? a MWE would help to understand your use case kl-18448:~ szampini$ cat t.py import sys import numpy as np from petsc4py import PETSc x = np.zeros(4, dtype=PETSc.ScalarType) print('initial ref count',sys.getrefcount(x)) v = PETSc.Vec().createWithArray(x) print('after create',sys.getrefcount(x)) # check if they share memory v.view() x[1] = 2 v.view() # free v.destroy() # you can also call del # del v print('after destroy',sys.getrefcount(x)) kl-18448:~ szampini$ python t.py initial ref count 2 after create 3 Vec Object: 1 MPI process type: seq 0. 0. 0. 0. Vec Object: 1 MPI process type: seq 0. 2. 0. 0. after destroy 2 Il giorno lun 6 mag 2024 alle ore 08:49 Samar Khatiwala > ha scritto: Hi Stefano, Thanks for looking into this. Since createWithArray calls VecCreateMPIWithArray which, as Matt noted and is documented (https://urldefense.us/v3/__https://petsc.org/main/manualpages/Vec/VecCreateMPIWithArray/__;!!G_uCfscf7eWS!fl-7I7xkuEmTFH8M9EaARIZL5vhGFWkIk5bVIGt0bkt_cYcuwPSwgOWaec1-DvYwiKWxA5q0HS0VTLBmD4NnSe1PfG057Y4SBd-8qs8$ ) doesn?t free the memory, then there?s a memory leak (and, furthermore, calling del on the original array will have no effect). Lisandro: would be great if you can provide some guidance. Thanks, Samar On May 3, 2024, at 12:45 PM, Stefano Zampini > wrote: While waiting for our Python wizard to shed light on this, I note that, from the documentation of PyArray_FROM_OTF https://urldefense.us/v3/__https://numpy.org/devdocs/user/c-info.how-to-extend.html*converting-an-arbitrary-sequence-object__;Iw!!G_uCfscf7eWS!fl-7I7xkuEmTFH8M9EaARIZL5vhGFWkIk5bVIGt0bkt_cYcuwPSwgOWaec1-DvYwiKWxA5q0HS0VTLBmD4NnSe1PfG057Y4S1ro4qy0$ , we have The object can be any Python object convertible to an ndarray. If the object is already (a subclass of) the ndarray that satisfies the requirements then a new reference is returned. I guess we should call "del" on the ndarray returned by iarray_s after having called self.set_attr('__array__', array) in this case, but let's wait for Lisandro to confirm Il giorno ven 3 mag 2024 alle ore 11:42 Samar Khatiwala > ha scritto: This Message Is From an External Sender This message came from outside your organization. Hi Matt, Thanks so much for the quick reply! Regarding #2, I put some debug statement in my code and what I find is that when I use createWithArray on my Cython-allocated numpy array, the destructor I set for it is no longer called when I delete the array. (If I don?t use createWithArray then the destructor is triggered.) I interpret that to suggest that the petsc4py Vec is somehow ?taking over? management of the numpy array. But I don?t understand where that could be happening. (I don?t think it has to do with the actual freeing of memory by PETSc's VecDestroy.) createWithArray calls iarray_s which in turn calls PyArray_FROM_OTF. Could it be there?s something going on there? The numpy documentation is unclear. Lisandro: do you have any thoughts on this? Thanks, Samar On May 2, 2024, at 11:56 PM, Matthew Knepley > wrote: On Thu, May 2, 2024 at 12:53?PM Samar Khatiwala > wrote: This Message Is From an External Sender This message came from outside your organization. Hello, I have a couple of questions about createWithArray in petsc4py: 1) What is the correct usage for creating a standard MPI Vec with it? Something like this seems to work but is it right?: On each rank do: a = np.zeros(localSize) v = PETSc.Vec().createWithArray(a, comm=PETSc.COMM_WORLD) Is that all it takes? That looks right to me. 2) Who ?owns? the underlying memory for a Vec created with the createWithArray method, i.e., who is responsible for managing it and doing garbage collection? In my problem, the numpy array is created in a Cython module where memory is allocated, and a pointer to it is associated with a numpy ndarray via PyArray_SimpleNewFromData and PyArray_SetBaseObject. I have a deallocator method of my own that is called when the numpy array is deleted/goes out of scope/whenever python does garbage collection. All of that works fine. But if I use this array to create a Vec with createWithArray what happens when the Vec is, e.g., destroyed? Will my deallocator be called? No. The PETSc struct will be deallocated, but the storage will not be touched. Thanks, Matt Or does petsc4py know that it doesn?t own the memory and won?t attempt to free it? I can?t quite figure out from the petsc4py code what is going on. And help would be appreciated. Thanks very much. Samar -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fl-7I7xkuEmTFH8M9EaARIZL5vhGFWkIk5bVIGt0bkt_cYcuwPSwgOWaec1-DvYwiKWxA5q0HS0VTLBmD4NnSe1PfG057Y4SmVADDr8$ -- Stefano -- Stefano -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed May 8 10:00:12 2024 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 8 May 2024 11:00:12 -0400 Subject: [petsc-users] HDF5 time step count In-Reply-To: <6aa6644b-c36b-4f4d-b843-d1a1a4adf7bd@auckland.ac.nz> References: <425d849a-eba3-6739-5e71-251e0f29b5fb@auckland.ac.nz> <8735pcd12k.fsf@jedbrown.org> <7639e2f6-d326-e04b-bd8e-53cecb8d794e@auckland.ac.nz> <220daca5-21ea-00df-bc51-ba6124d020ad@auckland.ac.nz> <6aa6644b-c36b-4f4d-b843-d1a1a4adf7bd@auckland.ac.nz> Message-ID: On Wed, May 1, 2024 at 11:03?PM Adrian Croucher wrote: > hi Matt & all, > > I just had a query from one of my users which prompted me to see if any > progress had been made on the issue below - using PETSc to get the number > of time steps in an HDF5 file. > > I can't see anything new in PETSc on this - I did try using > PetscViewerHDF5ReadSizes() to see if that would do it, but it seems it > doesn't. If I use that on the "time" dataset it just returns 1. > > Sorry about the delay. I had lost track of this. Can you look at branch knepley/feature-hdf5-seq-len I have not made a test yet, but if this works for you, I will make a test and merge it in. Thanks! Matt > Regards, Adrian > On 11/10/21 2:08 pm, Adrian Croucher wrote: > > On 10/11/21 11:59 AM, Matthew Knepley wrote: > > On Sun, Oct 10, 2021 at 6:51 PM Adrian Croucher > wrote: > >> hi >> >> Is there any way to query the PETSc HDF5 viewer to find the number of >> time steps in the file? >> >> A common use case I have is that an HDF5 file from a previous simulation >> is used to get initial conditions for a subsequent run. The most common >> thing you want to do is restart from the last set of results in the >> previous output. To do that you need to know how many time steps there >> are, so you can set the output index to be the last one. >> >> I thought maybe I could just query the size of the "time" dataset, but I >> can't even see any obvious way to do that using the viewer functions. >> > > There is nothing in there that does it right now. Do you know how to do it > in HDF5? > If so, I can put it in. Otherwise, I will have to learn more HDF5 :) > > > I haven't actually tried this myself but it looks like what you do is: > > 1) get the dataspace for the dataset (in our case the "time" dataset): > > hid_t dspace = H5Dget_space(dset); > > 2) Get the dimensions of the dataspace: > > const int ndims = 1; > > hsize_t dims[ndims]; > H5Sget_simple_extent_dims(dspace, dims, NULL); > > The first element of dims should be the number of time steps. Here I've > assumed the number of dimensions of the time dataset is 1. In general you > can instead query the rank of the dataspace using > H5Sget_simple_extent_ndims() to get the rank ndims. > > Regards, Adrian > > -- > Dr Adrian Croucher > Senior Research Fellow > Department of Engineering Science > Waipapa Taumata Rau / University of Auckland, New Zealand > email: a.croucher at auckland.ac.nz > tel: +64 (0)9 923 4611 > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!eWw8ZyC03JCY81KS4nuvF6OwPoyRvldIwdZsw_6BlM1OLpJ4cucgOSJ6wetJu-YCCDUSYq0ZI_4rgu343kFn$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From a.croucher at auckland.ac.nz Wed May 8 21:46:35 2024 From: a.croucher at auckland.ac.nz (Adrian Croucher) Date: Thu, 9 May 2024 14:46:35 +1200 Subject: [petsc-users] HDF5 time step count In-Reply-To: References: <425d849a-eba3-6739-5e71-251e0f29b5fb@auckland.ac.nz> <8735pcd12k.fsf@jedbrown.org> <7639e2f6-d326-e04b-bd8e-53cecb8d794e@auckland.ac.nz> <220daca5-21ea-00df-bc51-ba6124d020ad@auckland.ac.nz> <6aa6644b-c36b-4f4d-b843-d1a1a4adf7bd@auckland.ac.nz> Message-ID: <97876f22-d284-45d8-8993-893e3937004f@auckland.ac.nz> hi Matt, On 9/05/24 3:00 am, Matthew Knepley wrote: > Sorry about the delay. I had lost track of this. Can you look at branch > > ? knepley/feature-hdf5-seq-len > > I have not made a test yet, but if this works for you, I will make a > test and merge it in. Thanks for looking at that. I just tried to test it, but there doesn't seem to be a Fortran interface for DMGetOutputSequenceLength(). Another odd thing I found was that DMPlexConstructGhostCells() seemed to complain if I passed in PETSC_NULL_INTEGER for the parameter numGhostCells (which works ok in v3.21.1) - the compiler complained about a rank-1/scalar mismatch. Can you no longer pass in a null integer for that? It works if I declare a dummy integer and pass it in. - Adrian -- Dr Adrian Croucher Senior Research Fellow Department of Engineering Science Waipapa Taumata Rau / University of Auckland, New Zealand email:a.croucher at auckland.ac.nz tel: +64 (0)9 923 4611 -------------- next part -------------- An HTML attachment was scrubbed... URL: From cho at slac.stanford.edu Wed May 8 23:31:04 2024 From: cho at slac.stanford.edu (Ng, Cho-Kuen) Date: Thu, 9 May 2024 04:31:04 +0000 Subject: [petsc-users] About PC hpddm Message-ID: I used spack to install petsc with hpddm as follows. o spack install petsc+hpddm o spack install slepc ^petsc+hpddm I got the following runtime error of finding the library hpddm_petsc. argv0 -ksp_converged_reason -ksp_view_final_residual -ksp_type gmres -pc_type hpddm -ksp_monitor -log_view -options_left -device_view $$$$$$$$$$$$$$ buildmatrix petsc [6]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [6]PETSC ERROR: Petsc has generated inconsistent data [6]PETSC ERROR: /global/cfs/cdirs/ace3p/software/spack/var/spack/environments/demo1/.spack-env/view/lib/libhpddm_petsc not found Is there something wrong with the install procedure? Thanks, Cho -------------- next part -------------- An HTML attachment was scrubbed... URL: From pierre at joliv.et Thu May 9 01:01:12 2024 From: pierre at joliv.et (Pierre Jolivet) Date: Thu, 9 May 2024 08:01:12 +0200 Subject: [petsc-users] About PC hpddm In-Reply-To: References: Message-ID: <4E6CD92B-CB1B-4CAB-81B3-5E8E88EA3BB3@joliv.et> > On 9 May 2024, at 6:31?AM, Ng, Cho-Kuen via petsc-users wrote: > > This Message Is From an External Sender > This message came from outside your organization. > I used spack to install petsc with hpddm as follows. > > o spack install petsc+hpddm > o spack install slepc ^petsc+hpddm > > I got the following runtime error of finding the library hpddm_petsc. > > argv0 -ksp_converged_reason -ksp_view_final_residual -ksp_type gmres -pc_type hpddm -ksp_monitor -log_view -options_left -device_view > $$$$$$$$$$$$$$ buildmatrix petsc > [6]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [6]PETSC ERROR: Petsc has generated inconsistent data > [6]PETSC ERROR: /global/cfs/cdirs/ace3p/software/spack/var/spack/environments/demo1/.spack-env/view/lib/libhpddm_petsc not found > > Is there something wrong with the install procedure? The command lines look OK to me (but I don?t know Spack). I?m not sure where things are being installed. Could you please send the path to: 1) libpetsc.so 2) libslepc.so 3) libhpddm_petsc.so If there is a mismatch between some of these, you may want to set the environment variable SLEPC_DIR or the command line parameter -hpddm_dir. Thanks, Pierre > Thanks, > Cho -------------- next part -------------- An HTML attachment was scrubbed... URL: From cho at slac.stanford.edu Thu May 9 01:42:43 2024 From: cho at slac.stanford.edu (Ng, Cho-Kuen) Date: Thu, 9 May 2024 06:42:43 +0000 Subject: [petsc-users] About PC hpddm In-Reply-To: <4E6CD92B-CB1B-4CAB-81B3-5E8E88EA3BB3@joliv.et> References: <4E6CD92B-CB1B-4CAB-81B3-5E8E88EA3BB3@joliv.et> Message-ID: Pierre, petec and slepc libraries are found in the spack directory, but libhpddm_petsc is not. So it is not built during the spack install process. ... ... ./opt/spack/linux-sles15-zen3/gcc-12.3.0/petsc-3.21.0-tdljbqbgvde4zv6jr6tf7bbpae3rfkyg/lib/libpetsc.so ./var/spack/environments/demo1/.spack-env/._view/6rqhi7tgwui4fjo74fgkkyq4lklghged/lib/libpetsc.so ... ... ./opt/spack/linux-sles15-zen3/gcc-12.3.0/slepc-3.21.0-qhuhkznl5rlhqknsdqervlevup6xiuyt/lib/libslepc.so ./var/spack/environments/demo1/.spack-env/._view/6rqhi7tgwui4fjo74fgkkyq4lklghged/lib/libslepc.so Thanks, Cho ________________________________ From: Pierre Jolivet Sent: Wednesday, May 8, 2024 11:01 PM To: Ng, Cho-Kuen Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] About PC hpddm On 9 May 2024, at 6:31?AM, Ng, Cho-Kuen via petsc-users wrote: This Message Is From an External Sender This message came from outside your organization. I used spack to install petsc with hpddm as follows. o spack install petsc+hpddm o spack install slepc ^petsc+hpddm I got the following runtime error of finding the library hpddm_petsc. argv0 -ksp_converged_reason -ksp_view_final_residual -ksp_type gmres -pc_type hpddm -ksp_monitor -log_view -options_left -device_view $$$$$$$$$$$$$$ buildmatrix petsc [6]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [6]PETSC ERROR: Petsc has generated inconsistent data [6]PETSC ERROR: /global/cfs/cdirs/ace3p/software/spack/var/spack/environments/demo1/.spack-env/view/lib/libhpddm_petsc not found Is there something wrong with the install procedure? The command lines look OK to me (but I don?t know Spack). I?m not sure where things are being installed. Could you please send the path to: 1) libpetsc.so 2) libslepc.so 3) libhpddm_petsc.so If there is a mismatch between some of these, you may want to set the environment variable SLEPC_DIR or the command line parameter -hpddm_dir. Thanks, Pierre Thanks, Cho -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Thu May 9 01:49:39 2024 From: balay at mcs.anl.gov (Satish Balay) Date: Thu, 9 May 2024 01:49:39 -0500 (CDT) Subject: [petsc-users] About PC hpddm In-Reply-To: References: <4E6CD92B-CB1B-4CAB-81B3-5E8E88EA3BB3@joliv.et> Message-ID: Try: spack install slepc+hpddm ^petsc+hpddm Satish On Thu, 9 May 2024, Ng, Cho-Kuen via petsc-users wrote: > Pierre, > > petec and slepc libraries are found in the spack directory, but libhpddm_petsc is not. So it is not built during the spack install process. > > ... ... > ./opt/spack/linux-sles15-zen3/gcc-12.3.0/petsc-3.21.0-tdljbqbgvde4zv6jr6tf7bbpae3rfkyg/lib/libpetsc.so > ./var/spack/environments/demo1/.spack-env/._view/6rqhi7tgwui4fjo74fgkkyq4lklghged/lib/libpetsc.so > > ... ... > ./opt/spack/linux-sles15-zen3/gcc-12.3.0/slepc-3.21.0-qhuhkznl5rlhqknsdqervlevup6xiuyt/lib/libslepc.so > ./var/spack/environments/demo1/.spack-env/._view/6rqhi7tgwui4fjo74fgkkyq4lklghged/lib/libslepc.so > > Thanks, > Cho > ________________________________ > From: Pierre Jolivet > Sent: Wednesday, May 8, 2024 11:01 PM > To: Ng, Cho-Kuen > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] About PC hpddm > > > > On 9 May 2024, at 6:31?AM, Ng, Cho-Kuen via petsc-users wrote: > > This Message Is From an External Sender > This message came from outside your organization. > I used spack to install petsc with hpddm as follows. > > o spack install petsc+hpddm > o spack install slepc ^petsc+hpddm > > I got the following runtime error of finding the library hpddm_petsc. > > argv0 -ksp_converged_reason -ksp_view_final_residual -ksp_type gmres -pc_type hpddm -ksp_monitor -log_view -options_left -device_view > $$$$$$$$$$$$$$ buildmatrix petsc > [6]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [6]PETSC ERROR: Petsc has generated inconsistent data > [6]PETSC ERROR: /global/cfs/cdirs/ace3p/software/spack/var/spack/environments/demo1/.spack-env/view/lib/libhpddm_petsc not found > > Is there something wrong with the install procedure? > > The command lines look OK to me (but I don?t know Spack). > I?m not sure where things are being installed. > Could you please send the path to: > 1) libpetsc.so > 2) libslepc.so > 3) libhpddm_petsc.so > If there is a mismatch between some of these, you may want to set the environment variable SLEPC_DIR or the command line parameter -hpddm_dir. > > Thanks, > Pierre > > Thanks, > Cho > > From cho at slac.stanford.edu Thu May 9 02:47:14 2024 From: cho at slac.stanford.edu (Ng, Cho-Kuen) Date: Thu, 9 May 2024 07:47:14 +0000 Subject: [petsc-users] About PC hpddm In-Reply-To: References: <4E6CD92B-CB1B-4CAB-81B3-5E8E88EA3BB3@joliv.et> Message-ID: Satish, That works. I see that libhpddm_petsc.so is in the spack view lib. Will test how to use it. Thanks, Cho ________________________________ From: Satish Balay Sent: Wednesday, May 8, 2024 11:49 PM To: Ng, Cho-Kuen Cc: Pierre Jolivet ; petsc-users at mcs.anl.gov Subject: Re: [petsc-users] About PC hpddm Try: spack install slepc+hpddm ^petsc+hpddm Satish On Thu, 9 May 2024, Ng, Cho-Kuen via petsc-users wrote: > Pierre, > > petec and slepc libraries are found in the spack directory, but libhpddm_petsc is not. So it is not built during the spack install process. > > ... ... > ./opt/spack/linux-sles15-zen3/gcc-12.3.0/petsc-3.21.0-tdljbqbgvde4zv6jr6tf7bbpae3rfkyg/lib/libpetsc.so > ./var/spack/environments/demo1/.spack-env/._view/6rqhi7tgwui4fjo74fgkkyq4lklghged/lib/libpetsc.so > > ... ... > ./opt/spack/linux-sles15-zen3/gcc-12.3.0/slepc-3.21.0-qhuhkznl5rlhqknsdqervlevup6xiuyt/lib/libslepc.so > ./var/spack/environments/demo1/.spack-env/._view/6rqhi7tgwui4fjo74fgkkyq4lklghged/lib/libslepc.so > > Thanks, > Cho > ________________________________ > From: Pierre Jolivet > Sent: Wednesday, May 8, 2024 11:01 PM > To: Ng, Cho-Kuen > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] About PC hpddm > > > > On 9 May 2024, at 6:31?AM, Ng, Cho-Kuen via petsc-users wrote: > > This Message Is From an External Sender > This message came from outside your organization. > I used spack to install petsc with hpddm as follows. > > o spack install petsc+hpddm > o spack install slepc ^petsc+hpddm > > I got the following runtime error of finding the library hpddm_petsc. > > argv0 -ksp_converged_reason -ksp_view_final_residual -ksp_type gmres -pc_type hpddm -ksp_monitor -log_view -options_left -device_view > $$$$$$$$$$$$$$ buildmatrix petsc > [6]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [6]PETSC ERROR: Petsc has generated inconsistent data > [6]PETSC ERROR: /global/cfs/cdirs/ace3p/software/spack/var/spack/environments/demo1/.spack-env/view/lib/libhpddm_petsc not found > > Is there something wrong with the install procedure? > > The command lines look OK to me (but I don?t know Spack). > I?m not sure where things are being installed. > Could you please send the path to: > 1) libpetsc.so > 2) libslepc.so > 3) libhpddm_petsc.so > If there is a mismatch between some of these, you may want to set the environment variable SLEPC_DIR or the command line parameter -hpddm_dir. > > Thanks, > Pierre > > Thanks, > Cho > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu May 9 07:15:45 2024 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 9 May 2024 08:15:45 -0400 Subject: [petsc-users] HDF5 time step count In-Reply-To: <97876f22-d284-45d8-8993-893e3937004f@auckland.ac.nz> References: <425d849a-eba3-6739-5e71-251e0f29b5fb@auckland.ac.nz> <8735pcd12k.fsf@jedbrown.org> <7639e2f6-d326-e04b-bd8e-53cecb8d794e@auckland.ac.nz> <220daca5-21ea-00df-bc51-ba6124d020ad@auckland.ac.nz> <6aa6644b-c36b-4f4d-b843-d1a1a4adf7bd@auckland.ac.nz> <97876f22-d284-45d8-8993-893e3937004f@auckland.ac.nz> Message-ID: On Wed, May 8, 2024 at 10:46?PM Adrian Croucher wrote: > hi Matt, > On 9/05/24 3:00 am, Matthew Knepley wrote: > > Sorry about the delay. I had lost track of this. Can you look at branch > > knepley/feature-hdf5-seq-len > > I have not made a test yet, but if this works for you, I will make a test > and merge it in. > > Thanks for looking at that. > > I just tried to test it, but there doesn't seem to be a Fortran interface > for DMGetOutputSequenceLength(). > Pushed. > Another odd thing I found was that DMPlexConstructGhostCells() seemed to > complain if I passed in PETSC_NULL_INTEGER for the parameter numGhostCells > (which works ok in v3.21.1) - the compiler complained about a rank-1/scalar > mismatch. Can you no longer pass in a null integer for that? It works if I > declare a dummy integer and pass it in. > Oh, Barry has been redoing the Fortran stubs to try and get more automated. He might have eliminated the hand copy by mistake (since we cannot yet automate checking for NULL). I will ask him. For now, can we put a dummy in? Thanks, Matt > - Adrian > > -- > Dr Adrian Croucher > Senior Research Fellow > Department of Engineering Science > Waipapa Taumata Rau / University of Auckland, New Zealand > email: a.croucher at auckland.ac.nz > tel: +64 (0)9 923 4611 > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!YEiiqBTCh7zSAbsfhvbrdIgnFEYpbB4rUltLdnm3IhphEaC4qf6ny1HMQl-j374n_GwyG96OGNk1F-AzQUVb$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From sawsan.shatanawi at wsu.edu Thu May 9 13:52:54 2024 From: sawsan.shatanawi at wsu.edu (Shatanawi, Sawsan Muhammad) Date: Thu, 9 May 2024 18:52:54 +0000 Subject: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code In-Reply-To: References: <17B6144E-4F3A-4173-9818-4B03669736C4@petsc.dev> <9BD74F88-20FD-4021-B02A-195A58B72282@petsc.dev> Message-ID: Hello everyone, Thank you for your responses and feedback, I checked PFLOTRAN and found that it is a model to simulate groundwater flow, contaminant transport, and other subsurface processes. my goal is not to simulate the groundwater flow, my goal is to develop a code from scratch to simulate the groundwater flow with specific conditions, and then integrate this code with land surface models. Later, the simulation of this code will be on a large scale. I want PETSc to calculate the Jacobian because the system is large and has complex nonlinear behavior, and I don?t risk calculating the derivative by myself. My A-Matrix has parts of source terms that depend on the flow fields, and independent parts will be in the RHS vector. I hope I have answered your questions, and I apologize that I wasn?t clear from the beginning, I was trying to keep my descriptions brief. Bests, Sawsan ________________________________ From: Matthew Knepley Sent: Tuesday, May 7, 2024 5:17 PM To: Shatanawi, Sawsan Muhammad Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code [EXTERNAL EMAIL] On Tue, May 7, 2024 at 2:23?PM Shatanawi, Sawsan Muhammad via petsc-users > wrote: Hello everyone, I hope this email finds you well. My Name is Sawsan Shatanawi, and I was developing a Fortran code for simulating groundwater flow in a 3D system with nonlinear behavior. I solved the nonlinear system using the PCG solver and ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hello everyone, I hope this email finds you well. My Name is Sawsan Shatanawi, and I was developing a Fortran code for simulating groundwater flow in a 3D system with nonlinear behavior. I solved the nonlinear system using the PCG solver and Picard iteration, but I did not get good results although I checked my matrix and RHS and everything, I decided to change my solver to Newton Rapson method. I checked PETSc documents but I have a few questions: 1) My groundwater system is time-dependent, so should I use TS only instead of SNES? You could use TS, but it is not necessary. You could use SNES and your own timestepping. THe advantage of TS is that you can try many different timesteppers without recoding (just like you can try many different linear and nonlinear solvers). 2) My system has its deltaT, would using deltaT as dt affect my solver, or is it better to use TS-PETSc dt? Also, would using PETSc dt affect the simulation of the groundwater system It sounds like your dt comes from your timestepper. If you use TS, you would use the dt from that. 3) I want my Jacobian matrix to be calculated by PETSc automatically PETSc can calculate a full Jacobian for smaller problems, or a finite-difference Jacobian for any problem (but this impacts the solver). It should be straightfoward to code up the analytic Jacobian. Is there a reason it would be a problem? 4) Do I need to define and calculate the residual vector? Yes. My A-Matrix contains coefficients and external sources and my RHS vector includes the boundary conditions It is strange that your matrix would contain source terms. Do they depend on the flow fields? Barry is right, you should consider PFlotran, and at least know why it would not work for your problem if you don't use it. Thanks, Matt Please find the attached file contains a draft of my code Thank you in advance for your time and help. Best regards, Sawsan ________________________________ From: Shatanawi, Sawsan Muhammad > Sent: Tuesday, January 16, 2024 10:43 AM To: Junchao Zhang > Cc: Barry Smith >; Matthew Knepley >; Mark Adams >; petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code Hello all, Thank you for your valuable help. I will do your recommendations and hope it will run without any issues. Bests, Sawsan ________________________________ From: Junchao Zhang > Sent: Friday, January 12, 2024 8:46 AM To: Shatanawi, Sawsan Muhammad > Cc: Barry Smith >; Matthew Knepley >; Mark Adams >; petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code [EXTERNAL EMAIL] Hi, Sawsan, First in test_main.F90, you need to call VecGetArrayF90(temp_solution, H_vector, ierr) and VecRestoreArrayF90 (temp_solution, H_vector, ierr) as Barry mentioned. Secondly, in the loop of test_main.F90, it calls GW_solver(). Within it, it calls PetscInitialize()/PetscFinalize(). But without MPI being initialized, PetscInitialize()/PetscFinalize() can only be called once. do timestep =2 , NTSP call GW_boundary_conditions(timestep-1) !print *,HNEW(1,1,1) call GW_elevation() ! print *, GWTOP(2,2,2) call GW_conductance() ! print *, CC(2,2,2) call GW_recharge() ! print *, B_Rech(5,4) call GW_pumping(timestep-1) ! print *, B_pump(2,2,2) call GW_SW(timestep-1) print *,B_RIVER (2,2,2) call GW_solver(timestep-1,N) call GW_deallocate_loop() end do A solution is to delete PetscInitialize()/PetscFinalize() in GW_solver_try.F90 and move it to test_main.F90, outside the do loop. diff --git a/test_main.F90 b/test_main.F90 index b5997c55..107bd3ee 100644 --- a/test_main.F90 +++ b/test_main.F90 @@ -1,5 +1,6 @@ program test_GW +#include use petsc use GW_constants use GW_param_by_user @@ -8,6 +9,9 @@ program test_GW implicit none integer :: N integer :: timestep + PetscErrorCode ierr + + call PetscInitialize(ierr) call GW_domain(N) !print *, "N=",N !print *, DELTAT @@ -37,4 +41,5 @@ program test_GW end do print *, HNEW(NCOL,3,2) call GW_deallocate () + call PetscFinalize(ierr) end program test_GW With that, the MPI error will be fixed. The code could run to gw_deallocate () before abort. There are other memory errors. You can install/use valgrind to fix them. Run it with valgrind ./GW.exe and look through the output Thanks. --Junchao Zhang On Thu, Jan 11, 2024 at 10:49?PM Shatanawi, Sawsan Muhammad > wrote: Hello, Thank you all for your help. I have changed VecGetArray to VecGetArrayF90, and the location of destory call. but I want to make sure that VecGet ArrayF90 is to make a new array( vector) that I can use in the rest of my Fortran code? when I run it and debugged it, I got 5.2000000E-03 50.00000 10.00000 0.0000000E+00 PETSC: Attaching gdb to /weka/data/lab/richey/sawsan/GW_CODE/code2024/SS_GWM/./GW.exe of pid 33065 on display :0.0 on machine sn16 Unable to start debugger in xterm: No such file or directory 0.0000000E+00 Attempting to use an MPI routine after finalizing MPICH srun: error: sn16: task 0: Exited with exit code 1 [sawsan.shatanawi at login-p2n02 SS_GWM]$ gdb ./GW/exe GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-100.el7 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later > This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". For bug reporting instructions, please see: >... ./GW/exe: No such file or directory. (gdb) run Starting program: No executable file specified. Use the "file" or "exec-file" command. (gdb) bt No stack. (gdb) If the highlighted line is the error, I don't know why when I write gdb , it does not show me the location of error The code : sshatanawi/SS_GWM (github.com) I really appreciate your helps Sawsan ________________________________ From: Barry Smith > Sent: Wednesday, January 10, 2024 5:35 PM To: Junchao Zhang > Cc: Shatanawi, Sawsan Muhammad >; Mark Adams >; petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code [EXTERNAL EMAIL] On Jan 10, 2024, at 6:49?PM, Junchao Zhang > wrote: Hi, Sawsan, I could build your code and I also could gdb it. $ gdb ./GW.exe ... $ Thread 1 "GW.exe" received signal SIGSEGV, Segmentation fault. 0x00007ffff1e6d44f in vecgetarray_ (x=0x7fffffffa718, fa=0x0, ia=0x7fffffffa75c, ierr=0x0) at /scratch/jczhang/petsc/src/vec/vec/interface/ftn-custom/zvectorf.c:257 257 *ierr = VecGetArray(*x, &lx); (gdb) bt #0 0x00007ffff1e6d44f in vecgetarray_ (x=0x7fffffffa718, fa=0x0, ia=0x7fffffffa75c, ierr=0x0) at /scratch/jczhang/petsc/src/vec/vec/interface/ftn-custom/zvectorf.c:257 #1 0x000000000040b6e3 in gw_solver (t_s=1.40129846e-45, n=300) at GW_solver_try.F90:169 #2 0x000000000040c6a8 in test_gw () at test_main.F90:35 ierr=0x0 caused the segfault. See https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/VecGetArray/*vecgetarray__;Iw!!G_uCfscf7eWS!ZyzaNi6xVJMH9G4cGPFSeN-rj9OU0HopNoXpUCSfoIJPEL7ZTLZFhirDsSFgOcEeyKWSmo0cqE0gWlyrDqlueeni7CcptjGtGw$ , you should use VecGetArrayF90 instead. BTW, Barry, the code https://urldefense.us/v3/__https://github.com/sshatanawi/SS_GWM/blob/main/GW_solver_try.F90*L169__;Iw!!G_uCfscf7eWS!ZyzaNi6xVJMH9G4cGPFSeN-rj9OU0HopNoXpUCSfoIJPEL7ZTLZFhirDsSFgOcEeyKWSmo0cqE0gWlyrDqlueeni7Cc1mJt7xw$ has "call VecGetArray(temp_solution, H_vector, ierr)". I don't find petsc Fortran examples doing VecGetArray. Do we still support it? This is not the correct calling sequence for VecGetArray() from Fortran. Regardless, definitely should not be writing any new code that uses VecGetArray() from Fortran. Should use VecGetArrayF90(). --Junchao Zhang On Wed, Jan 10, 2024 at 2:38?PM Shatanawi, Sawsan Muhammad via petsc-users > wrote: Hello all, I hope you are doing well. Generally, I use gdb to debug the code. I got the attached error message. I have tried to add the flag -start_in_debugger in the make file, but it didn't work, so it seems I was doing it in the wrong way This is the link for the whole code: sshatanawi/SS_GWM (github.com) [https://urldefense.us/v3/__https://opengraph.githubassets.com/9eb6cd14baf12f04848ed209b6f502415eb531bdd7b3a5f9696af68663b870c0/sshatanawi/SS_GWM__;!!G_uCfscf7eWS!ZyzaNi6xVJMH9G4cGPFSeN-rj9OU0HopNoXpUCSfoIJPEL7ZTLZFhirDsSFgOcEeyKWSmo0cqE0gWlyrDqlueeni7CdtM497HQ$ ] GitHub - sshatanawi/SS_GWM Contribute to sshatanawi/SS_GWM development by creating an account on GitHub. github.com ? You can read the description of the code in " Model Desprciption.pdf" the compiling file is makefile_f90 where you can find the linked code files I really appreciate your help Bests, Sawsan ________________________________ From: Mark Adams > Sent: Friday, January 5, 2024 4:53 AM To: Shatanawi, Sawsan Muhammad > Cc: Matthew Knepley >; petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code [EXTERNAL EMAIL] This is a segv. As Matt said, you need to use a debugger for this or add print statements to narrow down the place where this happens. You will need to learn how to use debuggers to do your project so you might as well start now. If you have a machine with a GUI debugger that is easier but command line debuggers are good to learn anyway. I tend to run debuggers directly (eg, lldb ./a.out -- program-args ...) and use a GUI debugger (eg, Totalview or DDT) if available. Mark On Wed, Dec 20, 2023 at 10:02?PM Shatanawi, Sawsan Muhammad via petsc-users > wrote: Hello Matthew, Thank you for your help. I am sorry that I keep coming back with my error messages, but I reached a point that I don't know how to fix them, and I don't understand them easily. The list of errors is getting shorter, now I am getting the attached error messages Thank you again, Sawsan ________________________________ From: Matthew Knepley > Sent: Wednesday, December 20, 2023 6:54 PM To: Shatanawi, Sawsan Muhammad > Cc: Barry Smith >; petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code [EXTERNAL EMAIL] On Wed, Dec 20, 2023 at 9:49?PM Shatanawi, Sawsan Muhammad via petsc-users > wrote: Hello Barry, Thank you a lot for your help, Now I am getting the attached error message. Do not destroy the PC from KSPGetPC() THanks, Matt Bests, Sawsan ________________________________ From: Barry Smith > Sent: Wednesday, December 20, 2023 6:32 PM To: Shatanawi, Sawsan Muhammad > Cc: Mark Adams >; petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code [EXTERNAL EMAIL] Instead of call PCCreate(PETSC_COMM_WORLD, pc, ierr) call PCSetType(pc, PCILU,ierr) ! Choose a preconditioner type (ILU) call KSPSetPC(ksp, pc,ierr) ! Associate the preconditioner with the KSP solver do call KSPGetPC(ksp,pc,ierr) call PCSetType(pc, PCILU,ierr) Do not call KSPSetUp(). It will be taken care of automatically during the solve On Dec 20, 2023, at 8:52?PM, Shatanawi, Sawsan Muhammad via petsc-users > wrote: Hello, I don't think that I set preallocation values when I created the matrix, would you please have look at my code. It is just the petsc related part from my code. I was able to fix some of the error messages. Now I have a new set of error messages related to the KSP solver (attached) I appreciate your help Sawsan ________________________________ From: Mark Adams > Sent: Wednesday, December 20, 2023 6:44 AM To: Shatanawi, Sawsan Muhammad > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code [EXTERNAL EMAIL] Did you set preallocation values when you created the matrix? Don't do that. On Wed, Dec 20, 2023 at 9:36?AM Shatanawi, Sawsan Muhammad > wrote: Hello, I am trying to create a sparse matrix( which is as I believe a zero matrix) then adding some nonzero elements to it over a loop, then assembling it Get Outlook for iOS ________________________________ From: Mark Adams > Sent: Wednesday, December 20, 2023 2:48 AM To: Shatanawi, Sawsan Muhammad > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code [EXTERNAL EMAIL] I am guessing that you are creating a matrix, adding to it, finalizing it ("assembly"), and then adding to it again, which is fine, but you are adding new non-zeros to the sparsity pattern. If this is what you want then you can tell the matrix to let you do that. Otherwise you have a bug. Mark On Tue, Dec 19, 2023 at 9:50?PM Shatanawi, Sawsan Muhammad via petsc-users > wrote: Hello everyone, I hope this email finds you well. My Name is Sawsan Shatanawi, and I am currently working on developing a Fortran code for simulating groundwater flow in a 3D system. The code involves solving a nonlinear system, and I have created the matrix to be solved using the PCG solver and Picard iteration. However, when I tried to assign it as a PETSc matrix I started getting a lot of error messages. I am kindly asking if someone can help me, I would be happy to share my code with him/her. Please find the attached file contains a list of errors I have gotten Thank you in advance for your time and assistance. Best regards, Sawsan -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!ZyzaNi6xVJMH9G4cGPFSeN-rj9OU0HopNoXpUCSfoIJPEL7ZTLZFhirDsSFgOcEeyKWSmo0cqE0gWlyrDqlueeni7CeS4B6ueA$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!ZyzaNi6xVJMH9G4cGPFSeN-rj9OU0HopNoXpUCSfoIJPEL7ZTLZFhirDsSFgOcEeyKWSmo0cqE0gWlyrDqlueeni7CeS4B6ueA$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Thu May 9 14:05:35 2024 From: bsmith at petsc.dev (Barry Smith) Date: Thu, 9 May 2024 15:05:35 -0400 Subject: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code In-Reply-To: References: <17B6144E-4F3A-4173-9818-4B03669736C4@petsc.dev> <9BD74F88-20FD-4021-B02A-195A58B72282@petsc.dev> Message-ID: <0636729D-AE8D-4DE0-8606-E0948733814C@petsc.dev> > On May 9, 2024, at 2:52?PM, Shatanawi, Sawsan Muhammad via petsc-users wrote: > > This Message Is From an External Sender > This message came from outside your organization. > Hello everyone, > > Thank you for your responses and feedback, > > I checked PFLOTRAN and found that it is a model to simulate groundwater flow, contaminant transport, and other subsurface processes. > my goal is not to simulate the groundwater flow, my goal is to develop a code from scratch to simulate the groundwater flow with specific conditions, and then integrate this code with land surface models. > Later, the simulation of this code will be on a large scale. > > I want PETSc to calculate the Jacobian because the system is large and has complex nonlinear behavior, and I don?t risk calculating the derivative by myself. > My A-Matrix has parts of source terms that depend on the flow fields, and independent parts will be in the RHS vector. With coloring SNESComputeJacobianDefaultColor() PETSc can compute Jacobian's pretty efficiently. You do not to provide the residual function and you need to provide the nonzero pattern of the sparse Jacobian; that is what residual components f_i are coupled to what input variables in the array x_i. This information comes from your PDE and discretization and appears implicitly in your residual function. Barry > > I hope I have answered your questions, and I apologize that I wasn?t clear from the beginning, I was trying to keep my descriptions brief. > > Bests, > Sawsan > From: Matthew Knepley > > Sent: Tuesday, May 7, 2024 5:17 PM > To: Shatanawi, Sawsan Muhammad > > Cc: petsc-users at mcs.anl.gov > > Subject: Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code > > [EXTERNAL EMAIL] > On Tue, May 7, 2024 at 2:23?PM Shatanawi, Sawsan Muhammad via petsc-users > wrote: > This Message Is From an External Sender > This message came from outside your organization. > > Hello everyone, > > I hope this email finds you well. > > > My Name is Sawsan Shatanawi, and I was developing a Fortran code for simulating groundwater flow in a 3D system with nonlinear behavior. I solved the nonlinear system using the PCG solver and Picard iteration, but I did not get good results although I checked my matrix and RHS and everything, I decided to change my solver to Newton Rapson method. > I checked PETSc documents but I have a few questions: > 1) My groundwater system is time-dependent, so should I use TS only instead of SNES? > > You could use TS, but it is not necessary. You could use SNES and your own timestepping. THe advantage of TS is that you can try many different timesteppers without recoding (just like you can > try many different linear and nonlinear solvers). > > 2) My system has its deltaT, would using deltaT as dt affect my solver, or is it better to use TS-PETSc dt? Also, would using PETSc dt affect the simulation of the groundwater system > > It sounds like your dt comes from your timestepper. If you use TS, you would use the dt from that. > > 3) I want my Jacobian matrix to be calculated by PETSc automatically > > PETSc can calculate a full Jacobian for smaller problems, or a finite-difference Jacobian for any problem (but this impacts the solver). It should be straightfoward to code up the analytic Jacobian. Is there a reason it would be a problem? > > 4) Do I need to define and calculate the residual vector? > > Yes. > > My A-Matrix contains coefficients and external sources and my RHS vector includes the boundary conditions > > It is strange that your matrix would contain source terms. Do they depend on the flow fields? > > Barry is right, you should consider PFlotran, and at least know why it would not work for your problem if you don't use it. > > Thanks, > > Matt > > > Please find the attached file contains a draft of my code > > Thank you in advance for your time and help. > > Best regards, > > Sawsan > > > From: Shatanawi, Sawsan Muhammad > > Sent: Tuesday, January 16, 2024 10:43 AM > To: Junchao Zhang > > Cc: Barry Smith >; Matthew Knepley >; Mark Adams >; petsc-users at mcs.anl.gov > > Subject: Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code > > Hello all, > > Thank you for your valuable help. I will do your recommendations and hope it will run without any issues. > > Bests, > Sawsan > > From: Junchao Zhang > > Sent: Friday, January 12, 2024 8:46 AM > To: Shatanawi, Sawsan Muhammad > > Cc: Barry Smith >; Matthew Knepley >; Mark Adams >; petsc-users at mcs.anl.gov > > Subject: Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code > > [EXTERNAL EMAIL] > Hi, Sawsan, > First in test_main.F90, you need to call VecGetArrayF90(temp_solution, H_vector, ierr) and VecRestoreArrayF90 (temp_solution, H_vector, ierr) as Barry mentioned. > Secondly, in the loop of test_main.F90, it calls GW_solver(). Within it, it calls PetscInitialize()/PetscFinalize(). But without MPI being initialized, PetscInitialize()/PetscFinalize() can only be called once. > do timestep =2 , NTSP > call GW_boundary_conditions(timestep-1) > !print *,HNEW(1,1,1) > call GW_elevation() > ! print *, GWTOP(2,2,2) > call GW_conductance() > ! print *, CC(2,2,2) > call GW_recharge() > ! print *, B_Rech(5,4) > call GW_pumping(timestep-1) > ! print *, B_pump(2,2,2) > call GW_SW(timestep-1) > print *,B_RIVER (2,2,2) > call GW_solver(timestep-1,N) > call GW_deallocate_loop() > end do > > A solution is to delete PetscInitialize()/PetscFinalize() in GW_solver_try.F90 and move it to test_main.F90, outside the do loop. > > diff --git a/test_main.F90 b/test_main.F90 > index b5997c55..107bd3ee 100644 > --- a/test_main.F90 > +++ b/test_main.F90 > @@ -1,5 +1,6 @@ > program test_GW > > +#include > use petsc > use GW_constants > use GW_param_by_user > @@ -8,6 +9,9 @@ program test_GW > implicit none > integer :: N > integer :: timestep > + PetscErrorCode ierr > + > + call PetscInitialize(ierr) > call GW_domain(N) > !print *, "N=",N > !print *, DELTAT > @@ -37,4 +41,5 @@ program test_GW > end do > print *, HNEW(NCOL,3,2) > call GW_deallocate () > + call PetscFinalize(ierr) > end program test_GW > > With that, the MPI error will be fixed. The code could run to gw_deallocate () before abort. There are other memory errors. You can install/use valgrind to fix them. Run it with valgrind ./GW.exe and look through the output > > > Thanks. > --Junchao Zhang > > > On Thu, Jan 11, 2024 at 10:49?PM Shatanawi, Sawsan Muhammad > wrote: > Hello, > > Thank you all for your help. > > I have changed VecGetArray to VecGetArrayF90, and the location of destory call. but I want to make sure that VecGet ArrayF90 is to make a new array( vector) that I can use in the rest of my Fortran code? > > when I run it and debugged it, I got > > 5.2000000E-03 > 50.00000 > 10.00000 > 0.0000000E+00 > PETSC: Attaching gdb to /weka/data/lab/richey/sawsan/GW_CODE/code2024/SS_GWM/./GW.exe of pid 33065 on display :0.0 on machine sn16 > Unable to start debugger in xterm: No such file or directory > 0.0000000E+00 > Attempting to use an MPI routine after finalizing MPICH > srun: error: sn16: task 0: Exited with exit code 1 > [sawsan.shatanawi at login-p2n02 SS_GWM]$ gdb ./GW/exe > GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-100.el7 > Copyright (C) 2013 Free Software Foundation, Inc. > License GPLv3+: GNU GPL version 3 or later > > This is free software: you are free to change and redistribute it. > There is NO WARRANTY, to the extent permitted by law. Type "show copying" > and "show warranty" for details. > This GDB was configured as "x86_64-redhat-linux-gnu". > For bug reporting instructions, please see: > >... > ./GW/exe: No such file or directory. > (gdb) run > Starting program: > No executable file specified. > Use the "file" or "exec-file" command. > (gdb) bt > No stack. > (gdb) > > If the highlighted line is the error, I don't know why when I write gdb , it does not show me the location of error > The code : sshatanawi/SS_GWM (github.com) > > I really appreciate your helps > > Sawsan > From: Barry Smith > > Sent: Wednesday, January 10, 2024 5:35 PM > To: Junchao Zhang > > Cc: Shatanawi, Sawsan Muhammad >; Mark Adams >; petsc-users at mcs.anl.gov > > Subject: Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code > > [EXTERNAL EMAIL] > > >> On Jan 10, 2024, at 6:49?PM, Junchao Zhang > wrote: >> >> Hi, Sawsan, >> I could build your code and I also could gdb it. >> >> $ gdb ./GW.exe >> ... >> $ Thread 1 "GW.exe" received signal SIGSEGV, Segmentation fault. >> 0x00007ffff1e6d44f in vecgetarray_ (x=0x7fffffffa718, fa=0x0, ia=0x7fffffffa75c, ierr=0x0) at /scratch/jczhang/petsc/src/vec/vec/interface/ftn-custom/zvectorf.c:257 >> 257 *ierr = VecGetArray(*x, &lx); >> (gdb) bt >> #0 0x00007ffff1e6d44f in vecgetarray_ (x=0x7fffffffa718, fa=0x0, ia=0x7fffffffa75c, ierr=0x0) at /scratch/jczhang/petsc/src/vec/vec/interface/ftn-custom/zvectorf.c:257 >> #1 0x000000000040b6e3 in gw_solver (t_s=1.40129846e-45, n=300) at GW_solver_try.F90:169 >> #2 0x000000000040c6a8 in test_gw () at test_main.F90:35 >> >> ierr=0x0 caused the segfault. See https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/VecGetArray/*vecgetarray__;Iw!!G_uCfscf7eWS!dDnXrM4ZRvwOzTHj5gfqE4H6YFNE91xB_W_jxDV9ovdPFC-1DOJzuv429C_JeS_ZQwb58XaiXy-ws0k9qRm274Q$ , you should use VecGetArrayF90 instead. >> >> BTW, Barry, the code https://urldefense.us/v3/__https://github.com/sshatanawi/SS_GWM/blob/main/GW_solver_try.F90*L169__;Iw!!G_uCfscf7eWS!dDnXrM4ZRvwOzTHj5gfqE4H6YFNE91xB_W_jxDV9ovdPFC-1DOJzuv429C_JeS_ZQwb58XaiXy-ws0k9AD0UUjo$ has "call VecGetArray(temp_solution, H_vector, ierr)". I don't find petsc Fortran examples doing VecGetArray. Do we still support it? > > This is not the correct calling sequence for VecGetArray() from Fortran. > > Regardless, definitely should not be writing any new code that uses VecGetArray() from Fortran. Should use VecGetArrayF90(). > >> >> --Junchao Zhang >> >> >> On Wed, Jan 10, 2024 at 2:38?PM Shatanawi, Sawsan Muhammad via petsc-users > wrote: >> Hello all, >> >> I hope you are doing well. >> >> Generally, I use gdb to debug the code. >> I got the attached error message. >> >> I have tried to add the flag -start_in_debugger in the make file, but it didn't work, so it seems I was doing it in the wrong way >> >> This is the link for the whole code: sshatanawi/SS_GWM (github.com) >> >> GitHub - sshatanawi/SS_GWM >> Contribute to sshatanawi/SS_GWM development by creating an account on GitHub. >> github.com >> ? >> >> You can read the description of the code in " Model Desprciption.pdf" >> the compiling file is makefile_f90 where you can find the linked code files >> >> I really appreciate your help >> >> Bests, >> Sawsan >> From: Mark Adams > >> Sent: Friday, January 5, 2024 4:53 AM >> To: Shatanawi, Sawsan Muhammad > >> Cc: Matthew Knepley >; petsc-users at mcs.anl.gov > >> Subject: Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code >> >> [EXTERNAL EMAIL] >> This is a segv. As Matt said, you need to use a debugger for this or add print statements to narrow down the place where this happens. >> >> You will need to learn how to use debuggers to do your project so you might as well start now. >> >> If you have a machine with a GUI debugger that is easier but command line debuggers are good to learn anyway. >> >> I tend to run debuggers directly (eg, lldb ./a.out -- program-args ...) and use a GUI debugger (eg, Totalview or DDT) if available. >> >> Mark >> >> >> On Wed, Dec 20, 2023 at 10:02?PM Shatanawi, Sawsan Muhammad via petsc-users > wrote: >> Hello Matthew, >> >> Thank you for your help. I am sorry that I keep coming back with my error messages, but I reached a point that I don't know how to fix them, and I don't understand them easily. >> The list of errors is getting shorter, now I am getting the attached error messages >> >> Thank you again, >> >> Sawsan >> From: Matthew Knepley > >> Sent: Wednesday, December 20, 2023 6:54 PM >> To: Shatanawi, Sawsan Muhammad > >> Cc: Barry Smith >; petsc-users at mcs.anl.gov > >> Subject: Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code >> >> [EXTERNAL EMAIL] >> On Wed, Dec 20, 2023 at 9:49?PM Shatanawi, Sawsan Muhammad via petsc-users > wrote: >> Hello Barry, >> >> Thank you a lot for your help, Now I am getting the attached error message. >> >> Do not destroy the PC from KSPGetPC() >> >> THanks, >> >> Matt >> >> Bests, >> Sawsan >> From: Barry Smith > >> Sent: Wednesday, December 20, 2023 6:32 PM >> To: Shatanawi, Sawsan Muhammad > >> Cc: Mark Adams >; petsc-users at mcs.anl.gov > >> Subject: Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code >> >> [EXTERNAL EMAIL] >> >> Instead of >> >> call PCCreate(PETSC_COMM_WORLD, pc, ierr) >> call PCSetType(pc, PCILU,ierr) ! Choose a preconditioner type (ILU) >> call KSPSetPC(ksp, pc,ierr) ! Associate the preconditioner with the KSP solver >> >> do >> >> call KSPGetPC(ksp,pc,ierr) >> call PCSetType(pc, PCILU,ierr) >> >> Do not call KSPSetUp(). It will be taken care of automatically during the solve >> >> >> >>> On Dec 20, 2023, at 8:52?PM, Shatanawi, Sawsan Muhammad via petsc-users > wrote: >>> >>> Hello, >>> I don't think that I set preallocation values when I created the matrix, would you please have look at my code. It is just the petsc related part from my code. >>> I was able to fix some of the error messages. Now I have a new set of error messages related to the KSP solver (attached) >>> >>> I appreciate your help >>> >>> Sawsan >>> From: Mark Adams > >>> Sent: Wednesday, December 20, 2023 6:44 AM >>> To: Shatanawi, Sawsan Muhammad > >>> Cc: petsc-users at mcs.anl.gov > >>> Subject: Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code >>> >>> [EXTERNAL EMAIL] >>> Did you set preallocation values when you created the matrix? >>> Don't do that. >>> >>> On Wed, Dec 20, 2023 at 9:36?AM Shatanawi, Sawsan Muhammad > wrote: >>> Hello, >>> >>> I am trying to create a sparse matrix( which is as I believe a zero matrix) then adding some nonzero elements to it over a loop, then assembling it >>> >>> Get Outlook for iOS >>> From: Mark Adams > >>> Sent: Wednesday, December 20, 2023 2:48 AM >>> To: Shatanawi, Sawsan Muhammad > >>> Cc: petsc-users at mcs.anl.gov > >>> Subject: Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code >>> >>> [EXTERNAL EMAIL] >>> I am guessing that you are creating a matrix, adding to it, finalizing it ("assembly"), and then adding to it again, which is fine, but you are adding new non-zeros to the sparsity pattern. >>> If this is what you want then you can tell the matrix to let you do that. >>> Otherwise you have a bug. >>> >>> Mark >>> >>> On Tue, Dec 19, 2023 at 9:50?PM Shatanawi, Sawsan Muhammad via petsc-users > wrote: >>> Hello everyone, >>> >>> I hope this email finds you well. >>> >>> My Name is Sawsan Shatanawi, and I am currently working on developing a Fortran code for simulating groundwater flow in a 3D system. The code involves solving a nonlinear system, and I have created the matrix to be solved using the PCG solver and Picard iteration. However, when I tried to assign it as a PETSc matrix I started getting a lot of error messages. >>> >>> I am kindly asking if someone can help me, I would be happy to share my code with him/her. >>> >>> Please find the attached file contains a list of errors I have gotten >>> >>> Thank you in advance for your time and assistance. >>> Best regards, >>> >>> Sawsan >>> >>> >>> >> >> >> >> -- >> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >> -- Norbert Wiener >> >> https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!dDnXrM4ZRvwOzTHj5gfqE4H6YFNE91xB_W_jxDV9ovdPFC-1DOJzuv429C_JeS_ZQwb58XaiXy-ws0k9zTmU4k4$ > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!dDnXrM4ZRvwOzTHj5gfqE4H6YFNE91xB_W_jxDV9ovdPFC-1DOJzuv429C_JeS_ZQwb58XaiXy-ws0k9zTmU4k4$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Thu May 9 14:05:35 2024 From: bsmith at petsc.dev (Barry Smith) Date: Thu, 9 May 2024 15:05:35 -0400 Subject: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code In-Reply-To: References: <17B6144E-4F3A-4173-9818-4B03669736C4@petsc.dev> <9BD74F88-20FD-4021-B02A-195A58B72282@petsc.dev> Message-ID: <0636729D-AE8D-4DE0-8606-E0948733814C@petsc.dev> > On May 9, 2024, at 2:52?PM, Shatanawi, Sawsan Muhammad via petsc-users wrote: > > This Message Is From an External Sender > This message came from outside your organization. > Hello everyone, > > Thank you for your responses and feedback, > > I checked PFLOTRAN and found that it is a model to simulate groundwater flow, contaminant transport, and other subsurface processes. > my goal is not to simulate the groundwater flow, my goal is to develop a code from scratch to simulate the groundwater flow with specific conditions, and then integrate this code with land surface models. > Later, the simulation of this code will be on a large scale. > > I want PETSc to calculate the Jacobian because the system is large and has complex nonlinear behavior, and I don?t risk calculating the derivative by myself. > My A-Matrix has parts of source terms that depend on the flow fields, and independent parts will be in the RHS vector. With coloring SNESComputeJacobianDefaultColor() PETSc can compute Jacobian's pretty efficiently. You do not to provide the residual function and you need to provide the nonzero pattern of the sparse Jacobian; that is what residual components f_i are coupled to what input variables in the array x_i. This information comes from your PDE and discretization and appears implicitly in your residual function. Barry > > I hope I have answered your questions, and I apologize that I wasn?t clear from the beginning, I was trying to keep my descriptions brief. > > Bests, > Sawsan > From: Matthew Knepley > > Sent: Tuesday, May 7, 2024 5:17 PM > To: Shatanawi, Sawsan Muhammad > > Cc: petsc-users at mcs.anl.gov > > Subject: Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code > > [EXTERNAL EMAIL] > On Tue, May 7, 2024 at 2:23?PM Shatanawi, Sawsan Muhammad via petsc-users > wrote: > This Message Is From an External Sender > This message came from outside your organization. > > Hello everyone, > > I hope this email finds you well. > > > My Name is Sawsan Shatanawi, and I was developing a Fortran code for simulating groundwater flow in a 3D system with nonlinear behavior. I solved the nonlinear system using the PCG solver and Picard iteration, but I did not get good results although I checked my matrix and RHS and everything, I decided to change my solver to Newton Rapson method. > I checked PETSc documents but I have a few questions: > 1) My groundwater system is time-dependent, so should I use TS only instead of SNES? > > You could use TS, but it is not necessary. You could use SNES and your own timestepping. THe advantage of TS is that you can try many different timesteppers without recoding (just like you can > try many different linear and nonlinear solvers). > > 2) My system has its deltaT, would using deltaT as dt affect my solver, or is it better to use TS-PETSc dt? Also, would using PETSc dt affect the simulation of the groundwater system > > It sounds like your dt comes from your timestepper. If you use TS, you would use the dt from that. > > 3) I want my Jacobian matrix to be calculated by PETSc automatically > > PETSc can calculate a full Jacobian for smaller problems, or a finite-difference Jacobian for any problem (but this impacts the solver). It should be straightfoward to code up the analytic Jacobian. Is there a reason it would be a problem? > > 4) Do I need to define and calculate the residual vector? > > Yes. > > My A-Matrix contains coefficients and external sources and my RHS vector includes the boundary conditions > > It is strange that your matrix would contain source terms. Do they depend on the flow fields? > > Barry is right, you should consider PFlotran, and at least know why it would not work for your problem if you don't use it. > > Thanks, > > Matt > > > Please find the attached file contains a draft of my code > > Thank you in advance for your time and help. > > Best regards, > > Sawsan > > > From: Shatanawi, Sawsan Muhammad > > Sent: Tuesday, January 16, 2024 10:43 AM > To: Junchao Zhang > > Cc: Barry Smith >; Matthew Knepley >; Mark Adams >; petsc-users at mcs.anl.gov > > Subject: Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code > > Hello all, > > Thank you for your valuable help. I will do your recommendations and hope it will run without any issues. > > Bests, > Sawsan > > From: Junchao Zhang > > Sent: Friday, January 12, 2024 8:46 AM > To: Shatanawi, Sawsan Muhammad > > Cc: Barry Smith >; Matthew Knepley >; Mark Adams >; petsc-users at mcs.anl.gov > > Subject: Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code > > [EXTERNAL EMAIL] > Hi, Sawsan, > First in test_main.F90, you need to call VecGetArrayF90(temp_solution, H_vector, ierr) and VecRestoreArrayF90 (temp_solution, H_vector, ierr) as Barry mentioned. > Secondly, in the loop of test_main.F90, it calls GW_solver(). Within it, it calls PetscInitialize()/PetscFinalize(). But without MPI being initialized, PetscInitialize()/PetscFinalize() can only be called once. > do timestep =2 , NTSP > call GW_boundary_conditions(timestep-1) > !print *,HNEW(1,1,1) > call GW_elevation() > ! print *, GWTOP(2,2,2) > call GW_conductance() > ! print *, CC(2,2,2) > call GW_recharge() > ! print *, B_Rech(5,4) > call GW_pumping(timestep-1) > ! print *, B_pump(2,2,2) > call GW_SW(timestep-1) > print *,B_RIVER (2,2,2) > call GW_solver(timestep-1,N) > call GW_deallocate_loop() > end do > > A solution is to delete PetscInitialize()/PetscFinalize() in GW_solver_try.F90 and move it to test_main.F90, outside the do loop. > > diff --git a/test_main.F90 b/test_main.F90 > index b5997c55..107bd3ee 100644 > --- a/test_main.F90 > +++ b/test_main.F90 > @@ -1,5 +1,6 @@ > program test_GW > > +#include > use petsc > use GW_constants > use GW_param_by_user > @@ -8,6 +9,9 @@ program test_GW > implicit none > integer :: N > integer :: timestep > + PetscErrorCode ierr > + > + call PetscInitialize(ierr) > call GW_domain(N) > !print *, "N=",N > !print *, DELTAT > @@ -37,4 +41,5 @@ program test_GW > end do > print *, HNEW(NCOL,3,2) > call GW_deallocate () > + call PetscFinalize(ierr) > end program test_GW > > With that, the MPI error will be fixed. The code could run to gw_deallocate () before abort. There are other memory errors. You can install/use valgrind to fix them. Run it with valgrind ./GW.exe and look through the output > > > Thanks. > --Junchao Zhang > > > On Thu, Jan 11, 2024 at 10:49?PM Shatanawi, Sawsan Muhammad > wrote: > Hello, > > Thank you all for your help. > > I have changed VecGetArray to VecGetArrayF90, and the location of destory call. but I want to make sure that VecGet ArrayF90 is to make a new array( vector) that I can use in the rest of my Fortran code? > > when I run it and debugged it, I got > > 5.2000000E-03 > 50.00000 > 10.00000 > 0.0000000E+00 > PETSC: Attaching gdb to /weka/data/lab/richey/sawsan/GW_CODE/code2024/SS_GWM/./GW.exe of pid 33065 on display :0.0 on machine sn16 > Unable to start debugger in xterm: No such file or directory > 0.0000000E+00 > Attempting to use an MPI routine after finalizing MPICH > srun: error: sn16: task 0: Exited with exit code 1 > [sawsan.shatanawi at login-p2n02 SS_GWM]$ gdb ./GW/exe > GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-100.el7 > Copyright (C) 2013 Free Software Foundation, Inc. > License GPLv3+: GNU GPL version 3 or later > > This is free software: you are free to change and redistribute it. > There is NO WARRANTY, to the extent permitted by law. Type "show copying" > and "show warranty" for details. > This GDB was configured as "x86_64-redhat-linux-gnu". > For bug reporting instructions, please see: > >... > ./GW/exe: No such file or directory. > (gdb) run > Starting program: > No executable file specified. > Use the "file" or "exec-file" command. > (gdb) bt > No stack. > (gdb) > > If the highlighted line is the error, I don't know why when I write gdb , it does not show me the location of error > The code : sshatanawi/SS_GWM (github.com) > > I really appreciate your helps > > Sawsan > From: Barry Smith > > Sent: Wednesday, January 10, 2024 5:35 PM > To: Junchao Zhang > > Cc: Shatanawi, Sawsan Muhammad >; Mark Adams >; petsc-users at mcs.anl.gov > > Subject: Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code > > [EXTERNAL EMAIL] > > >> On Jan 10, 2024, at 6:49?PM, Junchao Zhang > wrote: >> >> Hi, Sawsan, >> I could build your code and I also could gdb it. >> >> $ gdb ./GW.exe >> ... >> $ Thread 1 "GW.exe" received signal SIGSEGV, Segmentation fault. >> 0x00007ffff1e6d44f in vecgetarray_ (x=0x7fffffffa718, fa=0x0, ia=0x7fffffffa75c, ierr=0x0) at /scratch/jczhang/petsc/src/vec/vec/interface/ftn-custom/zvectorf.c:257 >> 257 *ierr = VecGetArray(*x, &lx); >> (gdb) bt >> #0 0x00007ffff1e6d44f in vecgetarray_ (x=0x7fffffffa718, fa=0x0, ia=0x7fffffffa75c, ierr=0x0) at /scratch/jczhang/petsc/src/vec/vec/interface/ftn-custom/zvectorf.c:257 >> #1 0x000000000040b6e3 in gw_solver (t_s=1.40129846e-45, n=300) at GW_solver_try.F90:169 >> #2 0x000000000040c6a8 in test_gw () at test_main.F90:35 >> >> ierr=0x0 caused the segfault. See https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/VecGetArray/*vecgetarray__;Iw!!G_uCfscf7eWS!eMMx81X1rAnkLOCN8zBJ-u9Qm0CDez2uHoq4-lj4qxj96YvR6gbL9BLGMpPwTTKG3vTAPBQ3Wgq9Pi0P85VUt7A$ , you should use VecGetArrayF90 instead. >> >> BTW, Barry, the code https://urldefense.us/v3/__https://github.com/sshatanawi/SS_GWM/blob/main/GW_solver_try.F90*L169__;Iw!!G_uCfscf7eWS!eMMx81X1rAnkLOCN8zBJ-u9Qm0CDez2uHoq4-lj4qxj96YvR6gbL9BLGMpPwTTKG3vTAPBQ3Wgq9Pi0PaOJun9Q$ has "call VecGetArray(temp_solution, H_vector, ierr)". I don't find petsc Fortran examples doing VecGetArray. Do we still support it? > > This is not the correct calling sequence for VecGetArray() from Fortran. > > Regardless, definitely should not be writing any new code that uses VecGetArray() from Fortran. Should use VecGetArrayF90(). > >> >> --Junchao Zhang >> >> >> On Wed, Jan 10, 2024 at 2:38?PM Shatanawi, Sawsan Muhammad via petsc-users > wrote: >> Hello all, >> >> I hope you are doing well. >> >> Generally, I use gdb to debug the code. >> I got the attached error message. >> >> I have tried to add the flag -start_in_debugger in the make file, but it didn't work, so it seems I was doing it in the wrong way >> >> This is the link for the whole code: sshatanawi/SS_GWM (github.com) >> >> GitHub - sshatanawi/SS_GWM >> Contribute to sshatanawi/SS_GWM development by creating an account on GitHub. >> github.com >> ? >> >> You can read the description of the code in " Model Desprciption.pdf" >> the compiling file is makefile_f90 where you can find the linked code files >> >> I really appreciate your help >> >> Bests, >> Sawsan >> From: Mark Adams > >> Sent: Friday, January 5, 2024 4:53 AM >> To: Shatanawi, Sawsan Muhammad > >> Cc: Matthew Knepley >; petsc-users at mcs.anl.gov > >> Subject: Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code >> >> [EXTERNAL EMAIL] >> This is a segv. As Matt said, you need to use a debugger for this or add print statements to narrow down the place where this happens. >> >> You will need to learn how to use debuggers to do your project so you might as well start now. >> >> If you have a machine with a GUI debugger that is easier but command line debuggers are good to learn anyway. >> >> I tend to run debuggers directly (eg, lldb ./a.out -- program-args ...) and use a GUI debugger (eg, Totalview or DDT) if available. >> >> Mark >> >> >> On Wed, Dec 20, 2023 at 10:02?PM Shatanawi, Sawsan Muhammad via petsc-users > wrote: >> Hello Matthew, >> >> Thank you for your help. I am sorry that I keep coming back with my error messages, but I reached a point that I don't know how to fix them, and I don't understand them easily. >> The list of errors is getting shorter, now I am getting the attached error messages >> >> Thank you again, >> >> Sawsan >> From: Matthew Knepley > >> Sent: Wednesday, December 20, 2023 6:54 PM >> To: Shatanawi, Sawsan Muhammad > >> Cc: Barry Smith >; petsc-users at mcs.anl.gov > >> Subject: Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code >> >> [EXTERNAL EMAIL] >> On Wed, Dec 20, 2023 at 9:49?PM Shatanawi, Sawsan Muhammad via petsc-users > wrote: >> Hello Barry, >> >> Thank you a lot for your help, Now I am getting the attached error message. >> >> Do not destroy the PC from KSPGetPC() >> >> THanks, >> >> Matt >> >> Bests, >> Sawsan >> From: Barry Smith > >> Sent: Wednesday, December 20, 2023 6:32 PM >> To: Shatanawi, Sawsan Muhammad > >> Cc: Mark Adams >; petsc-users at mcs.anl.gov > >> Subject: Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code >> >> [EXTERNAL EMAIL] >> >> Instead of >> >> call PCCreate(PETSC_COMM_WORLD, pc, ierr) >> call PCSetType(pc, PCILU,ierr) ! Choose a preconditioner type (ILU) >> call KSPSetPC(ksp, pc,ierr) ! Associate the preconditioner with the KSP solver >> >> do >> >> call KSPGetPC(ksp,pc,ierr) >> call PCSetType(pc, PCILU,ierr) >> >> Do not call KSPSetUp(). It will be taken care of automatically during the solve >> >> >> >>> On Dec 20, 2023, at 8:52?PM, Shatanawi, Sawsan Muhammad via petsc-users > wrote: >>> >>> Hello, >>> I don't think that I set preallocation values when I created the matrix, would you please have look at my code. It is just the petsc related part from my code. >>> I was able to fix some of the error messages. Now I have a new set of error messages related to the KSP solver (attached) >>> >>> I appreciate your help >>> >>> Sawsan >>> From: Mark Adams > >>> Sent: Wednesday, December 20, 2023 6:44 AM >>> To: Shatanawi, Sawsan Muhammad > >>> Cc: petsc-users at mcs.anl.gov > >>> Subject: Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code >>> >>> [EXTERNAL EMAIL] >>> Did you set preallocation values when you created the matrix? >>> Don't do that. >>> >>> On Wed, Dec 20, 2023 at 9:36?AM Shatanawi, Sawsan Muhammad > wrote: >>> Hello, >>> >>> I am trying to create a sparse matrix( which is as I believe a zero matrix) then adding some nonzero elements to it over a loop, then assembling it >>> >>> Get Outlook for iOS >>> From: Mark Adams > >>> Sent: Wednesday, December 20, 2023 2:48 AM >>> To: Shatanawi, Sawsan Muhammad > >>> Cc: petsc-users at mcs.anl.gov > >>> Subject: Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code >>> >>> [EXTERNAL EMAIL] >>> I am guessing that you are creating a matrix, adding to it, finalizing it ("assembly"), and then adding to it again, which is fine, but you are adding new non-zeros to the sparsity pattern. >>> If this is what you want then you can tell the matrix to let you do that. >>> Otherwise you have a bug. >>> >>> Mark >>> >>> On Tue, Dec 19, 2023 at 9:50?PM Shatanawi, Sawsan Muhammad via petsc-users > wrote: >>> Hello everyone, >>> >>> I hope this email finds you well. >>> >>> My Name is Sawsan Shatanawi, and I am currently working on developing a Fortran code for simulating groundwater flow in a 3D system. The code involves solving a nonlinear system, and I have created the matrix to be solved using the PCG solver and Picard iteration. However, when I tried to assign it as a PETSc matrix I started getting a lot of error messages. >>> >>> I am kindly asking if someone can help me, I would be happy to share my code with him/her. >>> >>> Please find the attached file contains a list of errors I have gotten >>> >>> Thank you in advance for your time and assistance. >>> Best regards, >>> >>> Sawsan >>> >>> >>> >> >> >> >> -- >> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >> -- Norbert Wiener >> >> https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!eMMx81X1rAnkLOCN8zBJ-u9Qm0CDez2uHoq4-lj4qxj96YvR6gbL9BLGMpPwTTKG3vTAPBQ3Wgq9Pi0PwfbIzh4$ > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!eMMx81X1rAnkLOCN8zBJ-u9Qm0CDez2uHoq4-lj4qxj96YvR6gbL9BLGMpPwTTKG3vTAPBQ3Wgq9Pi0PwfbIzh4$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From a.croucher at auckland.ac.nz Fri May 10 00:01:17 2024 From: a.croucher at auckland.ac.nz (Adrian Croucher) Date: Fri, 10 May 2024 17:01:17 +1200 Subject: [petsc-users] HDF5 time step count In-Reply-To: References: <425d849a-eba3-6739-5e71-251e0f29b5fb@auckland.ac.nz> <8735pcd12k.fsf@jedbrown.org> <7639e2f6-d326-e04b-bd8e-53cecb8d794e@auckland.ac.nz> <220daca5-21ea-00df-bc51-ba6124d020ad@auckland.ac.nz> <6aa6644b-c36b-4f4d-b843-d1a1a4adf7bd@auckland.ac.nz> <97876f22-d284-45d8-8993-893e3937004f@auckland.ac.nz> Message-ID: <3e48cc78-c6fd-467a-a0f8-a67463a58ce3@auckland.ac.nz> hi Matt, On 10/05/24 12:15 am, Matthew Knepley wrote: > > I just tried to test it, but there doesn't seem to be a Fortran > interface for DMGetOutputSequenceLength(). > > Pushed. Thanks. I tried it out and the error below was raised, looks like it's when it tries to destroy the viewer. It still runs ok when DMGetOutputSequenceLength() isn't called. - Adrian [0]PETSC ERROR: Error in external library [0]PETSC ERROR: Error in HDF5 call H5Fclose() Status -1 [0]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!YwgL4tYSuYus7pShvp7QSDY_SmHzGbe32BN2ELvCuZw7P6ZOTBcfFsRiN3G0zVuU29FqKaNqYShrLLflVLcbn25Zbw8CAsGo$ for trouble shooting. [0]PETSC ERROR: Petsc Development GIT revision: v3.21.1-124-g2d06e2faec8? GIT Date: 2024-05-08 19:31:33 +0000 [0]PETSC ERROR: ./initial_test on a main-debug named EN438880 by acro018 Fri May 10 16:55:34 2024 [0]PETSC ERROR: Configure options --with-x --download-hdf5 --download-zlib --download-netcdf --download-pnetcdf --download-exodusii --download-triangle --download-ptscotch --download-chaco --download-hypre [0]PETSC ERROR: #1 PetscViewerFileClose_HDF5() at /home/acro018/software/PETSc/code/src/sys/classes/viewer/impls/hdf5/hdf5v.c:107 [0]PETSC ERROR: #2 PetscViewerDestroy_HDF5() at /home/acro018/software/PETSc/code/src/sys/classes/viewer/impls/hdf5/hdf5v.c:126 [0]PETSC ERROR: #3 PetscViewerDestroy() at /home/acro018/software/PETSc/code/src/sys/classes/viewer/interface/view.c:101 [0]PETSC ERROR: #4 ../src/initial.F90:497 -- Dr Adrian Croucher Senior Research Fellow Department of Engineering Science Waipapa Taumata Rau / University of Auckland, New Zealand email:a.croucher at auckland.ac.nz tel: +64 (0)9 923 4611 -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri May 10 11:12:40 2024 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 10 May 2024 12:12:40 -0400 Subject: [petsc-users] HDF5 time step count In-Reply-To: <3e48cc78-c6fd-467a-a0f8-a67463a58ce3@auckland.ac.nz> References: <425d849a-eba3-6739-5e71-251e0f29b5fb@auckland.ac.nz> <8735pcd12k.fsf@jedbrown.org> <7639e2f6-d326-e04b-bd8e-53cecb8d794e@auckland.ac.nz> <220daca5-21ea-00df-bc51-ba6124d020ad@auckland.ac.nz> <6aa6644b-c36b-4f4d-b843-d1a1a4adf7bd@auckland.ac.nz> <97876f22-d284-45d8-8993-893e3937004f@auckland.ac.nz> <3e48cc78-c6fd-467a-a0f8-a67463a58ce3@auckland.ac.nz> Message-ID: On Fri, May 10, 2024 at 1:01?AM Adrian Croucher wrote: > hi Matt, > On 10/05/24 12:15 am, Matthew Knepley wrote: > > I just tried to test it, but there doesn't seem to be a Fortran interface >> for DMGetOutputSequenceLength(). >> > Pushed. > > Thanks. I tried it out and the error below was raised, looks like it's > when it tries to destroy the viewer. It still runs ok when > DMGetOutputSequenceLength() isn't called. > > Sorry, it looks like I did not close the group. I pushed an update. If that is alright, I will merge it. Thanks Matt > - Adrian > > [0]PETSC ERROR: Error in external library > [0]PETSC ERROR: Error in HDF5 call H5Fclose() Status -1 > [0]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!cr7LJSXkch_z_JwHbQJLw6lOHNHIwlYHSs2EKuOgdO4WPY3yBsl64nAdyRiQX9blQC8KVSJZObA0YA9zG9uD$ for trouble shooting. > [0]PETSC ERROR: Petsc Development GIT revision: v3.21.1-124-g2d06e2faec8 > GIT Date: 2024-05-08 19:31:33 +0000 > [0]PETSC ERROR: ./initial_test on a main-debug named EN438880 by acro018 > Fri May 10 16:55:34 2024 > [0]PETSC ERROR: Configure options --with-x --download-hdf5 --download-zlib > --download-netcdf --download-pnetcdf --download-exodusii > --download-triangle --download-ptscotch --download-chaco --download-hypre > [0]PETSC ERROR: #1 PetscViewerFileClose_HDF5() at > /home/acro018/software/PETSc/code/src/sys/classes/viewer/impls/hdf5/hdf5v.c:107 > [0]PETSC ERROR: #2 PetscViewerDestroy_HDF5() at > /home/acro018/software/PETSc/code/src/sys/classes/viewer/impls/hdf5/hdf5v.c:126 > [0]PETSC ERROR: #3 PetscViewerDestroy() at > /home/acro018/software/PETSc/code/src/sys/classes/viewer/interface/view.c:101 > [0]PETSC ERROR: #4 ../src/initial.F90:497 > > -- > Dr Adrian Croucher > Senior Research Fellow > Department of Engineering Science > Waipapa Taumata Rau / University of Auckland, New Zealand > email: a.croucher at auckland.ac.nz > tel: +64 (0)9 923 4611 > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!cr7LJSXkch_z_JwHbQJLw6lOHNHIwlYHSs2EKuOgdO4WPY3yBsl64nAdyRiQX9blQC8KVSJZObA0YLHcR0C8$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From sawsan.shatanawi at wsu.edu Fri May 10 17:30:09 2024 From: sawsan.shatanawi at wsu.edu (Shatanawi, Sawsan Muhammad) Date: Fri, 10 May 2024 22:30:09 +0000 Subject: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code In-Reply-To: <0636729D-AE8D-4DE0-8606-E0948733814C@petsc.dev> References: <17B6144E-4F3A-4173-9818-4B03669736C4@petsc.dev> <9BD74F88-20FD-4021-B02A-195A58B72282@petsc.dev> <0636729D-AE8D-4DE0-8606-E0948733814C@petsc.dev> Message-ID: Good afternoon, I have tried SNESComputeJacobianDefaultColor(), but the arguments needed are confusing me. Would you please have a look at my code and the error messages I am getting? I didn't understand what the nonzero values of the sparse Jacobian would be. Thank you for your patience and help Bests, Sawsan ________________________________ From: Barry Smith Sent: Thursday, May 9, 2024 12:05 PM To: Shatanawi, Sawsan Muhammad Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code [EXTERNAL EMAIL] On May 9, 2024, at 2:52?PM, Shatanawi, Sawsan Muhammad via petsc-users wrote: This Message Is From an External Sender This message came from outside your organization. Hello everyone, Thank you for your responses and feedback, I checked PFLOTRAN and found that it is a model to simulate groundwater flow, contaminant transport, and other subsurface processes. my goal is not to simulate the groundwater flow, my goal is to develop a code from scratch to simulate the groundwater flow with specific conditions, and then integrate this code with land surface models. Later, the simulation of this code will be on a large scale. I want PETSc to calculate the Jacobian because the system is large and has complex nonlinear behavior, and I don?t risk calculating the derivative by myself. My A-Matrix has parts of source terms that depend on the flow fields, and independent parts will be in the RHS vector. With coloring SNESComputeJacobianDefaultColor() PETSc can compute Jacobian's pretty efficiently. You do not to provide the residual function and you need to provide the nonzero pattern of the sparse Jacobian; that is what residual components f_i are coupled to what input variables in the array x_i. This information comes from your PDE and discretization and appears implicitly in your residual function. Barry I hope I have answered your questions, and I apologize that I wasn?t clear from the beginning, I was trying to keep my descriptions brief. Bests, Sawsan ________________________________ From: Matthew Knepley > Sent: Tuesday, May 7, 2024 5:17 PM To: Shatanawi, Sawsan Muhammad > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code [EXTERNAL EMAIL] On Tue, May 7, 2024 at 2:23?PM Shatanawi, Sawsan Muhammad via petsc-users > wrote: This Message Is From an External Sender This message came from outside your organization. Hello everyone, I hope this email finds you well. My Name is Sawsan Shatanawi, and I was developing a Fortran code for simulating groundwater flow in a 3D system with nonlinear behavior. I solved the nonlinear system using the PCG solver and Picard iteration, but I did not get good results although I checked my matrix and RHS and everything, I decided to change my solver to Newton Rapson method. I checked PETSc documents but I have a few questions: 1) My groundwater system is time-dependent, so should I use TS only instead of SNES? You could use TS, but it is not necessary. You could use SNES and your own timestepping. THe advantage of TS is that you can try many different timesteppers without recoding (just like you can try many different linear and nonlinear solvers). 2) My system has its deltaT, would using deltaT as dt affect my solver, or is it better to use TS-PETSc dt? Also, would using PETSc dt affect the simulation of the groundwater system It sounds like your dt comes from your timestepper. If you use TS, you would use the dt from that. 3) I want my Jacobian matrix to be calculated by PETSc automatically PETSc can calculate a full Jacobian for smaller problems, or a finite-difference Jacobian for any problem (but this impacts the solver). It should be straightfoward to code up the analytic Jacobian. Is there a reason it would be a problem? 4) Do I need to define and calculate the residual vector? Yes. My A-Matrix contains coefficients and external sources and my RHS vector includes the boundary conditions It is strange that your matrix would contain source terms. Do they depend on the flow fields? Barry is right, you should consider PFlotran, and at least know why it would not work for your problem if you don't use it. Thanks, Matt Please find the attached file contains a draft of my code Thank you in advance for your time and help. Best regards, Sawsan ________________________________ From: Shatanawi, Sawsan Muhammad > Sent: Tuesday, January 16, 2024 10:43 AM To: Junchao Zhang > Cc: Barry Smith >; Matthew Knepley >; Mark Adams >; petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code Hello all, Thank you for your valuable help. I will do your recommendations and hope it will run without any issues. Bests, Sawsan ________________________________ From: Junchao Zhang > Sent: Friday, January 12, 2024 8:46 AM To: Shatanawi, Sawsan Muhammad > Cc: Barry Smith >; Matthew Knepley >; Mark Adams >; petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code [EXTERNAL EMAIL] Hi, Sawsan, First in test_main.F90, you need to call VecGetArrayF90(temp_solution, H_vector, ierr) and VecRestoreArrayF90 (temp_solution, H_vector, ierr) as Barry mentioned. Secondly, in the loop of test_main.F90, it calls GW_solver(). Within it, it calls PetscInitialize()/PetscFinalize(). But without MPI being initialized, PetscInitialize()/PetscFinalize() can only be called once. do timestep =2 , NTSP call GW_boundary_conditions(timestep-1) !print *,HNEW(1,1,1) call GW_elevation() ! print *, GWTOP(2,2,2) call GW_conductance() ! print *, CC(2,2,2) call GW_recharge() ! print *, B_Rech(5,4) call GW_pumping(timestep-1) ! print *, B_pump(2,2,2) call GW_SW(timestep-1) print *,B_RIVER (2,2,2) call GW_solver(timestep-1,N) call GW_deallocate_loop() end do A solution is to delete PetscInitialize()/PetscFinalize() in GW_solver_try.F90 and move it to test_main.F90, outside the do loop. diff --git a/test_main.F90 b/test_main.F90 index b5997c55..107bd3ee 100644 --- a/test_main.F90 +++ b/test_main.F90 @@ -1,5 +1,6 @@ program test_GW +#include use petsc use GW_constants use GW_param_by_user @@ -8,6 +9,9 @@ program test_GW implicit none integer :: N integer :: timestep + PetscErrorCode ierr + + call PetscInitialize(ierr) call GW_domain(N) !print *, "N=",N !print *, DELTAT @@ -37,4 +41,5 @@ program test_GW end do print *, HNEW(NCOL,3,2) call GW_deallocate () + call PetscFinalize(ierr) end program test_GW With that, the MPI error will be fixed. The code could run to gw_deallocate () before abort. There are other memory errors. You can install/use valgrind to fix them. Run it with valgrind ./GW.exe and look through the output Thanks. --Junchao Zhang On Thu, Jan 11, 2024 at 10:49?PM Shatanawi, Sawsan Muhammad > wrote: Hello, Thank you all for your help. I have changed VecGetArray to VecGetArrayF90, and the location of destory call. but I want to make sure that VecGet ArrayF90 is to make a new array( vector) that I can use in the rest of my Fortran code? when I run it and debugged it, I got 5.2000000E-03 50.00000 10.00000 0.0000000E+00 PETSC: Attaching gdb to /weka/data/lab/richey/sawsan/GW_CODE/code2024/SS_GWM/./GW.exe of pid 33065 on display :0.0 on machine sn16 Unable to start debugger in xterm: No such file or directory 0.0000000E+00 Attempting to use an MPI routine after finalizing MPICH srun: error: sn16: task 0: Exited with exit code 1 [sawsan.shatanawi at login-p2n02 SS_GWM]$ gdb ./GW/exe GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-100.el7 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later > This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". For bug reporting instructions, please see: >... ./GW/exe: No such file or directory. (gdb) run Starting program: No executable file specified. Use the "file" or "exec-file" command. (gdb) bt No stack. (gdb) If the highlighted line is the error, I don't know why when I write gdb , it does not show me the location of error The code : sshatanawi/SS_GWM (github.com) I really appreciate your helps Sawsan ________________________________ From: Barry Smith > Sent: Wednesday, January 10, 2024 5:35 PM To: Junchao Zhang > Cc: Shatanawi, Sawsan Muhammad >; Mark Adams >; petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code [EXTERNAL EMAIL] On Jan 10, 2024, at 6:49?PM, Junchao Zhang > wrote: Hi, Sawsan, I could build your code and I also could gdb it. $ gdb ./GW.exe ... $ Thread 1 "GW.exe" received signal SIGSEGV, Segmentation fault. 0x00007ffff1e6d44f in vecgetarray_ (x=0x7fffffffa718, fa=0x0, ia=0x7fffffffa75c, ierr=0x0) at /scratch/jczhang/petsc/src/vec/vec/interface/ftn-custom/zvectorf.c:257 257 *ierr = VecGetArray(*x, &lx); (gdb) bt #0 0x00007ffff1e6d44f in vecgetarray_ (x=0x7fffffffa718, fa=0x0, ia=0x7fffffffa75c, ierr=0x0) at /scratch/jczhang/petsc/src/vec/vec/interface/ftn-custom/zvectorf.c:257 #1 0x000000000040b6e3 in gw_solver (t_s=1.40129846e-45, n=300) at GW_solver_try.F90:169 #2 0x000000000040c6a8 in test_gw () at test_main.F90:35 ierr=0x0 caused the segfault. See https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/VecGetArray/*vecgetarray__;Iw!!G_uCfscf7eWS!cZQx_eAmsGrIt3zdZDz-EPdyxXGsLENPvRfcVFizzC_f1TFrp2l74tJO1z0Oi6_NmqrvV8_koK1hGcBM8EhGxfAu8k56c0nkpA$ , you should use VecGetArrayF90 instead. BTW, Barry, the code https://urldefense.us/v3/__https://github.com/sshatanawi/SS_GWM/blob/main/GW_solver_try.F90*L169__;Iw!!G_uCfscf7eWS!cZQx_eAmsGrIt3zdZDz-EPdyxXGsLENPvRfcVFizzC_f1TFrp2l74tJO1z0Oi6_NmqrvV8_koK1hGcBM8EhGxfAu8k4rql_Cyg$ has "call VecGetArray(temp_solution, H_vector, ierr)". I don't find petsc Fortran examples doing VecGetArray. Do we still support it? This is not the correct calling sequence for VecGetArray() from Fortran. Regardless, definitely should not be writing any new code that uses VecGetArray() from Fortran. Should use VecGetArrayF90(). --Junchao Zhang On Wed, Jan 10, 2024 at 2:38?PM Shatanawi, Sawsan Muhammad via petsc-users > wrote: Hello all, I hope you are doing well. Generally, I use gdb to debug the code. I got the attached error message. I have tried to add the flag -start_in_debugger in the make file, but it didn't work, so it seems I was doing it in the wrong way This is the link for the whole code: sshatanawi/SS_GWM (github.com) [https://urldefense.us/v3/__https://opengraph.githubassets.com/9eb6cd14baf12f04848ed209b6f502415eb531bdd7b3a5f9696af68663b870c0/sshatanawi/SS_GWM__;!!G_uCfscf7eWS!cZQx_eAmsGrIt3zdZDz-EPdyxXGsLENPvRfcVFizzC_f1TFrp2l74tJO1z0Oi6_NmqrvV8_koK1hGcBM8EhGxfAu8k7iIHAdBQ$ ] GitHub - sshatanawi/SS_GWM Contribute to sshatanawi/SS_GWM development by creating an account on GitHub. github.com ? You can read the description of the code in " Model Desprciption.pdf" the compiling file is makefile_f90 where you can find the linked code files I really appreciate your help Bests, Sawsan ________________________________ From: Mark Adams > Sent: Friday, January 5, 2024 4:53 AM To: Shatanawi, Sawsan Muhammad > Cc: Matthew Knepley >; petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code [EXTERNAL EMAIL] This is a segv. As Matt said, you need to use a debugger for this or add print statements to narrow down the place where this happens. You will need to learn how to use debuggers to do your project so you might as well start now. If you have a machine with a GUI debugger that is easier but command line debuggers are good to learn anyway. I tend to run debuggers directly (eg, lldb ./a.out -- program-args ...) and use a GUI debugger (eg, Totalview or DDT) if available. Mark On Wed, Dec 20, 2023 at 10:02?PM Shatanawi, Sawsan Muhammad via petsc-users > wrote: Hello Matthew, Thank you for your help. I am sorry that I keep coming back with my error messages, but I reached a point that I don't know how to fix them, and I don't understand them easily. The list of errors is getting shorter, now I am getting the attached error messages Thank you again, Sawsan ________________________________ From: Matthew Knepley > Sent: Wednesday, December 20, 2023 6:54 PM To: Shatanawi, Sawsan Muhammad > Cc: Barry Smith >; petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code [EXTERNAL EMAIL] On Wed, Dec 20, 2023 at 9:49?PM Shatanawi, Sawsan Muhammad via petsc-users > wrote: Hello Barry, Thank you a lot for your help, Now I am getting the attached error message. Do not destroy the PC from KSPGetPC() THanks, Matt Bests, Sawsan ________________________________ From: Barry Smith > Sent: Wednesday, December 20, 2023 6:32 PM To: Shatanawi, Sawsan Muhammad > Cc: Mark Adams >; petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code [EXTERNAL EMAIL] Instead of call PCCreate(PETSC_COMM_WORLD, pc, ierr) call PCSetType(pc, PCILU,ierr) ! Choose a preconditioner type (ILU) call KSPSetPC(ksp, pc,ierr) ! Associate the preconditioner with the KSP solver do call KSPGetPC(ksp,pc,ierr) call PCSetType(pc, PCILU,ierr) Do not call KSPSetUp(). It will be taken care of automatically during the solve On Dec 20, 2023, at 8:52?PM, Shatanawi, Sawsan Muhammad via petsc-users > wrote: Hello, I don't think that I set preallocation values when I created the matrix, would you please have look at my code. It is just the petsc related part from my code. I was able to fix some of the error messages. Now I have a new set of error messages related to the KSP solver (attached) I appreciate your help Sawsan ________________________________ From: Mark Adams > Sent: Wednesday, December 20, 2023 6:44 AM To: Shatanawi, Sawsan Muhammad > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code [EXTERNAL EMAIL] Did you set preallocation values when you created the matrix? Don't do that. On Wed, Dec 20, 2023 at 9:36?AM Shatanawi, Sawsan Muhammad > wrote: Hello, I am trying to create a sparse matrix( which is as I believe a zero matrix) then adding some nonzero elements to it over a loop, then assembling it Get Outlook for iOS ________________________________ From: Mark Adams > Sent: Wednesday, December 20, 2023 2:48 AM To: Shatanawi, Sawsan Muhammad > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code [EXTERNAL EMAIL] I am guessing that you are creating a matrix, adding to it, finalizing it ("assembly"), and then adding to it again, which is fine, but you are adding new non-zeros to the sparsity pattern. If this is what you want then you can tell the matrix to let you do that. Otherwise you have a bug. Mark On Tue, Dec 19, 2023 at 9:50?PM Shatanawi, Sawsan Muhammad via petsc-users > wrote: Hello everyone, I hope this email finds you well. My Name is Sawsan Shatanawi, and I am currently working on developing a Fortran code for simulating groundwater flow in a 3D system. The code involves solving a nonlinear system, and I have created the matrix to be solved using the PCG solver and Picard iteration. However, when I tried to assign it as a PETSc matrix I started getting a lot of error messages. I am kindly asking if someone can help me, I would be happy to share my code with him/her. Please find the attached file contains a list of errors I have gotten Thank you in advance for your time and assistance. Best regards, Sawsan -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!cZQx_eAmsGrIt3zdZDz-EPdyxXGsLENPvRfcVFizzC_f1TFrp2l74tJO1z0Oi6_NmqrvV8_koK1hGcBM8EhGxfAu8k7Aoxww_g$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!cZQx_eAmsGrIt3zdZDz-EPdyxXGsLENPvRfcVFizzC_f1TFrp2l74tJO1z0Oi6_NmqrvV8_koK1hGcBM8EhGxfAu8k7Aoxww_g$ -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: error messages.txt URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: Jacobian.txt URL: From knepley at gmail.com Sat May 11 03:56:02 2024 From: knepley at gmail.com (Matthew Knepley) Date: Sat, 11 May 2024 04:56:02 -0400 Subject: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code In-Reply-To: References: <17B6144E-4F3A-4173-9818-4B03669736C4@petsc.dev> <9BD74F88-20FD-4021-B02A-195A58B72282@petsc.dev> <0636729D-AE8D-4DE0-8606-E0948733814C@petsc.dev> Message-ID: On Fri, May 10, 2024 at 6:30?PM Shatanawi, Sawsan Muhammad via petsc-users < petsc-users at mcs.anl.gov> wrote: > Good afternoon, I have tried SNESComputeJacobianDefaultColor(), but the > arguments needed are confusing me. Would you please have a look at my code > and the error messages I am getting? I didn't understand what the nonzero > values of the sparse > ZjQcmQRYFpfptBannerStart > This Message Is From an External Sender > This message came from outside your organization. > > ZjQcmQRYFpfptBannerEnd > Good afternoon, > > I have tried SNESComputeJacobianDefaultColor(), but the arguments needed > are confusing me. > > Would you please have a look at my code and the error messages I am > getting? > I didn't understand what the nonzero values of the sparse Jacobian would > be. > 1) You are not checking any of the return values from PETSc calls. Look at the PETSc Fortran examples. You should wrap all PETSc calls in PetscCall() or PetscCallA(). 2) You are not intended to directly call SNESComputeJacobianDefaultColor(). PETSc will do this automatically if you do not set of Jacobian function. 3) As Barry points out, coloring will not work unless we understand the nonzero structure of your Jacobian. This can happen by either: a) Using a DM: This is the easiest. Find the type that matches your grid, or b) Preallocating your Jacobian: Here you give the nonzero structure of your Jacobian, but not the values. Currently you do not do this. Instead you just give the size, not the nonzero structure (columns for each row). Thanks, Matt > Thank you for your patience and help > Bests, > Sawsan > > > ------------------------------ > *From:* Barry Smith > *Sent:* Thursday, May 9, 2024 12:05 PM > *To:* Shatanawi, Sawsan Muhammad > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [petsc-users] Help with Integrating PETSc into Fortran > Groundwater Flow Simulation Code > > > *[EXTERNAL EMAIL]* > > > On May 9, 2024, at 2:52?PM, Shatanawi, Sawsan Muhammad via petsc-users < > petsc-users at mcs.anl.gov> wrote: > > This Message Is From an External Sender > This message came from outside your organization. > Hello everyone, > > Thank you for your responses and feedback, > > I checked PFLOTRAN and found that it is a model to simulate groundwater > flow, contaminant transport, and other subsurface processes. > my goal is not to simulate the groundwater flow, my goal is to develop a > code from scratch to simulate the groundwater flow with specific > conditions, and then integrate this code with land surface models. > Later, the simulation of this code will be on a large scale. > > I want PETSc to calculate the Jacobian because the system is large and has > complex nonlinear behavior, and I don?t risk calculating the derivative by > myself. > My A-Matrix has parts of source terms that depend on the flow fields, and > independent parts will be in the RHS vector. > > > With coloring SNESComputeJacobianDefaultColor() PETSc can compute > Jacobian's pretty efficiently. You do not to provide the residual function > and you need to provide the nonzero pattern of the sparse Jacobian; that is > what residual components f_i are coupled to what input variables in the > array x_i. This information comes from your PDE and discretization and > appears implicitly in your residual function. > > Barry > > > I hope I have answered your questions, and I apologize that I wasn?t clear > from the beginning, I was trying to keep my descriptions brief. > > Bests, > Sawsan > ------------------------------ > *From:* Matthew Knepley > *Sent:* Tuesday, May 7, 2024 5:17 PM > *To:* Shatanawi, Sawsan Muhammad > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [petsc-users] Help with Integrating PETSc into Fortran > Groundwater Flow Simulation Code > > *[EXTERNAL EMAIL]* > On Tue, May 7, 2024 at 2:23?PM Shatanawi, Sawsan Muhammad via petsc-users < > petsc-users at mcs.anl.gov> wrote: > > This Message Is From an External Sender > This message came from outside your organization. > > Hello everyone, > > > I hope this email finds you well. > > > My Name is Sawsan Shatanawi, and I was developing a Fortran code for > simulating groundwater flow in a 3D system with nonlinear behavior. I > solved the nonlinear system using the PCG solver and Picard iteration, but > I did not get good results although I checked my matrix and RHS and > everything, I decided to change my solver to Newton Rapson method. > I checked PETSc documents but I have a few questions: > 1) My groundwater system is time-dependent, so should I use TS only > instead of SNES? > > > You could use TS, but it is not necessary. You could use SNES and your own > timestepping. THe advantage of TS is that you can try many different > timesteppers without recoding (just like you can > try many different linear and nonlinear solvers). > > > 2) My system has its deltaT, would using deltaT as dt affect my solver, or > is it better to use TS-PETSc dt? Also, would using PETSc dt affect the > simulation of the groundwater system > > > It sounds like your dt comes from your timestepper. If you use TS, you > would use the dt from that. > > > 3) I want my Jacobian matrix to be calculated by PETSc automatically > > > PETSc can calculate a full Jacobian for smaller problems, or a > finite-difference Jacobian for any problem (but this impacts the solver). > It should be straightfoward to code up the analytic Jacobian. Is there a > reason it would be a problem? > > > 4) Do I need to define and calculate the residual vector? > > > Yes. > > > My A-Matrix contains coefficients and external sources and my RHS vector > includes the boundary conditions > > > It is strange that your matrix would contain source terms. Do they depend > on the flow fields? > > Barry is right, you should consider PFlotran, and at least know why it > would not work for your problem if you don't use it. > > Thanks, > > Matt > > > > Please find the attached file contains a draft of my code > > Thank you in advance for your time and help. > > Best regards, > > Sawsan > > ------------------------------ > *From:* Shatanawi, Sawsan Muhammad > *Sent:* Tuesday, January 16, 2024 10:43 AM > *To:* Junchao Zhang > *Cc:* Barry Smith ; Matthew Knepley ; > Mark Adams ; petsc-users at mcs.anl.gov < > petsc-users at mcs.anl.gov> > *Subject:* Re: [petsc-users] Help with Integrating PETSc into Fortran > Groundwater Flow Simulation Code > > Hello all, > > Thank you for your valuable help. I will do your recommendations and hope > it will run without any issues. > > Bests, > Sawsan > > ------------------------------ > *From:* Junchao Zhang > *Sent:* Friday, January 12, 2024 8:46 AM > *To:* Shatanawi, Sawsan Muhammad > *Cc:* Barry Smith ; Matthew Knepley ; > Mark Adams ; petsc-users at mcs.anl.gov < > petsc-users at mcs.anl.gov> > *Subject:* Re: [petsc-users] Help with Integrating PETSc into Fortran > Groundwater Flow Simulation Code > > *[EXTERNAL EMAIL]* > Hi, Sawsan, > First in test_main.F90, you need to call VecGetArrayF90(temp_solution, > H_vector, ierr) and VecRestoreArrayF90 (temp_solution, H_vector, ierr) as > Barry mentioned. > Secondly, in the loop of test_main.F90, it calls GW_solver(). Within > it, it calls PetscInitialize()/PetscFinalize(). But without MPI being > initialized, PetscInitialize()/PetscFinalize()* can only be called once.* > do timestep =2 , NTSP > call GW_boundary_conditions(timestep-1) > !print *,HNEW(1,1,1) > call GW_elevation() > ! print *, GWTOP(2,2,2) > call GW_conductance() > ! print *, CC(2,2,2) > call GW_recharge() > ! print *, B_Rech(5,4) > call GW_pumping(timestep-1) > ! print *, B_pump(2,2,2) > call GW_SW(timestep-1) > print *,B_RIVER (2,2,2) > call GW_solver(timestep-1,N) > call GW_deallocate_loop() > end do > > A solution is to delete PetscInitialize()/PetscFinalize() in > GW_solver_try.F90 and move it to test_main.F90, outside the do loop. > > diff --git a/test_main.F90 b/test_main.F90 > index b5997c55..107bd3ee 100644 > --- a/test_main.F90 > +++ b/test_main.F90 > @@ -1,5 +1,6 @@ > program test_GW > > +#include > use petsc > use GW_constants > use GW_param_by_user > @@ -8,6 +9,9 @@ program test_GW > implicit none > integer :: N > integer :: timestep > + PetscErrorCode ierr > + > + call PetscInitialize(ierr) > call GW_domain(N) > !print *, "N=",N > !print *, DELTAT > @@ -37,4 +41,5 @@ program test_GW > end do > print *, HNEW(NCOL,3,2) > call GW_deallocate () > + call PetscFinalize(ierr) > end program test_GW > > With that, the MPI error will be fixed. The code could run to > gw_deallocate () before abort. There are other memory errors. You can > install/use valgrind to fix them. Run it with valgrind ./GW.exe and look > through the output > > > Thanks. > --Junchao Zhang > > > On Thu, Jan 11, 2024 at 10:49?PM Shatanawi, Sawsan Muhammad < > sawsan.shatanawi at wsu.edu> wrote: > > Hello, > > Thank you all for your help. > > I have changed VecGetArray to VecGetArrayF90, and the location of destory > call. but I want to make sure that VecGet ArrayF90 is to make a new array( > vector) that I can use in the rest of my Fortran code? > > when I run it and debugged it, I got > > 5.2000000E-03 > 50.00000 > 10.00000 > 0.0000000E+00 > PETSC: Attaching gdb to > /weka/data/lab/richey/sawsan/GW_CODE/code2024/SS_GWM/./GW.exe of pid 33065 > on display :0.0 on machine sn16 > Unable to start debugger in xterm: No such file or directory > 0.0000000E+00 > Attempting to use an MPI routine after finalizing MPICH > srun: error: sn16: task 0: Exited with exit code 1 > [sawsan.shatanawi at login-p2n02 SS_GWM]$ gdb ./GW/exe > GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-100.el7 > Copyright (C) 2013 Free Software Foundation, Inc. > License GPLv3+: GNU GPL version 3 or later < > https://urldefense.us/v3/__http://gnu.org/licenses/gpl.html__;!!G_uCfscf7eWS!bkWeWEl3V-m5Tt01vwfw8xeVliXgzkronS0pUjz3mUShxViJYmF8w1Y_7iwFngCb9i79WfLvvIxFVlh0BDEY$ > > > > This is free software: you are free to change and redistribute it. > There is NO WARRANTY, to the extent permitted by law. Type "show copying" > and "show warranty" for details. > This GDB was configured as "x86_64-redhat-linux-gnu". > For bug reporting instructions, please see: > > >... > ./GW/exe: No such file or directory. > (gdb) run > Starting program: > No executable file specified. > Use the "file" or "exec-file" command. > (gdb) bt > No stack. > (gdb) > > If the highlighted line is the error, I don't know why when I write gdb , > it does not show me the location of error > The code : sshatanawi/SS_GWM (github.com) > > > > I really appreciate your helps > > Sawsan > ------------------------------ > *From:* Barry Smith > *Sent:* Wednesday, January 10, 2024 5:35 PM > *To:* Junchao Zhang > *Cc:* Shatanawi, Sawsan Muhammad ; Mark Adams < > mfadams at lbl.gov>; petsc-users at mcs.anl.gov > *Subject:* Re: [petsc-users] Help with Integrating PETSc into Fortran > Groundwater Flow Simulation Code > > *[EXTERNAL EMAIL]* > > > On Jan 10, 2024, at 6:49?PM, Junchao Zhang > wrote: > > Hi, Sawsan, > I could build your code and I also could gdb it. > > $ gdb ./GW.exe > ... > $ Thread 1 "GW.exe" received signal SIGSEGV, Segmentation fault. > 0x00007ffff1e6d44f in vecgetarray_ (x=0x7fffffffa718, fa=0x0, > ia=0x7fffffffa75c, ierr=0x0) at > /scratch/jczhang/petsc/src/vec/vec/interface/ftn-custom/zvectorf.c:257 > 257 *ierr = VecGetArray(*x, &lx); > (gdb) bt > #0 0x00007ffff1e6d44f in vecgetarray_ (x=0x7fffffffa718, fa=0x0, > ia=0x7fffffffa75c, ierr=0x0) at > /scratch/jczhang/petsc/src/vec/vec/interface/ftn-custom/zvectorf.c:257 > #1 0x000000000040b6e3 in gw_solver (t_s=1.40129846e-45, n=300) at > GW_solver_try.F90:169 > #2 0x000000000040c6a8 in test_gw () at test_main.F90:35 > > ierr=0x0 caused the segfault. See > https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/VecGetArray/*vecgetarray__;Iw!!G_uCfscf7eWS!bkWeWEl3V-m5Tt01vwfw8xeVliXgzkronS0pUjz3mUShxViJYmF8w1Y_7iwFngCb9i79WfLvvIxFVqLtJQNX$ > , > you should use VecGetArrayF90 instead. > > BTW, Barry, the code > https://urldefense.us/v3/__https://github.com/sshatanawi/SS_GWM/blob/main/GW_solver_try.F90*L169__;Iw!!G_uCfscf7eWS!bkWeWEl3V-m5Tt01vwfw8xeVliXgzkronS0pUjz3mUShxViJYmF8w1Y_7iwFngCb9i79WfLvvIxFVl16JdUb$ > > has "call VecGetArray(temp_solution, H_vector, ierr)". I don't find > petsc Fortran examples doing VecGetArray. Do we still support it? > > > This is not the correct calling sequence for VecGetArray() from > Fortran. > > Regardless, definitely should not be writing any new code that uses > VecGetArray() from Fortran. Should use VecGetArrayF90(). > > > --Junchao Zhang > > > On Wed, Jan 10, 2024 at 2:38?PM Shatanawi, Sawsan Muhammad via petsc-users > wrote: > > Hello all, > > I hope you are doing well. > > Generally, I use gdb to debug the code. > I got the attached error message. > > I have tried to add the flag -start_in_debugger in the make file, but it > didn't work, so it seems I was doing it in the wrong way > > This is the link for the whole code: sshatanawi/SS_GWM (github.com) > > > > GitHub - sshatanawi/SS_GWM > > Contribute to sshatanawi/SS_GWM development by creating an account on > GitHub. > github.com > > *?* > > You can read the description of the code in " Model Desprciption.pdf" > the compiling file is makefile_f90 where you can find the linked code > files > > I really appreciate your help > > Bests, > Sawsan > ------------------------------ > *From:* Mark Adams > *Sent:* Friday, January 5, 2024 4:53 AM > *To:* Shatanawi, Sawsan Muhammad > *Cc:* Matthew Knepley ; petsc-users at mcs.anl.gov < > petsc-users at mcs.anl.gov> > *Subject:* Re: [petsc-users] Help with Integrating PETSc into Fortran > Groundwater Flow Simulation Code > > *[EXTERNAL EMAIL]* > This is a segv. As Matt said, you need to use a debugger for this or add > print statements to narrow down the place where this happens. > > You will need to learn how to use debuggers to do your project so you > might as well start now. > > If you have a machine with a GUI debugger that is easier but command line > debuggers are good to learn anyway. > > I tend to run debuggers directly (eg, lldb ./a.out -- program-args ...) > and use a GUI debugger (eg, Totalview or DDT) if available. > > Mark > > > On Wed, Dec 20, 2023 at 10:02?PM Shatanawi, Sawsan Muhammad via > petsc-users wrote: > > Hello Matthew, > > Thank you for your help. I am sorry that I keep coming back with my error > messages, but I reached a point that I don't know how to fix them, and I > don't understand them easily. > The list of errors is getting shorter, now I am getting the attached error > messages > > Thank you again, > > Sawsan > ------------------------------ > *From:* Matthew Knepley > *Sent:* Wednesday, December 20, 2023 6:54 PM > *To:* Shatanawi, Sawsan Muhammad > *Cc:* Barry Smith ; petsc-users at mcs.anl.gov < > petsc-users at mcs.anl.gov> > *Subject:* Re: [petsc-users] Help with Integrating PETSc into Fortran > Groundwater Flow Simulation Code > > *[EXTERNAL EMAIL]* > On Wed, Dec 20, 2023 at 9:49?PM Shatanawi, Sawsan Muhammad via petsc-users > wrote: > > Hello Barry, > > Thank you a lot for your help, Now I am getting the attached error message. > > > Do not destroy the PC from KSPGetPC() > > THanks, > > Matt > > > Bests, > Sawsan > ------------------------------ > *From:* Barry Smith > *Sent:* Wednesday, December 20, 2023 6:32 PM > *To:* Shatanawi, Sawsan Muhammad > *Cc:* Mark Adams ; petsc-users at mcs.anl.gov < > petsc-users at mcs.anl.gov> > *Subject:* Re: [petsc-users] Help with Integrating PETSc into Fortran > Groundwater Flow Simulation Code > > *[EXTERNAL EMAIL]* > > Instead of > > call PCCreate(PETSC_COMM_WORLD, pc, ierr) > call PCSetType(pc, PCILU,ierr) ! Choose a preconditioner type (ILU) > call KSPSetPC(ksp, pc,ierr) ! Associate the preconditioner with the > KSP solver > > do > > call KSPGetPC(ksp,pc,ierr) > call PCSetType(pc, PCILU,ierr) > > Do not call KSPSetUp(). It will be taken care of automatically during the > solve > > > > On Dec 20, 2023, at 8:52?PM, Shatanawi, Sawsan Muhammad via petsc-users < > petsc-users at mcs.anl.gov> wrote: > > Hello, > I don't think that I set preallocation values when I created the matrix, > would you please have look at my code. It is just the petsc related part > from my code. > I was able to fix some of the error messages. Now I have a new set of > error messages related to the KSP solver (attached) > > I appreciate your help > > Sawsan > ------------------------------ > *From:* Mark Adams > *Sent:* Wednesday, December 20, 2023 6:44 AM > *To:* Shatanawi, Sawsan Muhammad > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [petsc-users] Help with Integrating PETSc into Fortran > Groundwater Flow Simulation Code > > *[EXTERNAL EMAIL]* > Did you set preallocation values when you created the matrix? > Don't do that. > > On Wed, Dec 20, 2023 at 9:36?AM Shatanawi, Sawsan Muhammad < > sawsan.shatanawi at wsu.edu> wrote: > > Hello, > > I am trying to create a sparse matrix( which is as I believe a zero > matrix) then adding some nonzero elements to it over a loop, then > assembling it > > Get Outlook for iOS > > ------------------------------ > *From:* Mark Adams > *Sent:* Wednesday, December 20, 2023 2:48 AM > *To:* Shatanawi, Sawsan Muhammad > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [petsc-users] Help with Integrating PETSc into Fortran > Groundwater Flow Simulation Code > > *[EXTERNAL EMAIL]* > I am guessing that you are creating a matrix, adding to it, finalizing it > ("assembly"), and then adding to it again, which is fine, but you are > adding new non-zeros to the sparsity pattern. > If this is what you want then you can tell the matrix to let you do that. > Otherwise you have a bug. > > Mark > > On Tue, Dec 19, 2023 at 9:50?PM Shatanawi, Sawsan Muhammad via petsc-users > wrote: > > Hello everyone, > > I hope this email finds you well. > > My Name is Sawsan Shatanawi, and I am currently working on developing a > Fortran code for simulating groundwater flow in a 3D system. The code > involves solving a nonlinear system, and I have created the matrix to be > solved using the PCG solver and Picard iteration. However, when I tried > to assign it as a PETSc matrix I started getting a lot of error messages. > > I am kindly asking if someone can help me, I would be happy to share my > code with him/her. > > Please find the attached file contains a list of errors I have gotten > > Thank you in advance for your time and assistance. > > Best regards, > > Sawsan > > > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!bkWeWEl3V-m5Tt01vwfw8xeVliXgzkronS0pUjz3mUShxViJYmF8w1Y_7iwFngCb9i79WfLvvIxFVmzEIgU0$ > > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!bkWeWEl3V-m5Tt01vwfw8xeVliXgzkronS0pUjz3mUShxViJYmF8w1Y_7iwFngCb9i79WfLvvIxFVmzEIgU0$ > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!bkWeWEl3V-m5Tt01vwfw8xeVliXgzkronS0pUjz3mUShxViJYmF8w1Y_7iwFngCb9i79WfLvvIxFVmzEIgU0$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From a.croucher at auckland.ac.nz Sun May 12 21:42:13 2024 From: a.croucher at auckland.ac.nz (Adrian Croucher) Date: Mon, 13 May 2024 14:42:13 +1200 Subject: [petsc-users] HDF5 time step count In-Reply-To: References: <425d849a-eba3-6739-5e71-251e0f29b5fb@auckland.ac.nz> <8735pcd12k.fsf@jedbrown.org> <7639e2f6-d326-e04b-bd8e-53cecb8d794e@auckland.ac.nz> <220daca5-21ea-00df-bc51-ba6124d020ad@auckland.ac.nz> <6aa6644b-c36b-4f4d-b843-d1a1a4adf7bd@auckland.ac.nz> <97876f22-d284-45d8-8993-893e3937004f@auckland.ac.nz> <3e48cc78-c6fd-467a-a0f8-a67463a58ce3@auckland.ac.nz> Message-ID: <745bf917-e30a-462d-9085-a03b75c69b66@auckland.ac.nz> hi Matt, On 11/05/24 4:12 am, Matthew Knepley wrote: > > Thanks. I tried it out and the error below was raised, looks like > it's when it tries to destroy the viewer. It still runs ok when > DMGetOutputSequenceLength() isn't called. > > Sorry, it looks like I did not close the group. I pushed an update. If > that is alright, I will merge it. Thanks, it does appear to work now. That's going to be very useful! - Adrian -- Dr Adrian Croucher Senior Research Fellow Department of Engineering Science Waipapa Taumata Rau / University of Auckland, New Zealand email:a.croucher at auckland.ac.nz tel: +64 (0)9 923 4611 -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon May 13 07:02:41 2024 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 13 May 2024 08:02:41 -0400 Subject: [petsc-users] HDF5 time step count In-Reply-To: <745bf917-e30a-462d-9085-a03b75c69b66@auckland.ac.nz> References: <425d849a-eba3-6739-5e71-251e0f29b5fb@auckland.ac.nz> <8735pcd12k.fsf@jedbrown.org> <7639e2f6-d326-e04b-bd8e-53cecb8d794e@auckland.ac.nz> <220daca5-21ea-00df-bc51-ba6124d020ad@auckland.ac.nz> <6aa6644b-c36b-4f4d-b843-d1a1a4adf7bd@auckland.ac.nz> <97876f22-d284-45d8-8993-893e3937004f@auckland.ac.nz> <3e48cc78-c6fd-467a-a0f8-a67463a58ce3@auckland.ac.nz> <745bf917-e30a-462d-9085-a03b75c69b66@auckland.ac.nz> Message-ID: On Sun, May 12, 2024 at 10:42?PM Adrian Croucher wrote: > hi Matt, > On 11/05/24 4:12 am, Matthew Knepley wrote: > > Thanks. I tried it out and the error below was raised, looks like it's >> when it tries to destroy the viewer. It still runs ok when >> DMGetOutputSequenceLength() isn't called. >> > Sorry, it looks like I did not close the group. I pushed an update. If > that is alright, I will merge it. > > Thanks, it does appear to work now. That's going to be very useful! > > Excellent! I will get it merged in. Thanks, Matt > - Adrian > > -- > Dr Adrian Croucher > Senior Research Fellow > Department of Engineering Science > Waipapa Taumata Rau / University of Auckland, New Zealand > email: a.croucher at auckland.ac.nz > tel: +64 (0)9 923 4611 > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fa_sX8leJlX2EqUMrX8Mgt-V_CgjZo1q5KtjgUiqXJKLzHcUH1Ko3Xgwt49Ji5tcIxwutLtuTM1N_Y15u93u$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From wurunjian at gmail.com Mon May 13 06:28:24 2024 From: wurunjian at gmail.com (Runjian Wu) Date: Mon, 13 May 2024 19:28:24 +0800 Subject: [petsc-users] VecGetArrayF90 vs. VecGetArrayReadF90 Message-ID: Hi all, I have a question about VecGetArrayReadF90(...). If I use VecGetArrayReadF90(...), I can still write entries into the array like VecGetArrayF90(...). Is it possible to report an error during compile process? Thanks, Runjian Wu -------------- next part -------------- An HTML attachment was scrubbed... URL: From wurunjian at gmail.com Mon May 13 06:47:48 2024 From: wurunjian at gmail.com (Runjian Wu) Date: Mon, 13 May 2024 19:47:48 +0800 Subject: [petsc-users] duplicated libs Message-ID: Hi all, After I compiled PETSc, I found some duplicated libs in the variable PETSC_EXTERNAL_LIB_BASIC, e.g., -lm, -lgfortran -lstdc++. I am curious how it happened and how to remove the duplicates? Thanks, Runjian Wu -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Mon May 13 08:31:45 2024 From: bsmith at petsc.dev (Barry Smith) Date: Mon, 13 May 2024 09:31:45 -0400 Subject: [petsc-users] duplicated libs In-Reply-To: References: Message-ID: <16654C9F-0761-4A68-A7E6-7DE5FA35E559@petsc.dev> Because the order of the libraries can be important, it is difficult for ./configure to remove unneeded duplicates automatically. You can manually remove duplicates by editing $PETSC_ARCH/lib/petsc/conf/petscvariables after running ./configure Barry > On May 13, 2024, at 7:47?AM, Runjian Wu wrote: > > This Message Is From an External Sender > This message came from outside your organization. > Hi all, > > After I compiled PETSc, I found some duplicated libs in the variable PETSC_EXTERNAL_LIB_BASIC, e.g., -lm, -lgfortran -lstdc++. I am curious how it happened and how to remove the duplicates? > > Thanks, > > Runjian Wu -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Mon May 13 08:34:33 2024 From: balay at mcs.anl.gov (Satish Balay) Date: Mon, 13 May 2024 08:34:33 -0500 (CDT) Subject: [petsc-users] duplicated libs In-Reply-To: References: Message-ID: <38fc058b-360d-073f-aac9-6a6d47e7fac2@mcs.anl.gov> what version of PETSc? What configure command? What do you have for PETSC_EXTERNAL_LIB_BASIC? You can send configure.log for your build to petsc-maint Generally duplicates should not cause grief. [as one needs them to overcome circular dependencies]. What issues are you seeing? [send relevant logs] If the duplicates are coming form "compiler library" detection part of configure - you can skip this part by listing the correct "compiler libraries" for ex: with gcc/gfortran: LIBS="-lgfortran -lstdc++" However -lgfortran could be required for gcc - but a duplicate for gfortran. Satish On Mon, 13 May 2024, Runjian Wu wrote: > Hi all, > > After I compiled PETSc, I found some duplicated libs in the variable > PETSC_EXTERNAL_LIB_BASIC, e.g., -lm, -lgfortran -lstdc++. I am curious how > it happened and how to remove the duplicates? > > Thanks, > > Runjian Wu > From bsmith at petsc.dev Mon May 13 08:35:53 2024 From: bsmith at petsc.dev (Barry Smith) Date: Mon, 13 May 2024 09:35:53 -0400 Subject: [petsc-users] VecGetArrayF90 vs. VecGetArrayReadF90 In-Reply-To: References: Message-ID: <21A28F71-57D4-48D2-9029-69B864B23C50@petsc.dev> I couldn't find a way in Fortran to declare an array as read-only. Is there such support? Barry > On May 13, 2024, at 7:28?AM, Runjian Wu wrote: > > This Message Is From an External Sender > This message came from outside your organization. > Hi all, > > I have a question about VecGetArrayReadF90(...). If I use VecGetArrayReadF90(...), I can still write entries into the array like VecGetArrayF90(...). Is it possible to report an error during compile process? > > Thanks, > > Runjian Wu -------------- next part -------------- An HTML attachment was scrubbed... URL: From liufield at gmail.com Mon May 13 10:12:41 2024 From: liufield at gmail.com (neil liu) Date: Mon, 13 May 2024 11:12:41 -0400 Subject: [petsc-users] Can not find the header file for OpenMPI Message-ID: Dear Petsc developers, I am trying to install Petsc with a preinstalled OpenMPi. ./configure --download-fblaslapack --with-mpi-dir=/usr/lib64/openmpi --with-mpi-incdir=/usr/include/openmpi-x86_64 But the final information shows, MPI: Version: 3 Includes: -I/usr/lib64/openmpi/include (*This is not the right include directory*) mpiexec: /usr/lib64/openmpi/bin/mpiexec Implementation: openmpi OMPI_VERSION: 4.1.1 Did I use the wrong way to include the incdir? Thanks, Xiaodong -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Mon May 13 10:32:25 2024 From: balay at mcs.anl.gov (Satish Balay) Date: Mon, 13 May 2024 10:32:25 -0500 (CDT) Subject: [petsc-users] Can not find the header file for OpenMPI In-Reply-To: References: Message-ID: On Mon, 13 May 2024, neil liu wrote: > Dear Petsc developers, > > I am trying to install Petsc with a preinstalled OpenMPi. > > ./configure --download-fblaslapack --with-mpi-dir=/usr/lib64/openmpi --with-mpi-dir=DIR is a bit unique [wrt other pkg-dir options]. It means: --with-cc=DIR/bin/mpicc --with-cxx=DIR/bin/mpicxx --with-fc=DIR/bin/mpif90 > --with-mpi-incdir=/usr/include/openmpi-x86_64 There is no such configure option > > But the final information shows, > MPI: > Version: 3 > Includes: -I/usr/lib64/openmpi/include (*This is not the right include > directory*) Can you send configure.log for this build [perhaps to petsc-maint] Satish > mpiexec: /usr/lib64/openmpi/bin/mpiexec > Implementation: openmpi > OMPI_VERSION: 4.1.1 > > Did I use the wrong way to include the incdir? > > Thanks, > Xiaodong > From liufield at gmail.com Mon May 13 10:36:55 2024 From: liufield at gmail.com (neil liu) Date: Mon, 13 May 2024 11:36:55 -0400 Subject: [petsc-users] Can not find the header file for OpenMPI In-Reply-To: References: Message-ID: Thanks, I tried the following, and it didn't work. ./configure --download-fblaslapack --with-mpi-lib=/usr/lib64/openmpi/lib --with-mpi-include=/usr/include/openmpi-x86_64 ============================================================================================= Configuring PETSc to compile on your system ============================================================================================= TESTING: check from config.libraries(config/BuildSystem/config/libraries.py:186) ********************************************************************************************* UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): --------------------------------------------------------------------------------------------- --with-mpi-lib=['/usr/lib64/openmpi/lib'] and --with-mpi-include=['/usr/include/openmpi-x86_64'] did not work ********************************************************************************************* On Mon, May 13, 2024 at 11:32?AM Satish Balay wrote: > > On Mon, 13 May 2024, neil liu wrote: > > > Dear Petsc developers, > > > > I am trying to install Petsc with a preinstalled OpenMPi. > > > > ./configure --download-fblaslapack --with-mpi-dir=/usr/lib64/openmpi > > --with-mpi-dir=DIR is a bit unique [wrt other pkg-dir options]. > > It means: > > --with-cc=DIR/bin/mpicc --with-cxx=DIR/bin/mpicxx --with-fc=DIR/bin/mpif90 > > > --with-mpi-incdir=/usr/include/openmpi-x86_64 > > There is no such configure option > > > > > But the final information shows, > > MPI: > > Version: 3 > > Includes: -I/usr/lib64/openmpi/include (*This is not the right > include > > directory*) > > Can you send configure.log for this build [perhaps to petsc-maint] > > Satish > > > mpiexec: /usr/lib64/openmpi/bin/mpiexec > > Implementation: openmpi > > OMPI_VERSION: 4.1.1 > > > > Did I use the wrong way to include the incdir? > > > > Thanks, > > Xiaodong > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: configure.log Type: text/x-log Size: 577594 bytes Desc: not available URL: From balay at mcs.anl.gov Mon May 13 10:42:19 2024 From: balay at mcs.anl.gov (Satish Balay) Date: Mon, 13 May 2024 10:42:19 -0500 (CDT) Subject: [petsc-users] Can not find the header file for OpenMPI In-Reply-To: References: Message-ID: <3b083eba-2b47-b8ad-932f-e0697a5b00d8@mcs.anl.gov> You are misinterpreting --with-mpi-include --with-mpi-lib options. Any particular reason you want to use these options instead of mpi compilers? >>>>>>> balay at p1 /home/balay $ mpicc -show gcc -I/home/balay/soft/mpich-4.0.1/include -L/home/balay/soft/mpich-4.0.1/lib -Wl,-rpath -Wl,/home/balay/soft/mpich-4.0.1/lib -Wl,--enable-new-dtags -lmpi balay at p1 /home/balay $ mpif90 -show gfortran -fallow-argument-mismatch -I/home/balay/soft/mpich-4.0.1/include -I/home/balay/soft/mpich-4.0.1/include -L/home/balay/soft/mpich-4.0.1/lib -lmpifort -Wl,-rpath -Wl,/home/balay/soft/mpich-4.0.1/lib -Wl,--enable-new-dtags -lmpi <<<<< i.e here I would use: --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --with-mpi-include=/home/balay/soft/mpich-4.0.1/include --with-mpi-lib="-L/home/balay/soft/mpich-4.0.1/lib -Wl,-rpath -Wl,/home/balay/soft/mpich-4.0.1/lib -lmpifort -lmpi" Satish On Mon, 13 May 2024, neil liu wrote: > Thanks, I tried the following, and it didn't work. > > ./configure --download-fblaslapack --with-mpi-lib=/usr/lib64/openmpi/lib > --with-mpi-include=/usr/include/openmpi-x86_64 > ============================================================================================= > Configuring PETSc to compile on your system > ============================================================================================= > TESTING: check from > config.libraries(config/BuildSystem/config/libraries.py:186) > ********************************************************************************************* > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for > details): > --------------------------------------------------------------------------------------------- > --with-mpi-lib=['/usr/lib64/openmpi/lib'] and > --with-mpi-include=['/usr/include/openmpi-x86_64'] did not work > ********************************************************************************************* > > On Mon, May 13, 2024 at 11:32?AM Satish Balay wrote: > > > > > On Mon, 13 May 2024, neil liu wrote: > > > > > Dear Petsc developers, > > > > > > I am trying to install Petsc with a preinstalled OpenMPi. > > > > > > ./configure --download-fblaslapack --with-mpi-dir=/usr/lib64/openmpi > > > > --with-mpi-dir=DIR is a bit unique [wrt other pkg-dir options]. > > > > It means: > > > > --with-cc=DIR/bin/mpicc --with-cxx=DIR/bin/mpicxx --with-fc=DIR/bin/mpif90 > > > > > --with-mpi-incdir=/usr/include/openmpi-x86_64 > > > > There is no such configure option > > > > > > > > But the final information shows, > > > MPI: > > > Version: 3 > > > Includes: -I/usr/lib64/openmpi/include (*This is not the right > > include > > > directory*) > > > > Can you send configure.log for this build [perhaps to petsc-maint] > > > > Satish > > > > > mpiexec: /usr/lib64/openmpi/bin/mpiexec > > > Implementation: openmpi > > > OMPI_VERSION: 4.1.1 > > > > > > Did I use the wrong way to include the incdir? > > > > > > Thanks, > > > Xiaodong > > > > > > > > From liufield at gmail.com Mon May 13 10:53:11 2024 From: liufield at gmail.com (neil liu) Date: Mon, 13 May 2024 11:53:11 -0400 Subject: [petsc-users] Can not find the header file for OpenMPI In-Reply-To: <3b083eba-2b47-b8ad-932f-e0697a5b00d8@mcs.anl.gov> References: <3b083eba-2b47-b8ad-932f-e0697a5b00d8@mcs.anl.gov> Message-ID: Thanks. I want to use the preinstalled OpenMPI to configure petsc. For the preinstalled OpenMPI, the lib and include dir are at different places. Then how should I configure with this preinstalled OpenMPI? Thanks, $ mpicc --show gcc -I/usr/include/openmpi-x86_64 -Wl,-rpath -Wl,/usr/lib64/openmpi/lib -Wl,--enable-new-dtags -L/usr/lib64/openmpi/lib -lmpi [liuneil at localhost petsc]$ mpif90 --show gfortran -I/usr/include/openmpi-x86_64 -I/usr/lib64/openmpi/lib -Wl,-rpath -Wl,/usr/lib64/openmpi/lib -Wl,--enable-new-dtags -L/usr/lib64/openmpi/lib -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi On Mon, May 13, 2024 at 11:42?AM Satish Balay wrote: > You are misinterpreting --with-mpi-include --with-mpi-lib options. > > Any particular reason you want to use these options instead of mpi > compilers? > > >>>>>>> > balay at p1 /home/balay > $ mpicc -show > gcc -I/home/balay/soft/mpich-4.0.1/include > -L/home/balay/soft/mpich-4.0.1/lib -Wl,-rpath > -Wl,/home/balay/soft/mpich-4.0.1/lib -Wl,--enable-new-dtags -lmpi > balay at p1 /home/balay > $ mpif90 -show > gfortran -fallow-argument-mismatch -I/home/balay/soft/mpich-4.0.1/include > -I/home/balay/soft/mpich-4.0.1/include -L/home/balay/soft/mpich-4.0.1/lib > -lmpifort -Wl,-rpath -Wl,/home/balay/soft/mpich-4.0.1/lib > -Wl,--enable-new-dtags -lmpi > <<<<< > > i.e here I would use: > > --with-cc=gcc --with-cxx=g++ --with-fc=gfortran > --with-mpi-include=/home/balay/soft/mpich-4.0.1/include > --with-mpi-lib="-L/home/balay/soft/mpich-4.0.1/lib -Wl,-rpath > -Wl,/home/balay/soft/mpich-4.0.1/lib -lmpifort -lmpi" > > Satish > > > > On Mon, 13 May 2024, neil liu wrote: > > > Thanks, I tried the following, and it didn't work. > > > > ./configure --download-fblaslapack --with-mpi-lib=/usr/lib64/openmpi/lib > > --with-mpi-include=/usr/include/openmpi-x86_64 > > > ============================================================================================= > > Configuring PETSc to compile on your system > > > ============================================================================================= > > TESTING: check from > > config.libraries(config/BuildSystem/config/libraries.py:186) > > > ********************************************************************************************* > > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for > > details): > > > --------------------------------------------------------------------------------------------- > > --with-mpi-lib=['/usr/lib64/openmpi/lib'] and > > --with-mpi-include=['/usr/include/openmpi-x86_64'] did not work > > > ********************************************************************************************* > > > > On Mon, May 13, 2024 at 11:32?AM Satish Balay wrote: > > > > > > > > On Mon, 13 May 2024, neil liu wrote: > > > > > > > Dear Petsc developers, > > > > > > > > I am trying to install Petsc with a preinstalled OpenMPi. > > > > > > > > ./configure --download-fblaslapack --with-mpi-dir=/usr/lib64/openmpi > > > > > > --with-mpi-dir=DIR is a bit unique [wrt other pkg-dir options]. > > > > > > It means: > > > > > > --with-cc=DIR/bin/mpicc --with-cxx=DIR/bin/mpicxx > --with-fc=DIR/bin/mpif90 > > > > > > > --with-mpi-incdir=/usr/include/openmpi-x86_64 > > > > > > There is no such configure option > > > > > > > > > > > But the final information shows, > > > > MPI: > > > > Version: 3 > > > > Includes: -I/usr/lib64/openmpi/include (*This is not the right > > > include > > > > directory*) > > > > > > Can you send configure.log for this build [perhaps to petsc-maint] > > > > > > Satish > > > > > > > mpiexec: /usr/lib64/openmpi/bin/mpiexec > > > > Implementation: openmpi > > > > OMPI_VERSION: 4.1.1 > > > > > > > > Did I use the wrong way to include the incdir? > > > > > > > > Thanks, > > > > Xiaodong > > > > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Mon May 13 10:57:04 2024 From: balay at mcs.anl.gov (Satish Balay) Date: Mon, 13 May 2024 10:57:04 -0500 (CDT) Subject: [petsc-users] Can not find the header file for OpenMPI In-Reply-To: References: <3b083eba-2b47-b8ad-932f-e0697a5b00d8@mcs.anl.gov> Message-ID: If you are using mpicc/mpif90 as compilers from you pre-installed MPI - you don't need to list --with-mpi-include, --with-mpi-lib options. As mentioned - you can do this either with: [if you have then in PATH] --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 or: --with-cc=MPI-DIR/bin/mpicc --with-cxx=MPI-DIR/bin/mpicxx --with-fc=MPI-DIR/bin/mpif90 or --with-mpi-dir=MPI-DIR If you are having trouble with any of the above modes - resend configure.log for it. Satish On Mon, 13 May 2024, neil liu wrote: > Thanks. I want to use the preinstalled OpenMPI to configure petsc. For the > preinstalled OpenMPI, the lib and include dir are at different places. > Then how should I configure with this preinstalled OpenMPI? > > Thanks, > > $ mpicc --show > gcc -I/usr/include/openmpi-x86_64 -Wl,-rpath -Wl,/usr/lib64/openmpi/lib > -Wl,--enable-new-dtags -L/usr/lib64/openmpi/lib -lmpi > [liuneil at localhost petsc]$ mpif90 --show > gfortran -I/usr/include/openmpi-x86_64 -I/usr/lib64/openmpi/lib -Wl,-rpath > -Wl,/usr/lib64/openmpi/lib -Wl,--enable-new-dtags -L/usr/lib64/openmpi/lib > -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi > > > On Mon, May 13, 2024 at 11:42?AM Satish Balay wrote: > > > You are misinterpreting --with-mpi-include --with-mpi-lib options. > > > > Any particular reason you want to use these options instead of mpi > > compilers? > > > > >>>>>>> > > balay at p1 /home/balay > > $ mpicc -show > > gcc -I/home/balay/soft/mpich-4.0.1/include > > -L/home/balay/soft/mpich-4.0.1/lib -Wl,-rpath > > -Wl,/home/balay/soft/mpich-4.0.1/lib -Wl,--enable-new-dtags -lmpi > > balay at p1 /home/balay > > $ mpif90 -show > > gfortran -fallow-argument-mismatch -I/home/balay/soft/mpich-4.0.1/include > > -I/home/balay/soft/mpich-4.0.1/include -L/home/balay/soft/mpich-4.0.1/lib > > -lmpifort -Wl,-rpath -Wl,/home/balay/soft/mpich-4.0.1/lib > > -Wl,--enable-new-dtags -lmpi > > <<<<< > > > > i.e here I would use: > > > > --with-cc=gcc --with-cxx=g++ --with-fc=gfortran > > --with-mpi-include=/home/balay/soft/mpich-4.0.1/include > > --with-mpi-lib="-L/home/balay/soft/mpich-4.0.1/lib -Wl,-rpath > > -Wl,/home/balay/soft/mpich-4.0.1/lib -lmpifort -lmpi" > > > > Satish > > > > > > > > On Mon, 13 May 2024, neil liu wrote: > > > > > Thanks, I tried the following, and it didn't work. > > > > > > ./configure --download-fblaslapack --with-mpi-lib=/usr/lib64/openmpi/lib > > > --with-mpi-include=/usr/include/openmpi-x86_64 > > > > > ============================================================================================= > > > Configuring PETSc to compile on your system > > > > > ============================================================================================= > > > TESTING: check from > > > config.libraries(config/BuildSystem/config/libraries.py:186) > > > > > ********************************************************************************************* > > > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for > > > details): > > > > > --------------------------------------------------------------------------------------------- > > > --with-mpi-lib=['/usr/lib64/openmpi/lib'] and > > > --with-mpi-include=['/usr/include/openmpi-x86_64'] did not work > > > > > ********************************************************************************************* > > > > > > On Mon, May 13, 2024 at 11:32?AM Satish Balay wrote: > > > > > > > > > > > On Mon, 13 May 2024, neil liu wrote: > > > > > > > > > Dear Petsc developers, > > > > > > > > > > I am trying to install Petsc with a preinstalled OpenMPi. > > > > > > > > > > ./configure --download-fblaslapack --with-mpi-dir=/usr/lib64/openmpi > > > > > > > > --with-mpi-dir=DIR is a bit unique [wrt other pkg-dir options]. > > > > > > > > It means: > > > > > > > > --with-cc=DIR/bin/mpicc --with-cxx=DIR/bin/mpicxx > > --with-fc=DIR/bin/mpif90 > > > > > > > > > --with-mpi-incdir=/usr/include/openmpi-x86_64 > > > > > > > > There is no such configure option > > > > > > > > > > > > > > But the final information shows, > > > > > MPI: > > > > > Version: 3 > > > > > Includes: -I/usr/lib64/openmpi/include (*This is not the right > > > > include > > > > > directory*) > > > > > > > > Can you send configure.log for this build [perhaps to petsc-maint] > > > > > > > > Satish > > > > > > > > > mpiexec: /usr/lib64/openmpi/bin/mpiexec > > > > > Implementation: openmpi > > > > > OMPI_VERSION: 4.1.1 > > > > > > > > > > Did I use the wrong way to include the incdir? > > > > > > > > > > Thanks, > > > > > Xiaodong > > > > > > > > > > > > > > > > > > > From liufield at gmail.com Mon May 13 11:13:19 2024 From: liufield at gmail.com (neil liu) Date: Mon, 13 May 2024 12:13:19 -0400 Subject: [petsc-users] Can not find the header file for OpenMPI In-Reply-To: References: <3b083eba-2b47-b8ad-932f-e0697a5b00d8@mcs.anl.gov> Message-ID: Thanks Satish. The first way worked.( --with-cc=MPI-DIR/bin/mpicc --with-cxx=MPI-DIR/bin/mpicxx --with-fc=MPI-DIR/bin/mpif90) I also tried the 2nd way, it didn't work. Thanks, XIaodong On Mon, May 13, 2024 at 11:57?AM Satish Balay wrote: > If you are using mpicc/mpif90 as compilers from you pre-installed MPI - > you don't need to list --with-mpi-include, --with-mpi-lib options. > > As mentioned - you can do this either with: > > [if you have then in PATH] --with-cc=mpicc --with-cxx=mpicxx > --with-fc=mpif90 > > or: > > --with-cc=MPI-DIR/bin/mpicc --with-cxx=MPI-DIR/bin/mpicxx > --with-fc=MPI-DIR/bin/mpif90 > > or > > --with-mpi-dir=MPI-DIR > > If you are having trouble with any of the above modes - resend > configure.log for it. > > Satish > > On Mon, 13 May 2024, neil liu wrote: > > > Thanks. I want to use the preinstalled OpenMPI to configure petsc. For > the > > preinstalled OpenMPI, the lib and include dir are at different places. > > Then how should I configure with this preinstalled OpenMPI? > > > > Thanks, > > > > $ mpicc --show > > gcc -I/usr/include/openmpi-x86_64 -Wl,-rpath -Wl,/usr/lib64/openmpi/lib > > -Wl,--enable-new-dtags -L/usr/lib64/openmpi/lib -lmpi > > [liuneil at localhost petsc]$ mpif90 --show > > gfortran -I/usr/include/openmpi-x86_64 -I/usr/lib64/openmpi/lib > -Wl,-rpath > > -Wl,/usr/lib64/openmpi/lib -Wl,--enable-new-dtags > -L/usr/lib64/openmpi/lib > > -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi > > > > > > On Mon, May 13, 2024 at 11:42?AM Satish Balay wrote: > > > > > You are misinterpreting --with-mpi-include --with-mpi-lib options. > > > > > > Any particular reason you want to use these options instead of mpi > > > compilers? > > > > > > >>>>>>> > > > balay at p1 /home/balay > > > $ mpicc -show > > > gcc -I/home/balay/soft/mpich-4.0.1/include > > > -L/home/balay/soft/mpich-4.0.1/lib -Wl,-rpath > > > -Wl,/home/balay/soft/mpich-4.0.1/lib -Wl,--enable-new-dtags -lmpi > > > balay at p1 /home/balay > > > $ mpif90 -show > > > gfortran -fallow-argument-mismatch > -I/home/balay/soft/mpich-4.0.1/include > > > -I/home/balay/soft/mpich-4.0.1/include > -L/home/balay/soft/mpich-4.0.1/lib > > > -lmpifort -Wl,-rpath -Wl,/home/balay/soft/mpich-4.0.1/lib > > > -Wl,--enable-new-dtags -lmpi > > > <<<<< > > > > > > i.e here I would use: > > > > > > --with-cc=gcc --with-cxx=g++ --with-fc=gfortran > > > --with-mpi-include=/home/balay/soft/mpich-4.0.1/include > > > --with-mpi-lib="-L/home/balay/soft/mpich-4.0.1/lib -Wl,-rpath > > > -Wl,/home/balay/soft/mpich-4.0.1/lib -lmpifort -lmpi" > > > > > > Satish > > > > > > > > > > > > On Mon, 13 May 2024, neil liu wrote: > > > > > > > Thanks, I tried the following, and it didn't work. > > > > > > > > ./configure --download-fblaslapack > --with-mpi-lib=/usr/lib64/openmpi/lib > > > > --with-mpi-include=/usr/include/openmpi-x86_64 > > > > > > > > ============================================================================================= > > > > Configuring PETSc to compile on your system > > > > > > > > ============================================================================================= > > > > TESTING: check from > > > > config.libraries(config/BuildSystem/config/libraries.py:186) > > > > > > > > ********************************************************************************************* > > > > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log > for > > > > details): > > > > > > > > --------------------------------------------------------------------------------------------- > > > > --with-mpi-lib=['/usr/lib64/openmpi/lib'] and > > > > --with-mpi-include=['/usr/include/openmpi-x86_64'] did not work > > > > > > > > ********************************************************************************************* > > > > > > > > On Mon, May 13, 2024 at 11:32?AM Satish Balay > wrote: > > > > > > > > > > > > > > On Mon, 13 May 2024, neil liu wrote: > > > > > > > > > > > Dear Petsc developers, > > > > > > > > > > > > I am trying to install Petsc with a preinstalled OpenMPi. > > > > > > > > > > > > ./configure --download-fblaslapack > --with-mpi-dir=/usr/lib64/openmpi > > > > > > > > > > --with-mpi-dir=DIR is a bit unique [wrt other pkg-dir options]. > > > > > > > > > > It means: > > > > > > > > > > --with-cc=DIR/bin/mpicc --with-cxx=DIR/bin/mpicxx > > > --with-fc=DIR/bin/mpif90 > > > > > > > > > > > --with-mpi-incdir=/usr/include/openmpi-x86_64 > > > > > > > > > > There is no such configure option > > > > > > > > > > > > > > > > > But the final information shows, > > > > > > MPI: > > > > > > Version: 3 > > > > > > Includes: -I/usr/lib64/openmpi/include (*This is not the > right > > > > > include > > > > > > directory*) > > > > > > > > > > Can you send configure.log for this build [perhaps to petsc-maint] > > > > > > > > > > Satish > > > > > > > > > > > mpiexec: /usr/lib64/openmpi/bin/mpiexec > > > > > > Implementation: openmpi > > > > > > OMPI_VERSION: 4.1.1 > > > > > > > > > > > > Did I use the wrong way to include the incdir? > > > > > > > > > > > > Thanks, > > > > > > Xiaodong > > > > > > > > > > > > > > > > > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: configure.log Type: text/x-log Size: 988220 bytes Desc: not available URL: From balay at mcs.anl.gov Mon May 13 12:39:25 2024 From: balay at mcs.anl.gov (Satish Balay) Date: Mon, 13 May 2024 12:39:25 -0500 (CDT) Subject: [petsc-users] Can not find the header file for OpenMPI In-Reply-To: References: <3b083eba-2b47-b8ad-932f-e0697a5b00d8@mcs.anl.gov> Message-ID: <99e9f80b-617c-d925-799a-9dbd9afc7265@mcs.anl.gov> On Mon, 13 May 2024, neil liu wrote: > I also tried the 2nd way, it didn't work. configure.log attached is successful. >>>>>>>> Configure Options: --configModules=PETSc.Configure --optionsModule=config.compilerOptions --download-fblaslapack --with-mpi-dir=/usr/lib64/openmpi Compilers: C Compiler: /usr/lib64/openmpi/bin/mpicc -fPIC -Wall -Wwrite-strings -Wno-unknown-pragmas -Wno-lto-type-mismatch -Wno-stringop-overflow -fstack-protector -fvisibility=hidden -g3 -O0 Version: gcc (GCC) 11.4.1 20231218 (Red Hat 11.4.1-3) C++ Compiler: /usr/lib64/openmpi/bin/mpicxx -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -Wno-lto-type-mismatch -Wno-psabi -fstack-protector -fvisibility=hidden -g -O0 -std=gnu++20 -fPIC Version: g++ (GCC) 11.4.1 20231218 (Red Hat 11.4.1-3) Fortran Compiler: /usr/lib64/openmpi/bin/mpif90 -fPIC -Wall -ffree-line-length-none -ffree-line-length-0 -Wno-lto-type-mismatch -Wno-unused-dummy-argument -g -O0 Version: GNU Fortran (GCC) 11.4.1 20231218 (Red Hat 11.4.1-3) <<<<< So the correct compilers from the specified mpi location /usr/lib64/openmpi got used. What problem are you seeing with this build? Satish From srvenkat at utexas.edu Mon May 13 12:40:11 2024 From: srvenkat at utexas.edu (Sreeram R Venkat) Date: Mon, 13 May 2024 12:40:11 -0500 Subject: [petsc-users] [SLEPc] Best method to compute all eigenvalues of a MatShell Message-ID: I have a MatShell object that computes matrix-vector products of a dense symmetric matrix of size NxN. The MatShell does not actually form the dense matrix, so it is never in memory/storage. For my application, N ranges from 1e4 to 1e5. I want to compute the full spectrum of this matrix. For an example with N ~1e4, I was able to use SLEPc's Krylov-Schur solver to get the spectrum in about 3 hours running on 6 A100 GPUs. There, I had set the MPD to 2000. Before moving on to larger matrices, I wanted to check whether this is the best way to go about it. I saw on other posts that for computing full spectra of dense matrices, it is generally better to go with LAPACK/SCALAPACK. Is the same true for MatShells of dense matrices? The matrix-vector products with the shell themselves are really cheap, so I can form the dense matrix with MatComputeOperator() and store it to later compute with another solver if needed. If SLEPc is not a bad option, what is a good way to select MPD/NCV? I do need the full spectrum here since I am trying to analyze how the spectral decay changes for different problem configurations. Thanks for your help, Sreeram -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Mon May 13 13:07:24 2024 From: balay at mcs.anl.gov (Satish Balay) Date: Mon, 13 May 2024 13:07:24 -0500 (CDT) Subject: [petsc-users] Can not find the header file for OpenMPI In-Reply-To: <99e9f80b-617c-d925-799a-9dbd9afc7265@mcs.anl.gov> References: <3b083eba-2b47-b8ad-932f-e0697a5b00d8@mcs.anl.gov> <99e9f80b-617c-d925-799a-9dbd9afc7265@mcs.anl.gov> Message-ID: <48bd0988-fa7f-b595-3dde-0652bc1ea6c8@mcs.anl.gov> > Includes: -I/usr/lib64/openmpi/include (*This is not the right include directory*) So this is the only issue? Does this dir not exist? If so, you can try the following fix: https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/merge_requests/7545__;!!G_uCfscf7eWS!a7OhWBQn8Zp5RfRnLqjH2JjyNYkp3CksuvTPa8FN5YXKhfigeeaNOYe29DFk4jQywze4XBM6KGVRw1AFXd6_vj0$ However if this dir exists [i.e when --with-pkg-dir=DIR is specified and DIR/include exists] - configure will have no way of figuring out that this location is incorrect. And it should cause no harm to include it in compile commands. Satish On Mon, 13 May 2024, Satish Balay via petsc-users wrote: > On Mon, 13 May 2024, neil liu wrote: > > > > I also tried the 2nd way, it didn't work. > > configure.log attached is successful. > > >>>>>>>> > Configure Options: --configModules=PETSc.Configure --optionsModule=config.compilerOptions --download-fblaslapack --with-mpi-dir=/usr/lib64/openmpi > > Compilers: > C Compiler: /usr/lib64/openmpi/bin/mpicc -fPIC -Wall -Wwrite-strings -Wno-unknown-pragmas -Wno-lto-type-mismatch -Wno-stringop-overflow -fstack-protector -fvisibility=hidden -g3 -O0 > Version: gcc (GCC) 11.4.1 20231218 (Red Hat 11.4.1-3) > C++ Compiler: /usr/lib64/openmpi/bin/mpicxx -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -Wno-lto-type-mismatch -Wno-psabi -fstack-protector -fvisibility=hidden -g -O0 -std=gnu++20 -fPIC > Version: g++ (GCC) 11.4.1 20231218 (Red Hat 11.4.1-3) > Fortran Compiler: /usr/lib64/openmpi/bin/mpif90 -fPIC -Wall -ffree-line-length-none -ffree-line-length-0 -Wno-lto-type-mismatch -Wno-unused-dummy-argument -g -O0 > Version: GNU Fortran (GCC) 11.4.1 20231218 (Red Hat 11.4.1-3) > <<<<< > > So the correct compilers from the specified mpi location /usr/lib64/openmpi got used. > > What problem are you seeing with this build? > > Satish > From knepley at gmail.com Mon May 13 13:10:54 2024 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 13 May 2024 14:10:54 -0400 Subject: [petsc-users] [SLEPc] Best method to compute all eigenvalues of a MatShell In-Reply-To: References: Message-ID: On Mon, May 13, 2024 at 1:40?PM Sreeram R Venkat wrote: > I have a MatShell object that computes matrix-vector products of a dense > symmetric matrix of size NxN. The MatShell does not actually form the dense > matrix, so it is never in memory/storage. For my application, N ranges from > 1e4 to 1e5. I want > ZjQcmQRYFpfptBannerStart > This Message Is From an External Sender > This message came from outside your organization. > > ZjQcmQRYFpfptBannerEnd > I have a MatShell object that computes matrix-vector products of a dense > symmetric matrix of size NxN. The MatShell does not actually form the dense > matrix, so it is never in memory/storage. For my application, N ranges from > 1e4 to 1e5. > > I want to compute the full spectrum of this matrix. For an example with N > ~1e4, I was able to use SLEPc's Krylov-Schur solver to get the spectrum in > about 3 hours running on 6 A100 GPUs. There, I had set the MPD to 2000. > Before moving on to larger matrices, I wanted to check whether this is the > best way to go about it. I saw on other posts that for computing full > spectra of dense matrices, it is generally better to go with > LAPACK/SCALAPACK. Is the same true for MatShells of dense matrices? The > matrix-vector products with the shell themselves are really cheap, so I can > form the dense matrix with MatComputeOperator() and store it to later > compute with another solver if needed. If SLEPc is not a bad option, what > is a good way to select MPD/NCV? > You can select LAPACK through SLEPc. I would definitely try it since it is easy. For the complete spectrum, it is likely to be faster (modulo how fast KS converges). Thanks, Matt > I do need the full spectrum here since I am trying to analyze how the > spectral decay changes for different problem configurations. > > Thanks for your help, > Sreeram > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!cLlaAsyk3G1fWIZGlTet7-bB1VMxzmbrmnjxP0IPtxxAA-mYn4vGvI9tzI2x7GXRMPqOjNBQZZrrDoH5JihG$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From jroman at dsic.upv.es Mon May 13 13:24:35 2024 From: jroman at dsic.upv.es (Jose E. Roman) Date: Mon, 13 May 2024 20:24:35 +0200 Subject: [petsc-users] [SLEPc] Best method to compute all eigenvalues of a MatShell In-Reply-To: References: Message-ID: An HTML attachment was scrubbed... URL: From liufield at gmail.com Mon May 13 13:53:13 2024 From: liufield at gmail.com (neil liu) Date: Mon, 13 May 2024 14:53:13 -0400 Subject: [petsc-users] Can not find the header file for OpenMPI In-Reply-To: <48bd0988-fa7f-b595-3dde-0652bc1ea6c8@mcs.anl.gov> References: <3b083eba-2b47-b8ad-932f-e0697a5b00d8@mcs.anl.gov> <99e9f80b-617c-d925-799a-9dbd9afc7265@mcs.anl.gov> <48bd0988-fa7f-b595-3dde-0652bc1ea6c8@mcs.anl.gov> Message-ID: Thanks a lot. That is the only issue. /usr/lib64/openmpi/include doesn't exist. The configuration can be done without issue with the 2nd method. But when I did make, the following warning came. I will study your link. Thanks, f951: Warning: Nonexistent include directory ?/usr/lib64/openmpi/include? [-Wmissing-include-dirs] CC linux-gnu-c-debug/obj/src/tao/unconstrained/impls/ntl/ntl.o CC linux-gnu-c-debug/obj/src/tao/util/tao_util.o FC linux-gnu-c-debug/obj/src/sys/objects/f2003-src/fsrc/optionenum.o f951: Warning: Nonexistent include directory ?/usr/lib64/openmpi/include? [-Wmissing-include-dirs] CC linux-gnu-c-debug/obj/src/tao/unconstrained/impls/owlqn/owlqn.o CC linux-gnu-c-debug/obj/src/tao/unconstrained/impls/ntr/ntr.o CC linux-gnu-c-debug/obj/src/tao/unconstrained/impls/nls/nls.o FC linux-gnu-c-debug/obj/src/dm/f90-mod/petscdmswarmmod.o f951: Warning: Nonexistent include directory ?/usr/lib64/openmpi/include? [-Wmissing-include-dirs] FC linux-gnu-c-debug/obj/src/dm/f90-mod/petscdmdamod.o f951: Warning: Nonexistent include directory ?/usr/lib64/openmpi/include? [-Wmissing-include-dirs] FC linux-gnu-c-debug/obj/src/dm/f90-mod/petscdmplexmod.o f951: Warning: Nonexistent include directory ?/usr/lib64/openmpi/include? [-Wmissing-include-dirs] FC linux-gnu-c-debug/obj/src/ksp/f90-mod/petsckspdefmod.o f951: Warning: Nonexistent include directory ?/usr/lib64/openmpi/include? [-Wmissing-include-dirs] FC linux-gnu-c-debug/obj/src/ksp/f90-mod/petscpcmod.o f951: Warning: Nonexistent include directory ?/usr/lib64/openmpi/include? [-Wmissing-include-dirs] FC linux-gnu-c-debug/obj/src/ksp/f90-mod/petsckspmod.o f951: Warning: Nonexistent include directory ?/usr/lib64/openmpi/include? [-Wmissing-include-dirs] FC linux-gnu-c-debug/obj/src/snes/f90-mod/petscsnesmod.o f951: Warning: Nonexistent include directory ?/usr/lib64/openmpi/include? [-Wmissing-include-dirs] FC linux-gnu-c-debug/obj/src/ts/f90-mod/petsctsmod.o f951: Warning: Nonexistent include directory ?/usr/lib64/openmpi/include? [-Wmissing-include-dirs] FC linux-gnu-c-debug/obj/src/tao/f90-mod/petsctaomod.o f951: Warning: Nonexistent include directory ?/usr/lib64/openmpi/include? [-Wmissing-include-dirs] On Mon, May 13, 2024 at 2:07?PM Satish Balay wrote: > > Includes: -I/usr/lib64/openmpi/include (*This is not the right > include directory*) > > So this is the only issue? Does this dir not exist? If so, you can try the > following fix: > > https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/merge_requests/7545__;!!G_uCfscf7eWS!c2QjEeOosTxvYgWhf7la2e2Y8u4qLZmaeLr2oVBLbPIxtR4uDBVQ984G_-0YLJTunEXOkWm2dRXZCq1GLGKAtQ$ > > However if this dir exists [i.e when --with-pkg-dir=DIR is specified and > DIR/include exists] - configure will have no way of figuring out that this > location is incorrect. And it should cause no harm to include it in compile > commands. > > Satish > > > On Mon, 13 May 2024, Satish Balay via petsc-users wrote: > > > On Mon, 13 May 2024, neil liu wrote: > > > > > > > I also tried the 2nd way, it didn't work. > > > > configure.log attached is successful. > > > > >>>>>>>> > > Configure Options: --configModules=PETSc.Configure > --optionsModule=config.compilerOptions --download-fblaslapack > --with-mpi-dir=/usr/lib64/openmpi > > > > Compilers: > > C Compiler: /usr/lib64/openmpi/bin/mpicc -fPIC -Wall > -Wwrite-strings -Wno-unknown-pragmas -Wno-lto-type-mismatch > -Wno-stringop-overflow -fstack-protector -fvisibility=hidden -g3 -O0 > > Version: gcc (GCC) 11.4.1 20231218 (Red Hat 11.4.1-3) > > C++ Compiler: /usr/lib64/openmpi/bin/mpicxx -Wall > -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas > -Wno-lto-type-mismatch -Wno-psabi -fstack-protector -fvisibility=hidden -g > -O0 -std=gnu++20 -fPIC > > Version: g++ (GCC) 11.4.1 20231218 (Red Hat 11.4.1-3) > > Fortran Compiler: /usr/lib64/openmpi/bin/mpif90 -fPIC -Wall > -ffree-line-length-none -ffree-line-length-0 -Wno-lto-type-mismatch > -Wno-unused-dummy-argument -g -O0 > > Version: GNU Fortran (GCC) 11.4.1 20231218 (Red Hat 11.4.1-3) > > <<<<< > > > > So the correct compilers from the specified mpi location > /usr/lib64/openmpi got used. > > > > What problem are you seeing with this build? > > > > Satish > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jroman at dsic.upv.es Mon May 13 13:54:02 2024 From: jroman at dsic.upv.es (Jose E. Roman) Date: Mon, 13 May 2024 20:54:02 +0200 Subject: [petsc-users] [SLEPc] Best method to compute all eigenvalues of a MatShell In-Reply-To: References: Message-ID: <8BACAD64-4266-4197-AAE5-930E07DA73E4@dsic.upv.es> An HTML attachment was scrubbed... URL: From srvenkat at utexas.edu Mon May 13 14:13:31 2024 From: srvenkat at utexas.edu (Sreeram R Venkat) Date: Mon, 13 May 2024 14:13:31 -0500 Subject: [petsc-users] [SLEPc] Best method to compute all eigenvalues of a MatShell In-Reply-To: <8BACAD64-4266-4197-AAE5-930E07DA73E4@dsic.upv.es> References: <8BACAD64-4266-4197-AAE5-930E07DA73E4@dsic.upv.es> Message-ID: Apologies, I accidentally hit "reply" instead of "reply-all." Thank you for the reference. Actually, I just tested that N ~ 1e4 case where I had saved the dense matrix to a python-readable format. Using scipy.linalg.eigvalsh, I got the eigenvalues in ~1.5 minutes. They agree with the ones I got from SLEPc as well. I believe the scipy routine just calls LAPACK in the backend, so I assume I would get similar results if I used the LAPACK solver in SLEPc. The MatShell object I am working with requires me to operate on multiple processes to fit all the data, so to use the LAPACK solver in SLEPc, I believe I would have to read/write the dense matrix as well. So at the cost of storing the dense matrix, I think the dense eigensolver seems to be the way to go. I will need to store at least one large dense matrix of this size for my application, so that should not be the limiting factor. Sreeram On Mon, May 13, 2024 at 1:54?PM Jose E. Roman wrote: > Please respond to the list. > > The mpd parameter means "maximum projected dimension". You can think of > the projected problem as the "sequential" part of the computation, that is > not parallelized ("small" dense eigenproblem). When you run with MPI, > everything will scale reasonably well except that part, so it is better to > keep it small, specially when you request many eigenvalues. A value > mpd=2000 might be too large, it may be better to reduce it to 500, say. The > paper https://urldefense.us/v3/__https://doi.org/10.1016/j.cpc.2010.09.007__;!!G_uCfscf7eWS!elC1IPQR-J0CwJ8mp-zPPEXvkzawO1RCvusBFlcaG2xfHHzqRFzOQqHgXXqpFf1EBfhJx3LzgTYotr_Jwv1_o0wyag$ includes a discussion on > the ncv and mpd paramters, mostly in terms of memory usage. > > Jose > > > El 13 may 2024, a las 20:33, Sreeram R Venkat > escribi?: > > > > Thank you for the suggestions. I will try out LAPACK/ELPA and the n/2 > method. For the latter, how should I choose the MPD? In the examples I > could find online, it looked like they were using something like mpd ~ > nev/10. > > > > Sreeram > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From srvenkat at utexas.edu Mon May 13 14:51:38 2024 From: srvenkat at utexas.edu (Sreeram R Venkat) Date: Mon, 13 May 2024 14:51:38 -0500 Subject: [petsc-users] [SLEPc] Best method to compute all eigenvalues of a MatShell In-Reply-To: References: <8BACAD64-4266-4197-AAE5-930E07DA73E4@dsic.upv.es> Message-ID: I also just ran with cupy.linalg.eigvalsh (which wraps cuSOLVER), and it only took 3.1 seconds. I will probably use this, but it is good to know about the SLEPc cases if I don't need the full spectrum or have sparse matrices, etc. Thanks, Sreeram On Mon, May 13, 2024 at 2:13?PM Sreeram R Venkat wrote: > > Apologies, I accidentally hit "reply" instead of "reply-all." > > Thank you for the reference. Actually, I just tested that N ~ 1e4 case > where I had saved the dense matrix to a python-readable format. Using > scipy.linalg.eigvalsh, I got the eigenvalues in ~1.5 minutes. They agree > with the ones I got from SLEPc as well. I believe the scipy routine just > calls LAPACK in the backend, so I assume I would get similar results if I > used the LAPACK solver in SLEPc. > > The MatShell object I am working with requires me to operate on multiple > processes to fit all the data, so to use the LAPACK solver in SLEPc, I > believe I would have to read/write the dense matrix as well. > > So at the cost of storing the dense matrix, I think the dense eigensolver > seems to be the way to go. I will need to store at least one large dense > matrix of this size for my application, so that should not be the limiting > factor. > > Sreeram > > On Mon, May 13, 2024 at 1:54?PM Jose E. Roman wrote: > >> Please respond to the list. >> >> The mpd parameter means "maximum projected dimension". You can think of >> the projected problem as the "sequential" part of the computation, that is >> not parallelized ("small" dense eigenproblem). When you run with MPI, >> everything will scale reasonably well except that part, so it is better to >> keep it small, specially when you request many eigenvalues. A value >> mpd=2000 might be too large, it may be better to reduce it to 500, say. The >> paper https://urldefense.us/v3/__https://doi.org/10.1016/j.cpc.2010.09.007__;!!G_uCfscf7eWS!f2yCdZSL_oAtFe32noZqGhwa3rbTgx34qUGo9HuXkP6nQm_-ryGpPFxDt_2hEv3HzLy7POL3H28uob9D4qchyj3kXA$ includes a discussion on >> the ncv and mpd paramters, mostly in terms of memory usage. >> >> Jose >> >> > El 13 may 2024, a las 20:33, Sreeram R Venkat >> escribi?: >> > >> > Thank you for the suggestions. I will try out LAPACK/ELPA and the n/2 >> method. For the latter, how should I choose the MPD? In the examples I >> could find online, it looked like they were using something like mpd ~ >> nev/10. >> > >> > Sreeram >> > >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From srvenkat at utexas.edu Mon May 13 19:16:43 2024 From: srvenkat at utexas.edu (Sreeram R Venkat) Date: Mon, 13 May 2024 19:16:43 -0500 Subject: [petsc-users] Using Compute-Sanitizer with PETSc Message-ID: I am trying to check my program for GPU memory leaks with the compute-sanitizer tool. If I run my application with: mpiexec -n 1 compute-sanitizer --tool memcheck --leak-check full ./a.out args I get the message: Error: No attachable process found. compute-sanitizer timed-out. Adding --target-processes all does not help. Is there anything else I should be doing? Thanks, Sreeram -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Mon May 13 20:32:57 2024 From: bsmith at petsc.dev (Barry Smith) Date: Mon, 13 May 2024 21:32:57 -0400 Subject: [petsc-users] Using Compute-Sanitizer with PETSc In-Reply-To: References: Message-ID: Depending on your mpi mpiexec is not needed so compute-sanitizer --tool memcheck --leak-check full ./a.out args may work > On May 13, 2024, at 8:16?PM, Sreeram R Venkat wrote: > > This Message Is From an External Sender > This message came from outside your organization. > I am trying to check my program for GPU memory leaks with the compute-sanitizer tool. If I run my application with: > mpiexec -n 1 compute-sanitizer --tool memcheck --leak-check full ./a.out args > > I get the message: > Error: No attachable process found. compute-sanitizer timed-out. > > Adding --target-processes all does not help. > > Is there anything else I should be doing? > > Thanks, > Sreeram -------------- next part -------------- An HTML attachment was scrubbed... URL: From a.croucher at auckland.ac.nz Mon May 13 20:33:02 2024 From: a.croucher at auckland.ac.nz (Adrian Croucher) Date: Tue, 14 May 2024 13:33:02 +1200 Subject: [petsc-users] GMSH entities Message-ID: An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon May 13 20:44:14 2024 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 13 May 2024 21:44:14 -0400 Subject: [petsc-users] GMSH entities In-Reply-To: References: Message-ID: On Mon, May 13, 2024 at 9:33?PM Adrian Croucher wrote: > hi, We often create meshes in GMSH format using the meshio library. This > works OK if we stick to GMSH file format 2. 2. If we use GMSH file format > 4. 1, DMPlex can't read them because it expects the "Entities" section to > be present: [0]PETSC > ZjQcmQRYFpfptBannerStart > This Message Is From an External Sender > This message came from outside your organization. > > ZjQcmQRYFpfptBannerEnd > > hi, > > We often create meshes in GMSH format using the meshio library. This > works OK if we stick to GMSH file format 2.2. > > If we use GMSH file format 4.1, DMPlex can't read them because it > expects the "Entities" section to be present: > > [0]PETSC ERROR: Unexpected data in file > [0]PETSC ERROR: File is not a valid Gmsh file, expecting $Entities, not > $Nodes > [0]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!ZfBS1KM5EBZ7ZJIu6lBKFcclVMmXteXsW8m9HBEZ5tIf0u_3duEFt9eXKF7FcorQSAQqD5SbJrYh4C8rX676S_IMI_sp6naX$ for trouble shooting. > [0]PETSC ERROR: Petsc Development GIT revision: > v3.21.1-124-g2d06e2faec8 GIT Date: 2024-05-08 19:31:33 +0000 > [0]PETSC ERROR: waiwera on a main-debug named EN438880 by acro018 Tue > May 14 11:25:54 2024 > [0]PETSC ERROR: Configure options --with-x --download-hdf5 > --download-zlib --download-netcdf --download-pnetcdf --download-exodusii > --download-triangle --download-ptscotch --download-chaco --download-hypre > [0]PETSC ERROR: #1 GmshExpect() at > /home/acro018/software/PETSc/code/src/dm/impls/plex/plexgmsh.c:270 > [0]PETSC ERROR: #2 DMPlexCreateGmsh() at > /home/acro018/software/PETSc/code/src/dm/impls/plex/plexgmsh.c:1608 > [0]PETSC ERROR: #3 DMPlexCreateGmshFromFile() at > /home/acro018/software/PETSc/code/src/dm/impls/plex/plexgmsh.c:1469 > [0]PETSC ERROR: #4 DMPlexCreateFromFile() at > /home/acro018/software/PETSc/code/src/dm/impls/plex/plexcreate.c:5804 > > By default meshio doesn't seem to write the Entities section. From what > I can gather, it is optional. > > Am I right that this section is not optional in DMPlex? > > I wish GMsh was clearer about what is optional: https://urldefense.us/v3/__https://gmsh.info/doc/texinfo/gmsh.html*MSH-file-format__;Iw!!G_uCfscf7eWS!fS2tI8aC3rUnGOv-LbLPj1PRyijAPB-EO54CwUBP6pNFD-VKeGE98pOHepNMSTP_krHBKTZNk92Bx01NrC-T$ They do talk about it, but not exhaustively. GMsh always writes and $Entities block from what I can tell. I can make it optional, it just might take until after the PETSc Meeting. Thanks, Matt - Adrian > > -- > Dr Adrian Croucher > Senior Research Fellow > Department of Engineering Science > Waipapa Taumata Rau / University of Auckland, New Zealand > email: a.croucher at auckland.ac.nz > tel: +64 (0)9 923 4611 > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fS2tI8aC3rUnGOv-LbLPj1PRyijAPB-EO54CwUBP6pNFD-VKeGE98pOHepNMSTP_krHBKTZNk92Bx1gj7y2J$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From wurunjian at gmail.com Mon May 13 21:00:28 2024 From: wurunjian at gmail.com (Runjian Wu) Date: Tue, 14 May 2024 10:00:28 +0800 Subject: [petsc-users] duplicated libs In-Reply-To: <16654C9F-0761-4A68-A7E6-7DE5FA35E559@petsc.dev> References: <16654C9F-0761-4A68-A7E6-7DE5FA35E559@petsc.dev> Message-ID: Thanks for your reply! Since I can manually remove duplicates, how about adding a function to automatically remove duplicates at the end of "configure" in the next PETSc version? Runjian On 5/13/2024 9:31 PM, Barry Smith wrote: Because the order of the libraries can be important, it is difficult for ./configure to remove unneeded duplicates automatically. You can manually remove duplicates by editing $PETSC_ARCH/lib/petsc/conf/petscvariables after running ./configure Barry On May 13, 2024, at 7:47?AM, Runjian Wu wrote: This Message Is From an External Sender This message came from outside your organization. Hi all, After I compiled PETSc, I found some duplicated libs in the variable PETSC_EXTERNAL_LIB_BASIC, e.g., -lm, -lgfortran -lstdc++. I am curious how it happened and how to remove the duplicates? Thanks, Runjian Wu -------------- next part -------------- An HTML attachment was scrubbed... URL: From a.croucher at auckland.ac.nz Mon May 13 21:04:32 2024 From: a.croucher at auckland.ac.nz (Adrian Croucher) Date: Tue, 14 May 2024 14:04:32 +1200 Subject: [petsc-users] GMSH entities In-Reply-To: References: Message-ID: <67984175-20da-4323-98d3-c2124dfb43ea@auckland.ac.nz> On 14/05/24 1:44 pm, Matthew Knepley wrote: > I wish GMsh was clearer about what is optional: > https://urldefense.us/v3/__https://gmsh.info/doc/texinfo/gmsh.html*MSH-file-format__;Iw!!G_uCfscf7eWS!aCypJMAzwWHAJLXGNJmSthDjbHcU-8_MdsaXZ4d1r1RKyL0bqIv5ZuLmQtV6ve4XcTjapf38-bsdDOLEDAlhWoaaGpmjDCYs$ > > > They do talk about it, but not exhaustively. GMsh always writes and > $Entities block from what I can tell. > I can make it optional, it just might take until after the PETSc Meeting. Looks like $Entities are optional: https://urldefense.us/v3/__https://gitlab.onelab.info/gmsh/gmsh/-/commit/b5feba2af57181ffa946d3f0c494b014603c6efa__;!!G_uCfscf7eWS!aCypJMAzwWHAJLXGNJmSthDjbHcU-8_MdsaXZ4d1r1RKyL0bqIv5ZuLmQtV6ve4XcTjapf38-bsdDOLEDAlhWoaaGjNKbSSU$ I can also load a GMSH 4.1 file without $Entities into GMSH itself and it doesn't complain, suggesting that they are indeed optional. If the $Entities aren't strictly needed for anything in DMPlex (which I'm guessing they aren't, as the GMSH file format 2.2 doesn't even have them) then it would be useful not to require them. - Adrian -- Dr Adrian Croucher Senior Research Fellow Department of Engineering Science Waipapa Taumata Rau / University of Auckland, New Zealand email:a.croucher at auckland.ac.nz tel: +64 (0)9 923 4611 -------------- next part -------------- An HTML attachment was scrubbed... URL: From wurunjian at gmail.com Mon May 13 21:15:14 2024 From: wurunjian at gmail.com (Runjian Wu) Date: Tue, 14 May 2024 10:15:14 +0800 Subject: [petsc-users] duplicated libs In-Reply-To: <38fc058b-360d-073f-aac9-6a6d47e7fac2@mcs.anl.gov> References: <38fc058b-360d-073f-aac9-6a6d47e7fac2@mcs.anl.gov> Message-ID: First clarification, NO issues happened. I am just curious about the duplicated libraries while I compile PFLOTRAN with PETSc. PETSc version: 3.16.2 Configure Options: --configModules=PETSc.Configure --optionsModule=config.compilerOptions --with-make-np=4 --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 --download-fblaslapack=yes --download-hdf5=/home/wurunjian/petsc_pkg/hdf5-1.12.1.tar.bz2 --download-hdf5-fortran-bindings=yes --download-metis=yes --download-parmetis=yes --CFLAGS=-O3 --CXXFLAGS=-O3 --FFLAGS=-O3 --with-debugging=0 PETSC_EXTERNAL_LIB_BASIC = -Wl,-rpath,/home/wurunjian/petsc/arch-linux2-c-opt/lib -L/home/wurunjian/petsc/arch-linux2-c-opt/lib -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/11 -L/usr/lib/gcc/x86_64-linux-gnu/11 -lflapack -lfblas -lhdf5hl_fortran -lhdf5_fortran -lhdf5_hl -lhdf5 -lparmetis -lmetis -lm -lstdc++ -ldl -lmpichfort -lmpich -lgfortran -lm -lgfortran -lm -lgcc_s -lquadmath -lstdc++ -ldl On Mon, May 13, 2024 at 9:34?PM Satish Balay wrote: > what version of PETSc? What configure command? What do you have for > PETSC_EXTERNAL_LIB_BASIC? > > You can send configure.log for your build to petsc-maint > > Generally duplicates should not cause grief. [as one needs them to > overcome circular dependencies]. > > What issues are you seeing? [send relevant logs] > > If the duplicates are coming form "compiler library" detection part of > configure - you can skip this part by listing the correct "compiler > libraries" > > for ex: with gcc/gfortran: > > LIBS="-lgfortran -lstdc++" > > However -lgfortran could be required for gcc - but a duplicate for > gfortran. > > Satish > > On Mon, 13 May 2024, Runjian Wu wrote: > > > Hi all, > > > > After I compiled PETSc, I found some duplicated libs in the variable > > PETSC_EXTERNAL_LIB_BASIC, e.g., -lm, -lgfortran -lstdc++. I am curious > how > > it happened and how to remove the duplicates? > > > > Thanks, > > > > Runjian Wu > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Mon May 13 21:33:16 2024 From: bsmith at petsc.dev (Barry Smith) Date: Mon, 13 May 2024 22:33:16 -0400 Subject: [petsc-users] VecGetArrayF90 vs. VecGetArrayReadF90 In-Reply-To: References: <21A28F71-57D4-48D2-9029-69B864B23C50@petsc.dev> Message-ID: It errors in C because the argument is labeled with const, but there does not seem to be a way in Fortran to indicate an array is read only. > On May 13, 2024, at 10:21?PM, Runjian Wu wrote: > > I only know the "intent(in)" attribute for the dummy arguments. > > In the counter function VecGetArrayRead(..), if I write values, will the compiler report an error? > > Runjian > > On Mon, May 13, 2024 at 9:36?PM Barry Smith > wrote: >> >> I couldn't find a way in Fortran to declare an array as read-only. Is there such support? >> >> Barry >> >> >>> On May 13, 2024, at 7:28?AM, Runjian Wu > wrote: >>> >>> This Message Is From an External Sender >>> This message came from outside your organization. >>> Hi all, >>> >>> I have a question about VecGetArrayReadF90(...). If I use VecGetArrayReadF90(...), I can still write entries into the array like VecGetArrayF90(...). Is it possible to report an error during compile process? >>> >>> Thanks, >>> >>> Runjian Wu >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Mon May 13 21:34:25 2024 From: bsmith at petsc.dev (Barry Smith) Date: Mon, 13 May 2024 22:34:25 -0400 Subject: [petsc-users] duplicated libs In-Reply-To: References: <16654C9F-0761-4A68-A7E6-7DE5FA35E559@petsc.dev> Message-ID: It is not always safe to remove duplicate libraries listed in different places in the list. Hence we cannot simply always remove them. Barry > On May 13, 2024, at 10:00?PM, Runjian Wu wrote: > > Thanks for your reply! Since I can manually remove duplicates, how about adding a function to automatically remove duplicates at the end of "configure" in the next PETSc version? > > > Runjian > > > > On 5/13/2024 9:31 PM, Barry Smith wrote: >> >> Because the order of the libraries can be important, it is difficult for ./configure to remove unneeded duplicates automatically. >> >> You can manually remove duplicates by editing $PETSC_ARCH/lib/petsc/conf/petscvariables after running ./configure >> >> Barry >> >> >> >>> On May 13, 2024, at 7:47?AM, Runjian Wu wrote: >>> >>> This Message Is From an External Sender >>> This message came from outside your organization. >>> Hi all, >>> >>> After I compiled PETSc, I found some duplicated libs in the variable PETSC_EXTERNAL_LIB_BASIC, e.g., -lm, -lgfortran -lstdc++. I am curious how it happened and how to remove the duplicates? >>> >>> Thanks, >>> >>> Runjian Wu >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From matteo.semplice at uninsubria.it Mon May 13 23:14:43 2024 From: matteo.semplice at uninsubria.it (Matteo Semplice) Date: Tue, 14 May 2024 06:14:43 +0200 Subject: [petsc-users] DMPlex periodic face coordinates Message-ID: Dear petsc-users, ??? I am playing with DMPlexGetCellCoordinates and observing that it returns correct periodic coordinates for cells, but not for faces. More precisely, adding PetscCall(DMPlexGetHeightStratum(dm, 1, &fStart, &fEnd)); ??? for (f = fStart; f < fEnd; ++f) { ????? const PetscScalar *array; ????? PetscScalar?????? *x = NULL; ????? PetscInt?????????? ndof; ????? PetscBool????????? isDG; ????? PetscCall(DMPlexGetCellCoordinates(dm, f, &isDG, &ndof, &array, &x)); ????? PetscCheck(ndof % cdim == 0, PETSC_COMM_SELF, PETSC_ERR_ARG_INCOMP, "ndof not divisible by cdim"); ????? PetscCall(PetscPrintf(PETSC_COMM_SELF, "Face #%" PetscInt_FMT " coordinates\n", f - fStart)); ????? for (PetscInt i = 0; i < ndof; i += cdim) PetscCall(PetscScalarView(cdim, &x[i], PETSC_VIEWER_STDOUT_SELF)); ????? PetscCall(DMPlexRestoreCellCoordinates(dm, f, &isDG, &ndof, &array, &x)); ??? } to src/dm/impls/plex/tutorials/ex8.c, I get $ ./ex8 -dm_plex_dim 2 -petscspace_degree 1 -dm_plex_simplex 0 -dm_plex_box_faces 3,2 -dm_plex_box_bd periodic,none -dm_view -view_coord DM Object: box 1 MPI process ? type: plex box in 2 dimensions: ? Number of 0-cells per rank: 9 ? Number of 1-cells per rank: 15 ? Number of 2-cells per rank: 6 Periodic mesh (PERIODIC, NONE) coordinates localized [...] Element #2 coordinates ?0:?? 6.6667e-01?? 0.0000e+00 ?0:?? 1.0000e+00?? 0.0000e+00 <<<----- correct ?0:?? 1.0000e+00?? 5.0000e-01 ?0:?? 6.6667e-01?? 5.0000e-01 [...] Face #0 coordinates ?0:?? 0.0000e+00?? 0.0000e+00 ?0:?? 3.3333e-01?? 0.0000e+00 Face #1 coordinates ?0:?? 3.3333e-01?? 0.0000e+00 ?0:?? 6.6667e-01?? 0.0000e+00 Face #2 coordinates ?0:?? 6.6667e-01?? 0.0000e+00 ?0:?? 0.0000e+00?? 0.0000e+00? <<<---- should be (0.66,0.00) and (1.00,0.00) Is there a way to recover correct periodic coordinates in the case of periodic DMPLex? Thanks in advance Matteo -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Tue May 14 00:09:02 2024 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 14 May 2024 00:09:02 -0500 (CDT) Subject: [petsc-users] duplicated libs In-Reply-To: References: <38fc058b-360d-073f-aac9-6a6d47e7fac2@mcs.anl.gov> Message-ID: <5597d213-d39a-f428-7b7f-91413590ce29@mcs.anl.gov> You can try using latest petsc version [3.21] - the list should be a bit more cleaner with it. balay at pj01:~/petsc$ ./configure --download-mpich --download-fblaslapack --download-hdf5 --with-hdf5-fortran-bindings --download-metis --download-parmetis COPTFLAGS=-O3 CXXOPTFLAGS=-O3 FOPTFLAGS=-O3 --with-debugging=0 balay at pj01:~/petsc$ grep ^PETSC_EXTERNAL_LIB_BASIC arch-linux-c-opt/lib/petsc/conf/petscvariables PETSC_EXTERNAL_LIB_BASIC = -Wl,-rpath,/home/balay/petsc/arch-linux-c-opt/lib -L/home/balay/petsc/arch-linux-c-opt/lib -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/13 -L/usr/lib/gcc/x86_64-redhat-linux/13 -lflapack -lfblas -lparmetis -lmetis -lhdf5hl_fortran -lhdf5_fortran -lhdf5_hl -lhdf5 -lm -lX11 -lmpifort -lmpi -lgfortran -lm -lgfortran -lm -lgcc_s -lquadmath -lstdc++ Satish On Tue, 14 May 2024, Runjian Wu wrote: > First clarification, NO issues happened. I am just curious about the > duplicated libraries while I compile PFLOTRAN with PETSc. > > PETSc version: 3.16.2 > > Configure Options: --configModules=PETSc.Configure > --optionsModule=config.compilerOptions --with-make-np=4 --with-cc=mpicc > --with-cxx=mpicxx --with-fc=mpif90 --download-fblaslapack=yes > --download-hdf5=/home/wurunjian/petsc_pkg/hdf5-1.12.1.tar.bz2 > --download-hdf5-fortran-bindings=yes --download-metis=yes > --download-parmetis=yes --CFLAGS=-O3 --CXXFLAGS=-O3 --FFLAGS=-O3 > --with-debugging=0 > > PETSC_EXTERNAL_LIB_BASIC = > -Wl,-rpath,/home/wurunjian/petsc/arch-linux2-c-opt/lib > -L/home/wurunjian/petsc/arch-linux2-c-opt/lib > -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/11 > -L/usr/lib/gcc/x86_64-linux-gnu/11 -lflapack -lfblas -lhdf5hl_fortran > -lhdf5_fortran -lhdf5_hl -lhdf5 -lparmetis -lmetis -lm -lstdc++ -ldl > -lmpichfort -lmpich -lgfortran -lm -lgfortran -lm -lgcc_s -lquadmath > -lstdc++ -ldl > > On Mon, May 13, 2024 at 9:34?PM Satish Balay wrote: > > > what version of PETSc? What configure command? What do you have for > > PETSC_EXTERNAL_LIB_BASIC? > > > > You can send configure.log for your build to petsc-maint > > > > Generally duplicates should not cause grief. [as one needs them to > > overcome circular dependencies]. > > > > What issues are you seeing? [send relevant logs] > > > > If the duplicates are coming form "compiler library" detection part of > > configure - you can skip this part by listing the correct "compiler > > libraries" > > > > for ex: with gcc/gfortran: > > > > LIBS="-lgfortran -lstdc++" > > > > However -lgfortran could be required for gcc - but a duplicate for > > gfortran. > > > > Satish > > > > On Mon, 13 May 2024, Runjian Wu wrote: > > > > > Hi all, > > > > > > After I compiled PETSc, I found some duplicated libs in the variable > > > PETSC_EXTERNAL_LIB_BASIC, e.g., -lm, -lgfortran -lstdc++. I am curious > > how > > > it happened and how to remove the duplicates? > > > > > > Thanks, > > > > > > Runjian Wu > > > > > > > > From balay at mcs.anl.gov Tue May 14 00:23:09 2024 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 14 May 2024 00:23:09 -0500 (CDT) Subject: [petsc-users] duplicated libs In-Reply-To: <5597d213-d39a-f428-7b7f-91413590ce29@mcs.anl.gov> References: <38fc058b-360d-073f-aac9-6a6d47e7fac2@mcs.anl.gov> <5597d213-d39a-f428-7b7f-91413590ce29@mcs.anl.gov> Message-ID: And to have cleaner list - specify the (appropriate) compiler libraries via LIBS option balay at pj01:~/petsc$ mpif90 -show gfortran -I/software/mpich-4.1.1/include -I/software/mpich-4.1.1/include -L/software/mpich-4.1.1/lib -lmpifort -Wl,-rpath -Wl,/software/mpich-4.1.1/lib -Wl,--enable-new-dtags -lmpi balay at pj01:~/petsc$ ./configure --download-fblaslapack --download-hdf5 --with-hdf5-fortran-bindings --download-metis --download-parmetis COPTFLAGS=-O3 CXXOPTFLAGS=-O3 FOPTFLAGS=-O3 --with-debugging=0 LIBS="-lmpifort -lmpi -lgfortran -lstdc++" balay at pj01:~/petsc$ grep ^PETSC_EXTERNAL_LIB_BASIC arch-linux-c-opt/lib/petsc/conf/petscvariables PETSC_EXTERNAL_LIB_BASIC = -Wl,-rpath,/home/balay/petsc/arch-linux-c-opt/lib -L/home/balay/petsc/arch-linux-c-opt/lib -lflapack -lfblas -lparmetis -lmetis -lhdf5hl_fortran -lhdf5_fortran -lhdf5_hl -lhdf5 -lm -lX11 -lquadmath -lmpifort -lmpi -lgfortran -lstdc++ Satish On Tue, 14 May 2024, Satish Balay via petsc-users wrote: > You can try using latest petsc version [3.21] - the list should be a bit more cleaner with it. > > balay at pj01:~/petsc$ ./configure --download-mpich --download-fblaslapack --download-hdf5 --with-hdf5-fortran-bindings --download-metis --download-parmetis COPTFLAGS=-O3 CXXOPTFLAGS=-O3 FOPTFLAGS=-O3 --with-debugging=0 > > balay at pj01:~/petsc$ grep ^PETSC_EXTERNAL_LIB_BASIC arch-linux-c-opt/lib/petsc/conf/petscvariables > PETSC_EXTERNAL_LIB_BASIC = -Wl,-rpath,/home/balay/petsc/arch-linux-c-opt/lib -L/home/balay/petsc/arch-linux-c-opt/lib -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/13 -L/usr/lib/gcc/x86_64-redhat-linux/13 -lflapack -lfblas -lparmetis -lmetis -lhdf5hl_fortran -lhdf5_fortran -lhdf5_hl -lhdf5 -lm -lX11 -lmpifort -lmpi -lgfortran -lm -lgfortran -lm -lgcc_s -lquadmath -lstdc++ > > Satish > > On Tue, 14 May 2024, Runjian Wu wrote: > > > First clarification, NO issues happened. I am just curious about the > > duplicated libraries while I compile PFLOTRAN with PETSc. > > > > PETSc version: 3.16.2 > > > > Configure Options: --configModules=PETSc.Configure > > --optionsModule=config.compilerOptions --with-make-np=4 --with-cc=mpicc > > --with-cxx=mpicxx --with-fc=mpif90 --download-fblaslapack=yes > > --download-hdf5=/home/wurunjian/petsc_pkg/hdf5-1.12.1.tar.bz2 > > --download-hdf5-fortran-bindings=yes --download-metis=yes > > --download-parmetis=yes --CFLAGS=-O3 --CXXFLAGS=-O3 --FFLAGS=-O3 > > --with-debugging=0 > > > > PETSC_EXTERNAL_LIB_BASIC = > > -Wl,-rpath,/home/wurunjian/petsc/arch-linux2-c-opt/lib > > -L/home/wurunjian/petsc/arch-linux2-c-opt/lib > > -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/11 > > -L/usr/lib/gcc/x86_64-linux-gnu/11 -lflapack -lfblas -lhdf5hl_fortran > > -lhdf5_fortran -lhdf5_hl -lhdf5 -lparmetis -lmetis -lm -lstdc++ -ldl > > -lmpichfort -lmpich -lgfortran -lm -lgfortran -lm -lgcc_s -lquadmath > > -lstdc++ -ldl > > > > On Mon, May 13, 2024 at 9:34?PM Satish Balay wrote: > > > > > what version of PETSc? What configure command? What do you have for > > > PETSC_EXTERNAL_LIB_BASIC? > > > > > > You can send configure.log for your build to petsc-maint > > > > > > Generally duplicates should not cause grief. [as one needs them to > > > overcome circular dependencies]. > > > > > > What issues are you seeing? [send relevant logs] > > > > > > If the duplicates are coming form "compiler library" detection part of > > > configure - you can skip this part by listing the correct "compiler > > > libraries" > > > > > > for ex: with gcc/gfortran: > > > > > > LIBS="-lgfortran -lstdc++" > > > > > > However -lgfortran could be required for gcc - but a duplicate for > > > gfortran. > > > > > > Satish > > > > > > On Mon, 13 May 2024, Runjian Wu wrote: > > > > > > > Hi all, > > > > > > > > After I compiled PETSc, I found some duplicated libs in the variable > > > > PETSC_EXTERNAL_LIB_BASIC, e.g., -lm, -lgfortran -lstdc++. I am curious > > > how > > > > it happened and how to remove the duplicates? > > > > > > > > Thanks, > > > > > > > > Runjian Wu > > > > > > > > > > > > > From wurunjian at gmail.com Tue May 14 02:29:18 2024 From: wurunjian at gmail.com (Runjian Wu) Date: Tue, 14 May 2024 15:29:18 +0800 Subject: [petsc-users] VecGetArrayF90 vs. VecGetArrayReadF90 In-Reply-To: References: <21A28F71-57D4-48D2-9029-69B864B23C50@petsc.dev> Message-ID: Thanks for your explanation! On Tue, May 14, 2024 at 10:33?AM Barry Smith wrote: > > It errors in C because the argument is labeled with const, but there > does not seem to be a way in Fortran to indicate an array is read only. > > > > On May 13, 2024, at 10:21?PM, Runjian Wu wrote: > > I only know the "intent(in)" attribute for the dummy arguments. > > In the counter function VecGetArrayRead(..), if I write values, will the > compiler report an error? > > Runjian > > On Mon, May 13, 2024 at 9:36?PM Barry Smith wrote: > >> >> I couldn't find a way in Fortran to declare an array as read-only. Is >> there such support? >> >> Barry >> >> >> On May 13, 2024, at 7:28?AM, Runjian Wu wrote: >> >> This Message Is From an External Sender >> This message came from outside your organization. >> Hi all, >> >> I have a question about VecGetArrayReadF90(...). If I use >> VecGetArrayReadF90(...), I can still write entries into the array like >> VecGetArrayF90(...). Is it possible to report an error during compile >> process? >> >> Thanks, >> >> Runjian Wu >> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wurunjian at gmail.com Tue May 14 02:30:58 2024 From: wurunjian at gmail.com (Runjian Wu) Date: Tue, 14 May 2024 15:30:58 +0800 Subject: [petsc-users] duplicated libs In-Reply-To: References: <16654C9F-0761-4A68-A7E6-7DE5FA35E559@petsc.dev> Message-ID: Sounds reasonable. Thank you very much! Runjian On Tue, May 14, 2024 at 10:34?AM Barry Smith wrote: > > It is not always safe to remove duplicate libraries listed in different > places in the list. Hence we cannot simply always remove them. > > Barry > > On May 13, 2024, at 10:00?PM, Runjian Wu wrote: > > Thanks for your reply! Since I can manually remove duplicates, how about > adding a function to automatically remove duplicates at the end of > "configure" in the next PETSc version? > > Runjian > > > On 5/13/2024 9:31 PM, Barry Smith wrote: > > > Because the order of the libraries can be important, it is difficult > for ./configure to remove unneeded duplicates automatically. > > You can manually remove duplicates by editing > $PETSC_ARCH/lib/petsc/conf/petscvariables after running ./configure > > Barry > > > > On May 13, 2024, at 7:47?AM, Runjian Wu > wrote: > > This Message Is From an External Sender > This message came from outside your organization. > Hi all, > > After I compiled PETSc, I found some duplicated libs in the variable > PETSC_EXTERNAL_LIB_BASIC, e.g., -lm, -lgfortran -lstdc++. I am curious how > it happened and how to remove the duplicates? > > Thanks, > > Runjian Wu > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wurunjian at gmail.com Tue May 14 03:30:05 2024 From: wurunjian at gmail.com (Runjian Wu) Date: Tue, 14 May 2024 16:30:05 +0800 Subject: [petsc-users] duplicated libs In-Reply-To: References: <38fc058b-360d-073f-aac9-6a6d47e7fac2@mcs.anl.gov> <5597d213-d39a-f428-7b7f-91413590ce29@mcs.anl.gov> Message-ID: Yes, it is indeed cleaner. Thanks for your explanation! Now I have a question. If I stick to the old PETSc version (v3.16.2), I can manually remove the duplicated libraries in PETSC_LIB and keep the order of libraries at the same time, there should be no compiling errors, correct? # From PFLOTRAN makefile pflotran : $(pflotran_obj) ${FLINKER} -o pflotran $(pflotran_obj) ${PETSC_LIB} ${LIBS} PETSC_LIB = -Wl,-rpath,/home/wurunjian/petsc/arch-linux2-c-opt/lib -L/home/wurunjian/petsc/arch-linux2-c-opt/lib -Wl,-rpath,/home/wurunjian/petsc/arch-linux2-c-opt/lib -L/home/wurunjian/petsc/arch-linux2-c-opt/lib -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/11 -L/usr/lib/gcc/x86_64-linux-gnu/11 -lpetsc -lflapack -lfblas -lhdf5hl_fortran -lhdf5_fortran -lhdf5_hl -lhdf5 -lparmetis -lmetis -lm -lstdc++ -ldl -lmpichfort -lmpich -lgfortran -lm -lgfortran -lm -lgcc_s -lquadmath -lstdc++ -ldl On Tue, May 14, 2024 at 1:23?PM Satish Balay wrote: > And to have cleaner list - specify the (appropriate) compiler libraries > via LIBS option > > balay at pj01:~/petsc$ mpif90 -show > gfortran -I/software/mpich-4.1.1/include -I/software/mpich-4.1.1/include > -L/software/mpich-4.1.1/lib -lmpifort -Wl,-rpath > -Wl,/software/mpich-4.1.1/lib -Wl,--enable-new-dtags -lmpi > > balay at pj01:~/petsc$ ./configure --download-fblaslapack --download-hdf5 > --with-hdf5-fortran-bindings --download-metis --download-parmetis > COPTFLAGS=-O3 CXXOPTFLAGS=-O3 FOPTFLAGS=-O3 --with-debugging=0 > LIBS="-lmpifort -lmpi -lgfortran -lstdc++" > > balay at pj01:~/petsc$ grep ^PETSC_EXTERNAL_LIB_BASIC > arch-linux-c-opt/lib/petsc/conf/petscvariables > PETSC_EXTERNAL_LIB_BASIC = > -Wl,-rpath,/home/balay/petsc/arch-linux-c-opt/lib > -L/home/balay/petsc/arch-linux-c-opt/lib -lflapack -lfblas -lparmetis > -lmetis -lhdf5hl_fortran -lhdf5_fortran -lhdf5_hl -lhdf5 -lm -lX11 > -lquadmath -lmpifort -lmpi -lgfortran -lstdc++ > > > Satish > > > On Tue, 14 May 2024, Satish Balay via petsc-users wrote: > > > You can try using latest petsc version [3.21] - the list should be a bit > more cleaner with it. > > > > balay at pj01:~/petsc$ ./configure --download-mpich --download-fblaslapack > --download-hdf5 --with-hdf5-fortran-bindings --download-metis > --download-parmetis COPTFLAGS=-O3 CXXOPTFLAGS=-O3 FOPTFLAGS=-O3 > --with-debugging=0 > > > > balay at pj01:~/petsc$ grep ^PETSC_EXTERNAL_LIB_BASIC > arch-linux-c-opt/lib/petsc/conf/petscvariables > > PETSC_EXTERNAL_LIB_BASIC = > -Wl,-rpath,/home/balay/petsc/arch-linux-c-opt/lib > -L/home/balay/petsc/arch-linux-c-opt/lib > -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/13 > -L/usr/lib/gcc/x86_64-redhat-linux/13 -lflapack -lfblas -lparmetis -lmetis > -lhdf5hl_fortran -lhdf5_fortran -lhdf5_hl -lhdf5 -lm -lX11 -lmpifort -lmpi > -lgfortran -lm -lgfortran -lm -lgcc_s -lquadmath -lstdc++ > > > > Satish > > > > On Tue, 14 May 2024, Runjian Wu wrote: > > > > > First clarification, NO issues happened. I am just curious about the > > > duplicated libraries while I compile PFLOTRAN with PETSc. > > > > > > PETSc version: 3.16.2 > > > > > > Configure Options: --configModules=PETSc.Configure > > > --optionsModule=config.compilerOptions --with-make-np=4 --with-cc=mpicc > > > --with-cxx=mpicxx --with-fc=mpif90 --download-fblaslapack=yes > > > --download-hdf5=/home/wurunjian/petsc_pkg/hdf5-1.12.1.tar.bz2 > > > --download-hdf5-fortran-bindings=yes --download-metis=yes > > > --download-parmetis=yes --CFLAGS=-O3 --CXXFLAGS=-O3 --FFLAGS=-O3 > > > --with-debugging=0 > > > > > > PETSC_EXTERNAL_LIB_BASIC = > > > -Wl,-rpath,/home/wurunjian/petsc/arch-linux2-c-opt/lib > > > -L/home/wurunjian/petsc/arch-linux2-c-opt/lib > > > -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/11 > > > -L/usr/lib/gcc/x86_64-linux-gnu/11 -lflapack -lfblas -lhdf5hl_fortran > > > -lhdf5_fortran -lhdf5_hl -lhdf5 -lparmetis -lmetis -lm -lstdc++ -ldl > > > -lmpichfort -lmpich -lgfortran -lm -lgfortran -lm -lgcc_s -lquadmath > > > -lstdc++ -ldl > > > > > > On Mon, May 13, 2024 at 9:34?PM Satish Balay > wrote: > > > > > > > what version of PETSc? What configure command? What do you have for > > > > PETSC_EXTERNAL_LIB_BASIC? > > > > > > > > You can send configure.log for your build to petsc-maint > > > > > > > > Generally duplicates should not cause grief. [as one needs them to > > > > overcome circular dependencies]. > > > > > > > > What issues are you seeing? [send relevant logs] > > > > > > > > If the duplicates are coming form "compiler library" detection part > of > > > > configure - you can skip this part by listing the correct "compiler > > > > libraries" > > > > > > > > for ex: with gcc/gfortran: > > > > > > > > LIBS="-lgfortran -lstdc++" > > > > > > > > However -lgfortran could be required for gcc - but a duplicate for > > > > gfortran. > > > > > > > > Satish > > > > > > > > On Mon, 13 May 2024, Runjian Wu wrote: > > > > > > > > > Hi all, > > > > > > > > > > After I compiled PETSc, I found some duplicated libs in the > variable > > > > > PETSC_EXTERNAL_LIB_BASIC, e.g., -lm, -lgfortran -lstdc++. I am > curious > > > > how > > > > > it happened and how to remove the duplicates? > > > > > > > > > > Thanks, > > > > > > > > > > Runjian Wu > > > > > > > > > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue May 14 08:02:40 2024 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 14 May 2024 09:02:40 -0400 Subject: [petsc-users] DMPlex periodic face coordinates In-Reply-To: References: Message-ID: On Tue, May 14, 2024 at 12:14?AM Matteo Semplice < matteo.semplice at uninsubria.it> wrote: > Dear petsc-users, I am playing with DMPlexGetCellCoordinates and observing > that it returns correct periodic coordinates for cells, but not for faces. > More precisely, adding PetscCall(DMPlexGetHeightStratum(dm, 1, &fStart, > &fEnd)); for > ZjQcmQRYFpfptBannerStart > This Message Is From an External Sender > This message came from outside your organization. > > ZjQcmQRYFpfptBannerEnd > > Dear petsc-users, > > I am playing with DMPlexGetCellCoordinates and observing that it > returns correct periodic coordinates for cells, but not for faces. > > More precisely, adding > > PetscCall(DMPlexGetHeightStratum(dm, 1, &fStart, &fEnd)); > for (f = fStart; f < fEnd; ++f) { > const PetscScalar *array; > PetscScalar *x = NULL; > PetscInt ndof; > PetscBool isDG; > > PetscCall(DMPlexGetCellCoordinates(dm, f, &isDG, &ndof, &array, &x)); > PetscCheck(ndof % cdim == 0, PETSC_COMM_SELF, PETSC_ERR_ARG_INCOMP, > "ndof not divisible by cdim"); > PetscCall(PetscPrintf(PETSC_COMM_SELF, "Face #%" PetscInt_FMT " > coordinates\n", f - fStart)); > for (PetscInt i = 0; i < ndof; i += cdim) > PetscCall(PetscScalarView(cdim, &x[i], PETSC_VIEWER_STDOUT_SELF)); > PetscCall(DMPlexRestoreCellCoordinates(dm, f, &isDG, &ndof, &array, > &x)); > } > > to src/dm/impls/plex/tutorials/ex8.c, I get > > $ ./ex8 -dm_plex_dim 2 -petscspace_degree 1 -dm_plex_simplex 0 > -dm_plex_box_faces 3,2 -dm_plex_box_bd periodic,none -dm_view -view_coord > DM Object: box 1 MPI process > type: plex > box in 2 dimensions: > Number of 0-cells per rank: 9 > Number of 1-cells per rank: 15 > Number of 2-cells per rank: 6 > Periodic mesh (PERIODIC, NONE) coordinates localized > > [...] > > Element #2 coordinates > 0: 6.6667e-01 0.0000e+00 > 0: 1.0000e+00 0.0000e+00 <<<----- correct > 0: 1.0000e+00 5.0000e-01 > 0: 6.6667e-01 5.0000e-01 > [...] > > Face #0 coordinates > 0: 0.0000e+00 0.0000e+00 > 0: 3.3333e-01 0.0000e+00 > Face #1 coordinates > 0: 3.3333e-01 0.0000e+00 > 0: 6.6667e-01 0.0000e+00 > Face #2 coordinates > 0: 6.6667e-01 0.0000e+00 > 0: 0.0000e+00 0.0000e+00 <<<---- should be (0.66,0.00) and > (1.00,0.00) > > Is there a way to recover correct periodic coordinates in the case of > periodic DMPLex? > > The way that periodic coordinates work is that it stores a DG coordinate field by cell. Faces default back to the vertices. You could think about also putting DG coordinates on faces, but no one had asked, and it is potentially expensive. If you really need them to keep going, face coordinates can be extracted from cell coordinates. Otherwise, I can do it after the PETSc Meeting. Or, we are happy to take contributions adding this. Thanks, Matt > Thanks in advance > > Matteo > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!Z9-i-9O8yBcNimBXW2alc9XHsn9MWWFwaobajlfkDcoghfrtZlVX4CRMW0BdrXaX2hOabPupZN8g9FrtoNf2$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue May 14 08:07:29 2024 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 14 May 2024 09:07:29 -0400 Subject: [petsc-users] GMSH entities In-Reply-To: <67984175-20da-4323-98d3-c2124dfb43ea@auckland.ac.nz> References: <67984175-20da-4323-98d3-c2124dfb43ea@auckland.ac.nz> Message-ID: On Mon, May 13, 2024 at 10:04?PM Adrian Croucher wrote: > On 14/05/24 1:44 pm, Matthew Knepley wrote: > > I wish GMsh was clearer about what is optional: > https://urldefense.us/v3/__https://gmsh.info/doc/texinfo/gmsh.html*MSH-file-format__;Iw!!G_uCfscf7eWS!YJN5how37EbmfjwDvfPAsVSCQdWejJn8symxZ83hj94omk6Mh9imh2qOrFqZbsZRM_3W3G5YIn5lZK2KzQlj$ > They do talk about it, but not exhaustively. GMsh always writes and > $Entities block from what I can tell. > I can make it optional, it just might take until after the PETSc Meeting. > > Looks like $Entities are optional: > > > https://urldefense.us/v3/__https://gitlab.onelab.info/gmsh/gmsh/-/commit/b5feba2af57181ffa946d3f0c494b014603c6efa__;!!G_uCfscf7eWS!YJN5how37EbmfjwDvfPAsVSCQdWejJn8symxZ83hj94omk6Mh9imh2qOrFqZbsZRM_3W3G5YIn5lZMbTrLlQ$ > > I can also load a GMSH 4.1 file without $Entities into GMSH itself and it > doesn't complain, suggesting that they are indeed optional. > > Yes, but they are not careful to specify when a file can be inconsistent. For instance, omitting the $Entities, but then specifying entity numbers in the $Nodes block. I think they also thought this was inconsistent, but then got user complaints. The minimal example they show does exactly this. > If the $Entities aren't strictly needed for anything in DMPlex (which I'm > guessing they aren't, as the GMSH file format 2.2 doesn't even have them) > then it would be useful not to require them. > I put in some code for this: https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/merge_requests/7546__;!!G_uCfscf7eWS!YJN5how37EbmfjwDvfPAsVSCQdWejJn8symxZ83hj94omk6Mh9imh2qOrFqZbsZRM_3W3G5YIn5lZHVFZ3e3$ It just ignores entity numbers when there is no section. Thanks, Matt > - Adrian > > -- > Dr Adrian Croucher > Senior Research Fellow > Department of Engineering Science > Waipapa Taumata Rau / University of Auckland, New Zealand > email: a.croucher at auckland.ac.nz > tel: +64 (0)9 923 4611 > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!YJN5how37EbmfjwDvfPAsVSCQdWejJn8symxZ83hj94omk6Mh9imh2qOrFqZbsZRM_3W3G5YIn5lZFEALcjs$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Tue May 14 08:56:36 2024 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 14 May 2024 08:56:36 -0500 (CDT) Subject: [petsc-users] duplicated libs In-Reply-To: References: <38fc058b-360d-073f-aac9-6a6d47e7fac2@mcs.anl.gov> <5597d213-d39a-f428-7b7f-91413590ce29@mcs.anl.gov> Message-ID: <7a6807a2-6b63-906d-d029-897d19159e54@mcs.anl.gov> On Tue, 14 May 2024, Runjian Wu wrote: 61;7601;1c > Yes, it is indeed cleaner. Thanks for your explanation! > > Now I have a question. If I stick to the old PETSc version (v3.16.2), I can > manually remove the duplicated libraries in PETSC_LIB and keep the order of > libraries at the same time, there should be no compiling errors, correct? Well - that's for you to verify - that it works for your setup. Some compilers have quirks. And its not clear to me if this trade-off is worth it. [esp - since the duplicates don't cause grief - and manually editing stuff - for each install/machine can be error prone, and makes pflotran makefile un-portable.] I think using LIBS [with the correct list] might be slightly better than manually modifying pflotran makefile. But you can use whatever mode you are comfortable with [that works for you] Satish > > # From PFLOTRAN makefile > pflotran : $(pflotran_obj) > ${FLINKER} -o pflotran $(pflotran_obj) ${PETSC_LIB} ${LIBS} > > PETSC_LIB = -Wl,-rpath,/home/wurunjian/petsc/arch-linux2-c-opt/lib > -L/home/wurunjian/petsc/arch-linux2-c-opt/lib > -Wl,-rpath,/home/wurunjian/petsc/arch-linux2-c-opt/lib > -L/home/wurunjian/petsc/arch-linux2-c-opt/lib > -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/11 > -L/usr/lib/gcc/x86_64-linux-gnu/11 -lpetsc -lflapack -lfblas > -lhdf5hl_fortran -lhdf5_fortran -lhdf5_hl -lhdf5 -lparmetis -lmetis -lm > -lstdc++ -ldl -lmpichfort -lmpich -lgfortran -lm -lgfortran -lm -lgcc_s > -lquadmath -lstdc++ -ldl > From knepley at gmail.com Tue May 14 13:05:23 2024 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 14 May 2024 14:05:23 -0400 Subject: [petsc-users] GMSH entities In-Reply-To: References: <67984175-20da-4323-98d3-c2124dfb43ea@auckland.ac.nz> Message-ID: On Tue, May 14, 2024 at 9:07?AM Matthew Knepley wrote: > On Mon, May 13, 2024 at 10:04?PM Adrian Croucher < > a.croucher at auckland.ac.nz> wrote: > >> On 14/05/24 1:44 pm, Matthew Knepley wrote: >> >> I wish GMsh was clearer about what is optional: >> https://urldefense.us/v3/__https://gmsh.info/doc/texinfo/gmsh.html*MSH-file-format__;Iw!!G_uCfscf7eWS!fewn3DodeIVBmUE15mfIIEWSAHMhFjmoORD8nz_G4QnsaxhItWNFYbe4Lm0noV3Fuzj8ep504ww-M4Z0oroq$ >> They do talk about it, but not exhaustively. GMsh always writes and >> $Entities block from what I can tell. >> I can make it optional, it just might take until after the PETSc Meeting. >> >> Looks like $Entities are optional: >> >> >> https://urldefense.us/v3/__https://gitlab.onelab.info/gmsh/gmsh/-/commit/b5feba2af57181ffa946d3f0c494b014603c6efa__;!!G_uCfscf7eWS!fewn3DodeIVBmUE15mfIIEWSAHMhFjmoORD8nz_G4QnsaxhItWNFYbe4Lm0noV3Fuzj8ep504ww-M57ksfou$ >> >> I can also load a GMSH 4.1 file without $Entities into GMSH itself and it >> doesn't complain, suggesting that they are indeed optional. >> >> Yes, but they are not careful to specify when a file can be inconsistent. > For instance, omitting the $Entities, but then specifying entity numbers in > the $Nodes block. I think they also thought this was inconsistent, but then > got user complaints. The minimal example they show does exactly this. > >> If the $Entities aren't strictly needed for anything in DMPlex (which I'm >> guessing they aren't, as the GMSH file format 2.2 doesn't even have them) >> then it would be useful not to require them. >> > I put in some code for this: > https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/merge_requests/7546__;!!G_uCfscf7eWS!fewn3DodeIVBmUE15mfIIEWSAHMhFjmoORD8nz_G4QnsaxhItWNFYbe4Lm0noV3Fuzj8ep504ww-M-tMQE24$ > > It just ignores entity numbers when there is no section. > This merged, so now it should be fixed for you. Thanks, Matt > Thanks, > > Matt > >> - Adrian >> >> -- >> Dr Adrian Croucher >> Senior Research Fellow >> Department of Engineering Science >> Waipapa Taumata Rau / University of Auckland, New Zealand >> email: a.croucher at auckland.ac.nz >> tel: +64 (0)9 923 4611 >> >> > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fewn3DodeIVBmUE15mfIIEWSAHMhFjmoORD8nz_G4QnsaxhItWNFYbe4Lm0noV3Fuzj8ep504ww-M7SPCKFm$ > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fewn3DodeIVBmUE15mfIIEWSAHMhFjmoORD8nz_G4QnsaxhItWNFYbe4Lm0noV3Fuzj8ep504ww-M7SPCKFm$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From srvenkat at utexas.edu Tue May 14 18:07:46 2024 From: srvenkat at utexas.edu (Sreeram R Venkat) Date: Tue, 14 May 2024 18:07:46 -0500 Subject: [petsc-users] Using CUDA Graphs with MatComputeOperator Message-ID: I have a MatShell object that I want to convert to a MATDENSECUDA. Normally, I use MatComputeOperator for this. However, I would now also like to use a CUDA Graph so that all the calls to MatMult are captured. I can wrap a code like for (int i = 0; i < N; i++) MatMult(A, x,y); in a CUDA Graph, and it runs fine. If I try to wrap MatComputeOperator in a graph, I get runtime errors like cuda error 906 (cudaErrorStreamCaptureImplicit) : operation would make the legacy stream depend on a capturing blocking stream I tried modifying the MatConvert_Shell routine to only put the graph around the main for loop, but that still gives the same errors. Is there a way to use CUDA Graphs here (either through a modified MatConvert_Shell or otherwise)? Thanks, Sreeram -------------- next part -------------- An HTML attachment was scrubbed... URL: From a.croucher at auckland.ac.nz Tue May 14 22:09:06 2024 From: a.croucher at auckland.ac.nz (Adrian Croucher) Date: Wed, 15 May 2024 15:09:06 +1200 Subject: [petsc-users] GMSH entities In-Reply-To: References: <67984175-20da-4323-98d3-c2124dfb43ea@auckland.ac.nz> Message-ID: hi Matt, On 15/05/24 6:05 am, Matthew Knepley wrote: > On Tue, May 14, 2024 at 9:07?AM Matthew Knepley wrote: > > On Mon, May 13, 2024 at 10:04?PM Adrian Croucher > wrote: > > On 14/05/24 1:44 pm, Matthew Knepley wrote: > >> I wish GMsh was clearer about what is optional: >> https://urldefense.us/v3/__https://gmsh.info/doc/texinfo/gmsh.html*MSH-file-format__;Iw!!G_uCfscf7eWS!dbZ8R9PicaYRx4Y2IkPxvCoLlJ4lpsMbIAFBKbKZK5h5-2OTX5Ne-AwgOBHv2mJaHqg9uo5orDQYs2PaoAcpDmS65Au1g9uw$ >> >> >> They do talk about it, but not exhaustively. GMsh always >> writes and $Entities block from what I can tell. >> I can make it optional, it just might take until after the >> PETSc Meeting. > > Looks like $Entities are optional: > > https://urldefense.us/v3/__https://gitlab.onelab.info/gmsh/gmsh/-/commit/b5feba2af57181ffa946d3f0c494b014603c6efa__;!!G_uCfscf7eWS!dbZ8R9PicaYRx4Y2IkPxvCoLlJ4lpsMbIAFBKbKZK5h5-2OTX5Ne-AwgOBHv2mJaHqg9uo5orDQYs2PaoAcpDmS65ISaNx6E$ > > > I can also load a GMSH 4.1 file without $Entities into GMSH > itself and it doesn't complain, suggesting that they are > indeed optional. > > Yes, but they are not careful to specify when a file can be > inconsistent. For instance, omitting the $Entities, but then > specifying entity numbers in the $Nodes block. I think they also > thought this was inconsistent, but then got user complaints. The > minimal example they show does exactly this. > > If the $Entities aren't strictly needed for anything in DMPlex > (which I'm guessing they aren't, as the GMSH file format 2.2 > doesn't even have them) then it would be useful not to require > them. > > I put in some code for this: > https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/merge_requests/7546__;!!G_uCfscf7eWS!dbZ8R9PicaYRx4Y2IkPxvCoLlJ4lpsMbIAFBKbKZK5h5-2OTX5Ne-AwgOBHv2mJaHqg9uo5orDQYs2PaoAcpDmS65MnfH9Hr$ > > > It just ignores entity numbers when there is no section. > > > This merged, so now it should be fixed for you. Thanks, that seems to fix the problem. Great! Cheers, Adrian -- Dr Adrian Croucher Senior Research Fellow Department of Engineering Science Waipapa Taumata Rau / University of Auckland, New Zealand email:a.croucher at auckland.ac.nz tel: +64 (0)9 923 4611 -------------- next part -------------- An HTML attachment was scrubbed... URL: From junchao.zhang at gmail.com Tue May 14 22:15:25 2024 From: junchao.zhang at gmail.com (Junchao Zhang) Date: Tue, 14 May 2024 22:15:25 -0500 Subject: [petsc-users] Using CUDA Graphs with MatComputeOperator In-Reply-To: References: Message-ID: I haven't used CUDA graph with PETSc. Do you happen to have a working example so we can debug? --Junchao Zhang On Tue, May 14, 2024 at 6:08?PM Sreeram R Venkat wrote: > I have a MatShell object that I want to convert to a MATDENSECUDA. > Normally, I use MatComputeOperator for this. However, I would now also like > to use a CUDA Graph so that all the calls to MatMult are captured. I can > wrap a code like for (int > ZjQcmQRYFpfptBannerStart > This Message Is From an External Sender > This message came from outside your organization. > > ZjQcmQRYFpfptBannerEnd > I have a MatShell object that I want to convert to a MATDENSECUDA. > Normally, I use MatComputeOperator for this. However, I would now also like > to use a CUDA Graph so that all the calls to MatMult are captured. I can > wrap a code like > > for (int i = 0; i < N; i++) > MatMult(A, x,y); > > in a CUDA Graph, and it runs fine. If I try to wrap MatComputeOperator in > a graph, I get runtime errors like > cuda error 906 (cudaErrorStreamCaptureImplicit) : operation would make > the legacy stream depend on a capturing blocking stream > > I tried modifying the MatConvert_Shell routine to only put the graph > around the main for loop, but that still gives the same errors. Is there a > way to use CUDA Graphs here (either through a modified MatConvert_Shell or > otherwise)? > > Thanks, > Sreeram > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wurunjian at gmail.com Wed May 15 06:05:29 2024 From: wurunjian at gmail.com (Runjian Wu) Date: Wed, 15 May 2024 19:05:29 +0800 Subject: [petsc-users] duplicated libs In-Reply-To: <7a6807a2-6b63-906d-d029-897d19159e54@mcs.anl.gov> References: <38fc058b-360d-073f-aac9-6a6d47e7fac2@mcs.anl.gov> <5597d213-d39a-f428-7b7f-91413590ce29@mcs.anl.gov> <7a6807a2-6b63-906d-d029-897d19159e54@mcs.anl.gov> Message-ID: Thanks for your kind explanation! Runjian On Tue, May 14, 2024 at 9:56?PM Satish Balay wrote: > On Tue, 14 May 2024, Runjian Wu wrote: > 61;7601;1c > > Yes, it is indeed cleaner. Thanks for your explanation! > > > > Now I have a question. If I stick to the old PETSc version (v3.16.2), I > can > > manually remove the duplicated libraries in PETSC_LIB and keep the order > of > > libraries at the same time, there should be no compiling errors, correct? > > Well - that's for you to verify - that it works for your setup. Some > compilers have quirks. > > And its not clear to me if this trade-off is worth it. [esp - since > the duplicates don't cause grief - and manually editing stuff - for > each install/machine can be error prone, and makes pflotran makefile > un-portable.] > > I think using LIBS [with the correct list] might be slightly better > than manually modifying pflotran makefile. But you can use whatever > mode you are comfortable with [that works for you] > > Satish > > > > > # From PFLOTRAN makefile > > pflotran : $(pflotran_obj) > > ${FLINKER} -o pflotran $(pflotran_obj) ${PETSC_LIB} ${LIBS} > > > > PETSC_LIB = -Wl,-rpath,/home/wurunjian/petsc/arch-linux2-c-opt/lib > > -L/home/wurunjian/petsc/arch-linux2-c-opt/lib > > -Wl,-rpath,/home/wurunjian/petsc/arch-linux2-c-opt/lib > > -L/home/wurunjian/petsc/arch-linux2-c-opt/lib > > -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/11 > > -L/usr/lib/gcc/x86_64-linux-gnu/11 -lpetsc -lflapack -lfblas > > -lhdf5hl_fortran -lhdf5_fortran -lhdf5_hl -lhdf5 -lparmetis -lmetis -lm > > -lstdc++ -ldl -lmpichfort -lmpich -lgfortran -lm -lgfortran -lm -lgcc_s > > -lquadmath -lstdc++ -ldl > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From matteo.semplice at uninsubria.it Wed May 15 11:07:26 2024 From: matteo.semplice at uninsubria.it (Matteo Semplice) Date: Wed, 15 May 2024 18:07:26 +0200 Subject: [petsc-users] DMPlex periodic face coordinates In-Reply-To: References: Message-ID: <250a9259-9be0-4d9f-a5c8-ea4dfa22c009@uninsubria.it> Il 14/05/24 15:02, Matthew Knepley ha scritto: > On Tue, May 14, 2024 at 12:14?AM Matteo Semplice > wrote: > > Dear petsc-users, I am playing with DMPlexGetCellCoordinates and > observing that it returns correct periodic coordinates for cells, > but not for faces. More precisely, adding > PetscCall(DMPlexGetHeightStratum(dm, 1, &fStart, &fEnd)); for > ZjQcmQRYFpfptBannerStart > This Message Is From an External Sender > This message came from outside your organization. > ZjQcmQRYFpfptBannerEnd > > Dear petsc-users, > > ??? I am playing with DMPlexGetCellCoordinates and observing that > it returns correct periodic coordinates for cells, but not for faces. > > More precisely, adding > > PetscCall(DMPlexGetHeightStratum(dm, 1, &fStart, &fEnd)); > ??? for (f = fStart; f < fEnd; ++f) { > ????? const PetscScalar *array; > ????? PetscScalar?????? *x = NULL; > ????? PetscInt?????????? ndof; > ????? PetscBool????????? isDG; > > ????? PetscCall(DMPlexGetCellCoordinates(dm, f, &isDG, &ndof, > &array, &x)); > ????? PetscCheck(ndof % cdim == 0, PETSC_COMM_SELF, > PETSC_ERR_ARG_INCOMP, "ndof not divisible by cdim"); > ????? PetscCall(PetscPrintf(PETSC_COMM_SELF, "Face #%" > PetscInt_FMT " coordinates\n", f - fStart)); > ????? for (PetscInt i = 0; i < ndof; i += cdim) > PetscCall(PetscScalarView(cdim, &x[i], PETSC_VIEWER_STDOUT_SELF)); > ????? PetscCall(DMPlexRestoreCellCoordinates(dm, f, &isDG, &ndof, > &array, &x)); > ??? } > > to src/dm/impls/plex/tutorials/ex8.c, I get > > $ ./ex8 -dm_plex_dim 2 -petscspace_degree 1 -dm_plex_simplex 0 > -dm_plex_box_faces 3,2 -dm_plex_box_bd periodic,none -dm_view > -view_coord > DM Object: box 1 MPI process > ? type: plex > box in 2 dimensions: > ? Number of 0-cells per rank: 9 > ? Number of 1-cells per rank: 15 > ? Number of 2-cells per rank: 6 > Periodic mesh (PERIODIC, NONE) coordinates localized > > [...] > > Element #2 coordinates > ?0:?? 6.6667e-01?? 0.0000e+00 > ?0:?? 1.0000e+00?? 0.0000e+00 <<<----- correct > ?0:?? 1.0000e+00?? 5.0000e-01 > ?0:?? 6.6667e-01?? 5.0000e-01 > [...] > > Face #0 coordinates > ?0:?? 0.0000e+00?? 0.0000e+00 > ?0:?? 3.3333e-01?? 0.0000e+00 > Face #1 coordinates > ?0:?? 3.3333e-01?? 0.0000e+00 > ?0:?? 6.6667e-01?? 0.0000e+00 > Face #2 coordinates > ?0:?? 6.6667e-01?? 0.0000e+00 > ?0:?? 0.0000e+00?? 0.0000e+00? <<<---- should be (0.66,0.00) and > (1.00,0.00) > > Is there a way to recover correct periodic coordinates in the case > of periodic DMPLex? > > The way that periodic coordinates work is that it stores a DG > coordinate field by cell. Faces default back to the vertices. You > could think about? also putting DG coordinates on faces, but no one > had asked, and it is potentially expensive. > > If you really need them to keep going, face coordinates can be > extracted from cell coordinates. Otherwise, I can do it after the > PETSc Meeting. Or, we are happy to take contributions adding this. Hi. I mainly wanted to check that this was expected and not a result of a setup of my DM. No need to change Petsc now except maybe issuing a warning if the fallback to continuous coordinates will give wrong results. As for my code, I think I can extract the info from the cell coordinates in a pre-processing phase and if this crazy FV methods I am implementing will survive for long enough, I'll think about contributing. Just to double-check, can I assume that the order in which DMPlexGetCellCoordinates returns the coordinates matches the ordering of vertices in DMPlexGetCone for the uninterpolated DM, right? Best ??? Matteo -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Wed May 15 11:28:39 2024 From: jed at jedbrown.org (Jed Brown) Date: Wed, 15 May 2024 10:28:39 -0600 Subject: [petsc-users] DMPlex periodic face coordinates In-Reply-To: <250a9259-9be0-4d9f-a5c8-ea4dfa22c009@uninsubria.it> References: <250a9259-9be0-4d9f-a5c8-ea4dfa22c009@uninsubria.it> Message-ID: <87le4brqdk.fsf@jedbrown.org> An HTML attachment was scrubbed... URL: From srvenkat at utexas.edu Wed May 15 19:03:41 2024 From: srvenkat at utexas.edu (Sreeram R Venkat) Date: Wed, 15 May 2024 19:03:41 -0500 Subject: [petsc-users] Using CUDA Graphs with MatComputeOperator In-Reply-To: References: Message-ID: I don't have a working example that I can easily send you since the MatShell I have involves a lot of other code. Essentially, I am trying to do something like this: bool graphCreated = false; cudaGraph_t graph; cudaGraphExec_t instance; if (!graphCreated) { gpuErrchk(cudaStreamBeginCapture(s, cudaStreamCaptureModeGlobal)); PetscCall(MatComputeOperator(ShellMat, MATDENSECUDA, &DenseMat)); gpuErrchk(cudaStreamEndCapture(s, &graph)); gpuErrchk(cudaGraphInstantiate(&instance, graph, NULL, NULL, 0)); graphCreated = true; } gpuErrchk(cudaGraphLaunch(instance, s)); gpuErrchk(cudaStreamSynchronize(s)); gpuErrchk(cudaGraphExecDestroy(instance)); gpuErrchk(cudaGraphDestroy(graph)); The ShellMat context has the CUDA Stream "s", and that stream is used in the shell computations. If I replace the MatComputeOperator() call with for (int i = 0; i < N; i++) MatMult(ShellMat, x, y); the code works inside the CUDA graph. I tried modifying the MatConvert_Shell() code and wrapping the main for loop that computes each column with the graph, but this still doesn't work. I think if you have a MatShell that does some trivial computation (like return y = x), that should be a good enough example to start with. For now, I ended up making my own version of MatComputeOperator that doesn't involve Mats or Vecs in the main loop (working directly with arrays) so I could have control over the stream operations. Thanks, Sreeram On Tue, May 14, 2024 at 10:15?PM Junchao Zhang wrote: > > I haven't used CUDA graph with PETSc. Do you happen to have a working > example so we can debug? > > --Junchao Zhang > > > On Tue, May 14, 2024 at 6:08?PM Sreeram R Venkat > wrote: > >> I have a MatShell object that I want to convert to a MATDENSECUDA. >> Normally, I use MatComputeOperator for this. However, I would now also like >> to use a CUDA Graph so that all the calls to MatMult are captured. I can >> wrap a code like for (int >> ZjQcmQRYFpfptBannerStart >> This Message Is From an External Sender >> This message came from outside your organization. >> >> ZjQcmQRYFpfptBannerEnd >> I have a MatShell object that I want to convert to a MATDENSECUDA. >> Normally, I use MatComputeOperator for this. However, I would now also like >> to use a CUDA Graph so that all the calls to MatMult are captured. I can >> wrap a code like >> >> for (int i = 0; i < N; i++) >> MatMult(A, x,y); >> >> in a CUDA Graph, and it runs fine. If I try to wrap MatComputeOperator in >> a graph, I get runtime errors like >> cuda error 906 (cudaErrorStreamCaptureImplicit) : operation would make >> the legacy stream depend on a capturing blocking stream >> >> I tried modifying the MatConvert_Shell routine to only put the graph >> around the main for loop, but that still gives the same errors. Is there a >> way to use CUDA Graphs here (either through a modified MatConvert_Shell or >> otherwise)? >> >> Thanks, >> Sreeram >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sawsan.shatanawi at wsu.edu Sat May 18 18:29:14 2024 From: sawsan.shatanawi at wsu.edu (Shatanawi, Sawsan Muhammad) Date: Sat, 18 May 2024 23:29:14 +0000 Subject: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code In-Reply-To: References: <17B6144E-4F3A-4173-9818-4B03669736C4@petsc.dev> <9BD74F88-20FD-4021-B02A-195A58B72282@petsc.dev> <0636729D-AE8D-4DE0-8606-E0948733814C@petsc.dev> Message-ID: Hello everyone, Thank you all for your feedback, it helped me a lot. I read the PETSc document and examples related to the Jacobian and modified my code. Now I am getting errors related to the memory access. I tried to debug but couldn't find out how to fix it. sshatanawi/SawSim: Sawsan's Simulation (SawSim) is a groundwater dynamics scheme to be integrated into Noah MP land surface model (github.com) is the link to the code in the GitHub, I would appreciate it if you could have a look at it and guide me how to fix it. I believe the problem is in the memory of LHS and res_vector because they are new vectors I created. Thank you in advance for your help, I really appreciate it Bests, Sawsan ________________________________ From: Matthew Knepley Sent: Saturday, May 11, 2024 1:56 AM To: Shatanawi, Sawsan Muhammad Cc: Barry Smith ; petsc-users at mcs.anl.gov Subject: Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code [EXTERNAL EMAIL] On Fri, May 10, 2024 at 6:30?PM Shatanawi, Sawsan Muhammad via petsc-users > wrote: Good afternoon, I have tried SNESComputeJacobianDefaultColor(), but the arguments needed are confusing me. Would you please have a look at my code and the error messages I am getting? I didn't understand what the nonzero values of the sparse ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Good afternoon, I have tried SNESComputeJacobianDefaultColor(), but the arguments needed are confusing me. Would you please have a look at my code and the error messages I am getting? I didn't understand what the nonzero values of the sparse Jacobian would be. 1) You are not checking any of the return values from PETSc calls. Look at the PETSc Fortran examples. You should wrap all PETSc calls in PetscCall() or PetscCallA(). 2) You are not intended to directly call SNESComputeJacobianDefaultColor(). PETSc will do this automatically if you do not set of Jacobian function. 3) As Barry points out, coloring will not work unless we understand the nonzero structure of your Jacobian. This can happen by either: a) Using a DM: This is the easiest. Find the type that matches your grid, or b) Preallocating your Jacobian: Here you give the nonzero structure of your Jacobian, but not the values. Currently you do not do this. Instead you just give the size, not the nonzero structure (columns for each row). Thanks, Matt Thank you for your patience and help Bests, Sawsan ________________________________ From: Barry Smith > Sent: Thursday, May 9, 2024 12:05 PM To: Shatanawi, Sawsan Muhammad > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code [EXTERNAL EMAIL] On May 9, 2024, at 2:52?PM, Shatanawi, Sawsan Muhammad via petsc-users > wrote: This Message Is From an External Sender This message came from outside your organization. Hello everyone, Thank you for your responses and feedback, I checked PFLOTRAN and found that it is a model to simulate groundwater flow, contaminant transport, and other subsurface processes. my goal is not to simulate the groundwater flow, my goal is to develop a code from scratch to simulate the groundwater flow with specific conditions, and then integrate this code with land surface models. Later, the simulation of this code will be on a large scale. I want PETSc to calculate the Jacobian because the system is large and has complex nonlinear behavior, and I don?t risk calculating the derivative by myself. My A-Matrix has parts of source terms that depend on the flow fields, and independent parts will be in the RHS vector. With coloring SNESComputeJacobianDefaultColor() PETSc can compute Jacobian's pretty efficiently. You do not to provide the residual function and you need to provide the nonzero pattern of the sparse Jacobian; that is what residual components f_i are coupled to what input variables in the array x_i. This information comes from your PDE and discretization and appears implicitly in your residual function. Barry I hope I have answered your questions, and I apologize that I wasn?t clear from the beginning, I was trying to keep my descriptions brief. Bests, Sawsan ________________________________ From: Matthew Knepley > Sent: Tuesday, May 7, 2024 5:17 PM To: Shatanawi, Sawsan Muhammad > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code [EXTERNAL EMAIL] On Tue, May 7, 2024 at 2:23?PM Shatanawi, Sawsan Muhammad via petsc-users > wrote: This Message Is From an External Sender This message came from outside your organization. Hello everyone, I hope this email finds you well. My Name is Sawsan Shatanawi, and I was developing a Fortran code for simulating groundwater flow in a 3D system with nonlinear behavior. I solved the nonlinear system using the PCG solver and Picard iteration, but I did not get good results although I checked my matrix and RHS and everything, I decided to change my solver to Newton Rapson method. I checked PETSc documents but I have a few questions: 1) My groundwater system is time-dependent, so should I use TS only instead of SNES? You could use TS, but it is not necessary. You could use SNES and your own timestepping. THe advantage of TS is that you can try many different timesteppers without recoding (just like you can try many different linear and nonlinear solvers). 2) My system has its deltaT, would using deltaT as dt affect my solver, or is it better to use TS-PETSc dt? Also, would using PETSc dt affect the simulation of the groundwater system It sounds like your dt comes from your timestepper. If you use TS, you would use the dt from that. 3) I want my Jacobian matrix to be calculated by PETSc automatically PETSc can calculate a full Jacobian for smaller problems, or a finite-difference Jacobian for any problem (but this impacts the solver). It should be straightfoward to code up the analytic Jacobian. Is there a reason it would be a problem? 4) Do I need to define and calculate the residual vector? Yes. My A-Matrix contains coefficients and external sources and my RHS vector includes the boundary conditions It is strange that your matrix would contain source terms. Do they depend on the flow fields? Barry is right, you should consider PFlotran, and at least know why it would not work for your problem if you don't use it. Thanks, Matt Please find the attached file contains a draft of my code Thank you in advance for your time and help. Best regards, Sawsan ________________________________ From: Shatanawi, Sawsan Muhammad > Sent: Tuesday, January 16, 2024 10:43 AM To: Junchao Zhang > Cc: Barry Smith >; Matthew Knepley >; Mark Adams >; petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code Hello all, Thank you for your valuable help. I will do your recommendations and hope it will run without any issues. Bests, Sawsan ________________________________ From: Junchao Zhang > Sent: Friday, January 12, 2024 8:46 AM To: Shatanawi, Sawsan Muhammad > Cc: Barry Smith >; Matthew Knepley >; Mark Adams >; petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code [EXTERNAL EMAIL] Hi, Sawsan, First in test_main.F90, you need to call VecGetArrayF90(temp_solution, H_vector, ierr) and VecRestoreArrayF90 (temp_solution, H_vector, ierr) as Barry mentioned. Secondly, in the loop of test_main.F90, it calls GW_solver(). Within it, it calls PetscInitialize()/PetscFinalize(). But without MPI being initialized, PetscInitialize()/PetscFinalize() can only be called once. do timestep =2 , NTSP call GW_boundary_conditions(timestep-1) !print *,HNEW(1,1,1) call GW_elevation() ! print *, GWTOP(2,2,2) call GW_conductance() ! print *, CC(2,2,2) call GW_recharge() ! print *, B_Rech(5,4) call GW_pumping(timestep-1) ! print *, B_pump(2,2,2) call GW_SW(timestep-1) print *,B_RIVER (2,2,2) call GW_solver(timestep-1,N) call GW_deallocate_loop() end do A solution is to delete PetscInitialize()/PetscFinalize() in GW_solver_try.F90 and move it to test_main.F90, outside the do loop. diff --git a/test_main.F90 b/test_main.F90 index b5997c55..107bd3ee 100644 --- a/test_main.F90 +++ b/test_main.F90 @@ -1,5 +1,6 @@ program test_GW +#include use petsc use GW_constants use GW_param_by_user @@ -8,6 +9,9 @@ program test_GW implicit none integer :: N integer :: timestep + PetscErrorCode ierr + + call PetscInitialize(ierr) call GW_domain(N) !print *, "N=",N !print *, DELTAT @@ -37,4 +41,5 @@ program test_GW end do print *, HNEW(NCOL,3,2) call GW_deallocate () + call PetscFinalize(ierr) end program test_GW With that, the MPI error will be fixed. The code could run to gw_deallocate () before abort. There are other memory errors. You can install/use valgrind to fix them. Run it with valgrind ./GW.exe and look through the output Thanks. --Junchao Zhang On Thu, Jan 11, 2024 at 10:49?PM Shatanawi, Sawsan Muhammad > wrote: Hello, Thank you all for your help. I have changed VecGetArray to VecGetArrayF90, and the location of destory call. but I want to make sure that VecGet ArrayF90 is to make a new array( vector) that I can use in the rest of my Fortran code? when I run it and debugged it, I got 5.2000000E-03 50.00000 10.00000 0.0000000E+00 PETSC: Attaching gdb to /weka/data/lab/richey/sawsan/GW_CODE/code2024/SS_GWM/./GW.exe of pid 33065 on display :0.0 on machine sn16 Unable to start debugger in xterm: No such file or directory 0.0000000E+00 Attempting to use an MPI routine after finalizing MPICH srun: error: sn16: task 0: Exited with exit code 1 [sawsan.shatanawi at login-p2n02 SS_GWM]$ gdb ./GW/exe GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-100.el7 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later > This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". For bug reporting instructions, please see: >... ./GW/exe: No such file or directory. (gdb) run Starting program: No executable file specified. Use the "file" or "exec-file" command. (gdb) bt No stack. (gdb) If the highlighted line is the error, I don't know why when I write gdb , it does not show me the location of error The code : sshatanawi/SS_GWM (github.com) I really appreciate your helps Sawsan ________________________________ From: Barry Smith > Sent: Wednesday, January 10, 2024 5:35 PM To: Junchao Zhang > Cc: Shatanawi, Sawsan Muhammad >; Mark Adams >; petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code [EXTERNAL EMAIL] On Jan 10, 2024, at 6:49?PM, Junchao Zhang > wrote: Hi, Sawsan, I could build your code and I also could gdb it. $ gdb ./GW.exe ... $ Thread 1 "GW.exe" received signal SIGSEGV, Segmentation fault. 0x00007ffff1e6d44f in vecgetarray_ (x=0x7fffffffa718, fa=0x0, ia=0x7fffffffa75c, ierr=0x0) at /scratch/jczhang/petsc/src/vec/vec/interface/ftn-custom/zvectorf.c:257 257 *ierr = VecGetArray(*x, &lx); (gdb) bt #0 0x00007ffff1e6d44f in vecgetarray_ (x=0x7fffffffa718, fa=0x0, ia=0x7fffffffa75c, ierr=0x0) at /scratch/jczhang/petsc/src/vec/vec/interface/ftn-custom/zvectorf.c:257 #1 0x000000000040b6e3 in gw_solver (t_s=1.40129846e-45, n=300) at GW_solver_try.F90:169 #2 0x000000000040c6a8 in test_gw () at test_main.F90:35 ierr=0x0 caused the segfault. See https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/VecGetArray/*vecgetarray__;Iw!!G_uCfscf7eWS!dytCOR2bN1sL_Mje4qGaOf_Aq9RPC4e0HSx7PRurwI9TU9luzRIcP2sPv9AwO7jvdbPOxwEnKa21nCxcVF28kGo5HWBtdqieFA$ , you should use VecGetArrayF90 instead. BTW, Barry, the code https://urldefense.us/v3/__https://github.com/sshatanawi/SS_GWM/blob/main/GW_solver_try.F90*L169__;Iw!!G_uCfscf7eWS!dytCOR2bN1sL_Mje4qGaOf_Aq9RPC4e0HSx7PRurwI9TU9luzRIcP2sPv9AwO7jvdbPOxwEnKa21nCxcVF28kGo5HWBYM08cBw$ has "call VecGetArray(temp_solution, H_vector, ierr)". I don't find petsc Fortran examples doing VecGetArray. Do we still support it? This is not the correct calling sequence for VecGetArray() from Fortran. Regardless, definitely should not be writing any new code that uses VecGetArray() from Fortran. Should use VecGetArrayF90(). --Junchao Zhang On Wed, Jan 10, 2024 at 2:38?PM Shatanawi, Sawsan Muhammad via petsc-users > wrote: Hello all, I hope you are doing well. Generally, I use gdb to debug the code. I got the attached error message. I have tried to add the flag -start_in_debugger in the make file, but it didn't work, so it seems I was doing it in the wrong way This is the link for the whole code: sshatanawi/SS_GWM (github.com) [https://urldefense.us/v3/__https://opengraph.githubassets.com/9eb6cd14baf12f04848ed209b6f502415eb531bdd7b3a5f9696af68663b870c0/sshatanawi/SS_GWM__;!!G_uCfscf7eWS!dytCOR2bN1sL_Mje4qGaOf_Aq9RPC4e0HSx7PRurwI9TU9luzRIcP2sPv9AwO7jvdbPOxwEnKa21nCxcVF28kGo5HWCf6xSywA$ ] GitHub - sshatanawi/SS_GWM Contribute to sshatanawi/SS_GWM development by creating an account on GitHub. github.com ? You can read the description of the code in " Model Desprciption.pdf" the compiling file is makefile_f90 where you can find the linked code files I really appreciate your help Bests, Sawsan ________________________________ From: Mark Adams > Sent: Friday, January 5, 2024 4:53 AM To: Shatanawi, Sawsan Muhammad > Cc: Matthew Knepley >; petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code [EXTERNAL EMAIL] This is a segv. As Matt said, you need to use a debugger for this or add print statements to narrow down the place where this happens. You will need to learn how to use debuggers to do your project so you might as well start now. If you have a machine with a GUI debugger that is easier but command line debuggers are good to learn anyway. I tend to run debuggers directly (eg, lldb ./a.out -- program-args ...) and use a GUI debugger (eg, Totalview or DDT) if available. Mark On Wed, Dec 20, 2023 at 10:02?PM Shatanawi, Sawsan Muhammad via petsc-users > wrote: Hello Matthew, Thank you for your help. I am sorry that I keep coming back with my error messages, but I reached a point that I don't know how to fix them, and I don't understand them easily. The list of errors is getting shorter, now I am getting the attached error messages Thank you again, Sawsan ________________________________ From: Matthew Knepley > Sent: Wednesday, December 20, 2023 6:54 PM To: Shatanawi, Sawsan Muhammad > Cc: Barry Smith >; petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code [EXTERNAL EMAIL] On Wed, Dec 20, 2023 at 9:49?PM Shatanawi, Sawsan Muhammad via petsc-users > wrote: Hello Barry, Thank you a lot for your help, Now I am getting the attached error message. Do not destroy the PC from KSPGetPC() THanks, Matt Bests, Sawsan ________________________________ From: Barry Smith > Sent: Wednesday, December 20, 2023 6:32 PM To: Shatanawi, Sawsan Muhammad > Cc: Mark Adams >; petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code [EXTERNAL EMAIL] Instead of call PCCreate(PETSC_COMM_WORLD, pc, ierr) call PCSetType(pc, PCILU,ierr) ! Choose a preconditioner type (ILU) call KSPSetPC(ksp, pc,ierr) ! Associate the preconditioner with the KSP solver do call KSPGetPC(ksp,pc,ierr) call PCSetType(pc, PCILU,ierr) Do not call KSPSetUp(). It will be taken care of automatically during the solve On Dec 20, 2023, at 8:52?PM, Shatanawi, Sawsan Muhammad via petsc-users > wrote: Hello, I don't think that I set preallocation values when I created the matrix, would you please have look at my code. It is just the petsc related part from my code. I was able to fix some of the error messages. Now I have a new set of error messages related to the KSP solver (attached) I appreciate your help Sawsan ________________________________ From: Mark Adams > Sent: Wednesday, December 20, 2023 6:44 AM To: Shatanawi, Sawsan Muhammad > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code [EXTERNAL EMAIL] Did you set preallocation values when you created the matrix? Don't do that. On Wed, Dec 20, 2023 at 9:36?AM Shatanawi, Sawsan Muhammad > wrote: Hello, I am trying to create a sparse matrix( which is as I believe a zero matrix) then adding some nonzero elements to it over a loop, then assembling it Get Outlook for iOS ________________________________ From: Mark Adams > Sent: Wednesday, December 20, 2023 2:48 AM To: Shatanawi, Sawsan Muhammad > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code [EXTERNAL EMAIL] I am guessing that you are creating a matrix, adding to it, finalizing it ("assembly"), and then adding to it again, which is fine, but you are adding new non-zeros to the sparsity pattern. If this is what you want then you can tell the matrix to let you do that. Otherwise you have a bug. Mark On Tue, Dec 19, 2023 at 9:50?PM Shatanawi, Sawsan Muhammad via petsc-users > wrote: Hello everyone, I hope this email finds you well. My Name is Sawsan Shatanawi, and I am currently working on developing a Fortran code for simulating groundwater flow in a 3D system. The code involves solving a nonlinear system, and I have created the matrix to be solved using the PCG solver and Picard iteration. However, when I tried to assign it as a PETSc matrix I started getting a lot of error messages. I am kindly asking if someone can help me, I would be happy to share my code with him/her. Please find the attached file contains a list of errors I have gotten Thank you in advance for your time and assistance. Best regards, Sawsan -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!dytCOR2bN1sL_Mje4qGaOf_Aq9RPC4e0HSx7PRurwI9TU9luzRIcP2sPv9AwO7jvdbPOxwEnKa21nCxcVF28kGo5HWC_FudztA$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!dytCOR2bN1sL_Mje4qGaOf_Aq9RPC4e0HSx7PRurwI9TU9luzRIcP2sPv9AwO7jvdbPOxwEnKa21nCxcVF28kGo5HWC_FudztA$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!dytCOR2bN1sL_Mje4qGaOf_Aq9RPC4e0HSx7PRurwI9TU9luzRIcP2sPv9AwO7jvdbPOxwEnKa21nCxcVF28kGo5HWC_FudztA$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sat May 18 18:35:20 2024 From: knepley at gmail.com (Matthew Knepley) Date: Sat, 18 May 2024 19:35:20 -0400 Subject: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code In-Reply-To: References: <17B6144E-4F3A-4173-9818-4B03669736C4@petsc.dev> <9BD74F88-20FD-4021-B02A-195A58B72282@petsc.dev> <0636729D-AE8D-4DE0-8606-E0948733814C@petsc.dev> Message-ID: On Sat, May 18, 2024 at 7:29?PM Shatanawi, Sawsan Muhammad < sawsan.shatanawi at wsu.edu> wrote: > Hello everyone, > > Thank you all for your feedback, it helped me a lot. > > I read the PETSc document and examples related to the Jacobian and > modified my code. > Now I am getting errors related to the memory access. I tried to debug > but couldn't find out how to fix it. > I think the right way to find the bug is to start with either a) Valgrind (https://urldefense.us/v3/__http://www.valgrind.org__;!!G_uCfscf7eWS!an0a5TBb_2F0Yl9de3r2Rr5Lde7DTjTZJaHL4BLVhBks6EHjOrKhtOsm9NFyMydeBwa6UGgfAXF46n-sW_2y$ ) or b) Address Sanitizer, which is a feature of the clang compiler (-fsanitize=address) You can also run with -malloc_debug in PETSc, which can detect some errors. Thanks, Matt > sshatanawi/SawSim: Sawsan's Simulation (SawSim) is a groundwater dynamics > scheme to be integrated into Noah MP land surface model (github.com) > is the link to the code in the > GitHub, I would appreciate it if you could have a look at it and guide me > how to fix it. > I believe the problem is in the memory of LHS and res_vector because they > are new vectors I created. > > Thank you in advance for your help, I really appreciate it > > Bests, > Sawsan > ------------------------------ > *From:* Matthew Knepley > *Sent:* Saturday, May 11, 2024 1:56 AM > *To:* Shatanawi, Sawsan Muhammad > *Cc:* Barry Smith ; petsc-users at mcs.anl.gov < > petsc-users at mcs.anl.gov> > *Subject:* Re: [petsc-users] Help with Integrating PETSc into Fortran > Groundwater Flow Simulation Code > > > *[EXTERNAL EMAIL]* > On Fri, May 10, 2024 at 6:30?PM Shatanawi, Sawsan Muhammad via petsc-users > wrote: > > Good afternoon, I have tried SNESComputeJacobianDefaultColor(), but the > arguments needed are confusing me. Would you please have a look at my code > and the error messages I am getting? I didn't understand what the nonzero > values of the sparse > ZjQcmQRYFpfptBannerStart > This Message Is From an External Sender > This message came from outside your organization. > > ZjQcmQRYFpfptBannerEnd > Good afternoon, > > I have tried SNESComputeJacobianDefaultColor(), but the arguments needed > are confusing me. > > Would you please have a look at my code and the error messages I am > getting? > I didn't understand what the nonzero values of the sparse Jacobian would > be. > > > 1) You are not checking any of the return values from PETSc calls. Look at > the PETSc Fortran examples. > You should wrap all PETSc calls in PetscCall() or PetscCallA(). > > 2) You are not intended to directly > call SNESComputeJacobianDefaultColor(). PETSc will do this automatically if > you do not set of Jacobian function. > > 3) As Barry points out, coloring will not work unless we understand the > nonzero structure of your Jacobian. This can happen by either: > > a) Using a DM: This is the easiest. Find the type that matches your > grid, or > > b) Preallocating your Jacobian: Here you give the nonzero structure of > your Jacobian, but not the values. > Currently you do not do this. Instead you just give the size, not > the nonzero structure (columns for > each row). > > Thanks, > > Matt > > > Thank you for your patience and help > Bests, > Sawsan > > > ------------------------------ > *From:* Barry Smith > *Sent:* Thursday, May 9, 2024 12:05 PM > *To:* Shatanawi, Sawsan Muhammad > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [petsc-users] Help with Integrating PETSc into Fortran > Groundwater Flow Simulation Code > > > *[EXTERNAL EMAIL]* > > > On May 9, 2024, at 2:52?PM, Shatanawi, Sawsan Muhammad via petsc-users < > petsc-users at mcs.anl.gov> wrote: > > This Message Is From an External Sender > This message came from outside your organization. > Hello everyone, > > Thank you for your responses and feedback, > > I checked PFLOTRAN and found that it is a model to simulate groundwater > flow, contaminant transport, and other subsurface processes. > my goal is not to simulate the groundwater flow, my goal is to develop a > code from scratch to simulate the groundwater flow with specific > conditions, and then integrate this code with land surface models. > Later, the simulation of this code will be on a large scale. > > I want PETSc to calculate the Jacobian because the system is large and has > complex nonlinear behavior, and I don?t risk calculating the derivative by > myself. > My A-Matrix has parts of source terms that depend on the flow fields, and > independent parts will be in the RHS vector. > > > With coloring SNESComputeJacobianDefaultColor() PETSc can compute > Jacobian's pretty efficiently. You do not to provide the residual function > and you need to provide the nonzero pattern of the sparse Jacobian; that is > what residual components f_i are coupled to what input variables in the > array x_i. This information comes from your PDE and discretization and > appears implicitly in your residual function. > > Barry > > > I hope I have answered your questions, and I apologize that I wasn?t clear > from the beginning, I was trying to keep my descriptions brief. > > Bests, > Sawsan > ------------------------------ > *From:* Matthew Knepley > *Sent:* Tuesday, May 7, 2024 5:17 PM > *To:* Shatanawi, Sawsan Muhammad > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [petsc-users] Help with Integrating PETSc into Fortran > Groundwater Flow Simulation Code > > *[EXTERNAL EMAIL]* > On Tue, May 7, 2024 at 2:23?PM Shatanawi, Sawsan Muhammad via petsc-users < > petsc-users at mcs.anl.gov> wrote: > > This Message Is From an External Sender > This message came from outside your organization. > > Hello everyone, > > > I hope this email finds you well. > > > My Name is Sawsan Shatanawi, and I was developing a Fortran code for > simulating groundwater flow in a 3D system with nonlinear behavior. I > solved the nonlinear system using the PCG solver and Picard iteration, but > I did not get good results although I checked my matrix and RHS and > everything, I decided to change my solver to Newton Rapson method. > I checked PETSc documents but I have a few questions: > 1) My groundwater system is time-dependent, so should I use TS only > instead of SNES? > > > You could use TS, but it is not necessary. You could use SNES and your own > timestepping. THe advantage of TS is that you can try many different > timesteppers without recoding (just like you can > try many different linear and nonlinear solvers). > > > 2) My system has its deltaT, would using deltaT as dt affect my solver, or > is it better to use TS-PETSc dt? Also, would using PETSc dt affect the > simulation of the groundwater system > > > It sounds like your dt comes from your timestepper. If you use TS, you > would use the dt from that. > > > 3) I want my Jacobian matrix to be calculated by PETSc automatically > > > PETSc can calculate a full Jacobian for smaller problems, or a > finite-difference Jacobian for any problem (but this impacts the solver). > It should be straightfoward to code up the analytic Jacobian. Is there a > reason it would be a problem? > > > 4) Do I need to define and calculate the residual vector? > > > Yes. > > > My A-Matrix contains coefficients and external sources and my RHS vector > includes the boundary conditions > > > It is strange that your matrix would contain source terms. Do they depend > on the flow fields? > > Barry is right, you should consider PFlotran, and at least know why it > would not work for your problem if you don't use it. > > Thanks, > > Matt > > > > Please find the attached file contains a draft of my code > > Thank you in advance for your time and help. > > Best regards, > > Sawsan > > ------------------------------ > *From:* Shatanawi, Sawsan Muhammad > *Sent:* Tuesday, January 16, 2024 10:43 AM > *To:* Junchao Zhang > *Cc:* Barry Smith ; Matthew Knepley ; > Mark Adams ; petsc-users at mcs.anl.gov < > petsc-users at mcs.anl.gov> > *Subject:* Re: [petsc-users] Help with Integrating PETSc into Fortran > Groundwater Flow Simulation Code > > Hello all, > > Thank you for your valuable help. I will do your recommendations and hope > it will run without any issues. > > Bests, > Sawsan > > ------------------------------ > *From:* Junchao Zhang > *Sent:* Friday, January 12, 2024 8:46 AM > *To:* Shatanawi, Sawsan Muhammad > *Cc:* Barry Smith ; Matthew Knepley ; > Mark Adams ; petsc-users at mcs.anl.gov < > petsc-users at mcs.anl.gov> > *Subject:* Re: [petsc-users] Help with Integrating PETSc into Fortran > Groundwater Flow Simulation Code > > *[EXTERNAL EMAIL]* > Hi, Sawsan, > First in test_main.F90, you need to call VecGetArrayF90(temp_solution, > H_vector, ierr) and VecRestoreArrayF90 (temp_solution, H_vector, ierr) as > Barry mentioned. > Secondly, in the loop of test_main.F90, it calls GW_solver(). Within > it, it calls PetscInitialize()/PetscFinalize(). But without MPI being > initialized, PetscInitialize()/PetscFinalize()* can only be called once.* > do timestep =2 , NTSP > call GW_boundary_conditions(timestep-1) > !print *,HNEW(1,1,1) > call GW_elevation() > ! print *, GWTOP(2,2,2) > call GW_conductance() > ! print *, CC(2,2,2) > call GW_recharge() > ! print *, B_Rech(5,4) > call GW_pumping(timestep-1) > ! print *, B_pump(2,2,2) > call GW_SW(timestep-1) > print *,B_RIVER (2,2,2) > call GW_solver(timestep-1,N) > call GW_deallocate_loop() > end do > > A solution is to delete PetscInitialize()/PetscFinalize() in > GW_solver_try.F90 and move it to test_main.F90, outside the do loop. > > diff --git a/test_main.F90 b/test_main.F90 > index b5997c55..107bd3ee 100644 > --- a/test_main.F90 > +++ b/test_main.F90 > @@ -1,5 +1,6 @@ > program test_GW > > +#include > use petsc > use GW_constants > use GW_param_by_user > @@ -8,6 +9,9 @@ program test_GW > implicit none > integer :: N > integer :: timestep > + PetscErrorCode ierr > + > + call PetscInitialize(ierr) > call GW_domain(N) > !print *, "N=",N > !print *, DELTAT > @@ -37,4 +41,5 @@ program test_GW > end do > print *, HNEW(NCOL,3,2) > call GW_deallocate () > + call PetscFinalize(ierr) > end program test_GW > > With that, the MPI error will be fixed. The code could run to > gw_deallocate () before abort. There are other memory errors. You can > install/use valgrind to fix them. Run it with valgrind ./GW.exe and look > through the output > > > Thanks. > --Junchao Zhang > > > On Thu, Jan 11, 2024 at 10:49?PM Shatanawi, Sawsan Muhammad < > sawsan.shatanawi at wsu.edu> wrote: > > Hello, > > Thank you all for your help. > > I have changed VecGetArray to VecGetArrayF90, and the location of destory > call. but I want to make sure that VecGet ArrayF90 is to make a new array( > vector) that I can use in the rest of my Fortran code? > > when I run it and debugged it, I got > > 5.2000000E-03 > 50.00000 > 10.00000 > 0.0000000E+00 > PETSC: Attaching gdb to > /weka/data/lab/richey/sawsan/GW_CODE/code2024/SS_GWM/./GW.exe of pid 33065 > on display :0.0 on machine sn16 > Unable to start debugger in xterm: No such file or directory > 0.0000000E+00 > Attempting to use an MPI routine after finalizing MPICH > srun: error: sn16: task 0: Exited with exit code 1 > [sawsan.shatanawi at login-p2n02 SS_GWM]$ gdb ./GW/exe > GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-100.el7 > Copyright (C) 2013 Free Software Foundation, Inc. > License GPLv3+: GNU GPL version 3 or later < > https://urldefense.us/v3/__http://gnu.org/licenses/gpl.html__;!!G_uCfscf7eWS!an0a5TBb_2F0Yl9de3r2Rr5Lde7DTjTZJaHL4BLVhBks6EHjOrKhtOsm9NFyMydeBwa6UGgfAXF46kYNzfsG$ > > > > This is free software: you are free to change and redistribute it. > There is NO WARRANTY, to the extent permitted by law. Type "show copying" > and "show warranty" for details. > This GDB was configured as "x86_64-redhat-linux-gnu". > For bug reporting instructions, please see: > > >... > ./GW/exe: No such file or directory. > (gdb) run > Starting program: > No executable file specified. > Use the "file" or "exec-file" command. > (gdb) bt > No stack. > (gdb) > > If the highlighted line is the error, I don't know why when I write gdb , > it does not show me the location of error > The code : sshatanawi/SS_GWM (github.com) > > > > I really appreciate your helps > > Sawsan > ------------------------------ > *From:* Barry Smith > *Sent:* Wednesday, January 10, 2024 5:35 PM > *To:* Junchao Zhang > *Cc:* Shatanawi, Sawsan Muhammad ; Mark Adams < > mfadams at lbl.gov>; petsc-users at mcs.anl.gov > *Subject:* Re: [petsc-users] Help with Integrating PETSc into Fortran > Groundwater Flow Simulation Code > > *[EXTERNAL EMAIL]* > > > On Jan 10, 2024, at 6:49?PM, Junchao Zhang > wrote: > > Hi, Sawsan, > I could build your code and I also could gdb it. > > $ gdb ./GW.exe > ... > $ Thread 1 "GW.exe" received signal SIGSEGV, Segmentation fault. > 0x00007ffff1e6d44f in vecgetarray_ (x=0x7fffffffa718, fa=0x0, > ia=0x7fffffffa75c, ierr=0x0) at > /scratch/jczhang/petsc/src/vec/vec/interface/ftn-custom/zvectorf.c:257 > 257 *ierr = VecGetArray(*x, &lx); > (gdb) bt > #0 0x00007ffff1e6d44f in vecgetarray_ (x=0x7fffffffa718, fa=0x0, > ia=0x7fffffffa75c, ierr=0x0) at > /scratch/jczhang/petsc/src/vec/vec/interface/ftn-custom/zvectorf.c:257 > #1 0x000000000040b6e3 in gw_solver (t_s=1.40129846e-45, n=300) at > GW_solver_try.F90:169 > #2 0x000000000040c6a8 in test_gw () at test_main.F90:35 > > ierr=0x0 caused the segfault. See > https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/VecGetArray/*vecgetarray__;Iw!!G_uCfscf7eWS!an0a5TBb_2F0Yl9de3r2Rr5Lde7DTjTZJaHL4BLVhBks6EHjOrKhtOsm9NFyMydeBwa6UGgfAXF46oes_GFt$ > , > you should use VecGetArrayF90 instead. > > BTW, Barry, the code > https://urldefense.us/v3/__https://github.com/sshatanawi/SS_GWM/blob/main/GW_solver_try.F90*L169__;Iw!!G_uCfscf7eWS!an0a5TBb_2F0Yl9de3r2Rr5Lde7DTjTZJaHL4BLVhBks6EHjOrKhtOsm9NFyMydeBwa6UGgfAXF46i3_INb_$ > > has "call VecGetArray(temp_solution, H_vector, ierr)". I don't find > petsc Fortran examples doing VecGetArray. Do we still support it? > > > This is not the correct calling sequence for VecGetArray() from > Fortran. > > Regardless, definitely should not be writing any new code that uses > VecGetArray() from Fortran. Should use VecGetArrayF90(). > > > --Junchao Zhang > > > On Wed, Jan 10, 2024 at 2:38?PM Shatanawi, Sawsan Muhammad via petsc-users > wrote: > > Hello all, > > I hope you are doing well. > > Generally, I use gdb to debug the code. > I got the attached error message. > > I have tried to add the flag -start_in_debugger in the make file, but it > didn't work, so it seems I was doing it in the wrong way > > This is the link for the whole code: sshatanawi/SS_GWM (github.com) > > > > GitHub - sshatanawi/SS_GWM > > Contribute to sshatanawi/SS_GWM development by creating an account on > GitHub. > github.com > > *?* > > You can read the description of the code in " Model Desprciption.pdf" > the compiling file is makefile_f90 where you can find the linked code > files > > I really appreciate your help > > Bests, > Sawsan > ------------------------------ > *From:* Mark Adams > *Sent:* Friday, January 5, 2024 4:53 AM > *To:* Shatanawi, Sawsan Muhammad > *Cc:* Matthew Knepley ; petsc-users at mcs.anl.gov < > petsc-users at mcs.anl.gov> > *Subject:* Re: [petsc-users] Help with Integrating PETSc into Fortran > Groundwater Flow Simulation Code > > *[EXTERNAL EMAIL]* > This is a segv. As Matt said, you need to use a debugger for this or add > print statements to narrow down the place where this happens. > > You will need to learn how to use debuggers to do your project so you > might as well start now. > > If you have a machine with a GUI debugger that is easier but command line > debuggers are good to learn anyway. > > I tend to run debuggers directly (eg, lldb ./a.out -- program-args ...) > and use a GUI debugger (eg, Totalview or DDT) if available. > > Mark > > > On Wed, Dec 20, 2023 at 10:02?PM Shatanawi, Sawsan Muhammad via > petsc-users wrote: > > Hello Matthew, > > Thank you for your help. I am sorry that I keep coming back with my error > messages, but I reached a point that I don't know how to fix them, and I > don't understand them easily. > The list of errors is getting shorter, now I am getting the attached error > messages > > Thank you again, > > Sawsan > ------------------------------ > *From:* Matthew Knepley > *Sent:* Wednesday, December 20, 2023 6:54 PM > *To:* Shatanawi, Sawsan Muhammad > *Cc:* Barry Smith ; petsc-users at mcs.anl.gov < > petsc-users at mcs.anl.gov> > *Subject:* Re: [petsc-users] Help with Integrating PETSc into Fortran > Groundwater Flow Simulation Code > > *[EXTERNAL EMAIL]* > On Wed, Dec 20, 2023 at 9:49?PM Shatanawi, Sawsan Muhammad via petsc-users > wrote: > > Hello Barry, > > Thank you a lot for your help, Now I am getting the attached error message. > > > Do not destroy the PC from KSPGetPC() > > THanks, > > Matt > > > Bests, > Sawsan > ------------------------------ > *From:* Barry Smith > *Sent:* Wednesday, December 20, 2023 6:32 PM > *To:* Shatanawi, Sawsan Muhammad > *Cc:* Mark Adams ; petsc-users at mcs.anl.gov < > petsc-users at mcs.anl.gov> > *Subject:* Re: [petsc-users] Help with Integrating PETSc into Fortran > Groundwater Flow Simulation Code > > *[EXTERNAL EMAIL]* > > Instead of > > call PCCreate(PETSC_COMM_WORLD, pc, ierr) > call PCSetType(pc, PCILU,ierr) ! Choose a preconditioner type (ILU) > call KSPSetPC(ksp, pc,ierr) ! Associate the preconditioner with the > KSP solver > > do > > call KSPGetPC(ksp,pc,ierr) > call PCSetType(pc, PCILU,ierr) > > Do not call KSPSetUp(). It will be taken care of automatically during the > solve > > > > On Dec 20, 2023, at 8:52?PM, Shatanawi, Sawsan Muhammad via petsc-users < > petsc-users at mcs.anl.gov> wrote: > > Hello, > I don't think that I set preallocation values when I created the matrix, > would you please have look at my code. It is just the petsc related part > from my code. > I was able to fix some of the error messages. Now I have a new set of > error messages related to the KSP solver (attached) > > I appreciate your help > > Sawsan > ------------------------------ > *From:* Mark Adams > *Sent:* Wednesday, December 20, 2023 6:44 AM > *To:* Shatanawi, Sawsan Muhammad > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [petsc-users] Help with Integrating PETSc into Fortran > Groundwater Flow Simulation Code > > *[EXTERNAL EMAIL]* > Did you set preallocation values when you created the matrix? > Don't do that. > > On Wed, Dec 20, 2023 at 9:36?AM Shatanawi, Sawsan Muhammad < > sawsan.shatanawi at wsu.edu> wrote: > > Hello, > > I am trying to create a sparse matrix( which is as I believe a zero > matrix) then adding some nonzero elements to it over a loop, then > assembling it > > Get Outlook for iOS > > ------------------------------ > *From:* Mark Adams > *Sent:* Wednesday, December 20, 2023 2:48 AM > *To:* Shatanawi, Sawsan Muhammad > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [petsc-users] Help with Integrating PETSc into Fortran > Groundwater Flow Simulation Code > > *[EXTERNAL EMAIL]* > I am guessing that you are creating a matrix, adding to it, finalizing it > ("assembly"), and then adding to it again, which is fine, but you are > adding new non-zeros to the sparsity pattern. > If this is what you want then you can tell the matrix to let you do that. > Otherwise you have a bug. > > Mark > > On Tue, Dec 19, 2023 at 9:50?PM Shatanawi, Sawsan Muhammad via petsc-users > wrote: > > Hello everyone, > > I hope this email finds you well. > > My Name is Sawsan Shatanawi, and I am currently working on developing a > Fortran code for simulating groundwater flow in a 3D system. The code > involves solving a nonlinear system, and I have created the matrix to be > solved using the PCG solver and Picard iteration. However, when I tried > to assign it as a PETSc matrix I started getting a lot of error messages. > > I am kindly asking if someone can help me, I would be happy to share my > code with him/her. > > Please find the attached file contains a list of errors I have gotten > > Thank you in advance for your time and assistance. > > Best regards, > > Sawsan > > > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!an0a5TBb_2F0Yl9de3r2Rr5Lde7DTjTZJaHL4BLVhBks6EHjOrKhtOsm9NFyMydeBwa6UGgfAXF46kPxN7eB$ > > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!an0a5TBb_2F0Yl9de3r2Rr5Lde7DTjTZJaHL4BLVhBks6EHjOrKhtOsm9NFyMydeBwa6UGgfAXF46kPxN7eB$ > > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!an0a5TBb_2F0Yl9de3r2Rr5Lde7DTjTZJaHL4BLVhBks6EHjOrKhtOsm9NFyMydeBwa6UGgfAXF46kPxN7eB$ > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!an0a5TBb_2F0Yl9de3r2Rr5Lde7DTjTZJaHL4BLVhBks6EHjOrKhtOsm9NFyMydeBwa6UGgfAXF46kPxN7eB$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From a.croucher at auckland.ac.nz Sun May 19 18:44:03 2024 From: a.croucher at auckland.ac.nz (Adrian Croucher) Date: Mon, 20 May 2024 11:44:03 +1200 Subject: [petsc-users] Modify matrix nonzero structure Message-ID: <9c4c4a7d-3da1-46ed-935e-b3d8cc80d474@auckland.ac.nz> An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Sun May 19 19:24:46 2024 From: bsmith at petsc.dev (Barry Smith) Date: Sun, 19 May 2024 20:24:46 -0400 Subject: [petsc-users] Modify matrix nonzero structure In-Reply-To: <9c4c4a7d-3da1-46ed-935e-b3d8cc80d474@auckland.ac.nz> References: <9c4c4a7d-3da1-46ed-935e-b3d8cc80d474@auckland.ac.nz> Message-ID: <4368DA96-759E-4831-8BCF-710F265A4728@petsc.dev> You can call MatSetOption(mat,MAT_NEW_NONZERO_LOCATION_ERR) then insert the new values. If it is just a handful of new insertions the extra time should be small. Making a copy of the matrix won't give you a new matrix that is any faster to insert into so best to just use the same matrix. Barry > On May 19, 2024, at 7:44?PM, Adrian Croucher wrote: > > This Message Is From an External Sender > This message came from outside your organization. > hi, > > I have a Jacobian matrix created using DMCreateMatrix(). What would be > the best way to add extra nonzero entries into it? > > I'm guessing that DMCreateMatrix() allocates the storage so the nonzero > structure can't really be easily modified. Would it be a case of > creating a new matrix, copying the nonzero entries from the original one > and then adding the extra ones, before calling MatSetUp() or similar? If > so, how exactly would you copy the nonzero structure from the original > matrix? > > Background: the flow problem I'm solving (on a DMPlex with finite volume > method) has complex source terms that depend on the solution (e.g. > pressure), and can also depend on other source terms. A simple example > is when fluid is extracted from one location, with a pressure-dependent > flow rate, and some of it is then reinjected in another location. This > can result in poor nonlinear solver convergence. I think the reason is > that there are effectively missing Jacobian entries in the row for the > reinjection cell, which should have an additional dependence on the > solution in the cell where fluid is extracted. > > - Adrian > > -- > Dr Adrian Croucher > Senior Research Fellow > Department of Engineering Science > Waipapa Taumata Rau / University of Auckland, New Zealand > email: a.croucher at auckland.ac.nz > tel: +64 (0)9 923 4611 > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sun May 19 19:41:52 2024 From: knepley at gmail.com (Matthew Knepley) Date: Sun, 19 May 2024 20:41:52 -0400 Subject: [petsc-users] Modify matrix nonzero structure In-Reply-To: <4368DA96-759E-4831-8BCF-710F265A4728@petsc.dev> References: <9c4c4a7d-3da1-46ed-935e-b3d8cc80d474@auckland.ac.nz> <4368DA96-759E-4831-8BCF-710F265A4728@petsc.dev> Message-ID: On Sun, May 19, 2024 at 8:25?PM Barry Smith wrote: > You can call MatSetOption(mat,MAT_NEW_NONZERO_LOCATION_ERR) then insert > the new values. If it is just a handful of new insertions the extra time > should be small. Making a copy of the matrix won't give you a new matrix > that is any faster to > ZjQcmQRYFpfptBannerStart > This Message Is From an External Sender > This message came from outside your organization. > > ZjQcmQRYFpfptBannerEnd > > You can call MatSetOption(mat,MAT_NEW_NONZERO_LOCATION_ERR) then insert > the new values. If it is just a handful of new insertions the extra time > should be small. > > Making a copy of the matrix won't give you a new matrix that is any > faster to insert into so best to just use the same matrix. > Let me add to Barry's answer. The preallocation infrastructure is now not strictly necessary. It is possible to just add all your nonzeros in and assembly, and the performance will be pretty good (uses hashing etc). So if just adding a few nonzeros does not work, we can go this route. Thanks, Matt > Barry > > > On May 19, 2024, at 7:44?PM, Adrian Croucher > wrote: > > This Message Is From an External Sender > This message came from outside your organization. > > hi, > > I have a Jacobian matrix created using DMCreateMatrix(). What would be > the best way to add extra nonzero entries into it? > > I'm guessing that DMCreateMatrix() allocates the storage so the nonzero > structure can't really be easily modified. Would it be a case of > creating a new matrix, copying the nonzero entries from the original one > and then adding the extra ones, before calling MatSetUp() or similar? If > so, how exactly would you copy the nonzero structure from the original > matrix? > > Background: the flow problem I'm solving (on a DMPlex with finite volume > method) has complex source terms that depend on the solution (e.g. > pressure), and can also depend on other source terms. A simple example > is when fluid is extracted from one location, with a pressure-dependent > flow rate, and some of it is then reinjected in another location. This > can result in poor nonlinear solver convergence. I think the reason is > that there are effectively missing Jacobian entries in the row for the > reinjection cell, which should have an additional dependence on the > solution in the cell where fluid is extracted. > > - Adrian > > -- > Dr Adrian Croucher > Senior Research Fellow > Department of Engineering Science > Waipapa Taumata Rau / University of Auckland, New Zealand > email: a.croucher at auckland.ac.nz > tel: +64 (0)9 923 4611 > > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!ZAicuPSGpbQ0tcP9ZNf7TPN7b-DE4XuRcqUtAUEsr_yk9p2EAWn5wSAIdkxE7SYHsDeBftI_qWDRw8aUfXLK$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From a.croucher at auckland.ac.nz Sun May 19 21:44:47 2024 From: a.croucher at auckland.ac.nz (Adrian Croucher) Date: Mon, 20 May 2024 14:44:47 +1200 Subject: [petsc-users] Modify matrix nonzero structure In-Reply-To: References: <9c4c4a7d-3da1-46ed-935e-b3d8cc80d474@auckland.ac.nz> <4368DA96-759E-4831-8BCF-710F265A4728@petsc.dev> Message-ID: Great, it sounds like this might be easier than I expected. Thanks very much. Did you have any thoughts on my diagnosis of the problem (the poor nonlinear solver convergence being caused by missing Jacobian elements representing interaction between the sources)? - Adrian On 20/05/24 12:41 pm, Matthew Knepley wrote: > On Sun, May 19, 2024 at 8:25?PM Barry Smith wrote: > > You can call MatSetOption(mat,MAT_NEW_NONZERO_LOCATION_ERR) then > insert the new values. If it is just a handful of new insertions > the extra time should be small. Making a copy of the matrix won't > give you a new matrix that is any faster to > ZjQcmQRYFpfptBannerStart > This Message Is From an External Sender > This message came from outside your organization. > ZjQcmQRYFpfptBannerEnd > > ? ?You can call MatSetOption(mat,MAT_NEW_NONZERO_LOCATION_ERR) > then insert the new values. If it is just a handful of new > insertions the extra time should be small. > > ? ? Making a copy of the matrix won't give you a new matrix that > is any faster to insert into so best to just use the same matrix. > > > Let me add to Barry's answer. The preallocation infrastructure is now > not strictly necessary. It is possible to just add all your nonzeros > in and assembly,? and the performance will be pretty good (uses > hashing etc). So if just adding a few nonzeros does not work, we can > go this route. > > ? Thanks, > > ? ? ?Matt > > ? Barry > > >> On May 19, 2024, at 7:44?PM, Adrian Croucher >> wrote: >> >> This Message Is From an External Sender >> This message came from outside your organization. >> hi, >> >> I have a Jacobian matrix created using DMCreateMatrix(). What would be >> the best way to add extra nonzero entries into it? >> >> I'm guessing that DMCreateMatrix() allocates the storage so the nonzero >> structure can't really be easily modified. Would it be a case of >> creating a new matrix, copying the nonzero entries from the original one >> and then adding the extra ones, before calling MatSetUp() or similar? If >> so, how exactly would you copy the nonzero structure from the original >> matrix? >> >> Background: the flow problem I'm solving (on a DMPlex with finite volume >> method) has complex source terms that depend on the solution (e.g. >> pressure), and can also depend on other source terms. A simple example >> is when fluid is extracted from one location, with a pressure-dependent >> flow rate, and some of it is then reinjected in another location. This >> can result in poor nonlinear solver convergence. I think the reason is >> that there are effectively missing Jacobian entries in the row for the >> reinjection cell, which should have an additional dependence on the >> solution in the cell where fluid is extracted. >> >> - Adrian > > -- Dr Adrian Croucher Senior Research Fellow Department of Engineering Science Waipapa Taumata Rau / University of Auckland, New Zealand email:a.croucher at auckland.ac.nz tel: +64 (0)9 923 4611 -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Sun May 19 22:25:00 2024 From: bsmith at petsc.dev (Barry Smith) Date: Sun, 19 May 2024 23:25:00 -0400 Subject: [petsc-users] Modify matrix nonzero structure In-Reply-To: References: <9c4c4a7d-3da1-46ed-935e-b3d8cc80d474@auckland.ac.nz> <4368DA96-759E-4831-8BCF-710F265A4728@petsc.dev> Message-ID: <5AEAC3D2-9548-443E-96CF-FE7A814CE0F9@petsc.dev> Certainly missing Jacobian entries can dramatically change the Newton direction and hence the convergence. Even if the optimal (in time) setup skips some Jacobian entries it is always good to have runs with all the entries to see the "best possible" convergence. Barry > On May 19, 2024, at 10:44?PM, Adrian Croucher wrote: > > Great, it sounds like this might be easier than I expected. Thanks very much. > > Did you have any thoughts on my diagnosis of the problem (the poor nonlinear solver convergence being caused by missing Jacobian elements representing interaction between the sources)? > > - Adrian > > On 20/05/24 12:41 pm, Matthew Knepley wrote: >> On Sun, May 19, 2024 at 8:25?PM Barry Smith > wrote: >>> This Message Is From an External Sender >>> This message came from outside your organization. >>> >>> >>> You can call MatSetOption(mat,MAT_NEW_NONZERO_LOCATION_ERR) then insert the new values. If it is just a handful of new insertions the extra time should be small. >>> >>> Making a copy of the matrix won't give you a new matrix that is any faster to insert into so best to just use the same matrix. >> >> Let me add to Barry's answer. The preallocation infrastructure is now not strictly necessary. It is possible to just add all your nonzeros in and assembly, and the performance will be pretty good (uses hashing etc). So if just adding a few nonzeros does not work, we can go this route. >> >> Thanks, >> >> Matt >> >>> Barry >>> >>> >>>> On May 19, 2024, at 7:44?PM, Adrian Croucher > wrote: >>>> >>>> This Message Is From an External Sender >>>> This message came from outside your organization. >>>> hi, >>>> >>>> I have a Jacobian matrix created using DMCreateMatrix(). What would be >>>> the best way to add extra nonzero entries into it? >>>> >>>> I'm guessing that DMCreateMatrix() allocates the storage so the nonzero >>>> structure can't really be easily modified. Would it be a case of >>>> creating a new matrix, copying the nonzero entries from the original one >>>> and then adding the extra ones, before calling MatSetUp() or similar? If >>>> so, how exactly would you copy the nonzero structure from the original >>>> matrix? >>>> >>>> Background: the flow problem I'm solving (on a DMPlex with finite volume >>>> method) has complex source terms that depend on the solution (e.g. >>>> pressure), and can also depend on other source terms. A simple example >>>> is when fluid is extracted from one location, with a pressure-dependent >>>> flow rate, and some of it is then reinjected in another location. This >>>> can result in poor nonlinear solver convergence. I think the reason is >>>> that there are effectively missing Jacobian entries in the row for the >>>> reinjection cell, which should have an additional dependence on the >>>> solution in the cell where fluid is extracted. >>>> >>>> - Adrian >> > -- > Dr Adrian Croucher > Senior Research Fellow > Department of Engineering Science > Waipapa Taumata Rau / University of Auckland, New Zealand > email: a.croucher at auckland.ac.nz > tel: +64 (0)9 923 4611 -------------- next part -------------- An HTML attachment was scrubbed... URL: