From knepley at gmail.com Thu Sep 1 04:31:49 2022 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 1 Sep 2022 05:31:49 -0400 Subject: [petsc-users] DMLabel Synchronize In-Reply-To: References: Message-ID: On Wed, Aug 31, 2022 at 9:30 PM Nicholas Arnold-Medabalimi < narnoldm at umich.edu> wrote: > Hi > > Thanks, this looks like what I need, but I don't seem to have it in my > release (3.17.4 from > https://ftp.mcs.anl.gov/pub/petsc/release-snapshots/petsc-3.17.4.tar.gz). > I see it in the current repository, though. Is the 3.17.4 tarball just > missing it, or will it be added in the next release? > Yes, it is in main (https://petsc.org/main/docs/changes/dev/) since it was an interface change. Our main branch is quite stable, and I would recommend it for development work. Thanks, Matt > Thanks > Nicholas > > On Wed, Aug 31, 2022 at 7:45 AM Matthew Knepley wrote: > >> On Tue, Aug 30, 2022 at 6:37 PM Nicholas Arnold-Medabalimi < >> narnoldm at umich.edu> wrote: >> >>> Hi >>> >>> Thanks for all the help. I'm looking to tag several cells based on >>> metrics occurring at run time. In addition to the metric-defined cells I >>> also want to label the surrounding cells. This is no problem as I can >>> traverse the DMPlex chart to get the neighboring cells. The issue only >>> arises if an identified cell is on a partition boundary. this leads to the >>> neighbor cell in the overlap halo being tagged. However, the root cell on >>> the adjacent partition is not tagged. Is there a built-in way to >>> synchronize the labels from the leafs to the roots? >>> >>> Right now, I iterate through the root and leaf cells of each processor, >>> but it is somewhat wasteful as I don't see a way for the root cell >>> partition to know if a corresponding leaf cell label value without >>> communicating it first. >>> >> >> What I think you want is this: >> >> https://petsc.org/main/docs/manualpages/DMLabel/DMLabelPropagateBegin/ >> >> I use this to do the same kind of labeling I think. It is setup to allow >> multiple rounds, >> so that you can use the communicated labeling to continue on the new rank. >> >> Will this work? >> >> Thanks, >> >> Matt >> >> >>> Thanks >>> Nicholas >>> >>> -- >>> Nicholas Arnold-Medabalimi >>> >>> Ph.D. Candidate >>> Computational Aeroscience Lab >>> University of Michigan >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> >> > > > -- > Nicholas Arnold-Medabalimi > > Ph.D. Candidate > Computational Aeroscience Lab > University of Michigan > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From narnoldm at umich.edu Thu Sep 1 23:53:31 2022 From: narnoldm at umich.edu (Nicholas Arnold-Medabalimi) Date: Fri, 2 Sep 2022 00:53:31 -0400 Subject: [petsc-users] DMLabel Synchronize In-Reply-To: References: Message-ID: I've been trying to update from the snapshot to the main branch on the gitlab but I'm having one issue. This does not occur using the snapshot petsc mentioned previously. I use a PetscViewer to read in a file. The appended function is used to fast-forward to the locations in the file. Looking at the error message, I thought maybe the file wasn't opening properly. However, I am able to read in and print the first line of the file using PetscViewerRead. As far as I can tell, the error is only due to the version change and I see something about the viewer in the compatibility notes, but I'm not sure how to resolve it. Any assistance would be appreciated since I can't use the gitlab version for the label functions without fixing this. PetscErrorCode FileFastForward(PetscViewer viewer, char delim, bool &eof) { int i = 0, ret = 0; char buffer; do PetscViewerRead(viewer, &(buffer), 1, &ret, PETSC_CHAR); while (ret > 0 && buffer != delim); if (ret == 0) { printf("End of file reached before delimiter found\n"); eof = false; } else eof = true; PetscFunctionReturn(0); } The viewer is opened using PetscViewerCreate(PETSC_COMM_WORLD, &viewer); PetscViewerSetType(viewer, PETSCVIEWERASCII); PetscViewerFileSetMode(viewer, FILE_MODE_READ); PetscViewerFileSetName(viewer, meshfile); The error that is thrown is [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Petsc has generated inconsistent data [0]PETSC ERROR: Invalid stack size 0, pop FileFastForward /home/narnoldm/code/solver-playground/version_upgrade/version_debug.cpp:35. (Note this line is the PetscFunctionReturn) [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting. [0]PETSC ERROR: Petsc Development GIT revision: unknown GIT Date: unknown [0]PETSC ERROR: /home/narnoldm/code/solver-playground/build/bin/version on a named Dragoon by narnoldm Fri Sep 2 00:34:02 2022 [0]PETSC ERROR: Configure options --prefix=/home/narnoldm/code/solver-playground/build/external/PETSc --with-hdf5-dir=/home/narnoldm/code/solver-playground/build/external/HDF5 --download-triangle --download-parmetis --download-metis --with-debugging=1 [0]PETSC ERROR: #1 FileFastForward() at /home/narnoldm/code/solver-playground/version_upgrade/version_debug.cpp:35 application called MPI_Abort(MPI_COMM_SELF, 77) - process 0 [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=77 : system msg for write_line failure : Bad file descriptor On Thu, Sep 1, 2022 at 5:32 AM Matthew Knepley wrote: > On Wed, Aug 31, 2022 at 9:30 PM Nicholas Arnold-Medabalimi < > narnoldm at umich.edu> wrote: > >> Hi >> >> Thanks, this looks like what I need, but I don't seem to have it in my >> release (3.17.4 from >> https://ftp.mcs.anl.gov/pub/petsc/release-snapshots/petsc-3.17.4.tar.gz). >> I see it in the current repository, though. Is the 3.17.4 tarball just >> missing it, or will it be added in the next release? >> > > Yes, it is in main (https://petsc.org/main/docs/changes/dev/) since it > was an interface change. Our main branch is quite stable, and I would > recommend > it for development work. > > Thanks, > > Matt > > >> Thanks >> Nicholas >> >> On Wed, Aug 31, 2022 at 7:45 AM Matthew Knepley >> wrote: >> >>> On Tue, Aug 30, 2022 at 6:37 PM Nicholas Arnold-Medabalimi < >>> narnoldm at umich.edu> wrote: >>> >>>> Hi >>>> >>>> Thanks for all the help. I'm looking to tag several cells based on >>>> metrics occurring at run time. In addition to the metric-defined cells I >>>> also want to label the surrounding cells. This is no problem as I can >>>> traverse the DMPlex chart to get the neighboring cells. The issue only >>>> arises if an identified cell is on a partition boundary. this leads to the >>>> neighbor cell in the overlap halo being tagged. However, the root cell on >>>> the adjacent partition is not tagged. Is there a built-in way to >>>> synchronize the labels from the leafs to the roots? >>>> >>>> Right now, I iterate through the root and leaf cells of each processor, >>>> but it is somewhat wasteful as I don't see a way for the root cell >>>> partition to know if a corresponding leaf cell label value without >>>> communicating it first. >>>> >>> >>> What I think you want is this: >>> >>> https://petsc.org/main/docs/manualpages/DMLabel/DMLabelPropagateBegin/ >>> >>> I use this to do the same kind of labeling I think. It is setup to allow >>> multiple rounds, >>> so that you can use the communicated labeling to continue on the new >>> rank. >>> >>> Will this work? >>> >>> Thanks, >>> >>> Matt >>> >>> >>>> Thanks >>>> Nicholas >>>> >>>> -- >>>> Nicholas Arnold-Medabalimi >>>> >>>> Ph.D. Candidate >>>> Computational Aeroscience Lab >>>> University of Michigan >>>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> https://www.cse.buffalo.edu/~knepley/ >>> >>> >> >> >> -- >> Nicholas Arnold-Medabalimi >> >> Ph.D. Candidate >> Computational Aeroscience Lab >> University of Michigan >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -- Nicholas Arnold-Medabalimi Ph.D. Candidate Computational Aeroscience Lab University of Michigan -------------- next part -------------- An HTML attachment was scrubbed... URL: From yyang85 at alumni.stanford.edu Fri Sep 2 00:50:16 2022 From: yyang85 at alumni.stanford.edu (Yuyun Yang) Date: Fri, 2 Sep 2022 13:50:16 +0800 Subject: [petsc-users] compile warnings Message-ID: Hello team, I recently reconfigured PETSc on my Mac terminal, and running makefile gives me the following warnings. Will these affect running the code, and how do I get rid of these warnings? ld: warning: dylib (/Users/yuyunyang/petsc/arch-darwin-c-debug/lib/libscalapack.dylib) was built for newer macOS version (11.6) than being linked (11.0) ld: warning: dylib (/Users/yuyunyang/petsc/arch-darwin-c-debug/lib/libsuperlu.dylib) was built for newer macOS version (11.6) than being linked (11.0) ld: warning: dylib (/Users/yuyunyang/petsc/arch-darwin-c-debug/lib/libparmetis.dylib) was built for newer macOS version (11.6) than being linked (11.0) ld: warning: dylib (/Users/yuyunyang/petsc/arch-darwin-c-debug/lib/libmetis.dylib) was built for newer macOS version (11.6) than being linked (11.0) Thank you! Yuyun -------------- next part -------------- An HTML attachment was scrubbed... URL: From jroman at dsic.upv.es Fri Sep 2 01:31:11 2022 From: jroman at dsic.upv.es (Jose E. Roman) Date: Fri, 2 Sep 2022 08:31:11 +0200 Subject: [petsc-users] compile warnings In-Reply-To: References: Message-ID: <56F56B21-8C51-4553-AAB2-BB67F2FFDCB1@dsic.upv.es> Set this in your environment: export MACOSX_DEPLOYMENT_TARGET=11.6 Jose > El 2 sept 2022, a las 7:50, Yuyun Yang escribi?: > > Hello team, > > I recently reconfigured PETSc on my Mac terminal, and running makefile gives me the following warnings. Will these affect running the code, and how do I get rid of these warnings? > > ld: warning: dylib (/Users/yuyunyang/petsc/arch-darwin-c-debug/lib/libscalapack.dylib) was built for newer macOS version (11.6) than being linked (11.0) > ld: warning: dylib (/Users/yuyunyang/petsc/arch-darwin-c-debug/lib/libsuperlu.dylib) was built for newer macOS version (11.6) than being linked (11.0) > ld: warning: dylib (/Users/yuyunyang/petsc/arch-darwin-c-debug/lib/libparmetis.dylib) was built for newer macOS version (11.6) than being linked (11.0) > ld: warning: dylib (/Users/yuyunyang/petsc/arch-darwin-c-debug/lib/libmetis.dylib) was built for newer macOS version (11.6) than being linked (11.0) > > Thank you! > Yuyun From knepley at gmail.com Fri Sep 2 06:27:32 2022 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 2 Sep 2022 07:27:32 -0400 Subject: [petsc-users] DMLabel Synchronize In-Reply-To: References: Message-ID: On Fri, Sep 2, 2022 at 12:53 AM Nicholas Arnold-Medabalimi < narnoldm at umich.edu> wrote: > I've been trying to update from the snapshot to the main branch on the > gitlab but I'm having one issue. This does not occur using the snapshot > petsc mentioned previously. > > I use a PetscViewer to read in a file. The appended function is used to > fast-forward to the locations in the file. Looking at the error message, I > thought maybe the file wasn't opening properly. However, I am able to read > in and print the first line of the file using PetscViewerRead. As far as I > can tell, the error is only due to the version change and I see something > about the viewer in the compatibility notes, but I'm not sure how to > resolve it. Any assistance would be appreciated since I can't use the > gitlab version for the label functions without fixing this. > You are missing "PetscFunctionBeginUser;" at the start of your function. This must be paired with all PetscFunctionReturn() calls since we keep track of the stack. We added this check in the new version. Thanks, Matt > > PetscErrorCode FileFastForward(PetscViewer viewer, char delim, bool &eof) > { > int i = 0, ret = 0; > char buffer; > do > PetscViewerRead(viewer, &(buffer), 1, &ret, PETSC_CHAR); > while (ret > 0 && buffer != delim); > if (ret == 0) > { > printf("End of file reached before delimiter found\n"); > eof = false; > } > else > eof = true; > PetscFunctionReturn(0); > } > The viewer is opened using > PetscViewerCreate(PETSC_COMM_WORLD, &viewer); > PetscViewerSetType(viewer, PETSCVIEWERASCII); > PetscViewerFileSetMode(viewer, FILE_MODE_READ); > PetscViewerFileSetName(viewer, meshfile); > The error that is thrown is > > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Petsc has generated inconsistent data > [0]PETSC ERROR: Invalid stack size 0, pop FileFastForward > /home/narnoldm/code/solver-playground/version_upgrade/version_debug.cpp:35. > (Note this line is the PetscFunctionReturn) > > [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting. > [0]PETSC ERROR: Petsc Development GIT revision: unknown GIT Date: unknown > [0]PETSC ERROR: /home/narnoldm/code/solver-playground/build/bin/version on > a named Dragoon by narnoldm Fri Sep 2 00:34:02 2022 > [0]PETSC ERROR: Configure options > --prefix=/home/narnoldm/code/solver-playground/build/external/PETSc > --with-hdf5-dir=/home/narnoldm/code/solver-playground/build/external/HDF5 > --download-triangle --download-parmetis --download-metis --with-debugging=1 > [0]PETSC ERROR: #1 FileFastForward() at > /home/narnoldm/code/solver-playground/version_upgrade/version_debug.cpp:35 > application called MPI_Abort(MPI_COMM_SELF, 77) - process 0 > [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=77 > : > system msg for write_line failure : Bad file descriptor > > > > On Thu, Sep 1, 2022 at 5:32 AM Matthew Knepley wrote: > >> On Wed, Aug 31, 2022 at 9:30 PM Nicholas Arnold-Medabalimi < >> narnoldm at umich.edu> wrote: >> >>> Hi >>> >>> Thanks, this looks like what I need, but I don't seem to have it in my >>> release (3.17.4 from >>> https://ftp.mcs.anl.gov/pub/petsc/release-snapshots/petsc-3.17.4.tar.gz). >>> I see it in the current repository, though. Is the 3.17.4 tarball just >>> missing it, or will it be added in the next release? >>> >> >> Yes, it is in main (https://petsc.org/main/docs/changes/dev/) since it >> was an interface change. Our main branch is quite stable, and I would >> recommend >> it for development work. >> >> Thanks, >> >> Matt >> >> >>> Thanks >>> Nicholas >>> >>> On Wed, Aug 31, 2022 at 7:45 AM Matthew Knepley >>> wrote: >>> >>>> On Tue, Aug 30, 2022 at 6:37 PM Nicholas Arnold-Medabalimi < >>>> narnoldm at umich.edu> wrote: >>>> >>>>> Hi >>>>> >>>>> Thanks for all the help. I'm looking to tag several cells based on >>>>> metrics occurring at run time. In addition to the metric-defined cells I >>>>> also want to label the surrounding cells. This is no problem as I can >>>>> traverse the DMPlex chart to get the neighboring cells. The issue only >>>>> arises if an identified cell is on a partition boundary. this leads to the >>>>> neighbor cell in the overlap halo being tagged. However, the root cell on >>>>> the adjacent partition is not tagged. Is there a built-in way to >>>>> synchronize the labels from the leafs to the roots? >>>>> >>>>> Right now, I iterate through the root and leaf cells of each >>>>> processor, but it is somewhat wasteful as I don't see a way for the root >>>>> cell partition to know if a corresponding leaf cell label value without >>>>> communicating it first. >>>>> >>>> >>>> What I think you want is this: >>>> >>>> >>>> https://petsc.org/main/docs/manualpages/DMLabel/DMLabelPropagateBegin/ >>>> >>>> I use this to do the same kind of labeling I think. It is setup to >>>> allow multiple rounds, >>>> so that you can use the communicated labeling to continue on the new >>>> rank. >>>> >>>> Will this work? >>>> >>>> Thanks, >>>> >>>> Matt >>>> >>>> >>>>> Thanks >>>>> Nicholas >>>>> >>>>> -- >>>>> Nicholas Arnold-Medabalimi >>>>> >>>>> Ph.D. Candidate >>>>> Computational Aeroscience Lab >>>>> University of Michigan >>>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>>> https://www.cse.buffalo.edu/~knepley/ >>>> >>>> >>> >>> >>> -- >>> Nicholas Arnold-Medabalimi >>> >>> Ph.D. Candidate >>> Computational Aeroscience Lab >>> University of Michigan >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> >> > > > -- > Nicholas Arnold-Medabalimi > > Ph.D. Candidate > Computational Aeroscience Lab > University of Michigan > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From narnoldm at umich.edu Sun Sep 4 16:50:00 2022 From: narnoldm at umich.edu (Nicholas Arnold-Medabalimi) Date: Sun, 4 Sep 2022 17:50:00 -0400 Subject: [petsc-users] DMLabel Synchronize In-Reply-To: References: Message-ID: Thank you. I'm able to access the label functions. Can you clarify what the Begin and End functions are doing? To clarify the behavior I am trying to achieve, I have identified the blue cell in the attached figure using metrics during the solve. I have already created a label across all the partitions I'm using to tag the identified and adjacent cells however I need the Root cell label to be updated to one when the initial identified cell is adjacent to its corresponding leaf. That doesn't seem to be happening with I use the propagate command, but I may be misusing them. I appreciate the assistance. [image: image.png] Sincerely Nicholas On Fri, Sep 2, 2022 at 7:27 AM Matthew Knepley wrote: > On Fri, Sep 2, 2022 at 12:53 AM Nicholas Arnold-Medabalimi < > narnoldm at umich.edu> wrote: > >> I've been trying to update from the snapshot to the main branch on the >> gitlab but I'm having one issue. This does not occur using the snapshot >> petsc mentioned previously. >> >> I use a PetscViewer to read in a file. The appended function is used to >> fast-forward to the locations in the file. Looking at the error message, I >> thought maybe the file wasn't opening properly. However, I am able to read >> in and print the first line of the file using PetscViewerRead. As far as I >> can tell, the error is only due to the version change and I see something >> about the viewer in the compatibility notes, but I'm not sure how to >> resolve it. Any assistance would be appreciated since I can't use the >> gitlab version for the label functions without fixing this. >> > > You are missing "PetscFunctionBeginUser;" at the start of your function. > This must be paired with all PetscFunctionReturn() calls since we keep > track of the stack. We added this check in the new version. > > Thanks, > > Matt > > >> >> PetscErrorCode FileFastForward(PetscViewer viewer, char delim, bool &eof) >> { >> int i = 0, ret = 0; >> char buffer; >> do >> PetscViewerRead(viewer, &(buffer), 1, &ret, PETSC_CHAR); >> while (ret > 0 && buffer != delim); >> if (ret == 0) >> { >> printf("End of file reached before delimiter found\n"); >> eof = false; >> } >> else >> eof = true; >> PetscFunctionReturn(0); >> } >> The viewer is opened using >> PetscViewerCreate(PETSC_COMM_WORLD, &viewer); >> PetscViewerSetType(viewer, PETSCVIEWERASCII); >> PetscViewerFileSetMode(viewer, FILE_MODE_READ); >> PetscViewerFileSetName(viewer, meshfile); >> The error that is thrown is >> >> [0]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> [0]PETSC ERROR: Petsc has generated inconsistent data >> [0]PETSC ERROR: Invalid stack size 0, pop FileFastForward >> /home/narnoldm/code/solver-playground/version_upgrade/version_debug.cpp:35. >> (Note this line is the PetscFunctionReturn) >> >> [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting. >> [0]PETSC ERROR: Petsc Development GIT revision: unknown GIT Date: unknown >> [0]PETSC ERROR: /home/narnoldm/code/solver-playground/build/bin/version >> on a named Dragoon by narnoldm Fri Sep 2 00:34:02 2022 >> [0]PETSC ERROR: Configure options >> --prefix=/home/narnoldm/code/solver-playground/build/external/PETSc >> --with-hdf5-dir=/home/narnoldm/code/solver-playground/build/external/HDF5 >> --download-triangle --download-parmetis --download-metis --with-debugging=1 >> [0]PETSC ERROR: #1 FileFastForward() at >> /home/narnoldm/code/solver-playground/version_upgrade/version_debug.cpp:35 >> application called MPI_Abort(MPI_COMM_SELF, 77) - process 0 >> [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=77 >> : >> system msg for write_line failure : Bad file descriptor >> >> >> >> On Thu, Sep 1, 2022 at 5:32 AM Matthew Knepley wrote: >> >>> On Wed, Aug 31, 2022 at 9:30 PM Nicholas Arnold-Medabalimi < >>> narnoldm at umich.edu> wrote: >>> >>>> Hi >>>> >>>> Thanks, this looks like what I need, but I don't seem to have it in my >>>> release (3.17.4 from >>>> https://ftp.mcs.anl.gov/pub/petsc/release-snapshots/petsc-3.17.4.tar.gz). >>>> I see it in the current repository, though. Is the 3.17.4 tarball just >>>> missing it, or will it be added in the next release? >>>> >>> >>> Yes, it is in main (https://petsc.org/main/docs/changes/dev/) since it >>> was an interface change. Our main branch is quite stable, and I would >>> recommend >>> it for development work. >>> >>> Thanks, >>> >>> Matt >>> >>> >>>> Thanks >>>> Nicholas >>>> >>>> On Wed, Aug 31, 2022 at 7:45 AM Matthew Knepley >>>> wrote: >>>> >>>>> On Tue, Aug 30, 2022 at 6:37 PM Nicholas Arnold-Medabalimi < >>>>> narnoldm at umich.edu> wrote: >>>>> >>>>>> Hi >>>>>> >>>>>> Thanks for all the help. I'm looking to tag several cells based on >>>>>> metrics occurring at run time. In addition to the metric-defined cells I >>>>>> also want to label the surrounding cells. This is no problem as I can >>>>>> traverse the DMPlex chart to get the neighboring cells. The issue only >>>>>> arises if an identified cell is on a partition boundary. this leads to the >>>>>> neighbor cell in the overlap halo being tagged. However, the root cell on >>>>>> the adjacent partition is not tagged. Is there a built-in way to >>>>>> synchronize the labels from the leafs to the roots? >>>>>> >>>>>> Right now, I iterate through the root and leaf cells of each >>>>>> processor, but it is somewhat wasteful as I don't see a way for the root >>>>>> cell partition to know if a corresponding leaf cell label value without >>>>>> communicating it first. >>>>>> >>>>> >>>>> What I think you want is this: >>>>> >>>>> >>>>> https://petsc.org/main/docs/manualpages/DMLabel/DMLabelPropagateBegin/ >>>>> >>>>> I use this to do the same kind of labeling I think. It is setup to >>>>> allow multiple rounds, >>>>> so that you can use the communicated labeling to continue on the new >>>>> rank. >>>>> >>>>> Will this work? >>>>> >>>>> Thanks, >>>>> >>>>> Matt >>>>> >>>>> >>>>>> Thanks >>>>>> Nicholas >>>>>> >>>>>> -- >>>>>> Nicholas Arnold-Medabalimi >>>>>> >>>>>> Ph.D. Candidate >>>>>> Computational Aeroscience Lab >>>>>> University of Michigan >>>>>> >>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their >>>>> experiments is infinitely more interesting than any results to which their >>>>> experiments lead. >>>>> -- Norbert Wiener >>>>> >>>>> https://www.cse.buffalo.edu/~knepley/ >>>>> >>>>> >>>> >>>> >>>> -- >>>> Nicholas Arnold-Medabalimi >>>> >>>> Ph.D. Candidate >>>> Computational Aeroscience Lab >>>> University of Michigan >>>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> https://www.cse.buffalo.edu/~knepley/ >>> >>> >> >> >> -- >> Nicholas Arnold-Medabalimi >> >> Ph.D. Candidate >> Computational Aeroscience Lab >> University of Michigan >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -- Nicholas Arnold-Medabalimi Ph.D. Candidate Computational Aeroscience Lab University of Michigan -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image.png Type: image/png Size: 50054 bytes Desc: not available URL: From tranduchan at gmail.com Tue Sep 6 10:59:51 2022 From: tranduchan at gmail.com (Han Tran) Date: Tue, 6 Sep 2022 09:59:51 -0600 Subject: [petsc-users] PETSc with gcc/gfortran Message-ID: <9834EB74-76E8-4D8D-8A69-B1ACCC769931@gmail.com> I used to compile my Fortran 90 code using intel compilers (i.e. intel and impi). However, when I tried to compile using gcc, it gave the following error. use petscksp 1 Fatal Error: File 'petscksp.mod' opened at (1) is not a GNU Fortran module file It would be appreciated if you could give any hints on this issue. Thank you so much! -Han -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Tue Sep 6 11:21:50 2022 From: bsmith at petsc.dev (Barry Smith) Date: Tue, 6 Sep 2022 12:21:50 -0400 Subject: [petsc-users] PETSc with gcc/gfortran In-Reply-To: <9834EB74-76E8-4D8D-8A69-B1ACCC769931@gmail.com> References: <9834EB74-76E8-4D8D-8A69-B1ACCC769931@gmail.com> Message-ID: This happens when the module files were created with one Fortran compiler, say ifort`, but it is attempting to read them with a different Fortran compiler, say gfortran. (You can never mix to different Fortran compilers in the same executable/library/module system. You need to use a different PETSC_ARCH and configure PETSc with --with-fc=gfortran to use the gfortran compiler Barry > On Sep 6, 2022, at 11:59 AM, Han Tran wrote: > > I used to compile my Fortran 90 code using intel compilers (i.e. intel and impi). However, when I tried to compile using gcc, it gave the following error. > > use petscksp > 1 > Fatal Error: File 'petscksp.mod' opened at (1) is not a GNU Fortran module file > > It would be appreciated if you could give any hints on this issue. Thank you so much! > > -Han -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Tue Sep 6 11:23:01 2022 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 6 Sep 2022 11:23:01 -0500 (CDT) Subject: [petsc-users] PETSc with gcc/gfortran In-Reply-To: <9834EB74-76E8-4D8D-8A69-B1ACCC769931@gmail.com> References: <9834EB74-76E8-4D8D-8A69-B1ACCC769931@gmail.com> Message-ID: <8a25523b-5d0-54ce-3d10-3ab4c39e7b98@mcs.anl.gov> Did you rebuild PETSc with gnu compilers? You can have both builds of PETSc (one with Intel compilers, the other with GNU compilers) by building each with a different PETSC_ARCH value ./configure PETSC_ARCH=arch-intel --with-cc=icc ... ..configure PETSC_ARCH=arch-gnu --with-cc=gcc ... And then switch your application to use appropriate build of PETSc - by switching PETSC_ARCH value [if using PETSc formatted makefiles] Satish On Tue, 6 Sep 2022, Han Tran wrote: > I used to compile my Fortran 90 code using intel compilers (i.e. intel and impi). However, when I tried to compile using gcc, it gave the following error. > > use petscksp > 1 > Fatal Error: File 'petscksp.mod' opened at (1) is not a GNU Fortran module file > > It would be appreciated if you could give any hints on this issue. Thank you so much! > > -Han From tranduchan at gmail.com Tue Sep 6 11:45:25 2022 From: tranduchan at gmail.com (Han Tran) Date: Tue, 6 Sep 2022 10:45:25 -0600 Subject: [petsc-users] PETSc with gcc/gfortran In-Reply-To: <8a25523b-5d0-54ce-3d10-3ab4c39e7b98@mcs.anl.gov> References: <9834EB74-76E8-4D8D-8A69-B1ACCC769931@gmail.com> <8a25523b-5d0-54ce-3d10-3ab4c39e7b98@mcs.anl.gov> Message-ID: <26CA2DC0-D4CC-4EBD-8818-C320C64F3DAB@gmail.com> Thank you, Satish and Barry! I will try to re-build PETSc with gcc compiler. Best regards, -Han > On Sep 6, 2022, at 10:23 AM, Satish Balay wrote: > > Did you rebuild PETSc with gnu compilers? > > You can have both builds of PETSc (one with Intel compilers, the other with GNU compilers) by building each with a different PETSC_ARCH value > > ./configure PETSC_ARCH=arch-intel --with-cc=icc ... > ..configure PETSC_ARCH=arch-gnu --with-cc=gcc ... > > > And then switch your application to use appropriate build of PETSc - by switching PETSC_ARCH value [if using PETSc formatted makefiles] > > Satish > > On Tue, 6 Sep 2022, Han Tran wrote: > >> I used to compile my Fortran 90 code using intel compilers (i.e. intel and impi). However, when I tried to compile using gcc, it gave the following error. >> >> use petscksp >> 1 >> Fatal Error: File 'petscksp.mod' opened at (1) is not a GNU Fortran module file >> >> It would be appreciated if you could give any hints on this issue. Thank you so much! >> >> -Han > From Jiannan_Tu at uml.edu Tue Sep 6 22:00:48 2022 From: Jiannan_Tu at uml.edu (Tu, Jiannan) Date: Wed, 7 Sep 2022 03:00:48 +0000 Subject: [petsc-users] Using matrix-free with KSP In-Reply-To: References: Message-ID: I am using TS IMEX to solve a large DAE system. The DAE is obtained by applying finite FV method to 3-D multi-specie ion/neutral fluids equations with magnetic induction equation. The Jacobian for stiff part is formed by using MatSetValuesStencil(). The Jacobian matrix is very sparse, no more than 10 non-zeros on each row. MatSetValuesStencil requires local to global mapping by calling ISLocaltoGlobalMapping(). Could you please instruct me how to use local to global mapping? I also tried using DMCreateMatrix() to create the Jacobian. While local to global mapping is not necessary, the matrix takes too much memory and requires 64-bit indices. I would prefer to take the advantage of sparsity of the Jacobian, pre-allocate the matrix to use as less as possible memory so that the code can be run on a multi-core desktop. Thank you very much for your advice. Jiannan -------------- next part -------------- An HTML attachment was scrubbed... URL: From hongzhang at anl.gov Tue Sep 6 22:10:11 2022 From: hongzhang at anl.gov (Zhang, Hong) Date: Wed, 7 Sep 2022 03:10:11 +0000 Subject: [petsc-users] Using matrix-free with KSP In-Reply-To: References: Message-ID: <040A2B58-7489-4EA1-A31B-4481169E874A@anl.gov> You can refer to the example src/ts/tutorials/advection-diffusion-reaction/ex5adj_mf.c which uses a matrix-free approach for a 2D multi-component reaction-diffusion PDE. In particular, you only need to look at MyIMatMult() and ignore the transposed version which is intended for adjoint calculations. Hong (Mr.) On Sep 6, 2022, at 10:00 PM, Tu, Jiannan > wrote: I am using TS IMEX to solve a large DAE system. The DAE is obtained by applying finite FV method to 3-D multi-specie ion/neutral fluids equations with magnetic induction equation. The Jacobian for stiff part is formed by using MatSetValuesStencil(). The Jacobian matrix is very sparse, no more than 10 non-zeros on each row. MatSetValuesStencil requires local to global mapping by calling ISLocaltoGlobalMapping(). Could you please instruct me how to use local to global mapping? I also tried using DMCreateMatrix() to create the Jacobian. While local to global mapping is not necessary, the matrix takes too much memory and requires 64-bit indices. I would prefer to take the advantage of sparsity of the Jacobian, pre-allocate the matrix to use as less as possible memory so that the code can be run on a multi-core desktop. Thank you very much for your advice. Jiannan -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Tue Sep 6 22:33:32 2022 From: bsmith at petsc.dev (Barry Smith) Date: Tue, 6 Sep 2022 23:33:32 -0400 Subject: [petsc-users] Using matrix-free with KSP In-Reply-To: References: Message-ID: <31AF5BF6-8AC4-4A76-9235-9C0E1BB3EE16@petsc.dev> > On Sep 6, 2022, at 11:00 PM, Tu, Jiannan wrote: > > I am using TS IMEX to solve a large DAE system. The DAE is obtained by applying finite FV method to 3-D multi-specie ion/neutral fluids equations with magnetic induction equation. The Jacobian for stiff part is formed by using MatSetValuesStencil(). The Jacobian matrix is very sparse, no more than 10 non-zeros on each row. MatSetValuesStencil requires local to global mapping by calling ISLocaltoGlobalMapping(). Could you please instruct me how to use local to global mapping? DMDA automatically sets up the ISLocaltoGlobalMapping() so you should not need to. > > > I also tried using DMCreateMatrix() to create the Jacobian. While local to global mapping is not necessary, the matrix takes too much memory and requires 64-bit indices. I would prefer to take the advantage of sparsity of the Jacobian, pre-allocate the matrix to use as less as possible memory so that the code can be run on a multi-core desktop. If using matrix-free does not work for you because the linear solves do not converge or converge too slowly. Then you might be able to decrease the memory used in the matrix. The DMCreateMatrix() does take advantage of sparsity and tries to preallocate only what it needs. Often what it preallocates is the best possible, but for multicomponent problems it has to assume there is full coupling within all the degrees of freedom that live at each at each grid point. How many components live at each grid point in your model and are they not all coupled to each other in the equations? If they are not fully coupled you can use either DMDASetBlockFills() or DMDASetBlockFillsSparse() to indicate the reduced coupling that actually exists for you model. Barry > > Thank you very much for your advice. > > Jiannan -------------- next part -------------- An HTML attachment was scrubbed... URL: From patrick.alken at geomag.info Tue Sep 6 23:18:41 2022 From: patrick.alken at geomag.info (Patrick Alken) Date: Tue, 6 Sep 2022 22:18:41 -0600 Subject: [petsc-users] [slepc] nan output for eigenvectors Message-ID: I sometimes get Nan output values in computed eigenvectors for the generalized symmetric eigenvalue problem produced by slepc. Is this a known issue, and is it related to the conditioning of the matrix pair (A,B)? Is there some way to compute a "condition number" of the matrix pair ahead of time to see if i have a good chance of getting stable eigenvectors out? In a possibly related issue, i am finding that petsc/slepc compiled with debugging vs optimization can produce very different eigenvectors for the same problem, while the eigenvalues are the same. The eigenvectors seem more accurate when I use the debugging version of the libraries. Could this be also a conditioning problem with the matrix pair? -------------- next part -------------- An HTML attachment was scrubbed... URL: From jroman at dsic.upv.es Wed Sep 7 02:20:50 2022 From: jroman at dsic.upv.es (Jose E. Roman) Date: Wed, 7 Sep 2022 09:20:50 +0200 Subject: [petsc-users] [slepc] nan output for eigenvectors In-Reply-To: References: Message-ID: > El 7 sept 2022, a las 6:18, Patrick Alken escribi?: > > I sometimes get Nan output values in computed eigenvectors for the generalized symmetric eigenvalue problem produced by slepc. Is this a known issue, and is it related to the conditioning of the matrix pair (A,B)? Is there some way to compute a "condition number" of the matrix pair ahead of time to see if i have a good chance of getting stable eigenvectors out? You should never get NaN. Can you send a reproducible example? > > In a possibly related issue, i am finding that petsc/slepc compiled with debugging vs optimization can produce very different eigenvectors for the same problem, while the eigenvalues are the same. The eigenvectors seem more accurate when I use the debugging version of the libraries. Could this be also a conditioning problem with the matrix pair? What do you mean more accurate? The residual norm computed with EPSComputeError() should be below the tolerance in both debugging and optimized versions. Jose From bsmith at petsc.dev Wed Sep 7 08:21:59 2022 From: bsmith at petsc.dev (Barry Smith) Date: Wed, 7 Sep 2022 09:21:59 -0400 Subject: [petsc-users] [slepc] nan output for eigenvectors In-Reply-To: References: Message-ID: <6E836BFB-EF2E-4E71-B6D9-B23473F3A4A1@petsc.dev> You can run with -fp_trap to have the program stop as soon as the first Nan or Inf appears, this can help track down why it is happening. In a debugger you can also set the debugger to trap on floating point exceptions (syntax is debugger dependent) to focus in on where it first happens. Barry > On Sep 7, 2022, at 3:20 AM, Jose E. Roman wrote: > > > >> El 7 sept 2022, a las 6:18, Patrick Alken escribi?: >> >> I sometimes get Nan output values in computed eigenvectors for the generalized symmetric eigenvalue problem produced by slepc. Is this a known issue, and is it related to the conditioning of the matrix pair (A,B)? Is there some way to compute a "condition number" of the matrix pair ahead of time to see if i have a good chance of getting stable eigenvectors out? > > You should never get NaN. Can you send a reproducible example? > >> >> In a possibly related issue, i am finding that petsc/slepc compiled with debugging vs optimization can produce very different eigenvectors for the same problem, while the eigenvalues are the same. The eigenvectors seem more accurate when I use the debugging version of the libraries. Could this be also a conditioning problem with the matrix pair? > > What do you mean more accurate? The residual norm computed with EPSComputeError() should be below the tolerance in both debugging and optimized versions. > > Jose > From knepley at gmail.com Wed Sep 7 08:44:12 2022 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 7 Sep 2022 09:44:12 -0400 Subject: [petsc-users] DMLabel Synchronize In-Reply-To: References: Message-ID: On Sun, Sep 4, 2022 at 5:50 PM Nicholas Arnold-Medabalimi < narnoldm at umich.edu> wrote: > Thank you. I'm able to access the label functions. Can you clarify what > the Begin and End functions are doing? > > To clarify the behavior I am trying to achieve, I have identified the blue > cell in the attached figure using metrics during the solve. I have already > created a label across all the partitions I'm using to tag the identified > and adjacent cells however I need the Root cell label to be updated to one > when the initial identified cell is adjacent to its corresponding leaf. > That doesn't seem to be happening with I use the propagate command, but I > may be misusing them. I appreciate the assistance. > I need to explain in more detail what is happening. Propagate ensures that all points involved in the PetscSF for points, meaning all shared points, have the same label values. In your diagram, it would appear that the 3 faces in red are shared, so label values on those faces would be made consistent across processes. >From your description, it would appear that you have at least 2 choices: 1) Label faces as well, with the rule that face labels cause any cell in the support to be labeled, and labeling a cell also labels its faces 2) Partition with a cell overlap Thanks, Matt > [image: image.png] > > > Sincerely > Nicholas > > > > On Fri, Sep 2, 2022 at 7:27 AM Matthew Knepley wrote: > >> On Fri, Sep 2, 2022 at 12:53 AM Nicholas Arnold-Medabalimi < >> narnoldm at umich.edu> wrote: >> >>> I've been trying to update from the snapshot to the main branch on the >>> gitlab but I'm having one issue. This does not occur using the snapshot >>> petsc mentioned previously. >>> >>> I use a PetscViewer to read in a file. The appended function is used to >>> fast-forward to the locations in the file. Looking at the error message, I >>> thought maybe the file wasn't opening properly. However, I am able to read >>> in and print the first line of the file using PetscViewerRead. As far as I >>> can tell, the error is only due to the version change and I see something >>> about the viewer in the compatibility notes, but I'm not sure how to >>> resolve it. Any assistance would be appreciated since I can't use the >>> gitlab version for the label functions without fixing this. >>> >> >> You are missing "PetscFunctionBeginUser;" at the start of your function. >> This must be paired with all PetscFunctionReturn() calls since we keep >> track of the stack. We added this check in the new version. >> >> Thanks, >> >> Matt >> >> >>> >>> PetscErrorCode FileFastForward(PetscViewer viewer, char delim, bool &eof >>> ) >>> { >>> int i = 0, ret = 0; >>> char buffer; >>> do >>> PetscViewerRead(viewer, &(buffer), 1, &ret, PETSC_CHAR); >>> while (ret > 0 && buffer != delim); >>> if (ret == 0) >>> { >>> printf("End of file reached before delimiter found\n"); >>> eof = false; >>> } >>> else >>> eof = true; >>> PetscFunctionReturn(0); >>> } >>> The viewer is opened using >>> PetscViewerCreate(PETSC_COMM_WORLD, &viewer); >>> PetscViewerSetType(viewer, PETSCVIEWERASCII); >>> PetscViewerFileSetMode(viewer, FILE_MODE_READ); >>> PetscViewerFileSetName(viewer, meshfile); >>> The error that is thrown is >>> >>> [0]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> [0]PETSC ERROR: Petsc has generated inconsistent data >>> [0]PETSC ERROR: Invalid stack size 0, pop FileFastForward >>> /home/narnoldm/code/solver-playground/version_upgrade/version_debug.cpp:35. >>> (Note this line is the PetscFunctionReturn) >>> >>> [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting. >>> [0]PETSC ERROR: Petsc Development GIT revision: unknown GIT Date: >>> unknown >>> [0]PETSC ERROR: /home/narnoldm/code/solver-playground/build/bin/version >>> on a named Dragoon by narnoldm Fri Sep 2 00:34:02 2022 >>> [0]PETSC ERROR: Configure options >>> --prefix=/home/narnoldm/code/solver-playground/build/external/PETSc >>> --with-hdf5-dir=/home/narnoldm/code/solver-playground/build/external/HDF5 >>> --download-triangle --download-parmetis --download-metis --with-debugging=1 >>> [0]PETSC ERROR: #1 FileFastForward() at >>> /home/narnoldm/code/solver-playground/version_upgrade/version_debug.cpp:35 >>> application called MPI_Abort(MPI_COMM_SELF, 77) - process 0 >>> [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=77 >>> : >>> system msg for write_line failure : Bad file descriptor >>> >>> >>> >>> On Thu, Sep 1, 2022 at 5:32 AM Matthew Knepley >>> wrote: >>> >>>> On Wed, Aug 31, 2022 at 9:30 PM Nicholas Arnold-Medabalimi < >>>> narnoldm at umich.edu> wrote: >>>> >>>>> Hi >>>>> >>>>> Thanks, this looks like what I need, but I don't seem to have it in my >>>>> release (3.17.4 from >>>>> https://ftp.mcs.anl.gov/pub/petsc/release-snapshots/petsc-3.17.4.tar.gz). >>>>> I see it in the current repository, though. Is the 3.17.4 tarball just >>>>> missing it, or will it be added in the next release? >>>>> >>>> >>>> Yes, it is in main (https://petsc.org/main/docs/changes/dev/) since it >>>> was an interface change. Our main branch is quite stable, and I would >>>> recommend >>>> it for development work. >>>> >>>> Thanks, >>>> >>>> Matt >>>> >>>> >>>>> Thanks >>>>> Nicholas >>>>> >>>>> On Wed, Aug 31, 2022 at 7:45 AM Matthew Knepley >>>>> wrote: >>>>> >>>>>> On Tue, Aug 30, 2022 at 6:37 PM Nicholas Arnold-Medabalimi < >>>>>> narnoldm at umich.edu> wrote: >>>>>> >>>>>>> Hi >>>>>>> >>>>>>> Thanks for all the help. I'm looking to tag several cells based on >>>>>>> metrics occurring at run time. In addition to the metric-defined cells I >>>>>>> also want to label the surrounding cells. This is no problem as I can >>>>>>> traverse the DMPlex chart to get the neighboring cells. The issue only >>>>>>> arises if an identified cell is on a partition boundary. this leads to the >>>>>>> neighbor cell in the overlap halo being tagged. However, the root cell on >>>>>>> the adjacent partition is not tagged. Is there a built-in way to >>>>>>> synchronize the labels from the leafs to the roots? >>>>>>> >>>>>>> Right now, I iterate through the root and leaf cells of each >>>>>>> processor, but it is somewhat wasteful as I don't see a way for the root >>>>>>> cell partition to know if a corresponding leaf cell label value without >>>>>>> communicating it first. >>>>>>> >>>>>> >>>>>> What I think you want is this: >>>>>> >>>>>> >>>>>> https://petsc.org/main/docs/manualpages/DMLabel/DMLabelPropagateBegin/ >>>>>> >>>>>> I use this to do the same kind of labeling I think. It is setup to >>>>>> allow multiple rounds, >>>>>> so that you can use the communicated labeling to continue on the new >>>>>> rank. >>>>>> >>>>>> Will this work? >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Matt >>>>>> >>>>>> >>>>>>> Thanks >>>>>>> Nicholas >>>>>>> >>>>>>> -- >>>>>>> Nicholas Arnold-Medabalimi >>>>>>> >>>>>>> Ph.D. Candidate >>>>>>> Computational Aeroscience Lab >>>>>>> University of Michigan >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> What most experimenters take for granted before they begin their >>>>>> experiments is infinitely more interesting than any results to which their >>>>>> experiments lead. >>>>>> -- Norbert Wiener >>>>>> >>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Nicholas Arnold-Medabalimi >>>>> >>>>> Ph.D. Candidate >>>>> Computational Aeroscience Lab >>>>> University of Michigan >>>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>>> https://www.cse.buffalo.edu/~knepley/ >>>> >>>> >>> >>> >>> -- >>> Nicholas Arnold-Medabalimi >>> >>> Ph.D. Candidate >>> Computational Aeroscience Lab >>> University of Michigan >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> >> > > > -- > Nicholas Arnold-Medabalimi > > Ph.D. Candidate > Computational Aeroscience Lab > University of Michigan > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image.png Type: image/png Size: 50054 bytes Desc: not available URL: From Jiannan_Tu at uml.edu Wed Sep 7 09:47:39 2022 From: Jiannan_Tu at uml.edu (Tu, Jiannan) Date: Wed, 7 Sep 2022 14:47:39 +0000 Subject: [petsc-users] Using matrix-free with KSP In-Reply-To: <31AF5BF6-8AC4-4A76-9235-9C0E1BB3EE16@petsc.dev> References: <31AF5BF6-8AC4-4A76-9235-9C0E1BB3EE16@petsc.dev> Message-ID: Barry and Hong, Thank you. There are 26 components at each grid and there are not fully coupled in terms of stiff functions. Mutual coupling is among about 6 components. I would prefer not using matrix-free since the Jacobina is not difficult to calculate and only up to 10 non-zeros at each row. I'll try DMDASetBlockFills() or DMDASetBlockFillsSparse() and see how they can reduce the memory usage. Jiannan ________________________________ From: Barry Smith Sent: Tuesday, September 6, 2022 11:33 PM To: Tu, Jiannan Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] Using matrix-free with KSP CAUTION: This email was sent from outside the UMass Lowell network. On Sep 6, 2022, at 11:00 PM, Tu, Jiannan > wrote: I am using TS IMEX to solve a large DAE system. The DAE is obtained by applying finite FV method to 3-D multi-specie ion/neutral fluids equations with magnetic induction equation. The Jacobian for stiff part is formed by using MatSetValuesStencil(). The Jacobian matrix is very sparse, no more than 10 non-zeros on each row. MatSetValuesStencil requires local to global mapping by calling ISLocaltoGlobalMapping(). Could you please instruct me how to use local to global mapping? DMDA automatically sets up the ISLocaltoGlobalMapping() so you should not need to. I also tried using DMCreateMatrix() to create the Jacobian. While local to global mapping is not necessary, the matrix takes too much memory and requires 64-bit indices. I would prefer to take the advantage of sparsity of the Jacobian, pre-allocate the matrix to use as less as possible memory so that the code can be run on a multi-core desktop. If using matrix-free does not work for you because the linear solves do not converge or converge too slowly. Then you might be able to decrease the memory used in the matrix. The DMCreateMatrix() does take advantage of sparsity and tries to preallocate only what it needs. Often what it preallocates is the best possible, but for multicomponent problems it has to assume there is full coupling within all the degrees of freedom that live at each at each grid point. How many components live at each grid point in your model and are they not all coupled to each other in the equations? If they are not fully coupled you can use either DMDASetBlockFills() or DMDASetBlockFillsSparse() to indicate the reduced coupling that actually exists for you model. Barry Thank you very much for your advice. Jiannan -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Wed Sep 7 10:53:50 2022 From: bsmith at petsc.dev (Barry Smith) Date: Wed, 7 Sep 2022 11:53:50 -0400 Subject: [petsc-users] Using matrix-free with KSP In-Reply-To: References: <31AF5BF6-8AC4-4A76-9235-9C0E1BB3EE16@petsc.dev> Message-ID: <7F5F237F-B73B-447E-A113-B46C697D4C37@petsc.dev> DMDASetBlockFills() or DMDASetBlockFillsSparse() (they are just different ways of providing the same information) will help you here enormously, your type of case is exactly what they were designed for. Barry > > On Sep 7, 2022, at 10:47 AM, Tu, Jiannan wrote: > > Barry and Hong, > > Thank you. > > There are 26 components at each grid and there are not fully coupled in terms of stiff functions. Mutual coupling is among about 6 components. > > I would prefer not using matrix-free since the Jacobina is not difficult to calculate and only up to 10 non-zeros at each row. I'll try DMDASetBlockFills() or DMDASetBlockFillsSparse() and see how they can reduce the memory usage. > > Jiannan > From: Barry Smith > > Sent: Tuesday, September 6, 2022 11:33 PM > To: Tu, Jiannan > > Cc: petsc-users at mcs.anl.gov > > Subject: Re: [petsc-users] Using matrix-free with KSP > > CAUTION: This email was sent from outside the UMass Lowell network. > > > >> On Sep 6, 2022, at 11:00 PM, Tu, Jiannan > wrote: >> >> I am using TS IMEX to solve a large DAE system. The DAE is obtained by applying finite FV method to 3-D multi-specie ion/neutral fluids equations with magnetic induction equation. The Jacobian for stiff part is formed by using MatSetValuesStencil(). The Jacobian matrix is very sparse, no more than 10 non-zeros on each row. MatSetValuesStencil requires local to global mapping by calling ISLocaltoGlobalMapping(). Could you please instruct me how to use local to global mapping? > > DMDA automatically sets up the ISLocaltoGlobalMapping() so you should not need to. >> >> >> I also tried using DMCreateMatrix() to create the Jacobian. While local to global mapping is not necessary, the matrix takes too much memory and requires 64-bit indices. I would prefer to take the advantage of sparsity of the Jacobian, pre-allocate the matrix to use as less as possible memory so that the code can be run on a multi-core desktop. > > If using matrix-free does not work for you because the linear solves do not converge or converge too slowly. Then you might be able to decrease the memory used in the matrix. The DMCreateMatrix() does take advantage of sparsity and tries to preallocate only what it needs. Often what it preallocates is the best possible, but for multicomponent problems it has to assume there is full coupling within all the degrees of freedom that live at each at each grid point. How many components live at each grid point in your model and are they not all coupled to each other in the equations? If they are not fully coupled you can use either DMDASetBlockFills() or DMDASetBlockFillsSparse() to indicate the reduced coupling that actually exists for you model. > > Barry > >> >> Thank you very much for your advice. >> >> Jiannan -------------- next part -------------- An HTML attachment was scrubbed... URL: From patrick.alken at geomag.info Wed Sep 7 21:31:01 2022 From: patrick.alken at geomag.info (Patrick Alken) Date: Wed, 7 Sep 2022 20:31:01 -0600 Subject: [petsc-users] [slepc] nan output for eigenvectors In-Reply-To: References: Message-ID: It looks like my problem may be in using the older gcc 7.5.0 compiler. The nans seem to go away when I compile petsc/slepc and my code on a newer gcc 11.2 compiler. Also, the petsc/slepc "make check" tests have errors on gcc 7.5 which disappear on 11.2. On 9/7/22 01:20, Jose E. Roman wrote: > >> El 7 sept 2022, a las 6:18, Patrick Alken escribi?: >> >> I sometimes get Nan output values in computed eigenvectors for the generalized symmetric eigenvalue problem produced by slepc. Is this a known issue, and is it related to the conditioning of the matrix pair (A,B)? Is there some way to compute a "condition number" of the matrix pair ahead of time to see if i have a good chance of getting stable eigenvectors out? > You should never get NaN. Can you send a reproducible example? > >> In a possibly related issue, i am finding that petsc/slepc compiled with debugging vs optimization can produce very different eigenvectors for the same problem, while the eigenvalues are the same. The eigenvectors seem more accurate when I use the debugging version of the libraries. Could this be also a conditioning problem with the matrix pair? > What do you mean more accurate? The residual norm computed with EPSComputeError() should be below the tolerance in both debugging and optimized versions. > > Jose > From quentin.chevalier at polytechnique.edu Thu Sep 8 06:43:15 2022 From: quentin.chevalier at polytechnique.edu (Quentin Chevalier) Date: Thu, 8 Sep 2022 13:43:15 +0200 Subject: [petsc-users] Read and write HDF5 files Message-ID: Hello PETSc users ! I'm trying to use PETSc routine to read and write complex vectors to binary files in parallel. I'm using dolfinx and petsc4py inside a docker container. My end goal is to write a dolfinx Function object to file, then read it again. My current code goes something like : viewer = pet.Viewer().createHDF5("file.h5", 'w', comm) function.vector.view(viewer) And fails with traceback : petsc4py.PETSc.Error: error code 86 [9] PetscViewerSetType() at /usr/local/petsc/src/sys/classes/viewer/interface/viewreg.c:435 [9] Unknown type. Check for miss-spelling or missing package: https://petsc.org/release/install/install/#external-packages [9] Unknown PetscViewer type given: hdf5 The relevant functions show on pydoc, what am I doing wrong ? Cheers, Quentin -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Sep 8 07:38:33 2022 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 8 Sep 2022 08:38:33 -0400 Subject: [petsc-users] Read and write HDF5 files In-Reply-To: References: Message-ID: On Thu, Sep 8, 2022 at 7:44 AM Quentin Chevalier < quentin.chevalier at polytechnique.edu> wrote: > Hello PETSc users ! > > I'm trying to use PETSc routine to read and write complex vectors to > binary files in parallel. > > I'm using dolfinx and petsc4py inside a docker container. My end goal is > to write a dolfinx Function object to file, then read it again. > > My current code goes something like : > viewer = pet.Viewer().createHDF5("file.h5", 'w', comm) > function.vector.view(viewer) > > And fails with traceback : > petsc4py.PETSc.Error: error code 86 > [9] PetscViewerSetType() at > /usr/local/petsc/src/sys/classes/viewer/interface/viewreg.c:435 > [9] Unknown type. Check for miss-spelling or missing package: > https://petsc.org/release/install/install/#external-packages > [9] Unknown PetscViewer type given: hdf5 > > The relevant functions show on pydoc, what am I doing wrong ? > You have not built PETSc with HDF5 support. In order to do this, you need to configure it using --with-hdf5-dir=/path/to/hdf5 or maybe --download-hdf5. Thanks, Matt > Cheers, > > Quentin > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Thu Sep 8 08:04:00 2022 From: jed at jedbrown.org (Jed Brown) Date: Thu, 08 Sep 2022 07:04:00 -0600 Subject: [petsc-users] Read and write HDF5 files In-Reply-To: References: Message-ID: <87pmg6rpa7.fsf@jedbrown.org> It sounds like the PETSc inside your container was not built with --download-hdf5 (or --with-hdf5). That can be fixed by updating the Dockerfile and rebuilding the image. Quentin Chevalier writes: > Hello PETSc users ! > > I'm trying to use PETSc routine to read and write complex vectors to binary > files in parallel. > > I'm using dolfinx and petsc4py inside a docker container. My end goal is to > write a dolfinx Function object to file, then read it again. > > My current code goes something like : > viewer = pet.Viewer().createHDF5("file.h5", 'w', comm) > function.vector.view(viewer) > > And fails with traceback : > petsc4py.PETSc.Error: error code 86 > [9] PetscViewerSetType() at > /usr/local/petsc/src/sys/classes/viewer/interface/viewreg.c:435 > [9] Unknown type. Check for miss-spelling or missing package: > https://petsc.org/release/install/install/#external-packages > [9] Unknown PetscViewer type given: hdf5 > > The relevant functions show on pydoc, what am I doing wrong ? > > Cheers, > > Quentin From Jiannan_Tu at uml.edu Thu Sep 8 12:47:22 2022 From: Jiannan_Tu at uml.edu (Tu, Jiannan) Date: Thu, 8 Sep 2022 17:47:22 +0000 Subject: [petsc-users] Using matrix-free with KSP In-Reply-To: <7F5F237F-B73B-447E-A113-B46C697D4C37@petsc.dev> References: <31AF5BF6-8AC4-4A76-9235-9C0E1BB3EE16@petsc.dev> <7F5F237F-B73B-447E-A113-B46C697D4C37@petsc.dev> Message-ID: Barry, Thank you very much. DMDASetBlockFills() needs two input arrays specifying sparsity pattern. In my case, I have 26 components at each grid. Is it correct that I need to have two arrays of 26x26 elements with value 1 for coupling and 0 for no coupling? Also how to determine when the value should be 1 or 0? Each component involves 7 stencils (i-1, j, k), (I, j, k), (i+1, j, k), (I, j-1, k), (I, j+1, k), (I, j, k-1), and (I, j, k+1), and couples with several other components. Jiannan Sent from Mail for Windows From: Barry Smith Sent: Wednesday, September 7, 2022 11:53 AM To: Tu, Jiannan Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] Using matrix-free with KSP CAUTION: This email was sent from outside the UMass Lowell network. DMDASetBlockFills() or DMDASetBlockFillsSparse() (they are just different ways of providing the same information) will help you here enormously, your type of case is exactly what they were designed for. Barry On Sep 7, 2022, at 10:47 AM, Tu, Jiannan > wrote: Barry and Hong, Thank you. There are 26 components at each grid and there are not fully coupled in terms of stiff functions. Mutual coupling is among about 6 components. I would prefer not using matrix-free since the Jacobina is not difficult to calculate and only up to 10 non-zeros at each row. I'll try DMDASetBlockFills() or DMDASetBlockFillsSparse() and see how they can reduce the memory usage. Jiannan From: Barry Smith > Sent: Tuesday, September 6, 2022 11:33 PM To: Tu, Jiannan > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Using matrix-free with KSP CAUTION: This email was sent from outside the UMass Lowell network. On Sep 6, 2022, at 11:00 PM, Tu, Jiannan > wrote: I am using TS IMEX to solve a large DAE system. The DAE is obtained by applying finite FV method to 3-D multi-specie ion/neutral fluids equations with magnetic induction equation. The Jacobian for stiff part is formed by using MatSetValuesStencil(). The Jacobian matrix is very sparse, no more than 10 non-zeros on each row. MatSetValuesStencil requires local to global mapping by calling ISLocaltoGlobalMapping(). Could you please instruct me how to use local to global mapping? DMDA automatically sets up the ISLocaltoGlobalMapping() so you should not need to. I also tried using DMCreateMatrix() to create the Jacobian. While local to global mapping is not necessary, the matrix takes too much memory and requires 64-bit indices. I would prefer to take the advantage of sparsity of the Jacobian, pre-allocate the matrix to use as less as possible memory so that the code can be run on a multi-core desktop. If using matrix-free does not work for you because the linear solves do not converge or converge too slowly. Then you might be able to decrease the memory used in the matrix. The DMCreateMatrix() does take advantage of sparsity and tries to preallocate only what it needs. Often what it preallocates is the best possible, but for multicomponent problems it has to assume there is full coupling within all the degrees of freedom that live at each at each grid point. How many components live at each grid point in your model and are they not all coupled to each other in the equations? If they are not fully coupled you can use either DMDASetBlockFills() or DMDASetBlockFillsSparse() to indicate the reduced coupling that actually exists for you model. Barry Thank you very much for your advice. Jiannan -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: E6340AADF8494DB7861EA4659E7713CE.png Type: image/png Size: 135 bytes Desc: E6340AADF8494DB7861EA4659E7713CE.png URL: From bsmith at petsc.dev Thu Sep 8 13:12:06 2022 From: bsmith at petsc.dev (Barry Smith) Date: Thu, 8 Sep 2022 14:12:06 -0400 Subject: [petsc-users] Using matrix-free with KSP In-Reply-To: References: <31AF5BF6-8AC4-4A76-9235-9C0E1BB3EE16@petsc.dev> <7F5F237F-B73B-447E-A113-B46C697D4C37@petsc.dev> Message-ID: <79BC638A-1064-42A1-AE04-40ACAA82323A@petsc.dev> > On Sep 8, 2022, at 1:47 PM, Tu, Jiannan wrote: > > Barry, > > Thank you very much. > > DMDASetBlockFills() needs two input arrays specifying sparsity pattern. In my case, I have 26 components at each grid. Is it correct that I need to have two arrays of 26x26 elements with value 1 for coupling and 0 for no coupling? Exactly > Also how to determine when the value should be 1 or 0? That is problem dependent. Given paper-and-pencil you know for your stencil which components are coupled with other components at each grid point and which components are coupled with components at neighboring grid points. Note the second array contains information about all the neighboring grid points. Above, below, right, left, front back. Normally the coupling is symmetric so the nonzero structures of those 6 blocks are identical. If, for you particular problem, the nonzero structures of the six are not identical you use the union of the nonzero structures of the six of them. 2d example say you have x,y,z at each grid point and the inter-point coupling is x is connected to y and y is connect to x then the first array is [ 1 1 0 ; 1 1; 0 ; 0 0 1]; the zeros are because x is not coupled to z and y is not coupled to z. For intra grid points, say x is coupled to z and itself, z is coupled to x and itself and y is not coupled to itself then [1 0 1; 0 0 0; 1 0 1]; the middle row is zero because y is not coupled to anything and the zeros in the first and last are because x and z are not coupled to y. > Each component involves 7 stencils (i-1, j, k), (I, j, k), (i+1, j, k), (I, j-1, k), (I, j+1, k), (I, j, k-1), and (I, j, k+1), and couples with several other components. > > Jiannan > > > > Sent from Mail for Windows > > From: Barry Smith > Sent: Wednesday, September 7, 2022 11:53 AM > To: Tu, Jiannan > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Using matrix-free with KSP > > CAUTION: This email was sent from outside the UMass Lowell network. > > > DMDASetBlockFills() or DMDASetBlockFillsSparse() (they are just different ways of providing the same information) will help you here enormously, your type of case is exactly what they were designed for. > > Barry > > > > On Sep 7, 2022, at 10:47 AM, Tu, Jiannan > wrote: > > Barry and Hong, > > Thank you. > > There are 26 components at each grid and there are not fully coupled in terms of stiff functions. Mutual coupling is among about 6 components. > > I would prefer not using matrix-free since the Jacobina is not difficult to calculate and only up to 10 non-zeros at each row. I'll try DMDASetBlockFills() or DMDASetBlockFillsSparse() and see how they can reduce the memory usage. > > Jiannan > > From: Barry Smith > > Sent: Tuesday, September 6, 2022 11:33 PM > To: Tu, Jiannan > > Cc: petsc-users at mcs.anl.gov > > Subject: Re: [petsc-users] Using matrix-free with KSP > > CAUTION: This email was sent from outside the UMass Lowell network. > > > > > On Sep 6, 2022, at 11:00 PM, Tu, Jiannan > wrote: > > I am using TS IMEX to solve a large DAE system. The DAE is obtained by applying finite FV method to 3-D multi-specie ion/neutral fluids equations with magnetic induction equation. The Jacobian for stiff part is formed by using MatSetValuesStencil(). The Jacobian matrix is very sparse, no more than 10 non-zeros on each row. MatSetValuesStencil requires local to global mapping by calling ISLocaltoGlobalMapping(). Could you please instruct me how to use local to global mapping? > > DMDA automatically sets up the ISLocaltoGlobalMapping() so you should not need to. > > I also tried using DMCreateMatrix() to create the Jacobian. While local to global mapping is not necessary, the matrix takes too much memory and requires 64-bit indices. I would prefer to take the advantage of sparsity of the Jacobian, pre-allocate the matrix to use as less as possible memory so that the code can be run on a multi-core desktop. > > If using matrix-free does not work for you because the linear solves do not converge or converge too slowly. Then you might be able to decrease the memory used in the matrix. The DMCreateMatrix() does take advantage of sparsity and tries to preallocate only what it needs. Often what it preallocates is the best possible, but for multicomponent problems it has to assume there is full coupling within all the degrees of freedom that live at each at each grid point. How many components live at each grid point in your model and are they not all coupled to each other in the equations? If they are not fully coupled you can use either DMDASetBlockFills() or DMDASetBlockFillsSparse() to indicate the reduced coupling that actually exists for you model. > > Barry > > > > Thank you very much for your advice. > > Jiannan -------------- next part -------------- An HTML attachment was scrubbed... URL: From heepark at sandia.gov Thu Sep 8 19:38:12 2022 From: heepark at sandia.gov (Park, Heeho) Date: Fri, 9 Sep 2022 00:38:12 +0000 Subject: [petsc-users] Make error --download-hdf5-fortran-bindings=1 Message-ID: Hi PETSc Developers, I am having trouble compiling with the --download-hdf5-fortran-bindings=yes option on Sandia?s HPC system. It compiles with make version 3.82, but fails to compile with make version 4.2.1 in the environment. The issues I?m having redirecting to make version 3.82 while having the more recent versions of gnu compilers like gnu/10.2.1 Is there a way around this? By the way, adding --download-make=1 did not help Attached is the configure log. Thank you for your help. heepark at boca1:/projects/ADSM/software/petsc-v3.17.2$ make -v GNU Make 4.2.1 Built for x86_64-redhat-linux-gnu Copyright (C) 1988-2016 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. heepark at boca1:/projects/ADSM/software/petsc-v3.17.2$ ./run_configure_gnu_O.sh ============================================================================================= Configuring PETSc to compile on your system ============================================================================================= ============================================================================================= Running configure on HDF5; this may take several minutes ============================================================================================= ============================================================================================= Running make on HDF5; this may take several minutes ============================================================================================= ******************************************************************************* UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): ------------------------------------------------------------------------------- Error running make; make install on HDF5 Best, Heeho D. Park, Ph.D. Computational Scientist & Engineer Center for Energy & Earth Systems Applied Systems Analysis and Research Dept. Sandia National Laboratories Email: heepark at sandia.gov Web: Homepage -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: configure.log Type: application/octet-stream Size: 3687170 bytes Desc: configure.log URL: From bsmith at petsc.dev Thu Sep 8 21:05:42 2022 From: bsmith at petsc.dev (Barry Smith) Date: Thu, 8 Sep 2022 22:05:42 -0400 Subject: [petsc-users] Make error --download-hdf5-fortran-bindings=1 In-Reply-To: References: Message-ID: <5A9F927A-9FC9-422B-8DE4-955E85B8C68C@petsc.dev> > On Sep 8, 2022, at 8:38 PM, Park, Heeho via petsc-users wrote: > > Hi PETSc Developers, > > I am having trouble compiling with the --download-hdf5-fortran-bindings=yes option on Sandia?s HPC system. > It compiles with make version 3.82, but fails to compile with make version 4.2.1 in the environment. > The issues I?m having redirecting to make version 3.82 while having the more recent versions of gnu compilers like gnu/10.2.1 ptest.f90:20:7: Fatal Error: Cannot read module file ?/opt/openmpi/4.1/gnu/lib/mpi.mod? opened at (1), because it was created by a different version of GNU Fortran compilation terminated. The MPI (Fortran side) cannot be used with the GFortran version you are trying to use. You need to have an MPI install compatible with the GNU Fortran version you want to use. I suggest showing this error to your sys-admins and asking for help in determining what MPI and Fortran compilers to use to get what you need. > Is there a way around this? By the way, adding --download-make=1 did not help > Attached is the configure log. Thank you for your help. > > heepark at boca1:/projects/ADSM/software/petsc-v3.17.2$ make -v > GNU Make 4.2.1 > Built for x86_64-redhat-linux-gnu > Copyright (C) 1988-2016 Free Software Foundation, Inc. > License GPLv3+: GNU GPL version 3 or later > > This is free software: you are free to change and redistribute it. > There is NO WARRANTY, to the extent permitted by law. > > heepark at boca1:/projects/ADSM/software/petsc-v3.17.2$ ./run_configure_gnu_O.sh > ============================================================================================= > Configuring PETSc to compile on your system > ============================================================================================= > ============================================================================================= > Running configure on HDF5; this may take several minutes > ============================================================================================= > ============================================================================================= > Running make on HDF5; this may take several minutes > ============================================================================================= > ******************************************************************************* > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): > ------------------------------------------------------------------------------- > Error running make; make install on HDF5 > > > Best, > > Heeho D. Park, Ph.D. > Computational Scientist & Engineer > Center for Energy & Earth Systems > Applied Systems Analysis and Research Dept. > Sandia National Laboratories > Email: heepark at sandia.gov > Web: Homepage > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From patrick.alken at geomag.info Fri Sep 9 01:09:50 2022 From: patrick.alken at geomag.info (Patrick Alken) Date: Fri, 9 Sep 2022 00:09:50 -0600 Subject: [petsc-users] [slepc] nan output for eigenvectors In-Reply-To: References: Message-ID: <540f0d7f-fae6-5fc2-1aec-f1cfd8de08f6@geomag.info> I have attached a minimal example, although it does have a dependency on the GSL library (sorry for this). The script mwe.sh shows how I am running the program with the various command line arguments. The program will write the computed eigenvectors to an output file called 'evec' in GNU octave / matlab format. I tried running it on 3 different systems with 3 different GCC versions. I am using the latest petsc downloaded from git, and SLEPc 3.17.2. Here are the results: GCC 7.5.0: produces nan values in the eigenvector output file, as well as extremely large and small values. In addition, the petsc/slepc 'make check' tests fail on this system. GCC 10.2.0: produces nan values in the eigenvector output file, as well as extremely large and small values. On this system, the petsc/slepc 'make check' tests pass. GCC 11.2.0: does not produce nan values, and the eigenvector values are all between about -0.01 and +0.01. On this system, the petsc/slepc 'make check' tests pass. At first I thought it was a compiler bug in the old GCC 7.5 version, but 10.2 also seems to have the same problem. I need to do more testing with 11.2. Any advice is appreciated! Patrick On 9/7/22 01:20, Jose E. Roman wrote: > >> El 7 sept 2022, a las 6:18, Patrick Alken escribi?: >> >> I sometimes get Nan output values in computed eigenvectors for the generalized symmetric eigenvalue problem produced by slepc. Is this a known issue, and is it related to the conditioning of the matrix pair (A,B)? Is there some way to compute a "condition number" of the matrix pair ahead of time to see if i have a good chance of getting stable eigenvectors out? > You should never get NaN. Can you send a reproducible example? > >> In a possibly related issue, i am finding that petsc/slepc compiled with debugging vs optimization can produce very different eigenvectors for the same problem, while the eigenvalues are the same. The eigenvectors seem more accurate when I use the debugging version of the libraries. Could this be also a conditioning problem with the matrix pair? > What do you mean more accurate? The residual norm computed with EPSComputeError() should be below the tolerance in both debugging and optimized versions. > > Jose > -------------- next part -------------- default: mwe include ${SLEPC_DIR}/lib/slepc/conf/slepc_common OBJECTS = mwe.o mwe: ${OBJECTS} ${CLINKER} -o mwe ${OBJECTS} ${SLEPC_EPS_LIB} -lgsl -lgslcblas -lm -------------- next part -------------- A non-text attachment was scrubbed... Name: mwe.c Type: text/x-csrc Size: 11082 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: mwe.sh Type: application/x-shellscript Size: 146 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: oct.c Type: text/x-csrc Size: 11933 bytes Desc: not available URL: From jroman at dsic.upv.es Fri Sep 9 02:21:47 2022 From: jroman at dsic.upv.es (Jose E. Roman) Date: Fri, 9 Sep 2022 09:21:47 +0200 Subject: [petsc-users] [slepc] nan output for eigenvectors In-Reply-To: <540f0d7f-fae6-5fc2-1aec-f1cfd8de08f6@geomag.info> References: <540f0d7f-fae6-5fc2-1aec-f1cfd8de08f6@geomag.info> Message-ID: <11CDEDB1-805A-418E-B8E0-5869E78D524B@dsic.upv.es> First, I would suggest that you add preallocation for SeqAIJ matrices, in this way you can run the program with 1 process: MatSeqAIJSetPreallocation(A, 0, d_nnz); MatSeqAIJSetPreallocation(B, 0, d_nnz); With 1 process, the output files are correct. But with more than one process the eigenvector file is wrong because you are writing only the local part of the vector at process 0, and the rest of values are uninitialized - this is probably causing the NaN's. To write a parallel vector to a file, you have to first gather the local parts in a sequential vector, see for instance VecScatterCreateToZero(). An easier way is to use PETSc/SLEPc functionality for writing the EPS solution (see section 2.5.4 of the SLEPc manual). For instance, the option -eps_view_vectors :evecs.m:ascii_matlab will write a script evecs.m that you can run in Matlab to load the vectors. You can also write it in binary format -eps_view_vectors binary:evecs.bin and then load the file with PetscBinaryRead.m (from $PETSC_DIR/share/petsc/matlab). Jose > El 9 sept 2022, a las 8:09, Patrick Alken escribi?: > > I have attached a minimal example, although it does have a dependency on the GSL library (sorry for this). The script mwe.sh shows how I am running the program with the various command line arguments. The program will write the computed eigenvectors to an output file called 'evec' in GNU octave / matlab format. > > I tried running it on 3 different systems with 3 different GCC versions. I am using the latest petsc downloaded from git, and SLEPc 3.17.2. Here are the results: > > GCC 7.5.0: produces nan values in the eigenvector output file, as well as extremely large and small values. In addition, the petsc/slepc 'make check' tests fail on this system. > > GCC 10.2.0: produces nan values in the eigenvector output file, as well as extremely large and small values. On this system, the petsc/slepc 'make check' tests pass. > > GCC 11.2.0: does not produce nan values, and the eigenvector values are all between about -0.01 and +0.01. On this system, the petsc/slepc 'make check' tests pass. > > At first I thought it was a compiler bug in the old GCC 7.5 version, but 10.2 also seems to have the same problem. I need to do more testing with 11.2. > > Any advice is appreciated! > > Patrick > > On 9/7/22 01:20, Jose E. Roman wrote: >> >>> El 7 sept 2022, a las 6:18, Patrick Alken escribi?: >>> >>> I sometimes get Nan output values in computed eigenvectors for the generalized symmetric eigenvalue problem produced by slepc. Is this a known issue, and is it related to the conditioning of the matrix pair (A,B)? Is there some way to compute a "condition number" of the matrix pair ahead of time to see if i have a good chance of getting stable eigenvectors out? >> You should never get NaN. Can you send a reproducible example? >> >>> In a possibly related issue, i am finding that petsc/slepc compiled with debugging vs optimization can produce very different eigenvectors for the same problem, while the eigenvalues are the same. The eigenvectors seem more accurate when I use the debugging version of the libraries. Could this be also a conditioning problem with the matrix pair? >> What do you mean more accurate? The residual norm computed with EPSComputeError() should be below the tolerance in both debugging and optimized versions. >> >> Jose > From quentin.chevalier at polytechnique.edu Fri Sep 9 08:54:55 2022 From: quentin.chevalier at polytechnique.edu (Quentin Chevalier) Date: Fri, 9 Sep 2022 15:54:55 +0200 Subject: [petsc-users] Read and write HDF5 files In-Reply-To: <87pmg6rpa7.fsf@jedbrown.org> References: <87pmg6rpa7.fsf@jedbrown.org> Message-ID: Thank you for the speedy answer ! Quentin CHEVALIER ? IA parcours recherche LadHyX - Ecole polytechnique __________ On Thu, 8 Sept 2022 at 15:04, Jed Brown wrote: > > It sounds like the PETSc inside your container was not built with --download-hdf5 (or --with-hdf5). That can be fixed by updating the Dockerfile and rebuilding the image. > > Quentin Chevalier writes: > > > Hello PETSc users ! > > > > I'm trying to use PETSc routine to read and write complex vectors to binary > > files in parallel. > > > > I'm using dolfinx and petsc4py inside a docker container. My end goal is to > > write a dolfinx Function object to file, then read it again. > > > > My current code goes something like : > > viewer = pet.Viewer().createHDF5("file.h5", 'w', comm) > > function.vector.view(viewer) > > > > And fails with traceback : > > petsc4py.PETSc.Error: error code 86 > > [9] PetscViewerSetType() at > > /usr/local/petsc/src/sys/classes/viewer/interface/viewreg.c:435 > > [9] Unknown type. Check for miss-spelling or missing package: > > https://petsc.org/release/install/install/#external-packages > > [9] Unknown PetscViewer type given: hdf5 > > > > The relevant functions show on pydoc, what am I doing wrong ? > > > > Cheers, > > > > Quentin From turane at gmail.com Fri Sep 9 06:56:24 2022 From: turane at gmail.com (erhan turan) Date: Fri, 9 Sep 2022 14:56:24 +0300 Subject: [petsc-users] multiple definition of `main' for a fortran example Message-ID: Hello, I recently installed the latest version of Petsc and tried to build a case with two fortran files (One main and one subroutine). I looked into similar examples and found one in petsc/src/dm/tutorials: ex13f90.F90. This code also needs a second file called ex13f90aux.F90 When I tried to build this example with "make ex13f90", I receive this error warning: src/dm/tutorials/ex13f90.F90:24: multiple definition of `main' src/dm/tutorials/ex13f90.F90:24: first defined here Hence, I was unable to run an example case from the tutorials folder. I tested in two different pc's, one with Ubuntu 16 and one with Ubuntu 18. Both cases use gnu compilers. Also test both mpich and openmpi. Are there anyone that saw a similar problem who knows a workaround to resolve this issue? Maybe a makefile example with multiple fortran files would be useful. Thanks. Best, Erhan -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Fri Sep 9 10:53:38 2022 From: balay at mcs.anl.gov (Satish Balay) Date: Fri, 9 Sep 2022 10:53:38 -0500 (CDT) Subject: [petsc-users] multiple definition of `main' for a fortran example In-Reply-To: References: Message-ID: <8a981f15-47e-d5a6-7f79-ae9b85f235e7@mcs.anl.gov> Looks like this makefile has a bug. Try the following change: diff --git a/src/dm/tutorials/makefile b/src/dm/tutorials/makefile index d169ab13df4..53bdf48284c 100644 --- a/src/dm/tutorials/makefile +++ b/src/dm/tutorials/makefile @@ -14,5 +14,4 @@ include ${PETSC_DIR}/lib/petsc/conf/variables include ${PETSC_DIR}/lib/petsc/conf/rules include ${PETSC_DIR}/lib/petsc/conf/test -ex13f90.o: ex13f90aux.o -ex13f90: ex13f90.o ex13f90aux.o +ex13f90: ex13f90aux.o Satish On Fri, 9 Sep 2022, erhan turan wrote: > Hello, > > I recently installed the latest version of Petsc and tried to build a case > with two fortran files (One main and one subroutine). I looked into similar > examples and found one in petsc/src/dm/tutorials: ex13f90.F90. This code > also needs a second file called ex13f90aux.F90 > > When I tried to build this example with "make ex13f90", I receive this > error warning: > > src/dm/tutorials/ex13f90.F90:24: multiple definition of `main' > src/dm/tutorials/ex13f90.F90:24: first defined here > > Hence, I was unable to run an example case from the tutorials folder. I > tested in two different pc's, one with Ubuntu 16 and one with Ubuntu 18. > Both cases use gnu compilers. Also test both mpich and openmpi. > > Are there anyone that saw a similar problem who knows a workaround to > resolve this issue? Maybe a makefile example with multiple fortran files > would be useful. Thanks. > > Best, > Erhan > From patrick.alken at geomag.info Fri Sep 9 11:50:32 2022 From: patrick.alken at geomag.info (Patrick Alken) Date: Fri, 9 Sep 2022 10:50:32 -0600 Subject: [petsc-users] [slepc] nan output for eigenvectors In-Reply-To: <11CDEDB1-805A-418E-B8E0-5869E78D524B@dsic.upv.es> References: <540f0d7f-fae6-5fc2-1aec-f1cfd8de08f6@geomag.info> <11CDEDB1-805A-418E-B8E0-5869E78D524B@dsic.upv.es> Message-ID: Jose, ? Thank you so much for looking at my code and suggesting the VecScatterCreateToZero() idea. This has fixed the nan problem, and my eigenvectors are looking much better now. Best regards, Patrick On 9/9/22 01:21, Jose E. Roman wrote: > First, I would suggest that you add preallocation for SeqAIJ matrices, in this way you can run the program with 1 process: > MatSeqAIJSetPreallocation(A, 0, d_nnz); > MatSeqAIJSetPreallocation(B, 0, d_nnz); > > With 1 process, the output files are correct. But with more than one process the eigenvector file is wrong because you are writing only the local part of the vector at process 0, and the rest of values are uninitialized - this is probably causing the NaN's. > > To write a parallel vector to a file, you have to first gather the local parts in a sequential vector, see for instance VecScatterCreateToZero(). An easier way is to use PETSc/SLEPc functionality for writing the EPS solution (see section 2.5.4 of the SLEPc manual). For instance, the option > -eps_view_vectors :evecs.m:ascii_matlab > will write a script evecs.m that you can run in Matlab to load the vectors. You can also write it in binary format > -eps_view_vectors binary:evecs.bin > and then load the file with PetscBinaryRead.m (from $PETSC_DIR/share/petsc/matlab). > > Jose > > >> El 9 sept 2022, a las 8:09, Patrick Alken escribi?: >> >> I have attached a minimal example, although it does have a dependency on the GSL library (sorry for this). The script mwe.sh shows how I am running the program with the various command line arguments. The program will write the computed eigenvectors to an output file called 'evec' in GNU octave / matlab format. >> >> I tried running it on 3 different systems with 3 different GCC versions. I am using the latest petsc downloaded from git, and SLEPc 3.17.2. Here are the results: >> >> GCC 7.5.0: produces nan values in the eigenvector output file, as well as extremely large and small values. In addition, the petsc/slepc 'make check' tests fail on this system. >> >> GCC 10.2.0: produces nan values in the eigenvector output file, as well as extremely large and small values. On this system, the petsc/slepc 'make check' tests pass. >> >> GCC 11.2.0: does not produce nan values, and the eigenvector values are all between about -0.01 and +0.01. On this system, the petsc/slepc 'make check' tests pass. >> >> At first I thought it was a compiler bug in the old GCC 7.5 version, but 10.2 also seems to have the same problem. I need to do more testing with 11.2. >> >> Any advice is appreciated! >> >> Patrick >> >> On 9/7/22 01:20, Jose E. Roman wrote: >>>> El 7 sept 2022, a las 6:18, Patrick Alken escribi?: >>>> >>>> I sometimes get Nan output values in computed eigenvectors for the generalized symmetric eigenvalue problem produced by slepc. Is this a known issue, and is it related to the conditioning of the matrix pair (A,B)? Is there some way to compute a "condition number" of the matrix pair ahead of time to see if i have a good chance of getting stable eigenvectors out? >>> You should never get NaN. Can you send a reproducible example? >>> >>>> In a possibly related issue, i am finding that petsc/slepc compiled with debugging vs optimization can produce very different eigenvectors for the same problem, while the eigenvalues are the same. The eigenvectors seem more accurate when I use the debugging version of the libraries. Could this be also a conditioning problem with the matrix pair? >>> What do you mean more accurate? The residual norm computed with EPSComputeError() should be below the tolerance in both debugging and optimized versions. >>> >>> Jose >> From bsmith at petsc.dev Sat Sep 10 12:09:50 2022 From: bsmith at petsc.dev (Barry Smith) Date: Sat, 10 Sep 2022 13:09:50 -0400 Subject: [petsc-users] Using matrix-free with KSP In-Reply-To: References: <31AF5BF6-8AC4-4A76-9235-9C0E1BB3EE16@petsc.dev> <7F5F237F-B73B-447E-A113-B46C697D4C37@petsc.dev> <79BC638A-1064-42A1-AE04-40ACAA82323A@petsc.dev> <12206964-1F03-45CE-9FDB-EC4B244CD1F8@petsc.dev> <15DD8C9E-C3A5-4758-9A1F-B9FA28B35825@petsc.dev> Message-ID: If you are using DMCreateMatrix() then MatMPIAIJSetPreallocation is not needed, it is automatically handled by the DM. It sounds like that the values you set in DMDASetBlockFills() did not include all the coupling, hence the preallocation was wrong, hence the time to insert values was too long. Barry > On Sep 10, 2022, at 8:16 AM, Tu, Jiannan wrote: > > Just found MatMPIAIJSetPreallocation is needed. Now Jacobin insertion and assembly is fast. > > From: Barry Smith > > Sent: Thursday, September 8, 2022 9:53 PM > To: Tu, Jiannan > > Subject: Re: [petsc-users] Using matrix-free with KSP > > CAUTION: This email was sent from outside the UMass Lowell network. > > > > > On Sep 8, 2022, at 8:34 PM, Tu, Jiannan > wrote: > > Barry, > > Thank you. > > I set up these two arrays according to the non-zero pattern of the Jacobian. The DMDASetBlockFills() is then called after the call to DMCreateMatrix(da, &A). > > Absolutely not! It must be called before DMCreateMatrix() so that the DM knows how to construct exactly the matrix you need. > > > > Nevertheless, the program execution is still killed with exit code 9, which I assume is due to overuse of the memory. My desktop machine has 32 GB RAM. > > I ran the code with a smaller mesh, 100x45x90, the result is the same. With that number of grids, 26 unknowns, and up to 10 non-zero at each matrix row, the estimated memory to store 100x45x90x26x10 elements should be less than 1 GB (double precision). I am wondering why the matrix still takes too much memory. Maybe I have to use matrix-free? > > Thank you, > Jiannan > > From: Barry Smith > > Sent: Thursday, September 8, 2022 4:19 PM > To: Tu, Jiannan > > Subject: Re: [petsc-users] Using matrix-free with KSP > > CAUTION: This email was sent from outside the UMass Lowell network. > > > > > > On Sep 8, 2022, at 3:39 PM, Tu, Jiannan > wrote: > > Barry, > > Thank you very much for the detailed description. > > So the first array, dfill, is for intra grids and the second one, ofill is for the inter-grid coupling? > > In the case for inter-grid coupling, a component is only coupled to itself, say x in your example is coupled to itself on above, below, right, left, front, and back, how to set values for the second array? > > Then the off-diagonal one would just have values on its diagonal. > > > > Jiannan > > From: Barry Smith > Sent: Thursday, September 8, 2022 2:12 PM > To: Tu, Jiannan > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Using matrix-free with KSP > > CAUTION: This email was sent from outside the UMass Lowell network. > > > > > > > On Sep 8, 2022, at 1:47 PM, Tu, Jiannan > wrote: > > Barry, > > Thank you very much. > > DMDASetBlockFills() needs two input arrays specifying sparsity pattern. In my case, I have 26 components at each grid. Is it correct that I need to have two arrays of 26x26 elements with value 1 for coupling and 0 for no coupling? > > Exactly > > > > > Also how to determine when the value should be 1 or 0? > > That is problem dependent. Given paper-and-pencil you know for your stencil which components are coupled with other components at each grid point and which components are coupled with components at neighboring grid points. > > Note the second array contains information about all the neighboring grid points. Above, below, right, left, front back. Normally the coupling is symmetric so the nonzero structures of those 6 blocks are identical. If, for you particular problem, the nonzero structures of the six are not identical you use the union of the nonzero structures of the six of them. > > > 2d example say you have x,y,z at each grid point and the inter-point coupling is x is connected to y and y is connect to x then the first array is > > [ 1 1 0 ; 1 1; 0 ; 0 0 1]; the zeros are because x is not coupled to z and y is not coupled to z. > > For intra grid points, say x is coupled to z and itself, z is coupled to x and itself and y is not coupled to itself then > > [1 0 1; 0 0 0; 1 0 1]; the middle row is zero because y is not coupled to anything and the zeros in the first and last are because x and z are not coupled to y. > > > > > > > Each component involves 7 stencils (i-1, j, k), (I, j, k), (i+1, j, k), (I, j-1, k), (I, j+1, k), (I, j, k-1), and (I, j, k+1), and couples with several other components. > > Jiannan > > > > Sent from Mail for Windows > > From: Barry Smith > Sent: Wednesday, September 7, 2022 11:53 AM > To: Tu, Jiannan > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Using matrix-free with KSP > > CAUTION: This email was sent from outside the UMass Lowell network. > > > DMDASetBlockFills() or DMDASetBlockFillsSparse() (they are just different ways of providing the same information) will help you here enormously, your type of case is exactly what they were designed for. > > Barry > > > > > > > On Sep 7, 2022, at 10:47 AM, Tu, Jiannan > wrote: > > Barry and Hong, > > Thank you. > > There are 26 components at each grid and there are not fully coupled in terms of stiff functions. Mutual coupling is among about 6 components. > > I would prefer not using matrix-free since the Jacobina is not difficult to calculate and only up to 10 non-zeros at each row. I'll try DMDASetBlockFills() or DMDASetBlockFillsSparse() and see how they can reduce the memory usage. > > Jiannan > > From: Barry Smith > > Sent: Tuesday, September 6, 2022 11:33 PM > To: Tu, Jiannan > > Cc: petsc-users at mcs.anl.gov > > Subject: Re: [petsc-users] Using matrix-free with KSP > > CAUTION: This email was sent from outside the UMass Lowell network. > > > > > > > > On Sep 6, 2022, at 11:00 PM, Tu, Jiannan > wrote: > > I am using TS IMEX to solve a large DAE system. The DAE is obtained by applying finite FV method to 3-D multi-specie ion/neutral fluids equations with magnetic induction equation. The Jacobian for stiff part is formed by using MatSetValuesStencil(). The Jacobian matrix is very sparse, no more than 10 non-zeros on each row. MatSetValuesStencil requires local to global mapping by calling ISLocaltoGlobalMapping(). Could you please instruct me how to use local to global mapping? > > DMDA automatically sets up the ISLocaltoGlobalMapping() so you should not need to. > > I also tried using DMCreateMatrix() to create the Jacobian. While local to global mapping is not necessary, the matrix takes too much memory and requires 64-bit indices. I would prefer to take the advantage of sparsity of the Jacobian, pre-allocate the matrix to use as less as possible memory so that the code can be run on a multi-core desktop. > > If using matrix-free does not work for you because the linear solves do not converge or converge too slowly. Then you might be able to decrease the memory used in the matrix. The DMCreateMatrix() does take advantage of sparsity and tries to preallocate only what it needs. Often what it preallocates is the best possible, but for multicomponent problems it has to assume there is full coupling within all the degrees of freedom that live at each at each grid point. How many components live at each grid point in your model and are they not all coupled to each other in the equations? If they are not fully coupled you can use either DMDASetBlockFills() or DMDASetBlockFillsSparse() to indicate the reduced coupling that actually exists for you model. > > Barry > > > > > > > Thank you very much for your advice. > > Jiannan -------------- next part -------------- An HTML attachment was scrubbed... URL: From turane at gmail.com Sat Sep 10 14:15:01 2022 From: turane at gmail.com (erhan turan) Date: Sat, 10 Sep 2022 19:15:01 +0000 Subject: [petsc-users] multiple definition of `main' for a fortran example In-Reply-To: <8a981f15-47e-d5a6-7f79-ae9b85f235e7@mcs.anl.gov> References: <8a981f15-47e-d5a6-7f79-ae9b85f235e7@mcs.anl.gov> Message-ID: Hi Satish, Fix works. Thanks! Erhan Erhan ________________________________ From: Satish Balay Sent: Friday, September 9, 2022 6:53:38 PM To: erhan turan Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] multiple definition of `main' for a fortran example Looks like this makefile has a bug. Try the following change: diff --git a/src/dm/tutorials/makefile b/src/dm/tutorials/makefile index d169ab13df4..53bdf48284c 100644 --- a/src/dm/tutorials/makefile +++ b/src/dm/tutorials/makefile @@ -14,5 +14,4 @@ include ${PETSC_DIR}/lib/petsc/conf/variables include ${PETSC_DIR}/lib/petsc/conf/rules include ${PETSC_DIR}/lib/petsc/conf/test -ex13f90.o: ex13f90aux.o -ex13f90: ex13f90.o ex13f90aux.o +ex13f90: ex13f90aux.o Satish On Fri, 9 Sep 2022, erhan turan wrote: > Hello, > > I recently installed the latest version of Petsc and tried to build a case > with two fortran files (One main and one subroutine). I looked into similar > examples and found one in petsc/src/dm/tutorials: ex13f90.F90. This code > also needs a second file called ex13f90aux.F90 > > When I tried to build this example with "make ex13f90", I receive this > error warning: > > src/dm/tutorials/ex13f90.F90:24: multiple definition of `main' > src/dm/tutorials/ex13f90.F90:24: first defined here > > Hence, I was unable to run an example case from the tutorials folder. I > tested in two different pc's, one with Ubuntu 16 and one with Ubuntu 18. > Both cases use gnu compilers. Also test both mpich and openmpi. > > Are there anyone that saw a similar problem who knows a workaround to > resolve this issue? Maybe a makefile example with multiple fortran files > would be useful. Thanks. > > Best, > Erhan > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Jiannan_Tu at uml.edu Sat Sep 10 21:14:19 2022 From: Jiannan_Tu at uml.edu (Tu, Jiannan) Date: Sun, 11 Sep 2022 02:14:19 +0000 Subject: [petsc-users] Using matrix-free with KSP In-Reply-To: References: <31AF5BF6-8AC4-4A76-9235-9C0E1BB3EE16@petsc.dev> <7F5F237F-B73B-447E-A113-B46C697D4C37@petsc.dev> <79BC638A-1064-42A1-AE04-40ACAA82323A@petsc.dev> <12206964-1F03-45CE-9FDB-EC4B244CD1F8@petsc.dev> <15DD8C9E-C3A5-4758-9A1F-B9FA28B35825@petsc.dev> Message-ID: Barry, I think I understand how to set values for the intra-grid coupling. But I am not clear about the way to set values for inter-grid coupling. When a component couples to itself at above, below, right, left, front and back grids, diagonal is set as 1 for the second array of DMDASetBlockFills(). How about if involving more grids? How does DMDA know where to allocate memory? In my case, a component couples to itself at the other end of the grid because of periodic boundary condition. There is an error message "Augment out of range . Insert new nonzero at global row/column (26, 520526) into matrix" because of this far away coupling. The column index 520526 is related to the other end of the grid in azimuthal direction. Thanks, Jiannan From: Barry Smith Sent: Saturday, September 10, 2022 1:10 PM To: Tu, Jiannan Cc: petsc-users Subject: Re: [petsc-users] Using matrix-free with KSP CAUTION: This email was sent from outside the UMass Lowell network. If you are using DMCreateMatrix() then MatMPIAIJSetPreallocation is not needed, it is automatically handled by the DM. It sounds like that the values you set in DMDASetBlockFills() did not include all the coupling, hence the preallocation was wrong, hence the time to insert values was too long. Barry On Sep 10, 2022, at 8:16 AM, Tu, Jiannan > wrote: Just found MatMPIAIJSetPreallocation is needed. Now Jacobin insertion and assembly is fast. From: Barry Smith > Sent: Thursday, September 8, 2022 9:53 PM To: Tu, Jiannan > Subject: Re: [petsc-users] Using matrix-free with KSP CAUTION: This email was sent from outside the UMass Lowell network. On Sep 8, 2022, at 8:34 PM, Tu, Jiannan > wrote: Barry, Thank you. I set up these two arrays according to the non-zero pattern of the Jacobian. The DMDASetBlockFills() is then called after the call to DMCreateMatrix(da, &A). Absolutely not! It must be called before DMCreateMatrix() so that the DM knows how to construct exactly the matrix you need. Nevertheless, the program execution is still killed with exit code 9, which I assume is due to overuse of the memory. My desktop machine has 32 GB RAM. I ran the code with a smaller mesh, 100x45x90, the result is the same. With that number of grids, 26 unknowns, and up to 10 non-zero at each matrix row, the estimated memory to store 100x45x90x26x10 elements should be less than 1 GB (double precision). I am wondering why the matrix still takes too much memory. Maybe I have to use matrix-free? Thank you, Jiannan From: Barry Smith > Sent: Thursday, September 8, 2022 4:19 PM To: Tu, Jiannan > Subject: Re: [petsc-users] Using matrix-free with KSP CAUTION: This email was sent from outside the UMass Lowell network. On Sep 8, 2022, at 3:39 PM, Tu, Jiannan > wrote: Barry, Thank you very much for the detailed description. So the first array, dfill, is for intra grids and the second one, ofill is for the inter-grid coupling? In the case for inter-grid coupling, a component is only coupled to itself, say x in your example is coupled to itself on above, below, right, left, front, and back, how to set values for the second array? Then the off-diagonal one would just have values on its diagonal. Jiannan From: Barry Smith Sent: Thursday, September 8, 2022 2:12 PM To: Tu, Jiannan Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] Using matrix-free with KSP CAUTION: This email was sent from outside the UMass Lowell network. On Sep 8, 2022, at 1:47 PM, Tu, Jiannan > wrote: Barry, Thank you very much. DMDASetBlockFills() needs two input arrays specifying sparsity pattern. In my case, I have 26 components at each grid. Is it correct that I need to have two arrays of 26x26 elements with value 1 for coupling and 0 for no coupling? Exactly Also how to determine when the value should be 1 or 0? That is problem dependent. Given paper-and-pencil you know for your stencil which components are coupled with other components at each grid point and which components are coupled with components at neighboring grid points. Note the second array contains information about all the neighboring grid points. Above, below, right, left, front back. Normally the coupling is symmetric so the nonzero structures of those 6 blocks are identical. If, for you particular problem, the nonzero structures of the six are not identical you use the union of the nonzero structures of the six of them. 2d example say you have x,y,z at each grid point and the inter-point coupling is x is connected to y and y is connect to x then the first array is [ 1 1 0 ; 1 1; 0 ; 0 0 1]; the zeros are because x is not coupled to z and y is not coupled to z. For intra grid points, say x is coupled to z and itself, z is coupled to x and itself and y is not coupled to itself then [1 0 1; 0 0 0; 1 0 1]; the middle row is zero because y is not coupled to anything and the zeros in the first and last are because x and z are not coupled to y. Each component involves 7 stencils (i-1, j, k), (I, j, k), (i+1, j, k), (I, j-1, k), (I, j+1, k), (I, j, k-1), and (I, j, k+1), and couples with several other components. Jiannan Sent from Mail for Windows From: Barry Smith Sent: Wednesday, September 7, 2022 11:53 AM To: Tu, Jiannan Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] Using matrix-free with KSP CAUTION: This email was sent from outside the UMass Lowell network. DMDASetBlockFills() or DMDASetBlockFillsSparse() (they are just different ways of providing the same information) will help you here enormously, your type of case is exactly what they were designed for. Barry On Sep 7, 2022, at 10:47 AM, Tu, Jiannan > wrote: Barry and Hong, Thank you. There are 26 components at each grid and there are not fully coupled in terms of stiff functions. Mutual coupling is among about 6 components. I would prefer not using matrix-free since the Jacobina is not difficult to calculate and only up to 10 non-zeros at each row. I'll try DMDASetBlockFills() or DMDASetBlockFillsSparse() and see how they can reduce the memory usage. Jiannan From: Barry Smith > Sent: Tuesday, September 6, 2022 11:33 PM To: Tu, Jiannan > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Using matrix-free with KSP CAUTION: This email was sent from outside the UMass Lowell network. On Sep 6, 2022, at 11:00 PM, Tu, Jiannan > wrote: I am using TS IMEX to solve a large DAE system. The DAE is obtained by applying finite FV method to 3-D multi-specie ion/neutral fluids equations with magnetic induction equation. The Jacobian for stiff part is formed by using MatSetValuesStencil(). The Jacobian matrix is very sparse, no more than 10 non-zeros on each row. MatSetValuesStencil requires local to global mapping by calling ISLocaltoGlobalMapping(). Could you please instruct me how to use local to global mapping? DMDA automatically sets up the ISLocaltoGlobalMapping() so you should not need to. I also tried using DMCreateMatrix() to create the Jacobian. While local to global mapping is not necessary, the matrix takes too much memory and requires 64-bit indices. I would prefer to take the advantage of sparsity of the Jacobian, pre-allocate the matrix to use as less as possible memory so that the code can be run on a multi-core desktop. If using matrix-free does not work for you because the linear solves do not converge or converge too slowly. Then you might be able to decrease the memory used in the matrix. The DMCreateMatrix() does take advantage of sparsity and tries to preallocate only what it needs. Often what it preallocates is the best possible, but for multicomponent problems it has to assume there is full coupling within all the degrees of freedom that live at each at each grid point. How many components live at each grid point in your model and are they not all coupled to each other in the equations? If they are not fully coupled you can use either DMDASetBlockFills() or DMDASetBlockFillsSparse() to indicate the reduced coupling that actually exists for you model. Barry Thank you very much for your advice. Jiannan -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Sat Sep 10 23:09:46 2022 From: bsmith at petsc.dev (Barry Smith) Date: Sun, 11 Sep 2022 00:09:46 -0400 Subject: [petsc-users] Using matrix-free with KSP In-Reply-To: References: <31AF5BF6-8AC4-4A76-9235-9C0E1BB3EE16@petsc.dev> <7F5F237F-B73B-447E-A113-B46C697D4C37@petsc.dev> <79BC638A-1064-42A1-AE04-40ACAA82323A@petsc.dev> <12206964-1F03-45CE-9FDB-EC4B244CD1F8@petsc.dev> <15DD8C9E-C3A5-4758-9A1F-B9FA28B35825@petsc.dev> Message-ID: Have you created the DMDA with the periodic boundary conditions arguments? Or are your boundary conditions not simply periodic? Compare DMCreateMatrix_DA_3d_MPIAIJ() and DMCreateMatrix_DA_3d_MPIAIJ_Fill() in src/dm/impls/da/fdda.c I think they both attempt to handle the periodic case. Barry > On Sep 10, 2022, at 10:14 PM, Tu, Jiannan wrote: > > Barry, > > I think I understand how to set values for the intra-grid coupling. But I am not clear about the way to set values for inter-grid coupling. When a component couples to itself at above, below, right, left, front and back grids, diagonal is set as 1 for the second array of DMDASetBlockFills(). How about if involving more grids? How does DMDA know where to allocate memory? In my case, a component couples to itself at the other end of the grid because of periodic boundary condition. There is an error message ?Augment out of range . Insert new nonzero at global row/column (26, 520526) into matrix? because of this far away coupling. The column index 520526 is related to the other end of the grid in azimuthal direction. > > Thanks, > Jiannan > > > > From: Barry Smith > > Sent: Saturday, September 10, 2022 1:10 PM > To: Tu, Jiannan > > Cc: petsc-users > > Subject: Re: [petsc-users] Using matrix-free with KSP > > CAUTION: This email was sent from outside the UMass Lowell network. > > > If you are using DMCreateMatrix() then MatMPIAIJSetPreallocation is not needed, it is automatically handled by the DM. > > It sounds like that the values you set in DMDASetBlockFills() did not include all the coupling, hence the preallocation was wrong, hence the time to insert values was too long. > > Barry > > > > On Sep 10, 2022, at 8:16 AM, Tu, Jiannan > wrote: > > Just found MatMPIAIJSetPreallocation is needed. Now Jacobin insertion and assembly is fast. > > From: Barry Smith > > Sent: Thursday, September 8, 2022 9:53 PM > To: Tu, Jiannan > > Subject: Re: [petsc-users] Using matrix-free with KSP > > CAUTION: This email was sent from outside the UMass Lowell network. > > > > > > On Sep 8, 2022, at 8:34 PM, Tu, Jiannan > wrote: > > Barry, > > Thank you. > > I set up these two arrays according to the non-zero pattern of the Jacobian. The DMDASetBlockFills() is then called after the call to DMCreateMatrix(da, &A). > > Absolutely not! It must be called before DMCreateMatrix() so that the DM knows how to construct exactly the matrix you need. > > > > > Nevertheless, the program execution is still killed with exit code 9, which I assume is due to overuse of the memory. My desktop machine has 32 GB RAM. > > I ran the code with a smaller mesh, 100x45x90, the result is the same. With that number of grids, 26 unknowns, and up to 10 non-zero at each matrix row, the estimated memory to store 100x45x90x26x10 elements should be less than 1 GB (double precision). I am wondering why the matrix still takes too much memory. Maybe I have to use matrix-free? > > Thank you, > Jiannan > > From: Barry Smith > > Sent: Thursday, September 8, 2022 4:19 PM > To: Tu, Jiannan > > Subject: Re: [petsc-users] Using matrix-free with KSP > > CAUTION: This email was sent from outside the UMass Lowell network. > > > > > > > On Sep 8, 2022, at 3:39 PM, Tu, Jiannan > wrote: > > Barry, > > Thank you very much for the detailed description. > > So the first array, dfill, is for intra grids and the second one, ofill is for the inter-grid coupling? > > In the case for inter-grid coupling, a component is only coupled to itself, say x in your example is coupled to itself on above, below, right, left, front, and back, how to set values for the second array? > > Then the off-diagonal one would just have values on its diagonal. > > > > > Jiannan > > From: Barry Smith > Sent: Thursday, September 8, 2022 2:12 PM > To: Tu, Jiannan > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Using matrix-free with KSP > > CAUTION: This email was sent from outside the UMass Lowell network. > > > > > > > > On Sep 8, 2022, at 1:47 PM, Tu, Jiannan > wrote: > > Barry, > > Thank you very much. > > DMDASetBlockFills() needs two input arrays specifying sparsity pattern. In my case, I have 26 components at each grid. Is it correct that I need to have two arrays of 26x26 elements with value 1 for coupling and 0 for no coupling? > > Exactly > > > > > > Also how to determine when the value should be 1 or 0? > > That is problem dependent. Given paper-and-pencil you know for your stencil which components are coupled with other components at each grid point and which components are coupled with components at neighboring grid points. > > Note the second array contains information about all the neighboring grid points. Above, below, right, left, front back. Normally the coupling is symmetric so the nonzero structures of those 6 blocks are identical. If, for you particular problem, the nonzero structures of the six are not identical you use the union of the nonzero structures of the six of them. > > > 2d example say you have x,y,z at each grid point and the inter-point coupling is x is connected to y and y is connect to x then the first array is > > [ 1 1 0 ; 1 1; 0 ; 0 0 1]; the zeros are because x is not coupled to z and y is not coupled to z. > > For intra grid points, say x is coupled to z and itself, z is coupled to x and itself and y is not coupled to itself then > > [1 0 1; 0 0 0; 1 0 1]; the middle row is zero because y is not coupled to anything and the zeros in the first and last are because x and z are not coupled to y. > > > > > > > > Each component involves 7 stencils (i-1, j, k), (I, j, k), (i+1, j, k), (I, j-1, k), (I, j+1, k), (I, j, k-1), and (I, j, k+1), and couples with several other components. > > Jiannan > > > > Sent from Mail for Windows > > From: Barry Smith > Sent: Wednesday, September 7, 2022 11:53 AM > To: Tu, Jiannan > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Using matrix-free with KSP > > CAUTION: This email was sent from outside the UMass Lowell network. > > > DMDASetBlockFills() or DMDASetBlockFillsSparse() (they are just different ways of providing the same information) will help you here enormously, your type of case is exactly what they were designed for. > > Barry > > > > > > > > On Sep 7, 2022, at 10:47 AM, Tu, Jiannan > wrote: > > Barry and Hong, > > Thank you. > > There are 26 components at each grid and there are not fully coupled in terms of stiff functions. Mutual coupling is among about 6 components. > > I would prefer not using matrix-free since the Jacobina is not difficult to calculate and only up to 10 non-zeros at each row. I'll try DMDASetBlockFills() or DMDASetBlockFillsSparse() and see how they can reduce the memory usage. > > Jiannan > > From: Barry Smith > > Sent: Tuesday, September 6, 2022 11:33 PM > To: Tu, Jiannan > > Cc: petsc-users at mcs.anl.gov > > Subject: Re: [petsc-users] Using matrix-free with KSP > > CAUTION: This email was sent from outside the UMass Lowell network. > > > > > > > > > On Sep 6, 2022, at 11:00 PM, Tu, Jiannan > wrote: > > I am using TS IMEX to solve a large DAE system. The DAE is obtained by applying finite FV method to 3-D multi-specie ion/neutral fluids equations with magnetic induction equation. The Jacobian for stiff part is formed by using MatSetValuesStencil(). The Jacobian matrix is very sparse, no more than 10 non-zeros on each row. MatSetValuesStencil requires local to global mapping by calling ISLocaltoGlobalMapping(). Could you please instruct me how to use local to global mapping? > > DMDA automatically sets up the ISLocaltoGlobalMapping() so you should not need to. > > I also tried using DMCreateMatrix() to create the Jacobian. While local to global mapping is not necessary, the matrix takes too much memory and requires 64-bit indices. I would prefer to take the advantage of sparsity of the Jacobian, pre-allocate the matrix to use as less as possible memory so that the code can be run on a multi-core desktop. > > If using matrix-free does not work for you because the linear solves do not converge or converge too slowly. Then you might be able to decrease the memory used in the matrix. The DMCreateMatrix() does take advantage of sparsity and tries to preallocate only what it needs. Often what it preallocates is the best possible, but for multicomponent problems it has to assume there is full coupling within all the degrees of freedom that live at each at each grid point. How many components live at each grid point in your model and are they not all coupled to each other in the equations? If they are not fully coupled you can use either DMDASetBlockFills() or DMDASetBlockFillsSparse() to indicate the reduced coupling that actually exists for you model. > > Barry > > > > > > > > Thank you very much for your advice. > > Jiannan -------------- next part -------------- An HTML attachment was scrubbed... URL: From Jiannan_Tu at uml.edu Sun Sep 11 08:56:23 2022 From: Jiannan_Tu at uml.edu (Tu, Jiannan) Date: Sun, 11 Sep 2022 13:56:23 +0000 Subject: [petsc-users] Using matrix-free with KSP In-Reply-To: References: <31AF5BF6-8AC4-4A76-9235-9C0E1BB3EE16@petsc.dev> <7F5F237F-B73B-447E-A113-B46C697D4C37@petsc.dev> <79BC638A-1064-42A1-AE04-40ACAA82323A@petsc.dev> <12206964-1F03-45CE-9FDB-EC4B244CD1F8@petsc.dev> <15DD8C9E-C3A5-4758-9A1F-B9FA28B35825@petsc.dev> Message-ID: Barry, Thank you. Yes, the DMDA is created with periodic BC. But it might not be the periodic BC since in azimuthal direction, the ghost grid -1 is actually grid Nmax, and Nmax+1 is grid 0. Still the problem is how to properly pre-allocate matrix. Otherwise either "Augument out of range" error or code stuck at inserting values if MatSetOption() is called. Jiannan From: Barry Smith Sent: Sunday, September 11, 2022 12:10 AM To: Tu, Jiannan Cc: petsc-users Subject: Re: [petsc-users] Using matrix-free with KSP CAUTION: This email was sent from outside the UMass Lowell network. Have you created the DMDA with the periodic boundary conditions arguments? Or are your boundary conditions not simply periodic? Compare DMCreateMatrix_DA_3d_MPIAIJ() and DMCreateMatrix_DA_3d_MPIAIJ_Fill() in src/dm/impls/da/fdda.c I think they both attempt to handle the periodic case. Barry On Sep 10, 2022, at 10:14 PM, Tu, Jiannan > wrote: Barry, I think I understand how to set values for the intra-grid coupling. But I am not clear about the way to set values for inter-grid coupling. When a component couples to itself at above, below, right, left, front and back grids, diagonal is set as 1 for the second array of DMDASetBlockFills(). How about if involving more grids? How does DMDA know where to allocate memory? In my case, a component couples to itself at the other end of the grid because of periodic boundary condition. There is an error message "Augment out of range . Insert new nonzero at global row/column (26, 520526) into matrix" because of this far away coupling. The column index 520526 is related to the other end of the grid in azimuthal direction. Thanks, Jiannan From: Barry Smith > Sent: Saturday, September 10, 2022 1:10 PM To: Tu, Jiannan > Cc: petsc-users > Subject: Re: [petsc-users] Using matrix-free with KSP CAUTION: This email was sent from outside the UMass Lowell network. If you are using DMCreateMatrix() then MatMPIAIJSetPreallocation is not needed, it is automatically handled by the DM. It sounds like that the values you set in DMDASetBlockFills() did not include all the coupling, hence the preallocation was wrong, hence the time to insert values was too long. Barry On Sep 10, 2022, at 8:16 AM, Tu, Jiannan > wrote: Just found MatMPIAIJSetPreallocation is needed. Now Jacobin insertion and assembly is fast. From: Barry Smith > Sent: Thursday, September 8, 2022 9:53 PM To: Tu, Jiannan > Subject: Re: [petsc-users] Using matrix-free with KSP CAUTION: This email was sent from outside the UMass Lowell network. On Sep 8, 2022, at 8:34 PM, Tu, Jiannan > wrote: Barry, Thank you. I set up these two arrays according to the non-zero pattern of the Jacobian. The DMDASetBlockFills() is then called after the call to DMCreateMatrix(da, &A). Absolutely not! It must be called before DMCreateMatrix() so that the DM knows how to construct exactly the matrix you need. Nevertheless, the program execution is still killed with exit code 9, which I assume is due to overuse of the memory. My desktop machine has 32 GB RAM. I ran the code with a smaller mesh, 100x45x90, the result is the same. With that number of grids, 26 unknowns, and up to 10 non-zero at each matrix row, the estimated memory to store 100x45x90x26x10 elements should be less than 1 GB (double precision). I am wondering why the matrix still takes too much memory. Maybe I have to use matrix-free? Thank you, Jiannan From: Barry Smith > Sent: Thursday, September 8, 2022 4:19 PM To: Tu, Jiannan > Subject: Re: [petsc-users] Using matrix-free with KSP CAUTION: This email was sent from outside the UMass Lowell network. On Sep 8, 2022, at 3:39 PM, Tu, Jiannan > wrote: Barry, Thank you very much for the detailed description. So the first array, dfill, is for intra grids and the second one, ofill is for the inter-grid coupling? In the case for inter-grid coupling, a component is only coupled to itself, say x in your example is coupled to itself on above, below, right, left, front, and back, how to set values for the second array? Then the off-diagonal one would just have values on its diagonal. Jiannan From: Barry Smith Sent: Thursday, September 8, 2022 2:12 PM To: Tu, Jiannan Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] Using matrix-free with KSP CAUTION: This email was sent from outside the UMass Lowell network. On Sep 8, 2022, at 1:47 PM, Tu, Jiannan > wrote: Barry, Thank you very much. DMDASetBlockFills() needs two input arrays specifying sparsity pattern. In my case, I have 26 components at each grid. Is it correct that I need to have two arrays of 26x26 elements with value 1 for coupling and 0 for no coupling? Exactly Also how to determine when the value should be 1 or 0? That is problem dependent. Given paper-and-pencil you know for your stencil which components are coupled with other components at each grid point and which components are coupled with components at neighboring grid points. Note the second array contains information about all the neighboring grid points. Above, below, right, left, front back. Normally the coupling is symmetric so the nonzero structures of those 6 blocks are identical. If, for you particular problem, the nonzero structures of the six are not identical you use the union of the nonzero structures of the six of them. 2d example say you have x,y,z at each grid point and the inter-point coupling is x is connected to y and y is connect to x then the first array is [ 1 1 0 ; 1 1; 0 ; 0 0 1]; the zeros are because x is not coupled to z and y is not coupled to z. For intra grid points, say x is coupled to z and itself, z is coupled to x and itself and y is not coupled to itself then [1 0 1; 0 0 0; 1 0 1]; the middle row is zero because y is not coupled to anything and the zeros in the first and last are because x and z are not coupled to y. Each component involves 7 stencils (i-1, j, k), (I, j, k), (i+1, j, k), (I, j-1, k), (I, j+1, k), (I, j, k-1), and (I, j, k+1), and couples with several other components. Jiannan Sent from Mail for Windows From: Barry Smith Sent: Wednesday, September 7, 2022 11:53 AM To: Tu, Jiannan Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] Using matrix-free with KSP CAUTION: This email was sent from outside the UMass Lowell network. DMDASetBlockFills() or DMDASetBlockFillsSparse() (they are just different ways of providing the same information) will help you here enormously, your type of case is exactly what they were designed for. Barry On Sep 7, 2022, at 10:47 AM, Tu, Jiannan > wrote: Barry and Hong, Thank you. There are 26 components at each grid and there are not fully coupled in terms of stiff functions. Mutual coupling is among about 6 components. I would prefer not using matrix-free since the Jacobina is not difficult to calculate and only up to 10 non-zeros at each row. I'll try DMDASetBlockFills() or DMDASetBlockFillsSparse() and see how they can reduce the memory usage. Jiannan From: Barry Smith > Sent: Tuesday, September 6, 2022 11:33 PM To: Tu, Jiannan > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Using matrix-free with KSP CAUTION: This email was sent from outside the UMass Lowell network. On Sep 6, 2022, at 11:00 PM, Tu, Jiannan > wrote: I am using TS IMEX to solve a large DAE system. The DAE is obtained by applying finite FV method to 3-D multi-specie ion/neutral fluids equations with magnetic induction equation. The Jacobian for stiff part is formed by using MatSetValuesStencil(). The Jacobian matrix is very sparse, no more than 10 non-zeros on each row. MatSetValuesStencil requires local to global mapping by calling ISLocaltoGlobalMapping(). Could you please instruct me how to use local to global mapping? DMDA automatically sets up the ISLocaltoGlobalMapping() so you should not need to. I also tried using DMCreateMatrix() to create the Jacobian. While local to global mapping is not necessary, the matrix takes too much memory and requires 64-bit indices. I would prefer to take the advantage of sparsity of the Jacobian, pre-allocate the matrix to use as less as possible memory so that the code can be run on a multi-core desktop. If using matrix-free does not work for you because the linear solves do not converge or converge too slowly. Then you might be able to decrease the memory used in the matrix. The DMCreateMatrix() does take advantage of sparsity and tries to preallocate only what it needs. Often what it preallocates is the best possible, but for multicomponent problems it has to assume there is full coupling within all the degrees of freedom that live at each at each grid point. How many components live at each grid point in your model and are they not all coupled to each other in the equations? If they are not fully coupled you can use either DMDASetBlockFills() or DMDASetBlockFillsSparse() to indicate the reduced coupling that actually exists for you model. Barry Thank you very much for your advice. Jiannan -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Sun Sep 11 11:09:36 2022 From: bsmith at petsc.dev (Barry Smith) Date: Sun, 11 Sep 2022 12:09:36 -0400 Subject: [petsc-users] Using matrix-free with KSP In-Reply-To: References: <31AF5BF6-8AC4-4A76-9235-9C0E1BB3EE16@petsc.dev> <7F5F237F-B73B-447E-A113-B46C697D4C37@petsc.dev> <79BC638A-1064-42A1-AE04-40ACAA82323A@petsc.dev> <12206964-1F03-45CE-9FDB-EC4B244CD1F8@petsc.dev> <15DD8C9E-C3A5-4758-9A1F-B9FA28B35825@petsc.dev> Message-ID: <21002A96-7F12-452D-94D1-A3B8265D982A@petsc.dev> > On Sep 11, 2022, at 9:56 AM, Tu, Jiannan wrote: > > Barry, > > Thank you. > > Yes, the DMDA is created with periodic BC. But it might not be the periodic BC since in azimuthal direction, the ghost grid -1 is actually grid Nmax, and Nmax+1 is grid 0. Consider the following, one has n points from 0 to n-1 where n = 4, so I think with your definition of Nmax, Nmax is 3, Nmax + 1 is 4, -1 0 1 2 3 4 + * * * * + Thus there is one grid point at each end. The * represent regular grid points and the + ghost locations. Is this your situation? This is what we call periodic with a stencil width of 1 with DMDA and the DMCreateMatrix() should automatically provide the correct preallocation and nonzero structure. Can you please do a MatView() on the result from DMCreateMatrix() for a very small grid before you set any values and you should see the locations for the coupling across the periodic boundary. It sounds like you have a more complicated cross periodic coupling, we need to understand it exactly so we can determine how to proceed. Barry > > Still the problem is how to properly pre-allocate matrix. Otherwise either ?Augument out of range? error or code stuck at inserting values if MatSetOption() is called. > > Jiannan > > From: Barry Smith > > Sent: Sunday, September 11, 2022 12:10 AM > To: Tu, Jiannan > > Cc: petsc-users > > Subject: Re: [petsc-users] Using matrix-free with KSP > > CAUTION: This email was sent from outside the UMass Lowell network. > > > Have you created the DMDA with the periodic boundary conditions arguments? Or are your boundary conditions not simply periodic? > > Compare DMCreateMatrix_DA_3d_MPIAIJ() and DMCreateMatrix_DA_3d_MPIAIJ_Fill() in src/dm/impls/da/fdda.c I think they both attempt to handle the periodic case. > > Barry > > > > > > On Sep 10, 2022, at 10:14 PM, Tu, Jiannan > wrote: > > Barry, > > I think I understand how to set values for the intra-grid coupling. But I am not clear about the way to set values for inter-grid coupling. When a component couples to itself at above, below, right, left, front and back grids, diagonal is set as 1 for the second array of DMDASetBlockFills(). How about if involving more grids? How does DMDA know where to allocate memory? In my case, a component couples to itself at the other end of the grid because of periodic boundary condition. There is an error message ?Augment out of range . Insert new nonzero at global row/column (26, 520526) into matrix? because of this far away coupling. The column index 520526 is related to the other end of the grid in azimuthal direction. > > Thanks, > Jiannan > > > > From: Barry Smith > > Sent: Saturday, September 10, 2022 1:10 PM > To: Tu, Jiannan > > Cc: petsc-users > > Subject: Re: [petsc-users] Using matrix-free with KSP > > CAUTION: This email was sent from outside the UMass Lowell network. > > > If you are using DMCreateMatrix() then MatMPIAIJSetPreallocation is not needed, it is automatically handled by the DM. > > It sounds like that the values you set in DMDASetBlockFills() did not include all the coupling, hence the preallocation was wrong, hence the time to insert values was too long. > > Barry > > > > > On Sep 10, 2022, at 8:16 AM, Tu, Jiannan > wrote: > > Just found MatMPIAIJSetPreallocation is needed. Now Jacobin insertion and assembly is fast. > > From: Barry Smith > > Sent: Thursday, September 8, 2022 9:53 PM > To: Tu, Jiannan > > Subject: Re: [petsc-users] Using matrix-free with KSP > > CAUTION: This email was sent from outside the UMass Lowell network. > > > > > > > On Sep 8, 2022, at 8:34 PM, Tu, Jiannan > wrote: > > Barry, > > Thank you. > > I set up these two arrays according to the non-zero pattern of the Jacobian. The DMDASetBlockFills() is then called after the call to DMCreateMatrix(da, &A). > > Absolutely not! It must be called before DMCreateMatrix() so that the DM knows how to construct exactly the matrix you need. > > > > > > Nevertheless, the program execution is still killed with exit code 9, which I assume is due to overuse of the memory. My desktop machine has 32 GB RAM. > > I ran the code with a smaller mesh, 100x45x90, the result is the same. With that number of grids, 26 unknowns, and up to 10 non-zero at each matrix row, the estimated memory to store 100x45x90x26x10 elements should be less than 1 GB (double precision). I am wondering why the matrix still takes too much memory. Maybe I have to use matrix-free? > > Thank you, > Jiannan > > From: Barry Smith > > Sent: Thursday, September 8, 2022 4:19 PM > To: Tu, Jiannan > > Subject: Re: [petsc-users] Using matrix-free with KSP > > CAUTION: This email was sent from outside the UMass Lowell network. > > > > > > > > On Sep 8, 2022, at 3:39 PM, Tu, Jiannan > wrote: > > Barry, > > Thank you very much for the detailed description. > > So the first array, dfill, is for intra grids and the second one, ofill is for the inter-grid coupling? > > In the case for inter-grid coupling, a component is only coupled to itself, say x in your example is coupled to itself on above, below, right, left, front, and back, how to set values for the second array? > > Then the off-diagonal one would just have values on its diagonal. > > > > > > Jiannan > > From: Barry Smith > Sent: Thursday, September 8, 2022 2:12 PM > To: Tu, Jiannan > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Using matrix-free with KSP > > CAUTION: This email was sent from outside the UMass Lowell network. > > > > > > > > > On Sep 8, 2022, at 1:47 PM, Tu, Jiannan > wrote: > > Barry, > > Thank you very much. > > DMDASetBlockFills() needs two input arrays specifying sparsity pattern. In my case, I have 26 components at each grid. Is it correct that I need to have two arrays of 26x26 elements with value 1 for coupling and 0 for no coupling? > > Exactly > > > > > > > Also how to determine when the value should be 1 or 0? > > That is problem dependent. Given paper-and-pencil you know for your stencil which components are coupled with other components at each grid point and which components are coupled with components at neighboring grid points. > > Note the second array contains information about all the neighboring grid points. Above, below, right, left, front back. Normally the coupling is symmetric so the nonzero structures of those 6 blocks are identical. If, for you particular problem, the nonzero structures of the six are not identical you use the union of the nonzero structures of the six of them. > > > 2d example say you have x,y,z at each grid point and the inter-point coupling is x is connected to y and y is connect to x then the first array is > > [ 1 1 0 ; 1 1; 0 ; 0 0 1]; the zeros are because x is not coupled to z and y is not coupled to z. > > For intra grid points, say x is coupled to z and itself, z is coupled to x and itself and y is not coupled to itself then > > [1 0 1; 0 0 0; 1 0 1]; the middle row is zero because y is not coupled to anything and the zeros in the first and last are because x and z are not coupled to y. > > > > > > > > > Each component involves 7 stencils (i-1, j, k), (I, j, k), (i+1, j, k), (I, j-1, k), (I, j+1, k), (I, j, k-1), and (I, j, k+1), and couples with several other components. > > Jiannan > > > > Sent from Mail for Windows > > From: Barry Smith > Sent: Wednesday, September 7, 2022 11:53 AM > To: Tu, Jiannan > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Using matrix-free with KSP > > CAUTION: This email was sent from outside the UMass Lowell network. > > > DMDASetBlockFills() or DMDASetBlockFillsSparse() (they are just different ways of providing the same information) will help you here enormously, your type of case is exactly what they were designed for. > > Barry > > > > > > > > > On Sep 7, 2022, at 10:47 AM, Tu, Jiannan > wrote: > > Barry and Hong, > > Thank you. > > There are 26 components at each grid and there are not fully coupled in terms of stiff functions. Mutual coupling is among about 6 components. > > I would prefer not using matrix-free since the Jacobina is not difficult to calculate and only up to 10 non-zeros at each row. I'll try DMDASetBlockFills() or DMDASetBlockFillsSparse() and see how they can reduce the memory usage. > > Jiannan > > From: Barry Smith > > Sent: Tuesday, September 6, 2022 11:33 PM > To: Tu, Jiannan > > Cc: petsc-users at mcs.anl.gov > > Subject: Re: [petsc-users] Using matrix-free with KSP > > CAUTION: This email was sent from outside the UMass Lowell network. > > > > > > > > > > On Sep 6, 2022, at 11:00 PM, Tu, Jiannan > wrote: > > I am using TS IMEX to solve a large DAE system. The DAE is obtained by applying finite FV method to 3-D multi-specie ion/neutral fluids equations with magnetic induction equation. The Jacobian for stiff part is formed by using MatSetValuesStencil(). The Jacobian matrix is very sparse, no more than 10 non-zeros on each row. MatSetValuesStencil requires local to global mapping by calling ISLocaltoGlobalMapping(). Could you please instruct me how to use local to global mapping? > > DMDA automatically sets up the ISLocaltoGlobalMapping() so you should not need to. > > I also tried using DMCreateMatrix() to create the Jacobian. While local to global mapping is not necessary, the matrix takes too much memory and requires 64-bit indices. I would prefer to take the advantage of sparsity of the Jacobian, pre-allocate the matrix to use as less as possible memory so that the code can be run on a multi-core desktop. > > If using matrix-free does not work for you because the linear solves do not converge or converge too slowly. Then you might be able to decrease the memory used in the matrix. The DMCreateMatrix() does take advantage of sparsity and tries to preallocate only what it needs. Often what it preallocates is the best possible, but for multicomponent problems it has to assume there is full coupling within all the degrees of freedom that live at each at each grid point. How many components live at each grid point in your model and are they not all coupled to each other in the equations? If they are not fully coupled you can use either DMDASetBlockFills() or DMDASetBlockFillsSparse() to indicate the reduced coupling that actually exists for you model. > > Barry > > > > > > > > > Thank you very much for your advice. > > Jiannan -------------- next part -------------- An HTML attachment was scrubbed... URL: From Jiannan_Tu at uml.edu Sun Sep 11 11:58:44 2022 From: Jiannan_Tu at uml.edu (Tu, Jiannan) Date: Sun, 11 Sep 2022 16:58:44 +0000 Subject: [petsc-users] Using matrix-free with KSP In-Reply-To: <21002A96-7F12-452D-94D1-A3B8265D982A@petsc.dev> References: <31AF5BF6-8AC4-4A76-9235-9C0E1BB3EE16@petsc.dev> <7F5F237F-B73B-447E-A113-B46C697D4C37@petsc.dev> <79BC638A-1064-42A1-AE04-40ACAA82323A@petsc.dev> <12206964-1F03-45CE-9FDB-EC4B244CD1F8@petsc.dev> <15DD8C9E-C3A5-4758-9A1F-B9FA28B35825@petsc.dev> <21002A96-7F12-452D-94D1-A3B8265D982A@petsc.dev> Message-ID: Barry, Thank you. Here is part of the jacbian, showing the non-zero structure (vals[nv] all set to 1 for simplicity). Np is the last regular grid in k-index, a3 is the number of grids also in k-index. Nth is the max j grid index. This may provide useful information for you to help me determine how to pre-allocate the matrix. Jiannan ------------------------------------------------------------ double vals [10]; MatStencil row, col [10]; for (k = zs; k < zs+zm; k++) { row.k=k; zk=k-zs; if (k == 0) km = Np; else km = k-1; if (k == Np) kp = 0; else kp = k+1; for (j = ys; j < ys+ym; j++) { row.j=j; if (j == 0) {jm = j; kcm = (k+a3/2) % a3;} else {jm = j-1; kcm = k;} if (j == Nth) {jp = Nth; kcp = (k+a3/2) % a3;} else {jp = j+1; kcp = k;} for (i = xs; i < xs+xm; i++) { row.i=i; for (ir = 0; ir < 26; ir++) { row.c=ir; nv=0; col[nv].k=k; col[nv].j=j; col[nv].i=im; //column at i-1 grid col[nv].c=ir; vals[nv] = 1.0; nv++; col[nv].k=k; col[nv].j=j; col[nv].i=i; col[nv].c=ir; vals[nv] = 1.0; nv++; col[nv].k=k; col[nv].j=j; col[nv].i=ip; //column at i+1 grid col[nv].c=ir; vals[nv] = 1.0; nv++; col[nv].k=kcm; col[nv].j=jm; //j-1 grid. wrapped to the other side of noth pole if j=0 col[nv].i=i; col[nv].c=ir; vals[nv] = 1.0; nv++; col[nv].k=kcp; col[nv].j=jp; //j-+ grid. wrapped to the other side of south pole if j=max col[nv].i=i; col[nv].c=ir; vals[nv] = 1.0; nv++; col[nv].k=km; //k-1 grid. if k=0 then km=Np col[nv].j=j; col[nv].i=i; col[nv].c=ir; vals[nv] = 1.0; nv++; col[nv].k=kp; //k+1 grid. if k=Np then kp=0 col[nv].j=j; col[nv].i=i; col[nv].c=ir; vals[nv] = 1.0; nv++; col[nv].k=k; col[nv].j=j; col[nv].i=i; col[nv].c=11; //intra couping to component 11 vals[nv] = 1.0; nv++; col[nv].k=k; col[nv].j=j; col[nv].i=i; col[nv].c=22; //intra couping to component 22 vals[nv] = 1.0; nv++; MatSetValuesStencil(Jac,1,&row,nv,col,vals,INSERT_VALUES); } } } } From: Barry Smith Sent: Sunday, September 11, 2022 12:10 PM To: Tu, Jiannan Cc: petsc-users Subject: Re: [petsc-users] Using matrix-free with KSP CAUTION: This email was sent from outside the UMass Lowell network. On Sep 11, 2022, at 9:56 AM, Tu, Jiannan > wrote: Barry, Thank you. Yes, the DMDA is created with periodic BC. But it might not be the periodic BC since in azimuthal direction, the ghost grid -1 is actually grid Nmax, and Nmax+1 is grid 0. Consider the following, one has n points from 0 to n-1 where n = 4, so I think with your definition of Nmax, Nmax is 3, Nmax + 1 is 4, -1 0 1 2 3 4 + * * * * + Thus there is one grid point at each end. The * represent regular grid points and the + ghost locations. Is this your situation? This is what we call periodic with a stencil width of 1 with DMDA and the DMCreateMatrix() should automatically provide the correct preallocation and nonzero structure. Can you please do a MatView() on the result from DMCreateMatrix() for a very small grid before you set any values and you should see the locations for the coupling across the periodic boundary. It sounds like you have a more complicated cross periodic coupling, we need to understand it exactly so we can determine how to proceed. Barry Still the problem is how to properly pre-allocate matrix. Otherwise either "Augument out of range" error or code stuck at inserting values if MatSetOption() is called. Jiannan From: Barry Smith > Sent: Sunday, September 11, 2022 12:10 AM To: Tu, Jiannan > Cc: petsc-users > Subject: Re: [petsc-users] Using matrix-free with KSP CAUTION: This email was sent from outside the UMass Lowell network. Have you created the DMDA with the periodic boundary conditions arguments? Or are your boundary conditions not simply periodic? Compare DMCreateMatrix_DA_3d_MPIAIJ() and DMCreateMatrix_DA_3d_MPIAIJ_Fill() in src/dm/impls/da/fdda.c I think they both attempt to handle the periodic case. Barry On Sep 10, 2022, at 10:14 PM, Tu, Jiannan > wrote: Barry, I think I understand how to set values for the intra-grid coupling. But I am not clear about the way to set values for inter-grid coupling. When a component couples to itself at above, below, right, left, front and back grids, diagonal is set as 1 for the second array of DMDASetBlockFills(). How about if involving more grids? How does DMDA know where to allocate memory? In my case, a component couples to itself at the other end of the grid because of periodic boundary condition. There is an error message "Augment out of range . Insert new nonzero at global row/column (26, 520526) into matrix" because of this far away coupling. The column index 520526 is related to the other end of the grid in azimuthal direction. Thanks, Jiannan From: Barry Smith > Sent: Saturday, September 10, 2022 1:10 PM To: Tu, Jiannan > Cc: petsc-users > Subject: Re: [petsc-users] Using matrix-free with KSP CAUTION: This email was sent from outside the UMass Lowell network. If you are using DMCreateMatrix() then MatMPIAIJSetPreallocation is not needed, it is automatically handled by the DM. It sounds like that the values you set in DMDASetBlockFills() did not include all the coupling, hence the preallocation was wrong, hence the time to insert values was too long. Barry On Sep 10, 2022, at 8:16 AM, Tu, Jiannan > wrote: Just found MatMPIAIJSetPreallocation is needed. Now Jacobin insertion and assembly is fast. From: Barry Smith > Sent: Thursday, September 8, 2022 9:53 PM To: Tu, Jiannan > Subject: Re: [petsc-users] Using matrix-free with KSP CAUTION: This email was sent from outside the UMass Lowell network. On Sep 8, 2022, at 8:34 PM, Tu, Jiannan > wrote: Barry, Thank you. I set up these two arrays according to the non-zero pattern of the Jacobian. The DMDASetBlockFills() is then called after the call to DMCreateMatrix(da, &A). Absolutely not! It must be called before DMCreateMatrix() so that the DM knows how to construct exactly the matrix you need. Nevertheless, the program execution is still killed with exit code 9, which I assume is due to overuse of the memory. My desktop machine has 32 GB RAM. I ran the code with a smaller mesh, 100x45x90, the result is the same. With that number of grids, 26 unknowns, and up to 10 non-zero at each matrix row, the estimated memory to store 100x45x90x26x10 elements should be less than 1 GB (double precision). I am wondering why the matrix still takes too much memory. Maybe I have to use matrix-free? Thank you, Jiannan From: Barry Smith > Sent: Thursday, September 8, 2022 4:19 PM To: Tu, Jiannan > Subject: Re: [petsc-users] Using matrix-free with KSP CAUTION: This email was sent from outside the UMass Lowell network. On Sep 8, 2022, at 3:39 PM, Tu, Jiannan > wrote: Barry, Thank you very much for the detailed description. So the first array, dfill, is for intra grids and the second one, ofill is for the inter-grid coupling? In the case for inter-grid coupling, a component is only coupled to itself, say x in your example is coupled to itself on above, below, right, left, front, and back, how to set values for the second array? Then the off-diagonal one would just have values on its diagonal. Jiannan From: Barry Smith Sent: Thursday, September 8, 2022 2:12 PM To: Tu, Jiannan Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] Using matrix-free with KSP CAUTION: This email was sent from outside the UMass Lowell network. On Sep 8, 2022, at 1:47 PM, Tu, Jiannan > wrote: Barry, Thank you very much. DMDASetBlockFills() needs two input arrays specifying sparsity pattern. In my case, I have 26 components at each grid. Is it correct that I need to have two arrays of 26x26 elements with value 1 for coupling and 0 for no coupling? Exactly Also how to determine when the value should be 1 or 0? That is problem dependent. Given paper-and-pencil you know for your stencil which components are coupled with other components at each grid point and which components are coupled with components at neighboring grid points. Note the second array contains information about all the neighboring grid points. Above, below, right, left, front back. Normally the coupling is symmetric so the nonzero structures of those 6 blocks are identical. If, for you particular problem, the nonzero structures of the six are not identical you use the union of the nonzero structures of the six of them. 2d example say you have x,y,z at each grid point and the inter-point coupling is x is connected to y and y is connect to x then the first array is [ 1 1 0 ; 1 1; 0 ; 0 0 1]; the zeros are because x is not coupled to z and y is not coupled to z. For intra grid points, say x is coupled to z and itself, z is coupled to x and itself and y is not coupled to itself then [1 0 1; 0 0 0; 1 0 1]; the middle row is zero because y is not coupled to anything and the zeros in the first and last are because x and z are not coupled to y. Each component involves 7 stencils (i-1, j, k), (I, j, k), (i+1, j, k), (I, j-1, k), (I, j+1, k), (I, j, k-1), and (I, j, k+1), and couples with several other components. Jiannan Sent from Mail for Windows From: Barry Smith Sent: Wednesday, September 7, 2022 11:53 AM To: Tu, Jiannan Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] Using matrix-free with KSP CAUTION: This email was sent from outside the UMass Lowell network. DMDASetBlockFills() or DMDASetBlockFillsSparse() (they are just different ways of providing the same information) will help you here enormously, your type of case is exactly what they were designed for. Barry On Sep 7, 2022, at 10:47 AM, Tu, Jiannan > wrote: Barry and Hong, Thank you. There are 26 components at each grid and there are not fully coupled in terms of stiff functions. Mutual coupling is among about 6 components. I would prefer not using matrix-free since the Jacobina is not difficult to calculate and only up to 10 non-zeros at each row. I'll try DMDASetBlockFills() or DMDASetBlockFillsSparse() and see how they can reduce the memory usage. Jiannan From: Barry Smith > Sent: Tuesday, September 6, 2022 11:33 PM To: Tu, Jiannan > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Using matrix-free with KSP CAUTION: This email was sent from outside the UMass Lowell network. On Sep 6, 2022, at 11:00 PM, Tu, Jiannan > wrote: I am using TS IMEX to solve a large DAE system. The DAE is obtained by applying finite FV method to 3-D multi-specie ion/neutral fluids equations with magnetic induction equation. The Jacobian for stiff part is formed by using MatSetValuesStencil(). The Jacobian matrix is very sparse, no more than 10 non-zeros on each row. MatSetValuesStencil requires local to global mapping by calling ISLocaltoGlobalMapping(). Could you please instruct me how to use local to global mapping? DMDA automatically sets up the ISLocaltoGlobalMapping() so you should not need to. I also tried using DMCreateMatrix() to create the Jacobian. While local to global mapping is not necessary, the matrix takes too much memory and requires 64-bit indices. I would prefer to take the advantage of sparsity of the Jacobian, pre-allocate the matrix to use as less as possible memory so that the code can be run on a multi-core desktop. If using matrix-free does not work for you because the linear solves do not converge or converge too slowly. Then you might be able to decrease the memory used in the matrix. The DMCreateMatrix() does take advantage of sparsity and tries to preallocate only what it needs. Often what it preallocates is the best possible, but for multicomponent problems it has to assume there is full coupling within all the degrees of freedom that live at each at each grid point. How many components live at each grid point in your model and are they not all coupled to each other in the equations? If they are not fully coupled you can use either DMDASetBlockFills() or DMDASetBlockFillsSparse() to indicate the reduced coupling that actually exists for you model. Barry Thank you very much for your advice. Jiannan -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Sun Sep 11 12:39:09 2022 From: bsmith at petsc.dev (Barry Smith) Date: Sun, 11 Sep 2022 13:39:09 -0400 Subject: [petsc-users] Using matrix-free with KSP In-Reply-To: References: <31AF5BF6-8AC4-4A76-9235-9C0E1BB3EE16@petsc.dev> <7F5F237F-B73B-447E-A113-B46C697D4C37@petsc.dev> <79BC638A-1064-42A1-AE04-40ACAA82323A@petsc.dev> <12206964-1F03-45CE-9FDB-EC4B244CD1F8@petsc.dev> <15DD8C9E-C3A5-4758-9A1F-B9FA28B35825@petsc.dev> <21002A96-7F12-452D-94D1-A3B8265D982A@petsc.dev> Message-ID: MatSetValuesStencil https://petsc.org/release/docs/manualpages/Mat/MatSetValuesStencil.html For periodic boundary conditions use negative indices for values to the left (below 0; that are to be obtained by wrapping values from right edge). For values to the right of the last entry using that index plus one etc to obtain values that obtained by wrapping the values from the left edge. This does not work for anything but the `DM_BOUNDARY_PERIODIC` boundary type. Thus I believe it should be > km = k-1; > kp = k+1; Does this not work? > On Sep 11, 2022, at 12:58 PM, Tu, Jiannan wrote: > > Barry, > > Thank you. Here is part of the jacbian, showing the non-zero structure (vals[nv] all set to 1 for simplicity). Np is the last regular grid in k-index, a3 is the number of grids also in k-index. Nth is the max j grid index. This may provide useful information for you to help me determine how to pre-allocate the matrix. > > Jiannan > > ------------------------------------------------------------ > double vals [10]; > MatStencil row, col [10]; > > for (k = zs; k < zs+zm; k++) { > row.k=k; zk=k-zs; > > if (k == 0) km = Np; else km = k-1; > if (k == Np) kp = 0; else kp = k+1; > > for (j = ys; j < ys+ym; j++) { > row.j=j; > > if (j == 0) {jm = j; kcm = (k+a3/2) % a3;} > else {jm = j-1; kcm = k;} > if (j == Nth) {jp = Nth; kcp = (k+a3/2) % a3;} > else {jp = j+1; kcp = k;} > > for (i = xs; i < xs+xm; i++) { > row.i=i; > > for (ir = 0; ir < 26; ir++) { > row.c=ir; nv=0; > > col[nv].k=k; col[nv].j=j; > col[nv].i=im; //column at i-1 grid > col[nv].c=ir; > vals[nv] = 1.0; > nv++; > > col[nv].k=k; col[nv].j=j; > col[nv].i=i; > col[nv].c=ir; > vals[nv] = 1.0; > nv++; > > col[nv].k=k; col[nv].j=j; > col[nv].i=ip; //column at i+1 grid > col[nv].c=ir; > vals[nv] = 1.0; > nv++; > > col[nv].k=kcm; col[nv].j=jm; //j-1 grid. wrapped to the other side of noth pole if j=0 > col[nv].i=i; col[nv].c=ir; > vals[nv] = 1.0; > nv++; > > col[nv].k=kcp; col[nv].j=jp; //j-+ grid. wrapped to the other side of south pole if j=max > col[nv].i=i; col[nv].c=ir; > vals[nv] = 1.0; > nv++; > > col[nv].k=km; //k-1 grid. if k=0 then km=Np > col[nv].j=j; > col[nv].i=i; col[nv].c=ir; > vals[nv] = 1.0; > nv++; > > col[nv].k=kp; //k+1 grid. if k=Np then kp=0 > col[nv].j=j; > col[nv].i=i; col[nv].c=ir; > vals[nv] = 1.0; > nv++; > > col[nv].k=k; col[nv].j=j; col[nv].i=i; > col[nv].c=11; //intra couping to component 11 > vals[nv] = 1.0; > nv++; > > col[nv].k=k; col[nv].j=j; col[nv].i=i; > col[nv].c=22; //intra couping to component 22 > vals[nv] = 1.0; > nv++; > > MatSetValuesStencil(Jac,1,&row,nv,col,vals,INSERT_VALUES); > } > } > } > } > > From: Barry Smith > > Sent: Sunday, September 11, 2022 12:10 PM > To: Tu, Jiannan > > Cc: petsc-users > > Subject: Re: [petsc-users] Using matrix-free with KSP > > CAUTION: This email was sent from outside the UMass Lowell network. > > > > > On Sep 11, 2022, at 9:56 AM, Tu, Jiannan > wrote: > > Barry, > > Thank you. > > Yes, the DMDA is created with periodic BC. But it might not be the periodic BC since in azimuthal direction, the ghost grid -1 is actually grid Nmax, and Nmax+1 is grid 0. > > Consider the following, one has n points from 0 to n-1 where n = 4, so I think with your definition of Nmax, Nmax is 3, Nmax + 1 is 4, > > > -1 0 1 2 3 4 > + * * * * + > > Thus there is one grid point at each end. The * represent regular grid points and the + ghost locations. Is this your situation? > > This is what we call periodic with a stencil width of 1 with DMDA and the DMCreateMatrix() should automatically provide the correct preallocation and nonzero structure. > > Can you please do a MatView() on the result from DMCreateMatrix() for a very small grid before you set any values and you should see the locations for the coupling across the periodic boundary. > > It sounds like you have a more complicated cross periodic coupling, we need to understand it exactly so we can determine how to proceed. > > Barry > > > > > Still the problem is how to properly pre-allocate matrix. Otherwise either ?Augument out of range? error or code stuck at inserting values if MatSetOption() is called. > > Jiannan > > From: Barry Smith > > Sent: Sunday, September 11, 2022 12:10 AM > To: Tu, Jiannan > > Cc: petsc-users > > Subject: Re: [petsc-users] Using matrix-free with KSP > > CAUTION: This email was sent from outside the UMass Lowell network. > > > Have you created the DMDA with the periodic boundary conditions arguments? Or are your boundary conditions not simply periodic? > > Compare DMCreateMatrix_DA_3d_MPIAIJ() and DMCreateMatrix_DA_3d_MPIAIJ_Fill() in src/dm/impls/da/fdda.c I think they both attempt to handle the periodic case. > > Barry > > > > > > > On Sep 10, 2022, at 10:14 PM, Tu, Jiannan > wrote: > > Barry, > > I think I understand how to set values for the intra-grid coupling. But I am not clear about the way to set values for inter-grid coupling. When a component couples to itself at above, below, right, left, front and back grids, diagonal is set as 1 for the second array of DMDASetBlockFills(). How about if involving more grids? How does DMDA know where to allocate memory? In my case, a component couples to itself at the other end of the grid because of periodic boundary condition. There is an error message ?Augment out of range . Insert new nonzero at global row/column (26, 520526) into matrix? because of this far away coupling. The column index 520526 is related to the other end of the grid in azimuthal direction. > > Thanks, > Jiannan > > > > From: Barry Smith > > Sent: Saturday, September 10, 2022 1:10 PM > To: Tu, Jiannan > > Cc: petsc-users > > Subject: Re: [petsc-users] Using matrix-free with KSP > > CAUTION: This email was sent from outside the UMass Lowell network. > > > If you are using DMCreateMatrix() then MatMPIAIJSetPreallocation is not needed, it is automatically handled by the DM. > > It sounds like that the values you set in DMDASetBlockFills() did not include all the coupling, hence the preallocation was wrong, hence the time to insert values was too long. > > Barry > > > > > > On Sep 10, 2022, at 8:16 AM, Tu, Jiannan > wrote: > > Just found MatMPIAIJSetPreallocation is needed. Now Jacobin insertion and assembly is fast. > > From: Barry Smith > > Sent: Thursday, September 8, 2022 9:53 PM > To: Tu, Jiannan > > Subject: Re: [petsc-users] Using matrix-free with KSP > > CAUTION: This email was sent from outside the UMass Lowell network. > > > > > > > > On Sep 8, 2022, at 8:34 PM, Tu, Jiannan > wrote: > > Barry, > > Thank you. > > I set up these two arrays according to the non-zero pattern of the Jacobian. The DMDASetBlockFills() is then called after the call to DMCreateMatrix(da, &A). > > Absolutely not! It must be called before DMCreateMatrix() so that the DM knows how to construct exactly the matrix you need. > > > > > > > Nevertheless, the program execution is still killed with exit code 9, which I assume is due to overuse of the memory. My desktop machine has 32 GB RAM. > > I ran the code with a smaller mesh, 100x45x90, the result is the same. With that number of grids, 26 unknowns, and up to 10 non-zero at each matrix row, the estimated memory to store 100x45x90x26x10 elements should be less than 1 GB (double precision). I am wondering why the matrix still takes too much memory. Maybe I have to use matrix-free? > > Thank you, > Jiannan > > From: Barry Smith > > Sent: Thursday, September 8, 2022 4:19 PM > To: Tu, Jiannan > > Subject: Re: [petsc-users] Using matrix-free with KSP > > CAUTION: This email was sent from outside the UMass Lowell network. > > > > > > > > > On Sep 8, 2022, at 3:39 PM, Tu, Jiannan > wrote: > > Barry, > > Thank you very much for the detailed description. > > So the first array, dfill, is for intra grids and the second one, ofill is for the inter-grid coupling? > > In the case for inter-grid coupling, a component is only coupled to itself, say x in your example is coupled to itself on above, below, right, left, front, and back, how to set values for the second array? > > Then the off-diagonal one would just have values on its diagonal. > > > > > > > Jiannan > > From: Barry Smith > Sent: Thursday, September 8, 2022 2:12 PM > To: Tu, Jiannan > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Using matrix-free with KSP > > CAUTION: This email was sent from outside the UMass Lowell network. > > > > > > > > > > On Sep 8, 2022, at 1:47 PM, Tu, Jiannan > wrote: > > Barry, > > Thank you very much. > > DMDASetBlockFills() needs two input arrays specifying sparsity pattern. In my case, I have 26 components at each grid. Is it correct that I need to have two arrays of 26x26 elements with value 1 for coupling and 0 for no coupling? > > Exactly > > > > > > > > Also how to determine when the value should be 1 or 0? > > That is problem dependent. Given paper-and-pencil you know for your stencil which components are coupled with other components at each grid point and which components are coupled with components at neighboring grid points. > > Note the second array contains information about all the neighboring grid points. Above, below, right, left, front back. Normally the coupling is symmetric so the nonzero structures of those 6 blocks are identical. If, for you particular problem, the nonzero structures of the six are not identical you use the union of the nonzero structures of the six of them. > > > 2d example say you have x,y,z at each grid point and the inter-point coupling is x is connected to y and y is connect to x then the first array is > > [ 1 1 0 ; 1 1; 0 ; 0 0 1]; the zeros are because x is not coupled to z and y is not coupled to z. > > For intra grid points, say x is coupled to z and itself, z is coupled to x and itself and y is not coupled to itself then > > [1 0 1; 0 0 0; 1 0 1]; the middle row is zero because y is not coupled to anything and the zeros in the first and last are because x and z are not coupled to y. > > > > > > > > > > Each component involves 7 stencils (i-1, j, k), (I, j, k), (i+1, j, k), (I, j-1, k), (I, j+1, k), (I, j, k-1), and (I, j, k+1), and couples with several other components. > > Jiannan > > > > Sent from Mail for Windows > > From: Barry Smith > Sent: Wednesday, September 7, 2022 11:53 AM > To: Tu, Jiannan > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Using matrix-free with KSP > > CAUTION: This email was sent from outside the UMass Lowell network. > > > DMDASetBlockFills() or DMDASetBlockFillsSparse() (they are just different ways of providing the same information) will help you here enormously, your type of case is exactly what they were designed for. > > Barry > > > > > > > > > > On Sep 7, 2022, at 10:47 AM, Tu, Jiannan > wrote: > > Barry and Hong, > > Thank you. > > There are 26 components at each grid and there are not fully coupled in terms of stiff functions. Mutual coupling is among about 6 components. > > I would prefer not using matrix-free since the Jacobina is not difficult to calculate and only up to 10 non-zeros at each row. I'll try DMDASetBlockFills() or DMDASetBlockFillsSparse() and see how they can reduce the memory usage. > > Jiannan > > From: Barry Smith > > Sent: Tuesday, September 6, 2022 11:33 PM > To: Tu, Jiannan > > Cc: petsc-users at mcs.anl.gov > > Subject: Re: [petsc-users] Using matrix-free with KSP > > CAUTION: This email was sent from outside the UMass Lowell network. > > > > > > > > > > > On Sep 6, 2022, at 11:00 PM, Tu, Jiannan > wrote: > > I am using TS IMEX to solve a large DAE system. The DAE is obtained by applying finite FV method to 3-D multi-specie ion/neutral fluids equations with magnetic induction equation. The Jacobian for stiff part is formed by using MatSetValuesStencil(). The Jacobian matrix is very sparse, no more than 10 non-zeros on each row. MatSetValuesStencil requires local to global mapping by calling ISLocaltoGlobalMapping(). Could you please instruct me how to use local to global mapping? > > DMDA automatically sets up the ISLocaltoGlobalMapping() so you should not need to. > > I also tried using DMCreateMatrix() to create the Jacobian. While local to global mapping is not necessary, the matrix takes too much memory and requires 64-bit indices. I would prefer to take the advantage of sparsity of the Jacobian, pre-allocate the matrix to use as less as possible memory so that the code can be run on a multi-core desktop. > > If using matrix-free does not work for you because the linear solves do not converge or converge too slowly. Then you might be able to decrease the memory used in the matrix. The DMCreateMatrix() does take advantage of sparsity and tries to preallocate only what it needs. Often what it preallocates is the best possible, but for multicomponent problems it has to assume there is full coupling within all the degrees of freedom that live at each at each grid point. How many components live at each grid point in your model and are they not all coupled to each other in the equations? If they are not fully coupled you can use either DMDASetBlockFills() or DMDASetBlockFillsSparse() to indicate the reduced coupling that actually exists for you model. > > Barry > > > > > > > > > > Thank you very much for your advice. > > Jiannan -------------- next part -------------- An HTML attachment was scrubbed... URL: From Jiannan_Tu at uml.edu Sun Sep 11 22:01:57 2022 From: Jiannan_Tu at uml.edu (Tu, Jiannan) Date: Mon, 12 Sep 2022 03:01:57 +0000 Subject: [petsc-users] Using matrix-free with KSP In-Reply-To: <21002A96-7F12-452D-94D1-A3B8265D982A@petsc.dev> References: <31AF5BF6-8AC4-4A76-9235-9C0E1BB3EE16@petsc.dev> <7F5F237F-B73B-447E-A113-B46C697D4C37@petsc.dev> <79BC638A-1064-42A1-AE04-40ACAA82323A@petsc.dev> <12206964-1F03-45CE-9FDB-EC4B244CD1F8@petsc.dev> <15DD8C9E-C3A5-4758-9A1F-B9FA28B35825@petsc.dev> <21002A96-7F12-452D-94D1-A3B8265D982A@petsc.dev> Message-ID: Barry, Creating DNDA with periodic boundary BC and using km=k-1; kp=k+1 even at the BCs solve the "augment out of range" problem for inserting elements at theae boundaries of k-index. Now the problem is due to wrapping around the north pole when j=0 and jm = -1 is reset as j=0 but on the other side of the pole with k-index replacing by kcm = [kmax+1)/2\ % (kmax+1). In my case, the "Augment out of range" happens at global row/column (26, 5265026), exactly at wrapping jm=-1 to j=0 to the other side. I am thinking about how to properly use ghost grid j = -1 by setting appropriate BC there and inserting at that location without wrapping. Jiannan From: Barry Smith Sent: Sunday, September 11, 2022 12:10 PM To: Tu, Jiannan Cc: petsc-users Subject: Re: [petsc-users] Using matrix-free with KSP CAUTION: This email was sent from outside the UMass Lowell network. On Sep 11, 2022, at 9:56 AM, Tu, Jiannan > wrote: Barry, Thank you. Yes, the DMDA is created with periodic BC. But it might not be the periodic BC since in azimuthal direction, the ghost grid -1 is actually grid Nmax, and Nmax+1 is grid 0. Consider the following, one has n points from 0 to n-1 where n = 4, so I think with your definition of Nmax, Nmax is 3, Nmax + 1 is 4, -1 0 1 2 3 4 + * * * * + Thus there is one grid point at each end. The * represent regular grid points and the + ghost locations. Is this your situation? This is what we call periodic with a stencil width of 1 with DMDA and the DMCreateMatrix() should automatically provide the correct preallocation and nonzero structure. Can you please do a MatView() on the result from DMCreateMatrix() for a very small grid before you set any values and you should see the locations for the coupling across the periodic boundary. It sounds like you have a more complicated cross periodic coupling, we need to understand it exactly so we can determine how to proceed. Barry Still the problem is how to properly pre-allocate matrix. Otherwise either "Augument out of range" error or code stuck at inserting values if MatSetOption() is called. Jiannan From: Barry Smith > Sent: Sunday, September 11, 2022 12:10 AM To: Tu, Jiannan > Cc: petsc-users > Subject: Re: [petsc-users] Using matrix-free with KSP CAUTION: This email was sent from outside the UMass Lowell network. Have you created the DMDA with the periodic boundary conditions arguments? Or are your boundary conditions not simply periodic? Compare DMCreateMatrix_DA_3d_MPIAIJ() and DMCreateMatrix_DA_3d_MPIAIJ_Fill() in src/dm/impls/da/fdda.c I think they both attempt to handle the periodic case. Barry On Sep 10, 2022, at 10:14 PM, Tu, Jiannan > wrote: Barry, I think I understand how to set values for the intra-grid coupling. But I am not clear about the way to set values for inter-grid coupling. When a component couples to itself at above, below, right, left, front and back grids, diagonal is set as 1 for the second array of DMDASetBlockFills(). How about if involving more grids? How does DMDA know where to allocate memory? In my case, a component couples to itself at the other end of the grid because of periodic boundary condition. There is an error message "Augment out of range . Insert new nonzero at global row/column (26, 520526) into matrix" because of this far away coupling. The column index 520526 is related to the other end of the grid in azimuthal direction. Thanks, Jiannan From: Barry Smith > Sent: Saturday, September 10, 2022 1:10 PM To: Tu, Jiannan > Cc: petsc-users > Subject: Re: [petsc-users] Using matrix-free with KSP CAUTION: This email was sent from outside the UMass Lowell network. If you are using DMCreateMatrix() then MatMPIAIJSetPreallocation is not needed, it is automatically handled by the DM. It sounds like that the values you set in DMDASetBlockFills() did not include all the coupling, hence the preallocation was wrong, hence the time to insert values was too long. Barry On Sep 10, 2022, at 8:16 AM, Tu, Jiannan > wrote: Just found MatMPIAIJSetPreallocation is needed. Now Jacobin insertion and assembly is fast. From: Barry Smith > Sent: Thursday, September 8, 2022 9:53 PM To: Tu, Jiannan > Subject: Re: [petsc-users] Using matrix-free with KSP CAUTION: This email was sent from outside the UMass Lowell network. On Sep 8, 2022, at 8:34 PM, Tu, Jiannan > wrote: Barry, Thank you. I set up these two arrays according to the non-zero pattern of the Jacobian. The DMDASetBlockFills() is then called after the call to DMCreateMatrix(da, &A). Absolutely not! It must be called before DMCreateMatrix() so that the DM knows how to construct exactly the matrix you need. Nevertheless, the program execution is still killed with exit code 9, which I assume is due to overuse of the memory. My desktop machine has 32 GB RAM. I ran the code with a smaller mesh, 100x45x90, the result is the same. With that number of grids, 26 unknowns, and up to 10 non-zero at each matrix row, the estimated memory to store 100x45x90x26x10 elements should be less than 1 GB (double precision). I am wondering why the matrix still takes too much memory. Maybe I have to use matrix-free? Thank you, Jiannan From: Barry Smith > Sent: Thursday, September 8, 2022 4:19 PM To: Tu, Jiannan > Subject: Re: [petsc-users] Using matrix-free with KSP CAUTION: This email was sent from outside the UMass Lowell network. On Sep 8, 2022, at 3:39 PM, Tu, Jiannan > wrote: Barry, Thank you very much for the detailed description. So the first array, dfill, is for intra grids and the second one, ofill is for the inter-grid coupling? In the case for inter-grid coupling, a component is only coupled to itself, say x in your example is coupled to itself on above, below, right, left, front, and back, how to set values for the second array? Then the off-diagonal one would just have values on its diagonal. Jiannan From: Barry Smith Sent: Thursday, September 8, 2022 2:12 PM To: Tu, Jiannan Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] Using matrix-free with KSP CAUTION: This email was sent from outside the UMass Lowell network. On Sep 8, 2022, at 1:47 PM, Tu, Jiannan > wrote: Barry, Thank you very much. DMDASetBlockFills() needs two input arrays specifying sparsity pattern. In my case, I have 26 components at each grid. Is it correct that I need to have two arrays of 26x26 elements with value 1 for coupling and 0 for no coupling? Exactly Also how to determine when the value should be 1 or 0? That is problem dependent. Given paper-and-pencil you know for your stencil which components are coupled with other components at each grid point and which components are coupled with components at neighboring grid points. Note the second array contains information about all the neighboring grid points. Above, below, right, left, front back. Normally the coupling is symmetric so the nonzero structures of those 6 blocks are identical. If, for you particular problem, the nonzero structures of the six are not identical you use the union of the nonzero structures of the six of them. 2d example say you have x,y,z at each grid point and the inter-point coupling is x is connected to y and y is connect to x then the first array is [ 1 1 0 ; 1 1; 0 ; 0 0 1]; the zeros are because x is not coupled to z and y is not coupled to z. For intra grid points, say x is coupled to z and itself, z is coupled to x and itself and y is not coupled to itself then [1 0 1; 0 0 0; 1 0 1]; the middle row is zero because y is not coupled to anything and the zeros in the first and last are because x and z are not coupled to y. Each component involves 7 stencils (i-1, j, k), (I, j, k), (i+1, j, k), (I, j-1, k), (I, j+1, k), (I, j, k-1), and (I, j, k+1), and couples with several other components. Jiannan Sent from Mail for Windows From: Barry Smith Sent: Wednesday, September 7, 2022 11:53 AM To: Tu, Jiannan Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] Using matrix-free with KSP CAUTION: This email was sent from outside the UMass Lowell network. DMDASetBlockFills() or DMDASetBlockFillsSparse() (they are just different ways of providing the same information) will help you here enormously, your type of case is exactly what they were designed for. Barry On Sep 7, 2022, at 10:47 AM, Tu, Jiannan > wrote: Barry and Hong, Thank you. There are 26 components at each grid and there are not fully coupled in terms of stiff functions. Mutual coupling is among about 6 components. I would prefer not using matrix-free since the Jacobina is not difficult to calculate and only up to 10 non-zeros at each row. I'll try DMDASetBlockFills() or DMDASetBlockFillsSparse() and see how they can reduce the memory usage. Jiannan From: Barry Smith > Sent: Tuesday, September 6, 2022 11:33 PM To: Tu, Jiannan > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Using matrix-free with KSP CAUTION: This email was sent from outside the UMass Lowell network. On Sep 6, 2022, at 11:00 PM, Tu, Jiannan > wrote: I am using TS IMEX to solve a large DAE system. The DAE is obtained by applying finite FV method to 3-D multi-specie ion/neutral fluids equations with magnetic induction equation. The Jacobian for stiff part is formed by using MatSetValuesStencil(). The Jacobian matrix is very sparse, no more than 10 non-zeros on each row. MatSetValuesStencil requires local to global mapping by calling ISLocaltoGlobalMapping(). Could you please instruct me how to use local to global mapping? DMDA automatically sets up the ISLocaltoGlobalMapping() so you should not need to. I also tried using DMCreateMatrix() to create the Jacobian. While local to global mapping is not necessary, the matrix takes too much memory and requires 64-bit indices. I would prefer to take the advantage of sparsity of the Jacobian, pre-allocate the matrix to use as less as possible memory so that the code can be run on a multi-core desktop. If using matrix-free does not work for you because the linear solves do not converge or converge too slowly. Then you might be able to decrease the memory used in the matrix. The DMCreateMatrix() does take advantage of sparsity and tries to preallocate only what it needs. Often what it preallocates is the best possible, but for multicomponent problems it has to assume there is full coupling within all the degrees of freedom that live at each at each grid point. How many components live at each grid point in your model and are they not all coupled to each other in the equations? If they are not fully coupled you can use either DMDASetBlockFills() or DMDASetBlockFillsSparse() to indicate the reduced coupling that actually exists for you model. Barry Thank you very much for your advice. Jiannan -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Mon Sep 12 08:31:09 2022 From: bsmith at petsc.dev (Barry Smith) Date: Mon, 12 Sep 2022 09:31:09 -0400 Subject: [petsc-users] Using matrix-free with KSP In-Reply-To: References: <31AF5BF6-8AC4-4A76-9235-9C0E1BB3EE16@petsc.dev> <7F5F237F-B73B-447E-A113-B46C697D4C37@petsc.dev> <79BC638A-1064-42A1-AE04-40ACAA82323A@petsc.dev> <12206964-1F03-45CE-9FDB-EC4B244CD1F8@petsc.dev> <15DD8C9E-C3A5-4758-9A1F-B9FA28B35825@petsc.dev> <21002A96-7F12-452D-94D1-A3B8265D982A@petsc.dev> Message-ID: I am not able to visualize the pattern you refer to but I am pretty sure it is not a type of connectivity that DMDA handles by default since DMDA is for relatively simply structured meshes. Here is what I suggest. The routine that does the preallocation for DMDA that you are using is in src/dm/impls/da/fdda.c called DMCreateMatrix_DA_3d_MPIAIJ_Fill(). On each MPI rank it determines the connectivity for each local grid vertex (calledthe variable row in the routine) with all the other grid vertices (these are stored temporarily in the array cols[]). The call to MatPreallocateSetLocal() takes this coupling information and stores it temporarily. The calls to PetscCall(MatSeqAIJSetPreallocation(J, 0, dnz)); PetscCall(MatMPIAIJSetPreallocation(J, 0, dnz, 0, onz)); MatPreallocateEnd(dnz, onz); PetscCall(MatSetLocalToGlobalMapping(J, ltog, ltog)); at the end of the loop over the local grid vertices then provide the information to the matrix so it now has the correct preallocation and correct local to global mapping needed by MatSetValuesStencil() The same loop structure is then called again where it inserts zero values into the matrix at all the correctly preallocated locations. You can copy this routine and modify it slightly to include the "special coupling" that your problem requires. I do not understand your formula for kcm so cannot tell you exactly what you need to change but it may correspond to remapping the kk values at the extremes of the loop for (kk = kstart; kk < kend + 1; kk++) { so that the values nc * (slot + ii + gnx * jj + gnx * gny * kk) correspond to the correct coupling location. I recommend drawing a picture of a case with a very small size for nx,ny, and nz so you can see exactly how the mapping takes place and trace through your modified code that it does what you need. Barry > On Sep 11, 2022, at 11:01 PM, Tu, Jiannan wrote: > > Barry, > > Creating DNDA with periodic boundary BC and using km=k-1; kp=k+1 even at the BCs solve the ?augment out of range? problem for inserting elements at theae boundaries of k-index. Now the problem is due to wrapping around the north pole when j=0 and jm = -1 is reset as j=0 but on the other side of the pole with k-index replacing by kcm = [kmax+1)/2\ % (kmax+1). In my case, the ?Augment out of range? happens at global row/column (26, 5265026), exactly at wrapping jm=-1 to j=0 to the other side. > > I am thinking about how to properly use ghost grid j = -1 by setting appropriate BC there and inserting at that location without wrapping. > > Jiannan > > From: Barry Smith > > Sent: Sunday, September 11, 2022 12:10 PM > To: Tu, Jiannan > > Cc: petsc-users > > Subject: Re: [petsc-users] Using matrix-free with KSP > > CAUTION: This email was sent from outside the UMass Lowell network. > > > > > On Sep 11, 2022, at 9:56 AM, Tu, Jiannan > wrote: > > Barry, > > Thank you. > > Yes, the DMDA is created with periodic BC. But it might not be the periodic BC since in azimuthal direction, the ghost grid -1 is actually grid Nmax, and Nmax+1 is grid 0. > > Consider the following, one has n points from 0 to n-1 where n = 4, so I think with your definition of Nmax, Nmax is 3, Nmax + 1 is 4, > > > -1 0 1 2 3 4 > + * * * * + > > Thus there is one grid point at each end. The * represent regular grid points and the + ghost locations. Is this your situation? > > This is what we call periodic with a stencil width of 1 with DMDA and the DMCreateMatrix() should automatically provide the correct preallocation and nonzero structure. > > Can you please do a MatView() on the result from DMCreateMatrix() for a very small grid before you set any values and you should see the locations for the coupling across the periodic boundary. > > It sounds like you have a more complicated cross periodic coupling, we need to understand it exactly so we can determine how to proceed. > > Barry > > > > > Still the problem is how to properly pre-allocate matrix. Otherwise either ?Augument out of range? error or code stuck at inserting values if MatSetOption() is called. > > Jiannan > > From: Barry Smith > > Sent: Sunday, September 11, 2022 12:10 AM > To: Tu, Jiannan > > Cc: petsc-users > > Subject: Re: [petsc-users] Using matrix-free with KSP > > CAUTION: This email was sent from outside the UMass Lowell network. > > > Have you created the DMDA with the periodic boundary conditions arguments? Or are your boundary conditions not simply periodic? > > Compare DMCreateMatrix_DA_3d_MPIAIJ() and DMCreateMatrix_DA_3d_MPIAIJ_Fill() in src/dm/impls/da/fdda.c I think they both attempt to handle the periodic case. > > Barry > > > > > > > On Sep 10, 2022, at 10:14 PM, Tu, Jiannan > wrote: > > Barry, > > I think I understand how to set values for the intra-grid coupling. But I am not clear about the way to set values for inter-grid coupling. When a component couples to itself at above, below, right, left, front and back grids, diagonal is set as 1 for the second array of DMDASetBlockFills(). How about if involving more grids? How does DMDA know where to allocate memory? In my case, a component couples to itself at the other end of the grid because of periodic boundary condition. There is an error message ?Augment out of range . Insert new nonzero at global row/column (26, 520526) into matrix? because of this far away coupling. The column index 520526 is related to the other end of the grid in azimuthal direction. > > Thanks, > Jiannan > > > > From: Barry Smith > > Sent: Saturday, September 10, 2022 1:10 PM > To: Tu, Jiannan > > Cc: petsc-users > > Subject: Re: [petsc-users] Using matrix-free with KSP > > CAUTION: This email was sent from outside the UMass Lowell network. > > > If you are using DMCreateMatrix() then MatMPIAIJSetPreallocation is not needed, it is automatically handled by the DM. > > It sounds like that the values you set in DMDASetBlockFills() did not include all the coupling, hence the preallocation was wrong, hence the time to insert values was too long. > > Barry > > > > > > On Sep 10, 2022, at 8:16 AM, Tu, Jiannan > wrote: > > Just found MatMPIAIJSetPreallocation is needed. Now Jacobin insertion and assembly is fast. > > From: Barry Smith > > Sent: Thursday, September 8, 2022 9:53 PM > To: Tu, Jiannan > > Subject: Re: [petsc-users] Using matrix-free with KSP > > CAUTION: This email was sent from outside the UMass Lowell network. > > > > > > > > On Sep 8, 2022, at 8:34 PM, Tu, Jiannan > wrote: > > Barry, > > Thank you. > > I set up these two arrays according to the non-zero pattern of the Jacobian. The DMDASetBlockFills() is then called after the call to DMCreateMatrix(da, &A). > > Absolutely not! It must be called before DMCreateMatrix() so that the DM knows how to construct exactly the matrix you need. > > > > > > > Nevertheless, the program execution is still killed with exit code 9, which I assume is due to overuse of the memory. My desktop machine has 32 GB RAM. > > I ran the code with a smaller mesh, 100x45x90, the result is the same. With that number of grids, 26 unknowns, and up to 10 non-zero at each matrix row, the estimated memory to store 100x45x90x26x10 elements should be less than 1 GB (double precision). I am wondering why the matrix still takes too much memory. Maybe I have to use matrix-free? > > Thank you, > Jiannan > > From: Barry Smith > > Sent: Thursday, September 8, 2022 4:19 PM > To: Tu, Jiannan > > Subject: Re: [petsc-users] Using matrix-free with KSP > > CAUTION: This email was sent from outside the UMass Lowell network. > > > > > > > > > On Sep 8, 2022, at 3:39 PM, Tu, Jiannan > wrote: > > Barry, > > Thank you very much for the detailed description. > > So the first array, dfill, is for intra grids and the second one, ofill is for the inter-grid coupling? > > In the case for inter-grid coupling, a component is only coupled to itself, say x in your example is coupled to itself on above, below, right, left, front, and back, how to set values for the second array? > > Then the off-diagonal one would just have values on its diagonal. > > > > > > > Jiannan > > From: Barry Smith > Sent: Thursday, September 8, 2022 2:12 PM > To: Tu, Jiannan > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Using matrix-free with KSP > > CAUTION: This email was sent from outside the UMass Lowell network. > > > > > > > > > > On Sep 8, 2022, at 1:47 PM, Tu, Jiannan > wrote: > > Barry, > > Thank you very much. > > DMDASetBlockFills() needs two input arrays specifying sparsity pattern. In my case, I have 26 components at each grid. Is it correct that I need to have two arrays of 26x26 elements with value 1 for coupling and 0 for no coupling? > > Exactly > > > > > > > > Also how to determine when the value should be 1 or 0? > > That is problem dependent. Given paper-and-pencil you know for your stencil which components are coupled with other components at each grid point and which components are coupled with components at neighboring grid points. > > Note the second array contains information about all the neighboring grid points. Above, below, right, left, front back. Normally the coupling is symmetric so the nonzero structures of those 6 blocks are identical. If, for you particular problem, the nonzero structures of the six are not identical you use the union of the nonzero structures of the six of them. > > > 2d example say you have x,y,z at each grid point and the inter-point coupling is x is connected to y and y is connect to x then the first array is > > [ 1 1 0 ; 1 1; 0 ; 0 0 1]; the zeros are because x is not coupled to z and y is not coupled to z. > > For intra grid points, say x is coupled to z and itself, z is coupled to x and itself and y is not coupled to itself then > > [1 0 1; 0 0 0; 1 0 1]; the middle row is zero because y is not coupled to anything and the zeros in the first and last are because x and z are not coupled to y. > > > > > > > > > > Each component involves 7 stencils (i-1, j, k), (I, j, k), (i+1, j, k), (I, j-1, k), (I, j+1, k), (I, j, k-1), and (I, j, k+1), and couples with several other components. > > Jiannan > > > > Sent from Mail for Windows > > From: Barry Smith > Sent: Wednesday, September 7, 2022 11:53 AM > To: Tu, Jiannan > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Using matrix-free with KSP > > CAUTION: This email was sent from outside the UMass Lowell network. > > > DMDASetBlockFills() or DMDASetBlockFillsSparse() (they are just different ways of providing the same information) will help you here enormously, your type of case is exactly what they were designed for. > > Barry > > > > > > > > > > On Sep 7, 2022, at 10:47 AM, Tu, Jiannan > wrote: > > Barry and Hong, > > Thank you. > > There are 26 components at each grid and there are not fully coupled in terms of stiff functions. Mutual coupling is among about 6 components. > > I would prefer not using matrix-free since the Jacobina is not difficult to calculate and only up to 10 non-zeros at each row. I'll try DMDASetBlockFills() or DMDASetBlockFillsSparse() and see how they can reduce the memory usage. > > Jiannan > > From: Barry Smith > > Sent: Tuesday, September 6, 2022 11:33 PM > To: Tu, Jiannan > > Cc: petsc-users at mcs.anl.gov > > Subject: Re: [petsc-users] Using matrix-free with KSP > > CAUTION: This email was sent from outside the UMass Lowell network. > > > > > > > > > > > On Sep 6, 2022, at 11:00 PM, Tu, Jiannan > wrote: > > I am using TS IMEX to solve a large DAE system. The DAE is obtained by applying finite FV method to 3-D multi-specie ion/neutral fluids equations with magnetic induction equation. The Jacobian for stiff part is formed by using MatSetValuesStencil(). The Jacobian matrix is very sparse, no more than 10 non-zeros on each row. MatSetValuesStencil requires local to global mapping by calling ISLocaltoGlobalMapping(). Could you please instruct me how to use local to global mapping? > > DMDA automatically sets up the ISLocaltoGlobalMapping() so you should not need to. > > I also tried using DMCreateMatrix() to create the Jacobian. While local to global mapping is not necessary, the matrix takes too much memory and requires 64-bit indices. I would prefer to take the advantage of sparsity of the Jacobian, pre-allocate the matrix to use as less as possible memory so that the code can be run on a multi-core desktop. > > If using matrix-free does not work for you because the linear solves do not converge or converge too slowly. Then you might be able to decrease the memory used in the matrix. The DMCreateMatrix() does take advantage of sparsity and tries to preallocate only what it needs. Often what it preallocates is the best possible, but for multicomponent problems it has to assume there is full coupling within all the degrees of freedom that live at each at each grid point. How many components live at each grid point in your model and are they not all coupled to each other in the equations? If they are not fully coupled you can use either DMDASetBlockFills() or DMDASetBlockFillsSparse() to indicate the reduced coupling that actually exists for you model. > > Barry > > > > > > > > > > Thank you very much for your advice. > > Jiannan -------------- next part -------------- An HTML attachment was scrubbed... URL: From shatrov.oleg.a at gmail.com Mon Sep 12 10:02:56 2022 From: shatrov.oleg.a at gmail.com (Oleg Shatrov) Date: Mon, 12 Sep 2022 18:02:56 +0300 Subject: [petsc-users] Problems with PCMGSetInterpolation Message-ID: Hello! Can somebody explain to me what I am doing wrong here? I took tutorial 28 from KSP folder and modified it a little in order to create manual DM interpolations and set them to PC. When i run the application I receive an error: [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Must call DMShellSetGlobalVector() or DMShellSetCreateGlobalVector() [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting. [0]PETSC ERROR: Petsc Development GIT revision: v3.17.3-770-gceb9926454c GIT Date: 2022-07-15 21:35:25 +0000 [0]PETSC ERROR: ./a.out on a named MacBook-Pro-Oleg.local by olegshatrov Mon Sep 12 17:53:13 2022 [0]PETSC ERROR: Configure options --prefix=/usr/local CFLAGS="-m64 -O0 -g" CXXFLAGS="-m64 -O0 -g" FCFLAGS="-m64 -O0 -g -ffree-line-length-none" --with-mem-debug=1 --with-debugging=1 --with-mpi-dir=/usr/local/ --with-moab=true [0]PETSC ERROR: #1 DMCreateGlobalVector_Shell() at /Users/olegshatrov/Documents/GitHub/petsc/src/dm/impls/shell/dmshell.c:206 [0]PETSC ERROR: #2 DMCreateGlobalVector() at /Users/olegshatrov/Documents/GitHub/petsc/src/dm/interface/dm.c:998 [0]PETSC ERROR: #3 DMGetGlobalVector() at /Users/olegshatrov/Documents/GitHub/petsc/src/dm/interface/dmget.c:161 [0]PETSC ERROR: #4 KSPCreateVecs() at /Users/olegshatrov/Documents/GitHub/petsc/src/ksp/ksp/interface/iterativ.c:1616 [0]PETSC ERROR: #5 PCSetUp_MG() at /Users/olegshatrov/Documents/GitHub/petsc/src/ksp/pc/impls/mg/mg.c:1129 [0]PETSC ERROR: #6 PCSetUp() at /Users/olegshatrov/Documents/GitHub/petsc/src/ksp/pc/interface/precon.c:991 [0]PETSC ERROR: #7 KSPSetUp() at /Users/olegshatrov/Documents/GitHub/petsc/src/ksp/ksp/interface/itfunc.c:401 [0]PETSC ERROR: #8 main() at ex28.c:67 [0]PETSC ERROR: No PETSc Option Table entries [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov---------- -------------------------------------------------------------------------- And this is strange, since I did not provide any Shell DM. When i comment out lines with setting interpolation matrix to PCMG everything works fine: [image: image.png] I am testing on MacOS 12.3. MPI: Open MPI: 4.1.4 gcc version 12.1.0 Program is compiled with mpicc ex28.c -I /usr/local/include/ -L /usr/local/lib -lpetsc Thanks in advance, Oleg -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image.png Type: image/png Size: 88252 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ex28.c Type: application/octet-stream Size: 4630 bytes Desc: not available URL: From bsmith at petsc.dev Mon Sep 12 11:23:06 2022 From: bsmith at petsc.dev (Barry Smith) Date: Mon, 12 Sep 2022 12:23:06 -0400 Subject: [petsc-users] Problems with PCMGSetInterpolation In-Reply-To: References: Message-ID: <640365F5-B65B-4910-B61F-C3D1CB22305E@petsc.dev> In PCSetUp_MG() it is creating needed vectors for each level. In order to create the vectors it needs to have a "template" of the vectors. It tries several ways to get such a template. If you have provided an vector for each level with PCMGSetRhs() it uses VecDuplicate for the needed vectors. If you provided a matrix for each level with PCMGSetOperators() it uses the MatCreateVectors() using that matrix If you have not provided either of those it tries DMCreateGlobalVector() for the DM associated with that level, if there is no DM it defaults to DMShell and then errors. Since it errors this seems to indicate you did not provide an operator for each level, nor a right hand side vector. Please find attached a patch that may resolve the issue for. In the petsc directory do patch -p1 < mg.patch make libs then rerun your example. Please let us know if it resolves the problem and if not send the complete error message Barry > On Sep 12, 2022, at 11:02 AM, Oleg Shatrov wrote: > > Hello! > > Can somebody explain to me what I am doing wrong here? > I took tutorial 28 from KSP folder and modified it a little in order to create manual DM interpolations and set them to PC. > When i run the application I receive an error: > > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [0]PETSC ERROR: Must call DMShellSetGlobalVector() or DMShellSetCreateGlobalVector() > [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting. > [0]PETSC ERROR: Petsc Development GIT revision: v3.17.3-770-gceb9926454c GIT Date: 2022-07-15 21:35:25 +0000 > [0]PETSC ERROR: ./a.out on a named MacBook-Pro-Oleg.local by olegshatrov Mon Sep 12 17:53:13 2022 > [0]PETSC ERROR: Configure options --prefix=/usr/local CFLAGS="-m64 -O0 -g" CXXFLAGS="-m64 -O0 -g" FCFLAGS="-m64 -O0 -g -ffree-line-length-none" --with-mem-debug=1 --with-debugging=1 --with-mpi-dir=/usr/local/ --with-moab=true > [0]PETSC ERROR: #1 DMCreateGlobalVector_Shell() at /Users/olegshatrov/Documents/GitHub/petsc/src/dm/impls/shell/dmshell.c:206 > [0]PETSC ERROR: #2 DMCreateGlobalVector() at /Users/olegshatrov/Documents/GitHub/petsc/src/dm/interface/dm.c:998 > [0]PETSC ERROR: #3 DMGetGlobalVector() at /Users/olegshatrov/Documents/GitHub/petsc/src/dm/interface/dmget.c:161 > [0]PETSC ERROR: #4 KSPCreateVecs() at /Users/olegshatrov/Documents/GitHub/petsc/src/ksp/ksp/interface/iterativ.c:1616 > [0]PETSC ERROR: #5 PCSetUp_MG() at /Users/olegshatrov/Documents/GitHub/petsc/src/ksp/pc/impls/mg/mg.c:1129 > [0]PETSC ERROR: #6 PCSetUp() at /Users/olegshatrov/Documents/GitHub/petsc/src/ksp/pc/interface/precon.c:991 > [0]PETSC ERROR: #7 KSPSetUp() at /Users/olegshatrov/Documents/GitHub/petsc/src/ksp/ksp/interface/itfunc.c:401 > [0]PETSC ERROR: #8 main() at ex28.c:67 > [0]PETSC ERROR: No PETSc Option Table entries > [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov---------- > -------------------------------------------------------------------------- > > And this is strange, since I did not provide any Shell DM. > When i comment out lines with setting interpolation matrix to PCMG everything works fine: > > > I am testing on MacOS 12.3. > MPI: Open MPI: 4.1.4 > gcc version 12.1.0 > Program is compiled with mpicc ex28.c -I /usr/local/include/ -L /usr/local/lib -lpetsc > > > Thanks in advance, > Oleg > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: mg.patch Type: application/octet-stream Size: 1532 bytes Desc: not available URL: -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Mon Sep 12 13:22:17 2022 From: bsmith at petsc.dev (Barry Smith) Date: Mon, 12 Sep 2022 14:22:17 -0400 Subject: [petsc-users] Problems with PCMGSetInterpolation In-Reply-To: References: <640365F5-B65B-4910-B61F-C3D1CB22305E@petsc.dev> Message-ID: <4D61D9CE-C2AF-4228-BCEC-067EFEF3EA31@petsc.dev> It made some progress :-) It would be quickest if you just sent me your code (you can use petsc-maint at mcs.anl.gov if you don't want it public). > Somehow I thought that setting ComputeOperators and ComputeRHS will be enough.. I think it is failing because though you provide code to fill up the matrices you never provide the matrices; it may be implicit in the use of KSPSetComputeOperators that you provide a DM that can create the matrices for each of the levels automatically inside the PCSetUp_MG. Barry > On Sep 12, 2022, at 1:26 PM, Oleg Shatrov wrote: > > Unfortunately it still failed. > > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [0]PETSC ERROR: Must call DMShellSetGlobalVector() or DMShellSetCreateGlobalVector() > [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting. > [0]PETSC ERROR: Petsc Development GIT revision: v3.17.3-770-gceb9926454c GIT Date: 2022-07-15 21:35:25 +0000 > [0]PETSC ERROR: ./a.out on a named MacBook-Pro-Oleg.local by olegshatrov Mon Sep 12 19:59:34 2022 > [0]PETSC ERROR: Configure options --prefix=/usr/local CFLAGS="-m64 -O0 -g" CXXFLAGS="-m64 -O0 -g" FCFLAGS="-m64 -O0 -g -ffree-line-length-none" --with-mem-debug=1 --with-debugging=1 --with-mpi-dir=/usr/local/ --with-moab=true > [0]PETSC ERROR: #1 DMCreateGlobalVector_Shell() at /Users/olegshatrov/Documents/GitHub/petsc/src/dm/impls/shell/dmshell.c:206 > [0]PETSC ERROR: #2 DMCreateGlobalVector() at /Users/olegshatrov/Documents/GitHub/petsc/src/dm/interface/dm.c:998 > [0]PETSC ERROR: #3 DMGetGlobalVector() at /Users/olegshatrov/Documents/GitHub/petsc/src/dm/interface/dmget.c:161 > [0]PETSC ERROR: #4 KSPCreateVecs() at /Users/olegshatrov/Documents/GitHub/petsc/src/ksp/ksp/interface/iterativ.c:1616 > [0]PETSC ERROR: #5 KSPSetWorkVecs() at /Users/olegshatrov/Documents/GitHub/petsc/src/ksp/ksp/interface/iterativ.c:1684 > [0]PETSC ERROR: #6 KSPSetUp_Chebyshev() at /Users/olegshatrov/Documents/GitHub/petsc/src/ksp/ksp/impls/cheby/cheby.c:46 > [0]PETSC ERROR: #7 KSPSetUp() at /Users/olegshatrov/Documents/GitHub/petsc/src/ksp/ksp/interface/itfunc.c:362 > [0]PETSC ERROR: #8 PCSetUp_MG() at /Users/olegshatrov/Documents/GitHub/petsc/src/ksp/pc/impls/mg/mg.c:1189 > [0]PETSC ERROR: #9 PCSetUp() at /Users/olegshatrov/Documents/GitHub/petsc/src/ksp/pc/interface/precon.c:991 > [0]PETSC ERROR: #10 KSPSetUp() at /Users/olegshatrov/Documents/GitHub/petsc/src/ksp/ksp/interface/itfunc.c:401 > [0]PETSC ERROR: #11 main() at ex28.c:67 > [0]PETSC ERROR: No PETSc Option Table entries > [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov---------- > -------------------------------------------------------------------------- > > Somehow I thought that setting ComputeOperators and ComputeRHS will be enough.. > > Oleg. > > > ??, 12 ????. 2022 ?. ? 19:23, Barry Smith >: > > In PCSetUp_MG() it is creating needed vectors for each level. In order to create the vectors it needs to have a "template" of the vectors. It tries > several ways to get such a template. > > If you have provided an vector for each level with PCMGSetRhs() it uses VecDuplicate for the needed vectors. > > If you provided a matrix for each level with PCMGSetOperators() it uses the MatCreateVectors() using that matrix > > If you have not provided either of those it tries DMCreateGlobalVector() for the DM associated with that level, if there is no DM it defaults to DMShell and then errors. > > Since it errors this seems to indicate you did not provide an operator for each level, nor a right hand side vector. > > Please find attached a patch that may resolve the issue for. > > In the petsc directory do > patch -p1 < mg.patch > make libs > > then rerun your example. Please let us know if it resolves the problem and if not send the complete error message > > Barry > > > > >> On Sep 12, 2022, at 11:02 AM, Oleg Shatrov > wrote: >> >> Hello! >> >> Can somebody explain to me what I am doing wrong here? >> I took tutorial 28 from KSP folder and modified it a little in order to create manual DM interpolations and set them to PC. >> When i run the application I receive an error: >> >> [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >> [0]PETSC ERROR: Must call DMShellSetGlobalVector() or DMShellSetCreateGlobalVector() >> [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting. >> [0]PETSC ERROR: Petsc Development GIT revision: v3.17.3-770-gceb9926454c GIT Date: 2022-07-15 21:35:25 +0000 >> [0]PETSC ERROR: ./a.out on a named MacBook-Pro-Oleg.local by olegshatrov Mon Sep 12 17:53:13 2022 >> [0]PETSC ERROR: Configure options --prefix=/usr/local CFLAGS="-m64 -O0 -g" CXXFLAGS="-m64 -O0 -g" FCFLAGS="-m64 -O0 -g -ffree-line-length-none" --with-mem-debug=1 --with-debugging=1 --with-mpi-dir=/usr/local/ --with-moab=true >> [0]PETSC ERROR: #1 DMCreateGlobalVector_Shell() at /Users/olegshatrov/Documents/GitHub/petsc/src/dm/impls/shell/dmshell.c:206 >> [0]PETSC ERROR: #2 DMCreateGlobalVector() at /Users/olegshatrov/Documents/GitHub/petsc/src/dm/interface/dm.c:998 >> [0]PETSC ERROR: #3 DMGetGlobalVector() at /Users/olegshatrov/Documents/GitHub/petsc/src/dm/interface/dmget.c:161 >> [0]PETSC ERROR: #4 KSPCreateVecs() at /Users/olegshatrov/Documents/GitHub/petsc/src/ksp/ksp/interface/iterativ.c:1616 >> [0]PETSC ERROR: #5 PCSetUp_MG() at /Users/olegshatrov/Documents/GitHub/petsc/src/ksp/pc/impls/mg/mg.c:1129 >> [0]PETSC ERROR: #6 PCSetUp() at /Users/olegshatrov/Documents/GitHub/petsc/src/ksp/pc/interface/precon.c:991 >> [0]PETSC ERROR: #7 KSPSetUp() at /Users/olegshatrov/Documents/GitHub/petsc/src/ksp/ksp/interface/itfunc.c:401 >> [0]PETSC ERROR: #8 main() at ex28.c:67 >> [0]PETSC ERROR: No PETSc Option Table entries >> [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov ---------- >> -------------------------------------------------------------------------- >> >> And this is strange, since I did not provide any Shell DM. >> When i comment out lines with setting interpolation matrix to PCMG everything works fine: >> >> >> I am testing on MacOS 12.3. >> MPI: Open MPI: 4.1.4 >> gcc version 12.1.0 >> Program is compiled with mpicc ex28.c -I /usr/local/include/ -L /usr/local/lib -lpetsc >> >> >> Thanks in advance, >> Oleg >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Jiannan_Tu at uml.edu Mon Sep 12 14:45:40 2022 From: Jiannan_Tu at uml.edu (Tu, Jiannan) Date: Mon, 12 Sep 2022 19:45:40 +0000 Subject: [petsc-users] Using matrix-free with KSP In-Reply-To: References: <31AF5BF6-8AC4-4A76-9235-9C0E1BB3EE16@petsc.dev> <7F5F237F-B73B-447E-A113-B46C697D4C37@petsc.dev> <79BC638A-1064-42A1-AE04-40ACAA82323A@petsc.dev> <12206964-1F03-45CE-9FDB-EC4B244CD1F8@petsc.dev> <15DD8C9E-C3A5-4758-9A1F-B9FA28B35825@petsc.dev> <21002A96-7F12-452D-94D1-A3B8265D982A@petsc.dev> Message-ID: Simulation domain is in spherical coordinates. The grid along co-latitude direction is that j = 0 on one side of the pole and j = -1 (in fact index 0) on the other side of the pole. The longitude index k for j = 0, say is 0 degree then for j = -1 is 180 degrees. kcm accounts for this change in k-index when going from one side of the pole to another side. I'll see if I can solve the problem with the method you suggested. Thank you very much. Jiannan ________________________________ From: Barry Smith Sent: Monday, September 12, 2022 9:31 AM To: Tu, Jiannan Cc: petsc-users Subject: Re: [petsc-users] Using matrix-free with KSP CAUTION: This email was sent from outside the UMass Lowell network. I am not able to visualize the pattern you refer to but I am pretty sure it is not a type of connectivity that DMDA handles by default since DMDA is for relatively simply structured meshes. Here is what I suggest. The routine that does the preallocation for DMDA that you are using is in src/dm/impls/da/fdda.c called DMCreateMatrix_DA_3d_MPIAIJ_Fill(). On each MPI rank it determines the connectivity for each local grid vertex (calledthe variable row in the routine) with all the other grid vertices (these are stored temporarily in the array cols[]). The call to MatPreallocateSetLocal() takes this coupling information and stores it temporarily. The calls to PetscCall(MatSeqAIJSetPreallocation(J, 0, dnz)); PetscCall(MatMPIAIJSetPreallocation(J, 0, dnz, 0, onz)); MatPreallocateEnd(dnz, onz); PetscCall(MatSetLocalToGlobalMapping(J, ltog, ltog)); at the end of the loop over the local grid vertices then provide the information to the matrix so it now has the correct preallocation and correct local to global mapping needed by MatSetValuesStencil() The same loop structure is then called again where it inserts zero values into the matrix at all the correctly preallocated locations. You can copy this routine and modify it slightly to include the "special coupling" that your problem requires. I do not understand your formula for kcm so cannot tell you exactly what you need to change but it may correspond to remapping the kk values at the extremes of the loop for (kk = kstart; kk < kend + 1; kk++) { so that the values nc * (slot + ii + gnx * jj + gnx * gny * kk) correspond to the correct coupling location. I recommend drawing a picture of a case with a very small size for nx,ny, and nz so you can see exactly how the mapping takes place and trace through your modified code that it does what you need. Barry On Sep 11, 2022, at 11:01 PM, Tu, Jiannan > wrote: Barry, Creating DNDA with periodic boundary BC and using km=k-1; kp=k+1 even at the BCs solve the ?augment out of range? problem for inserting elements at theae boundaries of k-index. Now the problem is due to wrapping around the north pole when j=0 and jm = -1 is reset as j=0 but on the other side of the pole with k-index replacing by kcm = [kmax+1)/2\ % (kmax+1). In my case, the ?Augment out of range? happens at global row/column (26, 5265026), exactly at wrapping jm=-1 to j=0 to the other side. I am thinking about how to properly use ghost grid j = -1 by setting appropriate BC there and inserting at that location without wrapping. Jiannan From: Barry Smith > Sent: Sunday, September 11, 2022 12:10 PM To: Tu, Jiannan > Cc: petsc-users > Subject: Re: [petsc-users] Using matrix-free with KSP CAUTION: This email was sent from outside the UMass Lowell network. On Sep 11, 2022, at 9:56 AM, Tu, Jiannan > wrote: Barry, Thank you. Yes, the DMDA is created with periodic BC. But it might not be the periodic BC since in azimuthal direction, the ghost grid -1 is actually grid Nmax, and Nmax+1 is grid 0. Consider the following, one has n points from 0 to n-1 where n = 4, so I think with your definition of Nmax, Nmax is 3, Nmax + 1 is 4, -1 0 1 2 3 4 + * * * * + Thus there is one grid point at each end. The * represent regular grid points and the + ghost locations. Is this your situation? This is what we call periodic with a stencil width of 1 with DMDA and the DMCreateMatrix() should automatically provide the correct preallocation and nonzero structure. Can you please do a MatView() on the result from DMCreateMatrix() for a very small grid before you set any values and you should see the locations for the coupling across the periodic boundary. It sounds like you have a more complicated cross periodic coupling, we need to understand it exactly so we can determine how to proceed. Barry Still the problem is how to properly pre-allocate matrix. Otherwise either ?Augument out of range? error or code stuck at inserting values if MatSetOption() is called. Jiannan From: Barry Smith > Sent: Sunday, September 11, 2022 12:10 AM To: Tu, Jiannan > Cc: petsc-users > Subject: Re: [petsc-users] Using matrix-free with KSP CAUTION: This email was sent from outside the UMass Lowell network. Have you created the DMDA with the periodic boundary conditions arguments? Or are your boundary conditions not simply periodic? Compare DMCreateMatrix_DA_3d_MPIAIJ() and DMCreateMatrix_DA_3d_MPIAIJ_Fill() in src/dm/impls/da/fdda.c I think they both attempt to handle the periodic case. Barry On Sep 10, 2022, at 10:14 PM, Tu, Jiannan > wrote: Barry, I think I understand how to set values for the intra-grid coupling. But I am not clear about the way to set values for inter-grid coupling. When a component couples to itself at above, below, right, left, front and back grids, diagonal is set as 1 for the second array of DMDASetBlockFills(). How about if involving more grids? How does DMDA know where to allocate memory? In my case, a component couples to itself at the other end of the grid because of periodic boundary condition. There is an error message ?Augment out of range . Insert new nonzero at global row/column (26, 520526) into matrix? because of this far away coupling. The column index 520526 is related to the other end of the grid in azimuthal direction. Thanks, Jiannan From: Barry Smith > Sent: Saturday, September 10, 2022 1:10 PM To: Tu, Jiannan > Cc: petsc-users > Subject: Re: [petsc-users] Using matrix-free with KSP CAUTION: This email was sent from outside the UMass Lowell network. If you are using DMCreateMatrix() then MatMPIAIJSetPreallocation is not needed, it is automatically handled by the DM. It sounds like that the values you set in DMDASetBlockFills() did not include all the coupling, hence the preallocation was wrong, hence the time to insert values was too long. Barry On Sep 10, 2022, at 8:16 AM, Tu, Jiannan > wrote: Just found MatMPIAIJSetPreallocation is needed. Now Jacobin insertion and assembly is fast. From: Barry Smith > Sent: Thursday, September 8, 2022 9:53 PM To: Tu, Jiannan > Subject: Re: [petsc-users] Using matrix-free with KSP CAUTION: This email was sent from outside the UMass Lowell network. On Sep 8, 2022, at 8:34 PM, Tu, Jiannan > wrote: Barry, Thank you. I set up these two arrays according to the non-zero pattern of the Jacobian. The DMDASetBlockFills() is then called after the call to DMCreateMatrix(da, &A). Absolutely not! It must be called before DMCreateMatrix() so that the DM knows how to construct exactly the matrix you need. Nevertheless, the program execution is still killed with exit code 9, which I assume is due to overuse of the memory. My desktop machine has 32 GB RAM. I ran the code with a smaller mesh, 100x45x90, the result is the same. With that number of grids, 26 unknowns, and up to 10 non-zero at each matrix row, the estimated memory to store 100x45x90x26x10 elements should be less than 1 GB (double precision). I am wondering why the matrix still takes too much memory. Maybe I have to use matrix-free? Thank you, Jiannan From: Barry Smith > Sent: Thursday, September 8, 2022 4:19 PM To: Tu, Jiannan > Subject: Re: [petsc-users] Using matrix-free with KSP CAUTION: This email was sent from outside the UMass Lowell network. On Sep 8, 2022, at 3:39 PM, Tu, Jiannan > wrote: Barry, Thank you very much for the detailed description. So the first array, dfill, is for intra grids and the second one, ofill is for the inter-grid coupling? In the case for inter-grid coupling, a component is only coupled to itself, say x in your example is coupled to itself on above, below, right, left, front, and back, how to set values for the second array? Then the off-diagonal one would just have values on its diagonal. Jiannan From: Barry Smith Sent: Thursday, September 8, 2022 2:12 PM To: Tu, Jiannan Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] Using matrix-free with KSP CAUTION: This email was sent from outside the UMass Lowell network. On Sep 8, 2022, at 1:47 PM, Tu, Jiannan > wrote: Barry, Thank you very much. DMDASetBlockFills() needs two input arrays specifying sparsity pattern. In my case, I have 26 components at each grid. Is it correct that I need to have two arrays of 26x26 elements with value 1 for coupling and 0 for no coupling? Exactly Also how to determine when the value should be 1 or 0? That is problem dependent. Given paper-and-pencil you know for your stencil which components are coupled with other components at each grid point and which components are coupled with components at neighboring grid points. Note the second array contains information about all the neighboring grid points. Above, below, right, left, front back. Normally the coupling is symmetric so the nonzero structures of those 6 blocks are identical. If, for you particular problem, the nonzero structures of the six are not identical you use the union of the nonzero structures of the six of them. 2d example say you have x,y,z at each grid point and the inter-point coupling is x is connected to y and y is connect to x then the first array is [ 1 1 0 ; 1 1; 0 ; 0 0 1]; the zeros are because x is not coupled to z and y is not coupled to z. For intra grid points, say x is coupled to z and itself, z is coupled to x and itself and y is not coupled to itself then [1 0 1; 0 0 0; 1 0 1]; the middle row is zero because y is not coupled to anything and the zeros in the first and last are because x and z are not coupled to y. Each component involves 7 stencils (i-1, j, k), (I, j, k), (i+1, j, k), (I, j-1, k), (I, j+1, k), (I, j, k-1), and (I, j, k+1), and couples with several other components. Jiannan Sent from Mail for Windows From: Barry Smith Sent: Wednesday, September 7, 2022 11:53 AM To: Tu, Jiannan Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] Using matrix-free with KSP CAUTION: This email was sent from outside the UMass Lowell network. DMDASetBlockFills() or DMDASetBlockFillsSparse() (they are just different ways of providing the same information) will help you here enormously, your type of case is exactly what they were designed for. Barry On Sep 7, 2022, at 10:47 AM, Tu, Jiannan > wrote: Barry and Hong, Thank you. There are 26 components at each grid and there are not fully coupled in terms of stiff functions. Mutual coupling is among about 6 components. I would prefer not using matrix-free since the Jacobina is not difficult to calculate and only up to 10 non-zeros at each row. I'll try DMDASetBlockFills() or DMDASetBlockFillsSparse() and see how they can reduce the memory usage. Jiannan From: Barry Smith > Sent: Tuesday, September 6, 2022 11:33 PM To: Tu, Jiannan > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Using matrix-free with KSP CAUTION: This email was sent from outside the UMass Lowell network. On Sep 6, 2022, at 11:00 PM, Tu, Jiannan > wrote: I am using TS IMEX to solve a large DAE system. The DAE is obtained by applying finite FV method to 3-D multi-specie ion/neutral fluids equations with magnetic induction equation. The Jacobian for stiff part is formed by using MatSetValuesStencil(). The Jacobian matrix is very sparse, no more than 10 non-zeros on each row. MatSetValuesStencil requires local to global mapping by calling ISLocaltoGlobalMapping(). Could you please instruct me how to use local to global mapping? DMDA automatically sets up the ISLocaltoGlobalMapping() so you should not need to. I also tried using DMCreateMatrix() to create the Jacobian. While local to global mapping is not necessary, the matrix takes too much memory and requires 64-bit indices. I would prefer to take the advantage of sparsity of the Jacobian, pre-allocate the matrix to use as less as possible memory so that the code can be run on a multi-core desktop. If using matrix-free does not work for you because the linear solves do not converge or converge too slowly. Then you might be able to decrease the memory used in the matrix. The DMCreateMatrix() does take advantage of sparsity and tries to preallocate only what it needs. Often what it preallocates is the best possible, but for multicomponent problems it has to assume there is full coupling within all the degrees of freedom that live at each at each grid point. How many components live at each grid point in your model and are they not all coupled to each other in the equations? If they are not fully coupled you can use either DMDASetBlockFills() or DMDASetBlockFillsSparse() to indicate the reduced coupling that actually exists for you model. Barry Thank you very much for your advice. Jiannan -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Mon Sep 12 15:17:46 2022 From: bsmith at petsc.dev (Barry Smith) Date: Mon, 12 Sep 2022 16:17:46 -0400 Subject: [petsc-users] Using matrix-free with KSP In-Reply-To: References: <31AF5BF6-8AC4-4A76-9235-9C0E1BB3EE16@petsc.dev> <7F5F237F-B73B-447E-A113-B46C697D4C37@petsc.dev> <79BC638A-1064-42A1-AE04-40ACAA82323A@petsc.dev> <12206964-1F03-45CE-9FDB-EC4B244CD1F8@petsc.dev> <15DD8C9E-C3A5-4758-9A1F-B9FA28B35825@petsc.dev> <21002A96-7F12-452D-94D1-A3B8265D982A@petsc.dev> Message-ID: <22852407-23A2-4D15-9117-E1C4577996A9@petsc.dev> I understand in principle, I just cannot wrap my head around a sphere :-) > On Sep 12, 2022, at 3:45 PM, Tu, Jiannan wrote: > > Simulation domain is in spherical coordinates. The grid along co-latitude direction is that j = 0 on one side of the pole and j = -1 (in fact index 0) on the other side of the pole. The longitude index k for j = 0, say is 0 degree then for j = -1 is 180 degrees. kcm accounts for this change in k-index when going from one side of the pole to another side. > > I'll see if I can solve the problem with the method you suggested. > > Thank you very much. > Jiannan > From: Barry Smith > > Sent: Monday, September 12, 2022 9:31 AM > To: Tu, Jiannan > > Cc: petsc-users > > Subject: Re: [petsc-users] Using matrix-free with KSP > > CAUTION: This email was sent from outside the UMass Lowell network. > > > I am not able to visualize the pattern you refer to but I am pretty sure it is not a type of connectivity that DMDA handles by default since DMDA is for relatively simply structured meshes. Here is what I suggest. > > The routine that does the preallocation for DMDA that you are using is in src/dm/impls/da/fdda.c called DMCreateMatrix_DA_3d_MPIAIJ_Fill(). On each MPI rank it determines the connectivity for each local grid vertex (calledthe variable row in the routine) with all the other grid vertices (these are stored temporarily in the array cols[]). > The call to MatPreallocateSetLocal() takes this coupling information and stores it temporarily. The calls to > > PetscCall(MatSeqAIJSetPreallocation(J, 0, dnz)); > PetscCall(MatMPIAIJSetPreallocation(J, 0, dnz, 0, onz)); > MatPreallocateEnd(dnz, onz); > PetscCall(MatSetLocalToGlobalMapping(J, ltog, ltog)); > > at the end of the loop over the local grid vertices then provide the information to the matrix so it now has the correct preallocation and correct local to global mapping needed by MatSetValuesStencil() > > The same loop structure is then called again where it inserts zero values into the matrix at all the correctly preallocated locations. > > You can copy this routine and modify it slightly to include the "special coupling" that your problem requires. I do not understand your formula for kcm so cannot tell you exactly what you need to change but it may correspond to remapping the kk values at the extremes of the loop for (kk = kstart; kk < kend + 1; kk++) { so that the > values nc * (slot + ii + gnx * jj + gnx * gny * kk) correspond to the correct coupling location. I recommend drawing a picture of a case with a very small size for nx,ny, and nz so you can see exactly how the mapping takes place and trace through your modified code that it does what you need. > > Barry > > > > >> On Sep 11, 2022, at 11:01 PM, Tu, Jiannan > wrote: >> >> Barry, >> >> Creating DNDA with periodic boundary BC and using km=k-1; kp=k+1 even at the BCs solve the ?augment out of range? problem for inserting elements at theae boundaries of k-index. Now the problem is due to wrapping around the north pole when j=0 and jm = -1 is reset as j=0 but on the other side of the pole with k-index replacing by kcm = [kmax+1)/2\ % (kmax+1). In my case, the ?Augment out of range? happens at global row/column (26, 5265026), exactly at wrapping jm=-1 to j=0 to the other side. >> >> I am thinking about how to properly use ghost grid j = -1 by setting appropriate BC there and inserting at that location without wrapping. >> >> Jiannan >> >> From: Barry Smith > >> Sent: Sunday, September 11, 2022 12:10 PM >> To: Tu, Jiannan > >> Cc: petsc-users > >> Subject: Re: [petsc-users] Using matrix-free with KSP >> >> CAUTION: This email was sent from outside the UMass Lowell network. >> >> >> >> >> On Sep 11, 2022, at 9:56 AM, Tu, Jiannan > wrote: >> >> Barry, >> >> Thank you. >> >> Yes, the DMDA is created with periodic BC. But it might not be the periodic BC since in azimuthal direction, the ghost grid -1 is actually grid Nmax, and Nmax+1 is grid 0. >> >> Consider the following, one has n points from 0 to n-1 where n = 4, so I think with your definition of Nmax, Nmax is 3, Nmax + 1 is 4, >> >> >> -1 0 1 2 3 4 >> + * * * * + >> >> Thus there is one grid point at each end. The * represent regular grid points and the + ghost locations. Is this your situation? >> >> This is what we call periodic with a stencil width of 1 with DMDA and the DMCreateMatrix() should automatically provide the correct preallocation and nonzero structure. >> >> Can you please do a MatView() on the result from DMCreateMatrix() for a very small grid before you set any values and you should see the locations for the coupling across the periodic boundary. >> >> It sounds like you have a more complicated cross periodic coupling, we need to understand it exactly so we can determine how to proceed. >> >> Barry >> >> >> >> >> Still the problem is how to properly pre-allocate matrix. Otherwise either ?Augument out of range? error or code stuck at inserting values if MatSetOption() is called. >> >> Jiannan >> >> From: Barry Smith > >> Sent: Sunday, September 11, 2022 12:10 AM >> To: Tu, Jiannan > >> Cc: petsc-users > >> Subject: Re: [petsc-users] Using matrix-free with KSP >> >> CAUTION: This email was sent from outside the UMass Lowell network. >> >> >> Have you created the DMDA with the periodic boundary conditions arguments? Or are your boundary conditions not simply periodic? >> >> Compare DMCreateMatrix_DA_3d_MPIAIJ() and DMCreateMatrix_DA_3d_MPIAIJ_Fill() in src/dm/impls/da/fdda.c I think they both attempt to handle the periodic case. >> >> Barry >> >> >> >> >> >> >> On Sep 10, 2022, at 10:14 PM, Tu, Jiannan > wrote: >> >> Barry, >> >> I think I understand how to set values for the intra-grid coupling. But I am not clear about the way to set values for inter-grid coupling. When a component couples to itself at above, below, right, left, front and back grids, diagonal is set as 1 for the second array of DMDASetBlockFills(). How about if involving more grids? How does DMDA know where to allocate memory? In my case, a component couples to itself at the other end of the grid because of periodic boundary condition. There is an error message ?Augment out of range . Insert new nonzero at global row/column (26, 520526) into matrix? because of this far away coupling. The column index 520526 is related to the other end of the grid in azimuthal direction. >> >> Thanks, >> Jiannan >> >> >> >> From: Barry Smith > >> Sent: Saturday, September 10, 2022 1:10 PM >> To: Tu, Jiannan > >> Cc: petsc-users > >> Subject: Re: [petsc-users] Using matrix-free with KSP >> >> CAUTION: This email was sent from outside the UMass Lowell network. >> >> >> If you are using DMCreateMatrix() then MatMPIAIJSetPreallocation is not needed, it is automatically handled by the DM. >> >> It sounds like that the values you set in DMDASetBlockFills() did not include all the coupling, hence the preallocation was wrong, hence the time to insert values was too long. >> >> Barry >> >> >> >> >> >> On Sep 10, 2022, at 8:16 AM, Tu, Jiannan > wrote: >> >> Just found MatMPIAIJSetPreallocation is needed. Now Jacobin insertion and assembly is fast. >> >> From: Barry Smith > >> Sent: Thursday, September 8, 2022 9:53 PM >> To: Tu, Jiannan > >> Subject: Re: [petsc-users] Using matrix-free with KSP >> >> CAUTION: This email was sent from outside the UMass Lowell network. >> >> >> >> >> >> >> >> On Sep 8, 2022, at 8:34 PM, Tu, Jiannan > wrote: >> >> Barry, >> >> Thank you. >> >> I set up these two arrays according to the non-zero pattern of the Jacobian. The DMDASetBlockFills() is then called after the call to DMCreateMatrix(da, &A). >> >> Absolutely not! It must be called before DMCreateMatrix() so that the DM knows how to construct exactly the matrix you need. >> >> >> >> >> >> >> Nevertheless, the program execution is still killed with exit code 9, which I assume is due to overuse of the memory. My desktop machine has 32 GB RAM. >> >> I ran the code with a smaller mesh, 100x45x90, the result is the same. With that number of grids, 26 unknowns, and up to 10 non-zero at each matrix row, the estimated memory to store 100x45x90x26x10 elements should be less than 1 GB (double precision). I am wondering why the matrix still takes too much memory. Maybe I have to use matrix-free? >> >> Thank you, >> Jiannan >> >> From: Barry Smith > >> Sent: Thursday, September 8, 2022 4:19 PM >> To: Tu, Jiannan > >> Subject: Re: [petsc-users] Using matrix-free with KSP >> >> CAUTION: This email was sent from outside the UMass Lowell network. >> >> >> >> >> >> >> >> >> On Sep 8, 2022, at 3:39 PM, Tu, Jiannan > wrote: >> >> Barry, >> >> Thank you very much for the detailed description. >> >> So the first array, dfill, is for intra grids and the second one, ofill is for the inter-grid coupling? >> >> In the case for inter-grid coupling, a component is only coupled to itself, say x in your example is coupled to itself on above, below, right, left, front, and back, how to set values for the second array? >> >> Then the off-diagonal one would just have values on its diagonal. >> >> >> >> >> >> >> Jiannan >> >> From: Barry Smith >> Sent: Thursday, September 8, 2022 2:12 PM >> To: Tu, Jiannan >> Cc: petsc-users at mcs.anl.gov >> Subject: Re: [petsc-users] Using matrix-free with KSP >> >> CAUTION: This email was sent from outside the UMass Lowell network. >> >> >> >> >> >> >> >> >> >> On Sep 8, 2022, at 1:47 PM, Tu, Jiannan > wrote: >> >> Barry, >> >> Thank you very much. >> >> DMDASetBlockFills() needs two input arrays specifying sparsity pattern. In my case, I have 26 components at each grid. Is it correct that I need to have two arrays of 26x26 elements with value 1 for coupling and 0 for no coupling? >> >> Exactly >> >> >> >> >> >> >> >> Also how to determine when the value should be 1 or 0? >> >> That is problem dependent. Given paper-and-pencil you know for your stencil which components are coupled with other components at each grid point and which components are coupled with components at neighboring grid points. >> >> Note the second array contains information about all the neighboring grid points. Above, below, right, left, front back. Normally the coupling is symmetric so the nonzero structures of those 6 blocks are identical. If, for you particular problem, the nonzero structures of the six are not identical you use the union of the nonzero structures of the six of them. >> >> >> 2d example say you have x,y,z at each grid point and the inter-point coupling is x is connected to y and y is connect to x then the first array is >> >> [ 1 1 0 ; 1 1; 0 ; 0 0 1]; the zeros are because x is not coupled to z and y is not coupled to z. >> >> For intra grid points, say x is coupled to z and itself, z is coupled to x and itself and y is not coupled to itself then >> >> [1 0 1; 0 0 0; 1 0 1]; the middle row is zero because y is not coupled to anything and the zeros in the first and last are because x and z are not coupled to y. >> >> >> >> >> >> >> >> >> >> Each component involves 7 stencils (i-1, j, k), (I, j, k), (i+1, j, k), (I, j-1, k), (I, j+1, k), (I, j, k-1), and (I, j, k+1), and couples with several other components. >> >> Jiannan >> >> >> >> Sent from Mail for Windows >> >> From: Barry Smith >> Sent: Wednesday, September 7, 2022 11:53 AM >> To: Tu, Jiannan >> Cc: petsc-users at mcs.anl.gov >> Subject: Re: [petsc-users] Using matrix-free with KSP >> >> CAUTION: This email was sent from outside the UMass Lowell network. >> >> >> DMDASetBlockFills() or DMDASetBlockFillsSparse() (they are just different ways of providing the same information) will help you here enormously, your type of case is exactly what they were designed for. >> >> Barry >> >> >> >> >> >> >> >> >> >> On Sep 7, 2022, at 10:47 AM, Tu, Jiannan > wrote: >> >> Barry and Hong, >> >> Thank you. >> >> There are 26 components at each grid and there are not fully coupled in terms of stiff functions. Mutual coupling is among about 6 components. >> >> I would prefer not using matrix-free since the Jacobina is not difficult to calculate and only up to 10 non-zeros at each row. I'll try DMDASetBlockFills() or DMDASetBlockFillsSparse() and see how they can reduce the memory usage. >> >> Jiannan >> >> From: Barry Smith > >> Sent: Tuesday, September 6, 2022 11:33 PM >> To: Tu, Jiannan > >> Cc: petsc-users at mcs.anl.gov > >> Subject: Re: [petsc-users] Using matrix-free with KSP >> >> CAUTION: This email was sent from outside the UMass Lowell network. >> >> >> >> >> >> >> >> >> >> >> On Sep 6, 2022, at 11:00 PM, Tu, Jiannan > wrote: >> >> I am using TS IMEX to solve a large DAE system. The DAE is obtained by applying finite FV method to 3-D multi-specie ion/neutral fluids equations with magnetic induction equation. The Jacobian for stiff part is formed by using MatSetValuesStencil(). The Jacobian matrix is very sparse, no more than 10 non-zeros on each row. MatSetValuesStencil requires local to global mapping by calling ISLocaltoGlobalMapping(). Could you please instruct me how to use local to global mapping? >> >> DMDA automatically sets up the ISLocaltoGlobalMapping() so you should not need to. >> >> I also tried using DMCreateMatrix() to create the Jacobian. While local to global mapping is not necessary, the matrix takes too much memory and requires 64-bit indices. I would prefer to take the advantage of sparsity of the Jacobian, pre-allocate the matrix to use as less as possible memory so that the code can be run on a multi-core desktop. >> >> If using matrix-free does not work for you because the linear solves do not converge or converge too slowly. Then you might be able to decrease the memory used in the matrix. The DMCreateMatrix() does take advantage of sparsity and tries to preallocate only what it needs. Often what it preallocates is the best possible, but for multicomponent problems it has to assume there is full coupling within all the degrees of freedom that live at each at each grid point. How many components live at each grid point in your model and are they not all coupled to each other in the equations? If they are not fully coupled you can use either DMDASetBlockFills() or DMDASetBlockFillsSparse() to indicate the reduced coupling that actually exists for you model. >> >> Barry >> >> >> >> >> >> >> >> >> >> Thank you very much for your advice. >> >> Jiannan -------------- next part -------------- An HTML attachment was scrubbed... URL: From Jiannan_Tu at uml.edu Mon Sep 12 23:03:27 2022 From: Jiannan_Tu at uml.edu (Tu, Jiannan) Date: Tue, 13 Sep 2022 04:03:27 +0000 Subject: [petsc-users] Using matrix-free with KSP In-Reply-To: <22852407-23A2-4D15-9117-E1C4577996A9@petsc.dev> References: <31AF5BF6-8AC4-4A76-9235-9C0E1BB3EE16@petsc.dev> <7F5F237F-B73B-447E-A113-B46C697D4C37@petsc.dev> <79BC638A-1064-42A1-AE04-40ACAA82323A@petsc.dev> <12206964-1F03-45CE-9FDB-EC4B244CD1F8@petsc.dev> <15DD8C9E-C3A5-4758-9A1F-B9FA28B35825@petsc.dev> <21002A96-7F12-452D-94D1-A3B8265D982A@petsc.dev> <22852407-23A2-4D15-9117-E1C4577996A9@petsc.dev> Message-ID: Indeed, it is not a standard grid structure that DMDA handles by default. Normally, a grid (k, j, i) has connectivity at grid (k, j-1, i). I understand if j > 0, DMDA correctly allocates memory for it. When j = 0, j-1 = -1, this (k, -1, i) grid is actually the grid across the pole with k-index = (k+a3/2) % a3, where a3 is the number of grids for k-index. Attached figure illustrates a pair of such grids. Assume red grid is at (k = 10, j = 0, i), its corresponding grid *blue dot) cross the pole is at (k = 55, j =0, i) if a3 = 90. The question is how to make DMDA allocate memory at such local column stencils like (col.k = 55, col.j = 0, col.i = i, col.c = 0), instead of the column stencil (col.k = 10, col.j = -1, col.i = i, col.c = 0). I tried insert at stencils like (col.k = 10, col.j = -1, col.i = i, col.c = 0). The matrix assemble has no issues but TSSolve failed. I guess probably due to wrong column locations of those Jacobian elements. Thank you, Jiannan From: Barry Smith Sent: Monday, September 12, 2022 4:18 PM To: Tu, Jiannan Cc: petsc-users Subject: Re: [petsc-users] Using matrix-free with KSP CAUTION: This email was sent from outside the UMass Lowell network. I understand in principle, I just cannot wrap my head around a sphere :-) On Sep 12, 2022, at 3:45 PM, Tu, Jiannan > wrote: Simulation domain is in spherical coordinates. The grid along co-latitude direction is that j = 0 on one side of the pole and j = -1 (in fact index 0) on the other side of the pole. The longitude index k for j = 0, say is 0 degree then for j = -1 is 180 degrees. kcm accounts for this change in k-index when going from one side of the pole to another side. I'll see if I can solve the problem with the method you suggested. Thank you very much. Jiannan ________________________________ From: Barry Smith > Sent: Monday, September 12, 2022 9:31 AM To: Tu, Jiannan > Cc: petsc-users > Subject: Re: [petsc-users] Using matrix-free with KSP CAUTION: This email was sent from outside the UMass Lowell network. I am not able to visualize the pattern you refer to but I am pretty sure it is not a type of connectivity that DMDA handles by default since DMDA is for relatively simply structured meshes. Here is what I suggest. The routine that does the preallocation for DMDA that you are using is in src/dm/impls/da/fdda.c called DMCreateMatrix_DA_3d_MPIAIJ_Fill(). On each MPI rank it determines the connectivity for each local grid vertex (calledthe variable row in the routine) with all the other grid vertices (these are stored temporarily in the array cols[]). The call to MatPreallocateSetLocal() takes this coupling information and stores it temporarily. The calls to PetscCall(MatSeqAIJSetPreallocation(J, 0, dnz)); PetscCall(MatMPIAIJSetPreallocation(J, 0, dnz, 0, onz)); MatPreallocateEnd(dnz, onz); PetscCall(MatSetLocalToGlobalMapping(J, ltog, ltog)); at the end of the loop over the local grid vertices then provide the information to the matrix so it now has the correct preallocation and correct local to global mapping needed by MatSetValuesStencil() The same loop structure is then called again where it inserts zero values into the matrix at all the correctly preallocated locations. You can copy this routine and modify it slightly to include the "special coupling" that your problem requires. I do not understand your formula for kcm so cannot tell you exactly what you need to change but it may correspond to remapping the kk values at the extremes of the loop for (kk = kstart; kk < kend + 1; kk++) { so that the values nc * (slot + ii + gnx * jj + gnx * gny * kk) correspond to the correct coupling location. I recommend drawing a picture of a case with a very small size for nx,ny, and nz so you can see exactly how the mapping takes place and trace through your modified code that it does what you need. Barry On Sep 11, 2022, at 11:01 PM, Tu, Jiannan > wrote: Barry, Creating DNDA with periodic boundary BC and using km=k-1; kp=k+1 even at the BCs solve the "augment out of range" problem for inserting elements at theae boundaries of k-index. Now the problem is due to wrapping around the north pole when j=0 and jm = -1 is reset as j=0 but on the other side of the pole with k-index replacing by kcm = [kmax+1)/2\ % (kmax+1). In my case, the "Augment out of range" happens at global row/column (26, 5265026), exactly at wrapping jm=-1 to j=0 to the other side. I am thinking about how to properly use ghost grid j = -1 by setting appropriate BC there and inserting at that location without wrapping. Jiannan From: Barry Smith > Sent: Sunday, September 11, 2022 12:10 PM To: Tu, Jiannan > Cc: petsc-users > Subject: Re: [petsc-users] Using matrix-free with KSP CAUTION: This email was sent from outside the UMass Lowell network. On Sep 11, 2022, at 9:56 AM, Tu, Jiannan > wrote: Barry, Thank you. Yes, the DMDA is created with periodic BC. But it might not be the periodic BC since in azimuthal direction, the ghost grid -1 is actually grid Nmax, and Nmax+1 is grid 0. Consider the following, one has n points from 0 to n-1 where n = 4, so I think with your definition of Nmax, Nmax is 3, Nmax + 1 is 4, -1 0 1 2 3 4 + * * * * + Thus there is one grid point at each end. The * represent regular grid points and the + ghost locations. Is this your situation? This is what we call periodic with a stencil width of 1 with DMDA and the DMCreateMatrix() should automatically provide the correct preallocation and nonzero structure. Can you please do a MatView() on the result from DMCreateMatrix() for a very small grid before you set any values and you should see the locations for the coupling across the periodic boundary. It sounds like you have a more complicated cross periodic coupling, we need to understand it exactly so we can determine how to proceed. Barry Still the problem is how to properly pre-allocate matrix. Otherwise either "Augument out of range" error or code stuck at inserting values if MatSetOption() is called. Jiannan From: Barry Smith > Sent: Sunday, September 11, 2022 12:10 AM To: Tu, Jiannan > Cc: petsc-users > Subject: Re: [petsc-users] Using matrix-free with KSP CAUTION: This email was sent from outside the UMass Lowell network. Have you created the DMDA with the periodic boundary conditions arguments? Or are your boundary conditions not simply periodic? Compare DMCreateMatrix_DA_3d_MPIAIJ() and DMCreateMatrix_DA_3d_MPIAIJ_Fill() in src/dm/impls/da/fdda.c I think they both attempt to handle the periodic case. Barry On Sep 10, 2022, at 10:14 PM, Tu, Jiannan > wrote: Barry, I think I understand how to set values for the intra-grid coupling. But I am not clear about the way to set values for inter-grid coupling. When a component couples to itself at above, below, right, left, front and back grids, diagonal is set as 1 for the second array of DMDASetBlockFills(). How about if involving more grids? How does DMDA know where to allocate memory? In my case, a component couples to itself at the other end of the grid because of periodic boundary condition. There is an error message "Augment out of range . Insert new nonzero at global row/column (26, 520526) into matrix" because of this far away coupling. The column index 520526 is related to the other end of the grid in azimuthal direction. Thanks, Jiannan From: Barry Smith > Sent: Saturday, September 10, 2022 1:10 PM To: Tu, Jiannan > Cc: petsc-users > Subject: Re: [petsc-users] Using matrix-free with KSP CAUTION: This email was sent from outside the UMass Lowell network. If you are using DMCreateMatrix() then MatMPIAIJSetPreallocation is not needed, it is automatically handled by the DM. It sounds like that the values you set in DMDASetBlockFills() did not include all the coupling, hence the preallocation was wrong, hence the time to insert values was too long. Barry On Sep 10, 2022, at 8:16 AM, Tu, Jiannan > wrote: Just found MatMPIAIJSetPreallocation is needed. Now Jacobin insertion and assembly is fast. From: Barry Smith > Sent: Thursday, September 8, 2022 9:53 PM To: Tu, Jiannan > Subject: Re: [petsc-users] Using matrix-free with KSP CAUTION: This email was sent from outside the UMass Lowell network. On Sep 8, 2022, at 8:34 PM, Tu, Jiannan > wrote: Barry, Thank you. I set up these two arrays according to the non-zero pattern of the Jacobian. The DMDASetBlockFills() is then called after the call to DMCreateMatrix(da, &A). Absolutely not! It must be called before DMCreateMatrix() so that the DM knows how to construct exactly the matrix you need. Nevertheless, the program execution is still killed with exit code 9, which I assume is due to overuse of the memory. My desktop machine has 32 GB RAM. I ran the code with a smaller mesh, 100x45x90, the result is the same. With that number of grids, 26 unknowns, and up to 10 non-zero at each matrix row, the estimated memory to store 100x45x90x26x10 elements should be less than 1 GB (double precision). I am wondering why the matrix still takes too much memory. Maybe I have to use matrix-free? Thank you, Jiannan From: Barry Smith > Sent: Thursday, September 8, 2022 4:19 PM To: Tu, Jiannan > Subject: Re: [petsc-users] Using matrix-free with KSP CAUTION: This email was sent from outside the UMass Lowell network. On Sep 8, 2022, at 3:39 PM, Tu, Jiannan > wrote: Barry, Thank you very much for the detailed description. So the first array, dfill, is for intra grids and the second one, ofill is for the inter-grid coupling? In the case for inter-grid coupling, a component is only coupled to itself, say x in your example is coupled to itself on above, below, right, left, front, and back, how to set values for the second array? Then the off-diagonal one would just have values on its diagonal. Jiannan From: Barry Smith Sent: Thursday, September 8, 2022 2:12 PM To: Tu, Jiannan Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] Using matrix-free with KSP CAUTION: This email was sent from outside the UMass Lowell network. On Sep 8, 2022, at 1:47 PM, Tu, Jiannan > wrote: Barry, Thank you very much. DMDASetBlockFills() needs two input arrays specifying sparsity pattern. In my case, I have 26 components at each grid. Is it correct that I need to have two arrays of 26x26 elements with value 1 for coupling and 0 for no coupling? Exactly Also how to determine when the value should be 1 or 0? That is problem dependent. Given paper-and-pencil you know for your stencil which components are coupled with other components at each grid point and which components are coupled with components at neighboring grid points. Note the second array contains information about all the neighboring grid points. Above, below, right, left, front back. Normally the coupling is symmetric so the nonzero structures of those 6 blocks are identical. If, for you particular problem, the nonzero structures of the six are not identical you use the union of the nonzero structures of the six of them. 2d example say you have x,y,z at each grid point and the inter-point coupling is x is connected to y and y is connect to x then the first array is [ 1 1 0 ; 1 1; 0 ; 0 0 1]; the zeros are because x is not coupled to z and y is not coupled to z. For intra grid points, say x is coupled to z and itself, z is coupled to x and itself and y is not coupled to itself then [1 0 1; 0 0 0; 1 0 1]; the middle row is zero because y is not coupled to anything and the zeros in the first and last are because x and z are not coupled to y. Each component involves 7 stencils (i-1, j, k), (I, j, k), (i+1, j, k), (I, j-1, k), (I, j+1, k), (I, j, k-1), and (I, j, k+1), and couples with several other components. Jiannan Sent from Mail for Windows From: Barry Smith Sent: Wednesday, September 7, 2022 11:53 AM To: Tu, Jiannan Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] Using matrix-free with KSP CAUTION: This email was sent from outside the UMass Lowell network. DMDASetBlockFills() or DMDASetBlockFillsSparse() (they are just different ways of providing the same information) will help you here enormously, your type of case is exactly what they were designed for. Barry On Sep 7, 2022, at 10:47 AM, Tu, Jiannan > wrote: Barry and Hong, Thank you. There are 26 components at each grid and there are not fully coupled in terms of stiff functions. Mutual coupling is among about 6 components. I would prefer not using matrix-free since the Jacobina is not difficult to calculate and only up to 10 non-zeros at each row. I'll try DMDASetBlockFills() or DMDASetBlockFillsSparse() and see how they can reduce the memory usage. Jiannan From: Barry Smith > Sent: Tuesday, September 6, 2022 11:33 PM To: Tu, Jiannan > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Using matrix-free with KSP CAUTION: This email was sent from outside the UMass Lowell network. On Sep 6, 2022, at 11:00 PM, Tu, Jiannan > wrote: I am using TS IMEX to solve a large DAE system. The DAE is obtained by applying finite FV method to 3-D multi-specie ion/neutral fluids equations with magnetic induction equation. The Jacobian for stiff part is formed by using MatSetValuesStencil(). The Jacobian matrix is very sparse, no more than 10 non-zeros on each row. MatSetValuesStencil requires local to global mapping by calling ISLocaltoGlobalMapping(). Could you please instruct me how to use local to global mapping? DMDA automatically sets up the ISLocaltoGlobalMapping() so you should not need to. I also tried using DMCreateMatrix() to create the Jacobian. While local to global mapping is not necessary, the matrix takes too much memory and requires 64-bit indices. I would prefer to take the advantage of sparsity of the Jacobian, pre-allocate the matrix to use as less as possible memory so that the code can be run on a multi-core desktop. If using matrix-free does not work for you because the linear solves do not converge or converge too slowly. Then you might be able to decrease the memory used in the matrix. The DMCreateMatrix() does take advantage of sparsity and tries to preallocate only what it needs. Often what it preallocates is the best possible, but for multicomponent problems it has to assume there is full coupling within all the degrees of freedom that live at each at each grid point. How many components live at each grid point in your model and are they not all coupled to each other in the equations? If they are not fully coupled you can use either DMDASetBlockFills() or DMDASetBlockFillsSparse() to indicate the reduced coupling that actually exists for you model. Barry Thank you very much for your advice. Jiannan -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: fig2.png Type: image/png Size: 19181 bytes Desc: fig2.png URL: From bsmith at petsc.dev Tue Sep 13 10:53:51 2022 From: bsmith at petsc.dev (Barry Smith) Date: Tue, 13 Sep 2022 11:53:51 -0400 Subject: [petsc-users] Using matrix-free with KSP In-Reply-To: References: <31AF5BF6-8AC4-4A76-9235-9C0E1BB3EE16@petsc.dev> <7F5F237F-B73B-447E-A113-B46C697D4C37@petsc.dev> <79BC638A-1064-42A1-AE04-40ACAA82323A@petsc.dev> <12206964-1F03-45CE-9FDB-EC4B244CD1F8@petsc.dev> <15DD8C9E-C3A5-4758-9A1F-B9FA28B35825@petsc.dev> <21002A96-7F12-452D-94D1-A3B8265D982A@petsc.dev> <22852407-23A2-4D15-9117-E1C4577996A9@petsc.dev> Message-ID: This is becoming a nightmare. You pointed out something I had not thought about. Not only is the nonzero structure different in your case but so is the local to global mapping. Because certain of your ghost points are connected to different global points than for the standard case, thus the calls to MatSetValuesStencil() will not work correctly for the nonstandard ghost points. The mapping is constructed in DMSetUp_DA_3D() and it is very tedious code. In theory, the idx[] entries could be patched just before the call to ISLocalToGlobalMappingCreate() but it may be tedious to figure out exactly which local points need to have their global point changed. In theory it is easy but the code is extremely tedious to get to exactly the place that needs change. I'm sorry, we may have been attempting to push the DMDA construct too far from its simple, save home territory and thus ending up in a morass. I suggest looking at the DMSetUp_DA_3D() to see if you can figure out how to get the specific ghost points properly mapped but I am not sure if that is a reasonable task. Otherwise, I have no good suggestions. If the local to global mapping cannot be fixed then the entire model of using MatSetValuesStencil falls apart. Barry > On Sep 13, 2022, at 12:03 AM, Tu, Jiannan wrote: > > Indeed, it is not a standard grid structure that DMDA handles by default. Normally, a grid (k, j, i) has connectivity at grid (k, j-1, i). I understand if j > 0, DMDA correctly allocates memory for it. When j = 0, j-1 = -1, this (k, -1, i) grid is actually the grid across the pole with k-index = (k+a3/2) % a3, where a3 is the number of grids for k-index. Attached figure illustrates a pair of such grids. Assume red grid is at (k = 10, j = 0, i), its corresponding grid *blue dot) cross the pole is at (k = 55, j =0, i) if a3 = 90. > > The question is how to make DMDA allocate memory at such local column stencils like (col.k = 55, col.j = 0, col.i = i, col.c = 0), instead of the column stencil (col.k = 10, col.j = -1, col.i = i, col.c = 0). > > I tried insert at stencils like (col.k = 10, col.j = -1, col.i = i, col.c = 0). The matrix assemble has no issues but TSSolve failed. I guess probably due to wrong column locations of those Jacobian elements. > > Thank you, > Jiannan > > From: Barry Smith > > Sent: Monday, September 12, 2022 4:18 PM > To: Tu, Jiannan > > Cc: petsc-users > > Subject: Re: [petsc-users] Using matrix-free with KSP > > CAUTION: This email was sent from outside the UMass Lowell network. > > > I understand in principle, I just cannot wrap my head around a sphere :-) > > > > On Sep 12, 2022, at 3:45 PM, Tu, Jiannan > wrote: > > Simulation domain is in spherical coordinates. The grid along co-latitude direction is that j = 0 on one side of the pole and j = -1 (in fact index 0) on the other side of the pole. The longitude index k for j = 0, say is 0 degree then for j = -1 is 180 degrees. kcm accounts for this change in k-index when going from one side of the pole to another side. > > I'll see if I can solve the problem with the method you suggested. > > Thank you very much. > Jiannan > From: Barry Smith > > Sent: Monday, September 12, 2022 9:31 AM > To: Tu, Jiannan > > Cc: petsc-users > > Subject: Re: [petsc-users] Using matrix-free with KSP > > CAUTION: This email was sent from outside the UMass Lowell network. > > > I am not able to visualize the pattern you refer to but I am pretty sure it is not a type of connectivity that DMDA handles by default since DMDA is for relatively simply structured meshes. Here is what I suggest. > > The routine that does the preallocation for DMDA that you are using is in src/dm/impls/da/fdda.c called DMCreateMatrix_DA_3d_MPIAIJ_Fill(). On each MPI rank it determines the connectivity for each local grid vertex (calledthe variable row in the routine) with all the other grid vertices (these are stored temporarily in the array cols[]). > The call to MatPreallocateSetLocal() takes this coupling information and stores it temporarily. The calls to > > PetscCall(MatSeqAIJSetPreallocation(J, 0, dnz)); > PetscCall(MatMPIAIJSetPreallocation(J, 0, dnz, 0, onz)); > MatPreallocateEnd(dnz, onz); > PetscCall(MatSetLocalToGlobalMapping(J, ltog, ltog)); > > at the end of the loop over the local grid vertices then provide the information to the matrix so it now has the correct preallocation and correct local to global mapping needed by MatSetValuesStencil() > > The same loop structure is then called again where it inserts zero values into the matrix at all the correctly preallocated locations. > > You can copy this routine and modify it slightly to include the "special coupling" that your problem requires. I do not understand your formula for kcm so cannot tell you exactly what you need to change but it may correspond to remapping the kk values at the extremes of the loop for (kk = kstart; kk < kend + 1; kk++) { so that the > values nc * (slot + ii + gnx * jj + gnx * gny * kk) correspond to the correct coupling location. I recommend drawing a picture of a case with a very small size for nx,ny, and nz so you can see exactly how the mapping takes place and trace through your modified code that it does what you need. > > Barry > > > > > > On Sep 11, 2022, at 11:01 PM, Tu, Jiannan > wrote: > > Barry, > > Creating DNDA with periodic boundary BC and using km=k-1; kp=k+1 even at the BCs solve the ?augment out of range? problem for inserting elements at theae boundaries of k-index. Now the problem is due to wrapping around the north pole when j=0 and jm = -1 is reset as j=0 but on the other side of the pole with k-index replacing by kcm = [kmax+1)/2\ % (kmax+1). In my case, the ?Augment out of range? happens at global row/column (26, 5265026), exactly at wrapping jm=-1 to j=0 to the other side. > > I am thinking about how to properly use ghost grid j = -1 by setting appropriate BC there and inserting at that location without wrapping. > > Jiannan > > From: Barry Smith > > Sent: Sunday, September 11, 2022 12:10 PM > To: Tu, Jiannan > > Cc: petsc-users > > Subject: Re: [petsc-users] Using matrix-free with KSP > > CAUTION: This email was sent from outside the UMass Lowell network. > > > > > On Sep 11, 2022, at 9:56 AM, Tu, Jiannan > wrote: > > Barry, > > Thank you. > > Yes, the DMDA is created with periodic BC. But it might not be the periodic BC since in azimuthal direction, the ghost grid -1 is actually grid Nmax, and Nmax+1 is grid 0. > > Consider the following, one has n points from 0 to n-1 where n = 4, so I think with your definition of Nmax, Nmax is 3, Nmax + 1 is 4, > > > -1 0 1 2 3 4 > + * * * * + > > Thus there is one grid point at each end. The * represent regular grid points and the + ghost locations. Is this your situation? > > This is what we call periodic with a stencil width of 1 with DMDA and the DMCreateMatrix() should automatically provide the correct preallocation and nonzero structure. > > Can you please do a MatView() on the result from DMCreateMatrix() for a very small grid before you set any values and you should see the locations for the coupling across the periodic boundary. > > It sounds like you have a more complicated cross periodic coupling, we need to understand it exactly so we can determine how to proceed. > > Barry > > > > > Still the problem is how to properly pre-allocate matrix. Otherwise either ?Augument out of range? error or code stuck at inserting values if MatSetOption() is called. > > Jiannan > > From: Barry Smith > > Sent: Sunday, September 11, 2022 12:10 AM > To: Tu, Jiannan > > Cc: petsc-users > > Subject: Re: [petsc-users] Using matrix-free with KSP > > CAUTION: This email was sent from outside the UMass Lowell network. > > > Have you created the DMDA with the periodic boundary conditions arguments? Or are your boundary conditions not simply periodic? > > Compare DMCreateMatrix_DA_3d_MPIAIJ() and DMCreateMatrix_DA_3d_MPIAIJ_Fill() in src/dm/impls/da/fdda.c I think they both attempt to handle the periodic case. > > Barry > > > > > > > On Sep 10, 2022, at 10:14 PM, Tu, Jiannan > wrote: > > Barry, > > I think I understand how to set values for the intra-grid coupling. But I am not clear about the way to set values for inter-grid coupling. When a component couples to itself at above, below, right, left, front and back grids, diagonal is set as 1 for the second array of DMDASetBlockFills(). How about if involving more grids? How does DMDA know where to allocate memory? In my case, a component couples to itself at the other end of the grid because of periodic boundary condition. There is an error message ?Augment out of range . Insert new nonzero at global row/column (26, 520526) into matrix? because of this far away coupling. The column index 520526 is related to the other end of the grid in azimuthal direction. > > Thanks, > Jiannan > > > > From: Barry Smith > > Sent: Saturday, September 10, 2022 1:10 PM > To: Tu, Jiannan > > Cc: petsc-users > > Subject: Re: [petsc-users] Using matrix-free with KSP > > CAUTION: This email was sent from outside the UMass Lowell network. > > > If you are using DMCreateMatrix() then MatMPIAIJSetPreallocation is not needed, it is automatically handled by the DM. > > It sounds like that the values you set in DMDASetBlockFills() did not include all the coupling, hence the preallocation was wrong, hence the time to insert values was too long. > > Barry > > > > > > On Sep 10, 2022, at 8:16 AM, Tu, Jiannan > wrote: > > Just found MatMPIAIJSetPreallocation is needed. Now Jacobin insertion and assembly is fast. > > From: Barry Smith > > Sent: Thursday, September 8, 2022 9:53 PM > To: Tu, Jiannan > > Subject: Re: [petsc-users] Using matrix-free with KSP > > CAUTION: This email was sent from outside the UMass Lowell network. > > > > > > > > On Sep 8, 2022, at 8:34 PM, Tu, Jiannan > wrote: > > Barry, > > Thank you. > > I set up these two arrays according to the non-zero pattern of the Jacobian. The DMDASetBlockFills() is then called after the call to DMCreateMatrix(da, &A). > > Absolutely not! It must be called before DMCreateMatrix() so that the DM knows how to construct exactly the matrix you need. > > > > > > > Nevertheless, the program execution is still killed with exit code 9, which I assume is due to overuse of the memory. My desktop machine has 32 GB RAM. > > I ran the code with a smaller mesh, 100x45x90, the result is the same. With that number of grids, 26 unknowns, and up to 10 non-zero at each matrix row, the estimated memory to store 100x45x90x26x10 elements should be less than 1 GB (double precision). I am wondering why the matrix still takes too much memory. Maybe I have to use matrix-free? > > Thank you, > Jiannan > > From: Barry Smith > > Sent: Thursday, September 8, 2022 4:19 PM > To: Tu, Jiannan > > Subject: Re: [petsc-users] Using matrix-free with KSP > > CAUTION: This email was sent from outside the UMass Lowell network. > > > > > > > > > On Sep 8, 2022, at 3:39 PM, Tu, Jiannan > wrote: > > Barry, > > Thank you very much for the detailed description. > > So the first array, dfill, is for intra grids and the second one, ofill is for the inter-grid coupling? > > In the case for inter-grid coupling, a component is only coupled to itself, say x in your example is coupled to itself on above, below, right, left, front, and back, how to set values for the second array? > > Then the off-diagonal one would just have values on its diagonal. > > > > > > > Jiannan > > From: Barry Smith > Sent: Thursday, September 8, 2022 2:12 PM > To: Tu, Jiannan > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Using matrix-free with KSP > > CAUTION: This email was sent from outside the UMass Lowell network. > > > > > > > > > > On Sep 8, 2022, at 1:47 PM, Tu, Jiannan > wrote: > > Barry, > > Thank you very much. > > DMDASetBlockFills() needs two input arrays specifying sparsity pattern. In my case, I have 26 components at each grid. Is it correct that I need to have two arrays of 26x26 elements with value 1 for coupling and 0 for no coupling? > > Exactly > > > > > > > > Also how to determine when the value should be 1 or 0? > > That is problem dependent. Given paper-and-pencil you know for your stencil which components are coupled with other components at each grid point and which components are coupled with components at neighboring grid points. > > Note the second array contains information about all the neighboring grid points. Above, below, right, left, front back. Normally the coupling is symmetric so the nonzero structures of those 6 blocks are identical. If, for you particular problem, the nonzero structures of the six are not identical you use the union of the nonzero structures of the six of them. > > > 2d example say you have x,y,z at each grid point and the inter-point coupling is x is connected to y and y is connect to x then the first array is > > [ 1 1 0 ; 1 1; 0 ; 0 0 1]; the zeros are because x is not coupled to z and y is not coupled to z. > > For intra grid points, say x is coupled to z and itself, z is coupled to x and itself and y is not coupled to itself then > > [1 0 1; 0 0 0; 1 0 1]; the middle row is zero because y is not coupled to anything and the zeros in the first and last are because x and z are not coupled to y. > > > > > > > > > > Each component involves 7 stencils (i-1, j, k), (I, j, k), (i+1, j, k), (I, j-1, k), (I, j+1, k), (I, j, k-1), and (I, j, k+1), and couples with several other components. > > Jiannan > > > > Sent from Mail for Windows > > From: Barry Smith > Sent: Wednesday, September 7, 2022 11:53 AM > To: Tu, Jiannan > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Using matrix-free with KSP > > CAUTION: This email was sent from outside the UMass Lowell network. > > > DMDASetBlockFills() or DMDASetBlockFillsSparse() (they are just different ways of providing the same information) will help you here enormously, your type of case is exactly what they were designed for. > > Barry > > > > > > > > > > On Sep 7, 2022, at 10:47 AM, Tu, Jiannan > wrote: > > Barry and Hong, > > Thank you. > > There are 26 components at each grid and there are not fully coupled in terms of stiff functions. Mutual coupling is among about 6 components. > > I would prefer not using matrix-free since the Jacobina is not difficult to calculate and only up to 10 non-zeros at each row. I'll try DMDASetBlockFills() or DMDASetBlockFillsSparse() and see how they can reduce the memory usage. > > Jiannan > > From: Barry Smith > > Sent: Tuesday, September 6, 2022 11:33 PM > To: Tu, Jiannan > > Cc: petsc-users at mcs.anl.gov > > Subject: Re: [petsc-users] Using matrix-free with KSP > > CAUTION: This email was sent from outside the UMass Lowell network. > > > > > > > > > > > On Sep 6, 2022, at 11:00 PM, Tu, Jiannan > wrote: > > I am using TS IMEX to solve a large DAE system. The DAE is obtained by applying finite FV method to 3-D multi-specie ion/neutral fluids equations with magnetic induction equation. The Jacobian for stiff part is formed by using MatSetValuesStencil(). The Jacobian matrix is very sparse, no more than 10 non-zeros on each row. MatSetValuesStencil requires local to global mapping by calling ISLocaltoGlobalMapping(). Could you please instruct me how to use local to global mapping? > > DMDA automatically sets up the ISLocaltoGlobalMapping() so you should not need to. > > I also tried using DMCreateMatrix() to create the Jacobian. While local to global mapping is not necessary, the matrix takes too much memory and requires 64-bit indices. I would prefer to take the advantage of sparsity of the Jacobian, pre-allocate the matrix to use as less as possible memory so that the code can be run on a multi-core desktop. > > If using matrix-free does not work for you because the linear solves do not converge or converge too slowly. Then you might be able to decrease the memory used in the matrix. The DMCreateMatrix() does take advantage of sparsity and tries to preallocate only what it needs. Often what it preallocates is the best possible, but for multicomponent problems it has to assume there is full coupling within all the degrees of freedom that live at each at each grid point. How many components live at each grid point in your model and are they not all coupled to each other in the equations? If they are not fully coupled you can use either DMDASetBlockFills() or DMDASetBlockFillsSparse() to indicate the reduced coupling that actually exists for you model. > > Barry > > > > > > > > > > Thank you very much for your advice. > > Jiannan > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Jiannan_Tu at uml.edu Tue Sep 13 12:35:41 2022 From: Jiannan_Tu at uml.edu (Tu, Jiannan) Date: Tue, 13 Sep 2022 17:35:41 +0000 Subject: [petsc-users] Using matrix-free with KSP In-Reply-To: References: <31AF5BF6-8AC4-4A76-9235-9C0E1BB3EE16@petsc.dev> <7F5F237F-B73B-447E-A113-B46C697D4C37@petsc.dev> <79BC638A-1064-42A1-AE04-40ACAA82323A@petsc.dev> <12206964-1F03-45CE-9FDB-EC4B244CD1F8@petsc.dev> <15DD8C9E-C3A5-4758-9A1F-B9FA28B35825@petsc.dev> <21002A96-7F12-452D-94D1-A3B8265D982A@petsc.dev> <22852407-23A2-4D15-9117-E1C4577996A9@petsc.dev> Message-ID: I see what you mean. It indeed will make the implementation very complicated. I thought about a solution for my case. The internal grids in j-index will start from 1, instead of 0. Index 0 is the index j = 1 across the north pole. What need to be solved are components at grid 1 to jmax-1. jmax is the grid across the south pole. At grids 0 and jmax, values are set equal to values at the grids j = 1 and j = jmax-1, respectively, on the other side, and at n time step but not the current n+1 time step. In this way we don't need jacobian location wrap around the pole but simply set it to 1 at j = 0 and j = jmax since the values at these locations already set. This is equivalent to that each row actually lacks columns at two ends of j-index. Anyway, thank you for all the helps. And appreciate your stimulating suggestions. Jiannan ________________________________ From: Barry Smith Sent: Tuesday, September 13, 2022 11:53 AM To: Tu, Jiannan Cc: petsc-users Subject: Re: [petsc-users] Using matrix-free with KSP CAUTION: This email was sent from outside the UMass Lowell network. This is becoming a nightmare. You pointed out something I had not thought about. Not only is the nonzero structure different in your case but so is the local to global mapping. Because certain of your ghost points are connected to different global points than for the standard case, thus the calls to MatSetValuesStencil() will not work correctly for the nonstandard ghost points. The mapping is constructed in DMSetUp_DA_3D() and it is very tedious code. In theory, the idx[] entries could be patched just before the call to ISLocalToGlobalMappingCreate() but it may be tedious to figure out exactly which local points need to have their global point changed. In theory it is easy but the code is extremely tedious to get to exactly the place that needs change. I'm sorry, we may have been attempting to push the DMDA construct too far from its simple, save home territory and thus ending up in a morass. I suggest looking at the DMSetUp_DA_3D() to see if you can figure out how to get the specific ghost points properly mapped but I am not sure if that is a reasonable task. Otherwise, I have no good suggestions. If the local to global mapping cannot be fixed then the entire model of using MatSetValuesStencil falls apart. Barry On Sep 13, 2022, at 12:03 AM, Tu, Jiannan > wrote: Indeed, it is not a standard grid structure that DMDA handles by default. Normally, a grid (k, j, i) has connectivity at grid (k, j-1, i). I understand if j > 0, DMDA correctly allocates memory for it. When j = 0, j-1 = -1, this (k, -1, i) grid is actually the grid across the pole with k-index = (k+a3/2) % a3, where a3 is the number of grids for k-index. Attached figure illustrates a pair of such grids. Assume red grid is at (k = 10, j = 0, i), its corresponding grid *blue dot) cross the pole is at (k = 55, j =0, i) if a3 = 90. The question is how to make DMDA allocate memory at such local column stencils like (col.k = 55, col.j = 0, col.i = i, col.c = 0), instead of the column stencil (col.k = 10, col.j = -1, col.i = i, col.c = 0). I tried insert at stencils like (col.k = 10, col.j = -1, col.i = i, col.c = 0). The matrix assemble has no issues but TSSolve failed. I guess probably due to wrong column locations of those Jacobian elements. Thank you, Jiannan From: Barry Smith > Sent: Monday, September 12, 2022 4:18 PM To: Tu, Jiannan > Cc: petsc-users > Subject: Re: [petsc-users] Using matrix-free with KSP CAUTION: This email was sent from outside the UMass Lowell network. I understand in principle, I just cannot wrap my head around a sphere :-) On Sep 12, 2022, at 3:45 PM, Tu, Jiannan > wrote: Simulation domain is in spherical coordinates. The grid along co-latitude direction is that j = 0 on one side of the pole and j = -1 (in fact index 0) on the other side of the pole. The longitude index k for j = 0, say is 0 degree then for j = -1 is 180 degrees. kcm accounts for this change in k-index when going from one side of the pole to another side. I'll see if I can solve the problem with the method you suggested. Thank you very much. Jiannan ________________________________ From: Barry Smith > Sent: Monday, September 12, 2022 9:31 AM To: Tu, Jiannan > Cc: petsc-users > Subject: Re: [petsc-users] Using matrix-free with KSP CAUTION: This email was sent from outside the UMass Lowell network. I am not able to visualize the pattern you refer to but I am pretty sure it is not a type of connectivity that DMDA handles by default since DMDA is for relatively simply structured meshes. Here is what I suggest. The routine that does the preallocation for DMDA that you are using is in src/dm/impls/da/fdda.c called DMCreateMatrix_DA_3d_MPIAIJ_Fill(). On each MPI rank it determines the connectivity for each local grid vertex (calledthe variable row in the routine) with all the other grid vertices (these are stored temporarily in the array cols[]). The call to MatPreallocateSetLocal() takes this coupling information and stores it temporarily. The calls to PetscCall(MatSeqAIJSetPreallocation(J, 0, dnz)); PetscCall(MatMPIAIJSetPreallocation(J, 0, dnz, 0, onz)); MatPreallocateEnd(dnz, onz); PetscCall(MatSetLocalToGlobalMapping(J, ltog, ltog)); at the end of the loop over the local grid vertices then provide the information to the matrix so it now has the correct preallocation and correct local to global mapping needed by MatSetValuesStencil() The same loop structure is then called again where it inserts zero values into the matrix at all the correctly preallocated locations. You can copy this routine and modify it slightly to include the "special coupling" that your problem requires. I do not understand your formula for kcm so cannot tell you exactly what you need to change but it may correspond to remapping the kk values at the extremes of the loop for (kk = kstart; kk < kend + 1; kk++) { so that the values nc * (slot + ii + gnx * jj + gnx * gny * kk) correspond to the correct coupling location. I recommend drawing a picture of a case with a very small size for nx,ny, and nz so you can see exactly how the mapping takes place and trace through your modified code that it does what you need. Barry On Sep 11, 2022, at 11:01 PM, Tu, Jiannan > wrote: Barry, Creating DNDA with periodic boundary BC and using km=k-1; kp=k+1 even at the BCs solve the ?augment out of range? problem for inserting elements at theae boundaries of k-index. Now the problem is due to wrapping around the north pole when j=0 and jm = -1 is reset as j=0 but on the other side of the pole with k-index replacing by kcm = [kmax+1)/2\ % (kmax+1). In my case, the ?Augment out of range? happens at global row/column (26, 5265026), exactly at wrapping jm=-1 to j=0 to the other side. I am thinking about how to properly use ghost grid j = -1 by setting appropriate BC there and inserting at that location without wrapping. Jiannan From: Barry Smith > Sent: Sunday, September 11, 2022 12:10 PM To: Tu, Jiannan > Cc: petsc-users > Subject: Re: [petsc-users] Using matrix-free with KSP CAUTION: This email was sent from outside the UMass Lowell network. On Sep 11, 2022, at 9:56 AM, Tu, Jiannan > wrote: Barry, Thank you. Yes, the DMDA is created with periodic BC. But it might not be the periodic BC since in azimuthal direction, the ghost grid -1 is actually grid Nmax, and Nmax+1 is grid 0. Consider the following, one has n points from 0 to n-1 where n = 4, so I think with your definition of Nmax, Nmax is 3, Nmax + 1 is 4, -1 0 1 2 3 4 + * * * * + Thus there is one grid point at each end. The * represent regular grid points and the + ghost locations. Is this your situation? This is what we call periodic with a stencil width of 1 with DMDA and the DMCreateMatrix() should automatically provide the correct preallocation and nonzero structure. Can you please do a MatView() on the result from DMCreateMatrix() for a very small grid before you set any values and you should see the locations for the coupling across the periodic boundary. It sounds like you have a more complicated cross periodic coupling, we need to understand it exactly so we can determine how to proceed. Barry Still the problem is how to properly pre-allocate matrix. Otherwise either ?Augument out of range? error or code stuck at inserting values if MatSetOption() is called. Jiannan From: Barry Smith > Sent: Sunday, September 11, 2022 12:10 AM To: Tu, Jiannan > Cc: petsc-users > Subject: Re: [petsc-users] Using matrix-free with KSP CAUTION: This email was sent from outside the UMass Lowell network. Have you created the DMDA with the periodic boundary conditions arguments? Or are your boundary conditions not simply periodic? Compare DMCreateMatrix_DA_3d_MPIAIJ() and DMCreateMatrix_DA_3d_MPIAIJ_Fill() in src/dm/impls/da/fdda.c I think they both attempt to handle the periodic case. Barry On Sep 10, 2022, at 10:14 PM, Tu, Jiannan > wrote: Barry, I think I understand how to set values for the intra-grid coupling. But I am not clear about the way to set values for inter-grid coupling. When a component couples to itself at above, below, right, left, front and back grids, diagonal is set as 1 for the second array of DMDASetBlockFills(). How about if involving more grids? How does DMDA know where to allocate memory? In my case, a component couples to itself at the other end of the grid because of periodic boundary condition. There is an error message ?Augment out of range . Insert new nonzero at global row/column (26, 520526) into matrix? because of this far away coupling. The column index 520526 is related to the other end of the grid in azimuthal direction. Thanks, Jiannan From: Barry Smith > Sent: Saturday, September 10, 2022 1:10 PM To: Tu, Jiannan > Cc: petsc-users > Subject: Re: [petsc-users] Using matrix-free with KSP CAUTION: This email was sent from outside the UMass Lowell network. If you are using DMCreateMatrix() then MatMPIAIJSetPreallocation is not needed, it is automatically handled by the DM. It sounds like that the values you set in DMDASetBlockFills() did not include all the coupling, hence the preallocation was wrong, hence the time to insert values was too long. Barry On Sep 10, 2022, at 8:16 AM, Tu, Jiannan > wrote: Just found MatMPIAIJSetPreallocation is needed. Now Jacobin insertion and assembly is fast. From: Barry Smith > Sent: Thursday, September 8, 2022 9:53 PM To: Tu, Jiannan > Subject: Re: [petsc-users] Using matrix-free with KSP CAUTION: This email was sent from outside the UMass Lowell network. On Sep 8, 2022, at 8:34 PM, Tu, Jiannan > wrote: Barry, Thank you. I set up these two arrays according to the non-zero pattern of the Jacobian. The DMDASetBlockFills() is then called after the call to DMCreateMatrix(da, &A). Absolutely not! It must be called before DMCreateMatrix() so that the DM knows how to construct exactly the matrix you need. Nevertheless, the program execution is still killed with exit code 9, which I assume is due to overuse of the memory. My desktop machine has 32 GB RAM. I ran the code with a smaller mesh, 100x45x90, the result is the same. With that number of grids, 26 unknowns, and up to 10 non-zero at each matrix row, the estimated memory to store 100x45x90x26x10 elements should be less than 1 GB (double precision). I am wondering why the matrix still takes too much memory. Maybe I have to use matrix-free? Thank you, Jiannan From: Barry Smith > Sent: Thursday, September 8, 2022 4:19 PM To: Tu, Jiannan > Subject: Re: [petsc-users] Using matrix-free with KSP CAUTION: This email was sent from outside the UMass Lowell network. On Sep 8, 2022, at 3:39 PM, Tu, Jiannan > wrote: Barry, Thank you very much for the detailed description. So the first array, dfill, is for intra grids and the second one, ofill is for the inter-grid coupling? In the case for inter-grid coupling, a component is only coupled to itself, say x in your example is coupled to itself on above, below, right, left, front, and back, how to set values for the second array? Then the off-diagonal one would just have values on its diagonal. Jiannan From: Barry Smith Sent: Thursday, September 8, 2022 2:12 PM To: Tu, Jiannan Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] Using matrix-free with KSP CAUTION: This email was sent from outside the UMass Lowell network. On Sep 8, 2022, at 1:47 PM, Tu, Jiannan > wrote: Barry, Thank you very much. DMDASetBlockFills() needs two input arrays specifying sparsity pattern. In my case, I have 26 components at each grid. Is it correct that I need to have two arrays of 26x26 elements with value 1 for coupling and 0 for no coupling? Exactly Also how to determine when the value should be 1 or 0? That is problem dependent. Given paper-and-pencil you know for your stencil which components are coupled with other components at each grid point and which components are coupled with components at neighboring grid points. Note the second array contains information about all the neighboring grid points. Above, below, right, left, front back. Normally the coupling is symmetric so the nonzero structures of those 6 blocks are identical. If, for you particular problem, the nonzero structures of the six are not identical you use the union of the nonzero structures of the six of them. 2d example say you have x,y,z at each grid point and the inter-point coupling is x is connected to y and y is connect to x then the first array is [ 1 1 0 ; 1 1; 0 ; 0 0 1]; the zeros are because x is not coupled to z and y is not coupled to z. For intra grid points, say x is coupled to z and itself, z is coupled to x and itself and y is not coupled to itself then [1 0 1; 0 0 0; 1 0 1]; the middle row is zero because y is not coupled to anything and the zeros in the first and last are because x and z are not coupled to y. Each component involves 7 stencils (i-1, j, k), (I, j, k), (i+1, j, k), (I, j-1, k), (I, j+1, k), (I, j, k-1), and (I, j, k+1), and couples with several other components. Jiannan Sent from Mail for Windows From: Barry Smith Sent: Wednesday, September 7, 2022 11:53 AM To: Tu, Jiannan Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] Using matrix-free with KSP CAUTION: This email was sent from outside the UMass Lowell network. DMDASetBlockFills() or DMDASetBlockFillsSparse() (they are just different ways of providing the same information) will help you here enormously, your type of case is exactly what they were designed for. Barry On Sep 7, 2022, at 10:47 AM, Tu, Jiannan > wrote: Barry and Hong, Thank you. There are 26 components at each grid and there are not fully coupled in terms of stiff functions. Mutual coupling is among about 6 components. I would prefer not using matrix-free since the Jacobina is not difficult to calculate and only up to 10 non-zeros at each row. I'll try DMDASetBlockFills() or DMDASetBlockFillsSparse() and see how they can reduce the memory usage. Jiannan From: Barry Smith > Sent: Tuesday, September 6, 2022 11:33 PM To: Tu, Jiannan > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Using matrix-free with KSP CAUTION: This email was sent from outside the UMass Lowell network. On Sep 6, 2022, at 11:00 PM, Tu, Jiannan > wrote: I am using TS IMEX to solve a large DAE system. The DAE is obtained by applying finite FV method to 3-D multi-specie ion/neutral fluids equations with magnetic induction equation. The Jacobian for stiff part is formed by using MatSetValuesStencil(). The Jacobian matrix is very sparse, no more than 10 non-zeros on each row. MatSetValuesStencil requires local to global mapping by calling ISLocaltoGlobalMapping(). Could you please instruct me how to use local to global mapping? DMDA automatically sets up the ISLocaltoGlobalMapping() so you should not need to. I also tried using DMCreateMatrix() to create the Jacobian. While local to global mapping is not necessary, the matrix takes too much memory and requires 64-bit indices. I would prefer to take the advantage of sparsity of the Jacobian, pre-allocate the matrix to use as less as possible memory so that the code can be run on a multi-core desktop. If using matrix-free does not work for you because the linear solves do not converge or converge too slowly. Then you might be able to decrease the memory used in the matrix. The DMCreateMatrix() does take advantage of sparsity and tries to preallocate only what it needs. Often what it preallocates is the best possible, but for multicomponent problems it has to assume there is full coupling within all the degrees of freedom that live at each at each grid point. How many components live at each grid point in your model and are they not all coupled to each other in the equations? If they are not fully coupled you can use either DMDASetBlockFills() or DMDASetBlockFillsSparse() to indicate the reduced coupling that actually exists for you model. Barry Thank you very much for your advice. Jiannan -------------- next part -------------- An HTML attachment was scrubbed... URL: From fujisan43 at gmail.com Wed Sep 21 03:03:39 2022 From: fujisan43 at gmail.com (fujisan) Date: Wed, 21 Sep 2022 10:03:39 +0200 Subject: [petsc-users] Problem solving Ax=b with rectangular matrix A Message-ID: I'm trying to solve Ax=b with a sparse rectangular matrix A (of size 33x17 in my test) using options '-ksp_type stcg -pc_type lu' on 1 or 2 cpus. And I always get an error saying "Incompatible vector local lengths" (see below). Here is the relevant lines of my code: program test ... ! Variable declarations PetscCallA(PetscInitialize(PETSC_NULL_CHARACTER,ierr)) PetscCall(MatCreate(PETSC_COMM_WORLD,A,ierr)) PetscCall(MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,m,n,ierr)) PetscCall(MatSetType(A,MATMPIAIJ,ierr)) PetscCall(MatSetFromOptions(A,ierr)) PetscCall(MatSetUp(A,ierr)) PetscCall(MatGetOwnershipRange(A,istart,iend,ierr)) do irow=istart,iend-1 ... Reading from file ... PetscCall(MatSetValues(A,1,irow,nzv,col,val,ADD_VALUES,ierr)) ... enddo PetscCall(MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY,ierr)) PetscCall(MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY,ierr)) ! Creating vectors x and b PetscCallA(MatCreateVecs(A,x,b,ierr)) ! Duplicating x in u. PetscCallA(VecDuplicate(x,u,ierr)) ! u is used to calculate b PetscCallA(VecSet(u,1.0,ierr)) PetscCallA(VecAssemblyBegin(u,ierr)) PetscCallA(VecAssemblyEnd(u,ierr)) ! Calculating Au = b PetscCallA(MatMult(A,u,b,ierr)) ! A.u = b PetscCallA(KSPSetType(ksp,KSPCG,ierr)) PetscCallA(KSPSetOperators(ksp,A,A,ierr)) PetscCallA(KSPSetFromOptions(ksp,ierr)) ! Solving Ax = b, x unknown PetscCallA(KSPSolve(ksp,b,x,ierr)) PetscCallA(VecDestroy(x,ierr)) PetscCallA(VecDestroy(u,ierr)) PetscCallA(VecDestroy(b,ierr)) PetscCallA(MatDestroy(A,ierr)) PetscCallA(KSPDestroy(ksp,ierr)) call PetscFinalize(ierr) end program The code reads a sparse matrix from a binary file. I also output the sizes of matrix A and vectors b, x, u. They all seem consistent. What am I doing wrong? Is it possible to solve Ax=b with A rectangular? Thank you in advance for your help. Have a nice day. Fuji Matrix size : m= 33 n= 17 cpu size: 1 Size of matrix A : 33 17 Size of vector b : 33 Size of vector x : 17 Size of vector u : 17 [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Arguments are incompatible [0]PETSC ERROR: Incompatible vector local lengths parameter # 1 local size 33 != parameter # 2 local size 17 [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting. [0]PETSC ERROR: Petsc Development GIT revision: v3.17.4-1341-g91b2b62a00 GIT Date: 2022-09-15 19:26:07 +0000 [0]PETSC ERROR: ./bin/solve on a x86_64 named master by fujisan Tue Sep 20 16:56:37 2022 [0]PETSC ERROR: Configure options --with-petsc-arch=x86_64 --COPTFLAGS="-g -O3" --FOPTFLAGS="-g -O3" --CXXOPTFLAGS="-g -O3" --with-debugging=0 --with-cc=mpiicc --with-cxx=mpiicpc --with-fc=mpiifort --with-single-library=1 --with-mpiexec=mpiexec --with-precision=double --with-fortran-interfaces=1 --with-make=1 --with-mpi=1 --with-mpi-compilers=1 --download-fblaslapack=0 --download-hypre=1 --download-cmake=0 --with-cmake=1 --download-metis=1 --download-parmetis=1 --download-ptscotch=0 --download-suitesparse=1 --download-triangle=1 --download-superlu=1 --download-superlu_dist=1 --download-scalapack=1 --download-mumps=1 --download-elemental=1 --download-spai=0 --download-parms=1 --download-moab=1 --download-chaco=0 --download-fftw=1 --with-petsc4py=1 --download-mpi4py=1 --download-saws --download-concurrencykit=1 --download-revolve=1 --download-cams=1 --download-p4est=0 --with-zlib=1 --download-mfem=1 --download-glvis=0 --with-opengl=0 --download-libpng=1 --download-libjpeg=1 --download-slepc=1 --download-hpddm=1 --download-bamg=1 --download-mmg=0 --download-parmmg=0 --download-htool=1 --download-egads=0 --download-opencascade=0 PETSC_ARCH=x86_64 [0]PETSC ERROR: #1 VecCopy() at /data/softs/petsc/src/vec/vec/interface/vector.c:1607 [0]PETSC ERROR: #2 KSPSolve_BiCG() at /data/softs/petsc/src/ksp/ksp/impls/bicg/bicg.c:40 [0]PETSC ERROR: #3 KSPSolve_Private() at /data/softs/petsc/src/ksp/ksp/interface/itfunc.c:877 [0]PETSC ERROR: #4 KSPSolve() at /data/softs/petsc/src/ksp/ksp/interface/itfunc.c:1048 [0]PETSC ERROR: #5 solve.F90:218 Abort(75) on node 0 (rank 0 in comm 16): application called MPI_Abort(MPI_COMM_SELF, 75) - process 0 -------------- next part -------------- An HTML attachment was scrubbed... URL: From pierre at joliv.et Wed Sep 21 05:20:27 2022 From: pierre at joliv.et (Pierre Jolivet) Date: Wed, 21 Sep 2022 12:20:27 +0200 Subject: [petsc-users] Problem solving Ax=b with rectangular matrix A In-Reply-To: References: Message-ID: Yes, but you need to use a KSP that handles rectangular Mat, such as KSPLSQR (-ksp_type lsqr). PCLU does not handle rectangular Pmat. The only PC that handle rectangular Pmat are PCQR, PCNONE. If you supply the normal equations as the Pmat for LSQR, then you can use ?standard? PC. You can have a look at https://petsc.org/main/src/ksp/ksp/tutorials/ex27.c.html that covers most of these cases. Thanks, Pierre (sorry for the earlier answer sent wrongfully to petsc-maint, please discard the previous email) > On 21 Sep 2022, at 10:03 AM, fujisan wrote: > > I'm trying to solve Ax=b with a sparse rectangular matrix A (of size 33x17 in my test) using > options '-ksp_type stcg -pc_type lu' on 1 or 2 cpus. > > And I always get an error saying "Incompatible vector local lengths" (see below). > > Here is the relevant lines of my code: > > program test > ... > ! Variable declarations > > PetscCallA(PetscInitialize(PETSC_NULL_CHARACTER,ierr)) > > PetscCall(MatCreate(PETSC_COMM_WORLD,A,ierr)) > PetscCall(MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,m,n,ierr)) > PetscCall(MatSetType(A,MATMPIAIJ,ierr)) > PetscCall(MatSetFromOptions(A,ierr)) > PetscCall(MatSetUp(A,ierr)) > PetscCall(MatGetOwnershipRange(A,istart,iend,ierr)) > > do irow=istart,iend-1 > ... Reading from file ... > PetscCall(MatSetValues(A,1,irow,nzv,col,val,ADD_VALUES,ierr)) > ... > enddo > > PetscCall(MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY,ierr)) > PetscCall(MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY,ierr)) > > ! Creating vectors x and b > PetscCallA(MatCreateVecs(A,x,b,ierr)) > > ! Duplicating x in u. > PetscCallA(VecDuplicate(x,u,ierr)) > > ! u is used to calculate b > PetscCallA(VecSet(u,1.0,ierr)) > > PetscCallA(VecAssemblyBegin(u,ierr)) > PetscCallA(VecAssemblyEnd(u,ierr)) > > ! Calculating Au = b > PetscCallA(MatMult(A,u,b,ierr)) ! A.u = b > > PetscCallA(KSPSetType(ksp,KSPCG,ierr)) > > PetscCallA(KSPSetOperators(ksp,A,A,ierr)) > > PetscCallA(KSPSetFromOptions(ksp,ierr)) > > ! Solving Ax = b, x unknown > PetscCallA(KSPSolve(ksp,b,x,ierr)) > > PetscCallA(VecDestroy(x,ierr)) > PetscCallA(VecDestroy(u,ierr)) > PetscCallA(VecDestroy(b,ierr)) > PetscCallA(MatDestroy(A,ierr)) > PetscCallA(KSPDestroy(ksp,ierr)) > > call PetscFinalize(ierr) > end program > > The code reads a sparse matrix from a binary file. > I also output the sizes of matrix A and vectors b, x, u. > They all seem consistent. > > What am I doing wrong? > Is it possible to solve Ax=b with A rectangular? > > Thank you in advance for your help. > Have a nice day. > > Fuji > > Matrix size : m= 33 n= 17 cpu size: 1 > Size of matrix A : 33 17 > Size of vector b : 33 > Size of vector x : 17 > Size of vector u : 17 > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [0]PETSC ERROR: Arguments are incompatible > [0]PETSC ERROR: Incompatible vector local lengths parameter # 1 local size 33 != parameter # 2 local size 17 > [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting. > [0]PETSC ERROR: Petsc Development GIT revision: v3.17.4-1341-g91b2b62a00 GIT Date: 2022-09-15 19:26:07 +0000 > [0]PETSC ERROR: ./bin/solve on a x86_64 named master by fujisan Tue Sep 20 16:56:37 2022 > [0]PETSC ERROR: Configure options --with-petsc-arch=x86_64 --COPTFLAGS="-g -O3" --FOPTFLAGS="-g -O3" --CXXOPTFLAGS="-g -O3" --with-debugging=0 --with-cc=mpiicc --with-cxx=mpiicpc --with-fc=mpiifort --with-single-library=1 --with-mpiexec=mpiexec --with-precision=double --with-fortran-interfaces=1 --with-make=1 --with-mpi=1 --with-mpi-compilers=1 --download-fblaslapack=0 --download-hypre=1 --download-cmake=0 --with-cmake=1 --download-metis=1 --download-parmetis=1 --download-ptscotch=0 --download-suitesparse=1 --download-triangle=1 --download-superlu=1 --download-superlu_dist=1 --download-scalapack=1 --download-mumps=1 --download-elemental=1 --download-spai=0 --download-parms=1 --download-moab=1 --download-chaco=0 --download-fftw=1 --with-petsc4py=1 --download-mpi4py=1 --download-saws --download-concurrencykit=1 --download-revolve=1 --download-cams=1 --download-p4est=0 --with-zlib=1 --download-mfem=1 --download-glvis=0 --with-opengl=0 --download-libpng=1 --download-libjpeg=1 --download-slepc=1 --download-hpddm=1 --download-bamg=1 --download-mmg=0 --download-parmmg=0 --download-htool=1 --download-egads=0 --download-opencascade=0 PETSC_ARCH=x86_64 > [0]PETSC ERROR: #1 VecCopy() at /data/softs/petsc/src/vec/vec/interface/vector.c:1607 > [0]PETSC ERROR: #2 KSPSolve_BiCG() at /data/softs/petsc/src/ksp/ksp/impls/bicg/bicg.c:40 > [0]PETSC ERROR: #3 KSPSolve_Private() at /data/softs/petsc/src/ksp/ksp/interface/itfunc.c:877 > [0]PETSC ERROR: #4 KSPSolve() at /data/softs/petsc/src/ksp/ksp/interface/itfunc.c:1048 > [0]PETSC ERROR: #5 solve.F90:218 > Abort(75) on node 0 (rank 0 in comm 16): application called MPI_Abort(MPI_COMM_SELF, 75) - process 0 > -------------- next part -------------- An HTML attachment was scrubbed... URL: From snailsoar at hotmail.com Wed Sep 21 06:41:10 2022 From: snailsoar at hotmail.com (feng wang) Date: Wed, 21 Sep 2022 11:41:10 +0000 Subject: [petsc-users] Slepc, shell matrix, parallel, halo exchange Message-ID: Hello, I am using Slepc with a shell matrix. The sequential version seems working and now I am trying to make it run in parallel. The partition of the domain is done, I am not sure how to do the halo exchange in the shell matrix in Slepc. I have a parallel version of matrix-free GMRES in my code with Petsc. I was using VecCreateGhostBlock to create vector with ghost cells, and then used VecGhostUpdateBegin/End for the halo exchange in the shell matrix, would this be the same for Slepc? Thanks, Feng -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Sep 21 06:58:23 2022 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 21 Sep 2022 07:58:23 -0400 Subject: [petsc-users] Slepc, shell matrix, parallel, halo exchange In-Reply-To: References: Message-ID: On Wed, Sep 21, 2022 at 7:41 AM feng wang wrote: > Hello, > > I am using Slepc with a shell matrix. The sequential version seems working > and now I am trying to make it run in parallel. > > The partition of the domain is done, I am not sure how to do the halo > exchange in the shell matrix in Slepc. I have a parallel version of > matrix-free GMRES in my code with Petsc. I was using VecCreateGhostBlock to > create vector with ghost cells, and then used VecGhostUpdateBegin/End for > the halo exchange in the shell matrix, would this be the same for Slepc? > That will be enough for the MatMult(). You would also have to use a SLEPc EPS that only needed MatMult(). Thanks, Matt > Thanks, > Feng > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From snailsoar at hotmail.com Wed Sep 21 07:09:43 2022 From: snailsoar at hotmail.com (feng wang) Date: Wed, 21 Sep 2022 12:09:43 +0000 Subject: [petsc-users] Slepc, shell matrix, parallel, halo exchange In-Reply-To: References: Message-ID: Thanks for your reply. For GMRES, I create a ghost vector and give it to KSPSolve. For Slepc, it only takes the shell matrix for EPSSetOperators. Suppose the shell matrix of the eigensolver defines MatMult(Mat m ,Vec x, Vec y), how does it know Vec x is a ghost vector and how many ghost cells there are? Thanks, Feng ________________________________ From: Matthew Knepley Sent: 21 September 2022 11:58 To: feng wang Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] Slepc, shell matrix, parallel, halo exchange On Wed, Sep 21, 2022 at 7:41 AM feng wang > wrote: Hello, I am using Slepc with a shell matrix. The sequential version seems working and now I am trying to make it run in parallel. The partition of the domain is done, I am not sure how to do the halo exchange in the shell matrix in Slepc. I have a parallel version of matrix-free GMRES in my code with Petsc. I was using VecCreateGhostBlock to create vector with ghost cells, and then used VecGhostUpdateBegin/End for the halo exchange in the shell matrix, would this be the same for Slepc? That will be enough for the MatMult(). You would also have to use a SLEPc EPS that only needed MatMult(). Thanks, Matt Thanks, Feng -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From jroman at dsic.upv.es Wed Sep 21 07:34:15 2022 From: jroman at dsic.upv.es (Jose E. Roman) Date: Wed, 21 Sep 2022 14:34:15 +0200 Subject: [petsc-users] Slepc, shell matrix, parallel, halo exchange In-Reply-To: References: Message-ID: <53363D7B-CCBD-4DAB-924E-1D5D56975828@dsic.upv.es> If you define the MATOP_CREATE_VECS operation in your shell matrix so that it creates a ghost vector, then all vectors within EPS will be ghost vectors, including those that are received as arguments of MatMult(). Not sure if this will work. A simpler solution is that you store a ghost vector in the context of your shell matrix, and then in MatMult() you receive a regular parallel vector x, then update the ghost points using the auxiliary ghost vector, do the computation and store the result in the regular parallel vector y. Jose > El 21 sept 2022, a las 14:09, feng wang escribi?: > > Thanks for your reply. > > For GMRES, I create a ghost vector and give it to KSPSolve. For Slepc, it only takes the shell matrix for EPSSetOperators. Suppose the shell matrix of the eigensolver defines MatMult(Mat m ,Vec x, Vec y), how does it know Vec x is a ghost vector and how many ghost cells there are? > > Thanks, > Feng > From: Matthew Knepley > Sent: 21 September 2022 11:58 > To: feng wang > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Slepc, shell matrix, parallel, halo exchange > > On Wed, Sep 21, 2022 at 7:41 AM feng wang wrote: > Hello, > > I am using Slepc with a shell matrix. The sequential version seems working and now I am trying to make it run in parallel. > > The partition of the domain is done, I am not sure how to do the halo exchange in the shell matrix in Slepc. I have a parallel version of matrix-free GMRES in my code with Petsc. I was using VecCreateGhostBlock to create vector with ghost cells, and then used VecGhostUpdateBegin/End for the halo exchange in the shell matrix, would this be the same for Slepc? > > That will be enough for the MatMult(). You would also have to use a SLEPc EPS that only needed MatMult(). > > Thanks, > > Matt > > Thanks, > Feng > > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ From snailsoar at hotmail.com Wed Sep 21 07:47:38 2022 From: snailsoar at hotmail.com (feng wang) Date: Wed, 21 Sep 2022 12:47:38 +0000 Subject: [petsc-users] Slepc, shell matrix, parallel, halo exchange In-Reply-To: <53363D7B-CCBD-4DAB-924E-1D5D56975828@dsic.upv.es> References: <53363D7B-CCBD-4DAB-924E-1D5D56975828@dsic.upv.es> Message-ID: Thanks Jose, I will try this and will come back to this thread if I have any issue. Besides, for EPSGetEigenpair, I guess each rank gets its portion of the eigenvector, and I need to put them together afterwards? Thanks, Feng ________________________________ From: Jose E. Roman Sent: 21 September 2022 12:34 To: feng wang Cc: Matthew Knepley ; petsc-users at mcs.anl.gov Subject: Re: [petsc-users] Slepc, shell matrix, parallel, halo exchange If you define the MATOP_CREATE_VECS operation in your shell matrix so that it creates a ghost vector, then all vectors within EPS will be ghost vectors, including those that are received as arguments of MatMult(). Not sure if this will work. A simpler solution is that you store a ghost vector in the context of your shell matrix, and then in MatMult() you receive a regular parallel vector x, then update the ghost points using the auxiliary ghost vector, do the computation and store the result in the regular parallel vector y. Jose > El 21 sept 2022, a las 14:09, feng wang escribi?: > > Thanks for your reply. > > For GMRES, I create a ghost vector and give it to KSPSolve. For Slepc, it only takes the shell matrix for EPSSetOperators. Suppose the shell matrix of the eigensolver defines MatMult(Mat m ,Vec x, Vec y), how does it know Vec x is a ghost vector and how many ghost cells there are? > > Thanks, > Feng > From: Matthew Knepley > Sent: 21 September 2022 11:58 > To: feng wang > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Slepc, shell matrix, parallel, halo exchange > > On Wed, Sep 21, 2022 at 7:41 AM feng wang wrote: > Hello, > > I am using Slepc with a shell matrix. The sequential version seems working and now I am trying to make it run in parallel. > > The partition of the domain is done, I am not sure how to do the halo exchange in the shell matrix in Slepc. I have a parallel version of matrix-free GMRES in my code with Petsc. I was using VecCreateGhostBlock to create vector with ghost cells, and then used VecGhostUpdateBegin/End for the halo exchange in the shell matrix, would this be the same for Slepc? > > That will be enough for the MatMult(). You would also have to use a SLEPc EPS that only needed MatMult(). > > Thanks, > > Matt > > Thanks, > Feng > > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From jroman at dsic.upv.es Wed Sep 21 07:52:17 2022 From: jroman at dsic.upv.es (Jose E. Roman) Date: Wed, 21 Sep 2022 14:52:17 +0200 Subject: [petsc-users] Slepc, shell matrix, parallel, halo exchange In-Reply-To: References: <53363D7B-CCBD-4DAB-924E-1D5D56975828@dsic.upv.es> Message-ID: > El 21 sept 2022, a las 14:47, feng wang escribi?: > > Thanks Jose, I will try this and will come back to this thread if I have any issue. > > Besides, for EPSGetEigenpair, I guess each rank gets its portion of the eigenvector, and I need to put them together afterwards? Eigenvectors are stored in parallel vectors, which are used in subsequent parallel computation in most applications. If for some reason you need to gather them in a single MPI process you can use e.g. VecScatterCreateToZero() > > Thanks, > Feng > > From: Jose E. Roman > Sent: 21 September 2022 12:34 > To: feng wang > Cc: Matthew Knepley ; petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Slepc, shell matrix, parallel, halo exchange > > If you define the MATOP_CREATE_VECS operation in your shell matrix so that it creates a ghost vector, then all vectors within EPS will be ghost vectors, including those that are received as arguments of MatMult(). Not sure if this will work. > > A simpler solution is that you store a ghost vector in the context of your shell matrix, and then in MatMult() you receive a regular parallel vector x, then update the ghost points using the auxiliary ghost vector, do the computation and store the result in the regular parallel vector y. > > Jose > > > > El 21 sept 2022, a las 14:09, feng wang escribi?: > > > > Thanks for your reply. > > > > For GMRES, I create a ghost vector and give it to KSPSolve. For Slepc, it only takes the shell matrix for EPSSetOperators. Suppose the shell matrix of the eigensolver defines MatMult(Mat m ,Vec x, Vec y), how does it know Vec x is a ghost vector and how many ghost cells there are? > > > > Thanks, > > Feng > > From: Matthew Knepley > > Sent: 21 September 2022 11:58 > > To: feng wang > > Cc: petsc-users at mcs.anl.gov > > Subject: Re: [petsc-users] Slepc, shell matrix, parallel, halo exchange > > > > On Wed, Sep 21, 2022 at 7:41 AM feng wang wrote: > > Hello, > > > > I am using Slepc with a shell matrix. The sequential version seems working and now I am trying to make it run in parallel. > > > > The partition of the domain is done, I am not sure how to do the halo exchange in the shell matrix in Slepc. I have a parallel version of matrix-free GMRES in my code with Petsc. I was using VecCreateGhostBlock to create vector with ghost cells, and then used VecGhostUpdateBegin/End for the halo exchange in the shell matrix, would this be the same for Slepc? > > > > That will be enough for the MatMult(). You would also have to use a SLEPc EPS that only needed MatMult(). > > > > Thanks, > > > > Matt > > > > Thanks, > > Feng > > > > > > > > > > -- > > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > > -- Norbert Wiener > > > > https://www.cse.buffalo.edu/~knepley/ From jroman at dsic.upv.es Wed Sep 21 08:07:15 2022 From: jroman at dsic.upv.es (Jose E. Roman) Date: Wed, 21 Sep 2022 15:07:15 +0200 Subject: [petsc-users] Slepc, shell matrix, parallel, halo exchange In-Reply-To: References: <53363D7B-CCBD-4DAB-924E-1D5D56975828@dsic.upv.es> Message-ID: <76162134-CDE9-42B9-8310-D9DD33D2F12D@dsic.upv.es> > El 21 sept 2022, a las 14:47, feng wang escribi?: > > Thanks Jose, I will try this and will come back to this thread if I have any issue. > > Besides, for EPSGetEigenpair, I guess each rank gets its portion of the eigenvector, and I need to put them together afterwards? Eigenvectors are stored in parallel vectors, which are used in subsequent parallel computation in most applications. If for some reason you need to gather them in a single MPI process you can use e.g. VecScatterCreateToZero() > > Thanks, > Feng > > From: Jose E. Roman > Sent: 21 September 2022 12:34 > To: feng wang > Cc: Matthew Knepley ; petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Slepc, shell matrix, parallel, halo exchange > > If you define the MATOP_CREATE_VECS operation in your shell matrix so that it creates a ghost vector, then all vectors within EPS will be ghost vectors, including those that are received as arguments of MatMult(). Not sure if this will work. > > A simpler solution is that you store a ghost vector in the context of your shell matrix, and then in MatMult() you receive a regular parallel vector x, then update the ghost points using the auxiliary ghost vector, do the computation and store the result in the regular parallel vector y. > > Jose > > > > El 21 sept 2022, a las 14:09, feng wang escribi?: > > > > Thanks for your reply. > > > > For GMRES, I create a ghost vector and give it to KSPSolve. For Slepc, it only takes the shell matrix for EPSSetOperators. Suppose the shell matrix of the eigensolver defines MatMult(Mat m ,Vec x, Vec y), how does it know Vec x is a ghost vector and how many ghost cells there are? > > > > Thanks, > > Feng > > From: Matthew Knepley > > Sent: 21 September 2022 11:58 > > To: feng wang > > Cc: petsc-users at mcs.anl.gov > > Subject: Re: [petsc-users] Slepc, shell matrix, parallel, halo exchange > > > > On Wed, Sep 21, 2022 at 7:41 AM feng wang wrote: > > Hello, > > > > I am using Slepc with a shell matrix. The sequential version seems working and now I am trying to make it run in parallel. > > > > The partition of the domain is done, I am not sure how to do the halo exchange in the shell matrix in Slepc. I have a parallel version of matrix-free GMRES in my code with Petsc. I was using VecCreateGhostBlock to create vector with ghost cells, and then used VecGhostUpdateBegin/End for the halo exchange in the shell matrix, would this be the same for Slepc? > > > > That will be enough for the MatMult(). You would also have to use a SLEPc EPS that only needed MatMult(). > > > > Thanks, > > > > Matt > > > > Thanks, > > Feng > > > > > > > > > > -- > > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > > -- Norbert Wiener > > > > https://www.cse.buffalo.edu/~knepley/ From snailsoar at hotmail.com Wed Sep 21 09:35:32 2022 From: snailsoar at hotmail.com (feng wang) Date: Wed, 21 Sep 2022 14:35:32 +0000 Subject: [petsc-users] Slepc, shell matrix, parallel, halo exchange In-Reply-To: <76162134-CDE9-42B9-8310-D9DD33D2F12D@dsic.upv.es> References: <53363D7B-CCBD-4DAB-924E-1D5D56975828@dsic.upv.es> <76162134-CDE9-42B9-8310-D9DD33D2F12D@dsic.upv.es> Message-ID: Hi Jose, For your 2nd suggestion on halo exchange, I get the idea and roughly know how to do it, but there are some implementation details which I am not quite sure. If I understand it correctly, in MatMult(Mat m ,Vec x, Vec y), Vec x is a normal parallel vector and it does not contain halo values. Suppose I create an auxiliary ghost vector x_g, then I assign the values of x to x_g. The values of the halo for each partition will not be assigned at this stage. But If I call VecGhostUpdateBegin/End(x_g, INSERT_VALUES, SCATTER_FORWARD), this will fill the values of the halo cells of x_g for each partition. Then x_g has local and halo cells assigned correctly and I can use x_g to do my computation. Is this what you mean? Thanks, Feng ________________________________ From: Jose E. Roman Sent: 21 September 2022 13:07 To: feng wang Cc: Matthew Knepley ; petsc-users at mcs.anl.gov Subject: Re: [petsc-users] Slepc, shell matrix, parallel, halo exchange > El 21 sept 2022, a las 14:47, feng wang escribi?: > > Thanks Jose, I will try this and will come back to this thread if I have any issue. > > Besides, for EPSGetEigenpair, I guess each rank gets its portion of the eigenvector, and I need to put them together afterwards? Eigenvectors are stored in parallel vectors, which are used in subsequent parallel computation in most applications. If for some reason you need to gather them in a single MPI process you can use e.g. VecScatterCreateToZero() > > Thanks, > Feng > > From: Jose E. Roman > Sent: 21 September 2022 12:34 > To: feng wang > Cc: Matthew Knepley ; petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Slepc, shell matrix, parallel, halo exchange > > If you define the MATOP_CREATE_VECS operation in your shell matrix so that it creates a ghost vector, then all vectors within EPS will be ghost vectors, including those that are received as arguments of MatMult(). Not sure if this will work. > > A simpler solution is that you store a ghost vector in the context of your shell matrix, and then in MatMult() you receive a regular parallel vector x, then update the ghost points using the auxiliary ghost vector, do the computation and store the result in the regular parallel vector y. > > Jose > > > > El 21 sept 2022, a las 14:09, feng wang escribi?: > > > > Thanks for your reply. > > > > For GMRES, I create a ghost vector and give it to KSPSolve. For Slepc, it only takes the shell matrix for EPSSetOperators. Suppose the shell matrix of the eigensolver defines MatMult(Mat m ,Vec x, Vec y), how does it know Vec x is a ghost vector and how many ghost cells there are? > > > > Thanks, > > Feng > > From: Matthew Knepley > > Sent: 21 September 2022 11:58 > > To: feng wang > > Cc: petsc-users at mcs.anl.gov > > Subject: Re: [petsc-users] Slepc, shell matrix, parallel, halo exchange > > > > On Wed, Sep 21, 2022 at 7:41 AM feng wang wrote: > > Hello, > > > > I am using Slepc with a shell matrix. The sequential version seems working and now I am trying to make it run in parallel. > > > > The partition of the domain is done, I am not sure how to do the halo exchange in the shell matrix in Slepc. I have a parallel version of matrix-free GMRES in my code with Petsc. I was using VecCreateGhostBlock to create vector with ghost cells, and then used VecGhostUpdateBegin/End for the halo exchange in the shell matrix, would this be the same for Slepc? > > > > That will be enough for the MatMult(). You would also have to use a SLEPc EPS that only needed MatMult(). > > > > Thanks, > > > > Matt > > > > Thanks, > > Feng > > > > > > > > > > -- > > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > > -- Norbert Wiener > > > > https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Sep 21 09:36:26 2022 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 21 Sep 2022 10:36:26 -0400 Subject: [petsc-users] Slepc, shell matrix, parallel, halo exchange In-Reply-To: References: <53363D7B-CCBD-4DAB-924E-1D5D56975828@dsic.upv.es> <76162134-CDE9-42B9-8310-D9DD33D2F12D@dsic.upv.es> Message-ID: On Wed, Sep 21, 2022 at 10:35 AM feng wang wrote: > Hi Jose, > > For your 2nd suggestion on halo exchange, I get the idea and roughly know > how to do it, but there are some implementation details which I am not > quite sure. > > If I understand it correctly, in MatMult(Mat m ,Vec x, Vec y), Vec *x* is > a normal parallel vector and it does not contain halo values. Suppose I > create an auxiliary ghost vector * x_g*, then I assign the values of *x* > to *x_g*. The values of the halo for each partition will not be assigned > at this stage. > > But If I call VecGhostUpdateBegin/End(*x_g*, INSERT_VALUES, > SCATTER_FORWARD), this will fill the values of the halo cells of *x_g *for > each partition. Then *x_g* has local and halo cells assigned correctly > and I can use *x_g* to do my computation. Is this what you mean? > Yes Matt > Thanks, > Feng > > ------------------------------ > *From:* Jose E. Roman > *Sent:* 21 September 2022 13:07 > *To:* feng wang > *Cc:* Matthew Knepley ; petsc-users at mcs.anl.gov < > petsc-users at mcs.anl.gov> > *Subject:* Re: [petsc-users] Slepc, shell matrix, parallel, halo exchange > > > > > El 21 sept 2022, a las 14:47, feng wang > escribi?: > > > > Thanks Jose, I will try this and will come back to this thread if I have > any issue. > > > > Besides, for EPSGetEigenpair, I guess each rank gets its portion of the > eigenvector, and I need to put them together afterwards? > > Eigenvectors are stored in parallel vectors, which are used in subsequent > parallel computation in most applications. If for some reason you need to > gather them in a single MPI process you can use e.g. > VecScatterCreateToZero() > > > > > Thanks, > > Feng > > > > From: Jose E. Roman > > Sent: 21 September 2022 12:34 > > To: feng wang > > Cc: Matthew Knepley ; petsc-users at mcs.anl.gov < > petsc-users at mcs.anl.gov> > > Subject: Re: [petsc-users] Slepc, shell matrix, parallel, halo exchange > > > > If you define the MATOP_CREATE_VECS operation in your shell matrix so > that it creates a ghost vector, then all vectors within EPS will be ghost > vectors, including those that are received as arguments of MatMult(). Not > sure if this will work. > > > > A simpler solution is that you store a ghost vector in the context of > your shell matrix, and then in MatMult() you receive a regular parallel > vector x, then update the ghost points using the auxiliary ghost vector, do > the computation and store the result in the regular parallel vector y. > > > > Jose > > > > > > > El 21 sept 2022, a las 14:09, feng wang > escribi?: > > > > > > Thanks for your reply. > > > > > > For GMRES, I create a ghost vector and give it to KSPSolve. For Slepc, > it only takes the shell matrix for EPSSetOperators. Suppose the shell > matrix of the eigensolver defines MatMult(Mat m ,Vec x, Vec y), how does it > know Vec x is a ghost vector and how many ghost cells there are? > > > > > > Thanks, > > > Feng > > > From: Matthew Knepley > > > Sent: 21 September 2022 11:58 > > > To: feng wang > > > Cc: petsc-users at mcs.anl.gov > > > Subject: Re: [petsc-users] Slepc, shell matrix, parallel, halo exchange > > > > > > On Wed, Sep 21, 2022 at 7:41 AM feng wang > wrote: > > > Hello, > > > > > > I am using Slepc with a shell matrix. The sequential version seems > working and now I am trying to make it run in parallel. > > > > > > The partition of the domain is done, I am not sure how to do the halo > exchange in the shell matrix in Slepc. I have a parallel version of > matrix-free GMRES in my code with Petsc. I was using VecCreateGhostBlock to > create vector with ghost cells, and then used VecGhostUpdateBegin/End for > the halo exchange in the shell matrix, would this be the same for Slepc? > > > > > > That will be enough for the MatMult(). You would also have to use a > SLEPc EPS that only needed MatMult(). > > > > > > Thanks, > > > > > > Matt > > > > > > Thanks, > > > Feng > > > > > > > > > > > > > > > -- > > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > > > -- Norbert Wiener > > > > > > https://www.cse.buffalo.edu/~knepley/ > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ablay94z at gmail.com Wed Sep 21 18:31:37 2022 From: ablay94z at gmail.com (Abylay Zhumekenov) Date: Thu, 22 Sep 2022 02:31:37 +0300 Subject: [petsc-users] PCApplySymmetricRight for PCBJACOBI Message-ID: Hello, I have been playing around with a block Jacobi preconditioner (PCJACOBI) with an incomplete Cholesky (PCICC) sub-preconditioner. Although you cannot set KSPSetPCSide to PC_SYMMETRIC for PCBJACOBI, you still can do it for PCICC. I was surprised to find that PCApplySymmetricLeft is properly applied to a vector. However, PCApplySymmetricRight throws an error: ... [0]PETSC ERROR: Object is in wrong state [0]PETSC ERROR: VecPlaceArray() was already called on this vector, without a call to VecResetArray() ... [0]PETSC ERROR: #3 PCApplySymmetricLeft_BJacobi_Singleblock() at /home/abylay/KAUST/Libraries/C/petsc/src/ksp/pc/impls/bjacobi/bjacobi.c:660 [0]PETSC ERROR: #4 PCApplySymmetricLeft() at /home/abylay/KAUST/Libraries/C/petsc/src/ksp/pc/interface/precon.c:532 ... The problem goes away if I add: ... PetscCall(VecResetArray(bjac->x)); PetscCall(VecResetArray(bjac->y)); ... at line 698 in source file bjacobi.c. I don't know if it is a bug, and how I should report it, just wanted let someone know if it is. Thanks. Abylay Zhumekenov -------------- next part -------------- An HTML attachment was scrubbed... URL: From bantingl at myumanitoba.ca Wed Sep 21 21:20:02 2022 From: bantingl at myumanitoba.ca (Lucas Banting) Date: Thu, 22 Sep 2022 02:20:02 +0000 Subject: [petsc-users] Help building Zoltan with gcc11 Message-ID: Hello all, I am having a problem building Zoltan using the "--download-Zoltan" using gcc11 (default version on Ubuntu 22.04). I have opened an issue on the PETSc gitlab: https://gitlab.com/petsc/petsc/-/issues/1248 Which also has my configure.log. The error is: Error: Type mismatch between actual argument at (1) and actual argument at (2) (CHARACTER(128)/INTEGER(4)). Which I believe is related to gcc10 changing Fortran compiler behaviour. Lucas -------------- next part -------------- An HTML attachment was scrubbed... URL: From broheimhelldawg29 at gmail.com Sat Sep 24 10:04:53 2022 From: broheimhelldawg29 at gmail.com (Bro H) Date: Sat, 24 Sep 2022 22:04:53 +0700 Subject: [petsc-users] PETSc with 64 bit indices and MKL Sparse BLAS fails to build Message-ID: Hello. I would like to build PETSc with support for MKL Sparse BLAS, so that I can use MATAIJMKL for improved performance, but I also need to use 64 bit indices. I'm trying to build using the following parameters: ./configure --force --prefix="/opt/libs/petsc" --with-precision=double --with-64-bit-indices --with-blas-lapack-dir=/opt/intel/oneapi/mkl/latest/ --with-mkl_sparse=1 --with-mkl_sparse_optimize=1 --with-debugging=0 --with-shared-libraries=1 --with-cxx=0 --with-mpi=1 --with-hdf5=1 --with-hdf5-dir="${HDF5_ROOT}" --with-openmp=1 Configuration fails when using --with-64-bit-indices simultaneously with --with-mkl_sparse=1 and/or --with-mkl_sparse_optimize=1 with the following possible errors: "Cannot use mkl_sparse with 64 bit integers, it is not coded for this capability" "Cannot use mkl_sparse_optimize with 64 bit integers, it is not coded for this capability" But doesn't the latest version of MKL Sparse BLAS support 64 bit indices? It is using MKL_INT in mkl_spblas.h. Is it possible to build PETSc with 64 bit indices and MKL Sparse BLAS? From bsmith at petsc.dev Sat Sep 24 15:54:54 2022 From: bsmith at petsc.dev (Barry Smith) Date: Sat, 24 Sep 2022 16:54:54 -0400 Subject: [petsc-users] PETSc with 64 bit indices and MKL Sparse BLAS fails to build In-Reply-To: References: Message-ID: It is possible they recently added support for using it with 64 bit integers. You would need to through their documents to see how to get mkl_spblas.h. to use 64 bit integers and if the library for 64 bit integer has a different name that would need to be linked to. You would need to remove the requires32bitint = 1 from the various config/BuildSystem/config/packages/mkl_*.py routines to get configure to accept MKL sparse with 64 bit integers. > On Sep 24, 2022, at 11:04 AM, Bro H wrote: > > Hello. > > I would like to build PETSc with support for MKL Sparse BLAS, so that > I can use MATAIJMKL for improved performance, but I also need to use > 64 bit indices. I'm trying to build using the following parameters: > > ./configure --force --prefix="/opt/libs/petsc" --with-precision=double > --with-64-bit-indices > --with-blas-lapack-dir=/opt/intel/oneapi/mkl/latest/ > --with-mkl_sparse=1 --with-mkl_sparse_optimize=1 --with-debugging=0 > --with-shared-libraries=1 --with-cxx=0 --with-mpi=1 --with-hdf5=1 > --with-hdf5-dir="${HDF5_ROOT}" --with-openmp=1 > > Configuration fails when using --with-64-bit-indices simultaneously > with --with-mkl_sparse=1 and/or --with-mkl_sparse_optimize=1 with the > following possible errors: > > "Cannot use mkl_sparse with 64 bit integers, it is not coded for this > capability" > "Cannot use mkl_sparse_optimize with 64 bit integers, it is not coded > for this capability" > > But doesn't the latest version of MKL Sparse BLAS support 64 bit > indices? It is using MKL_INT in mkl_spblas.h. > > Is it possible to build PETSc with 64 bit indices and MKL Sparse BLAS? From broheimhelldawg29 at gmail.com Sun Sep 25 08:57:19 2022 From: broheimhelldawg29 at gmail.com (Bro H) Date: Sun, 25 Sep 2022 20:57:19 +0700 Subject: [petsc-users] PETSc with 64 bit indices and MKL Sparse BLAS fails to build In-Reply-To: References: Message-ID: Sorry, forgot to send a copy to petsc-users mailing list when I first replied. My first reply is below. On Sun, Sep 25, 2022 at 11:37 AM Bro H wrote: > > Barry, thank you for answering. I did some further testing. My MKL > version is 20220002 as detected by PETSc. I tried to compile one of > the examples distributed with MKL > (${MKLROOT}/latest/examples/examples_core_c.tgz/c/sparse_blas/source/sparse_csr.c) > that contains calls to mkl_sparse_d_mv function from Sparse BLAS, > among others. It compiled and ran without errors with 64-bit indices. > > I used the following command to compile this example on Debian 11: > > gcc `mkl_link_tool --quiet -c gnu_c -o gomp -i ilp64 -opts` > sparse_csr.c `mkl_link_tool --quiet -c gnu_c -o gomp -i ilp64 -libs` > > mkl_link_tool is bundled with MKL. It returns: > > `mkl_link_tool --quiet -c gnu_c -o gomp -i ilp64 -opts` = -DMKL_ILP64 > -m64 -I"${MKLROOT}/include" > > `mkl_link_tool --quiet -c gnu_c -o gomp -i ilp64 -opts` = > -L${MKLROOT}/lib/intel64 -Wl,--no-as-needed -lmkl_intel_ilp64 > -lmkl_gnu_thread -lmkl_core -lgomp -lpthread -lm -ldl > > So in theory it should work when PetscInt is 64 bits, am I right? The > only problem is with the build scripts or source code in PETSc? > > Per your suggestion, I have tried to change requires32bitint to 0 in > mkl_sparse.py and mkl_sparse_optimize.py routines. PETSc used 64-bit > indices, but configuration also set MKL to 32-bit indices, so I added > --with-64-bit-blas-indices. I then ran into "Cannot use mkl_sparse > with 64-bit BLAS/Lapack indices" error, and so I tried to change > requires32bitintblas to 0 in config/BuildSystem/config/package.py. > Configuration was successful with both PETSc and MKL set to 64-bit > indices, however, it looks like PETSc was still using 32-bit indices > during compilation of aijmkl implementation, with warnings such as: > > petsc-3.17.4/src/mat/impls/aij/seq/aijmkl/aijmkl.c:282:98: warning: > passing argument 7 of ?mkl_sparse_d_export_csr? from incompatible > pointer type > > Are there any other modifications required to make PETSc work with > 64-bit MKL Sparse BLAS? > > On Sun, Sep 25, 2022 at 3:54 AM Barry Smith wrote: > > > > > > It is possible they recently added support for using it with 64 bit integers. You would need to through their documents to see how to get mkl_spblas.h. to use 64 bit integers and if the library for 64 bit integer has a different name that would need to be linked to. > > > > You would need to remove the requires32bitint = 1 from the various config/BuildSystem/config/packages/mkl_*.py routines to get configure to accept MKL sparse with 64 bit integers. > > > > > > > On Sep 24, 2022, at 11:04 AM, Bro H wrote: > > > > > > Hello. > > > > > > I would like to build PETSc with support for MKL Sparse BLAS, so that > > > I can use MATAIJMKL for improved performance, but I also need to use > > > 64 bit indices. I'm trying to build using the following parameters: > > > > > > ./configure --force --prefix="/opt/libs/petsc" --with-precision=double > > > --with-64-bit-indices > > > --with-blas-lapack-dir=/opt/intel/oneapi/mkl/latest/ > > > --with-mkl_sparse=1 --with-mkl_sparse_optimize=1 --with-debugging=0 > > > --with-shared-libraries=1 --with-cxx=0 --with-mpi=1 --with-hdf5=1 > > > --with-hdf5-dir="${HDF5_ROOT}" --with-openmp=1 > > > > > > Configuration fails when using --with-64-bit-indices simultaneously > > > with --with-mkl_sparse=1 and/or --with-mkl_sparse_optimize=1 with the > > > following possible errors: > > > > > > "Cannot use mkl_sparse with 64 bit integers, it is not coded for this > > > capability" > > > "Cannot use mkl_sparse_optimize with 64 bit integers, it is not coded > > > for this capability" > > > > > > But doesn't the latest version of MKL Sparse BLAS support 64 bit > > > indices? It is using MKL_INT in mkl_spblas.h. > > > > > > Is it possible to build PETSc with 64 bit indices and MKL Sparse BLAS? > > From bsmith at petsc.dev Sun Sep 25 09:50:21 2022 From: bsmith at petsc.dev (Barry Smith) Date: Sun, 25 Sep 2022 10:50:21 -0400 Subject: [petsc-users] PETSc with 64 bit indices and MKL Sparse BLAS fails to build In-Reply-To: References: Message-ID: Likely you can fix the problem by adding #if defined(PETSC_HAVE_MKL_INTEL_ILP64) #define MKL_ILP64 #endif before the #include in src/mat/impls/aij/seq/aijmkl/aijmkl.c and src/mat/impls/baij/seq/baijmkl/baijmkl.c Please let us know if this resolves the problem. Barry > On Sep 25, 2022, at 12:37 AM, Bro H wrote: > > Barry, thank you for answering. I did some further testing. My MKL > version is 20220002 as detected by PETSc. I tried to compile one of > the examples distributed with MKL > (${MKLROOT}/latest/examples/examples_core_c.tgz/c/sparse_blas/source/sparse_csr.c) > that contains calls to mkl_sparse_d_mv function from Sparse BLAS, > among others. It compiled and ran without errors with 64-bit indices. > > I used the following command to compile this example on Debian 11: > > gcc `mkl_link_tool --quiet -c gnu_c -o gomp -i ilp64 -opts` > sparse_csr.c `mkl_link_tool --quiet -c gnu_c -o gomp -i ilp64 -libs` > > mkl_link_tool is bundled with MKL. It returns: > > `mkl_link_tool --quiet -c gnu_c -o gomp -i ilp64 -opts` = -DMKL_ILP64 > -m64 -I"${MKLROOT}/include" > > `mkl_link_tool --quiet -c gnu_c -o gomp -i ilp64 -opts` = > -L${MKLROOT}/lib/intel64 -Wl,--no-as-needed -lmkl_intel_ilp64 > -lmkl_gnu_thread -lmkl_core -lgomp -lpthread -lm -ldl > > So in theory it should work when PetscInt is 64 bits, am I right? The > only problem is with the build scripts or source code in PETSc? > > Per your suggestion, I have tried to change requires32bitint to 0 in > mkl_sparse.py and mkl_sparse_optimize.py routines. PETSc used 64-bit > indices, but configuration also set MKL to 32-bit indices, so I added > --with-64-bit-blas-indices. I then ran into "Cannot use mkl_sparse > with 64-bit BLAS/Lapack indices" error, and so I tried to change > requires32bitintblas to 0 in config/BuildSystem/config/package.py. > Configuration was successful with both PETSc and MKL set to 64-bit > indices, however, it looks like PETSc was still using 32-bit indices > during compilation of aijmkl implementation, with warnings such as: > > petsc-3.17.4/src/mat/impls/aij/seq/aijmkl/aijmkl.c:282:98: warning: > passing argument 7 of ?mkl_sparse_d_export_csr? from incompatible > pointer type > > Are there any other modifications required to make PETSc work with > 64-bit MKL Sparse BLAS? > > On Sun, Sep 25, 2022 at 3:54 AM Barry Smith wrote: >> >> >> It is possible they recently added support for using it with 64 bit integers. You would need to through their documents to see how to get mkl_spblas.h. to use 64 bit integers and if the library for 64 bit integer has a different name that would need to be linked to. >> >> You would need to remove the requires32bitint = 1 from the various config/BuildSystem/config/packages/mkl_*.py routines to get configure to accept MKL sparse with 64 bit integers. >> >> >>> On Sep 24, 2022, at 11:04 AM, Bro H wrote: >>> >>> Hello. >>> >>> I would like to build PETSc with support for MKL Sparse BLAS, so that >>> I can use MATAIJMKL for improved performance, but I also need to use >>> 64 bit indices. I'm trying to build using the following parameters: >>> >>> ./configure --force --prefix="/opt/libs/petsc" --with-precision=double >>> --with-64-bit-indices >>> --with-blas-lapack-dir=/opt/intel/oneapi/mkl/latest/ >>> --with-mkl_sparse=1 --with-mkl_sparse_optimize=1 --with-debugging=0 >>> --with-shared-libraries=1 --with-cxx=0 --with-mpi=1 --with-hdf5=1 >>> --with-hdf5-dir="${HDF5_ROOT}" --with-openmp=1 >>> >>> Configuration fails when using --with-64-bit-indices simultaneously >>> with --with-mkl_sparse=1 and/or --with-mkl_sparse_optimize=1 with the >>> following possible errors: >>> >>> "Cannot use mkl_sparse with 64 bit integers, it is not coded for this >>> capability" >>> "Cannot use mkl_sparse_optimize with 64 bit integers, it is not coded >>> for this capability" >>> >>> But doesn't the latest version of MKL Sparse BLAS support 64 bit >>> indices? It is using MKL_INT in mkl_spblas.h. >>> >>> Is it possible to build PETSc with 64 bit indices and MKL Sparse BLAS? >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Sun Sep 25 09:55:48 2022 From: bsmith at petsc.dev (Barry Smith) Date: Sun, 25 Sep 2022 10:55:48 -0400 Subject: [petsc-users] PCApplySymmetricRight for PCBJACOBI (fwd) In-Reply-To: <418d72d5-7fd4-8fcf-1478-cbc7f69cfebb@mcs.anl.gov> References: <418d72d5-7fd4-8fcf-1478-cbc7f69cfebb@mcs.anl.gov> Message-ID: Thanks for the bug report; your fix is correct. I have corrected it in PETSc and also added support for multiple block per MPI rank in https://gitlab.com/petsc/petsc/-/merge_requests/5678 Barry > > > ---------- Forwarded message ---------- > Date: Thu, 22 Sep 2022 02:31:37 +0300 > From: Abylay Zhumekenov > To: petsc-users at mcs.anl.gov > Subject: [petsc-users] PCApplySymmetricRight for PCBJACOBI > > Hello, > > I have been playing around with a block Jacobi preconditioner (PCJACOBI) > with an incomplete Cholesky (PCICC) sub-preconditioner. Although you cannot > set KSPSetPCSide to PC_SYMMETRIC for PCBJACOBI, you still can do it for > PCICC. I was surprised to find that PCApplySymmetricLeft is properly > applied to a vector. However, PCApplySymmetricRight throws an error: > ... > [0]PETSC ERROR: Object is in wrong state > [0]PETSC ERROR: VecPlaceArray() was already called on this vector, > without a call to VecResetArray() > ... > [0]PETSC ERROR: #3 PCApplySymmetricLeft_BJacobi_Singleblock() at > /home/abylay/KAUST/Libraries/C/petsc/src/ksp/pc/impls/bjacobi/bjacobi.c:660 > [0]PETSC ERROR: #4 PCApplySymmetricLeft() at > /home/abylay/KAUST/Libraries/C/petsc/src/ksp/pc/interface/precon.c:532 > ... > The problem goes away if I add: > ... > PetscCall(VecResetArray(bjac->x)); > PetscCall(VecResetArray(bjac->y)); > ... > at line 698 in source file bjacobi.c. I don't know if it is a bug, and how > I should report it, just wanted let someone know if it is. > Thanks. > > Abylay Zhumekenov -------------- next part -------------- An HTML attachment was scrubbed... URL: From broheimhelldawg29 at gmail.com Sun Sep 25 14:47:08 2022 From: broheimhelldawg29 at gmail.com (Bro H) Date: Mon, 26 Sep 2022 02:47:08 +0700 Subject: [petsc-users] PETSc with 64 bit indices and MKL Sparse BLAS fails to build In-Reply-To: References: Message-ID: Barry, thanks, it appears to be working correctly after following your latest suggestion. I used the following configuration command: ./configure --force --prefix=/opt/libs/petsc COPTFLAGS="-O3" CXXOPTFLAGS="-O3" FOPTFLAGS="-O3" --with-precision=double --with-64-bit-indices --with-64-bit-blas-indices --known-64-bit-blas-indices=1 --with-blas-lapack-dir=/opt/intel/oneapi/mkl/latest/ --with-mkl_sparse=1 --with-mkl_sparse_optimize=1 --with-debugging=0 --with-shared-libraries=1 --with-cxx=0 --with-mpi=1 --with-hdf5=1 --with-hdf5-dir="${HDF5_ROOT}" --with-openmp=1 However, I still get warnings during build, with the exception that MKL Sparse BLAS now expects long long int instead of int for MKL_INT: warning: passing argument 7 of ?mkl_sparse_d_export_csr? from incompatible pointer type [-Wincompatible-pointer-types] note: expected ?long long int **? but argument is of type ?PetscInt **? {aka ?long int **?} Not a big problem for me right now since they are the same size in my case. But these warnings should probably be fixed in the future. Could you or other PETSc devs please add official support for MKL Sparse BLAS with 64-bit indices in an upcoming PETSc release? On Sun, Sep 25, 2022 at 9:50 PM Barry Smith wrote: > > > Likely you can fix the problem by adding > > #if defined(PETSC_HAVE_MKL_INTEL_ILP64) > #define MKL_ILP64 > #endif > > before the #include in > > src/mat/impls/aij/seq/aijmkl/aijmkl.c and > src/mat/impls/baij/seq/baijmkl/baijmkl.c > > Please let us know if this resolves the problem. > > Barry > > On Sep 25, 2022, at 12:37 AM, Bro H wrote: > > Barry, thank you for answering. I did some further testing. My MKL > version is 20220002 as detected by PETSc. I tried to compile one of > the examples distributed with MKL > (${MKLROOT}/latest/examples/examples_core_c.tgz/c/sparse_blas/source/sparse_csr.c) > that contains calls to mkl_sparse_d_mv function from Sparse BLAS, > among others. It compiled and ran without errors with 64-bit indices. > > I used the following command to compile this example on Debian 11: > > gcc `mkl_link_tool --quiet -c gnu_c -o gomp -i ilp64 -opts` > sparse_csr.c `mkl_link_tool --quiet -c gnu_c -o gomp -i ilp64 -libs` > > mkl_link_tool is bundled with MKL. It returns: > > `mkl_link_tool --quiet -c gnu_c -o gomp -i ilp64 -opts` = -DMKL_ILP64 > -m64 -I"${MKLROOT}/include" > > `mkl_link_tool --quiet -c gnu_c -o gomp -i ilp64 -opts` = > -L${MKLROOT}/lib/intel64 -Wl,--no-as-needed -lmkl_intel_ilp64 > -lmkl_gnu_thread -lmkl_core -lgomp -lpthread -lm -ldl > > So in theory it should work when PetscInt is 64 bits, am I right? The > only problem is with the build scripts or source code in PETSc? > > Per your suggestion, I have tried to change requires32bitint to 0 in > mkl_sparse.py and mkl_sparse_optimize.py routines. PETSc used 64-bit > indices, but configuration also set MKL to 32-bit indices, so I added > --with-64-bit-blas-indices. I then ran into "Cannot use mkl_sparse > with 64-bit BLAS/Lapack indices" error, and so I tried to change > requires32bitintblas to 0 in config/BuildSystem/config/package.py. > Configuration was successful with both PETSc and MKL set to 64-bit > indices, however, it looks like PETSc was still using 32-bit indices > during compilation of aijmkl implementation, with warnings such as: > > petsc-3.17.4/src/mat/impls/aij/seq/aijmkl/aijmkl.c:282:98: warning: > passing argument 7 of ?mkl_sparse_d_export_csr? from incompatible > pointer type > > Are there any other modifications required to make PETSc work with > 64-bit MKL Sparse BLAS? > > On Sun, Sep 25, 2022 at 3:54 AM Barry Smith wrote: > > > > It is possible they recently added support for using it with 64 bit integers. You would need to through their documents to see how to get mkl_spblas.h. to use 64 bit integers and if the library for 64 bit integer has a different name that would need to be linked to. > > You would need to remove the requires32bitint = 1 from the various config/BuildSystem/config/packages/mkl_*.py routines to get configure to accept MKL sparse with 64 bit integers. > > > On Sep 24, 2022, at 11:04 AM, Bro H wrote: > > Hello. > > I would like to build PETSc with support for MKL Sparse BLAS, so that > I can use MATAIJMKL for improved performance, but I also need to use > 64 bit indices. I'm trying to build using the following parameters: > > ./configure --force --prefix="/opt/libs/petsc" --with-precision=double > --with-64-bit-indices > --with-blas-lapack-dir=/opt/intel/oneapi/mkl/latest/ > --with-mkl_sparse=1 --with-mkl_sparse_optimize=1 --with-debugging=0 > --with-shared-libraries=1 --with-cxx=0 --with-mpi=1 --with-hdf5=1 > --with-hdf5-dir="${HDF5_ROOT}" --with-openmp=1 > > Configuration fails when using --with-64-bit-indices simultaneously > with --with-mkl_sparse=1 and/or --with-mkl_sparse_optimize=1 with the > following possible errors: > > "Cannot use mkl_sparse with 64 bit integers, it is not coded for this > capability" > "Cannot use mkl_sparse_optimize with 64 bit integers, it is not coded > for this capability" > > But doesn't the latest version of MKL Sparse BLAS support 64 bit > indices? It is using MKL_INT in mkl_spblas.h. > > Is it possible to build PETSc with 64 bit indices and MKL Sparse BLAS? > > > From ablay94z at gmail.com Sun Sep 25 15:13:51 2022 From: ablay94z at gmail.com (Abylay Zhumekenov) Date: Sun, 25 Sep 2022 23:13:51 +0300 Subject: [petsc-users] PCApplySymmetricRight for PCBJACOBI (fwd) In-Reply-To: References: <418d72d5-7fd4-8fcf-1478-cbc7f69cfebb@mcs.anl.gov> Message-ID: Great, thanks! On Sun, 25 Sep 2022, 17:55 Barry Smith, wrote: > > Thanks for the bug report; your fix is correct. I have corrected it in > PETSc and also added support for multiple block per MPI rank in > https://gitlab.com/petsc/petsc/-/merge_requests/5678 > > Barry > > > > > ---------- Forwarded message ---------- > Date: Thu, 22 Sep 2022 02:31:37 +0300 > From: Abylay Zhumekenov > To: petsc-users at mcs.anl.gov > Subject: [petsc-users] PCApplySymmetricRight for PCBJACOBI > > Hello, > > I have been playing around with a block Jacobi preconditioner (PCJACOBI) > with an incomplete Cholesky (PCICC) sub-preconditioner. Although you cannot > set KSPSetPCSide to PC_SYMMETRIC for PCBJACOBI, you still can do it for > PCICC. I was surprised to find that PCApplySymmetricLeft is properly > applied to a vector. However, PCApplySymmetricRight throws an error: > ... > [0]PETSC ERROR: Object is in wrong state > [0]PETSC ERROR: VecPlaceArray() was already called on this vector, > without a call to VecResetArray() > ... > [0]PETSC ERROR: #3 PCApplySymmetricLeft_BJacobi_Singleblock() at > /home/abylay/KAUST/Libraries/C/petsc/src/ksp/pc/impls/bjacobi/bjacobi.c:660 > [0]PETSC ERROR: #4 PCApplySymmetricLeft() at > /home/abylay/KAUST/Libraries/C/petsc/src/ksp/pc/interface/precon.c:532 > ... > The problem goes away if I add: > ... > PetscCall(VecResetArray(bjac->x)); > PetscCall(VecResetArray(bjac->y)); > ... > at line 698 in source file bjacobi.c. I don't know if it is a bug, and how > I should report it, just wanted let someone know if it is. > Thanks. > > Abylay Zhumekenov > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From la.abdala at gmail.com Sun Sep 25 15:49:12 2022 From: la.abdala at gmail.com (Laryssa Abdala) Date: Sun, 25 Sep 2022 16:49:12 -0400 Subject: [petsc-users] C++ error! MPI_Finalize() could not be located! Message-ID: Hello, I am trying to configure PETSc and have been getting the following message: TESTING: CxxMPICheck from config.packages.MPI(config/BuildSystem/config/packages/MPI.py:416) ******************************************************************************* UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): ------------------------------------------------------------------------------- C++ error! MPI_Finalize() could not be located! ******************************************************************************* I attached my configure.log file. Any suggestions on how to fix this issue would be appreciated. Thanks, Laryssa -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: configure.log Type: application/octet-stream Size: 985368 bytes Desc: not available URL: From bsmith at petsc.dev Sun Sep 25 16:06:46 2022 From: bsmith at petsc.dev (Barry Smith) Date: Sun, 25 Sep 2022 17:06:46 -0400 Subject: [petsc-users] C++ error! MPI_Finalize() could not be located! In-Reply-To: References: Message-ID: It appears you want to use MPI (if not pass --with-mpi=0 also). Thus you must either 1) have the MPI compiler wrappers in your path (mpicc, mpicxx, mpif90) or use --with-mpi-dir=somedirectory where MPI is installed and do NOT provide the compiler names (since MPI provides compiler wrappers). or 2) use the current configure options and add --download-mpich to have MPI installed for you. Barry > On Sep 25, 2022, at 4:49 PM, Laryssa Abdala wrote: > > Hello, > > I am trying to configure PETSc and have been getting the following message: > TESTING: CxxMPICheck from config.packages.MPI(config/BuildSystem/config/packages/MPI.py:416) ******************************************************************************* > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): > ------------------------------------------------------------------------------- > C++ error! MPI_Finalize() could not be located! > ******************************************************************************* > > I attached my configure.log file. Any suggestions on how to fix this issue would be appreciated. > > Thanks, > Laryssa > From la.abdala at gmail.com Sun Sep 25 16:30:02 2022 From: la.abdala at gmail.com (Laryssa Abdala) Date: Sun, 25 Sep 2022 17:30:02 -0400 Subject: [petsc-users] C++ error! MPI_Finalize() could not be located! In-Reply-To: References: Message-ID: Amazing. Thank you so much for your clear explanation, Barry! Laryssa On Sun, Sep 25, 2022 at 5:06 PM Barry Smith wrote: > > It appears you want to use MPI (if not pass --with-mpi=0 also). > > Thus you must either > > 1) have the MPI compiler wrappers in your path (mpicc, mpicxx, > mpif90) or use --with-mpi-dir=somedirectory where MPI is installed and do > NOT provide the compiler names (since MPI provides compiler wrappers). or > > 2) use the current configure options and add --download-mpich to have > MPI installed for you. > > Barry > > > > > > > On Sep 25, 2022, at 4:49 PM, Laryssa Abdala wrote: > > > > Hello, > > > > I am trying to configure PETSc and have been getting the following > message: > > TESTING: CxxMPICheck from > config.packages.MPI(config/BuildSystem/config/packages/MPI.py:416) > > > ******************************************************************************* > > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log > for details): > > > ------------------------------------------------------------------------------- > > C++ error! MPI_Finalize() could not be located! > > > ******************************************************************************* > > > > I attached my configure.log file. Any suggestions on how to fix this > issue would be appreciated. > > > > Thanks, > > Laryssa > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tlanyan at hotmail.com Sun Sep 25 22:02:38 2022 From: tlanyan at hotmail.com (=?utf-8?B?5pmT5bOwIOS9lQ==?=) Date: Mon, 26 Sep 2022 03:02:38 +0000 Subject: [petsc-users] Solve Linear System with Field Split Preconditioner Message-ID: Hi all, I have a linear system formed from structural mechanics, and there exists zero in the diagonal entries: A = (A00 A01 A10 A11), where A00 has inverse and the diagonal entries in A11 are all zero. The GMRES method with ILU preconditioner in PETSc was carried out to solve this system, and I got this error: "PC failed due to FACTOR_NUMERIC_ZEROPIVOT" I googled and found that Field Split preconditioner should be applied to solve this kind of systems. I passed -pc-type fieldsplit -pc_fieldsplit_detect_saddle_point options to program, and got another error: ?PC failed due to SUBPC_ERROR? After that, I tried to split the matrix by codes: IS is; ISCreateStride(PETSC_COMM_SELF, row number of A00, 0, 1, &is); PCFieldSplitSetIS(pc, ?0?, is); IS is2; ISCreateStride(PETSC_COMM_SELF, row number of A11, start row of A11, 1, &is2); PCFieldSplitSetIS(pc, ?1?, is2); Recompiled the codes and run it with additional options: -fieldsplit_0_ksp_type gmres -fieldsplit_0_pc_type ilu -fieldsplit_1_ksp_type gmres -fieldsplit_1_pc_type none To my surprise, the Krylov solver executed extremely slow. I googled and found the reason is that if size of A00 is much bigger than A11(e.g. A00 has 10000 rows and A11 has 50 rows), then Schur complement is inefficient. Could you help me to solve this kind of systems efficiently? Best regards, Xiaofeng From jed at jedbrown.org Sun Sep 25 23:16:07 2022 From: jed at jedbrown.org (Jed Brown) Date: Sun, 25 Sep 2022 22:16:07 -0600 Subject: [petsc-users] Solve Linear System with Field Split Preconditioner In-Reply-To: References: Message-ID: <87v8pahit4.fsf@jedbrown.org> The usual issue is that you need a preconditioner for the Schur complement S = A11 - A01 A00^{-1} A10. For incompressible elasticity, this S is spectrally equivalent to a scaled mass matrix. ?? ? writes: > Hi all, > > I have a linear system formed from structural mechanics, and there exists zero in the diagonal entries: > > A = (A00 A01 > A10 A11), where A00 has inverse and the diagonal entries in A11 are all zero. > > The GMRES method with ILU preconditioner in PETSc was carried out to solve this system, and I got this error: > > "PC failed due to FACTOR_NUMERIC_ZEROPIVOT" > > I googled and found that Field Split preconditioner should be applied to solve this kind of systems. I passed -pc-type fieldsplit -pc_fieldsplit_detect_saddle_point options to program, and got another error: > > ?PC failed due to SUBPC_ERROR? > > After that, I tried to split the matrix by codes: > > IS is; > ISCreateStride(PETSC_COMM_SELF, row number of A00, 0, 1, &is); > PCFieldSplitSetIS(pc, ?0?, is); > > IS is2; > ISCreateStride(PETSC_COMM_SELF, row number of A11, start row of A11, 1, &is2); > PCFieldSplitSetIS(pc, ?1?, is2); > > Recompiled the codes and run it with additional options: > > -fieldsplit_0_ksp_type gmres -fieldsplit_0_pc_type ilu -fieldsplit_1_ksp_type gmres -fieldsplit_1_pc_type none > > To my surprise, the Krylov solver executed extremely slow. I googled and found the reason is that if size of A00 is much bigger than A11(e.g. A00 has 10000 rows and A11 has 50 rows), then Schur complement is inefficient. > > Could you help me to solve this kind of systems efficiently? > > Best regards, > > Xiaofeng From tlanyan at hotmail.com Sun Sep 25 23:35:58 2022 From: tlanyan at hotmail.com (=?utf-8?B?5pmT5bOwIOS9lQ==?=) Date: Mon, 26 Sep 2022 04:35:58 +0000 Subject: [petsc-users] Solve Linear System with Field Split Preconditioner In-Reply-To: References: Message-ID: <819A69F5-6CB0-494E-80B2-18EB023FD048@hotmail.com> If assigned a preconditioner to A11 with this cmd options: -fieldsplit_0_ksp_type gmres -fieldsplit_0_pc_type ilu -fieldsplit_1_ksp_type gmres -fieldsplit_1_pc_type ilu Then I got this error: "Could not locate a solver type for factorization type ILU and matrix type schurcomplement" How could I specify a preconditioner for A11? BR, Xiaofeng On Sep 26, 2022, at 11:02, ?? ? > wrote: -fieldsplit_0_ksp_type gmres -fieldsplit_0_pc_type ilu -fieldsplit_1_ksp_type gmres -fieldsplit_1_pc_type none -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Mon Sep 26 01:13:18 2022 From: jed at jedbrown.org (Jed Brown) Date: Mon, 26 Sep 2022 00:13:18 -0600 Subject: [petsc-users] Solve Linear System with Field Split Preconditioner In-Reply-To: <819A69F5-6CB0-494E-80B2-18EB023FD048@hotmail.com> References: <819A69F5-6CB0-494E-80B2-18EB023FD048@hotmail.com> Message-ID: <87pmfihddt.fsf@jedbrown.org> This is the joy of factorization field-split methods. The actual Schur complement is dense, so we represent it implicitly. A common strategy is to assemble the mass matrix and drop it in the 11 block of the Pmat. You can check out some examples in the repository for incompressible flow (Stokes problems). The LSC (least squares commutator) is another option. You'll likely find that lumping diag(A00)^{-1} works poorly because the resulting operator behaves like a Laplacian rather than like a mass matrix. ?? ? writes: > If assigned a preconditioner to A11 with this cmd options: > > -fieldsplit_0_ksp_type gmres -fieldsplit_0_pc_type ilu -fieldsplit_1_ksp_type gmres -fieldsplit_1_pc_type ilu > > Then I got this error: > > "Could not locate a solver type for factorization type ILU and matrix type schurcomplement" > > How could I specify a preconditioner for A11? > > BR, > Xiaofeng > > > On Sep 26, 2022, at 11:02, ?? ? > wrote: > > -fieldsplit_0_ksp_type gmres -fieldsplit_0_pc_type ilu -fieldsplit_1_ksp_type gmres -fieldsplit_1_pc_type none From fujisan43 at gmail.com Mon Sep 26 02:06:25 2022 From: fujisan43 at gmail.com (fujisan) Date: Mon, 26 Sep 2022 09:06:25 +0200 Subject: [petsc-users] Problem solving Ax=b with rectangular matrix A In-Reply-To: References: Message-ID: Thank you Pierre, I used PCNONE along with KSPLSQR and it worked. But as for PCQR, it cannot be found. There is nothing about it in the documentation as well. Fuji On Wed, Sep 21, 2022 at 12:20 PM Pierre Jolivet wrote: > Yes, but you need to use a KSP that handles rectangular Mat, such as > KSPLSQR (-ksp_type lsqr). > PCLU does not handle rectangular Pmat. The only PC that handle rectangular > Pmat are PCQR, PCNONE. > If you supply the normal equations as the Pmat for LSQR, then you can use > ?standard? PC. > You can have a look at > https://petsc.org/main/src/ksp/ksp/tutorials/ex27.c.html that covers most > of these cases. > > Thanks, > Pierre > > (sorry for the earlier answer sent wrongfully to petsc-maint, please > discard the previous email) > > On 21 Sep 2022, at 10:03 AM, fujisan wrote: > > I'm trying to solve Ax=b with a sparse rectangular matrix A (of size 33x17 > in my test) using > options '-ksp_type stcg -pc_type lu' on 1 or 2 cpus. > > And I always get an error saying "Incompatible vector local lengths" (see > below). > > Here is the relevant lines of my code: > > program test > ... > ! Variable declarations > > PetscCallA(PetscInitialize(PETSC_NULL_CHARACTER,ierr)) > > PetscCall(MatCreate(PETSC_COMM_WORLD,A,ierr)) > PetscCall(MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,m,n,ierr)) > PetscCall(MatSetType(A,MATMPIAIJ,ierr)) > PetscCall(MatSetFromOptions(A,ierr)) > PetscCall(MatSetUp(A,ierr)) > PetscCall(MatGetOwnershipRange(A,istart,iend,ierr)) > > do irow=istart,iend-1 > ... Reading from file ... > PetscCall(MatSetValues(A,1,irow,nzv,col,val,ADD_VALUES,ierr)) > ... > enddo > > PetscCall(MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY,ierr)) > PetscCall(MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY,ierr)) > > ! Creating vectors x and b > PetscCallA(MatCreateVecs(A,x,b,ierr)) > > ! Duplicating x in u. > PetscCallA(VecDuplicate(x,u,ierr)) > > ! u is used to calculate b > PetscCallA(VecSet(u,1.0,ierr)) > > PetscCallA(VecAssemblyBegin(u,ierr)) > PetscCallA(VecAssemblyEnd(u,ierr)) > > ! Calculating Au = b > PetscCallA(MatMult(A,u,b,ierr)) ! A.u = b > > PetscCallA(KSPSetType(ksp,KSPCG,ierr)) > > PetscCallA(KSPSetOperators(ksp,A,A,ierr)) > > PetscCallA(KSPSetFromOptions(ksp,ierr)) > > ! Solving Ax = b, x unknown > PetscCallA(KSPSolve(ksp,b,x,ierr)) > > PetscCallA(VecDestroy(x,ierr)) > PetscCallA(VecDestroy(u,ierr)) > PetscCallA(VecDestroy(b,ierr)) > PetscCallA(MatDestroy(A,ierr)) > PetscCallA(KSPDestroy(ksp,ierr)) > > call PetscFinalize(ierr) > end program > > The code reads a sparse matrix from a binary file. > I also output the sizes of matrix A and vectors b, x, u. > They all seem consistent. > > What am I doing wrong? > Is it possible to solve Ax=b with A rectangular? > > Thank you in advance for your help. > Have a nice day. > > Fuji > > Matrix size : m= 33 n= 17 cpu size: 1 > Size of matrix A : 33 17 > Size of vector b : 33 > Size of vector x : 17 > Size of vector u : 17 > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Arguments are incompatible > [0]PETSC ERROR: Incompatible vector local lengths parameter # 1 local size > 33 != parameter # 2 local size 17 > [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting. > [0]PETSC ERROR: Petsc Development GIT revision: v3.17.4-1341-g91b2b62a00 > GIT Date: 2022-09-15 19:26:07 +0000 > [0]PETSC ERROR: ./bin/solve on a x86_64 named master by fujisan Tue Sep 20 > 16:56:37 2022 > [0]PETSC ERROR: Configure options --with-petsc-arch=x86_64 --COPTFLAGS="-g > -O3" --FOPTFLAGS="-g -O3" --CXXOPTFLAGS="-g -O3" --with-debugging=0 > --with-cc=mpiicc --with-cxx=mpiicpc --with-fc=mpiifort > --with-single-library=1 --with-mpiexec=mpiexec --with-precision=double > --with-fortran-interfaces=1 --with-make=1 --with-mpi=1 > --with-mpi-compilers=1 --download-fblaslapack=0 --download-hypre=1 > --download-cmake=0 --with-cmake=1 --download-metis=1 --download-parmetis=1 > --download-ptscotch=0 --download-suitesparse=1 --download-triangle=1 > --download-superlu=1 --download-superlu_dist=1 --download-scalapack=1 > --download-mumps=1 --download-elemental=1 --download-spai=0 > --download-parms=1 --download-moab=1 --download-chaco=0 --download-fftw=1 > --with-petsc4py=1 --download-mpi4py=1 --download-saws > --download-concurrencykit=1 --download-revolve=1 --download-cams=1 > --download-p4est=0 --with-zlib=1 --download-mfem=1 --download-glvis=0 > --with-opengl=0 --download-libpng=1 --download-libjpeg=1 --download-slepc=1 > --download-hpddm=1 --download-bamg=1 --download-mmg=0 --download-parmmg=0 > --download-htool=1 --download-egads=0 --download-opencascade=0 > PETSC_ARCH=x86_64 > [0]PETSC ERROR: #1 VecCopy() at > /data/softs/petsc/src/vec/vec/interface/vector.c:1607 > [0]PETSC ERROR: #2 KSPSolve_BiCG() at > /data/softs/petsc/src/ksp/ksp/impls/bicg/bicg.c:40 > [0]PETSC ERROR: #3 KSPSolve_Private() at > /data/softs/petsc/src/ksp/ksp/interface/itfunc.c:877 > [0]PETSC ERROR: #4 KSPSolve() at > /data/softs/petsc/src/ksp/ksp/interface/itfunc.c:1048 > [0]PETSC ERROR: #5 solve.F90:218 > Abort(75) on node 0 (rank 0 in comm 16): application called > MPI_Abort(MPI_COMM_SELF, 75) - process 0 > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jroman at dsic.upv.es Mon Sep 26 03:25:02 2022 From: jroman at dsic.upv.es (Jose E. Roman) Date: Mon, 26 Sep 2022 10:25:02 +0200 Subject: [petsc-users] Problem solving Ax=b with rectangular matrix A In-Reply-To: References: Message-ID: If the error message is "Could not locate a solver type for factorization type QR" then you should configure PETSc with --download-suitesparse Jose > El 26 sept 2022, a las 9:06, fujisan escribi?: > > Thank you Pierre, > > I used PCNONE along with KSPLSQR and it worked. > But as for PCQR, it cannot be found. There is nothing about it in the documentation as well. > > Fuji > > On Wed, Sep 21, 2022 at 12:20 PM Pierre Jolivet wrote: > Yes, but you need to use a KSP that handles rectangular Mat, such as KSPLSQR (-ksp_type lsqr). > PCLU does not handle rectangular Pmat. The only PC that handle rectangular Pmat are PCQR, PCNONE. > If you supply the normal equations as the Pmat for LSQR, then you can use ?standard? PC. > You can have a look at https://petsc.org/main/src/ksp/ksp/tutorials/ex27.c.html that covers most of these cases. > > Thanks, > Pierre > > (sorry for the earlier answer sent wrongfully to petsc-maint, please discard the previous email) > >> On 21 Sep 2022, at 10:03 AM, fujisan wrote: >> >> I'm trying to solve Ax=b with a sparse rectangular matrix A (of size 33x17 in my test) using >> options '-ksp_type stcg -pc_type lu' on 1 or 2 cpus. >> >> And I always get an error saying "Incompatible vector local lengths" (see below). >> >> Here is the relevant lines of my code: >> >> program test >> ... >> ! Variable declarations >> >> PetscCallA(PetscInitialize(PETSC_NULL_CHARACTER,ierr)) >> >> PetscCall(MatCreate(PETSC_COMM_WORLD,A,ierr)) >> PetscCall(MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,m,n,ierr)) >> PetscCall(MatSetType(A,MATMPIAIJ,ierr)) >> PetscCall(MatSetFromOptions(A,ierr)) >> PetscCall(MatSetUp(A,ierr)) >> PetscCall(MatGetOwnershipRange(A,istart,iend,ierr)) >> >> do irow=istart,iend-1 >> ... Reading from file ... >> PetscCall(MatSetValues(A,1,irow,nzv,col,val,ADD_VALUES,ierr)) >> ... >> enddo >> >> PetscCall(MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY,ierr)) >> PetscCall(MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY,ierr)) >> >> ! Creating vectors x and b >> PetscCallA(MatCreateVecs(A,x,b,ierr)) >> >> ! Duplicating x in u. >> PetscCallA(VecDuplicate(x,u,ierr)) >> >> ! u is used to calculate b >> PetscCallA(VecSet(u,1.0,ierr)) >> >> PetscCallA(VecAssemblyBegin(u,ierr)) >> PetscCallA(VecAssemblyEnd(u,ierr)) >> >> ! Calculating Au = b >> PetscCallA(MatMult(A,u,b,ierr)) ! A.u = b >> >> PetscCallA(KSPSetType(ksp,KSPCG,ierr)) >> >> PetscCallA(KSPSetOperators(ksp,A,A,ierr)) >> >> PetscCallA(KSPSetFromOptions(ksp,ierr)) >> >> ! Solving Ax = b, x unknown >> PetscCallA(KSPSolve(ksp,b,x,ierr)) >> >> PetscCallA(VecDestroy(x,ierr)) >> PetscCallA(VecDestroy(u,ierr)) >> PetscCallA(VecDestroy(b,ierr)) >> PetscCallA(MatDestroy(A,ierr)) >> PetscCallA(KSPDestroy(ksp,ierr)) >> >> call PetscFinalize(ierr) >> end program >> >> The code reads a sparse matrix from a binary file. >> I also output the sizes of matrix A and vectors b, x, u. >> They all seem consistent. >> >> What am I doing wrong? >> Is it possible to solve Ax=b with A rectangular? >> >> Thank you in advance for your help. >> Have a nice day. >> >> Fuji >> >> Matrix size : m= 33 n= 17 cpu size: 1 >> Size of matrix A : 33 17 >> Size of vector b : 33 >> Size of vector x : 17 >> Size of vector u : 17 >> [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >> [0]PETSC ERROR: Arguments are incompatible >> [0]PETSC ERROR: Incompatible vector local lengths parameter # 1 local size 33 != parameter # 2 local size 17 >> [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting. >> [0]PETSC ERROR: Petsc Development GIT revision: v3.17.4-1341-g91b2b62a00 GIT Date: 2022-09-15 19:26:07 +0000 >> [0]PETSC ERROR: ./bin/solve on a x86_64 named master by fujisan Tue Sep 20 16:56:37 2022 >> [0]PETSC ERROR: Configure options --with-petsc-arch=x86_64 --COPTFLAGS="-g -O3" --FOPTFLAGS="-g -O3" --CXXOPTFLAGS="-g -O3" --with-debugging=0 --with-cc=mpiicc --with-cxx=mpiicpc --with-fc=mpiifort --with-single-library=1 --with-mpiexec=mpiexec --with-precision=double --with-fortran-interfaces=1 --with-make=1 --with-mpi=1 --with-mpi-compilers=1 --download-fblaslapack=0 --download-hypre=1 --download-cmake=0 --with-cmake=1 --download-metis=1 --download-parmetis=1 --download-ptscotch=0 --download-suitesparse=1 --download-triangle=1 --download-superlu=1 --download-superlu_dist=1 --download-scalapack=1 --download-mumps=1 --download-elemental=1 --download-spai=0 --download-parms=1 --download-moab=1 --download-chaco=0 --download-fftw=1 --with-petsc4py=1 --download-mpi4py=1 --download-saws --download-concurrencykit=1 --download-revolve=1 --download-cams=1 --download-p4est=0 --with-zlib=1 --download-mfem=1 --download-glvis=0 --with-opengl=0 --download-libpng=1 --download-libjpeg=1 --download-slepc=1 --download-hpddm=1 --download-bamg=1 --download-mmg=0 --download-parmmg=0 --download-htool=1 --download-egads=0 --download-opencascade=0 PETSC_ARCH=x86_64 >> [0]PETSC ERROR: #1 VecCopy() at /data/softs/petsc/src/vec/vec/interface/vector.c:1607 >> [0]PETSC ERROR: #2 KSPSolve_BiCG() at /data/softs/petsc/src/ksp/ksp/impls/bicg/bicg.c:40 >> [0]PETSC ERROR: #3 KSPSolve_Private() at /data/softs/petsc/src/ksp/ksp/interface/itfunc.c:877 >> [0]PETSC ERROR: #4 KSPSolve() at /data/softs/petsc/src/ksp/ksp/interface/itfunc.c:1048 >> [0]PETSC ERROR: #5 solve.F90:218 >> Abort(75) on node 0 (rank 0 in comm 16): application called MPI_Abort(MPI_COMM_SELF, 75) - process 0 >> > From fujisan43 at gmail.com Mon Sep 26 03:47:05 2022 From: fujisan43 at gmail.com (fujisan) Date: Mon, 26 Sep 2022 10:47:05 +0200 Subject: [petsc-users] Problem solving Ax=b with rectangular matrix A In-Reply-To: References: Message-ID: I did configure Petsc with the option --download-suitesparse. The error is more like this: PETSC ERROR: Could not locate a solver type for factorization type QR and matrix type mpiaij. Fuji On Mon, Sep 26, 2022 at 10:25 AM Jose E. Roman wrote: > If the error message is "Could not locate a solver type for factorization > type QR" then you should configure PETSc with --download-suitesparse > > Jose > > > > El 26 sept 2022, a las 9:06, fujisan escribi?: > > > > Thank you Pierre, > > > > I used PCNONE along with KSPLSQR and it worked. > > But as for PCQR, it cannot be found. There is nothing about it in the > documentation as well. > > > > Fuji > > > > On Wed, Sep 21, 2022 at 12:20 PM Pierre Jolivet wrote: > > Yes, but you need to use a KSP that handles rectangular Mat, such as > KSPLSQR (-ksp_type lsqr). > > PCLU does not handle rectangular Pmat. The only PC that handle > rectangular Pmat are PCQR, PCNONE. > > If you supply the normal equations as the Pmat for LSQR, then you can > use ?standard? PC. > > You can have a look at > https://petsc.org/main/src/ksp/ksp/tutorials/ex27.c.html that covers most > of these cases. > > > > Thanks, > > Pierre > > > > (sorry for the earlier answer sent wrongfully to petsc-maint, please > discard the previous email) > > > >> On 21 Sep 2022, at 10:03 AM, fujisan wrote: > >> > >> I'm trying to solve Ax=b with a sparse rectangular matrix A (of size > 33x17 in my test) using > >> options '-ksp_type stcg -pc_type lu' on 1 or 2 cpus. > >> > >> And I always get an error saying "Incompatible vector local lengths" > (see below). > >> > >> Here is the relevant lines of my code: > >> > >> program test > >> ... > >> ! Variable declarations > >> > >> PetscCallA(PetscInitialize(PETSC_NULL_CHARACTER,ierr)) > >> > >> PetscCall(MatCreate(PETSC_COMM_WORLD,A,ierr)) > >> PetscCall(MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,m,n,ierr)) > >> PetscCall(MatSetType(A,MATMPIAIJ,ierr)) > >> PetscCall(MatSetFromOptions(A,ierr)) > >> PetscCall(MatSetUp(A,ierr)) > >> PetscCall(MatGetOwnershipRange(A,istart,iend,ierr)) > >> > >> do irow=istart,iend-1 > >> ... Reading from file ... > >> PetscCall(MatSetValues(A,1,irow,nzv,col,val,ADD_VALUES,ierr)) > >> ... > >> enddo > >> > >> PetscCall(MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY,ierr)) > >> PetscCall(MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY,ierr)) > >> > >> ! Creating vectors x and b > >> PetscCallA(MatCreateVecs(A,x,b,ierr)) > >> > >> ! Duplicating x in u. > >> PetscCallA(VecDuplicate(x,u,ierr)) > >> > >> ! u is used to calculate b > >> PetscCallA(VecSet(u,1.0,ierr)) > >> > >> PetscCallA(VecAssemblyBegin(u,ierr)) > >> PetscCallA(VecAssemblyEnd(u,ierr)) > >> > >> ! Calculating Au = b > >> PetscCallA(MatMult(A,u,b,ierr)) ! A.u = b > >> > >> PetscCallA(KSPSetType(ksp,KSPCG,ierr)) > >> > >> PetscCallA(KSPSetOperators(ksp,A,A,ierr)) > >> > >> PetscCallA(KSPSetFromOptions(ksp,ierr)) > >> > >> ! Solving Ax = b, x unknown > >> PetscCallA(KSPSolve(ksp,b,x,ierr)) > >> > >> PetscCallA(VecDestroy(x,ierr)) > >> PetscCallA(VecDestroy(u,ierr)) > >> PetscCallA(VecDestroy(b,ierr)) > >> PetscCallA(MatDestroy(A,ierr)) > >> PetscCallA(KSPDestroy(ksp,ierr)) > >> > >> call PetscFinalize(ierr) > >> end program > >> > >> The code reads a sparse matrix from a binary file. > >> I also output the sizes of matrix A and vectors b, x, u. > >> They all seem consistent. > >> > >> What am I doing wrong? > >> Is it possible to solve Ax=b with A rectangular? > >> > >> Thank you in advance for your help. > >> Have a nice day. > >> > >> Fuji > >> > >> Matrix size : m= 33 n= 17 cpu size: 1 > >> Size of matrix A : 33 17 > >> Size of vector b : 33 > >> Size of vector x : 17 > >> Size of vector u : 17 > >> [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > >> [0]PETSC ERROR: Arguments are incompatible > >> [0]PETSC ERROR: Incompatible vector local lengths parameter # 1 local > size 33 != parameter # 2 local size 17 > >> [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble > shooting. > >> [0]PETSC ERROR: Petsc Development GIT revision: > v3.17.4-1341-g91b2b62a00 GIT Date: 2022-09-15 19:26:07 +0000 > >> [0]PETSC ERROR: ./bin/solve on a x86_64 named master by fujisan Tue Sep > 20 16:56:37 2022 > >> [0]PETSC ERROR: Configure options --with-petsc-arch=x86_64 > --COPTFLAGS="-g -O3" --FOPTFLAGS="-g -O3" --CXXOPTFLAGS="-g -O3" > --with-debugging=0 --with-cc=mpiicc --with-cxx=mpiicpc --with-fc=mpiifort > --with-single-library=1 --with-mpiexec=mpiexec --with-precision=double > --with-fortran-interfaces=1 --with-make=1 --with-mpi=1 > --with-mpi-compilers=1 --download-fblaslapack=0 --download-hypre=1 > --download-cmake=0 --with-cmake=1 --download-metis=1 --download-parmetis=1 > --download-ptscotch=0 --download-suitesparse=1 --download-triangle=1 > --download-superlu=1 --download-superlu_dist=1 --download-scalapack=1 > --download-mumps=1 --download-elemental=1 --download-spai=0 > --download-parms=1 --download-moab=1 --download-chaco=0 --download-fftw=1 > --with-petsc4py=1 --download-mpi4py=1 --download-saws > --download-concurrencykit=1 --download-revolve=1 --download-cams=1 > --download-p4est=0 --with-zlib=1 --download-mfem=1 --download-glvis=0 > --with-opengl=0 --download-libpng=1 --download-libjpeg=1 --download-slepc=1 > --download-hpddm=1 --download-bamg=1 --download-mmg=0 --download-parmmg=0 > --download-htool=1 --download-egads=0 --download-opencascade=0 > PETSC_ARCH=x86_64 > >> [0]PETSC ERROR: #1 VecCopy() at > /data/softs/petsc/src/vec/vec/interface/vector.c:1607 > >> [0]PETSC ERROR: #2 KSPSolve_BiCG() at > /data/softs/petsc/src/ksp/ksp/impls/bicg/bicg.c:40 > >> [0]PETSC ERROR: #3 KSPSolve_Private() at > /data/softs/petsc/src/ksp/ksp/interface/itfunc.c:877 > >> [0]PETSC ERROR: #4 KSPSolve() at > /data/softs/petsc/src/ksp/ksp/interface/itfunc.c:1048 > >> [0]PETSC ERROR: #5 solve.F90:218 > >> Abort(75) on node 0 (rank 0 in comm 16): application called > MPI_Abort(MPI_COMM_SELF, 75) - process 0 > >> > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jroman at dsic.upv.es Mon Sep 26 03:53:20 2022 From: jroman at dsic.upv.es (Jose E. Roman) Date: Mon, 26 Sep 2022 10:53:20 +0200 Subject: [petsc-users] Problem solving Ax=b with rectangular matrix A In-Reply-To: References: Message-ID: <3023EA5A-19AB-4989-8E0D-533FD531783D@dsic.upv.es> The QR factorization from SuiteSparse is sequential only, cannot be used in parallel. In parallel you can try PCBJACOBI with a PCQR local preconditioner. Pierre may have additional suggestions. Jose > El 26 sept 2022, a las 10:47, fujisan escribi?: > > I did configure Petsc with the option --download-suitesparse. > > The error is more like this: > PETSC ERROR: Could not locate a solver type for factorization type QR and matrix type mpiaij. > > Fuji > > On Mon, Sep 26, 2022 at 10:25 AM Jose E. Roman wrote: > If the error message is "Could not locate a solver type for factorization type QR" then you should configure PETSc with --download-suitesparse > > Jose > > > > El 26 sept 2022, a las 9:06, fujisan escribi?: > > > > Thank you Pierre, > > > > I used PCNONE along with KSPLSQR and it worked. > > But as for PCQR, it cannot be found. There is nothing about it in the documentation as well. > > > > Fuji > > > > On Wed, Sep 21, 2022 at 12:20 PM Pierre Jolivet wrote: > > Yes, but you need to use a KSP that handles rectangular Mat, such as KSPLSQR (-ksp_type lsqr). > > PCLU does not handle rectangular Pmat. The only PC that handle rectangular Pmat are PCQR, PCNONE. > > If you supply the normal equations as the Pmat for LSQR, then you can use ?standard? PC. > > You can have a look at https://petsc.org/main/src/ksp/ksp/tutorials/ex27.c.html that covers most of these cases. > > > > Thanks, > > Pierre > > > > (sorry for the earlier answer sent wrongfully to petsc-maint, please discard the previous email) > > > >> On 21 Sep 2022, at 10:03 AM, fujisan wrote: > >> > >> I'm trying to solve Ax=b with a sparse rectangular matrix A (of size 33x17 in my test) using > >> options '-ksp_type stcg -pc_type lu' on 1 or 2 cpus. > >> > >> And I always get an error saying "Incompatible vector local lengths" (see below). > >> > >> Here is the relevant lines of my code: > >> > >> program test > >> ... > >> ! Variable declarations > >> > >> PetscCallA(PetscInitialize(PETSC_NULL_CHARACTER,ierr)) > >> > >> PetscCall(MatCreate(PETSC_COMM_WORLD,A,ierr)) > >> PetscCall(MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,m,n,ierr)) > >> PetscCall(MatSetType(A,MATMPIAIJ,ierr)) > >> PetscCall(MatSetFromOptions(A,ierr)) > >> PetscCall(MatSetUp(A,ierr)) > >> PetscCall(MatGetOwnershipRange(A,istart,iend,ierr)) > >> > >> do irow=istart,iend-1 > >> ... Reading from file ... > >> PetscCall(MatSetValues(A,1,irow,nzv,col,val,ADD_VALUES,ierr)) > >> ... > >> enddo > >> > >> PetscCall(MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY,ierr)) > >> PetscCall(MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY,ierr)) > >> > >> ! Creating vectors x and b > >> PetscCallA(MatCreateVecs(A,x,b,ierr)) > >> > >> ! Duplicating x in u. > >> PetscCallA(VecDuplicate(x,u,ierr)) > >> > >> ! u is used to calculate b > >> PetscCallA(VecSet(u,1.0,ierr)) > >> > >> PetscCallA(VecAssemblyBegin(u,ierr)) > >> PetscCallA(VecAssemblyEnd(u,ierr)) > >> > >> ! Calculating Au = b > >> PetscCallA(MatMult(A,u,b,ierr)) ! A.u = b > >> > >> PetscCallA(KSPSetType(ksp,KSPCG,ierr)) > >> > >> PetscCallA(KSPSetOperators(ksp,A,A,ierr)) > >> > >> PetscCallA(KSPSetFromOptions(ksp,ierr)) > >> > >> ! Solving Ax = b, x unknown > >> PetscCallA(KSPSolve(ksp,b,x,ierr)) > >> > >> PetscCallA(VecDestroy(x,ierr)) > >> PetscCallA(VecDestroy(u,ierr)) > >> PetscCallA(VecDestroy(b,ierr)) > >> PetscCallA(MatDestroy(A,ierr)) > >> PetscCallA(KSPDestroy(ksp,ierr)) > >> > >> call PetscFinalize(ierr) > >> end program > >> > >> The code reads a sparse matrix from a binary file. > >> I also output the sizes of matrix A and vectors b, x, u. > >> They all seem consistent. > >> > >> What am I doing wrong? > >> Is it possible to solve Ax=b with A rectangular? > >> > >> Thank you in advance for your help. > >> Have a nice day. > >> > >> Fuji > >> > >> Matrix size : m= 33 n= 17 cpu size: 1 > >> Size of matrix A : 33 17 > >> Size of vector b : 33 > >> Size of vector x : 17 > >> Size of vector u : 17 > >> [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > >> [0]PETSC ERROR: Arguments are incompatible > >> [0]PETSC ERROR: Incompatible vector local lengths parameter # 1 local size 33 != parameter # 2 local size 17 > >> [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting. > >> [0]PETSC ERROR: Petsc Development GIT revision: v3.17.4-1341-g91b2b62a00 GIT Date: 2022-09-15 19:26:07 +0000 > >> [0]PETSC ERROR: ./bin/solve on a x86_64 named master by fujisan Tue Sep 20 16:56:37 2022 > >> [0]PETSC ERROR: Configure options --with-petsc-arch=x86_64 --COPTFLAGS="-g -O3" --FOPTFLAGS="-g -O3" --CXXOPTFLAGS="-g -O3" --with-debugging=0 --with-cc=mpiicc --with-cxx=mpiicpc --with-fc=mpiifort --with-single-library=1 --with-mpiexec=mpiexec --with-precision=double --with-fortran-interfaces=1 --with-make=1 --with-mpi=1 --with-mpi-compilers=1 --download-fblaslapack=0 --download-hypre=1 --download-cmake=0 --with-cmake=1 --download-metis=1 --download-parmetis=1 --download-ptscotch=0 --download-suitesparse=1 --download-triangle=1 --download-superlu=1 --download-superlu_dist=1 --download-scalapack=1 --download-mumps=1 --download-elemental=1 --download-spai=0 --download-parms=1 --download-moab=1 --download-chaco=0 --download-fftw=1 --with-petsc4py=1 --download-mpi4py=1 --download-saws --download-concurrencykit=1 --download-revolve=1 --download-cams=1 --download-p4est=0 --with-zlib=1 --download-mfem=1 --download-glvis=0 --with-opengl=0 --download-libpng=1 --download-libjpeg=1 --download-slepc=1 --download-hpddm=1 --download-bamg=1 --download-mmg=0 --download-parmmg=0 --download-htool=1 --download-egads=0 --download-opencascade=0 PETSC_ARCH=x86_64 > >> [0]PETSC ERROR: #1 VecCopy() at /data/softs/petsc/src/vec/vec/interface/vector.c:1607 > >> [0]PETSC ERROR: #2 KSPSolve_BiCG() at /data/softs/petsc/src/ksp/ksp/impls/bicg/bicg.c:40 > >> [0]PETSC ERROR: #3 KSPSolve_Private() at /data/softs/petsc/src/ksp/ksp/interface/itfunc.c:877 > >> [0]PETSC ERROR: #4 KSPSolve() at /data/softs/petsc/src/ksp/ksp/interface/itfunc.c:1048 > >> [0]PETSC ERROR: #5 solve.F90:218 > >> Abort(75) on node 0 (rank 0 in comm 16): application called MPI_Abort(MPI_COMM_SELF, 75) - process 0 > >> > > > From fujisan43 at gmail.com Mon Sep 26 03:56:20 2022 From: fujisan43 at gmail.com (fujisan) Date: Mon, 26 Sep 2022 10:56:20 +0200 Subject: [petsc-users] Problem solving Ax=b with rectangular matrix A In-Reply-To: <3023EA5A-19AB-4989-8E0D-533FD531783D@dsic.upv.es> References: <3023EA5A-19AB-4989-8E0D-533FD531783D@dsic.upv.es> Message-ID: OK thank you. On Mon, Sep 26, 2022 at 10:53 AM Jose E. Roman wrote: > The QR factorization from SuiteSparse is sequential only, cannot be used > in parallel. > In parallel you can try PCBJACOBI with a PCQR local preconditioner. > Pierre may have additional suggestions. > > Jose > > > > El 26 sept 2022, a las 10:47, fujisan escribi?: > > > > I did configure Petsc with the option --download-suitesparse. > > > > The error is more like this: > > PETSC ERROR: Could not locate a solver type for factorization type QR > and matrix type mpiaij. > > > > Fuji > > > > On Mon, Sep 26, 2022 at 10:25 AM Jose E. Roman > wrote: > > If the error message is "Could not locate a solver type for > factorization type QR" then you should configure PETSc with > --download-suitesparse > > > > Jose > > > > > > > El 26 sept 2022, a las 9:06, fujisan escribi?: > > > > > > Thank you Pierre, > > > > > > I used PCNONE along with KSPLSQR and it worked. > > > But as for PCQR, it cannot be found. There is nothing about it in the > documentation as well. > > > > > > Fuji > > > > > > On Wed, Sep 21, 2022 at 12:20 PM Pierre Jolivet > wrote: > > > Yes, but you need to use a KSP that handles rectangular Mat, such as > KSPLSQR (-ksp_type lsqr). > > > PCLU does not handle rectangular Pmat. The only PC that handle > rectangular Pmat are PCQR, PCNONE. > > > If you supply the normal equations as the Pmat for LSQR, then you can > use ?standard? PC. > > > You can have a look at > https://petsc.org/main/src/ksp/ksp/tutorials/ex27.c.html that covers most > of these cases. > > > > > > Thanks, > > > Pierre > > > > > > (sorry for the earlier answer sent wrongfully to petsc-maint, please > discard the previous email) > > > > > >> On 21 Sep 2022, at 10:03 AM, fujisan wrote: > > >> > > >> I'm trying to solve Ax=b with a sparse rectangular matrix A (of size > 33x17 in my test) using > > >> options '-ksp_type stcg -pc_type lu' on 1 or 2 cpus. > > >> > > >> And I always get an error saying "Incompatible vector local lengths" > (see below). > > >> > > >> Here is the relevant lines of my code: > > >> > > >> program test > > >> ... > > >> ! Variable declarations > > >> > > >> PetscCallA(PetscInitialize(PETSC_NULL_CHARACTER,ierr)) > > >> > > >> PetscCall(MatCreate(PETSC_COMM_WORLD,A,ierr)) > > >> PetscCall(MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,m,n,ierr)) > > >> PetscCall(MatSetType(A,MATMPIAIJ,ierr)) > > >> PetscCall(MatSetFromOptions(A,ierr)) > > >> PetscCall(MatSetUp(A,ierr)) > > >> PetscCall(MatGetOwnershipRange(A,istart,iend,ierr)) > > >> > > >> do irow=istart,iend-1 > > >> ... Reading from file ... > > >> PetscCall(MatSetValues(A,1,irow,nzv,col,val,ADD_VALUES,ierr)) > > >> ... > > >> enddo > > >> > > >> PetscCall(MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY,ierr)) > > >> PetscCall(MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY,ierr)) > > >> > > >> ! Creating vectors x and b > > >> PetscCallA(MatCreateVecs(A,x,b,ierr)) > > >> > > >> ! Duplicating x in u. > > >> PetscCallA(VecDuplicate(x,u,ierr)) > > >> > > >> ! u is used to calculate b > > >> PetscCallA(VecSet(u,1.0,ierr)) > > >> > > >> PetscCallA(VecAssemblyBegin(u,ierr)) > > >> PetscCallA(VecAssemblyEnd(u,ierr)) > > >> > > >> ! Calculating Au = b > > >> PetscCallA(MatMult(A,u,b,ierr)) ! A.u = b > > >> > > >> PetscCallA(KSPSetType(ksp,KSPCG,ierr)) > > >> > > >> PetscCallA(KSPSetOperators(ksp,A,A,ierr)) > > >> > > >> PetscCallA(KSPSetFromOptions(ksp,ierr)) > > >> > > >> ! Solving Ax = b, x unknown > > >> PetscCallA(KSPSolve(ksp,b,x,ierr)) > > >> > > >> PetscCallA(VecDestroy(x,ierr)) > > >> PetscCallA(VecDestroy(u,ierr)) > > >> PetscCallA(VecDestroy(b,ierr)) > > >> PetscCallA(MatDestroy(A,ierr)) > > >> PetscCallA(KSPDestroy(ksp,ierr)) > > >> > > >> call PetscFinalize(ierr) > > >> end program > > >> > > >> The code reads a sparse matrix from a binary file. > > >> I also output the sizes of matrix A and vectors b, x, u. > > >> They all seem consistent. > > >> > > >> What am I doing wrong? > > >> Is it possible to solve Ax=b with A rectangular? > > >> > > >> Thank you in advance for your help. > > >> Have a nice day. > > >> > > >> Fuji > > >> > > >> Matrix size : m= 33 n= 17 cpu size: 1 > > >> Size of matrix A : 33 17 > > >> Size of vector b : 33 > > >> Size of vector x : 17 > > >> Size of vector u : 17 > > >> [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > > >> [0]PETSC ERROR: Arguments are incompatible > > >> [0]PETSC ERROR: Incompatible vector local lengths parameter # 1 local > size 33 != parameter # 2 local size 17 > > >> [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble > shooting. > > >> [0]PETSC ERROR: Petsc Development GIT revision: > v3.17.4-1341-g91b2b62a00 GIT Date: 2022-09-15 19:26:07 +0000 > > >> [0]PETSC ERROR: ./bin/solve on a x86_64 named master by fujisan Tue > Sep 20 16:56:37 2022 > > >> [0]PETSC ERROR: Configure options --with-petsc-arch=x86_64 > --COPTFLAGS="-g -O3" --FOPTFLAGS="-g -O3" --CXXOPTFLAGS="-g -O3" > --with-debugging=0 --with-cc=mpiicc --with-cxx=mpiicpc --with-fc=mpiifort > --with-single-library=1 --with-mpiexec=mpiexec --with-precision=double > --with-fortran-interfaces=1 --with-make=1 --with-mpi=1 > --with-mpi-compilers=1 --download-fblaslapack=0 --download-hypre=1 > --download-cmake=0 --with-cmake=1 --download-metis=1 --download-parmetis=1 > --download-ptscotch=0 --download-suitesparse=1 --download-triangle=1 > --download-superlu=1 --download-superlu_dist=1 --download-scalapack=1 > --download-mumps=1 --download-elemental=1 --download-spai=0 > --download-parms=1 --download-moab=1 --download-chaco=0 --download-fftw=1 > --with-petsc4py=1 --download-mpi4py=1 --download-saws > --download-concurrencykit=1 --download-revolve=1 --download-cams=1 > --download-p4est=0 --with-zlib=1 --download-mfem=1 --download-glvis=0 > --with-opengl=0 --download-libpng=1 --download-libjpeg=1 --download-slepc=1 > --download-hpddm=1 --download-bamg=1 --download-mmg=0 --download-parmmg=0 > --download-htool=1 --download-egads=0 --download-opencascade=0 > PETSC_ARCH=x86_64 > > >> [0]PETSC ERROR: #1 VecCopy() at > /data/softs/petsc/src/vec/vec/interface/vector.c:1607 > > >> [0]PETSC ERROR: #2 KSPSolve_BiCG() at > /data/softs/petsc/src/ksp/ksp/impls/bicg/bicg.c:40 > > >> [0]PETSC ERROR: #3 KSPSolve_Private() at > /data/softs/petsc/src/ksp/ksp/interface/itfunc.c:877 > > >> [0]PETSC ERROR: #4 KSPSolve() at > /data/softs/petsc/src/ksp/ksp/interface/itfunc.c:1048 > > >> [0]PETSC ERROR: #5 solve.F90:218 > > >> Abort(75) on node 0 (rank 0 in comm 16): application called > MPI_Abort(MPI_COMM_SELF, 75) - process 0 > > >> > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tlanyan at hotmail.com Mon Sep 26 04:20:03 2022 From: tlanyan at hotmail.com (=?utf-8?B?5pmT5bOwIOS9lQ==?=) Date: Mon, 26 Sep 2022 09:20:03 +0000 Subject: [petsc-users] Solve Linear System with Field Split Preconditioner In-Reply-To: <87pmfihddt.fsf@jedbrown.org> References: <819A69F5-6CB0-494E-80B2-18EB023FD048@hotmail.com> <87pmfihddt.fsf@jedbrown.org> Message-ID: <22A0172A-4AAA-4247-BA35-266555EDA4CF@hotmail.com> Are there other approaches to solve this kind of systems in PETSc except for field-split methods? Thanks, Xiaofeng On Sep 26, 2022, at 14:13, Jed Brown > wrote: This is the joy of factorization field-split methods. The actual Schur complement is dense, so we represent it implicitly. A common strategy is to assemble the mass matrix and drop it in the 11 block of the Pmat. You can check out some examples in the repository for incompressible flow (Stokes problems). The LSC (least squares commutator) is another option. You'll likely find that lumping diag(A00)^{-1} works poorly because the resulting operator behaves like a Laplacian rather than like a mass matrix. ?? ? > writes: If assigned a preconditioner to A11 with this cmd options: -fieldsplit_0_ksp_type gmres -fieldsplit_0_pc_type ilu -fieldsplit_1_ksp_type gmres -fieldsplit_1_pc_type ilu Then I got this error: "Could not locate a solver type for factorization type ILU and matrix type schurcomplement" How could I specify a preconditioner for A11? BR, Xiaofeng On Sep 26, 2022, at 11:02, ?? ? > wrote: -fieldsplit_0_ksp_type gmres -fieldsplit_0_pc_type ilu -fieldsplit_1_ksp_type gmres -fieldsplit_1_pc_type none -------------- next part -------------- An HTML attachment was scrubbed... URL: From pierre at joliv.et Mon Sep 26 04:45:02 2022 From: pierre at joliv.et (Pierre Jolivet) Date: Mon, 26 Sep 2022 11:45:02 +0200 Subject: [petsc-users] Problem solving Ax=b with rectangular matrix A In-Reply-To: References: <3023EA5A-19AB-4989-8E0D-533FD531783D@dsic.upv.es> Message-ID: <2C060B48-EB03-47F6-9E84-4A0A8E901770@joliv.et> I?m sorry, solving overdetermined systems, alongside (overlapping) domain decomposition preconditioners and solving systems with multiple right-hand sides, is one of the topic for which I need to stop pushing new features and update the users manual instead? The very basic documentation of PCQR is here: https://petsc.org/main/docs/manualpages/PC/PCQR (I?m guessing you are using the release documentation in which it?s indeed not present). Some comments about your problem of solving Ax=b with a rectangular matrix A. 1) if you switch to KSPLSQR, it is wrong to use KSPSetOperators(ksp, A, A). You can get away with murder if your PCType is PCNONE, but otherwise, you should always supply the normal equations as the Pmat (you will get runtime errors otherwise). To get the normal equations, you can use https://petsc.org/main/docs/manualpages/Mat/MatCreateNormal/ The following two points only applies if your Pmat is sparse (or sparse with some dense rows). 2) there are a couple of PC that handle MATNORMAL: PCNONE, PCQR, PCJACOBI, PCBJACOBI, PCASM, PCHPDDM Currently, PCQR needs SuiteSparse, and thus runs only if the Pmat is distributed on a single MPI process (if your Pmat is distributed on multiple processes, you should first use PCREDUNDANT). 3) if you intend to make your problem scale in sizes, most of these preconditioners will not be very robust. In that case, if your problem does not have any dense rows, you should either use PCHPDDM or MatConvert(Pmat, MATAIJ, PmatAIJ) and then use PCCHOLESKY, PCHYPRE or PCGAMG (you can have a look at https://epubs.siam.org/doi/abs/10.1137/21M1434891 for a comparison) If your problem has dense rows, I have somewhere the code to recast it into an augmented system then solved using PCFIELDSPLIT (see https://link.springer.com/article/10.1007/s11075-018-0478-2 ). I can send it to you if needed. Let me know if I can be of further assistance or if something is not clear to you. Thanks, Pierre > On 26 Sep 2022, at 10:56 AM, fujisan wrote: > > OK thank you. > > On Mon, Sep 26, 2022 at 10:53 AM Jose E. Roman > wrote: > The QR factorization from SuiteSparse is sequential only, cannot be used in parallel. > In parallel you can try PCBJACOBI with a PCQR local preconditioner. > Pierre may have additional suggestions. > > Jose > > > > El 26 sept 2022, a las 10:47, fujisan > escribi?: > > > > I did configure Petsc with the option --download-suitesparse. > > > > The error is more like this: > > PETSC ERROR: Could not locate a solver type for factorization type QR and matrix type mpiaij. > > > > Fuji > > > > On Mon, Sep 26, 2022 at 10:25 AM Jose E. Roman > wrote: > > If the error message is "Could not locate a solver type for factorization type QR" then you should configure PETSc with --download-suitesparse > > > > Jose > > > > > > > El 26 sept 2022, a las 9:06, fujisan > escribi?: > > > > > > Thank you Pierre, > > > > > > I used PCNONE along with KSPLSQR and it worked. > > > But as for PCQR, it cannot be found. There is nothing about it in the documentation as well. > > > > > > Fuji > > > > > > On Wed, Sep 21, 2022 at 12:20 PM Pierre Jolivet > wrote: > > > Yes, but you need to use a KSP that handles rectangular Mat, such as KSPLSQR (-ksp_type lsqr). > > > PCLU does not handle rectangular Pmat. The only PC that handle rectangular Pmat are PCQR, PCNONE. > > > If you supply the normal equations as the Pmat for LSQR, then you can use ?standard? PC. > > > You can have a look at https://petsc.org/main/src/ksp/ksp/tutorials/ex27.c.html that covers most of these cases. > > > > > > Thanks, > > > Pierre > > > > > > (sorry for the earlier answer sent wrongfully to petsc-maint, please discard the previous email) > > > > > >> On 21 Sep 2022, at 10:03 AM, fujisan > wrote: > > >> > > >> I'm trying to solve Ax=b with a sparse rectangular matrix A (of size 33x17 in my test) using > > >> options '-ksp_type stcg -pc_type lu' on 1 or 2 cpus. > > >> > > >> And I always get an error saying "Incompatible vector local lengths" (see below). > > >> > > >> Here is the relevant lines of my code: > > >> > > >> program test > > >> ... > > >> ! Variable declarations > > >> > > >> PetscCallA(PetscInitialize(PETSC_NULL_CHARACTER,ierr)) > > >> > > >> PetscCall(MatCreate(PETSC_COMM_WORLD,A,ierr)) > > >> PetscCall(MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,m,n,ierr)) > > >> PetscCall(MatSetType(A,MATMPIAIJ,ierr)) > > >> PetscCall(MatSetFromOptions(A,ierr)) > > >> PetscCall(MatSetUp(A,ierr)) > > >> PetscCall(MatGetOwnershipRange(A,istart,iend,ierr)) > > >> > > >> do irow=istart,iend-1 > > >> ... Reading from file ... > > >> PetscCall(MatSetValues(A,1,irow,nzv,col,val,ADD_VALUES,ierr)) > > >> ... > > >> enddo > > >> > > >> PetscCall(MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY,ierr)) > > >> PetscCall(MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY,ierr)) > > >> > > >> ! Creating vectors x and b > > >> PetscCallA(MatCreateVecs(A,x,b,ierr)) > > >> > > >> ! Duplicating x in u. > > >> PetscCallA(VecDuplicate(x,u,ierr)) > > >> > > >> ! u is used to calculate b > > >> PetscCallA(VecSet(u,1.0,ierr)) > > >> > > >> PetscCallA(VecAssemblyBegin(u,ierr)) > > >> PetscCallA(VecAssemblyEnd(u,ierr)) > > >> > > >> ! Calculating Au = b > > >> PetscCallA(MatMult(A,u,b,ierr)) ! A.u = b > > >> > > >> PetscCallA(KSPSetType(ksp,KSPCG,ierr)) > > >> > > >> PetscCallA(KSPSetOperators(ksp,A,A,ierr)) > > >> > > >> PetscCallA(KSPSetFromOptions(ksp,ierr)) > > >> > > >> ! Solving Ax = b, x unknown > > >> PetscCallA(KSPSolve(ksp,b,x,ierr)) > > >> > > >> PetscCallA(VecDestroy(x,ierr)) > > >> PetscCallA(VecDestroy(u,ierr)) > > >> PetscCallA(VecDestroy(b,ierr)) > > >> PetscCallA(MatDestroy(A,ierr)) > > >> PetscCallA(KSPDestroy(ksp,ierr)) > > >> > > >> call PetscFinalize(ierr) > > >> end program > > >> > > >> The code reads a sparse matrix from a binary file. > > >> I also output the sizes of matrix A and vectors b, x, u. > > >> They all seem consistent. > > >> > > >> What am I doing wrong? > > >> Is it possible to solve Ax=b with A rectangular? > > >> > > >> Thank you in advance for your help. > > >> Have a nice day. > > >> > > >> Fuji > > >> > > >> Matrix size : m= 33 n= 17 cpu size: 1 > > >> Size of matrix A : 33 17 > > >> Size of vector b : 33 > > >> Size of vector x : 17 > > >> Size of vector u : 17 > > >> [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > > >> [0]PETSC ERROR: Arguments are incompatible > > >> [0]PETSC ERROR: Incompatible vector local lengths parameter # 1 local size 33 != parameter # 2 local size 17 > > >> [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting. > > >> [0]PETSC ERROR: Petsc Development GIT revision: v3.17.4-1341-g91b2b62a00 GIT Date: 2022-09-15 19:26:07 +0000 > > >> [0]PETSC ERROR: ./bin/solve on a x86_64 named master by fujisan Tue Sep 20 16:56:37 2022 > > >> [0]PETSC ERROR: Configure options --with-petsc-arch=x86_64 --COPTFLAGS="-g -O3" --FOPTFLAGS="-g -O3" --CXXOPTFLAGS="-g -O3" --with-debugging=0 --with-cc=mpiicc --with-cxx=mpiicpc --with-fc=mpiifort --with-single-library=1 --with-mpiexec=mpiexec --with-precision=double --with-fortran-interfaces=1 --with-make=1 --with-mpi=1 --with-mpi-compilers=1 --download-fblaslapack=0 --download-hypre=1 --download-cmake=0 --with-cmake=1 --download-metis=1 --download-parmetis=1 --download-ptscotch=0 --download-suitesparse=1 --download-triangle=1 --download-superlu=1 --download-superlu_dist=1 --download-scalapack=1 --download-mumps=1 --download-elemental=1 --download-spai=0 --download-parms=1 --download-moab=1 --download-chaco=0 --download-fftw=1 --with-petsc4py=1 --download-mpi4py=1 --download-saws --download-concurrencykit=1 --download-revolve=1 --download-cams=1 --download-p4est=0 --with-zlib=1 --download-mfem=1 --download-glvis=0 --with-opengl=0 --download-libpng=1 --download-libjpeg=1 --download-slepc=1 --download-hpddm=1 --download-bamg=1 --download-mmg=0 --download-parmmg=0 --download-htool=1 --download-egads=0 --download-opencascade=0 PETSC_ARCH=x86_64 > > >> [0]PETSC ERROR: #1 VecCopy() at /data/softs/petsc/src/vec/vec/interface/vector.c:1607 > > >> [0]PETSC ERROR: #2 KSPSolve_BiCG() at /data/softs/petsc/src/ksp/ksp/impls/bicg/bicg.c:40 > > >> [0]PETSC ERROR: #3 KSPSolve_Private() at /data/softs/petsc/src/ksp/ksp/interface/itfunc.c:877 > > >> [0]PETSC ERROR: #4 KSPSolve() at /data/softs/petsc/src/ksp/ksp/interface/itfunc.c:1048 > > >> [0]PETSC ERROR: #5 solve.F90:218 > > >> Abort(75) on node 0 (rank 0 in comm 16): application called MPI_Abort(MPI_COMM_SELF, 75) - process 0 > > >> > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Sep 26 07:29:33 2022 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 26 Sep 2022 08:29:33 -0400 Subject: [petsc-users] Solve Linear System with Field Split Preconditioner In-Reply-To: <22A0172A-4AAA-4247-BA35-266555EDA4CF@hotmail.com> References: <819A69F5-6CB0-494E-80B2-18EB023FD048@hotmail.com> <87pmfihddt.fsf@jedbrown.org> <22A0172A-4AAA-4247-BA35-266555EDA4CF@hotmail.com> Message-ID: Another option are the PCPATCH solvers for multigrid, as shown in this paper: https://arxiv.org/abs/1912.08516 which I believe solves incompressible elasticity. There is an example in PETSc for Stokes I believe. Thanks, Matt On Mon, Sep 26, 2022 at 5:20 AM ?? ? wrote: > Are there other approaches to solve this kind of systems in PETSc except > for field-split methods? > > Thanks, > Xiaofeng > > On Sep 26, 2022, at 14:13, Jed Brown wrote: > > This is the joy of factorization field-split methods. The actual Schur > complement is dense, so we represent it implicitly. A common strategy is to > assemble the mass matrix and drop it in the 11 block of the Pmat. You can > check out some examples in the repository for incompressible flow (Stokes > problems). The LSC (least squares commutator) is another option. You'll > likely find that lumping diag(A00)^{-1} works poorly because the resulting > operator behaves like a Laplacian rather than like a mass matrix. > > ?? ? writes: > > If assigned a preconditioner to A11 with this cmd options: > > -fieldsplit_0_ksp_type gmres -fieldsplit_0_pc_type ilu > -fieldsplit_1_ksp_type gmres -fieldsplit_1_pc_type ilu > > Then I got this error: > > "Could not locate a solver type for factorization type ILU and matrix type > schurcomplement" > > How could I specify a preconditioner for A11? > > BR, > Xiaofeng > > > On Sep 26, 2022, at 11:02, ?? ? mailto:tlanyan at hotmail.com >> wrote: > > -fieldsplit_0_ksp_type gmres -fieldsplit_0_pc_type ilu > -fieldsplit_1_ksp_type gmres -fieldsplit_1_pc_type none > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From fujisan43 at gmail.com Mon Sep 26 07:52:21 2022 From: fujisan43 at gmail.com (fujisan) Date: Mon, 26 Sep 2022 14:52:21 +0200 Subject: [petsc-users] Problem solving Ax=b with rectangular matrix A In-Reply-To: <2C060B48-EB03-47F6-9E84-4A0A8E901770@joliv.et> References: <3023EA5A-19AB-4989-8E0D-533FD531783D@dsic.upv.es> <2C060B48-EB03-47F6-9E84-4A0A8E901770@joliv.et> Message-ID: Ok, Thank you. I didn't know about MatCreateNormal. In terms of computer performance, what is best to solve Ax=b with A rectangular? Is it to keep A rectangular and use KSPLSQR along with PCNONE or to convert to normal equations using MatCreateNormal and use another ksp type with another pc type? In the future, our A will be very sparse and will be something like up to 100 millions lines and 10 millions columns in size. I will study all that. Fuji On Mon, Sep 26, 2022 at 11:45 AM Pierre Jolivet wrote: > I?m sorry, solving overdetermined systems, alongside (overlapping) domain > decomposition preconditioners and solving systems with multiple right-hand > sides, is one of the topic for which I need to stop pushing new features > and update the users manual instead? > The very basic documentation of PCQR is here: > https://petsc.org/main/docs/manualpages/PC/PCQR (I?m guessing you are > using the release documentation in which it?s indeed not present). > Some comments about your problem of solving Ax=b with a rectangular matrix > A. > 1) if you switch to KSPLSQR, it is wrong to use KSPSetOperators(ksp, A, A). > You can get away with murder if your PCType is PCNONE, but otherwise, you > should always supply the normal equations as the Pmat (you will get runtime > errors otherwise). > To get the normal equations, you can use > https://petsc.org/main/docs/manualpages/Mat/MatCreateNormal/ > The following two points only applies if your Pmat is sparse (or sparse > with some dense rows). > 2) there are a couple of PC that handle MATNORMAL: PCNONE, PCQR, PCJACOBI, > PCBJACOBI, PCASM, PCHPDDM > Currently, PCQR needs SuiteSparse, and thus runs only if the Pmat is > distributed on a single MPI process (if your Pmat is distributed on > multiple processes, you should first use PCREDUNDANT). > 3) if you intend to make your problem scale in sizes, most of these > preconditioners will not be very robust. > In that case, if your problem does not have any dense rows, you should > either use PCHPDDM or MatConvert(Pmat, MATAIJ, PmatAIJ) and then use > PCCHOLESKY, PCHYPRE or PCGAMG (you can have a look at > https://epubs.siam.org/doi/abs/10.1137/21M1434891 for a comparison) > If your problem has dense rows, I have somewhere the code to recast it > into an augmented system then solved using PCFIELDSPLIT (see > https://link.springer.com/article/10.1007/s11075-018-0478-2). I can send > it to you if needed. > Let me know if I can be of further assistance or if something is not clear > to you. > > Thanks, > Pierre > > On 26 Sep 2022, at 10:56 AM, fujisan wrote: > > OK thank you. > > On Mon, Sep 26, 2022 at 10:53 AM Jose E. Roman wrote: > >> The QR factorization from SuiteSparse is sequential only, cannot be used >> in parallel. >> In parallel you can try PCBJACOBI with a PCQR local preconditioner. >> Pierre may have additional suggestions. >> >> Jose >> >> >> > El 26 sept 2022, a las 10:47, fujisan escribi?: >> > >> > I did configure Petsc with the option --download-suitesparse. >> > >> > The error is more like this: >> > PETSC ERROR: Could not locate a solver type for factorization type QR >> and matrix type mpiaij. >> > >> > Fuji >> > >> > On Mon, Sep 26, 2022 at 10:25 AM Jose E. Roman >> wrote: >> > If the error message is "Could not locate a solver type for >> factorization type QR" then you should configure PETSc with >> --download-suitesparse >> > >> > Jose >> > >> > >> > > El 26 sept 2022, a las 9:06, fujisan escribi?: >> > > >> > > Thank you Pierre, >> > > >> > > I used PCNONE along with KSPLSQR and it worked. >> > > But as for PCQR, it cannot be found. There is nothing about it in the >> documentation as well. >> > > >> > > Fuji >> > > >> > > On Wed, Sep 21, 2022 at 12:20 PM Pierre Jolivet >> wrote: >> > > Yes, but you need to use a KSP that handles rectangular Mat, such as >> KSPLSQR (-ksp_type lsqr). >> > > PCLU does not handle rectangular Pmat. The only PC that handle >> rectangular Pmat are PCQR, PCNONE. >> > > If you supply the normal equations as the Pmat for LSQR, then you can >> use ?standard? PC. >> > > You can have a look at >> https://petsc.org/main/src/ksp/ksp/tutorials/ex27.c.html that covers >> most of these cases. >> > > >> > > Thanks, >> > > Pierre >> > > >> > > (sorry for the earlier answer sent wrongfully to petsc-maint, please >> discard the previous email) >> > > >> > >> On 21 Sep 2022, at 10:03 AM, fujisan wrote: >> > >> >> > >> I'm trying to solve Ax=b with a sparse rectangular matrix A (of size >> 33x17 in my test) using >> > >> options '-ksp_type stcg -pc_type lu' on 1 or 2 cpus. >> > >> >> > >> And I always get an error saying "Incompatible vector local lengths" >> (see below). >> > >> >> > >> Here is the relevant lines of my code: >> > >> >> > >> program test >> > >> ... >> > >> ! Variable declarations >> > >> >> > >> PetscCallA(PetscInitialize(PETSC_NULL_CHARACTER,ierr)) >> > >> >> > >> PetscCall(MatCreate(PETSC_COMM_WORLD,A,ierr)) >> > >> PetscCall(MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,m,n,ierr)) >> > >> PetscCall(MatSetType(A,MATMPIAIJ,ierr)) >> > >> PetscCall(MatSetFromOptions(A,ierr)) >> > >> PetscCall(MatSetUp(A,ierr)) >> > >> PetscCall(MatGetOwnershipRange(A,istart,iend,ierr)) >> > >> >> > >> do irow=istart,iend-1 >> > >> ... Reading from file ... >> > >> PetscCall(MatSetValues(A,1,irow,nzv,col,val,ADD_VALUES,ierr)) >> > >> ... >> > >> enddo >> > >> >> > >> PetscCall(MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY,ierr)) >> > >> PetscCall(MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY,ierr)) >> > >> >> > >> ! Creating vectors x and b >> > >> PetscCallA(MatCreateVecs(A,x,b,ierr)) >> > >> >> > >> ! Duplicating x in u. >> > >> PetscCallA(VecDuplicate(x,u,ierr)) >> > >> >> > >> ! u is used to calculate b >> > >> PetscCallA(VecSet(u,1.0,ierr)) >> > >> >> > >> PetscCallA(VecAssemblyBegin(u,ierr)) >> > >> PetscCallA(VecAssemblyEnd(u,ierr)) >> > >> >> > >> ! Calculating Au = b >> > >> PetscCallA(MatMult(A,u,b,ierr)) ! A.u = b >> > >> >> > >> PetscCallA(KSPSetType(ksp,KSPCG,ierr)) >> > >> >> > >> PetscCallA(KSPSetOperators(ksp,A,A,ierr)) >> > >> >> > >> PetscCallA(KSPSetFromOptions(ksp,ierr)) >> > >> >> > >> ! Solving Ax = b, x unknown >> > >> PetscCallA(KSPSolve(ksp,b,x,ierr)) >> > >> >> > >> PetscCallA(VecDestroy(x,ierr)) >> > >> PetscCallA(VecDestroy(u,ierr)) >> > >> PetscCallA(VecDestroy(b,ierr)) >> > >> PetscCallA(MatDestroy(A,ierr)) >> > >> PetscCallA(KSPDestroy(ksp,ierr)) >> > >> >> > >> call PetscFinalize(ierr) >> > >> end program >> > >> >> > >> The code reads a sparse matrix from a binary file. >> > >> I also output the sizes of matrix A and vectors b, x, u. >> > >> They all seem consistent. >> > >> >> > >> What am I doing wrong? >> > >> Is it possible to solve Ax=b with A rectangular? >> > >> >> > >> Thank you in advance for your help. >> > >> Have a nice day. >> > >> >> > >> Fuji >> > >> >> > >> Matrix size : m= 33 n= 17 cpu size: 1 >> > >> Size of matrix A : 33 17 >> > >> Size of vector b : 33 >> > >> Size of vector x : 17 >> > >> Size of vector u : 17 >> > >> [0]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> > >> [0]PETSC ERROR: Arguments are incompatible >> > >> [0]PETSC ERROR: Incompatible vector local lengths parameter # 1 >> local size 33 != parameter # 2 local size 17 >> > >> [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble >> shooting. >> > >> [0]PETSC ERROR: Petsc Development GIT revision: >> v3.17.4-1341-g91b2b62a00 GIT Date: 2022-09-15 19:26:07 +0000 >> > >> [0]PETSC ERROR: ./bin/solve on a x86_64 named master by fujisan Tue >> Sep 20 16:56:37 2022 >> > >> [0]PETSC ERROR: Configure options --with-petsc-arch=x86_64 >> --COPTFLAGS="-g -O3" --FOPTFLAGS="-g -O3" --CXXOPTFLAGS="-g -O3" >> --with-debugging=0 --with-cc=mpiicc --with-cxx=mpiicpc --with-fc=mpiifort >> --with-single-library=1 --with-mpiexec=mpiexec --with-precision=double >> --with-fortran-interfaces=1 --with-make=1 --with-mpi=1 >> --with-mpi-compilers=1 --download-fblaslapack=0 --download-hypre=1 >> --download-cmake=0 --with-cmake=1 --download-metis=1 --download-parmetis=1 >> --download-ptscotch=0 --download-suitesparse=1 --download-triangle=1 >> --download-superlu=1 --download-superlu_dist=1 --download-scalapack=1 >> --download-mumps=1 --download-elemental=1 --download-spai=0 >> --download-parms=1 --download-moab=1 --download-chaco=0 --download-fftw=1 >> --with-petsc4py=1 --download-mpi4py=1 --download-saws >> --download-concurrencykit=1 --download-revolve=1 --download-cams=1 >> --download-p4est=0 --with-zlib=1 --download-mfem=1 --download-glvis=0 >> --with-opengl=0 --download-libpng=1 --download-libjpeg=1 --download-slepc=1 >> --download-hpddm=1 --download-bamg=1 --download-mmg=0 --download-parmmg=0 >> --download-htool=1 --download-egads=0 --download-opencascade=0 >> PETSC_ARCH=x86_64 >> > >> [0]PETSC ERROR: #1 VecCopy() at >> /data/softs/petsc/src/vec/vec/interface/vector.c:1607 >> > >> [0]PETSC ERROR: #2 KSPSolve_BiCG() at >> /data/softs/petsc/src/ksp/ksp/impls/bicg/bicg.c:40 >> > >> [0]PETSC ERROR: #3 KSPSolve_Private() at >> /data/softs/petsc/src/ksp/ksp/interface/itfunc.c:877 >> > >> [0]PETSC ERROR: #4 KSPSolve() at >> /data/softs/petsc/src/ksp/ksp/interface/itfunc.c:1048 >> > >> [0]PETSC ERROR: #5 solve.F90:218 >> > >> Abort(75) on node 0 (rank 0 in comm 16): application called >> MPI_Abort(MPI_COMM_SELF, 75) - process 0 >> > >> >> > > >> > >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Sep 26 07:58:42 2022 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 26 Sep 2022 08:58:42 -0400 Subject: [petsc-users] Problem solving Ax=b with rectangular matrix A In-Reply-To: References: <3023EA5A-19AB-4989-8E0D-533FD531783D@dsic.upv.es> <2C060B48-EB03-47F6-9E84-4A0A8E901770@joliv.et> Message-ID: On Mon, Sep 26, 2022 at 8:52 AM fujisan wrote: > Ok, Thank you. > I didn't know about MatCreateNormal. > > In terms of computer performance, what is best to solve Ax=b with A > rectangular? > Is it to keep A rectangular and use KSPLSQR along with PCNONE or > to convert to normal equations using MatCreateNormal and use another ksp > type with another pc type? > It depends on the spectral characteristics of the normal equations. Thanks, Matt > In the future, our A will be very sparse and will be something like up to > 100 millions lines and 10 millions columns in size. > > I will study all that. > > Fuji > > On Mon, Sep 26, 2022 at 11:45 AM Pierre Jolivet wrote: > >> I?m sorry, solving overdetermined systems, alongside (overlapping) domain >> decomposition preconditioners and solving systems with multiple right-hand >> sides, is one of the topic for which I need to stop pushing new features >> and update the users manual instead? >> The very basic documentation of PCQR is here: >> https://petsc.org/main/docs/manualpages/PC/PCQR (I?m guessing you are >> using the release documentation in which it?s indeed not present). >> Some comments about your problem of solving Ax=b with a rectangular >> matrix A. >> 1) if you switch to KSPLSQR, it is wrong to use KSPSetOperators(ksp, A, >> A). >> You can get away with murder if your PCType is PCNONE, but otherwise, you >> should always supply the normal equations as the Pmat (you will get runtime >> errors otherwise). >> To get the normal equations, you can use >> https://petsc.org/main/docs/manualpages/Mat/MatCreateNormal/ >> The following two points only applies if your Pmat is sparse (or sparse >> with some dense rows). >> 2) there are a couple of PC that handle MATNORMAL: PCNONE, PCQR, >> PCJACOBI, PCBJACOBI, PCASM, PCHPDDM >> Currently, PCQR needs SuiteSparse, and thus runs only if the Pmat is >> distributed on a single MPI process (if your Pmat is distributed on >> multiple processes, you should first use PCREDUNDANT). >> 3) if you intend to make your problem scale in sizes, most of these >> preconditioners will not be very robust. >> In that case, if your problem does not have any dense rows, you should >> either use PCHPDDM or MatConvert(Pmat, MATAIJ, PmatAIJ) and then use >> PCCHOLESKY, PCHYPRE or PCGAMG (you can have a look at >> https://epubs.siam.org/doi/abs/10.1137/21M1434891 for a comparison) >> If your problem has dense rows, I have somewhere the code to recast it >> into an augmented system then solved using PCFIELDSPLIT (see >> https://link.springer.com/article/10.1007/s11075-018-0478-2). I can send >> it to you if needed. >> Let me know if I can be of further assistance or if something is not >> clear to you. >> >> Thanks, >> Pierre >> >> On 26 Sep 2022, at 10:56 AM, fujisan wrote: >> >> OK thank you. >> >> On Mon, Sep 26, 2022 at 10:53 AM Jose E. Roman >> wrote: >> >>> The QR factorization from SuiteSparse is sequential only, cannot be used >>> in parallel. >>> In parallel you can try PCBJACOBI with a PCQR local preconditioner. >>> Pierre may have additional suggestions. >>> >>> Jose >>> >>> >>> > El 26 sept 2022, a las 10:47, fujisan escribi?: >>> > >>> > I did configure Petsc with the option --download-suitesparse. >>> > >>> > The error is more like this: >>> > PETSC ERROR: Could not locate a solver type for factorization type QR >>> and matrix type mpiaij. >>> > >>> > Fuji >>> > >>> > On Mon, Sep 26, 2022 at 10:25 AM Jose E. Roman >>> wrote: >>> > If the error message is "Could not locate a solver type for >>> factorization type QR" then you should configure PETSc with >>> --download-suitesparse >>> > >>> > Jose >>> > >>> > >>> > > El 26 sept 2022, a las 9:06, fujisan escribi?: >>> > > >>> > > Thank you Pierre, >>> > > >>> > > I used PCNONE along with KSPLSQR and it worked. >>> > > But as for PCQR, it cannot be found. There is nothing about it in >>> the documentation as well. >>> > > >>> > > Fuji >>> > > >>> > > On Wed, Sep 21, 2022 at 12:20 PM Pierre Jolivet >>> wrote: >>> > > Yes, but you need to use a KSP that handles rectangular Mat, such as >>> KSPLSQR (-ksp_type lsqr). >>> > > PCLU does not handle rectangular Pmat. The only PC that handle >>> rectangular Pmat are PCQR, PCNONE. >>> > > If you supply the normal equations as the Pmat for LSQR, then you >>> can use ?standard? PC. >>> > > You can have a look at >>> https://petsc.org/main/src/ksp/ksp/tutorials/ex27.c.html that covers >>> most of these cases. >>> > > >>> > > Thanks, >>> > > Pierre >>> > > >>> > > (sorry for the earlier answer sent wrongfully to petsc-maint, please >>> discard the previous email) >>> > > >>> > >> On 21 Sep 2022, at 10:03 AM, fujisan wrote: >>> > >> >>> > >> I'm trying to solve Ax=b with a sparse rectangular matrix A (of >>> size 33x17 in my test) using >>> > >> options '-ksp_type stcg -pc_type lu' on 1 or 2 cpus. >>> > >> >>> > >> And I always get an error saying "Incompatible vector local >>> lengths" (see below). >>> > >> >>> > >> Here is the relevant lines of my code: >>> > >> >>> > >> program test >>> > >> ... >>> > >> ! Variable declarations >>> > >> >>> > >> PetscCallA(PetscInitialize(PETSC_NULL_CHARACTER,ierr)) >>> > >> >>> > >> PetscCall(MatCreate(PETSC_COMM_WORLD,A,ierr)) >>> > >> PetscCall(MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,m,n,ierr)) >>> > >> PetscCall(MatSetType(A,MATMPIAIJ,ierr)) >>> > >> PetscCall(MatSetFromOptions(A,ierr)) >>> > >> PetscCall(MatSetUp(A,ierr)) >>> > >> PetscCall(MatGetOwnershipRange(A,istart,iend,ierr)) >>> > >> >>> > >> do irow=istart,iend-1 >>> > >> ... Reading from file ... >>> > >> >>> PetscCall(MatSetValues(A,1,irow,nzv,col,val,ADD_VALUES,ierr)) >>> > >> ... >>> > >> enddo >>> > >> >>> > >> PetscCall(MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY,ierr)) >>> > >> PetscCall(MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY,ierr)) >>> > >> >>> > >> ! Creating vectors x and b >>> > >> PetscCallA(MatCreateVecs(A,x,b,ierr)) >>> > >> >>> > >> ! Duplicating x in u. >>> > >> PetscCallA(VecDuplicate(x,u,ierr)) >>> > >> >>> > >> ! u is used to calculate b >>> > >> PetscCallA(VecSet(u,1.0,ierr)) >>> > >> >>> > >> PetscCallA(VecAssemblyBegin(u,ierr)) >>> > >> PetscCallA(VecAssemblyEnd(u,ierr)) >>> > >> >>> > >> ! Calculating Au = b >>> > >> PetscCallA(MatMult(A,u,b,ierr)) ! A.u = b >>> > >> >>> > >> PetscCallA(KSPSetType(ksp,KSPCG,ierr)) >>> > >> >>> > >> PetscCallA(KSPSetOperators(ksp,A,A,ierr)) >>> > >> >>> > >> PetscCallA(KSPSetFromOptions(ksp,ierr)) >>> > >> >>> > >> ! Solving Ax = b, x unknown >>> > >> PetscCallA(KSPSolve(ksp,b,x,ierr)) >>> > >> >>> > >> PetscCallA(VecDestroy(x,ierr)) >>> > >> PetscCallA(VecDestroy(u,ierr)) >>> > >> PetscCallA(VecDestroy(b,ierr)) >>> > >> PetscCallA(MatDestroy(A,ierr)) >>> > >> PetscCallA(KSPDestroy(ksp,ierr)) >>> > >> >>> > >> call PetscFinalize(ierr) >>> > >> end program >>> > >> >>> > >> The code reads a sparse matrix from a binary file. >>> > >> I also output the sizes of matrix A and vectors b, x, u. >>> > >> They all seem consistent. >>> > >> >>> > >> What am I doing wrong? >>> > >> Is it possible to solve Ax=b with A rectangular? >>> > >> >>> > >> Thank you in advance for your help. >>> > >> Have a nice day. >>> > >> >>> > >> Fuji >>> > >> >>> > >> Matrix size : m= 33 n= 17 cpu size: >>> 1 >>> > >> Size of matrix A : 33 17 >>> > >> Size of vector b : 33 >>> > >> Size of vector x : 17 >>> > >> Size of vector u : 17 >>> > >> [0]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> > >> [0]PETSC ERROR: Arguments are incompatible >>> > >> [0]PETSC ERROR: Incompatible vector local lengths parameter # 1 >>> local size 33 != parameter # 2 local size 17 >>> > >> [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble >>> shooting. >>> > >> [0]PETSC ERROR: Petsc Development GIT revision: >>> v3.17.4-1341-g91b2b62a00 GIT Date: 2022-09-15 19:26:07 +0000 >>> > >> [0]PETSC ERROR: ./bin/solve on a x86_64 named master by fujisan Tue >>> Sep 20 16:56:37 2022 >>> > >> [0]PETSC ERROR: Configure options --with-petsc-arch=x86_64 >>> --COPTFLAGS="-g -O3" --FOPTFLAGS="-g -O3" --CXXOPTFLAGS="-g -O3" >>> --with-debugging=0 --with-cc=mpiicc --with-cxx=mpiicpc --with-fc=mpiifort >>> --with-single-library=1 --with-mpiexec=mpiexec --with-precision=double >>> --with-fortran-interfaces=1 --with-make=1 --with-mpi=1 >>> --with-mpi-compilers=1 --download-fblaslapack=0 --download-hypre=1 >>> --download-cmake=0 --with-cmake=1 --download-metis=1 --download-parmetis=1 >>> --download-ptscotch=0 --download-suitesparse=1 --download-triangle=1 >>> --download-superlu=1 --download-superlu_dist=1 --download-scalapack=1 >>> --download-mumps=1 --download-elemental=1 --download-spai=0 >>> --download-parms=1 --download-moab=1 --download-chaco=0 --download-fftw=1 >>> --with-petsc4py=1 --download-mpi4py=1 --download-saws >>> --download-concurrencykit=1 --download-revolve=1 --download-cams=1 >>> --download-p4est=0 --with-zlib=1 --download-mfem=1 --download-glvis=0 >>> --with-opengl=0 --download-libpng=1 --download-libjpeg=1 --download-slepc=1 >>> --download-hpddm=1 --download-bamg=1 --download-mmg=0 --download-parmmg=0 >>> --download-htool=1 --download-egads=0 --download-opencascade=0 >>> PETSC_ARCH=x86_64 >>> > >> [0]PETSC ERROR: #1 VecCopy() at >>> /data/softs/petsc/src/vec/vec/interface/vector.c:1607 >>> > >> [0]PETSC ERROR: #2 KSPSolve_BiCG() at >>> /data/softs/petsc/src/ksp/ksp/impls/bicg/bicg.c:40 >>> > >> [0]PETSC ERROR: #3 KSPSolve_Private() at >>> /data/softs/petsc/src/ksp/ksp/interface/itfunc.c:877 >>> > >> [0]PETSC ERROR: #4 KSPSolve() at >>> /data/softs/petsc/src/ksp/ksp/interface/itfunc.c:1048 >>> > >> [0]PETSC ERROR: #5 solve.F90:218 >>> > >> Abort(75) on node 0 (rank 0 in comm 16): application called >>> MPI_Abort(MPI_COMM_SELF, 75) - process 0 >>> > >> >>> > > >>> > >>> >>> >> -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Mon Sep 26 08:03:07 2022 From: jed at jedbrown.org (Jed Brown) Date: Mon, 26 Sep 2022 07:03:07 -0600 Subject: [petsc-users] Solve Linear System with Field Split Preconditioner In-Reply-To: References: <819A69F5-6CB0-494E-80B2-18EB023FD048@hotmail.com> <87pmfihddt.fsf@jedbrown.org> <22A0172A-4AAA-4247-BA35-266555EDA4CF@hotmail.com> Message-ID: <87mtamgues.fsf@jedbrown.org> Xiaofeng, is your saddle point due to incompressibility or other constraints (like Lagrange multipliers for contact or multi-point constraints)? If incompressibility, are you working on structured or unstructured/non-nested meshes? Matthew Knepley writes: > Another option are the PCPATCH solvers for multigrid, as shown in this > paper: https://arxiv.org/abs/1912.08516 > which I believe solves incompressible elasticity. There is an example in > PETSc for Stokes I believe. > > Thanks, > > Matt > > On Mon, Sep 26, 2022 at 5:20 AM ?? ? wrote: > >> Are there other approaches to solve this kind of systems in PETSc except >> for field-split methods? >> >> Thanks, >> Xiaofeng >> >> On Sep 26, 2022, at 14:13, Jed Brown wrote: >> >> This is the joy of factorization field-split methods. The actual Schur >> complement is dense, so we represent it implicitly. A common strategy is to >> assemble the mass matrix and drop it in the 11 block of the Pmat. You can >> check out some examples in the repository for incompressible flow (Stokes >> problems). The LSC (least squares commutator) is another option. You'll >> likely find that lumping diag(A00)^{-1} works poorly because the resulting >> operator behaves like a Laplacian rather than like a mass matrix. >> >> ?? ? writes: >> >> If assigned a preconditioner to A11 with this cmd options: >> >> -fieldsplit_0_ksp_type gmres -fieldsplit_0_pc_type ilu >> -fieldsplit_1_ksp_type gmres -fieldsplit_1_pc_type ilu >> >> Then I got this error: >> >> "Could not locate a solver type for factorization type ILU and matrix type >> schurcomplement" >> >> How could I specify a preconditioner for A11? >> >> BR, >> Xiaofeng >> >> >> On Sep 26, 2022, at 11:02, ?? ? > mailto:tlanyan at hotmail.com >> wrote: >> >> -fieldsplit_0_ksp_type gmres -fieldsplit_0_pc_type ilu >> -fieldsplit_1_ksp_type gmres -fieldsplit_1_pc_type none >> >> >> > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ From pierre at joliv.et Mon Sep 26 08:20:04 2022 From: pierre at joliv.et (Pierre Jolivet) Date: Mon, 26 Sep 2022 15:20:04 +0200 Subject: [petsc-users] Problem solving Ax=b with rectangular matrix A In-Reply-To: References: <3023EA5A-19AB-4989-8E0D-533FD531783D@dsic.upv.es> <2C060B48-EB03-47F6-9E84-4A0A8E901770@joliv.et> Message-ID: > On 26 Sep 2022, at 2:52 PM, fujisan wrote: > > Ok, Thank you. > I didn't know about MatCreateNormal. > > In terms of computer performance, what is best to solve Ax=b with A rectangular? > Is it to keep A rectangular and use KSPLSQR along with PCNONE or > to convert to normal equations using MatCreateNormal and use another ksp type with another pc type? It?s extremely cheap to call MatCreateNormal(). It?s just an object that behaves like A^T A, but it?s actually not assembled. > In the future, our A will be very sparse and will be something like up to 100 millions lines and 10 millions columns in size. Following-up on Matt?s answer, the good news is that if A is indeed very sparse and if A^T A is sparse as well, then you have a couple of options. I?d be happy helping you out benchmark those. Some of the largest problems I have been investigating thus far are https://sparse.tamu.edu/Hardesty/Hardesty3 and https://sparse.tamu.edu/Mittelmann/cont11_l To this day, PETSc has no Mat partitioner for rectangular systems (like PaToH), so whenever you need to redistribute such systems, the normal equations must be formed explicitly. This can get quite costly, but if you already have an application with the proper partitioning, then the sky is the limit, I guess. Thanks, Pierre > I will study all that. > > Fuji > > On Mon, Sep 26, 2022 at 11:45 AM Pierre Jolivet > wrote: > I?m sorry, solving overdetermined systems, alongside (overlapping) domain decomposition preconditioners and solving systems with multiple right-hand sides, is one of the topic for which I need to stop pushing new features and update the users manual instead? > The very basic documentation of PCQR is here: https://petsc.org/main/docs/manualpages/PC/PCQR (I?m guessing you are using the release documentation in which it?s indeed not present). > Some comments about your problem of solving Ax=b with a rectangular matrix A. > 1) if you switch to KSPLSQR, it is wrong to use KSPSetOperators(ksp, A, A). > You can get away with murder if your PCType is PCNONE, but otherwise, you should always supply the normal equations as the Pmat (you will get runtime errors otherwise). > To get the normal equations, you can use https://petsc.org/main/docs/manualpages/Mat/MatCreateNormal/ > The following two points only applies if your Pmat is sparse (or sparse with some dense rows). > 2) there are a couple of PC that handle MATNORMAL: PCNONE, PCQR, PCJACOBI, PCBJACOBI, PCASM, PCHPDDM > Currently, PCQR needs SuiteSparse, and thus runs only if the Pmat is distributed on a single MPI process (if your Pmat is distributed on multiple processes, you should first use PCREDUNDANT). > 3) if you intend to make your problem scale in sizes, most of these preconditioners will not be very robust. > In that case, if your problem does not have any dense rows, you should either use PCHPDDM or MatConvert(Pmat, MATAIJ, PmatAIJ) and then use PCCHOLESKY, PCHYPRE or PCGAMG (you can have a look at https://epubs.siam.org/doi/abs/10.1137/21M1434891 for a comparison) > If your problem has dense rows, I have somewhere the code to recast it into an augmented system then solved using PCFIELDSPLIT (see https://link.springer.com/article/10.1007/s11075-018-0478-2 ). I can send it to you if needed. > Let me know if I can be of further assistance or if something is not clear to you. > > Thanks, > Pierre > >> On 26 Sep 2022, at 10:56 AM, fujisan > wrote: >> >> OK thank you. >> >> On Mon, Sep 26, 2022 at 10:53 AM Jose E. Roman > wrote: >> The QR factorization from SuiteSparse is sequential only, cannot be used in parallel. >> In parallel you can try PCBJACOBI with a PCQR local preconditioner. >> Pierre may have additional suggestions. >> >> Jose >> >> >> > El 26 sept 2022, a las 10:47, fujisan > escribi?: >> > >> > I did configure Petsc with the option --download-suitesparse. >> > >> > The error is more like this: >> > PETSC ERROR: Could not locate a solver type for factorization type QR and matrix type mpiaij. >> > >> > Fuji >> > >> > On Mon, Sep 26, 2022 at 10:25 AM Jose E. Roman > wrote: >> > If the error message is "Could not locate a solver type for factorization type QR" then you should configure PETSc with --download-suitesparse >> > >> > Jose >> > >> > >> > > El 26 sept 2022, a las 9:06, fujisan > escribi?: >> > > >> > > Thank you Pierre, >> > > >> > > I used PCNONE along with KSPLSQR and it worked. >> > > But as for PCQR, it cannot be found. There is nothing about it in the documentation as well. >> > > >> > > Fuji >> > > >> > > On Wed, Sep 21, 2022 at 12:20 PM Pierre Jolivet > wrote: >> > > Yes, but you need to use a KSP that handles rectangular Mat, such as KSPLSQR (-ksp_type lsqr). >> > > PCLU does not handle rectangular Pmat. The only PC that handle rectangular Pmat are PCQR, PCNONE. >> > > If you supply the normal equations as the Pmat for LSQR, then you can use ?standard? PC. >> > > You can have a look at https://petsc.org/main/src/ksp/ksp/tutorials/ex27.c.html that covers most of these cases. >> > > >> > > Thanks, >> > > Pierre >> > > >> > > (sorry for the earlier answer sent wrongfully to petsc-maint, please discard the previous email) >> > > >> > >> On 21 Sep 2022, at 10:03 AM, fujisan > wrote: >> > >> >> > >> I'm trying to solve Ax=b with a sparse rectangular matrix A (of size 33x17 in my test) using >> > >> options '-ksp_type stcg -pc_type lu' on 1 or 2 cpus. >> > >> >> > >> And I always get an error saying "Incompatible vector local lengths" (see below). >> > >> >> > >> Here is the relevant lines of my code: >> > >> >> > >> program test >> > >> ... >> > >> ! Variable declarations >> > >> >> > >> PetscCallA(PetscInitialize(PETSC_NULL_CHARACTER,ierr)) >> > >> >> > >> PetscCall(MatCreate(PETSC_COMM_WORLD,A,ierr)) >> > >> PetscCall(MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,m,n,ierr)) >> > >> PetscCall(MatSetType(A,MATMPIAIJ,ierr)) >> > >> PetscCall(MatSetFromOptions(A,ierr)) >> > >> PetscCall(MatSetUp(A,ierr)) >> > >> PetscCall(MatGetOwnershipRange(A,istart,iend,ierr)) >> > >> >> > >> do irow=istart,iend-1 >> > >> ... Reading from file ... >> > >> PetscCall(MatSetValues(A,1,irow,nzv,col,val,ADD_VALUES,ierr)) >> > >> ... >> > >> enddo >> > >> >> > >> PetscCall(MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY,ierr)) >> > >> PetscCall(MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY,ierr)) >> > >> >> > >> ! Creating vectors x and b >> > >> PetscCallA(MatCreateVecs(A,x,b,ierr)) >> > >> >> > >> ! Duplicating x in u. >> > >> PetscCallA(VecDuplicate(x,u,ierr)) >> > >> >> > >> ! u is used to calculate b >> > >> PetscCallA(VecSet(u,1.0,ierr)) >> > >> >> > >> PetscCallA(VecAssemblyBegin(u,ierr)) >> > >> PetscCallA(VecAssemblyEnd(u,ierr)) >> > >> >> > >> ! Calculating Au = b >> > >> PetscCallA(MatMult(A,u,b,ierr)) ! A.u = b >> > >> >> > >> PetscCallA(KSPSetType(ksp,KSPCG,ierr)) >> > >> >> > >> PetscCallA(KSPSetOperators(ksp,A,A,ierr)) >> > >> >> > >> PetscCallA(KSPSetFromOptions(ksp,ierr)) >> > >> >> > >> ! Solving Ax = b, x unknown >> > >> PetscCallA(KSPSolve(ksp,b,x,ierr)) >> > >> >> > >> PetscCallA(VecDestroy(x,ierr)) >> > >> PetscCallA(VecDestroy(u,ierr)) >> > >> PetscCallA(VecDestroy(b,ierr)) >> > >> PetscCallA(MatDestroy(A,ierr)) >> > >> PetscCallA(KSPDestroy(ksp,ierr)) >> > >> >> > >> call PetscFinalize(ierr) >> > >> end program >> > >> >> > >> The code reads a sparse matrix from a binary file. >> > >> I also output the sizes of matrix A and vectors b, x, u. >> > >> They all seem consistent. >> > >> >> > >> What am I doing wrong? >> > >> Is it possible to solve Ax=b with A rectangular? >> > >> >> > >> Thank you in advance for your help. >> > >> Have a nice day. >> > >> >> > >> Fuji >> > >> >> > >> Matrix size : m= 33 n= 17 cpu size: 1 >> > >> Size of matrix A : 33 17 >> > >> Size of vector b : 33 >> > >> Size of vector x : 17 >> > >> Size of vector u : 17 >> > >> [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >> > >> [0]PETSC ERROR: Arguments are incompatible >> > >> [0]PETSC ERROR: Incompatible vector local lengths parameter # 1 local size 33 != parameter # 2 local size 17 >> > >> [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting. >> > >> [0]PETSC ERROR: Petsc Development GIT revision: v3.17.4-1341-g91b2b62a00 GIT Date: 2022-09-15 19:26:07 +0000 >> > >> [0]PETSC ERROR: ./bin/solve on a x86_64 named master by fujisan Tue Sep 20 16:56:37 2022 >> > >> [0]PETSC ERROR: Configure options --with-petsc-arch=x86_64 --COPTFLAGS="-g -O3" --FOPTFLAGS="-g -O3" --CXXOPTFLAGS="-g -O3" --with-debugging=0 --with-cc=mpiicc --with-cxx=mpiicpc --with-fc=mpiifort --with-single-library=1 --with-mpiexec=mpiexec --with-precision=double --with-fortran-interfaces=1 --with-make=1 --with-mpi=1 --with-mpi-compilers=1 --download-fblaslapack=0 --download-hypre=1 --download-cmake=0 --with-cmake=1 --download-metis=1 --download-parmetis=1 --download-ptscotch=0 --download-suitesparse=1 --download-triangle=1 --download-superlu=1 --download-superlu_dist=1 --download-scalapack=1 --download-mumps=1 --download-elemental=1 --download-spai=0 --download-parms=1 --download-moab=1 --download-chaco=0 --download-fftw=1 --with-petsc4py=1 --download-mpi4py=1 --download-saws --download-concurrencykit=1 --download-revolve=1 --download-cams=1 --download-p4est=0 --with-zlib=1 --download-mfem=1 --download-glvis=0 --with-opengl=0 --download-libpng=1 --download-libjpeg=1 --download-slepc=1 --download-hpddm=1 --download-bamg=1 --download-mmg=0 --download-parmmg=0 --download-htool=1 --download-egads=0 --download-opencascade=0 PETSC_ARCH=x86_64 >> > >> [0]PETSC ERROR: #1 VecCopy() at /data/softs/petsc/src/vec/vec/interface/vector.c:1607 >> > >> [0]PETSC ERROR: #2 KSPSolve_BiCG() at /data/softs/petsc/src/ksp/ksp/impls/bicg/bicg.c:40 >> > >> [0]PETSC ERROR: #3 KSPSolve_Private() at /data/softs/petsc/src/ksp/ksp/interface/itfunc.c:877 >> > >> [0]PETSC ERROR: #4 KSPSolve() at /data/softs/petsc/src/ksp/ksp/interface/itfunc.c:1048 >> > >> [0]PETSC ERROR: #5 solve.F90:218 >> > >> Abort(75) on node 0 (rank 0 in comm 16): application called MPI_Abort(MPI_COMM_SELF, 75) - process 0 >> > >> >> > > >> > >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tlanyan at hotmail.com Mon Sep 26 09:08:27 2022 From: tlanyan at hotmail.com (=?gb2312?B?z/635SC6zg==?=) Date: Mon, 26 Sep 2022 14:08:27 +0000 Subject: [petsc-users] Solve Linear System with Field Split Preconditioner In-Reply-To: <87mtamgues.fsf@jedbrown.org> References: <819A69F5-6CB0-494E-80B2-18EB023FD048@hotmail.com> <87pmfihddt.fsf@jedbrown.org> <22A0172A-4AAA-4247-BA35-266555EDA4CF@hotmail.com> <87mtamgues.fsf@jedbrown.org> Message-ID: <22BCBB91-02A2-4440-BE9C-27F06D1E05CF@hotmail.com> Hello Jed, The saddle point is due to Lagrange multipliers, thus the size of A11 is much smaller than A00. Best Regards, Xiaofeng On Sep 26, 2022, at 21:03, Jed Brown > wrote: Lagrange multipliers -------------- next part -------------- An HTML attachment was scrubbed... URL: From tlanyan at hotmail.com Mon Sep 26 09:11:34 2022 From: tlanyan at hotmail.com (=?utf-8?B?5pmT5bOwIOS9lQ==?=) Date: Mon, 26 Sep 2022 14:11:34 +0000 Subject: [petsc-users] Solve Linear System with Field Split Preconditioner In-Reply-To: References: <819A69F5-6CB0-494E-80B2-18EB023FD048@hotmail.com> <87pmfihddt.fsf@jedbrown.org> <22A0172A-4AAA-4247-BA35-266555EDA4CF@hotmail.com> Message-ID: <2BA51259-4354-4C36-B976-BBD61F8290C3@hotmail.com> Hello Matt, Many thanks for your suggestion. BR, Xiaofeng On Sep 26, 2022, at 20:29, Matthew Knepley > wrote: Another option are the PCPATCH solvers for multigrid, as shown in this paper: https://arxiv.org/abs/1912.08516 which I believe solves incompressible elasticity. There is an example in PETSc for Stokes I believe. Thanks, Matt On Mon, Sep 26, 2022 at 5:20 AM ?? ? > wrote: Are there other approaches to solve this kind of systems in PETSc except for field-split methods? Thanks, Xiaofeng On Sep 26, 2022, at 14:13, Jed Brown > wrote: This is the joy of factorization field-split methods. The actual Schur complement is dense, so we represent it implicitly. A common strategy is to assemble the mass matrix and drop it in the 11 block of the Pmat. You can check out some examples in the repository for incompressible flow (Stokes problems). The LSC (least squares commutator) is another option. You'll likely find that lumping diag(A00)^{-1} works poorly because the resulting operator behaves like a Laplacian rather than like a mass matrix. ?? ? > writes: If assigned a preconditioner to A11 with this cmd options: -fieldsplit_0_ksp_type gmres -fieldsplit_0_pc_type ilu -fieldsplit_1_ksp_type gmres -fieldsplit_1_pc_type ilu Then I got this error: "Could not locate a solver type for factorization type ILU and matrix type schurcomplement" How could I specify a preconditioner for A11? BR, Xiaofeng On Sep 26, 2022, at 11:02, ?? ? > wrote: -fieldsplit_0_ksp_type gmres -fieldsplit_0_pc_type ilu -fieldsplit_1_ksp_type gmres -fieldsplit_1_pc_type none -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Mon Sep 26 10:11:57 2022 From: bsmith at petsc.dev (Barry Smith) Date: Mon, 26 Sep 2022 11:11:57 -0400 Subject: [petsc-users] Solve Linear System with Field Split Preconditioner In-Reply-To: <22BCBB91-02A2-4440-BE9C-27F06D1E05CF@hotmail.com> References: <819A69F5-6CB0-494E-80B2-18EB023FD048@hotmail.com> <87pmfihddt.fsf@jedbrown.org> <22A0172A-4AAA-4247-BA35-266555EDA4CF@hotmail.com> <87mtamgues.fsf@jedbrown.org> <22BCBB91-02A2-4440-BE9C-27F06D1E05CF@hotmail.com> Message-ID: What is your A00 operator? ILU is almost never a good choice for large scale problems. If it is an elliptic operator that using a PC of gamg may work well for the A00 preconditioner instead of ILU. Barry For moderate size problems you can use a PC type LU for AOO to help you understand the best preconditioner to use for the A11 (the Schur complement block), once you have a good preconditioner for the A11 block you would then go back and determine a good preconditioner for the A00 block. > On Sep 26, 2022, at 10:08 AM, ?? ? wrote: > > Hello Jed, > > The saddle point is due to Lagrange multipliers, thus the size of A11 is much smaller than A00. > > > > Best Regards, > > Xiaofeng > > >> On Sep 26, 2022, at 21:03, Jed Brown > wrote: >> >> Lagrange multipliers > -------------- next part -------------- An HTML attachment was scrubbed... URL: From junming.duan at epfl.ch Mon Sep 26 11:39:52 2022 From: junming.duan at epfl.ch (Duan Junming) Date: Mon, 26 Sep 2022 16:39:52 +0000 Subject: [petsc-users] Strange mpi timing and CPU load when -np > 2 Message-ID: Dear all, I am using PETSc 3.17.4 on a Linux server, compiling with --download-exodus --download-hdf5 --download-openmpi --download-triangle --with-fc=0 --with-debugging=0 PETSC_ARCH=arch-linux-c-opt COPTFLAGS="-g -O3" CXXOPTFLAGS="-g -O3". The strange thing is when I run my code with mpirun -np 1 ./main, the CPU time is 30s. When I use mpirun -np 2 ./main, the CPU time is 16s. It's OK. But when I use more than 2 CPUs, like mpirun -np 3 ./main, the CPU time is 30s. The output of command time is: real 0m30.189s, user 9m3.133s, sys 10m55.715s. I can also see that the CPU load is about 100% for each process when np = 2, but the CPU load goes to 2000%, 1000%, 1000% for each process (the server has 40 CPUs). Do you have any idea about this? Thanks in advance! -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Sep 26 11:57:09 2022 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 26 Sep 2022 12:57:09 -0400 Subject: [petsc-users] Strange mpi timing and CPU load when -np > 2 In-Reply-To: References: Message-ID: On Mon, Sep 26, 2022 at 12:40 PM Duan Junming via petsc-users < petsc-users at mcs.anl.gov> wrote: > Dear all, > > I am using PETSc 3.17.4 on a Linux server, compiling > with --download-exodus --download-hdf5 --download-openmpi > --download-triangle --with-fc=0 --with-debugging=0 > PETSC_ARCH=arch-linux-c-opt COPTFLAGS="-g -O3" CXXOPTFLAGS="-g -O3". > The strange thing is when I run my code with mpirun -np 1 ./main, the CPU > time is 30s. > When I use mpirun -np 2 ./main, the CPU time is 16s. It's OK. > But when I use more than 2 CPUs, like mpirun -np 3 ./main, the CPU time is > 30s. > The output of command time is: real 0m30.189s, user 9m3.133s, sys > 10m55.715s. > I can also see that the CPU load is about 100% for each process when np = > 2, but the CPU load goes to 2000%, 1000%, 1000% for each process (the > server has 40 CPUs). > Do you have any idea about this? > I believe this is an MPI implementation problem, in which there is a large penalty for oversubscription. I think you can try --download-mpich --download-mpich-pm=gforker which should be good for oversubscription. Thanks, Matt > Thanks in advance! > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Mon Sep 26 11:58:15 2022 From: bsmith at petsc.dev (Barry Smith) Date: Mon, 26 Sep 2022 12:58:15 -0400 Subject: [petsc-users] Strange mpi timing and CPU load when -np > 2 In-Reply-To: References: Message-ID: <60BBFA0F-E170-4530-B8F6-4388403515AF@petsc.dev> It is important to check out https://petsc.org/main/faq/?highlight=faq#why-is-my-parallel-solver-slower-than-my-sequential-solver-or-i-have-poor-speed-up In particular you will need to set appropriate binding for the mpiexec and should run the streams benchmark (make mpistreams) using the binding to find the potential performance of the system. If you are using a thread enabled BLAS/LAPACK that utilizes all the cores then you can get oversubscription and thus slow performance during BLAS/LAPACK calls. We try not to link with thread enabled BLAS/LAPACK by default. See https://petsc.org/main/docs/manual/blas-lapack/?highlight=thread%20blas Barry > On Sep 26, 2022, at 12:39 PM, Duan Junming via petsc-users wrote: > > Dear all, > > I am using PETSc 3.17.4 on a Linux server, compiling with --download-exodus --download-hdf5 --download-openmpi --download-triangle --with-fc=0 --with-debugging=0 PETSC_ARCH=arch-linux-c-opt COPTFLAGS="-g -O3" CXXOPTFLAGS="-g -O3". > The strange thing is when I run my code with mpirun -np 1 ./main, the CPU time is 30s. > When I use mpirun -np 2 ./main, the CPU time is 16s. It's OK. > But when I use more than 2 CPUs, like mpirun -np 3 ./main, the CPU time is 30s. > The output of command time is: real 0m30.189s, user 9m3.133s, sys 10m55.715s. > I can also see that the CPU load is about 100% for each process when np = 2, but the CPU load goes to 2000%, 1000%, 1000% for each process (the server has 40 CPUs). > Do you have any idea about this? > > Thanks in advance! -------------- next part -------------- An HTML attachment was scrubbed... URL: From tlanyan at hotmail.com Tue Sep 27 00:45:43 2022 From: tlanyan at hotmail.com (=?utf-8?B?5pmT5bOwIOS9lQ==?=) Date: Tue, 27 Sep 2022 05:45:43 +0000 Subject: [petsc-users] Solve Linear System with Field Split Preconditioner In-Reply-To: References: <819A69F5-6CB0-494E-80B2-18EB023FD048@hotmail.com> <87pmfihddt.fsf@jedbrown.org> <22A0172A-4AAA-4247-BA35-266555EDA4CF@hotmail.com> <87mtamgues.fsf@jedbrown.org> <22BCBB91-02A2-4440-BE9C-27F06D1E05CF@hotmail.com> Message-ID: Hi Barry, A00 is formed from elliptic operator. I tried GAMG with A00, but it was extremely slow to solve the system with field-split preconditioner(I?m not sure I did it with the right way). Thanks, Xiaofeng On Sep 26, 2022, at 23:11, Barry Smith > wrote: What is your A00 operator? ILU is almost never a good choice for large scale problems. If it is an elliptic operator that using a PC of gamg may work well for the A00 preconditioner instead of ILU. Barry For moderate size problems you can use a PC type LU for AOO to help you understand the best preconditioner to use for the A11 (the Schur complement block), once you have a good preconditioner for the A11 block you would then go back and determine a good preconditioner for the A00 block. On Sep 26, 2022, at 10:08 AM, ?? ? > wrote: Hello Jed, The saddle point is due to Lagrange multipliers, thus the size of A11 is much smaller than A00. Best Regards, Xiaofeng On Sep 26, 2022, at 21:03, Jed Brown > wrote: Lagrange multipliers -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Tue Sep 27 04:48:51 2022 From: mfadams at lbl.gov (Mark Adams) Date: Tue, 27 Sep 2022 05:48:51 -0400 Subject: [petsc-users] Solve Linear System with Field Split Preconditioner In-Reply-To: References: <819A69F5-6CB0-494E-80B2-18EB023FD048@hotmail.com> <87pmfihddt.fsf@jedbrown.org> <22A0172A-4AAA-4247-BA35-266555EDA4CF@hotmail.com> <87mtamgues.fsf@jedbrown.org> <22BCBB91-02A2-4440-BE9C-27F06D1E05CF@hotmail.com> Message-ID: what equations and discetizations are in A00? On Tue, Sep 27, 2022 at 1:45 AM ?? ? wrote: > Hi Barry, > > A00 is formed from elliptic operator. > > I tried GAMG with A00, but it was extremely slow to solve the system with > field-split preconditioner(I?m not sure I did it with the right way). > > Thanks, > Xiaofeng > > On Sep 26, 2022, at 23:11, Barry Smith wrote: > > > What is your A00 operator? ILU is almost never a good choice for large > scale problems. If it is an elliptic operator that using a PC of gamg may > work well for the A00 preconditioner instead of ILU. > > Barry > > For moderate size problems you can use a PC type LU for AOO to help you > understand the best preconditioner to use for the A11 (the Schur complement > block), once you have a good preconditioner for the A11 block you would > then go back and determine a good preconditioner for the A00 block. > > On Sep 26, 2022, at 10:08 AM, ?? ? wrote: > > Hello Jed, > > The saddle point is due to Lagrange multipliers, thus the size of A11 is > much smaller than A00. > > > > Best Regards, > > Xiaofeng > > > On Sep 26, 2022, at 21:03, Jed Brown wrote: > > Lagrange multipliers > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tlanyan at hotmail.com Tue Sep 27 05:20:32 2022 From: tlanyan at hotmail.com (=?utf-8?B?5pmT5bOwIOS9lQ==?=) Date: Tue, 27 Sep 2022 10:20:32 +0000 Subject: [petsc-users] Solve Linear System with Field Split Preconditioner In-Reply-To: References: <819A69F5-6CB0-494E-80B2-18EB023FD048@hotmail.com> <87pmfihddt.fsf@jedbrown.org> <22A0172A-4AAA-4247-BA35-266555EDA4CF@hotmail.com> <87mtamgues.fsf@jedbrown.org> <22BCBB91-02A2-4440-BE9C-27F06D1E05CF@hotmail.com> Message-ID: Hello, A00 comes from shell structures and discretized by FEM. Thanks, Xiaofeng On Sep 27, 2022, at 17:48, Mark Adams > wrote: what equations and discetizations are in A00? On Tue, Sep 27, 2022 at 1:45 AM ?? ? > wrote: Hi Barry, A00 is formed from elliptic operator. I tried GAMG with A00, but it was extremely slow to solve the system with field-split preconditioner(I?m not sure I did it with the right way). Thanks, Xiaofeng On Sep 26, 2022, at 23:11, Barry Smith > wrote: What is your A00 operator? ILU is almost never a good choice for large scale problems. If it is an elliptic operator that using a PC of gamg may work well for the A00 preconditioner instead of ILU. Barry For moderate size problems you can use a PC type LU for AOO to help you understand the best preconditioner to use for the A11 (the Schur complement block), once you have a good preconditioner for the A11 block you would then go back and determine a good preconditioner for the A00 block. On Sep 26, 2022, at 10:08 AM, ?? ? > wrote: Hello Jed, The saddle point is due to Lagrange multipliers, thus the size of A11 is much smaller than A00. Best Regards, Xiaofeng On Sep 26, 2022, at 21:03, Jed Brown > wrote: Lagrange multipliers -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Tue Sep 27 07:01:25 2022 From: mfadams at lbl.gov (Mark Adams) Date: Tue, 27 Sep 2022 08:01:25 -0400 Subject: [petsc-users] Solve Linear System with Field Split Preconditioner In-Reply-To: References: <819A69F5-6CB0-494E-80B2-18EB023FD048@hotmail.com> <87pmfihddt.fsf@jedbrown.org> <22A0172A-4AAA-4247-BA35-266555EDA4CF@hotmail.com> <87mtamgues.fsf@jedbrown.org> <22BCBB91-02A2-4440-BE9C-27F06D1E05CF@hotmail.com> Message-ID: Shells can be pretty hard. Start with an easy problem with well-shaped elements and probably best to start with a well-supported structure (eg, not a long cantilever). You should also configure with hypre and try that. I don't know if they deal with shells, but it is a well developed solver. For GAMG, you need to set the coordinates (PCSetCoordinates) which will create the rigid body modes for GAMG so that it can construct the rotational RBMs. This code has not been used for shells in a long time (petsc/src/ksp/pc/impls/gamg/agg.c:138), but it did once work (20 years ago in another code). This code constructs the 6 rigid body modes for 3D shell problems with my ording (6 dof, x,y,z,xx,yy,zz). Or you can construct the RBMs yourself and set them with https://petsc.org/release/docs/manualpages/Mat/MatNullSpaceCreateRigidBody.html There is some discussion here: https://www2.eecs.berkeley.edu/Pubs/TechRpts/2000/CSD-00-1103.pdf Mark On Tue, Sep 27, 2022 at 6:20 AM ?? ? wrote: > Hello, > > A00 comes from shell structures and discretized by FEM. > > Thanks, > Xiaofeng > > On Sep 27, 2022, at 17:48, Mark Adams wrote: > > what equations and discetizations are in A00? > > On Tue, Sep 27, 2022 at 1:45 AM ?? ? wrote: > >> Hi Barry, >> >> A00 is formed from elliptic operator. >> >> I tried GAMG with A00, but it was extremely slow to solve the system with >> field-split preconditioner(I?m not sure I did it with the right way). >> >> Thanks, >> Xiaofeng >> >> On Sep 26, 2022, at 23:11, Barry Smith wrote: >> >> >> What is your A00 operator? ILU is almost never a good choice for large >> scale problems. If it is an elliptic operator that using a PC of gamg may >> work well for the A00 preconditioner instead of ILU. >> >> Barry >> >> For moderate size problems you can use a PC type LU for AOO to help you >> understand the best preconditioner to use for the A11 (the Schur complement >> block), once you have a good preconditioner for the A11 block you would >> then go back and determine a good preconditioner for the A00 block. >> >> On Sep 26, 2022, at 10:08 AM, ?? ? wrote: >> >> Hello Jed, >> >> The saddle point is due to Lagrange multipliers, thus the size of A11 is >> much smaller than A00. >> >> >> >> Best Regards, >> >> Xiaofeng >> >> >> On Sep 26, 2022, at 21:03, Jed Brown wrote: >> >> Lagrange multipliers >> >> >> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Tue Sep 27 09:13:17 2022 From: bsmith at petsc.dev (Barry Smith) Date: Tue, 27 Sep 2022 10:13:17 -0400 Subject: [petsc-users] Solve Linear System with Field Split Preconditioner In-Reply-To: References: <819A69F5-6CB0-494E-80B2-18EB023FD048@hotmail.com> <87pmfihddt.fsf@jedbrown.org> <22A0172A-4AAA-4247-BA35-266555EDA4CF@hotmail.com> <87mtamgues.fsf@jedbrown.org> <22BCBB91-02A2-4440-BE9C-27F06D1E05CF@hotmail.com> Message-ID: <76065A6A-30FF-499E-BEDC-B1FA68C2056F@petsc.dev> Composed preconditioners (like field split) have multiple moving parts and you need to "tune" them for each part separately; you cannot just run the entire preconditioner, get slow convergence on the entire problem and then give up. So step one is to get a good preconditioner for the A00 block, and not worry about the entire fieldsplit yet (once you get good convergence on the A00 block you can tune the Schur complement preconditioner but without good convergence on the A00 block it makes no sense to try to tune the Schur complement preconditioner). You can run with options to monitor convergence of the A00 block and try to tune for that -fieldplit_0_ksp_monitor_true residual -fieldsplit_0_ksp_view -fieldsplit_0_pc_type gamg and control GAMG options with -fieldsplit_0_pc_gamg_* As Mark said, you first need to provide the coordinate and null space information for the A00 block to have any hope of good performance > On Sep 27, 2022, at 1:45 AM, ?? ? wrote: > > Hi Barry, > > A00 is formed from elliptic operator. > > I tried GAMG with A00, but it was extremely slow to solve the system with field-split preconditioner(I?m not sure I did it with the right way). > > Thanks, > Xiaofeng > >> On Sep 26, 2022, at 23:11, Barry Smith > wrote: >> >> >> What is your A00 operator? ILU is almost never a good choice for large scale problems. If it is an elliptic operator that using a PC of gamg may work well for the A00 preconditioner instead of ILU. >> >> Barry >> >> For moderate size problems you can use a PC type LU for AOO to help you understand the best preconditioner to use for the A11 (the Schur complement block), once you have a good preconditioner for the A11 block you would then go back and determine a good preconditioner for the A00 block. >> >>> On Sep 26, 2022, at 10:08 AM, ?? ? > wrote: >>> >>> Hello Jed, >>> >>> The saddle point is due to Lagrange multipliers, thus the size of A11 is much smaller than A00. >>> >>> >>> >>> Best Regards, >>> >>> Xiaofeng >>> >>> >>>> On Sep 26, 2022, at 21:03, Jed Brown > wrote: >>>> >>>> Lagrange multipliers >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tlanyan at hotmail.com Tue Sep 27 09:23:41 2022 From: tlanyan at hotmail.com (=?utf-8?B?5pmT5bOwIOS9lQ==?=) Date: Tue, 27 Sep 2022 14:23:41 +0000 Subject: [petsc-users] Solve Linear System with Field Split Preconditioner In-Reply-To: References: <819A69F5-6CB0-494E-80B2-18EB023FD048@hotmail.com> <87pmfihddt.fsf@jedbrown.org> <22A0172A-4AAA-4247-BA35-266555EDA4CF@hotmail.com> <87mtamgues.fsf@jedbrown.org> <22BCBB91-02A2-4440-BE9C-27F06D1E05CF@hotmail.com> Message-ID: <051B6F44-50D6-4176-B3AB-BF9725494516@hotmail.com> Many thanks for your concrete replies. They are very helpful. Best Wishes, Xiaofeng On Sep 27, 2022, at 20:01, Mark Adams > wrote: Shells can be pretty hard. Start with an easy problem with well-shaped elements and probably best to start with a well-supported structure (eg, not a long cantilever). You should also configure with hypre and try that. I don't know if they deal with shells, but it is a well developed solver. For GAMG, you need to set the coordinates (PCSetCoordinates) which will create the rigid body modes for GAMG so that it can construct the rotational RBMs. This code has not been used for shells in a long time (petsc/src/ksp/pc/impls/gamg/agg.c:138), but it did once work (20 years ago in another code). This code constructs the 6 rigid body modes for 3D shell problems with my ording (6 dof, x,y,z,xx,yy,zz). Or you can construct the RBMs yourself and set them with https://petsc.org/release/docs/manualpages/Mat/MatNullSpaceCreateRigidBody.html There is some discussion here: https://www2.eecs.berkeley.edu/Pubs/TechRpts/2000/CSD-00-1103.pdf Mark On Tue, Sep 27, 2022 at 6:20 AM ?? ? > wrote: Hello, A00 comes from shell structures and discretized by FEM. Thanks, Xiaofeng On Sep 27, 2022, at 17:48, Mark Adams > wrote: what equations and discetizations are in A00? On Tue, Sep 27, 2022 at 1:45 AM ?? ? > wrote: Hi Barry, A00 is formed from elliptic operator. I tried GAMG with A00, but it was extremely slow to solve the system with field-split preconditioner(I?m not sure I did it with the right way). Thanks, Xiaofeng On Sep 26, 2022, at 23:11, Barry Smith > wrote: What is your A00 operator? ILU is almost never a good choice for large scale problems. If it is an elliptic operator that using a PC of gamg may work well for the A00 preconditioner instead of ILU. Barry For moderate size problems you can use a PC type LU for AOO to help you understand the best preconditioner to use for the A11 (the Schur complement block), once you have a good preconditioner for the A11 block you would then go back and determine a good preconditioner for the A00 block. On Sep 26, 2022, at 10:08 AM, ?? ? > wrote: Hello Jed, The saddle point is due to Lagrange multipliers, thus the size of A11 is much smaller than A00. Best Regards, Xiaofeng On Sep 26, 2022, at 21:03, Jed Brown > wrote: Lagrange multipliers -------------- next part -------------- An HTML attachment was scrubbed... URL: From wangzj997 at foxmail.com Tue Sep 27 22:47:36 2022 From: wangzj997 at foxmail.com (=?gb18030?B?d2FuZ3pqOTk3?=) Date: Wed, 28 Sep 2022 11:47:36 +0800 Subject: [petsc-users] PETSc usage issues Message-ID: Dear PETSc development team? Currently, I am learning and trying to use PETSc's KSP to solve large-scale sparse linear systems Ax= b, where A is symmetric positive definite and nonsingular. However, the main diagonal of A contains many 0 items, which leads to the fact that many preconditioners cannot be used when using MPI for multi-process solution, and the number of iterations is large, and the convergence is slow. May I ask how to solve this problem? If it is necessary to make the main diagonal of A all non-zero items, is there any solution in PETSc? I would be grateful if you would reply and answer my question. Best Regards. -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Wed Sep 28 13:43:39 2022 From: bsmith at petsc.dev (Barry Smith) Date: Wed, 28 Sep 2022 14:43:39 -0400 Subject: [petsc-users] PETSc usage issues In-Reply-To: References: Message-ID: <3F04FF71-DC11-4CD4-A5E5-091A2757AC99@petsc.dev> -pc_type fieldsplit -pc_fieldsplit_detect_saddle_point -pc_fieldsplit_type schur Now there will be two additional decisions you need to make how to precondition the A00 block and the Schur complement. For the A00 block the option is -fieldsplit_0_pc_type something where depending on your problem gamg may be a good place to start -fieldsplit_0_ksp_type preonly (likely is the default) For the Schur complement start with just using the default and see how the convergence goes. Use -ksp_view to see all the parts of the preconditioner and -ksp_monitor_true_residual to see how it is coverging. Run with -help | grep fieldsplit to see all possible options and of course consult https://petsc.org/release/docs/manualpages/PC/PCFIELDSPLIT.html Barry > On Sep 27, 2022, at 11:47 PM, wangzj997 wrote: > > Dear PETSc development team? > > Currently, I am learning and trying to use <>PETSc <>'s KSP to solve large-scale sparse linear systems Ax= b, where A is symmetric positive definite and nonsingular. However, the main diagonal of A contains many 0 items, which leads to the fact that many preconditioners cannot be used when using MPI for multi-process solution, and the number of iterations is large, and the convergence is slow. May I ask how to solve this problem? If it is necessary to make the main diagonal of A all non-zero items, is there any solution in PETSc? > > I would be grateful if you would reply and answer my question. > > Best Regards. > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Sep 28 14:00:16 2022 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 28 Sep 2022 15:00:16 -0400 Subject: [petsc-users] PETSc usage issues In-Reply-To: References: Message-ID: On Wed, Sep 28, 2022 at 2:27 PM wangzj997 wrote: > Dear PETSc development team? > > Currently, I am learning and trying to use PETSc's KSP to solve > large-scale sparse linear systems Ax= b, where A is symmetric positive > definite > How do you know it is positive definite? This is unusual when you have 0 on the diagonal. Preconditioners depend sensitively on the matrix properties. Is this a discretization of some operator? Thanks, Matt > and nonsingular. However, the main diagonal of A contains many 0 items, > which leads to the fact that many preconditioners cannot be used when using > MPI for multi-process solution, and the number of iterations is large, and > the convergence is slow. May I ask how to solve this problem? If it is > necessary to make the main diagonal of A all non-zero items, is there any > solution in PETSc? > > I would be grateful if you would reply and answer my question. > > Best Regards. > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From bantingl at myumanitoba.ca Wed Sep 28 19:48:22 2022 From: bantingl at myumanitoba.ca (Lucas Banting) Date: Thu, 29 Sep 2022 00:48:22 +0000 Subject: [petsc-users] Using real build of Hypre with Complex Build of PETSc Message-ID: Hello, My application involves the complex-valued curl-curl equation. I use the complex build of PETSc, but would like to try HYPRE AMS as a real preconditioner, in the same way as this paper describes: https://www.research-collection.ethz.ch/bitstream/handle/20.500.11850/103394/geo2015-0013.1.pdf?sequence=2 My understanding of PETSc is that I can not mix complex and real scalars. Is there a way I can assemble a parallel, real valued matrix, from my complex PETSc matrix and use it with HYPRE? I understand that I can do this by using a real build of PETSc (like MFEM does), but would prefer to keep using the complex build of PETSc as that is what the rest of my application uses. Thanks, Lucas -------------- next part -------------- An HTML attachment was scrubbed... URL: From pierre at joliv.et Thu Sep 29 00:31:58 2022 From: pierre at joliv.et (Pierre Jolivet) Date: Thu, 29 Sep 2022 07:31:58 +0200 Subject: [petsc-users] Using real build of Hypre with Complex Build of PETSc In-Reply-To: References: Message-ID: <3F325D95-6507-498C-AA83-57D51015D1C2@joliv.et> Hello Lucas, Supposedly (I haven?t tried this myself for a while), HYPRE supports real-only complex-stored matrices. So as long as you assemble a complex Mat with a zero imaginary part, e.g., MatRealPart(Pmat), you could feed that to PCHYPRE. If you really want to use a real-only build of HYPRE, there could be a strategy using a PCSHELL, but it would require you to assemble all the HYPRE objects yourself, so I would first try the first strategy. Thanks, Pierre > On 29 Sep 2022, at 2:48 AM, Lucas Banting wrote: > > ? > Hello, > > My application involves the complex-valued curl-curl equation. > I use the complex build of PETSc, but would like to try HYPRE AMS as a real preconditioner, in the same way as this paper describes:https://www.research-collection.ethz.ch/bitstream/handle/20.500.11850/103394/geo2015-0013.1.pdf?sequence=2 > My understanding of PETSc is that I can not mix complex and real scalars. Is there a way I can assemble a parallel, real valued matrix, from my complex PETSc matrix and use it with HYPRE? > > I understand that I can do this by using a real build of PETSc (like MFEM does), but would prefer to keep using the complex build of PETSc as that is what the rest of my application uses. > > Thanks, > Lucas > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mi.mike1021 at gmail.com Fri Sep 30 15:13:50 2022 From: mi.mike1021 at gmail.com (Mike Michell) Date: Fri, 30 Sep 2022 15:13:50 -0500 Subject: [petsc-users] DMLocalToLocal with DMPlex in Fortran Message-ID: Hi, As a follow-up to this email thread, https://www.mail-archive.com/petsc-users at mcs.anl.gov/msg44070.html Are DMLocalToLocalBegin() and DMLocalToLocalEnd() really available for DMPlex with Fortran on the latest version of PETSc (3.17.99 from GitLab)? Matt commented that the Fortran bindings were updated so that those functions must be available in the latest version of PETSc, however, it seems still they are not working from my test with DMPlex in Fortran. Can anyone provide some comments? Probably I am missing some mandatory header file? Currently, I have headers; #include "petsc/finclude/petscvec.h" #include "petsc/finclude/petscdmplex.h" #include "petsc/finclude/petscdmlabel.h" #include "petsc/finclude/petscdm.h" Thanks, Mike -------------- next part -------------- An HTML attachment was scrubbed... URL: