[petsc-users] Problem coupling Petsc into OpenFOAM
Barry Smith
bsmith at petsc.dev
Thu Jun 25 20:54:44 CDT 2020
Mark,
Thanks for letting us know. We have added it to the PETSc package list. Should any issues come up we'll be glad to help support it.
Barry
> On Jun 25, 2020, at 4:53 AM, Mark Olesen <Mark.Olesen at esi-group.com> wrote:
>
> Late followup, but people may want to know that a PETSc/OpenFOAM interface is included in the pending OpenFOAM-v2006 release as a git submodule:
>
> https://develop.openfoam.com/modules/external-solver
>
> Cheers,
> /mark
>
> On 2019-05-24 12:06, Matthew Knepley via petsc-users wrote:
>> On Thu, May 23, 2019 at 10:41 PM Vu Q. Do via petsc-users <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>> wrote:
>> Hi all,
>> Thanks for your previous suggestion, I have been able to
>> successfully link Petsc to OpenFOAM. I have written a simple
>> interface and it works quite well in serial mode, but cannot run in
>> parallel. I have been thinking about this problem for weeks but
>> couldn't solve it. So I think maybe you could give me some idea. I
>> describe my problem below.
>> My interface is just a class named "petscSolver/"/, which is used to
>> convert an openfoam's matrix or blocked matrix to Petsc Mat, then
>> solve the matrix using Petsc's solver.
>> To use Petsc, an Openfoam's solver need to be recompiled after
>> adding the following lines to make file:
>> EXE_INC = \
>> ...
>> -I$(LIB_SRC)/petscSolver \
>> -I$(PETSC_DIR)/include \
>> -I$(PETSC_DIR)/$(PETSC_ARCH)/include
>> EXE_LIBS = \
>> ...
>> -L$(PETSC_DIR)/$(PETSC_ARCH)/lib -lpetsc
>> To run an openfoam's case in parallel, first I need to discretize
>> the domain into subdomains (e.g. 2 subdomains ), then use the
>> following command:
>> mpirun -np 2 mySolver -parallel
>> (where mpirun is literally mpiexec) The simulation crashed even
>> before doing anything and the error message is as in the attached image.
>> Run in the debugger and see where it is crashing. Its possible to screw up the MPI linking here, so that you
>> link OPENFOAM with one MPI and PETSc with another, or you call MPIInit() after you call PetscInitialize(), etc.
>> Thanks,
>> Matt
>> I have tested and realized that the solver can run in parallel as
>> normal by removing the two lines:
>> -I$(PETSC_DIR)/include \
>> -I$(PETSC_DIR)/$(PETSC_ARCH)/include
>> But then it is clearly no longer linked to Petsc.
>> I would appreciate any suggestion.
>> Screenshot from 2019-05-24 09-22-17.png
>> On Thu, Apr 11, 2019 at 1:37 PM Smith, Barry F. <bsmith at mcs.anl.gov
>> <mailto:bsmith at mcs.anl.gov>> wrote:
>> Mark,
>> Thanks for the clarifying email. My google searches didn't
>> locate the rheoTool you mention nor "a PRACE project running via
>> CINECA (Bologna)".
>> It would be nice if someday OpenFOAM had (either directly
>> or somehow with the modules directory) an interface to the PETSc
>> solvers. This would allow the use of a variety of other solvers
>> including hypre BoomerAMG, SuperLU_Dist, MUMPS, and even the use
>> of PETSc/ViennaCL GPU based solvers automatically from OpenFOAM.
>> Unfortunately the PETSc group doesn't have the resources or
>> expertise to develop and support such an interface ourselves. We
>> would, of course, try to answer emails about PETSc usage and
>> bugs for such an interface.
>> Barry
>> If OpenFOAM did have such an interface one thing we could
>> provide is the CI infrastructure for tracking changes to PETSc
>> that may effect OpenFOAM. For example we could automatically
>> build OpenFOAM each day with the latest master of PETSc thus
>> immediately detecting changes that effect the interface.
>> > On Apr 10, 2019, at 4:55 PM, Mark Olesen
>> <Mark.Olesen at esi-group.com <mailto:Mark.Olesen at esi-group.com>>
>> wrote:
>> >
>> > The paper that Barry mentioned gives some generalities, but
>> probably
>> > won't help much. There are some PETSc/OpenFOAM interfaces in
>> rheoTool
>> > that are probably much more helpful.
>> >
>> > As Barry also rightly noted, there are some config files in
>> the OpenFOAM
>> > tree that were put in some time ago for helping with setting
>> up PETSc
>> > and OpenFOAM. Assuming that you have set the appropriate
>> values in the
>> > etc/config.sh/petsc <http://config.sh/petsc> file you will be
>> able to use those when using wmake.
>> > For running you will still need to ensure that the
>> LD_LIBARY_PATH is set
>> > correctly. For example, what some build scripts exhibit:
>> >
>> > wmake(petsc) :
>> > ==> Before running, verify that PETSc libraries can be found
>> >
>> > Enable in the OpenFOAM etc/bashrc, define manually or try
>> with the
>> > following (POSIX shell):
>> >
>> > eval $(foamEtcFile -sh -config petsc -- -force)
>> >
>> > ==
>> >
>> >
>> > There is currently a PRACE project running via CINECA
>> (Bologna) with
>> > binding in PETSc as a runtime selectable linear solver in
>> OpenFOAM. This
>> > is still at the stage of early testing and performance
>> benchmarking.
>> >
>> > Cheers,
>> > /mark
>> >
>> > On 4/10/19 6:37 PM, Smith, Barry F. via petsc-users wrote:
>> >>
>> >> We don't know much about OpenFoam but
>> >>
>> >> 1) if I do a
>> >>
>> >> git grep -i petsc
>> >>
>> >> in the
>> https://develop.openfoam.com/Development/OpenFOAM-plus.git
>> repository I see various configuration files specifically for PETSc.
>> >>
>> >> etc/config.csh/petsc etc/config.sh/petsc
>> <http://config.sh/petsc> wmake/scripts/have_petsc
>> >>
>> >> so it appears that OpenFOAM has the tools to be linked
>> against PETSc (to me the documentation on how to use them is
>> rather terse). Are
>> >> you using these? If you have trouble with them perhaps
>> you can ask the OpenFOAM user community how to use them.
>> >>
>> >>
>> >> 2) if you are editing the Make/options file directly you
>> can try changing
>> >>
>> >> -L$(PETSC_DIR)/$(PETSC_ARCH)/lib -lpetsc
>> >>
>> >> to
>> >>
>> >> -Wl,-rpath,$(PETSC_DIR)/$(PETSC_ARCH)/lib -L$(PETSC_DIR)/$(PETSC_ARCH)/lib -lpetsc
>> >>
>> >>
>> >>
>> >> Note also that simply including petsc.h into the OpenFoam
>> source code and linking against -lpetsc will not immediately
>> allow calling the PETSc solvers from OpenFOAM. One needs to
>> write all the interface code that sets up and calls the PETSc
>> solvers from OpenFOAM. There is a paper
>> https://www.researchgate.net/publication/319045499_Insertion_of_PETSc_in_the_OpenFOAM_Framework
>> that describes at an abstract level how they wrote code that
>> calls the PETSc solvers from OpenFOAM but the source code that
>> actually does the work does not appear to be available.
>> >>
>> >> Note that PETSc is now at version 3.11 we recommend
>> working with that version (unless you already have a lot of code
>> that calls PETSc written with a previous version of PETSc, for
>> that we recommend first upgrading to petsc 3.11 and then
>> continuing to add code).
>> >>
>> >> Barry
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>> On Apr 10, 2019, at 8:23 AM, Balay, Satish via petsc-users
>> <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>> wrote:
>> >>>
>> >>> Runtime error? You might have to add the path to
>> $PETSC_ARCH/lib in LD_LIBRARY_PATH env variable
>> >>> or - to your link command. If linux/gcc - the linker option
>> is -Wl,-rpath,$PETSC_ARCH/lib
>> >>>
>> >>> If not - send detail logs.
>> >>>
>> >>> Satish
>> >>>
>> >>> On Wed, 10 Apr 2019, Vu Do Quoc via petsc-users wrote:
>> >>>
>> >>>> Hi all,
>> >>>>
>> >>>> I am trying to insert Petsc to OpenFOAM opensource software.
>> >>>> I have been successfully compiling Petsc with an available
>> solver in
>> >>>> OpenFOAM by linking it with the shared library
>> libpetsc.so. However, when I
>> >>>> call the solver to run a test case, I got an error saying
>> that:
>> >>>> "libpetsc.so cannot be found", even though the library
>> still exists in the
>> >>>> $PETSC_ARCH/lib folder.
>> >>>>
>> >>>> I have been struggling for weeks but still, have not been
>> able to figure it
>> >>>> out. Therefore I would be very grateful for any suggestion
>> to solve this
>> >>>> problem.
>> >>>>
>> >>>> Thanks in advance for your time,
>> >>>>
>> >>>> Best regards,
>> >>>>
>> >>>> Vu Do
>> >>>>
>> -- *Vu Q. Do
>> *------------------------------------------------
>> /Student of Aeronautical Engineering/
>> Programme de Formation d'Ingénieurs d'Excellence au Vietnam /- PFIEV/
>> School of Transportation Engineering/
>> /
>> Hanoi University of Science and Technology
>> 01 Dai Co Viet Avenue, Hanoi, Vietnam
>> E-mail: vu.doquochust at gmail.com <mailto:vu.doquochust at gmail.com>
>> --
>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>> -- Norbert Wiener
>> https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
>
> --
> Dr Mark OLESEN
> Principal Engineer, ESI-OpenCFD
> Engineering System International GmbH | Einsteinring 24 | 85609 Munich
> Mob. +49 171 9710 149
> www.openfoam.com | www.esi-group.com | mark.olesen at esi-group.com
More information about the petsc-users
mailing list