[petsc-users] Problem coupling Petsc into OpenFOAM
Mark Olesen
Mark.Olesen at esi-group.com
Thu Jun 25 04:53:20 CDT 2020
Late followup, but people may want to know that a PETSc/OpenFOAM
interface is included in the pending OpenFOAM-v2006 release as a git
submodule:
https://develop.openfoam.com/modules/external-solver
Cheers,
/mark
On 2019-05-24 12:06, Matthew Knepley via petsc-users wrote:
> On Thu, May 23, 2019 at 10:41 PM Vu Q. Do via petsc-users
> <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>> wrote:
>
> Hi all,
>
> Thanks for your previous suggestion, I have been able to
> successfully link Petsc to OpenFOAM. I have written a simple
> interface and it works quite well in serial mode, but cannot run in
> parallel. I have been thinking about this problem for weeks but
> couldn't solve it. So I think maybe you could give me some idea. I
> describe my problem below.
>
> My interface is just a class named "petscSolver/"/, which is used to
> convert an openfoam's matrix or blocked matrix to Petsc Mat, then
> solve the matrix using Petsc's solver.
> To use Petsc, an Openfoam's solver need to be recompiled after
> adding the following lines to make file:
>
> EXE_INC = \
> ...
> -I$(LIB_SRC)/petscSolver \
> -I$(PETSC_DIR)/include \
> -I$(PETSC_DIR)/$(PETSC_ARCH)/include
> EXE_LIBS = \
> ...
> -L$(PETSC_DIR)/$(PETSC_ARCH)/lib -lpetsc
>
> To run an openfoam's case in parallel, first I need to discretize
> the domain into subdomains (e.g. 2 subdomains ), then use the
> following command:
>
> mpirun -np 2 mySolver -parallel
>
> (where mpirun is literally mpiexec) The simulation crashed even
> before doing anything and the error message is as in the attached image.
>
>
> Run in the debugger and see where it is crashing. Its possible to screw
> up the MPI linking here, so that you
> link OPENFOAM with one MPI and PETSc with another, or you call MPIInit()
> after you call PetscInitialize(), etc.
>
> Thanks,
>
> Matt
>
> I have tested and realized that the solver can run in parallel as
> normal by removing the two lines:
> -I$(PETSC_DIR)/include \
> -I$(PETSC_DIR)/$(PETSC_ARCH)/include
> But then it is clearly no longer linked to Petsc.
>
> I would appreciate any suggestion.
>
> Screenshot from 2019-05-24 09-22-17.png
>
>
> On Thu, Apr 11, 2019 at 1:37 PM Smith, Barry F. <bsmith at mcs.anl.gov
> <mailto:bsmith at mcs.anl.gov>> wrote:
>
>
> Mark,
>
> Thanks for the clarifying email. My google searches didn't
> locate the rheoTool you mention nor "a PRACE project running via
> CINECA (Bologna)".
>
> It would be nice if someday OpenFOAM had (either directly
> or somehow with the modules directory) an interface to the PETSc
> solvers. This would allow the use of a variety of other solvers
> including hypre BoomerAMG, SuperLU_Dist, MUMPS, and even the use
> of PETSc/ViennaCL GPU based solvers automatically from OpenFOAM.
> Unfortunately the PETSc group doesn't have the resources or
> expertise to develop and support such an interface ourselves. We
> would, of course, try to answer emails about PETSc usage and
> bugs for such an interface.
>
> Barry
>
> If OpenFOAM did have such an interface one thing we could
> provide is the CI infrastructure for tracking changes to PETSc
> that may effect OpenFOAM. For example we could automatically
> build OpenFOAM each day with the latest master of PETSc thus
> immediately detecting changes that effect the interface.
>
>
>
>
> > On Apr 10, 2019, at 4:55 PM, Mark Olesen
> <Mark.Olesen at esi-group.com <mailto:Mark.Olesen at esi-group.com>>
> wrote:
> >
> > The paper that Barry mentioned gives some generalities, but
> probably
> > won't help much. There are some PETSc/OpenFOAM interfaces in
> rheoTool
> > that are probably much more helpful.
> >
> > As Barry also rightly noted, there are some config files in
> the OpenFOAM
> > tree that were put in some time ago for helping with setting
> up PETSc
> > and OpenFOAM. Assuming that you have set the appropriate
> values in the
> > etc/config.sh/petsc <http://config.sh/petsc> file you will be
> able to use those when using wmake.
> > For running you will still need to ensure that the
> LD_LIBARY_PATH is set
> > correctly. For example, what some build scripts exhibit:
> >
> > wmake(petsc) :
> > ==> Before running, verify that PETSc libraries can be found
> >
> > Enable in the OpenFOAM etc/bashrc, define manually or try
> with the
> > following (POSIX shell):
> >
> > eval $(foamEtcFile -sh -config petsc -- -force)
> >
> > ==
> >
> >
> > There is currently a PRACE project running via CINECA
> (Bologna) with
> > binding in PETSc as a runtime selectable linear solver in
> OpenFOAM. This
> > is still at the stage of early testing and performance
> benchmarking.
> >
> > Cheers,
> > /mark
> >
> > On 4/10/19 6:37 PM, Smith, Barry F. via petsc-users wrote:
> >>
> >> We don't know much about OpenFoam but
> >>
> >> 1) if I do a
> >>
> >> git grep -i petsc
> >>
> >> in the
> https://develop.openfoam.com/Development/OpenFOAM-plus.git
> repository I see various configuration files specifically for PETSc.
> >>
> >> etc/config.csh/petsc etc/config.sh/petsc
> <http://config.sh/petsc> wmake/scripts/have_petsc
> >>
> >> so it appears that OpenFOAM has the tools to be linked
> against PETSc (to me the documentation on how to use them is
> rather terse). Are
> >> you using these? If you have trouble with them perhaps
> you can ask the OpenFOAM user community how to use them.
> >>
> >>
> >> 2) if you are editing the Make/options file directly you
> can try changing
> >>
> >> -L$(PETSC_DIR)/$(PETSC_ARCH)/lib -lpetsc
> >>
> >> to
> >>
> >> -Wl,-rpath,$(PETSC_DIR)/$(PETSC_ARCH)/lib
> -L$(PETSC_DIR)/$(PETSC_ARCH)/lib -lpetsc
> >>
> >>
> >>
> >> Note also that simply including petsc.h into the OpenFoam
> source code and linking against -lpetsc will not immediately
> allow calling the PETSc solvers from OpenFOAM. One needs to
> write all the interface code that sets up and calls the PETSc
> solvers from OpenFOAM. There is a paper
> https://www.researchgate.net/publication/319045499_Insertion_of_PETSc_in_the_OpenFOAM_Framework
> that describes at an abstract level how they wrote code that
> calls the PETSc solvers from OpenFOAM but the source code that
> actually does the work does not appear to be available.
> >>
> >> Note that PETSc is now at version 3.11 we recommend
> working with that version (unless you already have a lot of code
> that calls PETSc written with a previous version of PETSc, for
> that we recommend first upgrading to petsc 3.11 and then
> continuing to add code).
> >>
> >> Barry
> >>
> >>
> >>
> >>
> >>
> >>> On Apr 10, 2019, at 8:23 AM, Balay, Satish via petsc-users
> <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>> wrote:
> >>>
> >>> Runtime error? You might have to add the path to
> $PETSC_ARCH/lib in LD_LIBRARY_PATH env variable
> >>> or - to your link command. If linux/gcc - the linker option
> is -Wl,-rpath,$PETSC_ARCH/lib
> >>>
> >>> If not - send detail logs.
> >>>
> >>> Satish
> >>>
> >>> On Wed, 10 Apr 2019, Vu Do Quoc via petsc-users wrote:
> >>>
> >>>> Hi all,
> >>>>
> >>>> I am trying to insert Petsc to OpenFOAM opensource software.
> >>>> I have been successfully compiling Petsc with an available
> solver in
> >>>> OpenFOAM by linking it with the shared library
> libpetsc.so. However, when I
> >>>> call the solver to run a test case, I got an error saying
> that:
> >>>> "libpetsc.so cannot be found", even though the library
> still exists in the
> >>>> $PETSC_ARCH/lib folder.
> >>>>
> >>>> I have been struggling for weeks but still, have not been
> able to figure it
> >>>> out. Therefore I would be very grateful for any suggestion
> to solve this
> >>>> problem.
> >>>>
> >>>> Thanks in advance for your time,
> >>>>
> >>>> Best regards,
> >>>>
> >>>> Vu Do
> >>>>
>
>
>
> --
> *Vu Q. Do
> *------------------------------------------------
> /Student of Aeronautical Engineering/
> Programme de Formation d'Ingénieurs d'Excellence au Vietnam /- PFIEV/
> School of Transportation Engineering/
> /
> Hanoi University of Science and Technology
> 01 Dai Co Viet Avenue, Hanoi, Vietnam
> E-mail: vu.doquochust at gmail.com <mailto:vu.doquochust at gmail.com>
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which
> their experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
--
Dr Mark OLESEN
Principal Engineer, ESI-OpenCFD
Engineering System International GmbH | Einsteinring 24 | 85609 Munich
Mob. +49 171 9710 149
www.openfoam.com | www.esi-group.com | mark.olesen at esi-group.com
More information about the petsc-users
mailing list