[petsc-users] Problem coupling Petsc into OpenFOAM

[Vu Q. Do]

 Hi all,

Thanks for your previous suggestion, I have been able to successfully link
Petsc to OpenFOAM. I have written a simple interface and it works quite
well in serial mode, but cannot run in parallel. I have been thinking about
this problem for weeks but couldn't solve it. So I think maybe you could
give me some idea. I describe my problem below.

My interface is just a class named "petscSolver*"*, which is used to
convert an openfoam's matrix or blocked matrix to Petsc Mat, then solve the
matrix using Petsc's solver.
To use Petsc, an Openfoam's solver need to be recompiled after adding the
following lines to make file:

EXE_INC = \
...
  -I$(LIB_SRC)/petscSolver \
  -I$(PETSC_DIR)/include \
  -I$(PETSC_DIR)/$(PETSC_ARCH)/include
EXE_LIBS = \
...
  -L$(PETSC_DIR)/$(PETSC_ARCH)/lib -lpetsc

To run an openfoam's case in parallel, first I need to discretize the
domain into subdomains (e.g. 2 subdomains ), then use the following command:

mpirun -np 2 mySolver -parallel

(where mpirun is literally mpiexec) The simulation crashed even before
doing anything and the error message is as in the attached image.
I have tested and realized that the solver can run in parallel as normal by
removing the two lines:
  -I$(PETSC_DIR)/include \
  -I$(PETSC_DIR)/$(PETSC_ARCH)/include
But then it is clearly no longer linked to Petsc.

I would appreciate any suggestion.

[image: Screenshot from 2019-05-24 09-22-17.png]


On Thu, Apr 11, 2019 at 1:37 PM Smith, Barry F. <bsmith at mcs.anl.gov> wrote:

>
>   Mark,
>
>     Thanks for the clarifying email. My google searches didn't locate the
> rheoTool you mention nor "a PRACE project running via CINECA (Bologna)".
>
>     It would be nice if someday OpenFOAM had (either directly or somehow
> with the modules directory) an interface to the PETSc solvers. This would
> allow the use of a variety of other solvers including hypre BoomerAMG,
> SuperLU_Dist, MUMPS, and even the use of PETSc/ViennaCL GPU based solvers
> automatically from OpenFOAM. Unfortunately the PETSc group doesn't have the
> resources or expertise to develop and support such an interface ourselves.
> We would, of course, try to answer emails about PETSc usage and bugs for
> such an interface.
>
>    Barry
>
>   If OpenFOAM did have such an interface one thing we could provide is the
> CI infrastructure for tracking changes to PETSc that may effect OpenFOAM.
> For example we could automatically build OpenFOAM each day with the latest
> master of PETSc thus immediately detecting changes that effect the
> interface.
>
>
>
>
> > On Apr 10, 2019, at 4:55 PM, Mark Olesen <Mark.Olesen at esi-group.com>
> wrote:
> >
> > The paper that Barry mentioned gives some generalities, but probably
> > won't help much. There are some PETSc/OpenFOAM interfaces in rheoTool
> > that are probably much more helpful.
> >
> > As Barry also rightly noted, there are some config files in the OpenFOAM
> > tree that were put in some time ago for helping with setting up PETSc
> > and OpenFOAM. Assuming that you have set the appropriate values in the
> > etc/config.sh/petsc file you will be able to use those when using
> wmake.
> > For running you will still need to ensure that the LD_LIBARY_PATH is set
> > correctly. For example, what some build scripts exhibit:
> >
> > wmake(petsc)  :
> > ==> Before running, verify that PETSc libraries can be found
> >
> > Enable in the OpenFOAM etc/bashrc, define manually or try with the
> > following (POSIX shell):
> >
> >     eval $(foamEtcFile -sh -config petsc -- -force)
> >
> > ==
> >
> >
> > There is currently a PRACE project running via CINECA (Bologna) with
> > binding in PETSc as a runtime selectable linear solver in OpenFOAM. This
> > is still at the stage of early testing and performance benchmarking.
> >
> > Cheers,
> > /mark
> >
> > On 4/10/19 6:37 PM, Smith, Barry F. via petsc-users wrote:
> >>
> >>   We don't know much about OpenFoam but
> >>
> >> 1)  if I do a
> >>
> >>   git grep -i petsc
> >>
> >>     in the https://develop.openfoam.com/Development/OpenFOAM-plus.git
> repository I see various configuration files specifically for PETSc.
> >>
> >> etc/config.csh/petsc  etc/config.sh/petsc wmake/scripts/have_petsc
> >>
> >>     so it appears that OpenFOAM has the tools to be linked against
> PETSc (to me the documentation on how to use them is rather terse). Are
> >>     you using these?  If you have trouble with them perhaps you can ask
> the OpenFOAM user community how to use them.
> >>
> >>
> >>   2)  if you are editing the Make/options file directly you can try
> changing
> >>
> >>      -L$(PETSC_DIR)/$(PETSC_ARCH)/lib -lpetsc
> >>
> >>      to
> >>
> >>      -Wl,-rpath,$(PETSC_DIR)/$(PETSC_ARCH)/lib
>  -L$(PETSC_DIR)/$(PETSC_ARCH)/lib -lpetsc
> >>
> >>
> >>
> >>    Note also that simply including petsc.h into the OpenFoam source
> code and linking against -lpetsc will not immediately allow calling the
> PETSc solvers from OpenFOAM. One needs to write all the interface code that
> sets up and calls the PETSc solvers from OpenFOAM. There is a paper
> https://www.researchgate.net/publication/319045499_Insertion_of_PETSc_in_the_OpenFOAM_Framework
> that describes at an abstract level how they wrote code that calls the
> PETSc solvers from OpenFOAM but the source code that actually does the work
> does not appear to be available.
> >>
> >>    Note that PETSc is now at version 3.11 we recommend working with
> that version (unless you already have a lot of code that calls PETSc
> written with a previous version of PETSc, for that we recommend first
> upgrading to petsc 3.11 and then continuing to add code).
> >>
> >>    Barry
> >>
> >>
> >>
> >>
> >>
> >>> On Apr 10, 2019, at 8:23 AM, Balay, Satish via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
> >>>
> >>> Runtime error? You might have to add the path to $PETSC_ARCH/lib in
> LD_LIBRARY_PATH env variable
> >>> or - to your link command. If linux/gcc - the linker option is
> -Wl,-rpath,$PETSC_ARCH/lib
> >>>
> >>> If not - send detail logs.
> >>>
> >>> Satish
> >>>
> >>> On Wed, 10 Apr 2019, Vu Do Quoc via petsc-users wrote:
> >>>
> >>>> Hi all,
> >>>>
> >>>> I am trying to insert Petsc to OpenFOAM opensource software.
> >>>> I have been successfully compiling Petsc with an available solver in
> >>>> OpenFOAM by linking it with the shared library libpetsc.so. However,
> when I
> >>>> call the solver to run a test case, I got an error saying that:
> >>>> "libpetsc.so cannot be found", even though the library still exists
> in the
> >>>> $PETSC_ARCH/lib folder.
> >>>>
> >>>> I have been struggling for weeks but still, have not been able to
> figure it
> >>>> out. Therefore I would be very grateful for any suggestion to solve
> this
> >>>> problem.
> >>>>
> >>>> Thanks in advance for your time,
> >>>>
> >>>> Best regards,
> >>>>
> >>>> Vu Do
> >>>>
>
>

-- 

*Vu Q. Do*------------------------------------------------
*Student of Aeronautical Engineering*
Programme de Formation d'Ingénieurs d'Excellence au Vietnam *- PFIEV*
School of Transportation Engineering
Hanoi University of Science and Technology
01 Dai Co Viet Avenue, Hanoi, Vietnam
E-mail: vu.doquochust at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190524/7be0fcb0/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Screenshot from 2019-05-24 09-22-17.png
Type: image/png
Size: 18904 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190524/7be0fcb0/attachment-0001.png>