[petsc-users] multiple definition of `main' with intel compilers

Alfredo Jaramillo ajaramillopalma at gmail.com
Fri May 22 10:57:05 CDT 2020 

Hello Satish and PETSc developers,

I'm sending this email in case somebody else is having the same problem
with the intel compilers.
In my personal computer (with Fedora 32, gcc 10). When I tried to execute
the resulting binary there appears the error message:

build/linux_icc/partrans_linux_icc_opt:
/opt/intel/clck/2019.8/lib/intel64/libstdc++.so.6: version `CXXABI_1.3.9'
not found (required by build/linux_icc/partrans_linux_icc_opt)

a workaround that worked for me was to delete the link
/opt/intel/clck/2019.8/lib/intel64/libstdc++.so.6 that was doing:

libstdc++.so.6 -> libstdc++.so.6.0.20 (in the folder
/opt/intel/clck/2019.8/lib/intel64/)

and replace it by a link to the default c++ library of my system:

sudo ln -s /usr/lib64/libstdc++.so.6.0.20 libstdc++.so.6

the code is running so such replacement seems to work fine.

best wishes
Alfredo

On Thu, May 21, 2020 at 10:25 PM Alfredo Jaramillo <
ajaramillopalma at gmail.com> wrote:

> I just read at the end of the section "compilers" here
> https://www.mcs.anl.gov/petsc/documentation/installation.html that one
> can indicate these libraries by doing
> ./configure --LIBS='-Bstatic -lifcore -Bdynamic'
>
> thank you very much for your help, Satish
>
> alfredo
>
> On Thu, May 21, 2020 at 10:08 PM Satish Balay <balay at mcs.anl.gov> wrote:
>
>> Configure attempts to automatically determine the language compatible
>> libraries by running 'icc -v' and 'ifort -v' - and uses what it says. [this
>> list can be different based on compiler options used]
>>
>> And PETSc examples work fine - so its not clear why fcoremt_pic is
>> causing issues with your app build.
>>
>> There are some differences in your build vs examples - so that could be
>> one reason. So its best to build PETSc and your app with the exact same
>> compilers and options and see if that works.
>>
>> Alternative is to specify to PETSc configure the compatibility libraries
>> so that it doesn't try to figure this out. [and add extra un-needed
>> libraries]
>>
>> With Intel compilers - its likely the additional configure option is:
>>
>> LIBS="-Bstatic -lifcore -Bdynamic"
>>
>> If this works - configure won't run 'ifort -v' and grab fcoremt_pic etc..
>>
>> Satish
>>
>> On Thu, 21 May 2020, Alfredo Jaramillo wrote:
>>
>> > In fact, removing -lifcore_pic from
>> PETSC_DIR/PETSC_ARCH/lib/petscvariables
>> > solved the problem (this was the first I traied and it worked), it
>> compiles
>> > and the program runs fine.
>> >
>> > Also, -lpetsc was being listed some times when compiling .c to .o, I
>> fixed
>> > that in the scripts.
>> >
>> > Now, I'm a bit lost about when fcoremt_pic is being linked or if it is
>> > necessary at all?
>> >
>> > thank you very much!
>> > alfredo
>> >
>> > On Thu, May 21, 2020 at 7:21 PM Satish Balay <balay at mcs.anl.gov> wrote:
>> >
>> > > For one - PETSc is built without -qopenmp flag - but your makefile is
>> > > using it. Intel compiler can link in with
>> > > different [incompatible] compiler libraries when some flags change
>> this
>> > > way [causing conflict].
>> > >
>> > > However - the issue could be:
>> > >
>> > > your makefile is listing PETSC_LIB [or however you are accessing petsc
>> > > info from petsc makefiles] redundantly.
>> > >
>> > > i.e
>> > >
>> > >  - its listing -lpetsc  etc when compiling .c to .o files [this
>> should not
>> > > happen]
>> > >  - its listing -lpetsc  etc twice [-lifcore_pic is listed once
>> though] -
>> > > don't know if this is the reason for the problem.
>> > >
>> > > You can try [manually] removing -lifcore_pic from
>> > > PETSC_DIR/PETSC_ARCH/lib/petscvariables - and see if this problem
>> goes away
>> > >
>> > > Satish
>> > >
>> > > On Thu, 21 May 2020, Alfredo Jaramillo wrote:
>> > >
>> > > > here is the output:
>> > > >
>> > > > alfredo.jaramillo at sdumont11 tutorials]$ make ex19
>> > > > mpiicc -fPIC -O3 -march=native -mtune=native  -fPIC -O3
>> -march=native
>> > > > -mtune=native  -I/scratch/simulreserv/softwares/petsc-3.13.0/include
>> > > >
>> > >
>> -I/scratch/simulreserv/softwares/petsc-3.13.0/x64-O3-3.13-intel2016-64/include
>> > > > -I/scratch/simulreserv/softwares/valgrind-3.15.0/include
>> > > >  -I/scratch/app/zlib/1.2.11/include    ex19.c
>> > > >
>> > >
>> -Wl,-rpath,/scratch/simulreserv/softwares/petsc-3.13.0/x64-O3-3.13-intel2016-64/lib
>> > > >
>> > >
>> -L/scratch/simulreserv/softwares/petsc-3.13.0/x64-O3-3.13-intel2016-64/lib
>> > > >
>> > >
>> -Wl,-rpath,/scratch/simulreserv/softwares/petsc-3.13.0/x64-O3-3.13-intel2016-64/lib
>> > > >
>> > >
>> -L/scratch/simulreserv/softwares/petsc-3.13.0/x64-O3-3.13-intel2016-64/lib
>> > > >
>> > >
>> -Wl,-rpath,/opt/intel/parallel_studio_xe_2016_update2/compilers_and_libraries_2016.2.181/linux/mpi/intel64/lib/release_mt
>> > > >
>> > >
>> -L/opt/intel/parallel_studio_xe_2016_update2/compilers_and_libraries_2016.2.181/linux/mpi/intel64/lib/release_mt
>> > > >
>> > >
>> -Wl,-rpath,/opt/intel/parallel_studio_xe_2016_update2/compilers_and_libraries_2016.2.181/linux/mpi/intel64/lib
>> > > >
>> > >
>> -L/opt/intel/parallel_studio_xe_2016_update2/compilers_and_libraries_2016.2.181/linux/mpi/intel64/lib
>> > > > -Wl,-rpath,/opt/intel/parallel_studio_xe_2016_update2/clck/
>> > > > 3.1.2.006/lib/intel64
>> -L/opt/intel/parallel_studio_xe_2016_update2/clck/
>> > > > 3.1.2.006/lib/intel64
>> > > >
>> > >
>> -Wl,-rpath,/opt/intel/parallel_studio_xe_2016_update2/compilers_and_libraries_2016.2.181/linux/ipp/lib/intel64
>> > > >
>> > >
>> -L/opt/intel/parallel_studio_xe_2016_update2/compilers_and_libraries_2016.2.181/linux/ipp/lib/intel64
>> > > >
>> > >
>> -Wl,-rpath,/opt/intel/parallel_studio_xe_2016_update2/compilers_and_libraries_2016.2.181/linux/compiler/lib/intel64
>> > > >
>> > >
>> -L/opt/intel/parallel_studio_xe_2016_update2/compilers_and_libraries_2016.2.181/linux/compiler/lib/intel64
>> > > >
>> > >
>> -Wl,-rpath,/opt/intel/parallel_studio_xe_2016_update2/compilers_and_libraries_2016.2.181/linux/mkl/lib/intel64
>> > > >
>> > >
>> -L/opt/intel/parallel_studio_xe_2016_update2/compilers_and_libraries_2016.2.181/linux/mkl/lib/intel64
>> > > >
>> > >
>> -Wl,-rpath,/opt/intel/parallel_studio_xe_2016_update2/compilers_and_libraries_2016.2.181/linux/tbb/lib/intel64/gcc4.4
>> > > >
>> > >
>> -L/opt/intel/parallel_studio_xe_2016_update2/compilers_and_libraries_2016.2.181/linux/tbb/lib/intel64/gcc4.4
>> > > >
>> > >
>> -Wl,-rpath,/opt/intel/parallel_studio_xe_2016_update2/compilers_and_libraries_2016.2.181/linux/daal/lib/intel64_lin
>> > > >
>> > >
>> -L/opt/intel/parallel_studio_xe_2016_update2/compilers_and_libraries_2016.2.181/linux/daal/lib/intel64_lin
>> > > >
>> > >
>> -Wl,-rpath,/opt/intel/parallel_studio_xe_2016_update2/compilers_and_libraries_2016.2.181/linux/tbb/lib/intel64_lin/gcc4.4
>> > > >
>> > >
>> -L/opt/intel/parallel_studio_xe_2016_update2/compilers_and_libraries_2016.2.181/linux/tbb/lib/intel64_lin/gcc4.4
>> > > >
>> > >
>> -Wl,-rpath,/opt/intel/parallel_studio_xe_2016_update2/compilers_and_libraries_2016.2.181/linux/compiler/lib/intel64_lin
>> > > >
>> > >
>> -L/opt/intel/parallel_studio_xe_2016_update2/compilers_and_libraries_2016.2.181/linux/compiler/lib/intel64_lin
>> > > > -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/4.8.5
>> > > > -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5
>> > > > -Wl,-rpath,/opt/intel/mpi-rt/5.1/intel64/lib/release_mt
>> > > > -Wl,-rpath,/opt/intel/mpi-rt/5.1/intel64/lib -lpetsc -lHYPRE
>> -lcmumps
>> > > > -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lscalapack
>> -lopenblas
>> > > > -lstdc++ -ldl -lmpifort -lmpi -lmpigi -lrt -lpthread -lifport
>> > > -lifcore_pic
>> > > > -limf -lsvml -lm -lipgo -lirc -lgcc_s -lirc_s -lquadmath -lstdc++
>> -ldl -o
>> > > > ex19
>> > > > [alfredo.jaramillo at sdumont11 tutorials]$ make ex5f
>> > > > mpiifort -fPIC -O3 -march=native -mtune=native
>> > > > -I/scratch/simulreserv/softwares/petsc-3.13.0/include
>> > > >
>> > >
>> -I/scratch/simulreserv/softwares/petsc-3.13.0/x64-O3-3.13-intel2016-64/include
>> > > > -I/scratch/simulreserv/softwares/valgrind-3.15.0/include
>> ex5f.F90
>> > > >
>> > >
>> -Wl,-rpath,/scratch/simulreserv/softwares/petsc-3.13.0/x64-O3-3.13-intel2016-64/lib
>> > > >
>> > >
>> -L/scratch/simulreserv/softwares/petsc-3.13.0/x64-O3-3.13-intel2016-64/lib
>> > > >
>> > >
>> -Wl,-rpath,/scratch/simulreserv/softwares/petsc-3.13.0/x64-O3-3.13-intel2016-64/lib
>> > > >
>> > >
>> -L/scratch/simulreserv/softwares/petsc-3.13.0/x64-O3-3.13-intel2016-64/lib
>> > > >
>> > >
>> -Wl,-rpath,/opt/intel/parallel_studio_xe_2016_update2/compilers_and_libraries_2016.2.181/linux/mpi/intel64/lib/release_mt
>> > > >
>> > >
>> -L/opt/intel/parallel_studio_xe_2016_update2/compilers_and_libraries_2016.2.181/linux/mpi/intel64/lib/release_mt
>> > > >
>> > >
>> -Wl,-rpath,/opt/intel/parallel_studio_xe_2016_update2/compilers_and_libraries_2016.2.181/linux/mpi/intel64/lib
>> > > >
>> > >
>> -L/opt/intel/parallel_studio_xe_2016_update2/compilers_and_libraries_2016.2.181/linux/mpi/intel64/lib
>> > > > -Wl,-rpath,/opt/intel/parallel_studio_xe_2016_update2/clck/
>> > > > 3.1.2.006/lib/intel64
>> -L/opt/intel/parallel_studio_xe_2016_update2/clck/
>> > > > 3.1.2.006/lib/intel64
>> > > >
>> > >
>> -Wl,-rpath,/opt/intel/parallel_studio_xe_2016_update2/compilers_and_libraries_2016.2.181/linux/ipp/lib/intel64
>> > > >
>> > >
>> -L/opt/intel/parallel_studio_xe_2016_update2/compilers_and_libraries_2016.2.181/linux/ipp/lib/intel64
>> > > >
>> > >
>> -Wl,-rpath,/opt/intel/parallel_studio_xe_2016_update2/compilers_and_libraries_2016.2.181/linux/compiler/lib/intel64
>> > > >
>> > >
>> -L/opt/intel/parallel_studio_xe_2016_update2/compilers_and_libraries_2016.2.181/linux/compiler/lib/intel64
>> > > >
>> > >
>> -Wl,-rpath,/opt/intel/parallel_studio_xe_2016_update2/compilers_and_libraries_2016.2.181/linux/mkl/lib/intel64
>> > > >
>> > >
>> -L/opt/intel/parallel_studio_xe_2016_update2/compilers_and_libraries_2016.2.181/linux/mkl/lib/intel64
>> > > >
>> > >
>> -Wl,-rpath,/opt/intel/parallel_studio_xe_2016_update2/compilers_and_libraries_2016.2.181/linux/tbb/lib/intel64/gcc4.4
>> > > >
>> > >
>> -L/opt/intel/parallel_studio_xe_2016_update2/compilers_and_libraries_2016.2.181/linux/tbb/lib/intel64/gcc4.4
>> > > >
>> > >
>> -Wl,-rpath,/opt/intel/parallel_studio_xe_2016_update2/compilers_and_libraries_2016.2.181/linux/daal/lib/intel64_lin
>> > > >
>> > >
>> -L/opt/intel/parallel_studio_xe_2016_update2/compilers_and_libraries_2016.2.181/linux/daal/lib/intel64_lin
>> > > >
>> > >
>> -Wl,-rpath,/opt/intel/parallel_studio_xe_2016_update2/compilers_and_libraries_2016.2.181/linux/tbb/lib/intel64_lin/gcc4.4
>> > > >
>> > >
>> -L/opt/intel/parallel_studio_xe_2016_update2/compilers_and_libraries_2016.2.181/linux/tbb/lib/intel64_lin/gcc4.4
>> > > >
>> > >
>> -Wl,-rpath,/opt/intel/parallel_studio_xe_2016_update2/compilers_and_libraries_2016.2.181/linux/compiler/lib/intel64_lin
>> > > >
>> > >
>> -L/opt/intel/parallel_studio_xe_2016_update2/compilers_and_libraries_2016.2.181/linux/compiler/lib/intel64_lin
>> > > > -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/4.8.5
>> > > > -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5
>> > > > -Wl,-rpath,/opt/intel/mpi-rt/5.1/intel64/lib/release_mt
>> > > > -Wl,-rpath,/opt/intel/mpi-rt/5.1/intel64/lib -lpetsc -lHYPRE
>> -lcmumps
>> > > > -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lscalapack
>> -lopenblas
>> > > > -lstdc++ -ldl -lmpifort -lmpi -lmpigi -lrt -lpthread -lifport
>> > > -lifcore_pic
>> > > > -limf -lsvml -lm -lipgo -lirc -lgcc_s -lirc_s -lquadmath -lstdc++
>> -ldl -o
>> > > > ex5f
>> > > >
>> > > > [alfredo.jaramillo at sdumont11 tutorials]$ ./ex5f
>> > > > Number of SNES iterations =     4
>> > > >
>> > > > [alfredo.jaramillo at sdumont11 tutorials]$ ./ex19
>> > > > lid velocity = 0.0625, prandtl # = 1., grashof # = 1.
>> > > > Number of SNES iterations = 2
>> > > >
>> > > > On Thu, May 21, 2020 at 6:53 PM Satish Balay <balay at mcs.anl.gov>
>> wrote:
>> > > >
>> > > > > Please copy/paste complete [compile] commands from:
>> > > > >
>> > > > > src/snes/tutorials/
>> > > > > make clean
>> > > > > make ex19
>> > > > > make ex5f
>> > > > >
>> > > > > Likely the link command used in your code is different than what
>> is
>> > > used
>> > > > > here - triggering errors.
>> > > > >
>> > > > > Satish
>> > > > >
>> > > > > On Thu, 21 May 2020, Alfredo Jaramillo wrote:
>> > > > >
>> > > > > > hello Satish, no the tests seem to be ok altough some error
>> related
>> > > to
>> > > > > mpd.
>> > > > > >
>> > > > > > ==============THE TESTS===================
>> > > > > >
>> > > > > > Running check examples to verify correct installation
>> > > > > > Using PETSC_DIR=/scratch/simulreserv/softwares/petsc-3.13.0 and
>> > > > > > PETSC_ARCH=x64-O3-3.13-intel2016-64
>> > > > > > Possible error running C/C++ src/snes/tutorials/ex19 with 1 MPI
>> > > process
>> > > > > > See http://www.mcs.anl.gov/petsc/documentation/faq.html
>> > > > > > mpiexec_sdumont11: cannot connect to local mpd
>> > > > > > (/tmp/mpd2.console_alfredo.jaramillo); possible causes:
>> > > > > >   1. no mpd is running on this host
>> > > > > >   2. an mpd is running but was started without a "console" (-n
>> > > option)
>> > > > > > Possible error running C/C++ src/snes/tutorials/ex19 with 2 MPI
>> > > processes
>> > > > > > See http://www.mcs.anl.gov/petsc/documentation/faq.html
>> > > > > > mpiexec_sdumont11: cannot connect to local mpd
>> > > > > > (/tmp/mpd2.console_alfredo.jaramillo); possible causes:
>> > > > > >   1. no mpd is running on this host
>> > > > > >   2. an mpd is running but was started without a "console" (-n
>> > > option)
>> > > > > > 1,5c1,3
>> > > > > > < lid velocity = 0.0016, prandtl # = 1., grashof # = 1.
>> > > > > > <   0 SNES Function norm 0.0406612
>> > > > > > <   1 SNES Function norm 4.12227e-06
>> > > > > > <   2 SNES Function norm 6.098e-11
>> > > > > > < Number of SNES iterations = 2
>> > > > > > ---
>> > > > > > > mpiexec_sdumont11: cannot connect to local mpd
>> > > > > > (/tmp/mpd2.console_alfredo.jaramillo); possible causes:
>> > > > > > >   1. no mpd is running on this host
>> > > > > > >   2. an mpd is running but was started without a "console" (-n
>> > > option)
>> > > > > > /scratch/simulreserv/softwares/petsc-3.13.0/src/snes/tutorials
>> > > > > > Possible problem with ex19 running with hypre, diffs above
>> > > > > > =========================================
>> > > > > > 1,9c1,3
>> > > > > > < lid velocity = 0.0625, prandtl # = 1., grashof # = 1.
>> > > > > > <   0 SNES Function norm 0.239155
>> > > > > > <     0 KSP Residual norm 0.239155
>> > > > > > <     1 KSP Residual norm < 1.e-11
>> > > > > > <   1 SNES Function norm 6.81968e-05
>> > > > > > <     0 KSP Residual norm 6.81968e-05
>> > > > > > <     1 KSP Residual norm < 1.e-11
>> > > > > > <   2 SNES Function norm < 1.e-11
>> > > > > > < Number of SNES iterations = 2
>> > > > > > ---
>> > > > > > > mpiexec_sdumont11: cannot connect to local mpd
>> > > > > > (/tmp/mpd2.console_alfredo.jaramillo); possible causes:
>> > > > > > >   1. no mpd is running on this host
>> > > > > > >   2. an mpd is running but was started without a "console" (-n
>> > > option)
>> > > > > > /scratch/simulreserv/softwares/petsc-3.13.0/src/snes/tutorials
>> > > > > > Possible problem with ex19 running with mumps, diffs above
>> > > > > > =========================================
>> > > > > > Possible error running Fortran example src/snes/tutorials/ex5f
>> with
>> > > 1 MPI
>> > > > > > process
>> > > > > > See http://www.mcs.anl.gov/petsc/documentation/faq.html
>> > > > > > mpiexec_sdumont11: cannot connect to local mpd
>> > > > > > (/tmp/mpd2.console_alfredo.jaramillo); possible causes:
>> > > > > >   1. no mpd is running on this host
>> > > > > >   2. an mpd is running but was started without a "console" (-n
>> > > option)
>> > > > > > Completed test examples
>> > > > > >
>> > > > > > ===============================
>> > > > > >
>> > > > > > I entered in src/snes/tutorials/ and executed "make ex5f". The
>> binary
>> > > > > exf5
>> > > > > > was created
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > On Thu, May 21, 2020 at 6:37 PM Satish Balay <balay at mcs.anl.gov
>> >
>> > > wrote:
>> > > > > >
>> > > > > > > Do you get this error when building PETSc examples [C and/or
>> > > fortran] -
>> > > > > > > when you build them with the corresponding petsc makefile?
>> > > > > > >
>> > > > > > > Can you send the log of the example compiles?
>> > > > > > >
>> > > > > > > Satish
>> > > > > > >
>> > > > > > > ---
>> > > > > > >
>> > > > > > > [the attachment got deleted - don't know by who..]
>> > > > > > >
>> > > > > > > DENIAL OF SERVICE ALERT
>> > > > > > >
>> > > > > > > A denial of service protection limit was exceeded. The file
>> has
>> > > been
>> > > > > > > removed.
>> > > > > > > Context: 'configure.log.7z'
>> > > > > > > Reason: The data size limit was exceeded
>> > > > > > > Limit: 10 MB
>> > > > > > > Ticket Number : 0c9c-5ec6-f30f-0001
>> > > > > > >
>> > > > > > >
>> > > > > > > For further information, contact your system administrator.
>> > > > > > > Copyright 1999-2014 McAfee, Inc.
>> > > > > > > All Rights Reserved.
>> > > > > > > http://www.mcafee.com
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > On Thu, 21 May 2020, Alfredo Jaramillo wrote:
>> > > > > > >
>> > > > > > > > dear PETSc team,
>> > > > > > > >
>> > > > > > > > I have compiled PETSc with a 2016 version of the intel
>> > > compilers. The
>> > > > > > > > installation went well, but when I tried to compile my code
>> the
>> > > > > following
>> > > > > > > > error appears in the final step of compilation (linking
>> with ld)
>> > > > > > > >
>> > > > > > > > ../build/linux_icc/obj_linux_icc_opt/main.o: In function
>> `main':
>> > > > > > > > main.c:(.text+0x0): multiple definition of `main'
>> > > > > > > >
>> > > > > > >
>> > > > >
>> > >
>> /opt/intel/parallel_studio_xe_2016_update2/compilers_and_libraries_2016.2.181/linux/compiler/lib/intel64/libifcore_pic.a(for_main.o):for_main.c:(.text+0x0):
>> > > > > > > > first defined here
>> > > > > > > >
>> > > > > > >
>> > > > >
>> > >
>> /opt/intel/parallel_studio_xe_2016_update2/compilers_and_libraries_2016.2.181/linux/compiler/lib/intel64/libifcore_pic.a(for_main.o):
>> > > > > > > > In function `main':
>> > > > > > > > for_main.c:(.text+0x3e): undefined reference to `MAIN__'
>> > > > > > > >
>> > > > > > > > I searched for this and I found that the option
>> "-nofor_main"
>> > > should
>> > > > > be
>> > > > > > > > added when compiling with ifort, but our code is written
>> only in
>> > > C an
>> > > > > > > C++.
>> > > > > > > > The FORTRAN compiler is used when PETSc compiles MUMPS. So
>> I dont
>> > > > > know if
>> > > > > > > > this would work for this case.
>> > > > > > > >
>> > > > > > > > The configure.log file and the log of the compilation
>> giving the
>> > > > > error
>> > > > > > > are
>> > > > > > > > attached to this message. These logs were obtained in a
>> cluster,
>> > > I'm
>> > > > > > > > getting the same error on my personal computer with a 2020
>> > > version
>> > > > > of the
>> > > > > > > > Intel Parallel Studio.
>> > > > > > > >
>> > > > > > > > thank you for any help on this
>> > > > > > > > Alfredo
>> > > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > > >
>> > > >
>> > >
>> > >
>> >
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20200522/9781ecd7/attachment-0001.html>

More information about the petsc-users mailing list