[petsc-users] Error while building PETSc with MATLAB

Satish Balay balay at mcs.anl.gov
Fri Sep 1 10:12:43 CDT 2023 

Also:

    '-known-64-bit-blas-indices=1',

Note: most externalpackages might not work in this mode.

[we can't really over come such dependency/conflicts across packages]

Satish

On Fri, 1 Sep 2023, Satish Balay via petsc-users wrote:

> Here is the matlab test that runs in CI 
> 
> https://gitlab.com/petsc/petsc/-/jobs/4904566768
> 
> config/examples/arch-ci-linux-matlab-ilp64.py
> 
> # Note: regular BLAS [with 32-bit integers] conflict with
> # MATLAB BLAS - hence requiring -known-64-bit-blas-indices=1
> 
> Ah - so you need to use the ilp64 blas/lapack with matlab  - to have a compatible build
> 
>     '--with-blaslapack-dir='+matlab_dir,
> 
> Satish
> 
> 
> On Fri, 1 Sep 2023, INTURU SRINIVAS 20PHD0548 via petsc-users wrote:
> 
> > Hi Amneet,
> > 
> > Without libmesh, even for PETSc with MATLAB is not working. It is showing
> > error for both 3.13.4 and 3.17.5 versions.
> > 
> > I am trying to install IBAMR in HPC cluster with libmesh and this is for
> > general usage not only for WEC. This I tried without linking Matlab with
> > PETSc. Here also I got the error while configuring libmesh 1.6.2 that PETSc
> > was not found but --enable-petsc-required is specified.
> > 
> > The reason for specifically asking for libmesh with PETSc and Matlab is in
> > the cfd-mpc-wecs application for complicated geometries it is mentioned to
> > build cfd-mpc-wecs with libmesh construct.
> > 
> > For the past 10 days I am trying these various things and ended up with
> > errors.
> > I want to solve this and proceed further to do my research work.
> > 
> > Please help me to solve this as soon as possible.
> > 
> > 
> > Thanks and Regards
> > Srinivas
> > 
> > On Fri, Sep 1, 2023, 19:48 Amneet Bhalla <mail2amneet at gmail.com> wrote:
> > 
> > > Hi Srinivas,
> > >
> > > As discussed earlier you don’t need libMesh for the WEC application. You
> > > can just work with PETSc 3.17.5 that builds with Matlab.
> > >
> > > Do you have a specific reason for wanting to build libMesh?
> > >
> > > Thanks,
> > > —Amneet
> > >
> > > On Thu, Aug 31, 2023 at 10:16 PM INTURU SRINIVAS 20PHD0548 via petsc-users
> > > <petsc-users at mcs.anl.gov> wrote:
> > >
> > >> Hi,
> > >>
> > >> I ran "make all" for petsc 3.13.4 by removing all occurrences of "-lut
> > >> -licudata -licui18n -licuuc". When I ran "make test" got the following
> > >> errors
> > >> in petsc/3.13.4/linux-debug:
> > >> TEST
> > >> linux-debug/tests/counts/tao_leastsquares_tutorials_matlab-matlab_ls_test.counts
> > >> not ok tao_leastsquares_tutorials_matlab-matlab_ls_test # Error code: 124
> > >> # Running problem 5
> > >> # [0]PETSC ERROR: --------------------- Error Message
> > >> --------------------------------------------------------------
> > >> # [0]PETSC ERROR: Error in external library
> > >> # [0]PETSC ERROR: Unable to start MATLAB engine on
> > >> # [0]PETSC ERROR: See
> > >> https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
> > >> shooting.
> > >> # [0]PETSC ERROR: Petsc Release Version 3.13.4, Aug 01, 2020
> > >> # [0]PETSC ERROR: ../matlab_ls_test on a linux-debug named MB108SMEC028
> > >> by vit Fri Sep  1 10:01:11 2023
> > >> # [0]PETSC ERROR: Configure options
> > >> --CC=/home/vit/sfw/linux/openmpi/4.1.4/bin/mpicc
> > >> --CXX=/home/vit/sfw/linux/openmpi/4.1.4/bin/mpicxx
> > >> --FC=/home/vit/sfw/linux/openmpi/4.1.4/bin/mpif90 --with-debugging=1
> > >> --download-hypre=1 --download-fblaslapack=1 --with-x=0
> > >> --with-matlab-dir=/usr/local/MATLAB/R2020b/ --with-matlab-engine=1
> > >> --with-matlab-engine-dir=/usr/local/MATLAB/R2020b/extern/engines/
> > >> # [0]PETSC ERROR: #1 PetscMatlabEngineCreate() line 67 in
> > >> /home/vit/sfw/petsc/3.13.4/src/sys/classes/matlabengine/matlab.c
> > >> # [0]PETSC ERROR: #2 main() line 126 in
> > >> /home/vit/sfw/petsc/3.13.4/src/tao/leastsquares/tutorials/matlab/matlab_ls_test.c
> > >> # [0]PETSC ERROR: PETSc Option Table entries:
> > >> # [0]PETSC ERROR: -prob_id 5
> > >> # [0]PETSC ERROR: -tao_smonitor
> > >> # [0]PETSC ERROR: ----------------End of Error Message -------send entire
> > >> error message to petsc-maint at mcs.anl.gov----------
> > >> #
> > >> --------------------------------------------------------------------------
> > >> # MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_SELF
> > >> # with errorcode 126076.
> > >> #
> > >> # NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
> > >> # You may or may not see output from other processes, depending on
> > >> # exactly when Open MPI kills them.
> > >> #
> > >> --------------------------------------------------------------------------
> > >>  ok tao_leastsquares_tutorials_matlab-matlab_ls_test # SKIP Command
> > >> failed so no diff
> > >>
> > >> in petsc/3.13.4/linux-opt
> > >>  TEST
> > >> linux-opt/tests/counts/tao_leastsquares_tutorials_matlab-matlab_ls_test.counts
> > >> not ok tao_leastsquares_tutorials_matlab-matlab_ls_test # Error code: 124
> > >> # Running problem 5
> > >> # [0]PETSC ERROR: --------------------- Error Message
> > >> --------------------------------------------------------------
> > >> # [0]PETSC ERROR: Error in external library
> > >> # [0]PETSC ERROR: Unable to start MATLAB engine on
> > >> # [0]PETSC ERROR: See
> > >> https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
> > >> shooting.
> > >> # [0]PETSC ERROR: Petsc Release Version 3.13.4, Aug 01, 2020
> > >> # [0]PETSC ERROR: ../matlab_ls_test on a linux-opt named MB108SMEC028 by
> > >> vit Fri Sep  1 10:34:12 2023
> > >> # [0]PETSC ERROR: Configure options
> > >> --CC=/home/vit/sfw/linux/openmpi/4.1.4/bin/mpicc
> > >> --CXX=/home/vit/sfw/linux/openmpi/4.1.4/bin/mpicxx
> > >> --FC=/home/vit/sfw/linux/openmpi/4.1.4/bin/mpif90 --COPTFLAGS=-O3
> > >> --CXXOPTFLAGS=-O3 --FOPTFLAGS=-O3 --PETSC_ARCH=linux-opt --with-debugging=0
> > >> --download-hypre=1 --with-x=0 --download-fblaslapack=1
> > >> --with-matlab-dir=/usr/local/MATLAB/R2020b/ --with-matlab-engine=1
> > >> --with-matlab-engine-dir=/usr/local/MATLAB/R2020b/extern/engines/
> > >> # [0]PETSC ERROR: #1 PetscMatlabEngineCreate() line 67 in
> > >> /home/vit/sfw/petsc/3.13.4/src/sys/classes/matlabengine/matlab.c
> > >> # [0]PETSC ERROR: #2 main() line 126 in
> > >> /home/vit/sfw/petsc/3.13.4/src/tao/leastsquares/tutorials/matlab/matlab_ls_test.c
> > >> # [0]PETSC ERROR: PETSc Option Table entries:
> > >> # [0]PETSC ERROR: -prob_id 5
> > >> # [0]PETSC ERROR: -tao_smonitor
> > >> # [0]PETSC ERROR: ----------------End of Error Message -------send entire
> > >> error message to petsc-maint at mcs.anl.gov----------
> > >> #
> > >> --------------------------------------------------------------------------
> > >> # MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_SELF
> > >> # with errorcode 126076.
> > >> #
> > >> # NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
> > >> # You may or may not see output from other processes, depending on
> > >> # exactly when Open MPI kills them.
> > >> #
> > >> --------------------------------------------------------------------------
> > >>  ok tao_leastsquares_tutorials_matlab-matlab_ls_test # SKIP Command
> > >> failed so no diff.
> > >>
> > >> The same errors occurred for petsc 3.17.5 version. I am sharing the
> > >> config.log and make.log for both petsc 3.13.4 and petsc 3.17.5. Please find
> > >> the attachment and do the needful.
> > >>
> > >> Thanks and regards
> > >> Srinivas
> > >>
> > >> On Tue, Aug 29, 2023 at 10:56 PM Satish Balay <balay at mcs.anl.gov> wrote:
> > >>
> > >>> Well - you sent in libmesh log not petsc's configure.log/make.log for
> > >>> petsc-3.17
> > >>>
> > >>> Anyway - with petsc-3.13 - you have:
> > >>>
> > >>> >>>>
> > >>>
> > >>> Matlab:
> > >>>   Includes: -I/usr/local/MATLAB/R2020b/extern/include
> > >>>   /usr/local/MATLAB/R2020b
> > >>> MatlabEngine:
> > >>>   Library:
> > >>> -Wl,-rpath,/usr/local/MATLAB/R2020b/sys/os/glnxa64:/usr/local/MATLAB/R2020b/bin/glnxa64:/usr/local/MATLAB/R2020b/extern/lib/glnxa64
> > >>> -L/usr/local/MATLAB/R2020b/bin/glnxa64
> > >>> -L/usr/local/MATLAB/R2020b/extern/lib/glnxa64 -leng -lmex -lmx -lmat -lut
> > >>> -licudata -licui18n -licuuc
> > >>>   Language used to compile PETSc: C
> > >>> <<<<<
> > >>>
> > >>> With petsc-3.19 (and matlab-R2022a) - we are seeing:
> > >>>
> > >>> https://gitlab.com/petsc/petsc/-/jobs/4904566768
> > >>>
> > >>> >>>
> > >>> Matlab:
> > >>>   Includes:
> > >>>  -I/nfs/gce/software/custom/linux-ubuntu22.04-x86_64/matlab/R2022a/extern/include
> > >>>   Libraries:
> > >>> -Wl,-rpath,/nfs/gce/software/custom/linux-ubuntu22.04-x86_64/matlab/R2022a/bin/glnxa64
> > >>> -L/nfs/gce/software/custom/linux-ubuntu22.04-x86_64/matlab/R2022a/bin/glnxa64
> > >>> -leng -lmex -lmx -lmat
> > >>>   Executable:
> > >>> /nfs/gce/software/custom/linux-ubuntu22.04-x86_64/matlab/R2022a
> > >>>   mex:
> > >>> /nfs/gce/software/custom/linux-ubuntu22.04-x86_64/matlab/R2022a/bin/mex
> > >>>   matlab:
> > >>> /nfs/gce/software/custom/linux-ubuntu22.04-x86_64/matlab/R2022a/bin/matlab
> > >>> -glnxa64
> > >>> <<<
> > >>>
> > >>> I.e "-lut -licudata -licui18n -licuuc" are not preset here. This might
> > >>> be a change wrt newer matlab versions.
> > >>>
> > >>> You can:
> > >>>
> > >>> - edit
> > >>> /home/vit/sfw/petsc/3.13.4/linux-opt/lib/petsc/conf/petscvariables and
> > >>> remove all occurrences of "-lut -licudata -licui18n -licuuc"
> > >>> - now run 'make all' in '/home/vit/sfw/petsc/3.13.4'
> > >>>
> > >>> And see if the build works now.
> > >>>
> > >>> Satish
> > >>>
> > >>> On Tue, 29 Aug 2023, INTURU SRINIVAS 20PHD0548 via petsc-users wrote:
> > >>>
> > >>> > I am sharing the log files while building petsc3.13.4 with matlab and
> > >>> also
> > >>> > the log file while building libmesh with petsc3.17.5 and matlab.
> > >>> Building
> > >>> > petsc 3.17.5 with matlab was done successfully. But libmesh is not
> > >>> able to
> > >>> > find the petsc
> > >>> > Please find the attachments.
> > >>> >
> > >>> > On Tue, Aug 29, 2023 at 7:31 PM Satish Balay <balay at mcs.anl.gov>
> > >>> wrote:
> > >>> >
> > >>> > > Send configure.log, make.log from both petsc-3.13 and 3.17 [or 3.19].
> > >>> > >
> > >>> > > [you can gzip them to make the logs friendly to mailing list - or
> > >>> send
> > >>> > > them to petsc-maint]
> > >>> > >
> > >>> > > And does test suite work with 3.17? [or 3.19?]
> > >>> > >
> > >>> > > Satish
> > >>> > >
> > >>> > > On Tue, 29 Aug 2023, INTURU SRINIVAS 20PHD0548 via petsc-users wrote:
> > >>> > >
> > >>> > > > I am sharing the make.log file while building petsc-3.13.4 with
> > >>> Matlab.
> > >>> > > > Please find the attachment and do the needful.
> > >>> > > >
> > >>> > > > On Tue, Aug 29, 2023 at 10:19 AM INTURU SRINIVAS 20PHD0548 <
> > >>> > > > inturu.srinivas2020 at vitstudent.ac.in> wrote:
> > >>> > > >
> > >>> > > > > I tried with petsc-3.17.5. During building of libmesh, the error
> > >>> shows
> > >>> > > > > petsc was not found
> > >>> > > > >
> > >>> > > > > On Mon, Aug 28, 2023 at 9:43 PM Satish Balay <balay at mcs.anl.gov>
> > >>> > > wrote:
> > >>> > > > >
> > >>> > > > >> https://ibamr.github.io/linux says petsc-3.17
> > >>> > > > >>
> > >>> > > > >> Here you are using 3.13
> > >>> > > > >>
> > >>> > > > >> Can you retry with petsc-3.17.5?
> > >>> > > > >>
> > >>> > > > >> Satish
> > >>> > > > >>
> > >>> > > > >> On Mon, 28 Aug 2023, INTURU SRINIVAS 20PHD0548 via petsc-users
> > >>> wrote:
> > >>> > > > >>
> > >>> > > > >> > Hello,
> > >>> > > > >> >
> > >>> > > > >> > I want to build PETSc with MATLAB for working on the
> > >>> simulation
> > >>> > > using
> > >>> > > > >> IBAMR
> > >>> > > > >> > open software. While building the PETSc, using the following
> > >>> > > > >> >
> > >>> > > > >> > export PETSC_DIR=$PWD
> > >>> > > > >> > export PETSC_ARCH=linux-debug
> > >>> > > > >> > ./configure \
> > >>> > > > >> >   --CC=$HOME/sfw/linux/openmpi/4.1.4/bin/mpicc \
> > >>> > > > >> >   --CXX=$HOME/sfw/linux/openmpi/4.1.4/bin/mpicxx \
> > >>> > > > >> >   --FC=$HOME/sfw/linux/openmpi/4.1.4/bin/mpif90 \
> > >>> > > > >> >   --with-debugging=1 \
> > >>> > > > >> >   --download-hypre=1 \
> > >>> > > > >> >   --download-fblaslapack=1 \
> > >>> > > > >> >   --with-x=0 \
> > >>> > > > >> >   --with-matlab-dir=/usr/local/MATLAB/R2020b/
> > >>> > > > >> >   --with-matlab-engine=1
> > >>> > > > >> >
> > >>>  --with-matlab-engine-dir=/usr/local/MATLAB/R2020b/extern/engines/
> > >>> > > > >> >
> > >>> > > > >> > make -j4
> > >>> > > > >> > make -j4 test
> > >>> > > > >> >
> > >>> > > > >> > I got the following error
> > >>> > > > >> > CLINKER
> > >>> > > > >>
> > >>> linux-debug/tests/tao/leastsquares/tutorials/matlab/matlab_ls_test
> > >>> > > > >> > /usr/bin/ld:
> > >>> > > > >> >
> > >>> > >
> > >>> linux-debug/tests/tao/leastsquares/tutorials/matlab/matlab_ls_test.o: in
> > >>> > > > >> > function `EvaluateResidual':
> > >>> > > > >> >
> > >>> > > > >>
> > >>> > >
> > >>> /home/vit/sfw/petsc/3.13.4/src/tao/leastsquares/tutorials/matlab/matlab_ls_test.c:32:
> > >>> > > > >> > undefined reference to `PetscMatlabEnginePut'
> > >>> > > > >> > /usr/bin/ld:
> > >>> > > > >> >
> > >>> > > > >>
> > >>> > >
> > >>> /home/vit/sfw/petsc/3.13.4/src/tao/leastsquares/tutorials/matlab/matlab_ls_test.c:33:
> > >>> > > > >> > undefined reference to `PetscMatlabEngineEvaluate'
> > >>> > > > >> > /usr/bin/ld:
> > >>> > > > >> >
> > >>> > > > >>
> > >>> > >
> > >>> /home/vit/sfw/petsc/3.13.4/src/tao/leastsquares/tutorials/matlab/matlab_ls_test.c:35:
> > >>> > > > >> > undefined reference to `PetscMatlabEngineGet'
> > >>> > > > >> > /usr/bin/ld:
> > >>> > > > >> >
> > >>> > >
> > >>> linux-debug/tests/tao/leastsquares/tutorials/matlab/matlab_ls_test.o: in
> > >>> > > > >> > function `EvaluateJacobian':
> > >>> > > > >> >
> > >>> > > > >>
> > >>> > >
> > >>> /home/vit/sfw/petsc/3.13.4/src/tao/leastsquares/tutorials/matlab/matlab_ls_test.c:46:
> > >>> > > > >> > undefined reference to `PetscMatlabEnginePut'
> > >>> > > > >> > /usr/bin/ld:
> > >>> > > > >> >
> > >>> > > > >>
> > >>> > >
> > >>> /home/vit/sfw/petsc/3.13.4/src/tao/leastsquares/tutorials/matlab/matlab_ls_test.c:47:
> > >>> > > > >> > undefined reference to `PetscMatlabEngineEvaluate'
> > >>> > > > >> > /usr/bin/ld:
> > >>> > > > >> >
> > >>> > > > >>
> > >>> > >
> > >>> /home/vit/sfw/petsc/3.13.4/src/tao/leastsquares/tutorials/matlab/matlab_ls_test.c:49:
> > >>> > > > >> > undefined reference to `PetscMatlabEngineGet'
> > >>> > > > >> > /usr/bin/ld:
> > >>> > > > >> >
> > >>> > >
> > >>> linux-debug/tests/tao/leastsquares/tutorials/matlab/matlab_ls_test.o: in
> > >>> > > > >> > function `TaoPounders':
> > >>> > > > >> >
> > >>> > > > >>
> > >>> > >
> > >>> /home/vit/sfw/petsc/3.13.4/src/tao/leastsquares/tutorials/matlab/matlab_ls_test.c:75:
> > >>> > > > >> > undefined reference to `PetscMatlabEngineGet'
> > >>> > > > >> > /usr/bin/ld:
> > >>> > > > >> >
> > >>> > >
> > >>> linux-debug/tests/tao/leastsquares/tutorials/matlab/matlab_ls_test.o: in
> > >>> > > > >> > function `main':
> > >>> > > > >> >
> > >>> > > > >>
> > >>> > >
> > >>> /home/vit/sfw/petsc/3.13.4/src/tao/leastsquares/tutorials/matlab/matlab_ls_test.c:126:
> > >>> > > > >> > undefined reference to `PetscMatlabEngineCreate'
> > >>> > > > >> > /usr/bin/ld:
> > >>> > > > >> >
> > >>> > > > >>
> > >>> > >
> > >>> /home/vit/sfw/petsc/3.13.4/src/tao/leastsquares/tutorials/matlab/matlab_ls_test.c:127:
> > >>> > > > >> > undefined reference to `PetscMatlabEngineEvaluate'
> > >>> > > > >> > /usr/bin/ld:
> > >>> > > > >> >
> > >>> > > > >>
> > >>> > >
> > >>> /home/vit/sfw/petsc/3.13.4/src/tao/leastsquares/tutorials/matlab/matlab_ls_test.c:139:
> > >>> > > > >> > undefined reference to `PetscMatlabEngineEvaluate'
> > >>> > > > >> > /usr/bin/ld:
> > >>> > > > >> >
> > >>> > > > >>
> > >>> > >
> > >>> /home/vit/sfw/petsc/3.13.4/src/tao/leastsquares/tutorials/matlab/matlab_ls_test.c:140:
> > >>> > > > >> > undefined reference to `PetscMatlabEngineGetArray'
> > >>> > > > >> > /usr/bin/ld:
> > >>> > > > >> >
> > >>> > > > >>
> > >>> > >
> > >>> /home/vit/sfw/petsc/3.13.4/src/tao/leastsquares/tutorials/matlab/matlab_ls_test.c:142:
> > >>> > > > >> > undefined reference to `PetscMatlabEngineGetArray'
> > >>> > > > >> > /usr/bin/ld:
> > >>> > > > >> >
> > >>> > > > >>
> > >>> > >
> > >>> /home/vit/sfw/petsc/3.13.4/src/tao/leastsquares/tutorials/matlab/matlab_ls_test.c:144:
> > >>> > > > >> > undefined reference to `PetscMatlabEngineGetArray'
> > >>> > > > >> > /usr/bin/ld:
> > >>> > > > >> >
> > >>> > > > >>
> > >>> > >
> > >>> /home/vit/sfw/petsc/3.13.4/src/tao/leastsquares/tutorials/matlab/matlab_ls_test.c:146:
> > >>> > > > >> > undefined reference to `PetscMatlabEngineGetArray'
> > >>> > > > >> > /usr/bin/ld:
> > >>> > > > >> >
> > >>> > > > >>
> > >>> > >
> > >>> /home/vit/sfw/petsc/3.13.4/src/tao/leastsquares/tutorials/matlab/matlab_ls_test.c:148:
> > >>> > > > >> > undefined reference to `PetscMatlabEngineGetArray'
> > >>> > > > >> > /usr/bin/ld:
> > >>> > > > >> >
> > >>> > > > >>
> > >>> > >
> > >>> /home/vit/sfw/petsc/3.13.4/src/tao/leastsquares/tutorials/matlab/matlab_ls_test.c:154:
> > >>> > > > >> > undefined reference to `PetscMatlabEngineEvaluate'
> > >>> > > > >> > /usr/bin/ld:
> > >>> > > > >> >
> > >>> > > > >>
> > >>> > >
> > >>> /home/vit/sfw/petsc/3.13.4/src/tao/leastsquares/tutorials/matlab/matlab_ls_test.c:157:
> > >>> > > > >> > undefined reference to `PetscMatlabEngineEvaluate'
> > >>> > > > >> > /usr/bin/ld:
> > >>> > > > >> >
> > >>> > > > >>
> > >>> > >
> > >>> /home/vit/sfw/petsc/3.13.4/src/tao/leastsquares/tutorials/matlab/matlab_ls_test.c:158:
> > >>> > > > >> > undefined reference to `PetscMatlabEngineDestroy'
> > >>> > > > >> >
> > >>> > > > >> > collect2: error: ld returned 1 exit status
> > >>> > > > >> > make: *** [gmakefile.test:185:
> > >>> > > > >> >
> > >>> linux-debug/tests/tao/leastsquares/tutorials/matlab/matlab_ls_test]
> > >>> > > > >> Error 1
> > >>> > > > >> > make: *** Waiting for unfinished jobs....
> > >>> > > > >> >
> > >>> > > > >> > Please help me to solve this issue
> > >>> > > > >> >
> > >>> > > > >> > Thank you
> > >>> > > > >> > Srinivas
> > >>> > > > >> >
> > >>> > > > >> >
> > >>> > > > >>
> > >>> > > > >>
> > >>> > > >
> > >>> > > >
> > >>> > >
> > >>> >
> > >>> >
> > >>>
> > >>
> > >>
> > >> *Disclaimer:This message was sent from Vellore Institute of Technology.
> > >> The contents of this email may contain legally protected confidential or
> > >> privileged information of “Vellore Institute of Technology”.  If you are
> > >> not the intended recipient, you should not disseminate, distribute or copy
> > >> this e-mail. Please notify the sender immediately and destroy all copies of
> > >> this message and any attachments. If you have received this email in error,
> > >> please promptly notify the sender by reply email and delete the original
> > >> email and any backup copies without reading them.*
> > >>
> > >
> > 
> > 
>

More information about the petsc-users mailing list