From ee331 at cam.ac.uk  Mon Aug  1 06:57:36 2022
From: ee331 at cam.ac.uk (E. Ekici)
Date: Mon, 1 Aug 2022 11:57:36 +0000
Subject: [petsc-users] About Preallocation for MPI matrix
Message-ID: <CWLP265MB2612C84B60B77544F495FF93B89A9@CWLP265MB2612.GBRP265.PROD.OUTLOOK.COM>

Hi,

I would like to preallocate my highly sparse matrices to get speed up during assembling.

Up to 70000x70000, it works fine in parallel.

But when I exceed around 70000 rows/columns, I receive following error at preallocation line;

petsc4py.PETSc.Error: error code 4031
[0] MatMPIAIJSetPreallocation() at /home/ekrem/Dev/Venvs/dolfinxcomplex/installation/petsc/src/mat/impls/aij/mpi/mpiaij.c:4039
[0] MatMPIAIJSetPreallocation_MPIAIJ() at /home/ekrem/Dev/Venvs/dolfinxcomplex/installation/petsc/src/mat/impls/aij/mpi/mpiaij.c:2845
[0] MatSeqAIJSetPreallocation() at /home/ekrem/Dev/Venvs/dolfinxcomplex/installation/petsc/src/mat/impls/aij/seq/aij.c:3963
[0] MatSeqAIJSetPreallocation_SeqAIJ() at /home/ekrem/Dev/Venvs/dolfinxcomplex/installation/petsc/src/mat/impls/aij/seq/aij.c:4031
[0] PetscMallocA() at /home/ekrem/Dev/Venvs/dolfinxcomplex/installation/petsc/src/sys/memory/mal.c:401
[0] PetscMallocAlign() at /home/ekrem/Dev/Venvs/dolfinxcomplex/installation/petsc/src/sys/memory/mal.c:49
[0] MatSeqAIJSetPreallocation_SeqAIJ

I have attached my sparsity pattern as CSR.png.

I use petsc4py for matrix generation, and here are the commands that I am using to generate matrix;

row # global row numbers
col = # global column numbers
val = # flattened 2D nonzero array

mat = PETSc.Mat().create(PETSc.COMM_WORLD)
mat.setSizes([(local_size, global_size), (local_size, global_size)])
mat.setType('mpiaij')
ONNZ = len(row)*np.ones(local_size,dtype=np.int32)
mat.setPreallocationNNZ([ONNZ,ONNZ]) # This line throws the error above
mat.setOption(PETSc.Mat.Option.NEW_NONZERO_ALLOCATION_ERR, False)
mat.setUp()
mat.setValues(row, col, val, addv=PETSc.InsertMode.ADD_VALUES)
mat.assemblyBegin()
mat.assemblyEnd()

Many thanks for your response in advance,

Kind regards

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220801/edc32816/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: CSR.png
Type: image/png
Size: 13851 bytes
Desc: CSR.png
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220801/edc32816/attachment-0001.png>

From knepley at gmail.com  Mon Aug  1 12:06:21 2022
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 1 Aug 2022 12:06:21 -0500
Subject: [petsc-users] About Preallocation for MPI matrix
In-Reply-To: <CWLP265MB2612C84B60B77544F495FF93B89A9@CWLP265MB2612.GBRP265.PROD.OUTLOOK.COM>
References: <CWLP265MB2612C84B60B77544F495FF93B89A9@CWLP265MB2612.GBRP265.PROD.OUTLOOK.COM>
Message-ID: <CAMYG4GnN8Bmy2DN-zy826qwyS3ESoJpgMuWWrgk3A3zr5S5N5w@mail.gmail.com>

On Mon, Aug 1, 2022 at 11:45 AM E. Ekici <ee331 at cam.ac.uk> wrote:

> Hi,
>
> I would like to preallocate my highly sparse matrices to get speed up
> during assembling.
>
> Up to 70000x70000, it works fine in parallel.
>
> But when I exceed around 70000 rows/columns, I receive following error at
> preallocation line;
>
> petsc4py.PETSc.Error: error code 4031
> [0] MatMPIAIJSetPreallocation() at
> /home/ekrem/Dev/Venvs/dolfinxcomplex/installation/petsc/src/mat/impls/aij/mpi/mpiaij.c:4039
> [0] MatMPIAIJSetPreallocation_MPIAIJ() at
> /home/ekrem/Dev/Venvs/dolfinxcomplex/installation/petsc/src/mat/impls/aij/mpi/mpiaij.c:2845
> [0] MatSeqAIJSetPreallocation() at
> /home/ekrem/Dev/Venvs/dolfinxcomplex/installation/petsc/src/mat/impls/aij/seq/aij.c:3963
> [0] MatSeqAIJSetPreallocation_SeqAIJ() at
> /home/ekrem/Dev/Venvs/dolfinxcomplex/installation/petsc/src/mat/impls/aij/seq/aij.c:4031
> [0] PetscMallocA() at
> /home/ekrem/Dev/Venvs/dolfinxcomplex/installation/petsc/src/sys/memory/mal.c:401
> [0] PetscMallocAlign() at
> /home/ekrem/Dev/Venvs/dolfinxcomplex/installation/petsc/src/sys/memory/mal.c:49
> [0] MatSeqAIJSetPreallocation_SeqAIJ
>
> I have attached my sparsity pattern as *CSR.png*.
>
> I use petsc4py for matrix generation, and here are the commands that I am
> using to generate matrix;
>
> row # global row numbers
> col = # global column numbers
> val = # flattened 2D nonzero array
>
> mat = PETSc.Mat().create(PETSc.COMM_WORLD)
> mat.setSizes([(local_size, global_size), (local_size, global_size)])
> mat.setType('mpiaij')
> ONNZ = len(row)*np.ones(local_size,dtype=np.int32)
>

Is len(row) == local_size? This looks like it is allocating a dense
diagonal block and a semi-dense off-diagonal block.
Is that true? If so, you are probably running out of memory.

  Thanks,

      Matt


> mat.setPreallocationNNZ([ONNZ,ONNZ]) # This line throws the error above
> mat.setOption(PETSc.Mat.Option.NEW_NONZERO_ALLOCATION_ERR, False)
> mat.setUp()
> mat.setValues(row, col, val, addv=PETSc.InsertMode.ADD_VALUES)
> mat.assemblyBegin()
> mat.assemblyEnd()
>
> Many thanks for your response in advance,
>
> Kind regards
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220801/84a4f283/attachment.html>

From mi.mike1021 at gmail.com  Mon Aug  1 21:55:28 2022
From: mi.mike1021 at gmail.com (Mike Michell)
Date: Mon, 1 Aug 2022 20:55:28 -0600
Subject: [petsc-users] DMPlex PETSCViewer for Surface Component
Message-ID: <CAEc7osb9DCAUOnUgCZ7S5fhAXfroi847W2dQbZVkd34pxW84Lw@mail.gmail.com>

Hi,

I am a user of DMPlex object in 3D grid topology. Currently the solution
field is printed out using viewer PETSCVIEWERVTK in .vtu format. By doing
that the entire volume components are written in the file. I was wondering
if there is an option that I can tell PETSc viewer to print out only
surface component, instead of the entire volume.

Best,
Mike
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220801/382f3309/attachment.html>

From jed at jedbrown.org  Mon Aug  1 22:59:12 2022
From: jed at jedbrown.org (Jed Brown)
Date: Mon, 01 Aug 2022 21:59:12 -0600
Subject: [petsc-users] DMPlex PETSCViewer for Surface Component
In-Reply-To: <CAEc7osb9DCAUOnUgCZ7S5fhAXfroi847W2dQbZVkd34pxW84Lw@mail.gmail.com>
References: <CAEc7osb9DCAUOnUgCZ7S5fhAXfroi847W2dQbZVkd34pxW84Lw@mail.gmail.com>
Message-ID: <87pmhje1nz.fsf@jedbrown.org>

I would create a sub-DM containing only the part you want to view.

Mike Michell <mi.mike1021 at gmail.com> writes:

> Hi,
>
> I am a user of DMPlex object in 3D grid topology. Currently the solution
> field is printed out using viewer PETSCVIEWERVTK in .vtu format. By doing
> that the entire volume components are written in the file. I was wondering
> if there is an option that I can tell PETSc viewer to print out only
> surface component, instead of the entire volume.
>
> Best,
> Mike

From balay at mcs.anl.gov  Mon Aug  1 23:29:58 2022
From: balay at mcs.anl.gov (Satish Balay)
Date: Mon, 1 Aug 2022 23:29:58 -0500 (CDT)
Subject: [petsc-users] petsc-3.17.4 now available
Message-ID: <562c6069-d7b3-b650-5d0-f88c44d6b612@mcs.anl.gov>

Dear PETSc users,

The patch release petsc-3.17.4 is now available for download.

http://www.mcs.anl.gov/petsc/download/index.html

Satish



From knepley at gmail.com  Tue Aug  2 09:38:16 2022
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 2 Aug 2022 09:38:16 -0500
Subject: [petsc-users] DMPlex PETSCViewer for Surface Component
In-Reply-To: <87pmhje1nz.fsf@jedbrown.org>
References: <CAEc7osb9DCAUOnUgCZ7S5fhAXfroi847W2dQbZVkd34pxW84Lw@mail.gmail.com>
	<87pmhje1nz.fsf@jedbrown.org>
Message-ID: <CAMYG4Gn7vL6syvYuUEwmYNnmLgF1LY7HZFd5Q9BVE9XmhZLMjA@mail.gmail.com>

Yes, you can use DMPlexCreateSubmesh() (and friends depending on exactly
what kind of submesh you want). This will allow you to create a vector over
only this mesh, and map your volumetric solution to that subvector. Then
you can view the subvector (which pulls in the submesh).

  Thanks,

     Matt

On Mon, Aug 1, 2022 at 10:59 PM Jed Brown <jed at jedbrown.org> wrote:

> I would create a sub-DM containing only the part you want to view.
>
> Mike Michell <mi.mike1021 at gmail.com> writes:
>
> > Hi,
> >
> > I am a user of DMPlex object in 3D grid topology. Currently the solution
> > field is printed out using viewer PETSCVIEWERVTK in .vtu format. By doing
> > that the entire volume components are written in the file. I was
> wondering
> > if there is an option that I can tell PETSc viewer to print out only
> > surface component, instead of the entire volume.
> >
> > Best,
> > Mike
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220802/1266e139/attachment.html>

From narnoldm at umich.edu  Tue Aug  2 10:27:10 2022
From: narnoldm at umich.edu (Nicholas Arnold-Medabalimi)
Date: Tue, 2 Aug 2022 11:27:10 -0400
Subject: [petsc-users] PetscSF clarification
Message-ID: <CAB1dcV7z2MGPtwjXV++dh_ZpOVHGStCQH2fRHRAhTQv4QxBc2g@mail.gmail.com>

Hello

I have been trying to follow how PetscSF works, and I'm observing some
behavior I don't quite understand. I have been looking at some of the past
petsc-user discussions involving PetsSF, and my understanding is that each
processor will have roots and leaves; roots are "owned" points, and leaves
are ghosts of another processor.

I have a setup when I've built the original mesh(6000 points) on just a
single processor and then distributed it using DMPlexDistrubute with an
overlap and then viewing the generated star forest.

My expectation is that roughly 1/p points will end up as roots on each
processor, with the overlap points being leaves on each processor. However,
instead I get the first processor having 6000 roots with ~ 1/p leaves
PetscSF Object: 4 MPI processes
  type: basic
  [0] Number of roots=6003, leaves=1587, remote ranks=1
  [0] 0 <- (0,0)
, and then each of the following processors having ~1/p leaves all pointed
at the root processor
  [1] Number of roots=0, leaves=1497, remote ranks=1
  [1] 0 <- (0,262)

Is this the expected outcome? At least to me, this implies the first
processor still "owns" all the points. I imagine I misunderstand something
here. Thank you for the assistance

Sincerely
Nicholas


-- 
Nicholas Arnold-Medabalimi

Ph.D. Candidate
Computational Aeroscience Lab
University of Michigan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220802/2b1734ed/attachment.html>

From toby.isaac at gmail.com  Tue Aug  2 10:54:09 2022
From: toby.isaac at gmail.com (Toby Isaac)
Date: Tue, 2 Aug 2022 10:54:09 -0500
Subject: [petsc-users] PetscSF clarification
In-Reply-To: <CAB1dcV7z2MGPtwjXV++dh_ZpOVHGStCQH2fRHRAhTQv4QxBc2g@mail.gmail.com>
References: <CAB1dcV7z2MGPtwjXV++dh_ZpOVHGStCQH2fRHRAhTQv4QxBc2g@mail.gmail.com>
Message-ID: <CAC2U1y_TjQqAODa_WMiN8UpxtyTQhxSE4wrd6HwHdTDZ9qpYEA@mail.gmail.com>

Hi Nicholas,

What command did you use to view the star forest?  What you are
showing looks like the PetscSF that is used to distribute the points
from the root, not the final PetscSF describing the points and their
duplicates once it has been distributed.

-- Toby

On Tue, Aug 2, 2022 at 10:28 AM Nicholas Arnold-Medabalimi
<narnoldm at umich.edu> wrote:
>
> Hello
>
> I have been trying to follow how PetscSF works, and I'm observing some behavior I don't quite understand. I have been looking at some of the past petsc-user discussions involving PetsSF, and my understanding is that each processor will have roots and leaves; roots are "owned" points, and leaves are ghosts of another processor.
>
> I have a setup when I've built the original mesh(6000 points) on just a single processor and then distributed it using DMPlexDistrubute with an overlap and then viewing the generated star forest.
>
> My expectation is that roughly 1/p points will end up as roots on each processor, with the overlap points being leaves on each processor. However, instead I get the first processor having 6000 roots with ~ 1/p leaves
> PetscSF Object: 4 MPI processes
>   type: basic
>   [0] Number of roots=6003, leaves=1587, remote ranks=1
>   [0] 0 <- (0,0)
> , and then each of the following processors having ~1/p leaves all pointed at the root processor
>   [1] Number of roots=0, leaves=1497, remote ranks=1
>   [1] 0 <- (0,262)
>
> Is this the expected outcome? At least to me, this implies the first processor still "owns" all the points. I imagine I misunderstand something here. Thank you for the assistance
>
> Sincerely
> Nicholas
>
>
> --
> Nicholas Arnold-Medabalimi
>
> Ph.D. Candidate
> Computational Aeroscience Lab
> University of Michigan

From toby.isaac at gmail.com  Tue Aug  2 11:11:44 2022
From: toby.isaac at gmail.com (Toby Isaac)
Date: Tue, 2 Aug 2022 11:11:44 -0500
Subject: [petsc-users] PetscSF clarification
In-Reply-To: <CAC2U1y_TjQqAODa_WMiN8UpxtyTQhxSE4wrd6HwHdTDZ9qpYEA@mail.gmail.com>
References: <CAB1dcV7z2MGPtwjXV++dh_ZpOVHGStCQH2fRHRAhTQv4QxBc2g@mail.gmail.com>
	<CAC2U1y_TjQqAODa_WMiN8UpxtyTQhxSE4wrd6HwHdTDZ9qpYEA@mail.gmail.com>
Message-ID: <CAC2U1y_JDkErPj2W7zf2wxUyw2Pt2RgA+hnyyKmf0R8o53gAQw@mail.gmail.com>

I think you want the PetscSF that can be obtained from calling
DMGetPointSF() <https://petsc.org/release/docs/manualpages/DM/DMGetPointSF.html>
on the mesh created by DMPlexDistribute(), not the PetscSF returned by
DMPlexDistribute() itself.

On Tue, Aug 2, 2022 at 10:54 AM Toby Isaac <toby.isaac at gmail.com> wrote:
>
> Hi Nicholas,
>
> What command did you use to view the star forest?  What you are
> showing looks like the PetscSF that is used to distribute the points
> from the root, not the final PetscSF describing the points and their
> duplicates once it has been distributed.
>
> -- Toby
>
> On Tue, Aug 2, 2022 at 10:28 AM Nicholas Arnold-Medabalimi
> <narnoldm at umich.edu> wrote:
> >
> > Hello
> >
> > I have been trying to follow how PetscSF works, and I'm observing some behavior I don't quite understand. I have been looking at some of the past petsc-user discussions involving PetsSF, and my understanding is that each processor will have roots and leaves; roots are "owned" points, and leaves are ghosts of another processor.
> >
> > I have a setup when I've built the original mesh(6000 points) on just a single processor and then distributed it using DMPlexDistrubute with an overlap and then viewing the generated star forest.
> >
> > My expectation is that roughly 1/p points will end up as roots on each processor, with the overlap points being leaves on each processor. However, instead I get the first processor having 6000 roots with ~ 1/p leaves
> > PetscSF Object: 4 MPI processes
> >   type: basic
> >   [0] Number of roots=6003, leaves=1587, remote ranks=1
> >   [0] 0 <- (0,0)
> > , and then each of the following processors having ~1/p leaves all pointed at the root processor
> >   [1] Number of roots=0, leaves=1497, remote ranks=1
> >   [1] 0 <- (0,262)
> >
> > Is this the expected outcome? At least to me, this implies the first processor still "owns" all the points. I imagine I misunderstand something here. Thank you for the assistance
> >
> > Sincerely
> > Nicholas
> >
> >
> > --
> > Nicholas Arnold-Medabalimi
> >
> > Ph.D. Candidate
> > Computational Aeroscience Lab
> > University of Michigan

From narnoldm at umich.edu  Tue Aug  2 11:41:11 2022
From: narnoldm at umich.edu (Nicholas Arnold-Medabalimi)
Date: Tue, 2 Aug 2022 12:41:11 -0400
Subject: [petsc-users] PetscSF clarification
In-Reply-To: <CAC2U1y_JDkErPj2W7zf2wxUyw2Pt2RgA+hnyyKmf0R8o53gAQw@mail.gmail.com>
References: <CAB1dcV7z2MGPtwjXV++dh_ZpOVHGStCQH2fRHRAhTQv4QxBc2g@mail.gmail.com>
	<CAC2U1y_TjQqAODa_WMiN8UpxtyTQhxSE4wrd6HwHdTDZ9qpYEA@mail.gmail.com>
	<CAC2U1y_JDkErPj2W7zf2wxUyw2Pt2RgA+hnyyKmf0R8o53gAQw@mail.gmail.com>
Message-ID: <CAB1dcV7EKAEc85WDvFFCVo+EFmKR2LeCgJhBbLj5XQY_MiPbuA@mail.gmail.com>

I see. I was under the impression that the PetscSF returned by
DMPlexDistribute was the resulting Star Forest. Getting the SF from the DM
after the distribution gets the results I am expecting. Thank you for the
help.



On Tue, Aug 2, 2022 at 12:12 PM Toby Isaac <toby.isaac at gmail.com> wrote:

> I think you want the PetscSF that can be obtained from calling
> DMGetPointSF() <
> https://petsc.org/release/docs/manualpages/DM/DMGetPointSF.html>
> on the mesh created by DMPlexDistribute(), not the PetscSF returned by
> DMPlexDistribute() itself.
>
> On Tue, Aug 2, 2022 at 10:54 AM Toby Isaac <toby.isaac at gmail.com> wrote:
> >
> > Hi Nicholas,
> >
> > What command did you use to view the star forest?  What you are
> > showing looks like the PetscSF that is used to distribute the points
> > from the root, not the final PetscSF describing the points and their
> > duplicates once it has been distributed.
> >
> > -- Toby
> >
> > On Tue, Aug 2, 2022 at 10:28 AM Nicholas Arnold-Medabalimi
> > <narnoldm at umich.edu> wrote:
> > >
> > > Hello
> > >
> > > I have been trying to follow how PetscSF works, and I'm observing some
> behavior I don't quite understand. I have been looking at some of the past
> petsc-user discussions involving PetsSF, and my understanding is that each
> processor will have roots and leaves; roots are "owned" points, and leaves
> are ghosts of another processor.
> > >
> > > I have a setup when I've built the original mesh(6000 points) on just
> a single processor and then distributed it using DMPlexDistrubute with an
> overlap and then viewing the generated star forest.
> > >
> > > My expectation is that roughly 1/p points will end up as roots on each
> processor, with the overlap points being leaves on each processor. However,
> instead I get the first processor having 6000 roots with ~ 1/p leaves
> > > PetscSF Object: 4 MPI processes
> > >   type: basic
> > >   [0] Number of roots=6003, leaves=1587, remote ranks=1
> > >   [0] 0 <- (0,0)
> > > , and then each of the following processors having ~1/p leaves all
> pointed at the root processor
> > >   [1] Number of roots=0, leaves=1497, remote ranks=1
> > >   [1] 0 <- (0,262)
> > >
> > > Is this the expected outcome? At least to me, this implies the first
> processor still "owns" all the points. I imagine I misunderstand something
> here. Thank you for the assistance
> > >
> > > Sincerely
> > > Nicholas
> > >
> > >
> > > --
> > > Nicholas Arnold-Medabalimi
> > >
> > > Ph.D. Candidate
> > > Computational Aeroscience Lab
> > > University of Michigan
>


-- 
Nicholas Arnold-Medabalimi

Ph.D. Candidate
Computational Aeroscience Lab
University of Michigan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220802/0a5cf098/attachment.html>

From bramkamp at nsc.liu.se  Tue Aug  2 12:18:56 2022
From: bramkamp at nsc.liu.se (Frank Bramkamp)
Date: Tue, 2 Aug 2022 19:18:56 +0200
Subject: [petsc-users] KSPAGMRES Question
Message-ID: <6F46D695-9CB9-41BC-BFA0-A567162E8C35@nsc.liu.se>

Dear PETSc team,

I have seen that there is the KSP method: KSPAGMRES,
https://petsc.org/release/docs/manualpages/KSP/KSPAGMRES.html <https://petsc.org/release/docs/manualpages/KSP/KSPAGMRES.html>

I wanted to test this method, as it also seems to reduce the amount of MPI communication, compared
to the standard GMRES. 


I supposed that the class is called ?KSPAGMRES?.
But in the include files petscksp.h and petsc/finclude/petscksp.h 
there is no definition for KSPAGMRES, just KSPDGMRES.

I wonder if the definition KSPAGMRES is simply missing, or do I have to call
DGMRES and set another option for AGMRES ?!

The standard GMRES has the problem that MPI_Allreduce gets expensive for 2048 cores.
Therefore I wanted to see, if AGMRES has a bit less communication, as this is mentioned in the description
of the method. 

Greetings, Frank Bramkamp



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220802/53c1f286/attachment.html>

From narnoldm at umich.edu  Tue Aug  2 15:55:23 2022
From: narnoldm at umich.edu (Nicholas Arnold-Medabalimi)
Date: Tue, 2 Aug 2022 16:55:23 -0400
Subject: [petsc-users] DMPlex total point counts
Message-ID: <CAB1dcV4KY9XGeYKsHr=avvyGOf53OMfGbfLMZUH=F0-Orn-4WQ@mail.gmail.com>

Hi

Currently, in my distributed DMPlex object, every so often, I want to look
at total cell counts. To do this in my current implementation, I get the
cell stratum from the DMplex and search through the PetscSF to remove any
overlapped (not owned) cells and then reduce over all the ranks to get the
total cell count. Is there a more graceful way of doing this already built
into DM?

Sincerely
Nicholas

-- 
Nicholas Arnold-Medabalimi

Ph.D. Candidate
Computational Aeroscience Lab
University of Michigan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220802/3084e4ef/attachment.html>

From bsmith at petsc.dev  Tue Aug  2 21:17:47 2022
From: bsmith at petsc.dev (Barry Smith)
Date: Tue, 2 Aug 2022 21:17:47 -0500
Subject: [petsc-users] KSPAGMRES Question
In-Reply-To: <6F46D695-9CB9-41BC-BFA0-A567162E8C35@nsc.liu.se>
References: <6F46D695-9CB9-41BC-BFA0-A567162E8C35@nsc.liu.se>
Message-ID: <54BDF6E6-1095-4734-BB38-C2B22A3D4189@petsc.dev>


  Ok, this is embarrassing, all the code is there and the documentation, but it is not wired up to be usable; I'm not sure how that happened. The code was contributed over two years ago, and I have no memory of why it would be there but not wired up. I wired it up and ran a simple test and got MPI errors, so I suspect that the code had serious bugs we did not understand and so was not wired up; 

  There are other pipelined GMRES KSPPGMRES and KSPPIPEFGMRES that are wired up and should be usable out of the box. Plus a pipelined bcgs. I suggest starting by trying those.



  Barry


> On Aug 2, 2022, at 12:18 PM, Frank Bramkamp <bramkamp at nsc.liu.se> wrote:
> 
> Dear PETSc team,
> 
> I have seen that there is the KSP method: KSPAGMRES,
> https://petsc.org/release/docs/manualpages/KSP/KSPAGMRES.html <https://petsc.org/release/docs/manualpages/KSP/KSPAGMRES.html>
> 
> I wanted to test this method, as it also seems to reduce the amount of MPI communication, compared
> to the standard GMRES. 
> 
> 
> I supposed that the class is called ?KSPAGMRES?.
> But in the include files petscksp.h and petsc/finclude/petscksp.h 
> there is no definition for KSPAGMRES, just KSPDGMRES.
> 
> I wonder if the definition KSPAGMRES is simply missing, or do I have to call
> DGMRES and set another option for AGMRES ?!
> 
> The standard GMRES has the problem that MPI_Allreduce gets expensive for 2048 cores.
> Therefore I wanted to see, if AGMRES has a bit less communication, as this is mentioned in the description
> of the method. 
> 
> Greetings, Frank Bramkamp
> 
> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220802/942f36db/attachment.html>

From toby.isaac at gmail.com  Tue Aug  2 21:18:10 2022
From: toby.isaac at gmail.com (Toby Isaac)
Date: Tue, 2 Aug 2022 21:18:10 -0500
Subject: [petsc-users] DMPlex total point counts
In-Reply-To: <CAB1dcV4KY9XGeYKsHr=avvyGOf53OMfGbfLMZUH=F0-Orn-4WQ@mail.gmail.com>
References: <CAB1dcV4KY9XGeYKsHr=avvyGOf53OMfGbfLMZUH=F0-Orn-4WQ@mail.gmail.com>
Message-ID: <CAC2U1y_rTqmcR_gqmqKXjjog99Bo793wrOF4yyrTAfvJ7UF+Xg@mail.gmail.com>

I don't know about graceful, but I don't think there is a more
efficient approach than what you have chosen.

On Tue, Aug 2, 2022 at 4:00 PM Nicholas Arnold-Medabalimi
<narnoldm at umich.edu> wrote:
>
> Hi
>
> Currently, in my distributed DMPlex object, every so often, I want to look at total cell counts. To do this in my current implementation, I get the cell stratum from the DMplex and search through the PetscSF to remove any overlapped (not owned) cells and then reduce over all the ranks to get the total cell count. Is there a more graceful way of doing this already built into DM?
>
> Sincerely
> Nicholas
>
> --
> Nicholas Arnold-Medabalimi
>
> Ph.D. Candidate
> Computational Aeroscience Lab
> University of Michigan

From wangxq2020 at mail.ustc.edu.cn  Wed Aug  3 03:01:35 2022
From: wangxq2020 at mail.ustc.edu.cn (wangxq2020 at mail.ustc.edu.cn)
Date: Wed, 3 Aug 2022 16:01:35 +0800 (GMT+08:00)
Subject: [petsc-users] Some issues about DMDA and MatSetValuesStencil
Message-ID: <2d5e9812.23151.18262b96c30.Coremail.wangxq2020@mail.ustc.edu.cn>

Hello!
I am a beginner about petsc?and I'm studing the DMDA recently. I hava a exercise code below, i run it with 4 processes? why is the size of A matrix is 15?15? The user manual said A is the Jobian matix, is there any detailed material that explain the process? Another question is the setting values for A. I don't understand why a loop using the grid distribution information of each process can assign a value to A. The MatStencil mentioned in the user manual is a Data structure (C struct) for storing information about a single row or column of a matrix, why MatStencil contains i and j at the same time, and the row only needs i. Very much looking forward to reply?thanks.


  DM grid;
  ierr = DMDACreate2d            // IN:
    ( comm,                      // collective on this communicator
      DM_BOUNDARY_NONE,DM_BOUNDARY_NONE, // no periodicity and such
      DMDA_STENCIL_STAR,         // no cross connections
      4,4,                 // global size 100x100; can be changed with options
      PETSC_DECIDE,PETSC_DECIDE, // processors in each direction
      1,                         // degree of freedom per node
      1,                         // stencil width
      NULL,NULL,                 // arrays of local sizes in each direction
      &grid                      // OUT: resulting object
      ); CHKERRQ(ierr);
  ierr = DMSetUp(grid); CHKERRQ(ierr);


  Mat A;
  ierr = DMCreateMatrix(grid,&A); CHKERRQ(ierr);
  DMDALocalInfo  info;
  ierr = DMDAGetLocalInfo(grid,&info);CHKERRQ(ierr);
  for (int j=info.ys; j<info.ys+info.ym; j++) {
    for (int i=info.xs; i<info.xs+info.xm; i++) {
      MatStencil  row = {0},col[5] = {{0}};
      PetscScalar v[5];
      PetscInt    ncols = 0;
      row.j        = j; row.i = i;
      printf("procno: %d;j:%d; i:%d\n",procno,j,i);
      /**** local connection: diagonal element ****/
      col[ncols].j = j; col[ncols].i = i; v[ncols++] = 4.;
      printf("procno: %d, j:%d. col[%d].j = %d, col[%d].i = %d, v[%d]=4\n",procno,j,ncols-1,j,ncols-1,i,ncols);
      /* boundaries: top row */
      if (i>0)         {
        col[ncols].j = j;   col[ncols].i = i-1; v[ncols++] = -1.;
        printf("procno: %d, j:%d. col[%d].j = %d, col[%d].i = %d, v[%d]=4\n",procno,j,ncols-1,j,ncols-1,i-1,ncols);
        }
      if (j>0){
        col[ncols].j = j-1; col[ncols].i = i;   v[ncols++] = -1.;
        printf("procno: %d, j:%d. col[%d].j = %d, col[%d].i = %d, v[%d]=4\n",procno,j,ncols-1,j-1,ncols-1,i,ncols);
        }
      if (j<info.my-1) {
        col[ncols].j = j+1; col[ncols].i = i;   v[ncols++] = -1.;
        printf("procno: %d, j:%d. col[%d].j = %d, col[%d].i = %d, v[%d]=-1\n",procno,j,ncols-1,j+1,ncols-1,i+1,ncols);
        }


      /* boundary: bottom row */
      if (i<info.mx-1) {
        col[ncols].j = j;   col[ncols].i = i+1; v[ncols++] = -1.;
        printf("procno: %d, j:%d. col[%d].j = %d, col[%d].i = %d, v[%d]=-1\n",procno,j,ncols-1,j,ncols-1,i+1,ncols);
        }
      ierr = MatSetValuesStencil(A,1,&row,ncols,col,v,INSERT_VALUES);CHKERRQ(ierr);
    }
  }
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220803/5e0f03c4/attachment.html>

From knepley at gmail.com  Wed Aug  3 09:32:54 2022
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 3 Aug 2022 09:32:54 -0500
Subject: [petsc-users] DMPlex total point counts
In-Reply-To: <CAB1dcV4KY9XGeYKsHr=avvyGOf53OMfGbfLMZUH=F0-Orn-4WQ@mail.gmail.com>
References: <CAB1dcV4KY9XGeYKsHr=avvyGOf53OMfGbfLMZUH=F0-Orn-4WQ@mail.gmail.com>
Message-ID: <CAMYG4GkGCwXxwT9=G+D90V9DT0EZ1+TFZTa1PrA9xF3kOYMoTg@mail.gmail.com>

On Tue, Aug 2, 2022 at 4:00 PM Nicholas Arnold-Medabalimi <
narnoldm at umich.edu> wrote:

> Hi
>
> Currently, in my distributed DMPlex object, every so often, I want to look
> at total cell counts. To do this in my current implementation, I get the
> cell stratum from the DMplex and search through the PetscSF to remove any
> overlapped (not owned) cells and then reduce over all the ranks to get the
> total cell count. Is there a more graceful way of doing this already built
> into DM?
>

If all you want is the global cell count, this is probably the easiest way.
If you want a global numbering, you can use

  https://petsc.org/main/docs/manualpages/DMPLEX/DMPlexGetCellNumbering/

  Thanks,

     Matt


> Sincerely
> Nicholas
>
> --
> Nicholas Arnold-Medabalimi
>
> Ph.D. Candidate
> Computational Aeroscience Lab
> University of Michigan
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220803/f3cb2012/attachment.html>

From knepley at gmail.com  Wed Aug  3 09:39:45 2022
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 3 Aug 2022 09:39:45 -0500
Subject: [petsc-users] Some issues about DMDA and MatSetValuesStencil
In-Reply-To: <2d5e9812.23151.18262b96c30.Coremail.wangxq2020@mail.ustc.edu.cn>
References: <2d5e9812.23151.18262b96c30.Coremail.wangxq2020@mail.ustc.edu.cn>
Message-ID: <CAMYG4GnWmPiyb6pJe49j1_equ8XgjWKimaaga=u2bUD4qKTaJw@mail.gmail.com>

On Wed, Aug 3, 2022 at 9:17 AM wangxq2020--- via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Hello!
> I am a beginner about petsc?and I'm studing the DMDA recently. I hava a
> exercise code below, i run it with 4 processes? why is the size of A matrix
> is 15?15?
>

Are you sure it is not 16x16? The grid has 4 vertices in both the x- and
y-directions, so 16 vertices. You have 1 dof per vertex. This means that a
function
over this grid has 16 variables, so vectors have length 16. The Jacobian
relates the (linearized) change in the residual for a change in each
variables,
so it is a matrix of size 16 x 16.

  Thanks,

      Matt


> The user manual said A is the Jobian matix, is there any detailed material
> that explain the process? Another question is the setting values for A. I
> don't understand why a loop using the grid distribution information of each
> process can assign a value to A. The MatStencil mentioned in the user
> manual is a Data structure (C struct) for storing information about a
> single row or column of a matrix, why MatStencil contains i and j at the
> same time, and the row only needs i. Very much looking forward to
> reply?thanks.
>
>   DM grid;
>   ierr = DMDACreate2d            // IN:
>     ( comm,                      // collective on this communicator
>       DM_BOUNDARY_NONE,DM_BOUNDARY_NONE, // no periodicity and such
>       DMDA_STENCIL_STAR,         // no cross connections
>       4,4,                 // global size 100x100; can be changed with
> options
>       PETSC_DECIDE,PETSC_DECIDE, // processors in each direction
>       1,                         // degree of freedom per node
>       1,                         // stencil width
>       NULL,NULL,                 // arrays of local sizes in each direction
>       &grid                      // OUT: resulting object
>       ); CHKERRQ(ierr);
>   ierr = DMSetUp(grid); CHKERRQ(ierr);
>
>   Mat A;
>   ierr = DMCreateMatrix(grid,&A); CHKERRQ(ierr);
>   DMDALocalInfo  info;
>   ierr = DMDAGetLocalInfo(grid,&info);CHKERRQ(ierr);
>   for (int j=info.ys; j<info.ys+info.ym; j++) {
>     for (int i=info.xs; i<info.xs+info.xm; i++) {
>       MatStencil  row = {0},col[5] = {{0}};
>       PetscScalar v[5];
>       PetscInt    ncols = 0;
>       row.j        = j; row.i = i;
>       printf("procno: %d;j:%d; i:%d\n",procno,j,i);
>       /**** local connection: diagonal element ****/
>       col[ncols].j = j; col[ncols].i = i; v[ncols++] = 4.;
>       printf("procno: %d, j:%d. col[%d].j = %d, col[%d].i = %d,
> v[%d]=4\n",procno,j,ncols-1,j,ncols-1,i,ncols);
>       /* boundaries: top row */
>       if (i>0)         {
>         col[ncols].j = j;   col[ncols].i = i-1; v[ncols++] = -1.;
>         printf("procno: %d, j:%d. col[%d].j = %d, col[%d].i = %d,
> v[%d]=4\n",procno,j,ncols-1,j,ncols-1,i-1,ncols);
>         }
>       if (j>0){
>         col[ncols].j = j-1; col[ncols].i = i;   v[ncols++] = -1.;
>         printf("procno: %d, j:%d. col[%d].j = %d, col[%d].i = %d,
> v[%d]=4\n",procno,j,ncols-1,j-1,ncols-1,i,ncols);
>         }
>       if (j<info.my-1) {
>         col[ncols].j = j+1; col[ncols].i = i;   v[ncols++] = -1.;
>         printf("procno: %d, j:%d. col[%d].j = %d, col[%d].i = %d,
> v[%d]=-1\n",procno,j,ncols-1,j+1,ncols-1,i+1,ncols);
>         }
>
>       /* boundary: bottom row */
>       if (i<info.mx-1) {
>         col[ncols].j = j;   col[ncols].i = i+1; v[ncols++] = -1.;
>         printf("procno: %d, j:%d. col[%d].j = %d, col[%d].i = %d,
> v[%d]=-1\n",procno,j,ncols-1,j,ncols-1,i+1,ncols);
>         }
>       ierr =
> MatSetValuesStencil(A,1,&row,ncols,col,v,INSERT_VALUES);CHKERRQ(ierr);
>     }
>   }
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220803/0ea4081b/attachment-0001.html>

From SimonZhang at lbl.gov  Thu Aug  4 11:52:24 2022
From: SimonZhang at lbl.gov (Simon Zhang)
Date: Thu, 4 Aug 2022 09:52:24 -0700
Subject: [petsc-users] PETSc on GCC
Message-ID: <CADb1vSuvHOCCTk0Zc+XUExQqtEAYwODODqmhizoNs3U5aaCM_A@mail.gmail.com>

To whom it may concern,

I'm trying to install PETSc on my computer and PETSc is compliant with GCC
but something keeps going wrong at the compilation step. It's returning the
following message.

PETSc requires c99 compiler! Configure could not determine compatible
compiler flag.

How can I fix this problem?

Thank you,
Simon Zhang
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220804/9e333ffc/attachment.html>

From balay at mcs.anl.gov  Thu Aug  4 12:05:16 2022
From: balay at mcs.anl.gov (Satish Balay)
Date: Thu, 4 Aug 2022 12:05:16 -0500 (CDT)
Subject: [petsc-users] PETSc on GCC
In-Reply-To: <CADb1vSuvHOCCTk0Zc+XUExQqtEAYwODODqmhizoNs3U5aaCM_A@mail.gmail.com>
References: <CADb1vSuvHOCCTk0Zc+XUExQqtEAYwODODqmhizoNs3U5aaCM_A@mail.gmail.com>
Message-ID: <e960d078-8711-eb55-e9d7-7bb9e650d2d8@mcs.anl.gov>

please send the corresponding configure.log

Satish

On Thu, 4 Aug 2022, Simon Zhang wrote:

> To whom it may concern,
> 
> I'm trying to install PETSc on my computer and PETSc is compliant with GCC
> but something keeps going wrong at the compilation step. It's returning the
> following message.
> 
> PETSc requires c99 compiler! Configure could not determine compatible
> compiler flag.
> 
> How can I fix this problem?
> 
> Thank you,
> Simon Zhang
> 


From SimonZhang at lbl.gov  Thu Aug  4 12:38:28 2022
From: SimonZhang at lbl.gov (Simon Zhang)
Date: Thu, 4 Aug 2022 10:38:28 -0700
Subject: [petsc-users] PETSc on GCC
In-Reply-To: <e960d078-8711-eb55-e9d7-7bb9e650d2d8@mcs.anl.gov>
References: <CADb1vSuvHOCCTk0Zc+XUExQqtEAYwODODqmhizoNs3U5aaCM_A@mail.gmail.com>
	<e960d078-8711-eb55-e9d7-7bb9e650d2d8@mcs.anl.gov>
Message-ID: <CADb1vSucvzhXSnC8BXzze3XxBumKJcwmE0qwpOYkMTjcqJEDoQ@mail.gmail.com>

On Thu, Aug 4, 2022 at 10:05 AM Satish Balay <balay at mcs.anl.gov> wrote:

> please send the corresponding configure.log
>
> Satish
>
> On Thu, 4 Aug 2022, Simon Zhang wrote:
>
> > To whom it may concern,
> >
> > I'm trying to install PETSc on my computer and PETSc is compliant with
> GCC
> > but something keeps going wrong at the compilation step. It's returning
> the
> > following message.
> >
> > PETSc requires c99 compiler! Configure could not determine compatible
> > compiler flag.
> >
> > How can I fix this problem?
> >
> > Thank you,
> > Simon Zhang
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220804/02b013b1/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: configure.log
Type: text/x-log
Size: 188201 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220804/02b013b1/attachment-0001.bin>

From balay at mcs.anl.gov  Thu Aug  4 14:21:08 2022
From: balay at mcs.anl.gov (Satish Balay)
Date: Thu, 4 Aug 2022 14:21:08 -0500 (CDT)
Subject: [petsc-users] PETSc on GCC
In-Reply-To: <CADb1vSucvzhXSnC8BXzze3XxBumKJcwmE0qwpOYkMTjcqJEDoQ@mail.gmail.com>
References: <CADb1vSuvHOCCTk0Zc+XUExQqtEAYwODODqmhizoNs3U5aaCM_A@mail.gmail.com>
	<e960d078-8711-eb55-e9d7-7bb9e650d2d8@mcs.anl.gov>
	<CADb1vSucvzhXSnC8BXzze3XxBumKJcwmE0qwpOYkMTjcqJEDoQ@mail.gmail.com>
Message-ID: <1812bbc5-4cd7-58f2-342-95dc28a7e425@mcs.anl.gov>

> gcc (Ubuntu 4.8.4-2ubuntu1~14.04.4) 4.8.4

Do you have a newer gcc version available for use on this machine?

>>>>>>
/tmp/petsc-7thm2vwz/config.setCompilers/conftest.c: At top level:
cc1: warning: unrecognized command line option "-Wno-lto-type-mismatch" [enabled by default]

            Rejecting compiler flag -std=c99 due to
<<<<<<

Hm - looks configure logic here is failing.

-Wno-lto-type-mismatch checked went through earlier [so this flag was accepted] - but then when a test is retried with "-Wno-lto-type-mismatch -std=c99" its failing

Perhaps one way to work around is to use configure option: CFLAGS=-std=c99

But its better to use a more recent gcc than 4.8.4

Satish

On Thu, 4 Aug 2022, Simon Zhang wrote:

> On Thu, Aug 4, 2022 at 10:05 AM Satish Balay <balay at mcs.anl.gov> wrote:
> 
> > please send the corresponding configure.log
> >
> > Satish
> >
> > On Thu, 4 Aug 2022, Simon Zhang wrote:
> >
> > > To whom it may concern,
> > >
> > > I'm trying to install PETSc on my computer and PETSc is compliant with
> > GCC
> > > but something keeps going wrong at the compilation step. It's returning
> > the
> > > following message.
> > >
> > > PETSc requires c99 compiler! Configure could not determine compatible
> > > compiler flag.
> > >
> > > How can I fix this problem?
> > >
> > > Thank you,
> > > Simon Zhang
> > >
> >
> >
> 


From SimonZhang at lbl.gov  Thu Aug  4 15:46:16 2022
From: SimonZhang at lbl.gov (Simon Zhang)
Date: Thu, 4 Aug 2022 13:46:16 -0700
Subject: [petsc-users] PETSc on GCC
In-Reply-To: <1812bbc5-4cd7-58f2-342-95dc28a7e425@mcs.anl.gov>
References: <CADb1vSuvHOCCTk0Zc+XUExQqtEAYwODODqmhizoNs3U5aaCM_A@mail.gmail.com>
	<e960d078-8711-eb55-e9d7-7bb9e650d2d8@mcs.anl.gov>
	<CADb1vSucvzhXSnC8BXzze3XxBumKJcwmE0qwpOYkMTjcqJEDoQ@mail.gmail.com>
	<1812bbc5-4cd7-58f2-342-95dc28a7e425@mcs.anl.gov>
Message-ID: <CADb1vSvN=1b0+8WCgskV5csmV7Zo4jE2caDWY44dBZr9z10vUQ@mail.gmail.com>

I do have mpicc installed and have CFLAGS=-std=c99 but I'm still unable to
configure.

On Thu, Aug 4, 2022 at 12:21 PM Satish Balay <balay at mcs.anl.gov> wrote:

> > gcc (Ubuntu 4.8.4-2ubuntu1~14.04.4) 4.8.4
>
> Do you have a newer gcc version available for use on this machine?
>
> >>>>>>
> /tmp/petsc-7thm2vwz/config.setCompilers/conftest.c: At top level:
> cc1: warning: unrecognized command line option "-Wno-lto-type-mismatch"
> [enabled by default]
>
>             Rejecting compiler flag -std=c99 due to
> <<<<<<
>
> Hm - looks configure logic here is failing.
>
> -Wno-lto-type-mismatch checked went through earlier [so this flag was
> accepted] - but then when a test is retried with "-Wno-lto-type-mismatch
> -std=c99" its failing
>
> Perhaps one way to work around is to use configure option: CFLAGS=-std=c99
>
> But its better to use a more recent gcc than 4.8.4
>
> Satish
>
> On Thu, 4 Aug 2022, Simon Zhang wrote:
>
> > On Thu, Aug 4, 2022 at 10:05 AM Satish Balay <balay at mcs.anl.gov> wrote:
> >
> > > please send the corresponding configure.log
> > >
> > > Satish
> > >
> > > On Thu, 4 Aug 2022, Simon Zhang wrote:
> > >
> > > > To whom it may concern,
> > > >
> > > > I'm trying to install PETSc on my computer and PETSc is compliant
> with
> > > GCC
> > > > but something keeps going wrong at the compilation step. It's
> returning
> > > the
> > > > following message.
> > > >
> > > > PETSc requires c99 compiler! Configure could not determine compatible
> > > > compiler flag.
> > > >
> > > > How can I fix this problem?
> > > >
> > > > Thank you,
> > > > Simon Zhang
> > > >
> > >
> > >
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220804/844cc71a/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: configure.log
Type: text/x-log
Size: 640285 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220804/844cc71a/attachment-0001.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Screenshot from 2022-08-04 13-41-19.png
Type: image/png
Size: 341469 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220804/844cc71a/attachment-0001.png>

From balay at mcs.anl.gov  Thu Aug  4 15:51:49 2022
From: balay at mcs.anl.gov (Satish Balay)
Date: Thu, 4 Aug 2022 15:51:49 -0500 (CDT)
Subject: [petsc-users] PETSc on GCC
In-Reply-To: <CADb1vSvN=1b0+8WCgskV5csmV7Zo4jE2caDWY44dBZr9z10vUQ@mail.gmail.com>
References: <CADb1vSuvHOCCTk0Zc+XUExQqtEAYwODODqmhizoNs3U5aaCM_A@mail.gmail.com>
	<e960d078-8711-eb55-e9d7-7bb9e650d2d8@mcs.anl.gov>
	<CADb1vSucvzhXSnC8BXzze3XxBumKJcwmE0qwpOYkMTjcqJEDoQ@mail.gmail.com>
	<1812bbc5-4cd7-58f2-342-95dc28a7e425@mcs.anl.gov>
	<CADb1vSvN=1b0+8WCgskV5csmV7Zo4jE2caDWY44dBZr9z10vUQ@mail.gmail.com>
Message-ID: <552333ce-85e-13cb-b823-ab14497b05@mcs.anl.gov>

> Configure Options: --configModules=PETSc.Configure --optionsModule=config.compilerOptions CFLAGS=-std=c99 --with-cc=mpicc --with-cxx=g++ --download-fblaslapack=yes --download-hypre=yes -with-x=0 --with-c++support=yes --with-mpi=yes --with-hypre=yes --prefix=/petsc --with-c2html=0 --with-ssl=0

Try:

./configure CFLAGS=-std=c99 --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 --download-fblaslapack=yes --download-hypre=yes -with-x=0 --with-c2html=0 --with-ssl=0

Satish

On Thu, 4 Aug 2022, Simon Zhang wrote:

> I do have mpicc installed and have CFLAGS=-std=c99 but I'm still unable to
> configure.
> 
> On Thu, Aug 4, 2022 at 12:21 PM Satish Balay <balay at mcs.anl.gov> wrote:
> 
> > > gcc (Ubuntu 4.8.4-2ubuntu1~14.04.4) 4.8.4
> >
> > Do you have a newer gcc version available for use on this machine?
> >
> > >>>>>>
> > /tmp/petsc-7thm2vwz/config.setCompilers/conftest.c: At top level:
> > cc1: warning: unrecognized command line option "-Wno-lto-type-mismatch"
> > [enabled by default]
> >
> >             Rejecting compiler flag -std=c99 due to
> > <<<<<<
> >
> > Hm - looks configure logic here is failing.
> >
> > -Wno-lto-type-mismatch checked went through earlier [so this flag was
> > accepted] - but then when a test is retried with "-Wno-lto-type-mismatch
> > -std=c99" its failing
> >
> > Perhaps one way to work around is to use configure option: CFLAGS=-std=c99
> >
> > But its better to use a more recent gcc than 4.8.4
> >
> > Satish
> >
> > On Thu, 4 Aug 2022, Simon Zhang wrote:
> >
> > > On Thu, Aug 4, 2022 at 10:05 AM Satish Balay <balay at mcs.anl.gov> wrote:
> > >
> > > > please send the corresponding configure.log
> > > >
> > > > Satish
> > > >
> > > > On Thu, 4 Aug 2022, Simon Zhang wrote:
> > > >
> > > > > To whom it may concern,
> > > > >
> > > > > I'm trying to install PETSc on my computer and PETSc is compliant
> > with
> > > > GCC
> > > > > but something keeps going wrong at the compilation step. It's
> > returning
> > > > the
> > > > > following message.
> > > > >
> > > > > PETSc requires c99 compiler! Configure could not determine compatible
> > > > > compiler flag.
> > > > >
> > > > > How can I fix this problem?
> > > > >
> > > > > Thank you,
> > > > > Simon Zhang
> > > > >
> > > >
> > > >
> > >
> >
> >
> 


From rlmackie862 at gmail.com  Thu Aug  4 15:53:04 2022
From: rlmackie862 at gmail.com (Randall Mackie)
Date: Thu, 4 Aug 2022 13:53:04 -0700
Subject: [petsc-users] PETSc on GCC
In-Reply-To: <552333ce-85e-13cb-b823-ab14497b05@mcs.anl.gov>
References: <CADb1vSuvHOCCTk0Zc+XUExQqtEAYwODODqmhizoNs3U5aaCM_A@mail.gmail.com>
	<e960d078-8711-eb55-e9d7-7bb9e650d2d8@mcs.anl.gov>
	<CADb1vSucvzhXSnC8BXzze3XxBumKJcwmE0qwpOYkMTjcqJEDoQ@mail.gmail.com>
	<1812bbc5-4cd7-58f2-342-95dc28a7e425@mcs.anl.gov>
	<CADb1vSvN=1b0+8WCgskV5csmV7Zo4jE2caDWY44dBZr9z10vUQ@mail.gmail.com>
	<552333ce-85e-13cb-b823-ab14497b05@mcs.anl.gov>
Message-ID: <F047C509-EF16-45EE-AD62-999F7AFC5167@gmail.com>

Hi Simon,

I think you might actually need CFLAGS=?-std=gnu99?

Randy M.


> On Aug 4, 2022, at 1:51 PM, Satish Balay via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
>> Configure Options: --configModules=PETSc.Configure --optionsModule=config.compilerOptions CFLAGS=-std=c99 --with-cc=mpicc --with-cxx=g++ --download-fblaslapack=yes --download-hypre=yes -with-x=0 --with-c++support=yes --with-mpi=yes --with-hypre=yes --prefix=/petsc --with-c2html=0 --with-ssl=0
> 
> Try:
> 
> ./configure CFLAGS=-std=c99 --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 --download-fblaslapack=yes --download-hypre=yes -with-x=0 --with-c2html=0 --with-ssl=0
> 
> Satish
> 
> On Thu, 4 Aug 2022, Simon Zhang wrote:
> 
>> I do have mpicc installed and have CFLAGS=-std=c99 but I'm still unable to
>> configure.
>> 
>> On Thu, Aug 4, 2022 at 12:21 PM Satish Balay <balay at mcs.anl.gov> wrote:
>> 
>>>> gcc (Ubuntu 4.8.4-2ubuntu1~14.04.4) 4.8.4
>>> 
>>> Do you have a newer gcc version available for use on this machine?
>>> 
>>>>>>>>> 
>>> /tmp/petsc-7thm2vwz/config.setCompilers/conftest.c: At top level:
>>> cc1: warning: unrecognized command line option "-Wno-lto-type-mismatch"
>>> [enabled by default]
>>> 
>>>            Rejecting compiler flag -std=c99 due to
>>> <<<<<<
>>> 
>>> Hm - looks configure logic here is failing.
>>> 
>>> -Wno-lto-type-mismatch checked went through earlier [so this flag was
>>> accepted] - but then when a test is retried with "-Wno-lto-type-mismatch
>>> -std=c99" its failing
>>> 
>>> Perhaps one way to work around is to use configure option: CFLAGS=-std=c99
>>> 
>>> But its better to use a more recent gcc than 4.8.4
>>> 
>>> Satish
>>> 
>>> On Thu, 4 Aug 2022, Simon Zhang wrote:
>>> 
>>>> On Thu, Aug 4, 2022 at 10:05 AM Satish Balay <balay at mcs.anl.gov> wrote:
>>>> 
>>>>> please send the corresponding configure.log
>>>>> 
>>>>> Satish
>>>>> 
>>>>> On Thu, 4 Aug 2022, Simon Zhang wrote:
>>>>> 
>>>>>> To whom it may concern,
>>>>>> 
>>>>>> I'm trying to install PETSc on my computer and PETSc is compliant
>>> with
>>>>> GCC
>>>>>> but something keeps going wrong at the compilation step. It's
>>> returning
>>>>> the
>>>>>> following message.
>>>>>> 
>>>>>> PETSc requires c99 compiler! Configure could not determine compatible
>>>>>> compiler flag.
>>>>>> 
>>>>>> How can I fix this problem?
>>>>>> 
>>>>>> Thank you,
>>>>>> Simon Zhang
>>>>>> 
>>>>> 
>>>>> 
>>>> 
>>> 
>>> 
>> 
> 


From bantingl at myumanitoba.ca  Thu Aug  4 15:55:50 2022
From: bantingl at myumanitoba.ca (Lucas Banting)
Date: Thu, 4 Aug 2022 20:55:50 +0000
Subject: [petsc-users] Question about speed of MatMatSolveTranspose with
 MUMPS
Message-ID: <YTBPR01MB2448E9CD564153D9D75F90ACB19F9@YTBPR01MB2448.CANPRD01.PROD.OUTLOOK.COM>

Hello,

I am using MUMPS to solve a system with multiple right hand sides.
I have noticed that the MatMatSolve function appears to be quite a bit faster than the MatMatSolveTranspose function.
I was trying to look through the petsc source to see where the MatMatSolveTranspose_MUMPS function was but couldn't find it.

Is PETSc just doing MatSolveTranspose_MUMPS for each right hand side when doing MatMatSolveTranspose with MUMPS?
If an accelerated version of MatMatSolveTranspose existed, it would be very useful for my application.

Regards,
Lucas

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220804/a6bb5ad5/attachment-0001.html>

From bsmith at petsc.dev  Thu Aug  4 16:20:18 2022
From: bsmith at petsc.dev (Barry Smith)
Date: Thu, 4 Aug 2022 16:20:18 -0500
Subject: [petsc-users] Question about speed of MatMatSolveTranspose with
 MUMPS
In-Reply-To: <YTBPR01MB2448E9CD564153D9D75F90ACB19F9@YTBPR01MB2448.CANPRD01.PROD.OUTLOOK.COM>
References: <YTBPR01MB2448E9CD564153D9D75F90ACB19F9@YTBPR01MB2448.CANPRD01.PROD.OUTLOOK.COM>
Message-ID: <77E3BA05-765A-4012-9053-8D98E254C12D@petsc.dev>


  Yes, it appears no MUMPS matmatsolvetranspose is wired in. Does MUMPS provide such a function? If so it can be added with a few lines of code.

  Barry


> On Aug 4, 2022, at 3:55 PM, Lucas Banting <bantingl at myumanitoba.ca> wrote:
> 
> Hello,
> 
> I am using MUMPS to solve a system with multiple right hand sides.
> I have noticed that the MatMatSolve function appears to be quite a bit faster than the MatMatSolveTranspose function.
> I was trying to look through the petsc source to see where the MatMatSolveTranspose_MUMPS function was but couldn't find it.
> 
> Is PETSc just doing MatSolveTranspose_MUMPS for each right hand side when doing MatMatSolveTranspose with MUMPS?
> If an accelerated version of MatMatSolveTranspose existed, it would be very useful for my application.
> 
> Regards,
> Lucas

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220804/117bc4e5/attachment.html>

From junchao.zhang at gmail.com  Thu Aug  4 17:21:02 2022
From: junchao.zhang at gmail.com (Junchao Zhang)
Date: Thu, 4 Aug 2022 17:21:02 -0500
Subject: [petsc-users] Question about speed of MatMatSolveTranspose with
 MUMPS
In-Reply-To: <77E3BA05-765A-4012-9053-8D98E254C12D@petsc.dev>
References: <YTBPR01MB2448E9CD564153D9D75F90ACB19F9@YTBPR01MB2448.CANPRD01.PROD.OUTLOOK.COM>
	<77E3BA05-765A-4012-9053-8D98E254C12D@petsc.dev>
Message-ID: <CA+MQGp8zoLCBme9kRdz+9=bHB-dep0Hzddq9uf1n89=NYVrDyw@mail.gmail.com>

It seems yes, from mump manual


ICNTL(9) computes the solution using A or AT

Phase: accessed by the host during the solve phase.

Possible values :

1    : AX = B is solved.

!= 1 : ATX = B is solved.



On Thu, Aug 4, 2022 at 4:20 PM Barry Smith <bsmith at petsc.dev> wrote:

>
>   Yes, it appears no MUMPS matmatsolvetranspose is wired in. Does MUMPS
> provide such a function? If so it can be added with a few lines of code.
>
>   Barry
>
>
> On Aug 4, 2022, at 3:55 PM, Lucas Banting <bantingl at myumanitoba.ca> wrote:
>
> Hello,
>
> I am using MUMPS to solve a system with multiple right hand sides.
> I have noticed that the MatMatSolve function appears to be quite a bit
> faster than the MatMatSolveTranspose function.
> I was trying to look through the petsc source to see where the
> MatMatSolveTranspose_MUMPS function was but couldn't find it.
>
> Is PETSc just doing MatSolveTranspose_MUMPS for each right hand side when
> doing MatMatSolveTranspose with MUMPS?
> If an accelerated version of MatMatSolveTranspose existed, it would be
> very useful for my application.
>
> Regards,
> Lucas
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220804/dab9c6a5/attachment.html>

From hzhang at mcs.anl.gov  Thu Aug  4 18:02:21 2022
From: hzhang at mcs.anl.gov (Zhang, Hong)
Date: Thu, 4 Aug 2022 23:02:21 +0000
Subject: [petsc-users] Question about speed of MatMatSolveTranspose with
 MUMPS
In-Reply-To: <CA+MQGp8zoLCBme9kRdz+9=bHB-dep0Hzddq9uf1n89=NYVrDyw@mail.gmail.com>
References: <YTBPR01MB2448E9CD564153D9D75F90ACB19F9@YTBPR01MB2448.CANPRD01.PROD.OUTLOOK.COM>
	<77E3BA05-765A-4012-9053-8D98E254C12D@petsc.dev>
	<CA+MQGp8zoLCBme9kRdz+9=bHB-dep0Hzddq9uf1n89=NYVrDyw@mail.gmail.com>
Message-ID: <SA1PR09MB86071ABEED395775E2A1D370889F9@SA1PR09MB8607.namprd09.prod.outlook.com>

Lucas,
Solving
A X = B^T  #MatMatTransposeSolve() -- supported by petsc/mumps interface
or
A^T X = B #MatTransposeMatSolve() -- not supported
which one do you want ?
Hong
________________________________
From: petsc-users <petsc-users-bounces at mcs.anl.gov> on behalf of Junchao Zhang <junchao.zhang at gmail.com>
Sent: Thursday, August 4, 2022 5:21 PM
To: Barry Smith <bsmith at petsc.dev>
Cc: petsc-users at mcs.anl.gov <petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] Question about speed of MatMatSolveTranspose with MUMPS


It seems yes, from mump manual


ICNTL(9) computes the solution using A or AT

Phase: accessed by the host during the solve phase.

Possible values :

1    : AX = B is solved.

!= 1 : ATX = B is solved.


On Thu, Aug 4, 2022 at 4:20 PM Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>> wrote:

  Yes, it appears no MUMPS matmatsolvetranspose is wired in. Does MUMPS provide such a function? If so it can be added with a few lines of code.

  Barry


On Aug 4, 2022, at 3:55 PM, Lucas Banting <bantingl at myumanitoba.ca<mailto:bantingl at myumanitoba.ca>> wrote:

Hello,

I am using MUMPS to solve a system with multiple right hand sides.
I have noticed that the MatMatSolve function appears to be quite a bit faster than the MatMatSolveTranspose function.
I was trying to look through the petsc source to see where the MatMatSolveTranspose_MUMPS function was but couldn't find it.

Is PETSc just doing MatSolveTranspose_MUMPS for each right hand side when doing MatMatSolveTranspose with MUMPS?
If an accelerated version of MatMatSolveTranspose existed, it would be very useful for my application.

Regards,
Lucas

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220804/b5891c4b/attachment-0001.html>

From bantingl at myumanitoba.ca  Thu Aug  4 19:29:25 2022
From: bantingl at myumanitoba.ca (Lucas Banting)
Date: Fri, 5 Aug 2022 00:29:25 +0000
Subject: [petsc-users] Question about speed of MatMatSolveTranspose with
 MUMPS
In-Reply-To: <SA1PR09MB86071ABEED395775E2A1D370889F9@SA1PR09MB8607.namprd09.prod.outlook.com>
References: <YTBPR01MB2448E9CD564153D9D75F90ACB19F9@YTBPR01MB2448.CANPRD01.PROD.OUTLOOK.COM>
	<77E3BA05-765A-4012-9053-8D98E254C12D@petsc.dev>
	<CA+MQGp8zoLCBme9kRdz+9=bHB-dep0Hzddq9uf1n89=NYVrDyw@mail.gmail.com>
	<SA1PR09MB86071ABEED395775E2A1D370889F9@SA1PR09MB8607.namprd09.prod.outlook.com>
Message-ID: <YTBPR01MB24481D2156365B5CA4863681B19E9@YTBPR01MB2448.CANPRD01.PROD.OUTLOOK.COM>

Hi Hong,

The second option, I want to solve A^T X = B.
The MatMatSolveTranspose() function in petsc.

Lucas
________________________________
From: petsc-users <petsc-users-bounces at mcs.anl.gov> on behalf of Zhang, Hong via petsc-users <petsc-users at mcs.anl.gov>
Sent: Thursday, August 4, 2022 6:02:21 PM
To: Junchao Zhang <junchao.zhang at gmail.com>; Barry Smith <bsmith at petsc.dev>
Cc: petsc-users at mcs.anl.gov <petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] Question about speed of MatMatSolveTranspose with MUMPS

Caution: This message was sent from outside the University of Manitoba.

Lucas,
Solving
A X = B^T  #MatMatTransposeSolve() -- supported by petsc/mumps interface
or
A^T X = B #MatTransposeMatSolve() -- not supported
which one do you want ?
Hong
________________________________
From: petsc-users <petsc-users-bounces at mcs.anl.gov> on behalf of Junchao Zhang <junchao.zhang at gmail.com>
Sent: Thursday, August 4, 2022 5:21 PM
To: Barry Smith <bsmith at petsc.dev>
Cc: petsc-users at mcs.anl.gov <petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] Question about speed of MatMatSolveTranspose with MUMPS


It seems yes, from mump manual


ICNTL(9) computes the solution using A or AT

Phase: accessed by the host during the solve phase.

Possible values :

1    : AX = B is solved.

!= 1 : ATX = B is solved.


On Thu, Aug 4, 2022 at 4:20 PM Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>> wrote:

  Yes, it appears no MUMPS matmatsolvetranspose is wired in. Does MUMPS provide such a function? If so it can be added with a few lines of code.

  Barry


On Aug 4, 2022, at 3:55 PM, Lucas Banting <bantingl at myumanitoba.ca<mailto:bantingl at myumanitoba.ca>> wrote:

Hello,

I am using MUMPS to solve a system with multiple right hand sides.
I have noticed that the MatMatSolve function appears to be quite a bit faster than the MatMatSolveTranspose function.
I was trying to look through the petsc source to see where the MatMatSolveTranspose_MUMPS function was but couldn't find it.

Is PETSc just doing MatSolveTranspose_MUMPS for each right hand side when doing MatMatSolveTranspose with MUMPS?
If an accelerated version of MatMatSolveTranspose existed, it would be very useful for my application.

Regards,
Lucas

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220805/249994fb/attachment.html>

From hzhang at mcs.anl.gov  Thu Aug  4 21:59:32 2022
From: hzhang at mcs.anl.gov (Zhang, Hong)
Date: Fri, 5 Aug 2022 02:59:32 +0000
Subject: [petsc-users] Question about speed of MatMatSolveTranspose with
 MUMPS
In-Reply-To: <YTBPR01MB24481D2156365B5CA4863681B19E9@YTBPR01MB2448.CANPRD01.PROD.OUTLOOK.COM>
References: <YTBPR01MB2448E9CD564153D9D75F90ACB19F9@YTBPR01MB2448.CANPRD01.PROD.OUTLOOK.COM>
	<77E3BA05-765A-4012-9053-8D98E254C12D@petsc.dev>
	<CA+MQGp8zoLCBme9kRdz+9=bHB-dep0Hzddq9uf1n89=NYVrDyw@mail.gmail.com>
	<SA1PR09MB86071ABEED395775E2A1D370889F9@SA1PR09MB8607.namprd09.prod.outlook.com>
	<YTBPR01MB24481D2156365B5CA4863681B19E9@YTBPR01MB2448.CANPRD01.PROD.OUTLOOK.COM>
Message-ID: <SA1PR09MB8607432D8AE6401E5E32DAA8889E9@SA1PR09MB8607.namprd09.prod.outlook.com>

Lucas,
I added MatMatSolveTranspose_MUMPS()
https://gitlab.com/petsc/petsc/-/merge_requests/5500
[https://gitlab.com/uploads/-/system/project/avatar/13882401/PETSc_RBG-logo.png]<https://gitlab.com/petsc/petsc/-/merge_requests/5500>
add MatMatSolveTranspose_MUMPS() (!5500) ? Merge requests ? PETSc / petsc ? GitLab<https://gitlab.com/petsc/petsc/-/merge_requests/5500>
Add MatMatSolveTranspose_MUMPS() Requested-by: Lucas Banting bantingl at myumanitoba.ca
gitlab.com
It is in branch hzhang/add-MatMatSolveTranspose_MUMPS. You may give it a try.

MatMatSolveTranspose() is a bad name for solving A^t X = B.
We should rename it as MatTransposeMatSolve().
Hong
________________________________
From: Lucas Banting <bantingl at myumanitoba.ca>
Sent: Thursday, August 4, 2022 7:29 PM
To: Zhang, Hong <hzhang at mcs.anl.gov>; Junchao Zhang <junchao.zhang at gmail.com>; Barry Smith <bsmith at petsc.dev>
Cc: petsc-users at mcs.anl.gov <petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] Question about speed of MatMatSolveTranspose with MUMPS

Hi Hong,

The second option, I want to solve A^T X = B.
The MatMatSolveTranspose() function in petsc.

Lucas
________________________________
From: petsc-users <petsc-users-bounces at mcs.anl.gov> on behalf of Zhang, Hong via petsc-users <petsc-users at mcs.anl.gov>
Sent: Thursday, August 4, 2022 6:02:21 PM
To: Junchao Zhang <junchao.zhang at gmail.com>; Barry Smith <bsmith at petsc.dev>
Cc: petsc-users at mcs.anl.gov <petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] Question about speed of MatMatSolveTranspose with MUMPS

Caution: This message was sent from outside the University of Manitoba.

Lucas,
Solving
A X = B^T  #MatMatTransposeSolve() -- supported by petsc/mumps interface
or
A^T X = B #MatTransposeMatSolve() -- not supported
which one do you want ?
Hong
________________________________
From: petsc-users <petsc-users-bounces at mcs.anl.gov> on behalf of Junchao Zhang <junchao.zhang at gmail.com>
Sent: Thursday, August 4, 2022 5:21 PM
To: Barry Smith <bsmith at petsc.dev>
Cc: petsc-users at mcs.anl.gov <petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] Question about speed of MatMatSolveTranspose with MUMPS


It seems yes, from mump manual


ICNTL(9) computes the solution using A or AT

Phase: accessed by the host during the solve phase.

Possible values :

1    : AX = B is solved.

!= 1 : ATX = B is solved.


On Thu, Aug 4, 2022 at 4:20 PM Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>> wrote:

  Yes, it appears no MUMPS matmatsolvetranspose is wired in. Does MUMPS provide such a function? If so it can be added with a few lines of code.

  Barry


On Aug 4, 2022, at 3:55 PM, Lucas Banting <bantingl at myumanitoba.ca<mailto:bantingl at myumanitoba.ca>> wrote:

Hello,

I am using MUMPS to solve a system with multiple right hand sides.
I have noticed that the MatMatSolve function appears to be quite a bit faster than the MatMatSolveTranspose function.
I was trying to look through the petsc source to see where the MatMatSolveTranspose_MUMPS function was but couldn't find it.

Is PETSc just doing MatSolveTranspose_MUMPS for each right hand side when doing MatMatSolveTranspose with MUMPS?
If an accelerated version of MatMatSolveTranspose existed, it would be very useful for my application.

Regards,
Lucas

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220805/ff5662e7/attachment-0001.html>

From patrick.alken at geomag.info  Thu Aug  4 22:04:31 2022
From: patrick.alken at geomag.info (Patrick Alken)
Date: Thu, 4 Aug 2022 21:04:31 -0600
Subject: [petsc-users] [SLEPc] generalized symmetric eigensystem
Message-ID: <3c217462-cc35-0d2b-9c7b-88bd6f2f0b3f@geomag.info>

I am trying to solve a generalized symmetric eigensystem problem using 
SLEPc, and I am getting much different eigenvalues compared with the 
dense LAPACK solver DSYGVD. I have attached a minimal working example 
(MWE). The code relies on the external libraries GSL, Lapack and Lapacke.

The matrices (A,B) come from discretizing the Laplacian in spherical 
coordinates using a radial basis function method. Here are the 
eigenvalues computed by the dense LAPACK solver:

=== LAPACK eigenvalues ===
eval(0) = 1.309567289750e+00
eval(1) = 1.648674331144e+00
eval(2) = 1.709879393690e+00
eval(3) = 1.709879393690e+00
eval(4) = 2.142210251638e+00
eval(5) = 2.142210251638e+00
eval(6) = 2.145726457896e+00
eval(7) = 2.216526311324e+00
eval(8) = 2.216526311324e+00
eval(9) = 2.695837030236e+00


Here are the eigenvalues computed by SLEPc, using the command:

$ ./mwe -eps_nev 10 -eps_smallest_magnitude -eps_tol 1e-8

Number of requested eigenvalues: 10
 ?Linear eigensolve converged (10 eigenpairs) due to CONVERGED_TOL; 
iterations 196
 ?---------------------- --------------------
 ??????????? k???????????? ||Ax-kBx||/||kx||
 ?---------------------- --------------------
 ??????? 0.094520??????????????? 906.031
 ??????? 0.177268??????????????? 363.291
 ??????? 1.324501??????????????? 61.9577
 ??????? 1.786472??????????????? 35.7082
 ??????? 2.187857??????????????? 32.8364
 ?????? -2.886905??????????????? 27.7314
 ??????? 2.899206??????????????? 24.2224
 ??????? 4.195222??????????????? 20.3007
 ??????? 4.787192??????????????? 12.8346
 ??????? 7.221589??????????????? 9.58513
 ?---------------------- --------------------

As we can see, the results are quite different than LAPACK and the error 
norms in the right column are quite large. Using a stricter tolerance 
(eps_tol) does not seem to help. Can anyone suggest how to diagnose and 
fix the problem?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: laplace3d.c
Type: text/x-csrc
Size: 15082 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220804/0333098a/attachment.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: le_laplace3d.h
Type: text/x-chdr
Size: 1045 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220804/0333098a/attachment-0001.bin>
-------------- next part --------------
default: mwe

include ${SLEPC_DIR}/lib/slepc/conf/slepc_common

lib_dir = /home/palken/usr/lib
common_libs = -L${lib_dir} ${lib_dir}/libgsl.a -lm ${lib_dir}/liblapacke.a ${lib_dir}/liblapack.a

OBJECTS = mwe.o laplace3d.o

mwe: ${OBJECTS}
	${CLINKER} -o mwe ${OBJECTS} ${SLEPC_EPS_LIB} ${common_libs}
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mwe.c
Type: text/x-csrc
Size: 3374 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220804/0333098a/attachment-0002.bin>

From pierre at joliv.et  Fri Aug  5 00:47:00 2022
From: pierre at joliv.et (Pierre Jolivet)
Date: Fri, 5 Aug 2022 07:47:00 +0200
Subject: [petsc-users] [SLEPc] generalized symmetric eigensystem
In-Reply-To: <3c217462-cc35-0d2b-9c7b-88bd6f2f0b3f@geomag.info>
References: <3c217462-cc35-0d2b-9c7b-88bd6f2f0b3f@geomag.info>
Message-ID: <7DC5CF36-1C2B-42E7-93C9-251874ADFC3E@joliv.et>



> On 5 Aug 2022, at 5:04 AM, Patrick Alken <patrick.alken at geomag.info> wrote:
> 
> I am trying to solve a generalized symmetric eigensystem problem using SLEPc, and I am getting much different eigenvalues compared with the dense LAPACK solver DSYGVD. I have attached a minimal working example (MWE). The code relies on the external libraries GSL, Lapack and Lapacke.
> 
> The matrices (A,B) come from discretizing the Laplacian in spherical coordinates using a radial basis function method. Here are the eigenvalues computed by the dense LAPACK solver:
> 
> === LAPACK eigenvalues ===
> eval(0) = 1.309567289750e+00
> eval(1) = 1.648674331144e+00
> eval(2) = 1.709879393690e+00
> eval(3) = 1.709879393690e+00
> eval(4) = 2.142210251638e+00
> eval(5) = 2.142210251638e+00
> eval(6) = 2.145726457896e+00
> eval(7) = 2.216526311324e+00
> eval(8) = 2.216526311324e+00
> eval(9) = 2.695837030236e+00
> 
> 
> Here are the eigenvalues computed by SLEPc, using the command:
> 
> $ ./mwe -eps_nev 10 -eps_smallest_magnitude -eps_tol 1e-8
> 
> Number of requested eigenvalues: 10
>  Linear eigensolve converged (10 eigenpairs) due to CONVERGED_TOL; iterations 196
>  ---------------------- --------------------
>             k             ||Ax-kBx||/||kx||
>  ---------------------- --------------------
>         0.094520                906.031
>         0.177268                363.291
>         1.324501                61.9577
>         1.786472                35.7082
>         2.187857                32.8364
>        -2.886905                27.7314
>         2.899206                24.2224
>         4.195222                20.3007
>         4.787192                12.8346
>         7.221589                9.58513
>  ---------------------- --------------------
> 
> As we can see, the results are quite different than LAPACK and the error norms in the right column are quite large. Using a stricter tolerance (eps_tol) does not seem to help. Can anyone suggest how to diagnose and fix the problem?

You probably want to use a spectral transformation to find the smallest magnitude eigenpairs.
I cannot try your code because of its dependencies, but I bet that if you replace -eps_smallest_magnitude by -eps_target 0.0 -st_type sinvert, you?ll get a match with the LAPACK eigenvalues.

Thanks,
Pierre

> <laplace3d.c><le_laplace3d.h><Makefile.txt><mwe.c>


From jroman at dsic.upv.es  Fri Aug  5 01:32:38 2022
From: jroman at dsic.upv.es (Jose E. Roman)
Date: Fri, 5 Aug 2022 08:32:38 +0200
Subject: [petsc-users] [SLEPc] generalized symmetric eigensystem
In-Reply-To: <7DC5CF36-1C2B-42E7-93C9-251874ADFC3E@joliv.et>
References: <3c217462-cc35-0d2b-9c7b-88bd6f2f0b3f@geomag.info>
	<7DC5CF36-1C2B-42E7-93C9-251874ADFC3E@joliv.et>
Message-ID: <CF6DE534-E64C-4D5E-8115-2881B47121A6@dsic.upv.es>

You are setting the matrix values in the lower triangular part only. You are assuming that SLEPc uses the same storage as LAPACK, but this is not true. EPSSolve() does not work correctly because you are specifying a symmetric problem EPS_GHEP but your matrices are not symmetric. You should set the upper triangular part as well, or use the special storage SBAIJ. Anyway, your matrices are dense, so it is better that you use a DENSE storage format. Note that SLEPc is more appropriate for working with sparse matrices, not dense.

With both upper and lower triangular parts:

$ ./mwe -eps_nev 10 -eps_target 0 -st_type sinvert -mat_type dense
 Number of requested eigenvalues: 10
 Linear eigensolve converged (14 eigenpairs) due to CONVERGED_TOL; iterations 6
 ---------------------- --------------------
            k             ||Ax-kBx||/||kx||
 ---------------------- --------------------
        1.309567            5.65098e-14
        1.648674            1.67452e-09
        1.709879            5.74072e-10
        1.709879            7.52919e-09
        2.142210             1.1093e-10
        2.142210            3.24847e-09
        2.145726            1.68857e-10
        2.216526            9.25784e-11
        2.216526            2.22353e-09
        2.695837            2.30037e-09
        2.712654            6.93094e-10
        2.720031            1.63047e-09
        2.734190            8.88298e-12
        2.805326            1.31304e-09
 ---------------------- --------------------

As Pierre said, you want to avoid -eps_smallest_magnitude, see https://slepc.upv.es/documentation/faq.htm#faq12

Jose


> El 5 ago 2022, a las 7:47, Pierre Jolivet <pierre at joliv.et> escribi?:
> 
> 
> 
>> On 5 Aug 2022, at 5:04 AM, Patrick Alken <patrick.alken at geomag.info> wrote:
>> 
>> I am trying to solve a generalized symmetric eigensystem problem using SLEPc, and I am getting much different eigenvalues compared with the dense LAPACK solver DSYGVD. I have attached a minimal working example (MWE). The code relies on the external libraries GSL, Lapack and Lapacke.
>> 
>> The matrices (A,B) come from discretizing the Laplacian in spherical coordinates using a radial basis function method. Here are the eigenvalues computed by the dense LAPACK solver:
>> 
>> === LAPACK eigenvalues ===
>> eval(0) = 1.309567289750e+00
>> eval(1) = 1.648674331144e+00
>> eval(2) = 1.709879393690e+00
>> eval(3) = 1.709879393690e+00
>> eval(4) = 2.142210251638e+00
>> eval(5) = 2.142210251638e+00
>> eval(6) = 2.145726457896e+00
>> eval(7) = 2.216526311324e+00
>> eval(8) = 2.216526311324e+00
>> eval(9) = 2.695837030236e+00
>> 
>> 
>> Here are the eigenvalues computed by SLEPc, using the command:
>> 
>> $ ./mwe -eps_nev 10 -eps_smallest_magnitude -eps_tol 1e-8
>> 
>> Number of requested eigenvalues: 10
>> Linear eigensolve converged (10 eigenpairs) due to CONVERGED_TOL; iterations 196
>> ---------------------- --------------------
>>            k             ||Ax-kBx||/||kx||
>> ---------------------- --------------------
>>        0.094520                906.031
>>        0.177268                363.291
>>        1.324501                61.9577
>>        1.786472                35.7082
>>        2.187857                32.8364
>>       -2.886905                27.7314
>>        2.899206                24.2224
>>        4.195222                20.3007
>>        4.787192                12.8346
>>        7.221589                9.58513
>> ---------------------- --------------------
>> 
>> As we can see, the results are quite different than LAPACK and the error norms in the right column are quite large. Using a stricter tolerance (eps_tol) does not seem to help. Can anyone suggest how to diagnose and fix the problem?
> 
> You probably want to use a spectral transformation to find the smallest magnitude eigenpairs.
> I cannot try your code because of its dependencies, but I bet that if you replace -eps_smallest_magnitude by -eps_target 0.0 -st_type sinvert, you?ll get a match with the LAPACK eigenvalues.
> 
> Thanks,
> Pierre
> 
>> <laplace3d.c><le_laplace3d.h><Makefile.txt><mwe.c>


From mi.mike1021 at gmail.com  Fri Aug  5 08:09:32 2022
From: mi.mike1021 at gmail.com (Mike Michell)
Date: Fri, 5 Aug 2022 08:09:32 -0500
Subject: [petsc-users] DMPlex PETSCViewer for Surface Component
In-Reply-To: <CAMYG4Gn7vL6syvYuUEwmYNnmLgF1LY7HZFd5Q9BVE9XmhZLMjA@mail.gmail.com>
References: <CAEc7osb9DCAUOnUgCZ7S5fhAXfroi847W2dQbZVkd34pxW84Lw@mail.gmail.com>
	<87pmhje1nz.fsf@jedbrown.org>
	<CAMYG4Gn7vL6syvYuUEwmYNnmLgF1LY7HZFd5Q9BVE9XmhZLMjA@mail.gmail.com>
Message-ID: <CAEc7osZNp9BOuXFiHrzbhwvjDPHiKJst4E9RS_EXimnFoUJJ=Q@mail.gmail.com>

Thank you Jed and Matt. That is exactly what I wanted to do.
I was wondering if there is an example of mapping between DMPLEXs.
I have several requirements of the mapping for my code, such as
- mapping solution vectors on cell-centroids into vertices
- mapping from volume dmplex into sub-dm for wall surface region

Any example will help.

Many thanks,
Mike


> Yes, you can use DMPlexCreateSubmesh() (and friends depending on exactly
> what kind of submesh you want). This will allow you to create a vector over
> only this mesh, and map your volumetric solution to that subvector. Then
> you can view the subvector (which pulls in the submesh).
>
>   Thanks,
>
>      Matt
>
> On Mon, Aug 1, 2022 at 10:59 PM Jed Brown <jed at jedbrown.org> wrote:
>
>> I would create a sub-DM containing only the part you want to view.
>>
>> Mike Michell <mi.mike1021 at gmail.com> writes:
>>
>> > Hi,
>> >
>> > I am a user of DMPlex object in 3D grid topology. Currently the solution
>> > field is printed out using viewer PETSCVIEWERVTK in .vtu format. By
>> doing
>> > that the entire volume components are written in the file. I was
>> wondering
>> > if there is an option that I can tell PETSc viewer to print out only
>> > surface component, instead of the entire volume.
>> >
>> > Best,
>> > Mike
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220805/d4e9ada1/attachment.html>

From patrick.alken at geomag.info  Fri Aug  5 09:31:53 2022
From: patrick.alken at geomag.info (Patrick Alken)
Date: Fri, 5 Aug 2022 08:31:53 -0600
Subject: [petsc-users] [SLEPc] generalized symmetric eigensystem
In-Reply-To: <CF6DE534-E64C-4D5E-8115-2881B47121A6@dsic.upv.es>
References: <3c217462-cc35-0d2b-9c7b-88bd6f2f0b3f@geomag.info>
	<7DC5CF36-1C2B-42E7-93C9-251874ADFC3E@joliv.et>
	<CF6DE534-E64C-4D5E-8115-2881B47121A6@dsic.upv.es>
Message-ID: <10386404-58e2-6fb6-1e0c-9f29bed80f5d@geomag.info>

Thank you both, I am getting the correct results now with these changes.

On 8/5/22 00:32, Jose E. Roman wrote:
> You are setting the matrix values in the lower triangular part only. You are assuming that SLEPc uses the same storage as LAPACK, but this is not true. EPSSolve() does not work correctly because you are specifying a symmetric problem EPS_GHEP but your matrices are not symmetric. You should set the upper triangular part as well, or use the special storage SBAIJ. Anyway, your matrices are dense, so it is better that you use a DENSE storage format. Note that SLEPc is more appropriate for working with sparse matrices, not dense.
>
> With both upper and lower triangular parts:
>
> $ ./mwe -eps_nev 10 -eps_target 0 -st_type sinvert -mat_type dense
>   Number of requested eigenvalues: 10
>   Linear eigensolve converged (14 eigenpairs) due to CONVERGED_TOL; iterations 6
>   ---------------------- --------------------
>              k             ||Ax-kBx||/||kx||
>   ---------------------- --------------------
>          1.309567            5.65098e-14
>          1.648674            1.67452e-09
>          1.709879            5.74072e-10
>          1.709879            7.52919e-09
>          2.142210             1.1093e-10
>          2.142210            3.24847e-09
>          2.145726            1.68857e-10
>          2.216526            9.25784e-11
>          2.216526            2.22353e-09
>          2.695837            2.30037e-09
>          2.712654            6.93094e-10
>          2.720031            1.63047e-09
>          2.734190            8.88298e-12
>          2.805326            1.31304e-09
>   ---------------------- --------------------
>
> As Pierre said, you want to avoid -eps_smallest_magnitude, see https://slepc.upv.es/documentation/faq.htm#faq12
>
> Jose
>
>
>> El 5 ago 2022, a las 7:47, Pierre Jolivet <pierre at joliv.et> escribi?:
>>
>>
>>
>>> On 5 Aug 2022, at 5:04 AM, Patrick Alken <patrick.alken at geomag.info> wrote:
>>>
>>> I am trying to solve a generalized symmetric eigensystem problem using SLEPc, and I am getting much different eigenvalues compared with the dense LAPACK solver DSYGVD. I have attached a minimal working example (MWE). The code relies on the external libraries GSL, Lapack and Lapacke.
>>>
>>> The matrices (A,B) come from discretizing the Laplacian in spherical coordinates using a radial basis function method. Here are the eigenvalues computed by the dense LAPACK solver:
>>>
>>> === LAPACK eigenvalues ===
>>> eval(0) = 1.309567289750e+00
>>> eval(1) = 1.648674331144e+00
>>> eval(2) = 1.709879393690e+00
>>> eval(3) = 1.709879393690e+00
>>> eval(4) = 2.142210251638e+00
>>> eval(5) = 2.142210251638e+00
>>> eval(6) = 2.145726457896e+00
>>> eval(7) = 2.216526311324e+00
>>> eval(8) = 2.216526311324e+00
>>> eval(9) = 2.695837030236e+00
>>>
>>>
>>> Here are the eigenvalues computed by SLEPc, using the command:
>>>
>>> $ ./mwe -eps_nev 10 -eps_smallest_magnitude -eps_tol 1e-8
>>>
>>> Number of requested eigenvalues: 10
>>> Linear eigensolve converged (10 eigenpairs) due to CONVERGED_TOL; iterations 196
>>> ---------------------- --------------------
>>>             k             ||Ax-kBx||/||kx||
>>> ---------------------- --------------------
>>>         0.094520                906.031
>>>         0.177268                363.291
>>>         1.324501                61.9577
>>>         1.786472                35.7082
>>>         2.187857                32.8364
>>>        -2.886905                27.7314
>>>         2.899206                24.2224
>>>         4.195222                20.3007
>>>         4.787192                12.8346
>>>         7.221589                9.58513
>>> ---------------------- --------------------
>>>
>>> As we can see, the results are quite different than LAPACK and the error norms in the right column are quite large. Using a stricter tolerance (eps_tol) does not seem to help. Can anyone suggest how to diagnose and fix the problem?
>> You probably want to use a spectral transformation to find the smallest magnitude eigenpairs.
>> I cannot try your code because of its dependencies, but I bet that if you replace -eps_smallest_magnitude by -eps_target 0.0 -st_type sinvert, you?ll get a match with the LAPACK eigenvalues.
>>
>> Thanks,
>> Pierre
>>
>>> <laplace3d.c><le_laplace3d.h><Makefile.txt><mwe.c>

From apersaud at lbl.gov  Fri Aug  5 14:55:28 2022
From: apersaud at lbl.gov (Arun Persaud)
Date: Fri, 5 Aug 2022 12:55:28 -0700
Subject: [petsc-users] problem linking to petsc
Message-ID: <56430ec9-18fa-7344-a1fc-f2c238bef5d2@lbl.gov>

Hello

I'm trying to compile a plasma simulation package (in fortran + mpi) 
that uses PETSC and HYPRE.

I'm running on linux (opensuse tumbleweed) and using the distribution 
provided compilers and openmpi4 packages.

I git cloned petsc and then compiled with the following commands


./configure --prefix=/home/arun 
--with-mpi-dir=/usr/lib64/mpi/gcc/openmpi4 
--with-hypre-lib="-L/usr/lib64/mpi/gcc/openmpi4/lib64 -lHYPRE" 
--with-hypre-include=/usr/lib64/mpi/gcc/openmpi4/include/hypre/ 
--download-fblaslapack

make PETSC_DIR=/home/arun/src/petsc PETSC_ARCH=arch-linux-c-debug all

make PETSC_DIR=/home/arun/src/petsc PETSC_ARCH=arch-linux-c-debug install

And this worked without errors, but when I try to use the petsc library 
to compile and link the simulation package, I'm getting:

pic2d_CurProblemValues.f90:(.text+0x11e5d): undefined reference to 
`petscinitialize_'
collect2: error: ld returned 1 exit status
make: *** [Makefile:50: edipic2d] Error 1


pic2d_CurProblemValues.f90 seems to have the same code included that is 
given in the howto on how to use PETSC.

also, if I check the shared library, I get the following:

arun at apersaud:~/src/petsc> nm arch-linux-c-debug/lib/libpetsc.so |grep 
-i petscinit
00000000003531e6 t PetscInitFortran_Private
000000000032e9ac T PetscInitialize
0000000002bc82e8 B PetscInitializeCalled
000000000032c22d t PetscInitialize_Common
0000000000354ac3 T petscinitialized_
0000000000327d71 T PetscInitialized
00000000001213e5 t PetscInitialize_DynamicLibraries
000000000035389d T petscinitializef_
0000000000353e38 T petscinitializefortran_
0000000000353cd2 T PetscInitializeFortran
00000000003279a8 T PetscInitializeNoArguments
0000000000327138 T PetscInitializeNoPointers
00000000000fd2ce T __petscsys_MOD_petscinitializenoarguments
00000000000fd313 T __petscsys_MOD_petscinitializenohelp
00000000000fd3e9 T __petscsys_MOD_petscinitializewithhelp

which seems to indicate that the initialize function "petscinitialize_" 
is missing.

Is there a compiler flag I need to set to or a configure flag when 
compiling petsc?

Not sure where things are going wrong here and how to debug this.

Any suggestions would be appreciated.

Thanks

Arun


From bsmith at petsc.dev  Fri Aug  5 15:42:44 2022
From: bsmith at petsc.dev (Barry Smith)
Date: Fri, 5 Aug 2022 16:42:44 -0400
Subject: [petsc-users] problem linking to petsc
In-Reply-To: <56430ec9-18fa-7344-a1fc-f2c238bef5d2@lbl.gov>
References: <56430ec9-18fa-7344-a1fc-f2c238bef5d2@lbl.gov>
Message-ID: <7308917B-31CA-4186-912C-5C336F7D3ED2@petsc.dev>

 
  The secret is in the 

> 00000000000fd2ce T __petscsys_MOD_petscinitializenoarguments
> 00000000000fd313 T __petscsys_MOD_petscinitializenohelp
> 00000000000fd3e9 T __petscsys_MOD_petscinitializewithhelp

it looks like the application code was last used with a much older version of PETSc that did not use Fortran 90 modules as much.

You will need to locate in the Fortran code where PetscInitialize is called and make sure the main/or subroutine has a pair like

#include <petsc/finclude/petscsnes.h>
      use petscsnes

at the top. See the, for example,  src/*/tutorials/*.F* files

The snes maybe ksp or ts depending how the application uses PETSc.

There maybe other updates needed to modernize the PETSc calls to add the use in various places.
 Barry


> On Aug 5, 2022, at 3:55 PM, Arun Persaud <apersaud at lbl.gov> wrote:
> 
> Hello
> 
> I'm trying to compile a plasma simulation package (in fortran + mpi) that uses PETSC and HYPRE.
> 
> I'm running on linux (opensuse tumbleweed) and using the distribution provided compilers and openmpi4 packages.
> 
> I git cloned petsc and then compiled with the following commands
> 
> 
> ./configure --prefix=/home/arun --with-mpi-dir=/usr/lib64/mpi/gcc/openmpi4 --with-hypre-lib="-L/usr/lib64/mpi/gcc/openmpi4/lib64 -lHYPRE" --with-hypre-include=/usr/lib64/mpi/gcc/openmpi4/include/hypre/ --download-fblaslapack
> 
> make PETSC_DIR=/home/arun/src/petsc PETSC_ARCH=arch-linux-c-debug all
> 
> make PETSC_DIR=/home/arun/src/petsc PETSC_ARCH=arch-linux-c-debug install
> 
> And this worked without errors, but when I try to use the petsc library to compile and link the simulation package, I'm getting:
> 
> pic2d_CurProblemValues.f90:(.text+0x11e5d): undefined reference to `petscinitialize_'
> collect2: error: ld returned 1 exit status
> make: *** [Makefile:50: edipic2d] Error 1
> 
> 
> pic2d_CurProblemValues.f90 seems to have the same code included that is given in the howto on how to use PETSC.
> 
> also, if I check the shared library, I get the following:
> 
> arun at apersaud:~/src/petsc> nm arch-linux-c-debug/lib/libpetsc.so |grep -i petscinit
> 00000000003531e6 t PetscInitFortran_Private
> 000000000032e9ac T PetscInitialize
> 0000000002bc82e8 B PetscInitializeCalled
> 000000000032c22d t PetscInitialize_Common
> 0000000000354ac3 T petscinitialized_
> 0000000000327d71 T PetscInitialized
> 00000000001213e5 t PetscInitialize_DynamicLibraries
> 000000000035389d T petscinitializef_
> 0000000000353e38 T petscinitializefortran_
> 0000000000353cd2 T PetscInitializeFortran
> 00000000003279a8 T PetscInitializeNoArguments
> 0000000000327138 T PetscInitializeNoPointers
> 00000000000fd2ce T __petscsys_MOD_petscinitializenoarguments
> 00000000000fd313 T __petscsys_MOD_petscinitializenohelp
> 00000000000fd3e9 T __petscsys_MOD_petscinitializewithhelp
> 
> which seems to indicate that the initialize function "petscinitialize_" is missing.
> 
> Is there a compiler flag I need to set to or a configure flag when compiling petsc?
> 
> Not sure where things are going wrong here and how to debug this.
> 
> Any suggestions would be appreciated.
> 
> Thanks
> 
> Arun
> 


From apersaud at lbl.gov  Fri Aug  5 20:03:05 2022
From: apersaud at lbl.gov (Arun Persaud)
Date: Fri, 5 Aug 2022 18:03:05 -0700
Subject: [petsc-users] problem linking to petsc
In-Reply-To: <7308917B-31CA-4186-912C-5C336F7D3ED2@petsc.dev>
References: <56430ec9-18fa-7344-a1fc-f2c238bef5d2@lbl.gov>
	<7308917B-31CA-4186-912C-5C336F7D3ED2@petsc.dev>
Message-ID: <733dd23d-87ee-76c5-706f-eac4ecf18d4d@lbl.gov>

Hello Barry

On 8/5/22 13:42, Barry Smith wrote:
> [...]
> #include <petsc/finclude/petscsnes.h>
>        use petscsnes
> 
> at the top. See the, for example,  src/*/tutorials/*.F* files
 >[...]

Thanks for the pointers. Got it to run now!

have a great weekend

Arun



From narnoldm at umich.edu  Mon Aug  8 09:29:43 2022
From: narnoldm at umich.edu (Nicholas Arnold-Medabalimi)
Date: Mon, 8 Aug 2022 10:29:43 -0400
Subject: [petsc-users] PetscSF remote ranks
Message-ID: <CAB1dcV4CAW2FkGtaAdKky2es4mkgrSP+Qa41-0Yapdzgsw+e1g@mail.gmail.com>

Hi

When I look at the SFView, I can see there is a value for remote ranks,
which are the overlapped processors. Is there a function to get this value
from the SF? I can't seem to find the relative function in the SF function
list. I can get all the other SFView values from
PetscSFGetGraph. Is the best way to search through iremote and count
the distinct values?

Sincerely
Nicholas
-- 
Nicholas Arnold-Medabalimi

Ph.D. Candidate
Computational Aeroscience Lab
University of Michigan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220808/f0ef0104/attachment.html>

From narnoldm at umich.edu  Mon Aug  8 10:56:21 2022
From: narnoldm at umich.edu (Nicholas Arnold-Medabalimi)
Date: Mon, 8 Aug 2022 11:56:21 -0400
Subject: [petsc-users] DMPlex total point counts
In-Reply-To: <CAMYG4GkGCwXxwT9=G+D90V9DT0EZ1+TFZTa1PrA9xF3kOYMoTg@mail.gmail.com>
References: <CAB1dcV4KY9XGeYKsHr=avvyGOf53OMfGbfLMZUH=F0-Orn-4WQ@mail.gmail.com>
	<CAMYG4GkGCwXxwT9=G+D90V9DT0EZ1+TFZTa1PrA9xF3kOYMoTg@mail.gmail.com>
Message-ID: <CAB1dcV45rFHJ6WZnoGoncc_0zeVhrwXEEvGKjBhZv8Ur-X3U-A@mail.gmail.com>

Thanks, this helped a great deal.

I wondered if there is any issue in using the IS generated by
DMPlexGetCellNumbering to count the cells instead? It looks like the IS
values will be -(globalID+1) if they are not on the local processor, so I
just count negative values instead of having to check the SF.

Sincerely
Nicholas

On Wed, Aug 3, 2022 at 10:33 AM Matthew Knepley <knepley at gmail.com> wrote:

> On Tue, Aug 2, 2022 at 4:00 PM Nicholas Arnold-Medabalimi <
> narnoldm at umich.edu> wrote:
>
>> Hi
>>
>> Currently, in my distributed DMPlex object, every so often, I want to
>> look at total cell counts. To do this in my current implementation, I get
>> the cell stratum from the DMplex and search through the PetscSF to remove
>> any overlapped (not owned) cells and then reduce over all the ranks to get
>> the total cell count. Is there a more graceful way of doing this already
>> built into DM?
>>
>
> If all you want is the global cell count, this is probably the easiest
> way. If you want a global numbering, you can use
>
>   https://petsc.org/main/docs/manualpages/DMPLEX/DMPlexGetCellNumbering/
>
>   Thanks,
>
>      Matt
>
>
>> Sincerely
>> Nicholas
>>
>> --
>> Nicholas Arnold-Medabalimi
>>
>> Ph.D. Candidate
>> Computational Aeroscience Lab
>> University of Michigan
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>


-- 
Nicholas Arnold-Medabalimi

Ph.D. Candidate
Computational Aeroscience Lab
University of Michigan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220808/2911a9d3/attachment.html>

From junchao.zhang at gmail.com  Mon Aug  8 11:43:30 2022
From: junchao.zhang at gmail.com (Junchao Zhang)
Date: Mon, 8 Aug 2022 11:43:30 -0500
Subject: [petsc-users] PetscSF remote ranks
In-Reply-To: <CAB1dcV4CAW2FkGtaAdKky2es4mkgrSP+Qa41-0Yapdzgsw+e1g@mail.gmail.com>
References: <CAB1dcV4CAW2FkGtaAdKky2es4mkgrSP+Qa41-0Yapdzgsw+e1g@mail.gmail.com>
Message-ID: <CA+MQGp8PZvUindjwLCrDRoW1eh986HYEUcXjHH9SkaFm7vXRhQ@mail.gmail.com>

See https://petsc.org/main/docs/manualpages/PetscSF/PetscSFGetRootRanks/
and https://petsc.org/main/docs/manualpages/PetscSF/PetscSFGetLeafRanks/

--Junchao Zhang


On Mon, Aug 8, 2022 at 9:30 AM Nicholas Arnold-Medabalimi <
narnoldm at umich.edu> wrote:

> Hi
>
> When I look at the SFView, I can see there is a value for remote ranks,
> which are the overlapped processors. Is there a function to get this value
> from the SF? I can't seem to find the relative function in the SF function
> list. I can get all the other SFView values from
> PetscSFGetGraph. Is the best way to search through iremote and count
> the distinct values?
>
> Sincerely
> Nicholas
> --
> Nicholas Arnold-Medabalimi
>
> Ph.D. Candidate
> Computational Aeroscience Lab
> University of Michigan
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220808/8167e4cf/attachment.html>

From saigre at math.unistra.fr  Tue Aug  9 04:21:05 2022
From: saigre at math.unistra.fr (Thomas Saigre)
Date: Tue, 9 Aug 2022 11:21:05 +0200
Subject: [petsc-users] Issue to set values to a matrix in parallel in python
Message-ID: <8a5330c2-4dc1-6627-26d1-c79bfc677bcc@math.unistra.fr>

Hi,

I've been trying for a few weeks to construct a matrix from a list of 
vectors, unsuccessfully.
Here is my problem : I have a list l a petsc4py.Vec, each vector has a 
size n, and I have d vectors. I want to "cast" these vectors to a 
petsc4py.Mat Z of shape (n,d), where Z[:, i] = l[i] (using NumPy notation)

Here is the code I'm using :

import sys
from petsc4py import PETSc
n = 5
d = 10

l = []?? # creation of the list of vectors
for i in range(d):
 ??? v = PETSc.Vec().create()
 ??? v.setSizes(n)
 ??? v.setFromOptions()
 ??? v.set(i)
 ??? l.append(v)

Z = PETSc.Mat().create()
Z.setSizes([n, d])
Z.setFromOptions()
Z.setUp()
for i, v in enumerate(l):
 ??? Z.setValues(range(n), i, v)
Z.assemble()
Z.view()??? # to display the matrix in the terminal

In sequential, the result is correct :

Mat Object: 1 MPI processes
 ? type: seqaij
row 0: (0, 0.)? (1, 1.)? (2, 2.)? (3, 3.)? (4, 4.)? (5, 5.)? (6, 6.)? 
(7, 7.)? (8, 8.)? (9, 9.)
row 1: (0, 0.)? (1, 1.)? (2, 2.)? (3, 3.)? (4, 4.)? (5, 5.)? (6, 6.)? 
(7, 7.)? (8, 8.)? (9, 9.)
row 2: (0, 0.)? (1, 1.)? (2, 2.)? (3, 3.)? (4, 4.)? (5, 5.)? (6, 6.)? 
(7, 7.)? (8, 8.)? (9, 9.)
row 3: (0, 0.)? (1, 1.)? (2, 2.)? (3, 3.)? (4, 4.)? (5, 5.)? (6, 6.)? 
(7, 7.)? (8, 8.)? (9, 9.)
row 4: (0, 0.)? (1, 1.)? (2, 2.)? (3, 3.)? (4, 4.)? (5, 5.)? (6, 6.)? 
(7, 7.)? (8, 8.)? (9, 9.)

 ?but when I run it using the command mpirun -np 2 python3 file.py, I 
get the following error, about incompatible array sizes (I did not 
manage to understand what ni, nj and nv correspond to...)

Traceback (most recent call last):
 ? File "/home/Documents/code/tests/file.py", line 31, in <module>
 ??? Z.setValues(list(range(n)), i, v)
 ? File "PETSc/Mat.pyx", line 888, in petsc4py.PETSc.Mat.setValues
 ? File "PETSc/petscmat.pxi", line 828, in petsc4py.PETSc.matsetvalues
ValueError: incompatible array sizes: ni=5, nj=1, nv=3
Traceback (most recent call last):
 ? File "/home/saigre/Documents/code/tests/t2.py", line 31, in <module>
 ??? Z.setValues(list(range(n)), i, v)
 ? File "PETSc/Mat.pyx", line 888, in petsc4py.PETSc.Mat.setValues
 ? File "PETSc/petscmat.pxi", line 828, in petsc4py.PETSc.matsetvalues
ValueError: incompatible array sizes: ni=5, nj=1, nv=2

Two weeks ago, I made a post on stack overflow 
(https://stackoverflow.com/questions/73124230/convert-a-list-of-vector-to-a-matrix-with-petsc4py). 
I tried using the apt packages, and I also compiled from the sources, 
but I get the same error.

I someone has an idea how to succeed in it, I'm all ears !

Thanks,

Thomas

From stefano.zampini at gmail.com  Tue Aug  9 10:08:36 2022
From: stefano.zampini at gmail.com (Stefano Zampini)
Date: Tue, 9 Aug 2022 17:08:36 +0200
Subject: [petsc-users] Issue to set values to a matrix in parallel in
 python
In-Reply-To: <8a5330c2-4dc1-6627-26d1-c79bfc677bcc@math.unistra.fr>
References: <8a5330c2-4dc1-6627-26d1-c79bfc677bcc@math.unistra.fr>
Message-ID: <021E152D-AC6D-4E42-AD98-254CFA9FD07B@gmail.com>

PETSc distributes matrices and vectors in parallel. Take a look at https://petsc.org/release/docs/manualpages/Vec/VecGetOwnershipRange.html

> On Aug 9, 2022, at 11:21 AM, Thomas Saigre <saigre at math.unistra.fr> wrote:
> 
> Hi,
> 
> I've been trying for a few weeks to construct a matrix from a list of vectors, unsuccessfully.
> Here is my problem : I have a list l a petsc4py.Vec, each vector has a size n, and I have d vectors. I want to "cast" these vectors to a petsc4py.Mat Z of shape (n,d), where Z[:, i] = l[i] (using NumPy notation)
> 
> Here is the code I'm using :
> 
> import sys
> from petsc4py import PETSc
> n = 5
> d = 10
> 
> l = []   # creation of the list of vectors
> for i in range(d):
>     v = PETSc.Vec().create()
>     v.setSizes(n)
>     v.setFromOptions()
>     v.set(i)
>     l.append(v)
> 
> Z = PETSc.Mat().create()
> Z.setSizes([n, d])
> Z.setFromOptions()
> Z.setUp()
> for i, v in enumerate(l):
>     Z.setValues(range(n), i, v)
> Z.assemble()
> Z.view()    # to display the matrix in the terminal
> 
> In sequential, the result is correct :
> 
> Mat Object: 1 MPI processes
>   type: seqaij
> row 0: (0, 0.)  (1, 1.)  (2, 2.)  (3, 3.)  (4, 4.)  (5, 5.)  (6, 6.)  (7, 7.)  (8, 8.)  (9, 9.)
> row 1: (0, 0.)  (1, 1.)  (2, 2.)  (3, 3.)  (4, 4.)  (5, 5.)  (6, 6.)  (7, 7.)  (8, 8.)  (9, 9.)
> row 2: (0, 0.)  (1, 1.)  (2, 2.)  (3, 3.)  (4, 4.)  (5, 5.)  (6, 6.)  (7, 7.)  (8, 8.)  (9, 9.)
> row 3: (0, 0.)  (1, 1.)  (2, 2.)  (3, 3.)  (4, 4.)  (5, 5.)  (6, 6.)  (7, 7.)  (8, 8.)  (9, 9.)
> row 4: (0, 0.)  (1, 1.)  (2, 2.)  (3, 3.)  (4, 4.)  (5, 5.)  (6, 6.)  (7, 7.)  (8, 8.)  (9, 9.)
> 
>  but when I run it using the command mpirun -np 2 python3 file.py, I get the following error, about incompatible array sizes (I did not manage to understand what ni, nj and nv correspond to...)
> 
> Traceback (most recent call last):
>   File "/home/Documents/code/tests/file.py", line 31, in <module>
>     Z.setValues(list(range(n)), i, v)
>   File "PETSc/Mat.pyx", line 888, in petsc4py.PETSc.Mat.setValues
>   File "PETSc/petscmat.pxi", line 828, in petsc4py.PETSc.matsetvalues
> ValueError: incompatible array sizes: ni=5, nj=1, nv=3
> Traceback (most recent call last):
>   File "/home/saigre/Documents/code/tests/t2.py", line 31, in <module>
>     Z.setValues(list(range(n)), i, v)
>   File "PETSc/Mat.pyx", line 888, in petsc4py.PETSc.Mat.setValues
>   File "PETSc/petscmat.pxi", line 828, in petsc4py.PETSc.matsetvalues
> ValueError: incompatible array sizes: ni=5, nj=1, nv=2
> 
> Two weeks ago, I made a post on stack overflow (https://stackoverflow.com/questions/73124230/convert-a-list-of-vector-to-a-matrix-with-petsc4py). I tried using the apt packages, and I also compiled from the sources, but I get the same error.
> 
> I someone has an idea how to succeed in it, I'm all ears !
> 
> Thanks,
> 
> Thomas


From knepley at gmail.com  Tue Aug  9 10:27:52 2022
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 9 Aug 2022 11:27:52 -0400
Subject: [petsc-users] Issue to set values to a matrix in parallel in
 python
In-Reply-To: <021E152D-AC6D-4E42-AD98-254CFA9FD07B@gmail.com>
References: <8a5330c2-4dc1-6627-26d1-c79bfc677bcc@math.unistra.fr>
	<021E152D-AC6D-4E42-AD98-254CFA9FD07B@gmail.com>
Message-ID: <CAMYG4GnepbiXz3thRP4AOQ=7XLTQC1ehYLHnLy5NNro=S7Cg9Q@mail.gmail.com>

On Tue, Aug 9, 2022 at 11:12 AM Stefano Zampini <stefano.zampini at gmail.com>
wrote:

> PETSc distributes matrices and vectors in parallel. Take a look at
> https://petsc.org/release/docs/manualpages/Vec/VecGetOwnershipRange.html


Longer explanation:

Vectors are distributed by giving a stretch of contiguous rows to each
process. Dense matrices also give a stretch of contiguous rows to each
process.
This means that each matrix column would correspond to a vector. The
easiest way I see to do this is to use

  https://petsc.org/main/docs/manualpages/Mat/MatDenseGetColumnVec/

and then VecCopy() each vector into the correct column. If you want to
share memory, allocate the dense matrix first, and make you vectors using
this
routine.

  Thanks,

      Matt


>
> > On Aug 9, 2022, at 11:21 AM, Thomas Saigre <saigre at math.unistra.fr>
> wrote:
> >
> > Hi,
> >
> > I've been trying for a few weeks to construct a matrix from a list of
> vectors, unsuccessfully.
> > Here is my problem : I have a list l a petsc4py.Vec, each vector has a
> size n, and I have d vectors. I want to "cast" these vectors to a
> petsc4py.Mat Z of shape (n,d), where Z[:, i] = l[i] (using NumPy notation)
> >
> > Here is the code I'm using :
> >
> > import sys
> > from petsc4py import PETSc
> > n = 5
> > d = 10
> >
> > l = []   # creation of the list of vectors
> > for i in range(d):
> >     v = PETSc.Vec().create()
> >     v.setSizes(n)
> >     v.setFromOptions()
> >     v.set(i)
> >     l.append(v)
> >
> > Z = PETSc.Mat().create()
> > Z.setSizes([n, d])
> > Z.setFromOptions()
> > Z.setUp()
> > for i, v in enumerate(l):
> >     Z.setValues(range(n), i, v)
> > Z.assemble()
> > Z.view()    # to display the matrix in the terminal
> >
> > In sequential, the result is correct :
> >
> > Mat Object: 1 MPI processes
> >   type: seqaij
> > row 0: (0, 0.)  (1, 1.)  (2, 2.)  (3, 3.)  (4, 4.)  (5, 5.)  (6, 6.)
> (7, 7.)  (8, 8.)  (9, 9.)
> > row 1: (0, 0.)  (1, 1.)  (2, 2.)  (3, 3.)  (4, 4.)  (5, 5.)  (6, 6.)
> (7, 7.)  (8, 8.)  (9, 9.)
> > row 2: (0, 0.)  (1, 1.)  (2, 2.)  (3, 3.)  (4, 4.)  (5, 5.)  (6, 6.)
> (7, 7.)  (8, 8.)  (9, 9.)
> > row 3: (0, 0.)  (1, 1.)  (2, 2.)  (3, 3.)  (4, 4.)  (5, 5.)  (6, 6.)
> (7, 7.)  (8, 8.)  (9, 9.)
> > row 4: (0, 0.)  (1, 1.)  (2, 2.)  (3, 3.)  (4, 4.)  (5, 5.)  (6, 6.)
> (7, 7.)  (8, 8.)  (9, 9.)
> >
> >  but when I run it using the command mpirun -np 2 python3 file.py, I get
> the following error, about incompatible array sizes (I did not manage to
> understand what ni, nj and nv correspond to...)
> >
> > Traceback (most recent call last):
> >   File "/home/Documents/code/tests/file.py", line 31, in <module>
> >     Z.setValues(list(range(n)), i, v)
> >   File "PETSc/Mat.pyx", line 888, in petsc4py.PETSc.Mat.setValues
> >   File "PETSc/petscmat.pxi", line 828, in petsc4py.PETSc.matsetvalues
> > ValueError: incompatible array sizes: ni=5, nj=1, nv=3
> > Traceback (most recent call last):
> >   File "/home/saigre/Documents/code/tests/t2.py", line 31, in <module>
> >     Z.setValues(list(range(n)), i, v)
> >   File "PETSc/Mat.pyx", line 888, in petsc4py.PETSc.Mat.setValues
> >   File "PETSc/petscmat.pxi", line 828, in petsc4py.PETSc.matsetvalues
> > ValueError: incompatible array sizes: ni=5, nj=1, nv=2
> >
> > Two weeks ago, I made a post on stack overflow (
> https://stackoverflow.com/questions/73124230/convert-a-list-of-vector-to-a-matrix-with-petsc4py).
> I tried using the apt packages, and I also compiled from the sources, but I
> get the same error.
> >
> > I someone has an idea how to succeed in it, I'm all ears !
> >
> > Thanks,
> >
> > Thomas
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220809/de1afc9c/attachment-0001.html>

From elias.karabelas at uni-graz.at  Wed Aug 10 02:35:41 2022
From: elias.karabelas at uni-graz.at (Karabelas, Elias (elias.karabelas@uni-graz.at))
Date: Wed, 10 Aug 2022 07:35:41 +0000
Subject: [petsc-users] 3D Model Coupled to 0D Model
Message-ID: <4eb1f2e7-64ac-a3af-7ff9-276674f55c32@uni-graz.at>

Hey all,

so I have a nonlinear problem that can be abstractly written as

| F(u,P) |           | 0 |

|        |    =      |   |

| G(u,P) |           | 0 |


Here u is a variable that comes from a discretization of a PDE and P are four scalars that come from some coupled attached ODEs (P = (P1,P2,P3,P4) )

I know abstract how to apply Newton-Raphson in this context as the Jacobian is simply

d_u F   d_P F

d_u G   d_P G

where d_P F and d_u G are formed from 4 Vecs resp, and d_P G is a 4x4 matrix.

So what I have troubles with is how I could squeeze something like this into an SNES context, at the moment I'm doing a Schur-Complement for Solving this problem for each Newton solve. This however entails, that I'm, solving (d_u F)^-1 to a very low tolerance inside the SC.
In the End I want to have something that can work with an inexact Newton method, but I don't know which would be the correct tool (MATSHELL for the jacobian maybe?) to squeeze this into an SNES.

Any ideas?

Best regards
Elias


--
Dr. Elias Karabelas
Research Associate
University of Graz
Institute of Mathematics and Scientific Computing
Heinrichstra?e 36
A-8010 Graz
Austria

Phone: +43 316 380 8546
Email: elias.karabelas at uni-graz.at<mailto:elias.karabelas at uni-graz.at>
Web:  https://ccl.medunigraz.at/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220810/c22aa58e/attachment.html>

From jed at jedbrown.org  Wed Aug 10 07:17:22 2022
From: jed at jedbrown.org (Jed Brown)
Date: Wed, 10 Aug 2022 06:17:22 -0600
Subject: [petsc-users] 3D Model Coupled to 0D Model
In-Reply-To: <4eb1f2e7-64ac-a3af-7ff9-276674f55c32@uni-graz.at>
References: <4eb1f2e7-64ac-a3af-7ff9-276674f55c32@uni-graz.at>
Message-ID: <87a68c8f8t.fsf@jedbrown.org>

Have you considered PCFieldSplit? The SCHUR variant gives you all sorts of approximate factorization schemes, though you might find that an additive or multiplicative split is almost as good for this problem. 

"Karabelas, Elias (elias.karabelas at uni-graz.at)"	<elias.karabelas at uni-graz.at> writes:

> Hey all,
>
> so I have a nonlinear problem that can be abstractly written as
>
> | F(u,P) |           | 0 |
>
> |        |    =      |   |
>
> | G(u,P) |           | 0 |
>
>
> Here u is a variable that comes from a discretization of a PDE and P are four scalars that come from some coupled attached ODEs (P = (P1,P2,P3,P4) )
>
> I know abstract how to apply Newton-Raphson in this context as the Jacobian is simply
>
> d_u F   d_P F
>
> d_u G   d_P G
>
> where d_P F and d_u G are formed from 4 Vecs resp, and d_P G is a 4x4 matrix.
>
> So what I have troubles with is how I could squeeze something like this into an SNES context, at the moment I'm doing a Schur-Complement for Solving this problem for each Newton solve. This however entails, that I'm, solving (d_u F)^-1 to a very low tolerance inside the SC.
> In the End I want to have something that can work with an inexact Newton method, but I don't know which would be the correct tool (MATSHELL for the jacobian maybe?) to squeeze this into an SNES.
>
> Any ideas?
>
> Best regards
> Elias
>
>
> --
> Dr. Elias Karabelas
> Research Associate
> University of Graz
> Institute of Mathematics and Scientific Computing
> Heinrichstra?e 36
> A-8010 Graz
> Austria
>
> Phone: +43 316 380 8546
> Email: elias.karabelas at uni-graz.at<mailto:elias.karabelas at uni-graz.at>
> Web:  https://ccl.medunigraz.at/

From narnoldm at umich.edu  Wed Aug 10 20:59:52 2022
From: narnoldm at umich.edu (Nicholas Arnold-Medabalimi)
Date: Wed, 10 Aug 2022 21:59:52 -0400
Subject: [petsc-users] PetscSF leaf ordering behaviour
Message-ID: <CAB1dcV5LpR7G8ng4XCgi3wPCT-NTFNX_qkQ2cGY5xUojx1ALEA@mail.gmail.com>

Hi

I have been working on writing an output of a DMPlex object to interface
with an existing code that has some strict requirements. I need each
process to output its partition vertices and cell map but also note the
point number of the cell on the adjacent processor for the boundary faces.

I have two different setups I'm pursuing with different challenges.

1) The DMPlex is distributed with an overlap of 0. This makes the vertex
output trivial as only I don't need to worry about the halo vertices.
However, to get the cell from the other processor, I have to check the star
forest on both processors, figure out the face on the sending processor,
then grab its support cell and trade it with the host processor.

*If I continue on this path I'd say the question would be, is there an
easier way to get the support cell from the adjacent processor when you
don't have an overlap?*

Right now, what I am doing is getting the RootRanks of the DM SF and the
inverse of this SF. I do this because with an overlap of 0, the root
dependency only goes one way on the boundaries, and I need to have the
cells on each side sent to the opposite processor.


2) The DMPlex has an overlap of 1. This makes grabbing the neighboring cell
local point value trivial roots using the support of the faces combined
with the SF. But the issue is now outputting only the vertices of the
process without messing up the cell maps.

*The question here is how are the vertices in the DMPlex organized by
default?* I have noticed that the leaf vertices are always at the end of
the DMPlex stratum.* Is this a guaranteed behavior and then within the leaf
cells when there is an overlap are they organized in any set way? *I'm
hoping that for each layer of overlap those leaves are appended. That way I
can just take the vertices up until the first halo of vertices. Otherwise,
I would search through the list to check if they are in the core partition
and output them, and then account for any removed vertex in the cell maps.

I appreciate the input and help.

Sincerely
Nicholas



-- 
Nicholas Arnold-Medabalimi

Ph.D. Candidate
Computational Aeroscience Lab
University of Michigan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220810/d8694c03/attachment.html>

From gongding at cn.cogenda.com  Thu Aug 11 04:42:19 2022
From: gongding at cn.cogenda.com (Gong Ding)
Date: Thu, 11 Aug 2022 17:42:19 +0800
Subject: [petsc-users] CPARDISO halt on petsc3.17
Message-ID: <22f0095c-a008-0512-7d77-2e64470397e1@cn.cogenda.com>

Hi petsc developer,

MKL (version 20200004) cpardiso halt on parallel environment since petsc 
3.15.

we tested that 3.12 and 3.14 works? but 3.15~3.17 (latest) halt.

Dose anyone meet the same trouble?

Gong Ding




From knepley at gmail.com  Thu Aug 11 06:49:22 2022
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 11 Aug 2022 07:49:22 -0400
Subject: [petsc-users] PetscSF leaf ordering behaviour
In-Reply-To: <CAB1dcV5LpR7G8ng4XCgi3wPCT-NTFNX_qkQ2cGY5xUojx1ALEA@mail.gmail.com>
References: <CAB1dcV5LpR7G8ng4XCgi3wPCT-NTFNX_qkQ2cGY5xUojx1ALEA@mail.gmail.com>
Message-ID: <CAMYG4GnFj0dyFsLjpmBU1VEqrZdkte=bPR9Og5Kn62ozDnTTCQ@mail.gmail.com>

On Wed, Aug 10, 2022 at 10:00 PM Nicholas Arnold-Medabalimi <
narnoldm at umich.edu> wrote:

> Hi
>
> I have been working on writing an output of a DMPlex object to interface
> with an existing code that has some strict requirements. I need each
> process to output its partition vertices and cell map but also note the
> point number of the cell on the adjacent processor for the boundary faces.
>
> I have two different setups I'm pursuing with different challenges.
>
> 1) The DMPlex is distributed with an overlap of 0. This makes the vertex
> output trivial as only I don't need to worry about the halo vertices.
> However, to get the cell from the other processor, I have to check the star
> forest on both processors, figure out the face on the sending processor,
> then grab its support cell and trade it with the host processor.
>
> *If I continue on this path I'd say the question would be, is there an
> easier way to get the support cell from the adjacent processor when you
> don't have an overlap?*
>
> Right now, what I am doing is getting the RootRanks of the DM SF and the
> inverse of this SF. I do this because with an overlap of 0, the root
> dependency only goes one way on the boundaries, and I need to have the
> cells on each side sent to the opposite processor.
>

YOu end up doing 90% of the work we do for overlap here. I cannot see an
easier way right now.


> 2) The DMPlex has an overlap of 1. This makes grabbing the neighboring
> cell local point value trivial roots using the support of the faces
> combined with the SF. But the issue is now outputting only the vertices of
> the process without messing up the cell maps.
>
> *The question here is how are the vertices in the DMPlex organized by
> default?* I have noticed that the leaf vertices are always at the end of
> the DMPlex stratum.* Is this a guaranteed behavior and then within the
> leaf cells when there is an overlap are they organized in any set way?*
>

No, I do not guarantee that leaf vertices are at the end, or leaf cells.
You can always use DMPlexPermute() to get that behavior if you want. It
sounds
like you want to do that. I know Firedrake does a similar thing. Let me
know if this does not work for you.

  Thanks,

     Matt


> I'm hoping that for each layer of overlap those leaves are appended. That
> way I can just take the vertices up until the first halo of vertices.
> Otherwise, I would search through the list to check if they are in the core
> partition and output them, and then account for any removed vertex in the
> cell maps.
>
> I appreciate the input and help.
>
> Sincerely
> Nicholas
>
>
>
> --
> Nicholas Arnold-Medabalimi
>
> Ph.D. Candidate
> Computational Aeroscience Lab
> University of Michigan
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220811/c310d600/attachment.html>

From bsmith at petsc.dev  Thu Aug 11 10:00:36 2022
From: bsmith at petsc.dev (Barry Smith)
Date: Thu, 11 Aug 2022 11:00:36 -0400
Subject: [petsc-users] CPARDISO halt on petsc3.17
In-Reply-To: <22f0095c-a008-0512-7d77-2e64470397e1@cn.cogenda.com>
References: <22f0095c-a008-0512-7d77-2e64470397e1@cn.cogenda.com>
Message-ID: <4805AD7A-DDE9-42B1-A382-AD16A40F93F4@petsc.dev>


  What do you mean halt? Does it hang, seemingly running forever with no output, does it crash and print an error message (please send the entire error message; cut and paste). Is the matrix something you can share so we can try to reproduce?

  Barry


> On Aug 11, 2022, at 5:42 AM, Gong Ding <gongding at cn.cogenda.com> wrote:
> 
> Hi petsc developer,
> 
> MKL (version 20200004) cpardiso halt on parallel environment since petsc 3.15.
> 
> we tested that 3.12 and 3.14 works  but 3.15~3.17 (latest) halt.
> 
> Dose anyone meet the same trouble?
> 
> Gong Ding
> 
> 
> 


From bsmith at petsc.dev  Thu Aug 11 12:41:52 2022
From: bsmith at petsc.dev (Barry Smith)
Date: Thu, 11 Aug 2022 13:41:52 -0400
Subject: [petsc-users] CPARDISO halt on petsc3.17
In-Reply-To: <c68fd611-b45d-4e58-b7e6-ab5d6890f610@cn.cogenda.com>
References: <22f0095c-a008-0512-7d77-2e64470397e1@cn.cogenda.com>
	<4805AD7A-DDE9-42B1-A382-AD16A40F93F4@petsc.dev>
	<c68fd611-b45d-4e58-b7e6-ab5d6890f610@cn.cogenda.com>
Message-ID: <F3252EC7-ECCF-4A79-A0B0-BDD7F3112999@petsc.dev>


-with-mpi-dir=/usr/local/mpich-3.4.2/ 


> #0  0x00007fede65066b3 in MPIR_Barrier.part.0 () from /usr/local/mpich/lib/libmpi.so.12
> #1  0x00007fede65075fe in PMPI_Barrier () from /usr/local/mpich/lib/libmpi.so.12
> #2  0x00007fede4629409 in MKLMPI_Barrier () from /opt/intel/mkl/lib/intel64/libmkl_blacs_intelmpi_lp64.so


There seem to be three MPI floating around? The libmkl_blacs_intelmpi_lp64.so seems to indicate it expects and Intel MPI? The /usr/local/mpich/lib/libmpi.so.12 seems to indicate that the MPI is MPICH installed at /usr/local/mpich but the -with-mpi-dir=/usr/local/mpich-3.4.2/  seems to indicate you are building PETSc with yet a different MPI.

Given the libmkl_blacs_intelmpi_lp64.so I think you need to start by figuring out how to do the configure using the Intel MPI only.

It is hanging because of 

> #2  0x00007fede4629409 in MKLMPI_Barrier () from /opt/intel/mkl/lib/intel64/libmkl_blacs_intelmpi_lp64.so

versus 

> #2  0x00007f3537829532 in MKLMPI_Bcast () from /opt/intel/mkl/lib/intel64/libmkl_blacs_intelmpi_lp64.so

since this is inside the MKL code I don't see why PETSc would cause impossible configuration. Nor do I see why the same code should run with earlier PETSc versions, are the configure options and the resulting libraries used (MKL, MPI etc) exactly the same for all PETSc versions you tested with?



> On Aug 11, 2022, at 12:42 PM, Gong Ding <gongding at cn.cogenda.com> wrote:
> 
> The petsc-3.17.4 is configured on unbuntu 2022.04
> 
> with gcc-9 and icc version 19.1.3.304 (gcc version 9.4.0 compatibility)
> 
> Here is the config and make script
> 
> source /opt/intel/bin/iccvars.sh intel64
> export PETSC_ARCH=arch-linux2-c-opt
> export MPICH_CC=icc
> export MPICH_CXX=icpc
> export MPICH_F77=ifort
> export MPICH_F90=ifort
> python configure  --with-blaslapack-include=/opt/intel/mkl/include --with-blaslapack-lib="-L/opt/intel/mkl/lib/intel64/ -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lpthread -lm -ldl" --with-blacs-include=/opt/intel/mkl/include --with-blacs-lib="-L/opt/intel/mkl/lib/intel64/ -lmkl_blacs_intelmpi_lp64 -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lpthread -lm -ldl" --with-scalapack-include=/opt/intel/mkl/include --with-scalapack-lib="-L/opt/intel/mkl/lib/intel64/ -lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64 -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lpthread -lm -ldl" --with-mkl_pardiso-include=/opt/intel/mkl/include --with-mkl_pardiso-lib="-L/opt/intel/mkl/lib/intel64/ -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lpthread -lm -ldl" --with-mkl_cpardiso-include=/opt/intel/mkl/include --with-mkl_cpardiso-lib="-L/opt/intel/mkl/lib/intel64/ -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lpthread -lm -ldl" --download-mumps=1 --download-parmetis=1 --download-metis=1 --download-ptscotch=1 --with-shared-libraries=1 --with-debugging=0 --with-x=0 --with-mpi-dir=/usr/local/mpich-3.4.2/ --download-superlu=1 --download-superlu_dist=1 --download-suitesparse=1   --with-vendor-compilers=intel COPTFLAGS="-O3 -mavx2" CXXOPTFLAGS="-O3 -mavx2" FOPTFLAGS="-O3 -mavx2" --force
> make
> 
> 
> The CPARDISO is configured as
> 
> ierr = KSPSetType (ksp, (char*) KSPPREONLY); assert(!ierr);
> ierr = PCSetType  (pc, (char*) PCLU); assert(!ierr);
> ierr = PCFactorSetMatSolverType (pc, "mkl_cpardiso"); assert(!ierr);
> 
>  "-mat_mkl_cpardiso_2"  "2"     // "Fill-in reducing ordering for the input matrix", 2 for nested dissection
>  "-mat_mkl_cpardiso_8"  "20"  // "Iterative refinement step"
>  "-mat_mkl_cpardiso_13"  "1"  // "Improved accuracy using (non-) symmetric weighted matching"
> 
> 
> When run the linear solver in parallel, with -info argument,
> 
> the code will last print
> 
> [0] PCSetUp(): Setting up PC for first time
> 
> and run into endless loop
> 
> If run the code with -n 2, with two process  411465 and 411466
> 
> gdb attach shows
> 
> (gdb) attach 411465
> Attaching to process 411465
> [New LWP 411469]
> [New LWP 411470]
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
> 0x00007fede65066b3 in MPIR_Barrier.part.0 () from /usr/local/mpich/lib/libmpi.so.12
> (gdb) bt
> #0  0x00007fede65066b3 in MPIR_Barrier.part.0 () from /usr/local/mpich/lib/libmpi.so.12
> #1  0x00007fede65075fe in PMPI_Barrier () from /usr/local/mpich/lib/libmpi.so.12
> #2  0x00007fede4629409 in MKLMPI_Barrier () from /opt/intel/mkl/lib/intel64/libmkl_blacs_intelmpi_lp64.so
> #3  0x00007fede0732b65 in mkl_pds_lp64_cpardiso_mpi_barrier () from /opt/intel/mkl/lib/intel64/libmkl_core.so
> #4  0x00007feddfdb1a62 in mkl_pds_lp64_cluster_sparse_solver () from /opt/intel/mkl/lib/intel64/libmkl_core.so
> #5  0x00007fede7d7f689 in MatFactorNumeric_MKL_CPARDISO () from /usr/local/petsc-3.17.4/arch-linux2-c-opt/lib/libpetsc.so.3.17
> #6  0x00007fede7912f2d in MatLUFactorNumeric () from /usr/local/petsc-3.17.4/arch-linux2-c-opt/lib/libpetsc.so.3.17
> #7  0x00007fede839da46 in PCSetUp_LU () from /usr/local/petsc-3.17.4/arch-linux2-c-opt/lib/libpetsc.so.3.17
> #8  0x00007fede846fd82 in PCSetUp () from /usr/local/petsc-3.17.4/arch-linux2-c-opt/lib/libpetsc.so.3.17
> #9  0x00007fede84b1e5a in KSPSetUp () from /usr/local/petsc-3.17.4/arch-linux2-c-opt/lib/libpetsc.so.3.17
> #10 0x00007fede84adc89 in KSPSolve_Private () from /usr/local/petsc-3.17.4/arch-linux2-c-opt/lib/libpetsc.so.3.17
> #11 0x00007fede84b66c0 in KSPSolve () from /usr/local/petsc-3.17.4/arch-linux2-c-opt/lib/libpetsc.so.3.17
> #12 0x00007fede86174a3 in SNESSolve_NEWTONLS () from /usr/local/petsc-3.17.4/arch-linux2-c-opt/lib/libpetsc.so.3.17
> #13 0x00007fede85c62d8 in SNESSolve () from /usr/local/petsc-3.17.4/arch-linux2-c-opt/lib/libpetsc.so.3.17
> 
> 
> (gdb) attach 411466
> Attaching to process 411466
> [New LWP 411467]
> [New LWP 411468]
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
> 0x00007f3539638583 in MPIR_Bcast.part.0 () from /usr/local/mpich/lib/libmpi.so.12
> (gdb) bt
> #0  0x00007f3539638583 in MPIR_Bcast.part.0 () from /usr/local/mpich/lib/libmpi.so.12
> #1  0x00007f3539639b90 in PMPI_Bcast () from /usr/local/mpich/lib/libmpi.so.12
> #2  0x00007f3537829532 in MKLMPI_Bcast () from /opt/intel/mkl/lib/intel64/libmkl_blacs_intelmpi_lp64.so
> #3  0x00007f3532faa561 in mkl_pds_lp64_factorize_slave () from /opt/intel/mkl/lib/intel64/libmkl_core.so
> #4  0x00007f3532fb16be in mkl_pds_lp64_cluster_sparse_solver () from /opt/intel/mkl/lib/intel64/libmkl_core.so
> #5  0x00007f353aeac689 in MatFactorNumeric_MKL_CPARDISO () from /usr/local/petsc-3.17.4/arch-linux2-c-opt/lib/libpetsc.so.3.17
> #6  0x00007f353aa3ff2d in MatLUFactorNumeric () from /usr/local/petsc-3.17.4/arch-linux2-c-opt/lib/libpetsc.so.3.17
> #7  0x00007f353b4caa46 in PCSetUp_LU () from /usr/local/petsc-3.17.4/arch-linux2-c-opt/lib/libpetsc.so.3.17
> #8  0x00007f353b59cd82 in PCSetUp () from /usr/local/petsc-3.17.4/arch-linux2-c-opt/lib/libpetsc.so.3.17
> #9  0x00007f353b5dee5a in KSPSetUp () from /usr/local/petsc-3.17.4/arch-linux2-c-opt/lib/libpetsc.so.3.17
> #10 0x00007f353b5dac89 in KSPSolve_Private () from /usr/local/petsc-3.17.4/arch-linux2-c-opt/lib/libpetsc.so.3.17
> #11 0x00007f353b5e36c0 in KSPSolve () from /usr/local/petsc-3.17.4/arch-linux2-c-opt/lib/libpetsc.so.3.17
> #12 0x00007f353b7444a3 in SNESSolve_NEWTONLS () from /usr/local/petsc-3.17.4/arch-linux2-c-opt/lib/libpetsc.so.3.17
> #13 0x00007f353b6f32d8 in SNESSolve () from /usr/local/petsc-3.17.4/arch-linux2-c-opt/lib/libpetsc.so.3.17
> 
> 
> Hope above information is enough.
> 
> 
> Gong Ding
> 
> 
> 
> 
> 
> On 2022/8/11 23:00, Barry Smith wrote:
>>   What do you mean halt? Does it hang, seemingly running forever with no output, does it crash and print an error message (please send the entire error message; cut and paste). Is the matrix something you can share so we can try to reproduce?
>> 
>>   Barry
>> 
>> 
>>> On Aug 11, 2022, at 5:42 AM, Gong Ding <gongding at cn.cogenda.com> wrote:
>>> 
>>> Hi petsc developer,
>>> 
>>> MKL (version 20200004) cpardiso halt on parallel environment since petsc 3.15.
>>> 
>>> we tested that 3.12 and 3.14 works  but 3.15~3.17 (latest) halt.
>>> 
>>> Dose anyone meet the same trouble?
>>> 
>>> Gong Ding
>>> 
>>> 
>>> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220811/319d0c32/attachment.html>

From bsmith at petsc.dev  Thu Aug 11 13:04:18 2022
From: bsmith at petsc.dev (Barry Smith)
Date: Thu, 11 Aug 2022 14:04:18 -0400
Subject: [petsc-users] CPARDISO halt on petsc3.17
In-Reply-To: <2ac32e6c-c31a-ef99-60d2-c37f03bfc34d@cn.cogenda.com>
References: <22f0095c-a008-0512-7d77-2e64470397e1@cn.cogenda.com>
	<4805AD7A-DDE9-42B1-A382-AD16A40F93F4@petsc.dev>
	<c68fd611-b45d-4e58-b7e6-ab5d6890f610@cn.cogenda.com>
	<F3252EC7-ECCF-4A79-A0B0-BDD7F3112999@petsc.dev>
	<2ac32e6c-c31a-ef99-60d2-c37f03bfc34d@cn.cogenda.com>
Message-ID: <352AF509-90EC-4DC6-BF94-FADB91574531@petsc.dev>


  Is this the first call to the MKL solver? 

  Any chance the matrix entries or layout across ranks (different number of local rows) are different in the current run and the previous run?

  Can you save the matrix with MatView() using the binary viewer and get it to us?



> On Aug 11, 2022, at 1:57 PM, Gong Ding <gongding at cn.cogenda.com> wrote:
> 
> On 2022/8/12 01:41, Barry Smith wrote:
>> 
>> -with-mpi-dir=/usr/local/mpich-3.4.2/ 
>> 
>> 
>>> #0  0x00007fede65066b3 in MPIR_Barrier.part.0 () from /usr/local/mpich/lib/libmpi.so.12
>>> #1  0x00007fede65075fe in PMPI_Barrier () from /usr/local/mpich/lib/libmpi.so.12
>>> #2  0x00007fede4629409 in MKLMPI_Barrier () from /opt/intel/mkl/lib/intel64/libmkl_blacs_intelmpi_lp64.so
>> 
>> 
>> There seem to be three MPI floating around? The libmkl_blacs_intelmpi_lp64.so seems to indicate it expects and Intel MPI? The /usr/local/mpich/lib/libmpi.so.12 seems to indicate that the MPI is MPICH installed at /usr/local/mpich but the -with-mpi-dir=/usr/local/mpich-3.4.2/  seems to indicate you are building PETSc with yet a different MPI.
> I have a symbolic link  /usr/local/mpich to  /usr/local/mpich-3.4.2/, its the same 
> 
> 
> 
>> 
>> Given the libmkl_blacs_intelmpi_lp64.so I think you need to start by figuring out how to do the configure using the Intel MPI only.
>> 
>> It is hanging because of 
>> 
>>> #2  0x00007fede4629409 in MKLMPI_Barrier () from /opt/intel/mkl/lib/intel64/libmkl_blacs_intelmpi_lp64.so
>> 
>> versus 
>> 
>>> #2  0x00007f3537829532 in MKLMPI_Bcast () from /opt/intel/mkl/lib/intel64/libmkl_blacs_intelmpi_lp64.so
>> 
>> since this is inside the MKL code I don't see why PETSc would cause impossible configuration. Nor do I see why the same code should run with earlier PETSc versions, are the configure options and the resulting libraries used (MKL, MPI etc) exactly the same for all PETSc versions you tested with?
>> 
> I had used exactly the same configuration from 3.12 to 3.17.   The halt happens from 3.15.
> Anyway, thanks to point out that libmkl_blacs_intelmpi_lp64 may caused the problem. I will try to debug it.
> 
> 
> 
>> 
>>> On Aug 11, 2022, at 12:42 PM, Gong Ding <gongding at cn.cogenda.com <mailto:gongding at cn.cogenda.com>> wrote:
>>> 
>>> The petsc-3.17.4 is configured on unbuntu 2022.04
>>> 
>>> with gcc-9 and icc version 19.1.3.304 (gcc version 9.4.0 compatibility)
>>> 
>>> Here is the config and make script
>>> 
>>> source /opt/intel/bin/iccvars.sh intel64
>>> export PETSC_ARCH=arch-linux2-c-opt
>>> export MPICH_CC=icc
>>> export MPICH_CXX=icpc
>>> export MPICH_F77=ifort
>>> export MPICH_F90=ifort
>>> python configure  --with-blaslapack-include=/opt/intel/mkl/include --with-blaslapack-lib="-L/opt/intel/mkl/lib/intel64/ -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lpthread -lm -ldl" --with-blacs-include=/opt/intel/mkl/include --with-blacs-lib="-L/opt/intel/mkl/lib/intel64/ -lmkl_blacs_intelmpi_lp64 -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lpthread -lm -ldl" --with-scalapack-include=/opt/intel/mkl/include --with-scalapack-lib="-L/opt/intel/mkl/lib/intel64/ -lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64 -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lpthread -lm -ldl" --with-mkl_pardiso-include=/opt/intel/mkl/include --with-mkl_pardiso-lib="-L/opt/intel/mkl/lib/intel64/ -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lpthread -lm -ldl" --with-mkl_cpardiso-include=/opt/intel/mkl/include --with-mkl_cpardiso-lib="-L/opt/intel/mkl/lib/intel64/ -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lpthread -lm -ldl" --download-mumps=1 --download-parmetis=1 --download-metis=1 --download-ptscotch=1 --with-shared-libraries=1 --with-debugging=0 --with-x=0 --with-mpi-dir=/usr/local/mpich-3.4.2/ --download-superlu=1 --download-superlu_dist=1 --download-suitesparse=1   --with-vendor-compilers=intel COPTFLAGS="-O3 -mavx2" CXXOPTFLAGS="-O3 -mavx2" FOPTFLAGS="-O3 -mavx2" --force
>>> make
>>> 
>>> 
>>> The CPARDISO is configured as
>>> 
>>> ierr = KSPSetType (ksp, (char*) KSPPREONLY); assert(!ierr);
>>> ierr = PCSetType  (pc, (char*) PCLU); assert(!ierr);
>>> ierr = PCFactorSetMatSolverType (pc, "mkl_cpardiso"); assert(!ierr);
>>> 
>>>  "-mat_mkl_cpardiso_2"  "2"     // "Fill-in reducing ordering for the input matrix", 2 for nested dissection
>>>  "-mat_mkl_cpardiso_8"  "20"  // "Iterative refinement step"
>>>  "-mat_mkl_cpardiso_13"  "1"  // "Improved accuracy using (non-) symmetric weighted matching"
>>> 
>>> 
>>> When run the linear solver in parallel, with -info argument,
>>> 
>>> the code will last print
>>> 
>>> [0] PCSetUp(): Setting up PC for first time
>>> 
>>> and run into endless loop
>>> 
>>> If run the code with -n 2, with two process  411465 and 411466
>>> 
>>> gdb attach shows
>>> 
>>> (gdb) attach 411465
>>> Attaching to process 411465
>>> [New LWP 411469]
>>> [New LWP 411470]
>>> [Thread debugging using libthread_db enabled]
>>> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
>>> 0x00007fede65066b3 in MPIR_Barrier.part.0 () from /usr/local/mpich/lib/libmpi.so.12
>>> (gdb) bt
>>> #0  0x00007fede65066b3 in MPIR_Barrier.part.0 () from /usr/local/mpich/lib/libmpi.so.12
>>> #1  0x00007fede65075fe in PMPI_Barrier () from /usr/local/mpich/lib/libmpi.so.12
>>> #2  0x00007fede4629409 in MKLMPI_Barrier () from /opt/intel/mkl/lib/intel64/libmkl_blacs_intelmpi_lp64.so
>>> #3  0x00007fede0732b65 in mkl_pds_lp64_cpardiso_mpi_barrier () from /opt/intel/mkl/lib/intel64/libmkl_core.so
>>> #4  0x00007feddfdb1a62 in mkl_pds_lp64_cluster_sparse_solver () from /opt/intel/mkl/lib/intel64/libmkl_core.so
>>> #5  0x00007fede7d7f689 in MatFactorNumeric_MKL_CPARDISO () from /usr/local/petsc-3.17.4/arch-linux2-c-opt/lib/libpetsc.so.3.17
>>> #6  0x00007fede7912f2d in MatLUFactorNumeric () from /usr/local/petsc-3.17.4/arch-linux2-c-opt/lib/libpetsc.so.3.17
>>> #7  0x00007fede839da46 in PCSetUp_LU () from /usr/local/petsc-3.17.4/arch-linux2-c-opt/lib/libpetsc.so.3.17
>>> #8  0x00007fede846fd82 in PCSetUp () from /usr/local/petsc-3.17.4/arch-linux2-c-opt/lib/libpetsc.so.3.17
>>> #9  0x00007fede84b1e5a in KSPSetUp () from /usr/local/petsc-3.17.4/arch-linux2-c-opt/lib/libpetsc.so.3.17
>>> #10 0x00007fede84adc89 in KSPSolve_Private () from /usr/local/petsc-3.17.4/arch-linux2-c-opt/lib/libpetsc.so.3.17
>>> #11 0x00007fede84b66c0 in KSPSolve () from /usr/local/petsc-3.17.4/arch-linux2-c-opt/lib/libpetsc.so.3.17
>>> #12 0x00007fede86174a3 in SNESSolve_NEWTONLS () from /usr/local/petsc-3.17.4/arch-linux2-c-opt/lib/libpetsc.so.3.17
>>> #13 0x00007fede85c62d8 in SNESSolve () from /usr/local/petsc-3.17.4/arch-linux2-c-opt/lib/libpetsc.so.3.17
>>> 
>>> 
>>> (gdb) attach 411466
>>> Attaching to process 411466
>>> [New LWP 411467]
>>> [New LWP 411468]
>>> [Thread debugging using libthread_db enabled]
>>> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
>>> 0x00007f3539638583 in MPIR_Bcast.part.0 () from /usr/local/mpich/lib/libmpi.so.12
>>> (gdb) bt
>>> #0  0x00007f3539638583 in MPIR_Bcast.part.0 () from /usr/local/mpich/lib/libmpi.so.12
>>> #1  0x00007f3539639b90 in PMPI_Bcast () from /usr/local/mpich/lib/libmpi.so.12
>>> #2  0x00007f3537829532 in MKLMPI_Bcast () from /opt/intel/mkl/lib/intel64/libmkl_blacs_intelmpi_lp64.so
>>> #3  0x00007f3532faa561 in mkl_pds_lp64_factorize_slave () from /opt/intel/mkl/lib/intel64/libmkl_core.so
>>> #4  0x00007f3532fb16be in mkl_pds_lp64_cluster_sparse_solver () from /opt/intel/mkl/lib/intel64/libmkl_core.so
>>> #5  0x00007f353aeac689 in MatFactorNumeric_MKL_CPARDISO () from /usr/local/petsc-3.17.4/arch-linux2-c-opt/lib/libpetsc.so.3.17
>>> #6  0x00007f353aa3ff2d in MatLUFactorNumeric () from /usr/local/petsc-3.17.4/arch-linux2-c-opt/lib/libpetsc.so.3.17
>>> #7  0x00007f353b4caa46 in PCSetUp_LU () from /usr/local/petsc-3.17.4/arch-linux2-c-opt/lib/libpetsc.so.3.17
>>> #8  0x00007f353b59cd82 in PCSetUp () from /usr/local/petsc-3.17.4/arch-linux2-c-opt/lib/libpetsc.so.3.17
>>> #9  0x00007f353b5dee5a in KSPSetUp () from /usr/local/petsc-3.17.4/arch-linux2-c-opt/lib/libpetsc.so.3.17
>>> #10 0x00007f353b5dac89 in KSPSolve_Private () from /usr/local/petsc-3.17.4/arch-linux2-c-opt/lib/libpetsc.so.3.17
>>> #11 0x00007f353b5e36c0 in KSPSolve () from /usr/local/petsc-3.17.4/arch-linux2-c-opt/lib/libpetsc.so.3.17
>>> #12 0x00007f353b7444a3 in SNESSolve_NEWTONLS () from /usr/local/petsc-3.17.4/arch-linux2-c-opt/lib/libpetsc.so.3.17
>>> #13 0x00007f353b6f32d8 in SNESSolve () from /usr/local/petsc-3.17.4/arch-linux2-c-opt/lib/libpetsc.so.3.17
>>> 
>>> 
>>> Hope above information is enough.
>>> 
>>> 
>>> Gong Ding
>>> 
>>> 
>>> 
>>> 
>>> 
>>> On 2022/8/11 23:00, Barry Smith wrote:
>>>>   What do you mean halt? Does it hang, seemingly running forever with no output, does it crash and print an error message (please send the entire error message; cut and paste). Is the matrix something you can share so we can try to reproduce?
>>>> 
>>>>   Barry
>>>> 
>>>> 
>>>>> On Aug 11, 2022, at 5:42 AM, Gong Ding <gongding at cn.cogenda.com <mailto:gongding at cn.cogenda.com>> wrote:
>>>>> 
>>>>> Hi petsc developer,
>>>>> 
>>>>> MKL (version 20200004) cpardiso halt on parallel environment since petsc 3.15.
>>>>> 
>>>>> we tested that 3.12 and 3.14 works  but 3.15~3.17 (latest) halt.
>>>>> 
>>>>> Dose anyone meet the same trouble?
>>>>> 
>>>>> Gong Ding
>>>>> 
>>>>> 
>>>>> 
>> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220811/78fc4d88/attachment-0001.html>

From nguyenquynh9537 at gmail.com  Fri Aug 12 01:21:33 2022
From: nguyenquynh9537 at gmail.com (Quynh Nguyen)
Date: Fri, 12 Aug 2022 15:21:33 +0900
Subject: [petsc-users] DMDA ghosted boundary
Message-ID: <CAFma+BnGu-cLjLpWn4H07vHzi-_8jAWPYf0W+VzXOa3OD-c4CQ@mail.gmail.com>

Dear petsc developers and users,

I am a new petsc user ^_^.
I have some questions about using DMDA_BOUNDARY_GHOSTED.

1. I have 2D Poisson code with domain (nx, ny), and I used
DMDA_BOUNDARY_NONE, then the size of the matrix is (nx*ny)X(nx*ny). *what
is the size of the matrix if I used  DMDA_BOUNDARY_GHOSTED? is it the same
size as DMDA_BOUNDARY_NONE*?

2. if the matrix size of DMDA_BOUNDARY_GHOSTED isn't the same as
DMDA_BOUNDARY_NONE, *how i can view the matrix including ghost points*? I
ask this one because I used MatView(P, PETSC_VIEWER_STDOUT_WORLD) to show
the matrix but excluded ghost points.

I hope someone can explain my wonder :)

P/S: maybe, those question is simple and a bit stupid ^_^.

Have a nice day,

-- 
Quynh Nguyen
Graduate student, Ocean Engineering.
School of Naval Architecture and Ocean Engineering, University of Ulsan.
-----------------------------------------------------------------------------------------
Mobile: (+82) 010-9394-5595
Email: nguyenquynh9537 at gmail.com; nguyenthiquynhb_t58 at hus.edu.vn
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220812/47b95738/attachment.html>

From knepley at gmail.com  Fri Aug 12 05:26:55 2022
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 12 Aug 2022 06:26:55 -0400
Subject: [petsc-users] DMDA ghosted boundary
In-Reply-To: <CAFma+BnGu-cLjLpWn4H07vHzi-_8jAWPYf0W+VzXOa3OD-c4CQ@mail.gmail.com>
References: <CAFma+BnGu-cLjLpWn4H07vHzi-_8jAWPYf0W+VzXOa3OD-c4CQ@mail.gmail.com>
Message-ID: <CAMYG4GkAywr933zE2XtMWd=0DqbjxH7BtmixijF1mp1AnBx-Pg@mail.gmail.com>

On Fri, Aug 12, 2022 at 2:21 AM Quynh Nguyen <nguyenquynh9537 at gmail.com>
wrote:

> Dear petsc developers and users,
>
> I am a new petsc user ^_^.
> I have some questions about using DMDA_BOUNDARY_GHOSTED.
>
> 1. I have 2D Poisson code with domain (nx, ny), and I used
> DMDA_BOUNDARY_NONE, then the size of the matrix is (nx*ny)X(nx*ny). *what
> is the size of the matrix if I used  DMDA_BOUNDARY_GHOSTED? is it the same
> size as DMDA_BOUNDARY_NONE*?
>

Yes, the matrix should be the same size, but local vectors have extra ghost
node slots.


> 2. if the matrix size of DMDA_BOUNDARY_GHOSTED isn't the same as
> DMDA_BOUNDARY_NONE, *how i can view the matrix including ghost points*? I
> ask this one because I used MatView(P, PETSC_VIEWER_STDOUT_WORLD) to show
> the matrix but excluded ghost points.
>

Ghost nodes are there so that stencil operations on local vectors work at
boundaries. You fill in the slots with boundary information.

  Thanks,

     Matt


> I hope someone can explain my wonder :)
>
> P/S: maybe, those question is simple and a bit stupid ^_^.
>
> Have a nice day,
>
> --
> Quynh Nguyen
> Graduate student, Ocean Engineering.
> School of Naval Architecture and Ocean Engineering, University of Ulsan.
>
> -----------------------------------------------------------------------------------------
> Mobile: (+82) 010-9394-5595
> Email: nguyenquynh9537 at gmail.com; nguyenthiquynhb_t58 at hus.edu.vn
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220812/e2cbe5d1/attachment.html>

From nguyenquynh9537 at gmail.com  Fri Aug 12 06:18:53 2022
From: nguyenquynh9537 at gmail.com (nguyenquynh9537)
Date: Fri, 12 Aug 2022 20:18:53 +0900
Subject: [petsc-users] DMDA ghosted boundary
Message-ID: <62f63721.170a0220.f21c0.29af@mx.google.com>

I got it, thank you so much!
null
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220812/f4fd3f58/attachment.html>

From d.scott at epcc.ed.ac.uk  Fri Aug 12 08:27:26 2022
From: d.scott at epcc.ed.ac.uk (David Scott)
Date: Fri, 12 Aug 2022 14:27:26 +0100
Subject: [petsc-users] DMDA ghosted boundary
In-Reply-To: <CAMYG4GkAywr933zE2XtMWd=0DqbjxH7BtmixijF1mp1AnBx-Pg@mail.gmail.com>
References: <CAFma+BnGu-cLjLpWn4H07vHzi-_8jAWPYf0W+VzXOa3OD-c4CQ@mail.gmail.com>
	<CAMYG4GkAywr933zE2XtMWd=0DqbjxH7BtmixijF1mp1AnBx-Pg@mail.gmail.com>
Message-ID: <b7f9ec88-a7df-6eb8-ebda-e96e1223e9a6@epcc.ed.ac.uk>



On 12/08/2022 11:26, Matthew Knepley wrote:
This email was sent to you by someone outside the University.
You should only click on links or attachments if you are certain that the email is genuine and the content is safe.
On Fri, Aug 12, 2022 at 2:21 AM Quynh Nguyen <nguyenquynh9537 at gmail.com<mailto:nguyenquynh9537 at gmail.com>> wrote:
Dear petsc developers and users,

I am a new petsc user ^_^.
I have some questions about using DMDA_BOUNDARY_GHOSTED.

1. I have 2D Poisson code with domain (nx, ny), and I used  DMDA_BOUNDARY_NONE, then the size of the matrix is (nx*ny)X(nx*ny). what is the size of the matrix if I used  DMDA_BOUNDARY_GHOSTED? is it the same size as DMDA_BOUNDARY_NONE?

Yes, the matrix should be the same size, but local vectors have extra ghost node slots.

2. if the matrix size of DMDA_BOUNDARY_GHOSTED isn't the same as DMDA_BOUNDARY_NONE, how i can view the matrix including ghost points? I ask this one because I used MatView(P, PETSC_VIEWER_STDOUT_WORLD) to show the matrix but excluded ghost points.

Ghost nodes are there so that stencil operations on local vectors work at boundaries. You fill in the slots with boundary information.

I can see how this works if you are writing your own solver but can it be used with a KSP? In the case of periodic boundary conditions the boundary values are "filled in" for you by PETSc. Is there any way for a user to supply boundary information to a PETSc solver when using DMDA_BOUNDARY_GHOSTED?

Thanks,

David

  Thanks,

     Matt

I hope someone can explain my wonder :)

P/S: maybe, those question is simple and a bit stupid ^_^.

Have a nice day,

--
Quynh Nguyen
Graduate student, Ocean Engineering.
School of Naval Architecture and Ocean Engineering, University of Ulsan.
-----------------------------------------------------------------------------------------
Mobile: (+82) 010-9394-5595
Email: nguyenquynh9537 at gmail.com<mailto:nguyenquynh9537 at gmail.com>; nguyenthiquynhb_t58 at hus.edu.vn<mailto:nguyenthiquynhb_t58 at hus.edu.vn>


--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/<http://www.cse.buffalo.edu/~knepley/>

The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Is e buidheann carthannais a th? ann an Oilthigh Dh?n ?ideann, cl?raichte an Alba, ?ireamh cl?raidh SC005336.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220812/192a21ba/attachment-0001.html>

From ajaramillopalma at gmail.com  Fri Aug 12 09:28:27 2022
From: ajaramillopalma at gmail.com (Alfredo Jaramillo)
Date: Fri, 12 Aug 2022 08:28:27 -0600
Subject: [petsc-users] MatView to file
Message-ID: <CAKoAOWpqdnnC-A0bQOrCeYmtYOC-guNLBfXha7kpDX4CvNWWmw@mail.gmail.com>

Dear developers,

I'm writing a sparse matrix into a file by doing

    if (dump_mat) {
        PetscViewer viewer;
        PetscViewerASCIIOpen(PETSC_COMM_WORLD,"mat-par-aux.m",&viewer);
        PetscViewerPushFormat(viewer, PETSC_VIEWER_ASCII_MATLAB);
        MatView(A,viewer);
    }

This works perfectly for small cases.
The program crashes for a case where the matrix A is of order 1 million but
with only 4 million non-zero elements.

Maybe at some point petsc is full-sizing A?

Thank you,
Alfredo
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220812/26897e35/attachment.html>

From mfadams at lbl.gov  Fri Aug 12 11:02:10 2022
From: mfadams at lbl.gov (Mark Adams)
Date: Fri, 12 Aug 2022 12:02:10 -0400
Subject: [petsc-users] MatView to file
In-Reply-To: <CAKoAOWpqdnnC-A0bQOrCeYmtYOC-guNLBfXha7kpDX4CvNWWmw@mail.gmail.com>
References: <CAKoAOWpqdnnC-A0bQOrCeYmtYOC-guNLBfXha7kpDX4CvNWWmw@mail.gmail.com>
Message-ID: <CADOhEh5LBDieYhPiCH5U1z-wJGLkF6uNyFMxRoTN-H7AxzPWtA@mail.gmail.com>

You also want:

    PetscCall(PetscViewerPopFormat(viewer));
    PetscCall(PetscViewerDestroy(&viewer));

This should not be a problem.
If this is a segv and you configure it with   '--with-debugging=1', you
should get a stack trace, which would help immensely.
Or run in a debugger to get a stack trace.

Thanks,
Mark



On Fri, Aug 12, 2022 at 11:26 AM Alfredo Jaramillo <
ajaramillopalma at gmail.com> wrote:

> Dear developers,
>
> I'm writing a sparse matrix into a file by doing
>
>     if (dump_mat) {
>         PetscViewer viewer;
>         PetscViewerASCIIOpen(PETSC_COMM_WORLD,"mat-par-aux.m",&viewer);
>         PetscViewerPushFormat(viewer, PETSC_VIEWER_ASCII_MATLAB);
>         MatView(A,viewer);
>     }
>
> This works perfectly for small cases.
> The program crashes for a case where the matrix A is of order 1 million
> but with only 4 million non-zero elements.
>
> Maybe at some point petsc is full-sizing A?
>
> Thank you,
> Alfredo
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220812/4615ba93/attachment.html>

From ajaramillopalma at gmail.com  Fri Aug 12 11:59:54 2022
From: ajaramillopalma at gmail.com (Alfredo Jaramillo)
Date: Fri, 12 Aug 2022 10:59:54 -0600
Subject: [petsc-users] MatView to file
In-Reply-To: <CADOhEh5LBDieYhPiCH5U1z-wJGLkF6uNyFMxRoTN-H7AxzPWtA@mail.gmail.com>
References: <CAKoAOWpqdnnC-A0bQOrCeYmtYOC-guNLBfXha7kpDX4CvNWWmw@mail.gmail.com>
	<CADOhEh5LBDieYhPiCH5U1z-wJGLkF6uNyFMxRoTN-H7AxzPWtA@mail.gmail.com>
Message-ID: <CAKoAOWoDFOg32Pu=uSU8E6-1yGeqNLYiX9X09gX5TQFZ1bFdug@mail.gmail.com>

Hello Mark,
Thank you, I added the lines that you sent.
This only happens when running the code with more than 1 process. With only
1 MPI process the matrix is printed out.
With 2 processes or more I observed the program begins to allocate RAM
until it exceeds the computer capacity (32GB) so I wasn't able to get the
stack trace.

However, I was able to reproduce the problem by compiling
src/ksp/ksp/tutorials/ex54.c.html
<https://petsc.org/release/src/ksp/ksp/tutorials/ex54.c.html> (modifying
line 144) and running it with

mpirun -np 2 ex54 -ne 1000

This gives a sparse matrix of order ~1 million. When running ex54 with only
one MPI process I don't observe this excessive allocation and the matrix is
printed out.

Thanks,
Alfredo



On Fri, Aug 12, 2022 at 10:02 AM Mark Adams <mfadams at lbl.gov> wrote:

> You also want:
>
>     PetscCall(PetscViewerPopFormat(viewer));
>     PetscCall(PetscViewerDestroy(&viewer));
>
> This should not be a problem.
> If this is a segv and you configure it with   '--with-debugging=1', you
> should get a stack trace, which would help immensely.
> Or run in a debugger to get a stack trace.
>
> Thanks,
> Mark
>
>
>
> On Fri, Aug 12, 2022 at 11:26 AM Alfredo Jaramillo <
> ajaramillopalma at gmail.com> wrote:
>
>> Dear developers,
>>
>> I'm writing a sparse matrix into a file by doing
>>
>>     if (dump_mat) {
>>         PetscViewer viewer;
>>         PetscViewerASCIIOpen(PETSC_COMM_WORLD,"mat-par-aux.m",&viewer);
>>         PetscViewerPushFormat(viewer, PETSC_VIEWER_ASCII_MATLAB);
>>         MatView(A,viewer);
>>     }
>>
>> This works perfectly for small cases.
>> The program crashes for a case where the matrix A is of order 1 million
>> but with only 4 million non-zero elements.
>>
>> Maybe at some point petsc is full-sizing A?
>>
>> Thank you,
>> Alfredo
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220812/7191b5cc/attachment.html>

From david.knezevic at akselos.com  Fri Aug 12 12:13:55 2022
From: david.knezevic at akselos.com (David Knezevic)
Date: Fri, 12 Aug 2022 13:13:55 -0400
Subject: [petsc-users] Configure error with PETSc 3.17.4 on Ubuntu 22.04
Message-ID: <CAJCWK9DPbW36NoCxVGEx=-ugqj24J3LaQT=cRu0w18k0hLZgTA@mail.gmail.com>

I get a configure error when building PETSc 3.17.4 on Ubuntu 22.04. The
configure.log is attached.

This error seems to only occur when we use --with-clanguage=cxx. We need to
use that object when building with complex numbers.

Guidance on how to resolve this would be most appreciated.

Thanks,
David
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220812/621ea849/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: configure.log
Type: text/x-log
Size: 1263308 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220812/621ea849/attachment-0001.bin>

From knepley at gmail.com  Fri Aug 12 12:15:45 2022
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 12 Aug 2022 13:15:45 -0400
Subject: [petsc-users] DMDA ghosted boundary
In-Reply-To: <b7f9ec88-a7df-6eb8-ebda-e96e1223e9a6@epcc.ed.ac.uk>
References: <CAFma+BnGu-cLjLpWn4H07vHzi-_8jAWPYf0W+VzXOa3OD-c4CQ@mail.gmail.com>
	<CAMYG4GkAywr933zE2XtMWd=0DqbjxH7BtmixijF1mp1AnBx-Pg@mail.gmail.com>
	<b7f9ec88-a7df-6eb8-ebda-e96e1223e9a6@epcc.ed.ac.uk>
Message-ID: <CAMYG4GmzzvfqPuXRBE9n1ZFZ5sCU=Ao95KNNPtPCvD4O7_4ZrQ@mail.gmail.com>

On Fri, Aug 12, 2022 at 9:27 AM David Scott <d.scott at epcc.ed.ac.uk> wrote:

> On 12/08/2022 11:26, Matthew Knepley wrote:
>
> This email was sent to you by someone outside the University.
> You should only click on links or attachments if you are certain that the
> email is genuine and the content is safe.
> On Fri, Aug 12, 2022 at 2:21 AM Quynh Nguyen <nguyenquynh9537 at gmail.com>
> wrote:
>
>> Dear petsc developers and users,
>>
>> I am a new petsc user ^_^.
>> I have some questions about using DMDA_BOUNDARY_GHOSTED.
>>
>> 1. I have 2D Poisson code with domain (nx, ny), and I used
>> DMDA_BOUNDARY_NONE, then the size of the matrix is (nx*ny)X(nx*ny). *what
>> is the size of the matrix if I used  DMDA_BOUNDARY_GHOSTED? is it the same
>> size as DMDA_BOUNDARY_NONE*?
>>
>
> Yes, the matrix should be the same size, but local vectors have extra
> ghost node slots.
>
>
>> 2. if the matrix size of DMDA_BOUNDARY_GHOSTED isn't the same as
>> DMDA_BOUNDARY_NONE, *how i can view the matrix including ghost points*?
>> I ask this one because I used MatView(P, PETSC_VIEWER_STDOUT_WORLD) to show
>> the matrix but excluded ghost points.
>>
>
> Ghost nodes are there so that stencil operations on local vectors work at
> boundaries. You fill in the slots with boundary information.
>
> I can see how this works if you are writing your own solver but can it be
> used with a KSP? In the case of periodic boundary conditions the boundary
> values are "filled in" for you by PETSc. Is there any way for a user to
> supply boundary information to a PETSc solver when using
> DMDA_BOUNDARY_GHOSTED?
>

Right now, there is not. This is mainly due to the lack of consensus for
boundary condition specification. Periodic conditions are simple in this
respect.

For example, Plex has such a thing

  https://petsc.org/main/docs/manualpages/DM/DMAddBoundary/

that can be used to fill in these constrained spots

  https://petsc.org/main/docs/manualpages/DMPLEX/DMPlexInsertBoundaryValues/

But there have been many complaints about this interface (and I suspect any
interface we choose).

Maybe there is something that we could write that would make it easier for
you with DMDA?

  Thanks,

     Matt


> Thanks,
>
> David
>
>   Thanks,
>
>      Matt
>
>
>> I hope someone can explain my wonder :)
>>
>> P/S: maybe, those question is simple and a bit stupid ^_^.
>>
>> Have a nice day,
>>
>> --
>> Quynh Nguyen
>> Graduate student, Ocean Engineering.
>> School of Naval Architecture and Ocean Engineering, University of Ulsan.
>>
>> -----------------------------------------------------------------------------------------
>> Mobile: (+82) 010-9394-5595
>> Email: nguyenquynh9537 at gmail.com; nguyenthiquynhb_t58 at hus.edu.vn
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
>
> The University of Edinburgh is a charitable body, registered in Scotland,
> with registration number SC005336. Is e buidheann carthannais a th? ann an
> Oilthigh Dh?n ?ideann, cl?raichte an Alba, ?ireamh cl?raidh SC005336.
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220812/0e2226b1/attachment.html>

From knepley at gmail.com  Fri Aug 12 12:23:59 2022
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 12 Aug 2022 13:23:59 -0400
Subject: [petsc-users] Configure error with PETSc 3.17.4 on Ubuntu 22.04
In-Reply-To: <CAJCWK9DPbW36NoCxVGEx=-ugqj24J3LaQT=cRu0w18k0hLZgTA@mail.gmail.com>
References: <CAJCWK9DPbW36NoCxVGEx=-ugqj24J3LaQT=cRu0w18k0hLZgTA@mail.gmail.com>
Message-ID: <CAMYG4G=dTFFPcmN+m6SoJffmnpXeU9E-6XHrjZT2Np3mmnW2Nw@mail.gmail.com>

On Fri, Aug 12, 2022 at 1:14 PM David Knezevic via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> I get a configure error when building PETSc 3.17.4 on Ubuntu 22.04. The
> configure.log is attached.
>
> This error seems to only occur when we use --with-clanguage=cxx. We need
> to use that object when building with complex numbers.
>
> Guidance on how to resolve this would be most appreciated.
>

1. I think your existing MUMPS build is incompatible (but it is hard for us
to tell). Remove it

  rm -rf $PETSC_DIR/$PETSC_ARCH/externalpackages/git.mumps

and reconfigure

2. You can also use C99 for complex numbers, but your application code
might already be using C++.

  Thanks,

     Matt


> Thanks,
> David
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220812/6c74aae3/attachment.html>

From balay at mcs.anl.gov  Fri Aug 12 12:25:37 2022
From: balay at mcs.anl.gov (Satish Balay)
Date: Fri, 12 Aug 2022 12:25:37 -0500 (CDT)
Subject: [petsc-users] Configure error with PETSc 3.17.4 on Ubuntu 22.04
In-Reply-To: <CAJCWK9DPbW36NoCxVGEx=-ugqj24J3LaQT=cRu0w18k0hLZgTA@mail.gmail.com>
References: <CAJCWK9DPbW36NoCxVGEx=-ugqj24J3LaQT=cRu0w18k0hLZgTA@mail.gmail.com>
Message-ID: <5a7ef623-35de-4061-db56-4ca7b6cd288e@mcs.anl.gov>

We've seen issues with Ubuntu built MPI. Can you try --download-mpich and see if that helps?

Also  you don't need --with-clanguage=cxx for complex. PETSc can be built with c99 compilex - i.e "--with-clanguage=c --with-scalar-type=complex" should work.

Satish

On Fri, 12 Aug 2022, David Knezevic via petsc-users wrote:

> I get a configure error when building PETSc 3.17.4 on Ubuntu 22.04. The
> configure.log is attached.
> 
> This error seems to only occur when we use --with-clanguage=cxx. We need to
> use that object when building with complex numbers.
> 
> Guidance on how to resolve this would be most appreciated.
> 
> Thanks,
> David
> 


From david.knezevic at akselos.com  Fri Aug 12 12:32:10 2022
From: david.knezevic at akselos.com (David Knezevic)
Date: Fri, 12 Aug 2022 13:32:10 -0400
Subject: [petsc-users] Configure error with PETSc 3.17.4 on Ubuntu 22.04
In-Reply-To: <CAMYG4G=dTFFPcmN+m6SoJffmnpXeU9E-6XHrjZT2Np3mmnW2Nw@mail.gmail.com>
References: <CAJCWK9DPbW36NoCxVGEx=-ugqj24J3LaQT=cRu0w18k0hLZgTA@mail.gmail.com>
	<CAMYG4G=dTFFPcmN+m6SoJffmnpXeU9E-6XHrjZT2Np3mmnW2Nw@mail.gmail.com>
Message-ID: <CAJCWK9CgbWhUGKV+b+OS-oRoyAdq_dR_-zrFmavQWDQEyKys7Q@mail.gmail.com>

Regarding 1.: We're using --download-mumps, so this is the mumps that was
installed during config. There was no mumps build prior to that. Does that
change anything?

Regarding 2.: We need --with-clanguage=cxx since we're also using an
external C++ library (libMesh). We've been using --with-clanguage=cxx in
PETSc with libMesh for years...



On Fri, Aug 12, 2022 at 1:24 PM Matthew Knepley <knepley at gmail.com> wrote:

> On Fri, Aug 12, 2022 at 1:14 PM David Knezevic via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
>
>> I get a configure error when building PETSc 3.17.4 on Ubuntu 22.04. The
>> configure.log is attached.
>>
>> This error seems to only occur when we use --with-clanguage=cxx. We need
>> to use that object when building with complex numbers.
>>
>> Guidance on how to resolve this would be most appreciated.
>>
>
> 1. I think your existing MUMPS build is incompatible (but it is hard for
> us to tell). Remove it
>
>   rm -rf $PETSC_DIR/$PETSC_ARCH/externalpackages/git.mumps
>
> and reconfigure
>
> 2. You can also use C99 for complex numbers, but your application code
> might already be using C++.
>
>   Thanks,
>
>      Matt
>
>
>> Thanks,
>> David
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220812/7d6cf591/attachment.html>

From knepley at gmail.com  Fri Aug 12 12:47:04 2022
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 12 Aug 2022 13:47:04 -0400
Subject: [petsc-users] Configure error with PETSc 3.17.4 on Ubuntu 22.04
In-Reply-To: <CAJCWK9CgbWhUGKV+b+OS-oRoyAdq_dR_-zrFmavQWDQEyKys7Q@mail.gmail.com>
References: <CAJCWK9DPbW36NoCxVGEx=-ugqj24J3LaQT=cRu0w18k0hLZgTA@mail.gmail.com>
	<CAMYG4G=dTFFPcmN+m6SoJffmnpXeU9E-6XHrjZT2Np3mmnW2Nw@mail.gmail.com>
	<CAJCWK9CgbWhUGKV+b+OS-oRoyAdq_dR_-zrFmavQWDQEyKys7Q@mail.gmail.com>
Message-ID: <CAMYG4GksDL-cYoKDQsCP9s1rKuUMO0jz_RroE9YVstbFy088sA@mail.gmail.com>

On Fri, Aug 12, 2022 at 1:32 PM David Knezevic <david.knezevic at akselos.com>
wrote:

> Regarding 1.: We're using --download-mumps, so this is the mumps that was
> installed during config. There was no mumps build prior to that. Does that
> change anything?
>

I did not see the build in the log. Can you try removing and reconfiguring?
Then I can see everything.

  Thanks,

     Matt


> Regarding 2.: We need --with-clanguage=cxx since we're also using an
> external C++ library (libMesh). We've been using --with-clanguage=cxx in
> PETSc with libMesh for years...
>
>
>
> On Fri, Aug 12, 2022 at 1:24 PM Matthew Knepley <knepley at gmail.com> wrote:
>
>> On Fri, Aug 12, 2022 at 1:14 PM David Knezevic via petsc-users <
>> petsc-users at mcs.anl.gov> wrote:
>>
>>> I get a configure error when building PETSc 3.17.4 on Ubuntu 22.04. The
>>> configure.log is attached.
>>>
>>> This error seems to only occur when we use --with-clanguage=cxx. We need
>>> to use that object when building with complex numbers.
>>>
>>> Guidance on how to resolve this would be most appreciated.
>>>
>>
>> 1. I think your existing MUMPS build is incompatible (but it is hard for
>> us to tell). Remove it
>>
>>   rm -rf $PETSC_DIR/$PETSC_ARCH/externalpackages/git.mumps
>>
>> and reconfigure
>>
>> 2. You can also use C99 for complex numbers, but your application code
>> might already be using C++.
>>
>>   Thanks,
>>
>>      Matt
>>
>>
>>> Thanks,
>>> David
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/~knepley/>
>>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220812/f8152c00/attachment.html>

From david.knezevic at akselos.com  Fri Aug 12 12:51:41 2022
From: david.knezevic at akselos.com (David Knezevic)
Date: Fri, 12 Aug 2022 13:51:41 -0400
Subject: [petsc-users] Configure error with PETSc 3.17.4 on Ubuntu 22.04
In-Reply-To: <5a7ef623-35de-4061-db56-4ca7b6cd288e@mcs.anl.gov>
References: <CAJCWK9DPbW36NoCxVGEx=-ugqj24J3LaQT=cRu0w18k0hLZgTA@mail.gmail.com>
	<5a7ef623-35de-4061-db56-4ca7b6cd288e@mcs.anl.gov>
Message-ID: <CAJCWK9A3z3q+_j9JBszbbmfyHzE+=Kd1FhuCz3qyL=piAcb7cw@mail.gmail.com>

Thanks for this suggestion, using --download-mpich got it working!

Thanks,
David



On Fri, Aug 12, 2022 at 1:25 PM Satish Balay <balay at mcs.anl.gov> wrote:

> We've seen issues with Ubuntu built MPI. Can you try --download-mpich and
> see if that helps?
>
> Also  you don't need --with-clanguage=cxx for complex. PETSc can be built
> with c99 compilex - i.e "--with-clanguage=c --with-scalar-type=complex"
> should work.
>
> Satish
>
> On Fri, 12 Aug 2022, David Knezevic via petsc-users wrote:
>
> > I get a configure error when building PETSc 3.17.4 on Ubuntu 22.04. The
> > configure.log is attached.
> >
> > This error seems to only occur when we use --with-clanguage=cxx. We need
> to
> > use that object when building with complex numbers.
> >
> > Guidance on how to resolve this would be most appreciated.
> >
> > Thanks,
> > David
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220812/7fc5f869/attachment.html>

From mfadams at lbl.gov  Fri Aug 12 14:42:23 2022
From: mfadams at lbl.gov (Mark Adams)
Date: Fri, 12 Aug 2022 15:42:23 -0400
Subject: [petsc-users] MatView to file
In-Reply-To: <CAKoAOWoDFOg32Pu=uSU8E6-1yGeqNLYiX9X09gX5TQFZ1bFdug@mail.gmail.com>
References: <CAKoAOWpqdnnC-A0bQOrCeYmtYOC-guNLBfXha7kpDX4CvNWWmw@mail.gmail.com>
	<CADOhEh5LBDieYhPiCH5U1z-wJGLkF6uNyFMxRoTN-H7AxzPWtA@mail.gmail.com>
	<CAKoAOWoDFOg32Pu=uSU8E6-1yGeqNLYiX9X09gX5TQFZ1bFdug@mail.gmail.com>
Message-ID: <CADOhEh6rqhPqhhu7bonH4Pr54tVM2OZHxgap+uo7Rf+qxtpChA@mail.gmail.com>

With 4 million elements you are nowhere near the 32 but integer limit of 2B
or 32Gb of memory.

See the https://petsc.org/main/docs/manualpages/Mat/MatView
You should go to binary format when doing large matrices.

Mark

On Fri, Aug 12, 2022 at 1:00 PM Alfredo Jaramillo <ajaramillopalma at gmail.com>
wrote:

> Hello Mark,
> Thank you, I added the lines that you sent.
> This only happens when running the code with more than 1 process. With
> only 1 MPI process the matrix is printed out.
> With 2 processes or more I observed the program begins to allocate RAM
> until it exceeds the computer capacity (32GB) so I wasn't able to get the
> stack trace.
>
> However, I was able to reproduce the problem by compiling
> src/ksp/ksp/tutorials/ex54.c.html
> <https://petsc.org/release/src/ksp/ksp/tutorials/ex54.c.html> (modifying
> line 144) and running it with
>
> mpirun -np 2 ex54 -ne 1000
>
> This gives a sparse matrix of order ~1 million. When running ex54 with
> only one MPI process I don't observe this excessive allocation and the
> matrix is printed out.
>
> Thanks,
> Alfredo
>
>
>
> On Fri, Aug 12, 2022 at 10:02 AM Mark Adams <mfadams at lbl.gov> wrote:
>
>> You also want:
>>
>>     PetscCall(PetscViewerPopFormat(viewer));
>>     PetscCall(PetscViewerDestroy(&viewer));
>>
>> This should not be a problem.
>> If this is a segv and you configure it with   '--with-debugging=1', you
>> should get a stack trace, which would help immensely.
>> Or run in a debugger to get a stack trace.
>>
>> Thanks,
>> Mark
>>
>>
>>
>> On Fri, Aug 12, 2022 at 11:26 AM Alfredo Jaramillo <
>> ajaramillopalma at gmail.com> wrote:
>>
>>> Dear developers,
>>>
>>> I'm writing a sparse matrix into a file by doing
>>>
>>>     if (dump_mat) {
>>>         PetscViewer viewer;
>>>         PetscViewerASCIIOpen(PETSC_COMM_WORLD,"mat-par-aux.m",&viewer);
>>>         PetscViewerPushFormat(viewer, PETSC_VIEWER_ASCII_MATLAB);
>>>         MatView(A,viewer);
>>>     }
>>>
>>> This works perfectly for small cases.
>>> The program crashes for a case where the matrix A is of order 1 million
>>> but with only 4 million non-zero elements.
>>>
>>> Maybe at some point petsc is full-sizing A?
>>>
>>> Thank you,
>>> Alfredo
>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220812/6bac7af2/attachment-0001.html>

From ajaramillopalma at gmail.com  Fri Aug 12 14:47:03 2022
From: ajaramillopalma at gmail.com (Alfredo Jaramillo)
Date: Fri, 12 Aug 2022 13:47:03 -0600
Subject: [petsc-users] MatView to file
In-Reply-To: <CADOhEh6rqhPqhhu7bonH4Pr54tVM2OZHxgap+uo7Rf+qxtpChA@mail.gmail.com>
References: <CAKoAOWpqdnnC-A0bQOrCeYmtYOC-guNLBfXha7kpDX4CvNWWmw@mail.gmail.com>
	<CADOhEh5LBDieYhPiCH5U1z-wJGLkF6uNyFMxRoTN-H7AxzPWtA@mail.gmail.com>
	<CAKoAOWoDFOg32Pu=uSU8E6-1yGeqNLYiX9X09gX5TQFZ1bFdug@mail.gmail.com>
	<CADOhEh6rqhPqhhu7bonH4Pr54tVM2OZHxgap+uo7Rf+qxtpChA@mail.gmail.com>
Message-ID: <CAKoAOWodj2xKeA32f9dpsYU7YP7sEBne+N+4YXCoHBzrHJL-tg@mail.gmail.com>

Hello Mark,
But why should this depend on the number of processes?
thanks
Alfredo

On Fri, Aug 12, 2022 at 1:42 PM Mark Adams <mfadams at lbl.gov> wrote:

> With 4 million elements you are nowhere near the 32 but integer limit of
> 2B or 32Gb of memory.
>
> See the https://petsc.org/main/docs/manualpages/Mat/MatView
> You should go to binary format when doing large matrices.
>
> Mark
>
> On Fri, Aug 12, 2022 at 1:00 PM Alfredo Jaramillo <
> ajaramillopalma at gmail.com> wrote:
>
>> Hello Mark,
>> Thank you, I added the lines that you sent.
>> This only happens when running the code with more than 1 process. With
>> only 1 MPI process the matrix is printed out.
>> With 2 processes or more I observed the program begins to allocate RAM
>> until it exceeds the computer capacity (32GB) so I wasn't able to get the
>> stack trace.
>>
>> However, I was able to reproduce the problem by compiling
>> src/ksp/ksp/tutorials/ex54.c.html
>> <https://petsc.org/release/src/ksp/ksp/tutorials/ex54.c.html> (modifying
>> line 144) and running it with
>>
>> mpirun -np 2 ex54 -ne 1000
>>
>> This gives a sparse matrix of order ~1 million. When running ex54 with
>> only one MPI process I don't observe this excessive allocation and the
>> matrix is printed out.
>>
>> Thanks,
>> Alfredo
>>
>>
>>
>> On Fri, Aug 12, 2022 at 10:02 AM Mark Adams <mfadams at lbl.gov> wrote:
>>
>>> You also want:
>>>
>>>     PetscCall(PetscViewerPopFormat(viewer));
>>>     PetscCall(PetscViewerDestroy(&viewer));
>>>
>>> This should not be a problem.
>>> If this is a segv and you configure it with   '--with-debugging=1', you
>>> should get a stack trace, which would help immensely.
>>> Or run in a debugger to get a stack trace.
>>>
>>> Thanks,
>>> Mark
>>>
>>>
>>>
>>> On Fri, Aug 12, 2022 at 11:26 AM Alfredo Jaramillo <
>>> ajaramillopalma at gmail.com> wrote:
>>>
>>>> Dear developers,
>>>>
>>>> I'm writing a sparse matrix into a file by doing
>>>>
>>>>     if (dump_mat) {
>>>>         PetscViewer viewer;
>>>>         PetscViewerASCIIOpen(PETSC_COMM_WORLD,"mat-par-aux.m",&viewer);
>>>>         PetscViewerPushFormat(viewer, PETSC_VIEWER_ASCII_MATLAB);
>>>>         MatView(A,viewer);
>>>>     }
>>>>
>>>> This works perfectly for small cases.
>>>> The program crashes for a case where the matrix A is of order 1 million
>>>> but with only 4 million non-zero elements.
>>>>
>>>> Maybe at some point petsc is full-sizing A?
>>>>
>>>> Thank you,
>>>> Alfredo
>>>>
>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220812/66d1a258/attachment.html>

From pierre at joliv.et  Fri Aug 12 14:50:47 2022
From: pierre at joliv.et (Pierre Jolivet)
Date: Fri, 12 Aug 2022 21:50:47 +0200
Subject: [petsc-users] MatView to file
In-Reply-To: <CAKoAOWodj2xKeA32f9dpsYU7YP7sEBne+N+4YXCoHBzrHJL-tg@mail.gmail.com>
References: <CAKoAOWpqdnnC-A0bQOrCeYmtYOC-guNLBfXha7kpDX4CvNWWmw@mail.gmail.com>
	<CADOhEh5LBDieYhPiCH5U1z-wJGLkF6uNyFMxRoTN-H7AxzPWtA@mail.gmail.com>
	<CAKoAOWoDFOg32Pu=uSU8E6-1yGeqNLYiX9X09gX5TQFZ1bFdug@mail.gmail.com>
	<CADOhEh6rqhPqhhu7bonH4Pr54tVM2OZHxgap+uo7Rf+qxtpChA@mail.gmail.com>
	<CAKoAOWodj2xKeA32f9dpsYU7YP7sEBne+N+4YXCoHBzrHJL-tg@mail.gmail.com>
Message-ID: <3279B874-9DEB-45DE-8C8D-3CBA585E311B@joliv.et>


> On 12 Aug 2022, at 9:47 PM, Alfredo Jaramillo <ajaramillopalma at gmail.com> wrote:
> 
> Hello Mark,
> But why should this depend on the number of processes?

Because with non-binary formats, the matrix is centralized on the first process, which can become very costly.

Thanks,
Pierre

> thanks
> Alfredo
> 
> On Fri, Aug 12, 2022 at 1:42 PM Mark Adams <mfadams at lbl.gov <mailto:mfadams at lbl.gov>> wrote:
> With 4 million elements you are nowhere near the 32 but integer limit of 2B or 32Gb of memory.
> 
> See the https://petsc.org/main/docs/manualpages/Mat/MatView <https://petsc.org/main/docs/manualpages/Mat/MatView>
> You should go to binary format when doing large matrices.
> 
> Mark
> 
> On Fri, Aug 12, 2022 at 1:00 PM Alfredo Jaramillo <ajaramillopalma at gmail.com <mailto:ajaramillopalma at gmail.com>> wrote:
> Hello Mark,
> Thank you, I added the lines that you sent.
> This only happens when running the code with more than 1 process. With only 1 MPI process the matrix is printed out.
> With 2 processes or more I observed the program begins to allocate RAM until it exceeds the computer capacity (32GB) so I wasn't able to get the stack trace.
> 
> However, I was able to reproduce the problem by compiling src/ksp/ksp/tutorials/ex54.c.html <https://petsc.org/release/src/ksp/ksp/tutorials/ex54.c.html> (modifying line 144) and running it with
> 
> mpirun -np 2 ex54 -ne 1000
> 
> This gives a sparse matrix of order ~1 million. When running ex54 with only one MPI process I don't observe this excessive allocation and the matrix is printed out.
> 
> Thanks,
> Alfredo
> 
> 
> 
> On Fri, Aug 12, 2022 at 10:02 AM Mark Adams <mfadams at lbl.gov <mailto:mfadams at lbl.gov>> wrote:
> You also want:
> 
>     PetscCall(PetscViewerPopFormat(viewer));
>     PetscCall(PetscViewerDestroy(&viewer));
> 
> This should not be a problem.
> If this is a segv and you configure it with   '--with-debugging=1', you should get a stack trace, which would help immensely.
> Or run in a debugger to get a stack trace.
> 
> Thanks,
> Mark
> 
> 
> On Fri, Aug 12, 2022 at 11:26 AM Alfredo Jaramillo <ajaramillopalma at gmail.com <mailto:ajaramillopalma at gmail.com>> wrote:
> Dear developers,
> 
> I'm writing a sparse matrix into a file by doing 
> 
>     if (dump_mat) {
>         PetscViewer viewer;
>         PetscViewerASCIIOpen(PETSC_COMM_WORLD,"mat-par-aux.m",&viewer);
>         PetscViewerPushFormat(viewer, PETSC_VIEWER_ASCII_MATLAB);
>         MatView(A,viewer);
>     }
> 
> This works perfectly for small cases.
> The program crashes for a case where the matrix A is of order 1 million but with only 4 million non-zero elements.
> 
> Maybe at some point petsc is full-sizing A?
> 
> Thank you,
> Alfredo

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220812/ad426fc3/attachment.html>

From mfadams at lbl.gov  Fri Aug 12 14:53:35 2022
From: mfadams at lbl.gov (Mark Adams)
Date: Fri, 12 Aug 2022 15:53:35 -0400
Subject: [petsc-users] MatView to file
In-Reply-To: <CAKoAOWodj2xKeA32f9dpsYU7YP7sEBne+N+4YXCoHBzrHJL-tg@mail.gmail.com>
References: <CAKoAOWpqdnnC-A0bQOrCeYmtYOC-guNLBfXha7kpDX4CvNWWmw@mail.gmail.com>
	<CADOhEh5LBDieYhPiCH5U1z-wJGLkF6uNyFMxRoTN-H7AxzPWtA@mail.gmail.com>
	<CAKoAOWoDFOg32Pu=uSU8E6-1yGeqNLYiX9X09gX5TQFZ1bFdug@mail.gmail.com>
	<CADOhEh6rqhPqhhu7bonH4Pr54tVM2OZHxgap+uo7Rf+qxtpChA@mail.gmail.com>
	<CAKoAOWodj2xKeA32f9dpsYU7YP7sEBne+N+4YXCoHBzrHJL-tg@mail.gmail.com>
Message-ID: <CADOhEh6_e2H2mdk8mu7NVDs7c_sdPjjHzSXzs7RvEvg9VjitMw@mail.gmail.com>

So your build/machine works with ex54.
That is a nine-point stencil so about 2x larger than your problem, but your
problem fails.
Is that correct?

On Fri, Aug 12, 2022 at 3:47 PM Alfredo Jaramillo <ajaramillopalma at gmail.com>
wrote:

> Hello Mark,
> But why should this depend on the number of processes?
> thanks
> Alfredo
>
> On Fri, Aug 12, 2022 at 1:42 PM Mark Adams <mfadams at lbl.gov> wrote:
>
>> With 4 million elements you are nowhere near the 32 but integer limit of
>> 2B or 32Gb of memory.
>>
>> See the https://petsc.org/main/docs/manualpages/Mat/MatView
>> You should go to binary format when doing large matrices.
>>
>> Mark
>>
>> On Fri, Aug 12, 2022 at 1:00 PM Alfredo Jaramillo <
>> ajaramillopalma at gmail.com> wrote:
>>
>>> Hello Mark,
>>> Thank you, I added the lines that you sent.
>>> This only happens when running the code with more than 1 process. With
>>> only 1 MPI process the matrix is printed out.
>>> With 2 processes or more I observed the program begins to allocate RAM
>>> until it exceeds the computer capacity (32GB) so I wasn't able to get the
>>> stack trace.
>>>
>>> However, I was able to reproduce the problem by compiling
>>> src/ksp/ksp/tutorials/ex54.c.html
>>> <https://petsc.org/release/src/ksp/ksp/tutorials/ex54.c.html>
>>> (modifying line 144) and running it with
>>>
>>> mpirun -np 2 ex54 -ne 1000
>>>
>>> This gives a sparse matrix of order ~1 million. When running ex54 with
>>> only one MPI process I don't observe this excessive allocation and the
>>> matrix is printed out.
>>>
>>> Thanks,
>>> Alfredo
>>>
>>>
>>>
>>> On Fri, Aug 12, 2022 at 10:02 AM Mark Adams <mfadams at lbl.gov> wrote:
>>>
>>>> You also want:
>>>>
>>>>     PetscCall(PetscViewerPopFormat(viewer));
>>>>     PetscCall(PetscViewerDestroy(&viewer));
>>>>
>>>> This should not be a problem.
>>>> If this is a segv and you configure it with   '--with-debugging=1', you
>>>> should get a stack trace, which would help immensely.
>>>> Or run in a debugger to get a stack trace.
>>>>
>>>> Thanks,
>>>> Mark
>>>>
>>>>
>>>>
>>>> On Fri, Aug 12, 2022 at 11:26 AM Alfredo Jaramillo <
>>>> ajaramillopalma at gmail.com> wrote:
>>>>
>>>>> Dear developers,
>>>>>
>>>>> I'm writing a sparse matrix into a file by doing
>>>>>
>>>>>     if (dump_mat) {
>>>>>         PetscViewer viewer;
>>>>>         PetscViewerASCIIOpen(PETSC_COMM_WORLD,"mat-par-aux.m",&viewer);
>>>>>         PetscViewerPushFormat(viewer, PETSC_VIEWER_ASCII_MATLAB);
>>>>>         MatView(A,viewer);
>>>>>     }
>>>>>
>>>>> This works perfectly for small cases.
>>>>> The program crashes for a case where the matrix A is of order 1
>>>>> million but with only 4 million non-zero elements.
>>>>>
>>>>> Maybe at some point petsc is full-sizing A?
>>>>>
>>>>> Thank you,
>>>>> Alfredo
>>>>>
>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220812/aa9b31b1/attachment-0001.html>

From bsmith at petsc.dev  Fri Aug 12 16:55:31 2022
From: bsmith at petsc.dev (Barry Smith)
Date: Fri, 12 Aug 2022 17:55:31 -0400
Subject: [petsc-users] DMDA ghosted boundary
In-Reply-To: <b7f9ec88-a7df-6eb8-ebda-e96e1223e9a6@epcc.ed.ac.uk>
References: <CAFma+BnGu-cLjLpWn4H07vHzi-_8jAWPYf0W+VzXOa3OD-c4CQ@mail.gmail.com>
	<CAMYG4GkAywr933zE2XtMWd=0DqbjxH7BtmixijF1mp1AnBx-Pg@mail.gmail.com>
	<b7f9ec88-a7df-6eb8-ebda-e96e1223e9a6@epcc.ed.ac.uk>
Message-ID: <DA3FE1B8-5AB8-44CD-8B4A-4F70CF364AB5@petsc.dev>



> On Aug 12, 2022, at 9:27 AM, David Scott <d.scott at epcc.ed.ac.uk> wrote:
> 
> 
> 
> On 12/08/2022 11:26, Matthew Knepley wrote:
>> This email was sent to you by someone outside the University.
>> You should only click on links or attachments if you are certain that the email is genuine and the content is safe.
>> On Fri, Aug 12, 2022 at 2:21 AM Quynh Nguyen <nguyenquynh9537 at gmail.com <mailto:nguyenquynh9537 at gmail.com>> wrote:
>> Dear petsc developers and users,
>> 
>> I am a new petsc user ^_^.
>> I have some questions about using DMDA_BOUNDARY_GHOSTED.
>> 
>> 1. I have 2D Poisson code with domain (nx, ny), and I used  DMDA_BOUNDARY_NONE, then the size of the matrix is (nx*ny)X(nx*ny). what is the size of the matrix if I used  DMDA_BOUNDARY_GHOSTED? is it the same size as DMDA_BOUNDARY_NONE?
>> 
>> Yes, the matrix should be the same size, but local vectors have extra ghost node slots.
>>  
>> 2. if the matrix size of DMDA_BOUNDARY_GHOSTED isn't the same as DMDA_BOUNDARY_NONE, how i can view the matrix including ghost points? I ask this one because I used MatView(P, PETSC_VIEWER_STDOUT_WORLD) to show the matrix but excluded ghost points.
>> 
>> Ghost nodes are there so that stencil operations on local vectors work at boundaries. You fill in the slots with boundary information.
>> 
> I can see how this works if you are writing your own solver but can it be used with a KSP? In the case of periodic boundary conditions the boundary values are "filled in" for you by PETSc. Is there any way for a user to supply boundary information to a PETSc solver when using DMDA_BOUNDARY_GHOSTED?

 It is up to you to determine what values to put into those locations and put them in before you apply your "stencil" computation to evaluate the residual. You just stick values in those locations, likely after doing a DMDAVecGetArray().


> 
> Thanks,
> 
> David
> 
>>   Thanks,
>> 
>>      Matt
>>  
>> I hope someone can explain my wonder :)
>> 
>> P/S: maybe, those question is simple and a bit stupid ^_^. 
>> 
>> Have a nice day,
>> 
>> -- 
>> Quynh Nguyen
>> Graduate student, Ocean Engineering.
>> School of Naval Architecture and Ocean Engineering, University of Ulsan.
>> -----------------------------------------------------------------------------------------
>> Mobile: (+82) 010-9394-5595 
>> Email: nguyenquynh9537 at gmail.com <mailto:nguyenquynh9537 at gmail.com>; nguyenthiquynhb_t58 at hus.edu.vn <mailto:nguyenthiquynhb_t58 at hus.edu.vn>
>> 
>> -- 
>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>> -- Norbert Wiener
>> 
>> https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
> 
> The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Is e buidheann carthannais a th? ann an Oilthigh Dh?n ?ideann, cl?raichte an Alba, ?ireamh cl?raidh SC005336.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220812/0fc26f14/attachment.html>

From bsmith at petsc.dev  Fri Aug 12 17:13:49 2022
From: bsmith at petsc.dev (Barry Smith)
Date: Fri, 12 Aug 2022 18:13:49 -0400
Subject: [petsc-users] MatView to file
In-Reply-To: <3279B874-9DEB-45DE-8C8D-3CBA585E311B@joliv.et>
References: <CAKoAOWpqdnnC-A0bQOrCeYmtYOC-guNLBfXha7kpDX4CvNWWmw@mail.gmail.com>
	<CADOhEh5LBDieYhPiCH5U1z-wJGLkF6uNyFMxRoTN-H7AxzPWtA@mail.gmail.com>
	<CAKoAOWoDFOg32Pu=uSU8E6-1yGeqNLYiX9X09gX5TQFZ1bFdug@mail.gmail.com>
	<CADOhEh6rqhPqhhu7bonH4Pr54tVM2OZHxgap+uo7Rf+qxtpChA@mail.gmail.com>
	<CAKoAOWodj2xKeA32f9dpsYU7YP7sEBne+N+4YXCoHBzrHJL-tg@mail.gmail.com>
	<3279B874-9DEB-45DE-8C8D-3CBA585E311B@joliv.et>
Message-ID: <B5360454-FBEE-4350-B67D-C1A45F3289BB@petsc.dev>



> On Aug 12, 2022, at 3:50 PM, Pierre Jolivet <pierre at joliv.et> wrote:
> 
> 
>> On 12 Aug 2022, at 9:47 PM, Alfredo Jaramillo <ajaramillopalma at gmail.com <mailto:ajaramillopalma at gmail.com>> wrote:
>> 
>> Hello Mark,
>> But why should this depend on the number of processes?
> 
> Because with non-binary formats, the matrix is centralized on the first process, which can become very costly.

! On. We expect one to use a binary format for anything but trivially sized matrices, we don't consider ASCII a reasonable format for large matrices.


> 
> Thanks,
> Pierre
> 
>> thanks
>> Alfredo
>> 
>> On Fri, Aug 12, 2022 at 1:42 PM Mark Adams <mfadams at lbl.gov <mailto:mfadams at lbl.gov>> wrote:
>> With 4 million elements you are nowhere near the 32 but integer limit of 2B or 32Gb of memory.
>> 
>> See the https://petsc.org/main/docs/manualpages/Mat/MatView <https://petsc.org/main/docs/manualpages/Mat/MatView>
>> You should go to binary format when doing large matrices.
>> 
>> Mark
>> 
>> On Fri, Aug 12, 2022 at 1:00 PM Alfredo Jaramillo <ajaramillopalma at gmail.com <mailto:ajaramillopalma at gmail.com>> wrote:
>> Hello Mark,
>> Thank you, I added the lines that you sent.
>> This only happens when running the code with more than 1 process. With only 1 MPI process the matrix is printed out.
>> With 2 processes or more I observed the program begins to allocate RAM until it exceeds the computer capacity (32GB) so I wasn't able to get the stack trace.
>> 
>> However, I was able to reproduce the problem by compiling src/ksp/ksp/tutorials/ex54.c.html <https://petsc.org/release/src/ksp/ksp/tutorials/ex54.c.html> (modifying line 144) and running it with
>> 
>> mpirun -np 2 ex54 -ne 1000
>> 
>> This gives a sparse matrix of order ~1 million. When running ex54 with only one MPI process I don't observe this excessive allocation and the matrix is printed out.
>> 
>> Thanks,
>> Alfredo
>> 
>> 
>> 
>> On Fri, Aug 12, 2022 at 10:02 AM Mark Adams <mfadams at lbl.gov <mailto:mfadams at lbl.gov>> wrote:
>> You also want:
>> 
>>     PetscCall(PetscViewerPopFormat(viewer));
>>     PetscCall(PetscViewerDestroy(&viewer));
>> 
>> This should not be a problem.
>> If this is a segv and you configure it with   '--with-debugging=1', you should get a stack trace, which would help immensely.
>> Or run in a debugger to get a stack trace.
>> 
>> Thanks,
>> Mark
>> 
>> 
>> On Fri, Aug 12, 2022 at 11:26 AM Alfredo Jaramillo <ajaramillopalma at gmail.com <mailto:ajaramillopalma at gmail.com>> wrote:
>> Dear developers,
>> 
>> I'm writing a sparse matrix into a file by doing 
>> 
>>     if (dump_mat) {
>>         PetscViewer viewer;
>>         PetscViewerASCIIOpen(PETSC_COMM_WORLD,"mat-par-aux.m",&viewer);
>>         PetscViewerPushFormat(viewer, PETSC_VIEWER_ASCII_MATLAB);
>>         MatView(A,viewer);
>>     }
>> 
>> This works perfectly for small cases.
>> The program crashes for a case where the matrix A is of order 1 million but with only 4 million non-zero elements.
>> 
>> Maybe at some point petsc is full-sizing A?
>> 
>> Thank you,
>> Alfredo
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220812/eac5949b/attachment-0001.html>

From patrick.alken at geomag.info  Sun Aug 14 23:12:20 2022
From: patrick.alken at geomag.info (Patrick Alken)
Date: Sun, 14 Aug 2022 22:12:20 -0600
Subject: [petsc-users] Parallel assembly of a matrix problem
Message-ID: <e8c23b6d-7c55-5a23-91c4-99cbb8356cc1@geomag.info>

I am trying to assemble a matrix in parallel in Petsc, and it is working 
for 1 processor, but when I try running it with 2 processors I get 
errors like the following:

----

[1]PETSC ERROR: --------------------- Error Message 
--------------------------------------------------------------
[1]PETSC ERROR: Argument out of range
[1]PETSC ERROR: New nonzero at (0,207) caused a malloc
Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to turn 
off this check
[1]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[1]PETSC ERROR: Petsc Release Version 3.17.4, unknown
[1]PETSC ERROR: #1 MatSetValues_MPIAIJ() at 
/data/palken/petsc/src/mat/impls/aij/mpi/mpiaij.c:506

----

and then many MatSetValues errors follow.

I have attached a minimal working example and a Makefile. My code does 
depend on the GSL library also. I think there must be a problem in how I 
set up the d_nnz and o_nnz arrays for the MatMPIAIJSetPreallocation() 
function (lines 211-235 of mwe.c).

I am building a matrix using radial basis functions with compact 
support, so the matrix entries are zero when ||Xi - Xj|| >= c, where c 
is the RBF shape parameter. This is accounted for on line 224 and 254 of 
the code. As far as I can tell I am consistent in the way I define the 
nnz arrays and the way I call MatSetValues, so I don't know why Petsc 
would need to malloc additional memory during the assembly process.

Any pointers would be greatly appreciated.

Patrick
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mwe.c
Type: text/x-csrc
Size: 6728 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220814/d929d5ef/attachment.bin>
-------------- next part --------------
default: mwe

include ${SLEPC_DIR}/lib/slepc/conf/slepc_common

SLEPC_INCLUDE += -I..

lib_dir = /home/palken/usr/lib
common_libs = -L${lib_dir} ${lib_dir}/libgsl.a -lm -lgslcblas ${lib_dir}/liblapacke.a ${lib_dir}/liblapack.a -lcsparse

OBJECTS = mwe.o

mwe: ${OBJECTS}
	${CLINKER} -o mwe ${OBJECTS} ${SLEPC_EPS_LIB} ${common_libs}

From d.scott at epcc.ed.ac.uk  Mon Aug 15 03:02:35 2022
From: d.scott at epcc.ed.ac.uk (David Scott)
Date: Mon, 15 Aug 2022 09:02:35 +0100
Subject: [petsc-users] DMDA ghosted boundary
In-Reply-To: <CAMYG4GmzzvfqPuXRBE9n1ZFZ5sCU=Ao95KNNPtPCvD4O7_4ZrQ@mail.gmail.com>
References: <CAFma+BnGu-cLjLpWn4H07vHzi-_8jAWPYf0W+VzXOa3OD-c4CQ@mail.gmail.com>
	<CAMYG4GkAywr933zE2XtMWd=0DqbjxH7BtmixijF1mp1AnBx-Pg@mail.gmail.com>
	<b7f9ec88-a7df-6eb8-ebda-e96e1223e9a6@epcc.ed.ac.uk>
	<CAMYG4GmzzvfqPuXRBE9n1ZFZ5sCU=Ao95KNNPtPCvD4O7_4ZrQ@mail.gmail.com>
Message-ID: <19e1feb8-e6b9-98aa-7858-2cedb53397f0@epcc.ed.ac.uk>



On 12/08/2022 18:15, Matthew Knepley wrote:
> This email was sent to you by someone outside the University.
> You should only click on links or attachments if you are certain that 
> the email is genuine and the content is safe.
> On Fri, Aug 12, 2022 at 9:27 AM David Scott <d.scott at epcc.ed.ac.uk> wrote:
>
>     On 12/08/2022 11:26, Matthew Knepley wrote:
>>     This email was sent to you by someone outside the University.
>>     You should only click on links or attachments if you are certain
>>     that the email is genuine and the content is safe.
>>     On Fri, Aug 12, 2022 at 2:21 AM Quynh Nguyen
>>     <nguyenquynh9537 at gmail.com> wrote:
>>
>>         Dear petsc developers and users,
>>
>>         I am a new petsc user ^_^.
>>         I have some questions about using DMDA_BOUNDARY_GHOSTED.
>>
>>         1. I have 2D Poisson code with domain (nx, ny), and I used?
>>         DMDA_BOUNDARY_NONE, then the size of the matrix is
>>         (nx*ny)X(nx*ny). */what is the size of the matrix if I used
>>         DMDA_BOUNDARY_GHOSTED? is it?the?same size as
>>         DMDA_BOUNDARY_NONE/*?
>>
>>
>>     Yes, the matrix should be the same size, but local vectors have
>>     extra ghost node slots.
>>
>>         2. if the matrix size of DMDA_BOUNDARY_GHOSTED isn't the same
>>         as DMDA_BOUNDARY_NONE, */how i can view the matrix
>>         including?ghost points/*? I ask this one because?I
>>         used?MatView(P, PETSC_VIEWER_STDOUT_WORLD) to show the matrix
>>         but excluded?ghost points.
>>
>>
>>     Ghost nodes are there so that stencil operations on local vectors
>>     work at boundaries. You fill in the slots with boundary information.
>>
>     I can see how this works if you are writing your own solver but
>     can it be used with a KSP? In the case of periodic boundary
>     conditions the boundary values are "filled in" for you by PETSc.
>     Is there any way for a user to supply boundary information to a
>     PETSc solver when using DMDA_BOUNDARY_GHOSTED?
>
>
> Right now, there is not. This is mainly due to the lack of consensus 
> for boundary condition specification. Periodic conditions are simple 
> in this respect.
>
> For example, Plex has such a thing
>
> https://petsc.org/main/docs/manualpages/DM/DMAddBoundary/
>
> that can be used to fill in these constrained spots
>
> https://petsc.org/main/docs/manualpages/DMPLEX/DMPlexInsertBoundaryValues/
>
> But there have been many complaints about this interface (and I 
> suspect any interface we choose).
>
> Maybe there is something that we could write that would make it easier 
> for you with DMDA?
>
The way I had imagined it working was that for each dimension the user 
could register a function to be to be called at the start of every 
iteration (in a similar fashion to adding a monitor). Not all of the 
dimensions would have to be specified as DMDA_BOUNDARY_GHOSTED and thus 
require a function to be registered. This function would have access to 
the DMDA and the local vector so it could determine if ghost locations 
were present and, if so, set appropriate values. Does this make sense?

Thanks,

David

> ? Thanks,
>
> ? ? ?Matt
>
>     Thanks,
>
>     David
>
>>     ? Thanks,
>>
>>     ? ? ?Matt
>>
>>         I hope someone?can explain my wonder :)
>>
>>         P/S: maybe, those question is simple?and a bit stupid ^_^.
>>
>>         Have a nice day,
>>
>>         -- 
>>         Quynh Nguyen
>>         Graduate student, Ocean Engineering.
>>         School of Naval Architecture and Ocean Engineering,
>>         University of Ulsan.
>>         -----------------------------------------------------------------------------------------
>>         Mobile: (+82) 010-9394-5595
>>         Email: nguyenquynh9537 at gmail.com; nguyenthiquynhb_t58 at hus.edu.vn
>>
>>
>>
>>     -- 
>>     What most experimenters take for granted before they begin their
>>     experiments is infinitely more interesting than any results to
>>     which their experiments lead.
>>     -- Norbert Wiener
>>
>>     https://www.cse.buffalo.edu/~knepley/
>>     <http://www.cse.buffalo.edu/~knepley/>
>
>     The University of Edinburgh is a charitable body, registered in
>     Scotland, with registration number SC005336. Is e buidheann
>     carthannais a th? ann an Oilthigh Dh?n ?ideann, cl?raichte an
>     Alba, ?ireamh cl?raidh SC005336.
>
>
>
> -- 
> What most experimenters take for granted before they begin their 
> experiments is infinitely more interesting than any results to which 
> their experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/ 
> <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220815/dff4afa2/attachment-0001.html>

From mfadams at lbl.gov  Mon Aug 15 05:29:10 2022
From: mfadams at lbl.gov (Mark Adams)
Date: Mon, 15 Aug 2022 06:29:10 -0400
Subject: [petsc-users] Parallel assembly of a matrix problem
In-Reply-To: <e8c23b6d-7c55-5a23-91c4-99cbb8356cc1@geomag.info>
References: <e8c23b6d-7c55-5a23-91c4-99cbb8356cc1@geomag.info>
Message-ID: <CADOhEh5PQ=G+xkkP9k11sHU7QQYWJhgMXkT4zvXM7u5-Oo7UKQ@mail.gmail.com>

You might need to fix this:

if (i - j < m)


to


if (i - j < m && i - j >= 0)


Mark


On Mon, Aug 15, 2022 at 12:12 AM Patrick Alken <patrick.alken at geomag.info>
wrote:

> I am trying to assemble a matrix in parallel in Petsc, and it is working
> for 1 processor, but when I try running it with 2 processors I get
> errors like the following:
>
> ----
>
> [1]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [1]PETSC ERROR: Argument out of range
> [1]PETSC ERROR: New nonzero at (0,207) caused a malloc
> Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to turn
> off this check
> [1]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
> [1]PETSC ERROR: Petsc Release Version 3.17.4, unknown
> [1]PETSC ERROR: #1 MatSetValues_MPIAIJ() at
> /data/palken/petsc/src/mat/impls/aij/mpi/mpiaij.c:506
>
> ----
>
> and then many MatSetValues errors follow.
>
> I have attached a minimal working example and a Makefile. My code does
> depend on the GSL library also. I think there must be a problem in how I
> set up the d_nnz and o_nnz arrays for the MatMPIAIJSetPreallocation()
> function (lines 211-235 of mwe.c).
>
> I am building a matrix using radial basis functions with compact
> support, so the matrix entries are zero when ||Xi - Xj|| >= c, where c
> is the RBF shape parameter. This is accounted for on line 224 and 254 of
> the code. As far as I can tell I am consistent in the way I define the
> nnz arrays and the way I call MatSetValues, so I don't know why Petsc
> would need to malloc additional memory during the assembly process.
>
> Any pointers would be greatly appreciated.
>
> Patrick
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220815/88c4d8b5/attachment.html>

From bsmith at petsc.dev  Mon Aug 15 08:21:32 2022
From: bsmith at petsc.dev (Barry Smith)
Date: Mon, 15 Aug 2022 09:21:32 -0400
Subject: [petsc-users] DMDA ghosted boundary
In-Reply-To: <19e1feb8-e6b9-98aa-7858-2cedb53397f0@epcc.ed.ac.uk>
References: <CAFma+BnGu-cLjLpWn4H07vHzi-_8jAWPYf0W+VzXOa3OD-c4CQ@mail.gmail.com>
	<CAMYG4GkAywr933zE2XtMWd=0DqbjxH7BtmixijF1mp1AnBx-Pg@mail.gmail.com>
	<b7f9ec88-a7df-6eb8-ebda-e96e1223e9a6@epcc.ed.ac.uk>
	<CAMYG4GmzzvfqPuXRBE9n1ZFZ5sCU=Ao95KNNPtPCvD4O7_4ZrQ@mail.gmail.com>
	<19e1feb8-e6b9-98aa-7858-2cedb53397f0@epcc.ed.ac.uk>
Message-ID: <004989A2-A6DE-4492-8BCA-79BA8F4E93A9@petsc.dev>


  Something like 

  DMDASetComputeGhostBoundary(DM dm, s(DM,Vec,Vec,void*),d(void*))

where 

s(DM dm, Vec g, Vec l, void*ctx) {
  User code that fills in appropriate places in l. 

d(void* ctx) {
Optional: Destroys optional user context when DM is destroyed.
}

and s() gets called inside of DMGlobalToLocalEnd() just before it returns? 

Yes, this seems doable. But I note that you can simply perform your s() function on the local vector immediately after you call DMGlobalToLocalEnd() and get the same affect.

Barry


> On Aug 15, 2022, at 4:02 AM, David Scott <d.scott at epcc.ed.ac.uk> wrote:
> 
> 
> 
> On 12/08/2022 18:15, Matthew Knepley wrote:
>> This email was sent to you by someone outside the University.
>> You should only click on links or attachments if you are certain that the email is genuine and the content is safe.
>> On Fri, Aug 12, 2022 at 9:27 AM David Scott <d.scott at epcc.ed.ac.uk <mailto:d.scott at epcc.ed.ac.uk>> wrote:
>> On 12/08/2022 11:26, Matthew Knepley wrote:
>>> This email was sent to you by someone outside the University.
>>> You should only click on links or attachments if you are certain that the email is genuine and the content is safe.
>>> On Fri, Aug 12, 2022 at 2:21 AM Quynh Nguyen <nguyenquynh9537 at gmail.com <mailto:nguyenquynh9537 at gmail.com>> wrote:
>>> Dear petsc developers and users,
>>> 
>>> I am a new petsc user ^_^.
>>> I have some questions about using DMDA_BOUNDARY_GHOSTED.
>>> 
>>> 1. I have 2D Poisson code with domain (nx, ny), and I used  DMDA_BOUNDARY_NONE, then the size of the matrix is (nx*ny)X(nx*ny). what is the size of the matrix if I used  DMDA_BOUNDARY_GHOSTED? is it the same size as DMDA_BOUNDARY_NONE?
>>> 
>>> Yes, the matrix should be the same size, but local vectors have extra ghost node slots.
>>>  
>>> 2. if the matrix size of DMDA_BOUNDARY_GHOSTED isn't the same as DMDA_BOUNDARY_NONE, how i can view the matrix including ghost points? I ask this one because I used MatView(P, PETSC_VIEWER_STDOUT_WORLD) to show the matrix but excluded ghost points.
>>> 
>>> Ghost nodes are there so that stencil operations on local vectors work at boundaries. You fill in the slots with boundary information.
>>> 
>> I can see how this works if you are writing your own solver but can it be used with a KSP? In the case of periodic boundary conditions the boundary values are "filled in" for you by PETSc. Is there any way for a user to supply boundary information to a PETSc solver when using DMDA_BOUNDARY_GHOSTED?
>> 
>> Right now, there is not. This is mainly due to the lack of consensus for boundary condition specification. Periodic conditions are simple in this respect.
>> 
>> For example, Plex has such a thing
>> 
>>   https://petsc.org/main/docs/manualpages/DM/DMAddBoundary/ <https://petsc.org/main/docs/manualpages/DM/DMAddBoundary/>
>> 
>> that can be used to fill in these constrained spots
>> 
>>   https://petsc.org/main/docs/manualpages/DMPLEX/DMPlexInsertBoundaryValues/ <https://petsc.org/main/docs/manualpages/DMPLEX/DMPlexInsertBoundaryValues/>
>> 
>> But there have been many complaints about this interface (and I suspect any interface we choose).
>> 
>> Maybe there is something that we could write that would make it easier for you with DMDA?
>> 
> The way I had imagined it working was that for each dimension the user could register a function to be to be called at the start of every iteration (in a similar fashion to adding a monitor). Not all of the dimensions would have to be specified as DMDA_BOUNDARY_GHOSTED and thus require a function to be registered. This function would have access to the DMDA and the local vector so it could determine if ghost locations were present and, if so, set appropriate values. Does this make sense?
> 
> Thanks,
> 
> David
> 
>>   Thanks,
>> 
>>      Matt
>>  
>> Thanks,
>> 
>> David
>> 
>>>   Thanks,
>>> 
>>>      Matt
>>>  
>>> I hope someone can explain my wonder :)
>>> 
>>> P/S: maybe, those question is simple and a bit stupid ^_^. 
>>> 
>>> Have a nice day,
>>> 
>>> -- 
>>> Quynh Nguyen
>>> Graduate student, Ocean Engineering.
>>> School of Naval Architecture and Ocean Engineering, University of Ulsan.
>>> -----------------------------------------------------------------------------------------
>>> Mobile: (+82) 010-9394-5595 
>>> Email: nguyenquynh9537 at gmail.com <mailto:nguyenquynh9537 at gmail.com>; nguyenthiquynhb_t58 at hus.edu.vn <mailto:nguyenthiquynhb_t58 at hus.edu.vn>
>>> 
>>> -- 
>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>> -- Norbert Wiener
>>> 
>>> https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
>> 
>> The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Is e buidheann carthannais a th? ann an Oilthigh Dh?n ?ideann, cl?raichte an Alba, ?ireamh cl?raidh SC005336.
>> 
>> 
>> -- 
>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>> -- Norbert Wiener
>> 
>> https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220815/2863555e/attachment-0001.html>

From bsmith at petsc.dev  Mon Aug 15 08:28:55 2022
From: bsmith at petsc.dev (Barry Smith)
Date: Mon, 15 Aug 2022 09:28:55 -0400
Subject: [petsc-users] Parallel assembly of a matrix problem
In-Reply-To: <e8c23b6d-7c55-5a23-91c4-99cbb8356cc1@geomag.info>
References: <e8c23b6d-7c55-5a23-91c4-99cbb8356cc1@geomag.info>
Message-ID: <7FE6C0E9-CEB5-4ACD-B71B-6DD3A6FBCE0F@petsc.dev>


    There are two sets of tools for helping compute the preallocation in a "pre-compute" step that "count" what you will need later efficiently so that you don't need to "figure out" the preallocation information yourself. They are both slightly clumsy, unfortunately, but should work fine.

 https://petsc.org/main/docs/manualpages/Mat/MATPREALLOCATOR/?highlight=matpreallocator <https://petsc.org/main/docs/manualpages/Mat/MATPREALLOCATOR/?highlight=matpreallocator>

https://petsc.org/main/docs/manualpages/Mat/MatPreallocateBegin/?highlight=matpreallocatebegin <https://petsc.org/main/docs/manualpages/Mat/MatPreallocateBegin/?highlight=matpreallocatebegin> 

> On Aug 15, 2022, at 12:12 AM, Patrick Alken <patrick.alken at geomag.info> wrote:
> 
> I am trying to assemble a matrix in parallel in Petsc, and it is working for 1 processor, but when I try running it with 2 processors I get errors like the following:
> 
> ----
> 
> [1]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [1]PETSC ERROR: Argument out of range
> [1]PETSC ERROR: New nonzero at (0,207) caused a malloc
> Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to turn off this check
> [1]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
> [1]PETSC ERROR: Petsc Release Version 3.17.4, unknown
> [1]PETSC ERROR: #1 MatSetValues_MPIAIJ() at /data/palken/petsc/src/mat/impls/aij/mpi/mpiaij.c:506
> 
> ----
> 
> and then many MatSetValues errors follow.
> 
> I have attached a minimal working example and a Makefile. My code does depend on the GSL library also. I think there must be a problem in how I set up the d_nnz and o_nnz arrays for the MatMPIAIJSetPreallocation() function (lines 211-235 of mwe.c).
> 
> I am building a matrix using radial basis functions with compact support, so the matrix entries are zero when ||Xi - Xj|| >= c, where c is the RBF shape parameter. This is accounted for on line 224 and 254 of the code. As far as I can tell I am consistent in the way I define the nnz arrays and the way I call MatSetValues, so I don't know why Petsc would need to malloc additional memory during the assembly process.
> 
> Any pointers would be greatly appreciated.
> 
> Patrick
> <mwe.c><Makefile.txt>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220815/0fe474e9/attachment.html>

From lei.xu at siat.ac.cn  Sun Aug 14 22:30:52 2022
From: lei.xu at siat.ac.cn (lei.xu at siat.ac.cn)
Date: Mon, 15 Aug 2022 11:30:52 +0800
Subject: [petsc-users] Solution vector in a component-wise order
Message-ID: <202208151130516427426@siat.ac.cn>

Dear all,

I am trying to compute a matrix for KSP based on DMDACreate2d. Can I organize the solution vector in a component-wise (field-splitting) order instead of point-wise (field-coupling) order.

Best wishes,

Lei



Lei Xu,   PhD
Assistant Professor

Laboratory for Engineering and Scientific Computing
Shenzhen Institutes of Advanced Technology
Chinese Academy of Sciences
Address: 1068 Xueyuan Avenue, Shenzhen University Town, Shenzhen, Guangdong (518055), P. R. China
E-mail:  lei.xu at siat.ac.cn
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220815/b3e0c7c1/attachment.html>

From bsmith at petsc.dev  Mon Aug 15 10:14:49 2022
From: bsmith at petsc.dev (Barry Smith)
Date: Mon, 15 Aug 2022 11:14:49 -0400
Subject: [petsc-users] Solution vector in a component-wise order
In-Reply-To: <202208151130516427426@siat.ac.cn>
References: <202208151130516427426@siat.ac.cn>
Message-ID: <AE191971-1002-4D68-AFD6-5C9AC5FE2114@petsc.dev>


  Not really. There are tools to make it easier to do computations on individual fields in the field-coupling order such as https://petsc.org/release/docs/manualpages/Vec/VecStrideGatherAll.html <https://petsc.org/release/docs/manualpages/Vec/VecStrideGatherAll.html> and its many friends and relations. 

  What task or set of computations motivates you to want to use a field-split storage format as the base format instead of the field-coupling?

  Barry


> On Aug 14, 2022, at 11:30 PM, lei.xu at siat.ac.cn wrote:
> 
> Dear all,
> 
> I am trying to compute a matrix for KSP based on DMDACreate2d. Can I organize the solution vector in a component-wise (field-splitting) order instead of point-wise (field-coupling) order.
> 
> Best wishes,
> 
> Lei
> 
> Lei Xu,   PhD
> Assistant Professor
> 
> Laboratory for Engineering and Scientific Computing
> Shenzhen Institutes of Advanced Technology
> Chinese Academy of Sciences
> Address: 1068 Xueyuan Avenue, Shenzhen University Town, Shenzhen, Guangdong (518055), P. R. China
> E-mail:  lei.xu at siat.ac.cn <mailto:lei.xu at siat.ac.cn>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220815/95c0b2d6/attachment.html>

From Jiannan_Tu at uml.edu  Tue Aug 16 08:40:07 2022
From: Jiannan_Tu at uml.edu (Tu, Jiannan)
Date: Tue, 16 Aug 2022 13:40:07 +0000
Subject: [petsc-users] Using matrix-free with KSP
In-Reply-To: <BY5PR02MB6471E386057DDDE98AD05DFAE7809@BY5PR02MB6471.namprd02.prod.outlook.com>
References: <BY5PR02MB6471969F80FA5F732FC74D87E7D79@BY5PR02MB6471.namprd02.prod.outlook.com>
	<B55CFC2C-6364-4B78-8E99-9D53700743B8@petsc.dev>
	<BY5PR02MB6471BDF90783FE3A0EA176CBE7809@BY5PR02MB6471.namprd02.prod.outlook.com>
	<87iloa49kt.fsf@jedbrown.org>
	<BY5PR02MB6471C0ED04769D02046F68ACE7809@BY5PR02MB6471.namprd02.prod.outlook.com>
	<00EBE4B8-22E5-408F-B56D-14226A278D7E@petsc.dev>
	<BY5PR02MB6471E386057DDDE98AD05DFAE7809@BY5PR02MB6471.namprd02.prod.outlook.com>
Message-ID: <BY5PR02MB647140084C7E7CDDB6BC2A11E76B9@BY5PR02MB6471.namprd02.prod.outlook.com>

I am trying to apply a user-defined preconditioner to speed up the convergency of matrix-free KSP solver. The user's manual provides an example of how to call the user-defined preconditioner routine. The routine has PC, input Vec and output Vec as arguments. I want to use part of the Jacobian as preconditioner matrix, the elements of which can be calculated using input Vec. My question is what the role of output Vec from the routine is in matrix-vector product? Or I just have the preconditioner matrix elements calculated in the routine and then Petsc handles applying the preconditioner to matrix-vector product? I used PCSetType(pc, PCSHELL) and PCShellSetApply.

Thank you and appreciate your help!

Jiannan
________________________________
From: Tu, Jiannan <Jiannan_Tu at uml.edu>
Sent: Wednesday, July 6, 2022 2:34 PM
To: Barry Smith <bsmith at petsc.dev>
Cc: Jed Brown <jed at jedbrown.org>; petsc-users at mcs.anl.gov <petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] Using matrix-free with KSP

Barry,

VecScatterCreateToAll solves the problem! The code now gives the correct answer with multiple processes.

Thank you all so much!

Jiannan
________________________________
From: Barry Smith <bsmith at petsc.dev>
Sent: Wednesday, July 6, 2022 2:08 PM
To: Tu, Jiannan <Jiannan_Tu at uml.edu>
Cc: Jed Brown <jed at jedbrown.org>; petsc-users at mcs.anl.gov <petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] Using matrix-free with KSP


  So your operator is a "dense" operator in that its matrix representation would be essentially dense. In that case, you can use specialized routines to do this efficiently. You can use VecScatterCreateToAll() and then VecScatterBegin/End to do the communication.

  Barry



On Jul 6, 2022, at 1:39 PM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

Jed, thank you very much for the reply.

I'm not sure GlobaltoLocal will work. Say the solution vector is of length 10. Two processes then each process can see 5 elements of the vector. For A x = b (10 equations), each process performs matrix-vector product for five equations. And for each equation i , sum_j A_ij *  x_j = b_i requires the process has access to all 10 unknows x_i (i=0,1,...,9).

Can VecScatter and PetscSF make the entire vector accessible to each process? I am reading the manual and trying to understand how VecScatter and PetscSF work.

Jiannan
________________________________
From: Jed Brown <jed at jedbrown.org<mailto:jed at jedbrown.org>>
Sent: Wednesday, July 6, 2022 10:09 AM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>; Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] Using matrix-free with KSP

You'll usually have a GlobalToLocal operation for each rank to get the halo data it needs, then either a LocalToGlobal to collect the result (e.g., finite element methods) or the local compute will write directly into the owned portion of the global vector. If you're doing the communication "raw", then you may find VecScatter or PetscSF useful to perform the necessary communication.

"Tu, Jiannan" <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> writes:

> Hi Barry,
>
> Following your instructions I implemented the matrix-free method to solve a large linear equation system resulted from a surface integration equation. The KSP solver works fine for single process, but it is problematic with multiple processes. The problem is that each process only can access its own part of solution vector so that each process only conducts part of matrix-vector multiplication. MPI can be used to assemble these partial matrix-vector multiplication together.  Does Petsc provide any ways to implement multi-process matrix-free method?
>
> Thanks,
> Jiannan
> ________________________________
> From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
> Sent: Tuesday, May 24, 2022 2:12 PM
> To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
> Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
> Subject: Re: [petsc-users] Using matrix-free with KSP
>
> This e-mail originated from outside the UMass Lowell network.
> ________________________________
>
>    You can use MatCreateMFFD https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMatCreateMFFD%2F&amp;data=05%7C01%7CJiannan_Tu%40uml.edu%7Cab0859d8d154471590bc08da5f5918d6%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637927133525745809%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=A7wqegTfh94No5BpDiWLK3VxOuR44U2wlWHVm2k7l60%3D&amp;reserved=0<https://urldefense.com/v3/__https://petsc.org/main/docs/manualpages/Mat/MatCreateMFFD/__;!!PVKG_VDCxu5g!stxq2NWGVjUihvHBQC9Rlk0aHVoowcy0PTQYMcQxQ4DqOEC05KSw6TmstSXKckiUgelHzy4ue-d10-zlXkw$><https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMatCreateMFFD%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7C60675432e0b5409f004908da5f7a8b29%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637927277198462552%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=7SEmNO9TybDxwnhwcmyW1vFVc2eo7ys8JGBv9%2F%2FR2eI%3D&reserved=0> MatMFFDSetFunction MatSetFromOptions MatMFFDSetBase and provide the matrix to KSP. Note you will need to use -pc_type none or provide your own custom preconditioner withhttps://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FPC%2FPCSHELL%2F&amp;data=05%7C01%7CJiannan_Tu%40uml.edu%7Cab0859d8d154471590bc08da5f5918d6%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637927133525745809%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=Gb%2F9uFG3jp%2FbfWaSh2cSEQLItHS9CuLwarzN0KcNiJY%3D&amp;reserved=0<https://urldefense.com/v3/__https://petsc.org/main/docs/manualpages/PC/PCSHELL/__;!!PVKG_VDCxu5g!stxq2NWGVjUihvHBQC9Rlk0aHVoowcy0PTQYMcQxQ4DqOEC05KSw6TmstSXKckiUgelHzy4ue-d1cA4fKGw$><https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FPC%2FPCSHELL%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7C60675432e0b5409f004908da5f7a8b29%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637927277198462552%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=FDGqXk2UbO%2FYef5n21SIAUYrte093iKgL2p%2FwoA3qec%3D&reserved=0>
>
>
>
> On May 24, 2022, at 1:21 PM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu><mailto:Jiannan_Tu at uml.edu>> wrote:
>
> I want to use a matrix-free matrix to solve a large linear equation system because the matrix is too large to be stored. Petsc user manual describes matrix-free method for SNES with examples. The matrix-free matrices section explains how to set up such a matrix, but I can't find any example of matrix-free method with KSP. I am wondering how to apply the method to KSP iterative solver in parallel.
>
> I greatly appreciate your help for me to understand this topic.
>
> Jiannan Tu

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220816/3833db84/attachment.html>

From bsmith at petsc.dev  Tue Aug 16 09:10:49 2022
From: bsmith at petsc.dev (Barry Smith)
Date: Tue, 16 Aug 2022 10:10:49 -0400
Subject: [petsc-users] Using matrix-free with KSP
In-Reply-To: <BY5PR02MB647140084C7E7CDDB6BC2A11E76B9@BY5PR02MB6471.namprd02.prod.outlook.com>
References: <BY5PR02MB6471969F80FA5F732FC74D87E7D79@BY5PR02MB6471.namprd02.prod.outlook.com>
	<B55CFC2C-6364-4B78-8E99-9D53700743B8@petsc.dev>
	<BY5PR02MB6471BDF90783FE3A0EA176CBE7809@BY5PR02MB6471.namprd02.prod.outlook.com>
	<87iloa49kt.fsf@jedbrown.org>
	<BY5PR02MB6471C0ED04769D02046F68ACE7809@BY5PR02MB6471.namprd02.prod.outlook.com>
	<00EBE4B8-22E5-408F-B56D-14226A278D7E@petsc.dev>
	<BY5PR02MB6471E386057DDDE98AD05DFAE7809@BY5PR02MB6471.namprd02.prod.outlook.com>
	<BY5PR02MB647140084C7E7CDDB6BC2A11E76B9@BY5PR02MB6471.namprd02.prod.outlook.com>
Message-ID: <A6629DE1-5A7F-414E-B164-A69877158675@petsc.dev>


  I don't fully understand your question so I will answer questions that I do know the answer to :-)

  The function you provide to PCShellSetApply() applies the preconditioner to the input vector and puts the result in the output vector.  This input vector changes at every application of the preconditioner so should not be used in constructing the preconditioner.

  If you are using the standard PETSc matrix-free matrix-vector product via finite difference, -snes_mf_operator or via MatCreateSNESMF() or you are using a MatShell, the matrix-vector product routines input changes for each multiply and should not be used in constructing the preconditioner.

  The function you provide with SNESSetJacobian() is where you can compute anything you need to help build your preconditioner. The input vector to that function is the location at which the Jacobian is needed and is what you should use to build your approximate/part of Jacobian. You can store your approximate/part of Jacobian that you need to compute in the pmat and then in a PCShellSetUp() function that you provide you will compute what you will need latter to apply the preconditioner in your PCShellSetApply() function. Note that the values you put in the pmat do not need to be the full Jacobian; they can be anything you want since this pmat is not used to apply the matrix vector product.

  If this is unclear, feel free to ask additional questions.

  Barry


> On Aug 16, 2022, at 9:40 AM, Tu, Jiannan <Jiannan_Tu at uml.edu> wrote:
> 
> I am trying to apply a user-defined preconditioner to speed up the convergency of matrix-free KSP solver. The user's manual provides an example of how to call the user-defined preconditioner routine. The routine has PC, input Vec and output Vec as arguments. I want to use part of the Jacobian as preconditioner matrix, the elements of which can be calculated using input Vec. My question is what the role of output Vec from the routine is in matrix-vector product? Or I just have the preconditioner matrix elements calculated in the routine and then Petsc handles applying the preconditioner to matrix-vector product? I used PCSetType(pc, PCSHELL) and PCShellSetApply.
> 
> Thank you and appreciate your help!
> 
> Jiannan 
> From: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
> Sent: Wednesday, July 6, 2022 2:34 PM
> To: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>>
> Cc: Jed Brown <jed at jedbrown.org <mailto:jed at jedbrown.org>>; petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>>
> Subject: Re: [petsc-users] Using matrix-free with KSP
>  
> Barry,
> 
> VecScatterCreateToAll solves the problem! The code now gives the correct answer with multiple processes. 
> 
> Thank you all so much!
> 
> Jiannan
> From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>>
> Sent: Wednesday, July 6, 2022 2:08 PM
> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
> Cc: Jed Brown <jed at jedbrown.org <mailto:jed at jedbrown.org>>; petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>>
> Subject: Re: [petsc-users] Using matrix-free with KSP
>  
> 
>   So your operator is a "dense" operator in that its matrix representation would be essentially dense. In that case, you can use specialized routines to do this efficiently. You can use VecScatterCreateToAll() and then VecScatterBegin/End to do the communication.
> 
>   Barry
> 
> 
> 
>> On Jul 6, 2022, at 1:39 PM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> wrote:
>> 
>> Jed, thank you very much for the reply. 
>> 
>> I'm not sure GlobaltoLocal will work. Say the solution vector is of length 10. Two processes then each process can see 5 elements of the vector. For A x = b (10 equations), each process performs matrix-vector product for five equations. And for each equation i , sum_j A_ij *  x_j = b_i requires the process has access to all 10 unknows x_i (i=0,1,...,9). 
>> 
>> Can VecScatter and PetscSF make the entire vector accessible to each process? I am reading the manual and trying to understand how VecScatter and PetscSF work.
>> 
>> Jiannan
>> From: Jed Brown <jed at jedbrown.org <mailto:jed at jedbrown.org>>
>> Sent: Wednesday, July 6, 2022 10:09 AM
>> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>; Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>>
>> Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>>
>> Subject: Re: [petsc-users] Using matrix-free with KSP
>>  
>> You'll usually have a GlobalToLocal operation for each rank to get the halo data it needs, then either a LocalToGlobal to collect the result (e.g., finite element methods) or the local compute will write directly into the owned portion of the global vector. If you're doing the communication "raw", then you may find VecScatter or PetscSF useful to perform the necessary communication.
>> 
>> "Tu, Jiannan" <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> writes:
>> 
>> > Hi Barry,
>> >
>> > Following your instructions I implemented the matrix-free method to solve a large linear equation system resulted from a surface integration equation. The KSP solver works fine for single process, but it is problematic with multiple processes. The problem is that each process only can access its own part of solution vector so that each process only conducts part of matrix-vector multiplication. MPI can be used to assemble these partial matrix-vector multiplication together.  Does Petsc provide any ways to implement multi-process matrix-free method?
>> >
>> > Thanks,
>> > Jiannan
>> > ________________________________
>> > From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>>
>> > Sent: Tuesday, May 24, 2022 2:12 PM
>> > To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
>> > Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>>
>> > Subject: Re: [petsc-users] Using matrix-free with KSP
>> >
>> > This e-mail originated from outside the UMass Lowell network.
>> > ________________________________
>> >
>> >    You can use MatCreateMFFD https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMatCreateMFFD%2F&amp;data=05%7C01%7CJiannan_Tu%40uml.edu%7Cab0859d8d154471590bc08da5f5918d6%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637927133525745809%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=A7wqegTfh94No5BpDiWLK3VxOuR44U2wlWHVm2k7l60%3D&amp;reserved=0<https://urldefense.com/v3/__https://petsc.org/main/docs/manualpages/Mat/MatCreateMFFD/__;!!PVKG_VDCxu5g!stxq2NWGVjUihvHBQC9Rlk0aHVoowcy0PTQYMcQxQ4DqOEC05KSw6TmstSXKckiUgelHzy4ue-d10-zlXkw$> <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMatCreateMFFD%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7C60675432e0b5409f004908da5f7a8b29%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637927277198462552%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=7SEmNO9TybDxwnhwcmyW1vFVc2eo7ys8JGBv9%2F%2FR2eI%3D&reserved=0> MatMFFDSetFunction MatSetFromOptions MatMFFDSetBase and provide the matrix to KSP. Note you will need to use -pc_type none or provide your own custom preconditioner withhttps://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FPC%2FPCSHELL%2F&amp;data=05%7C01%7CJiannan_Tu%40uml.edu%7Cab0859d8d154471590bc08da5f5918d6%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637927133525745809%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=Gb%2F9uFG3jp%2FbfWaSh2cSEQLItHS9CuLwarzN0KcNiJY%3D&amp;reserved=0<https://urldefense.com/v3/__https://petsc.org/main/docs/manualpages/PC/PCSHELL/__;!!PVKG_VDCxu5g!stxq2NWGVjUihvHBQC9Rlk0aHVoowcy0PTQYMcQxQ4DqOEC05KSw6TmstSXKckiUgelHzy4ue-d1cA4fKGw$> <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FPC%2FPCSHELL%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7C60675432e0b5409f004908da5f7a8b29%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637927277198462552%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=FDGqXk2UbO%2FYef5n21SIAUYrte093iKgL2p%2FwoA3qec%3D&reserved=0>
>> >
>> >
>> >
>> > On May 24, 2022, at 1:21 PM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu><mailto:Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>> wrote:
>> >
>> > I want to use a matrix-free matrix to solve a large linear equation system because the matrix is too large to be stored. Petsc user manual describes matrix-free method for SNES with examples. The matrix-free matrices section explains how to set up such a matrix, but I can't find any example of matrix-free method with KSP. I am wondering how to apply the method to KSP iterative solver in parallel.
>> >
>> > I greatly appreciate your help for me to understand this topic.
>> >
>> > Jiannan Tu

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220816/14884696/attachment-0001.html>

From snarayanan1 at altair.com  Tue Aug 16 11:27:01 2022
From: snarayanan1 at altair.com (Sidarth Narayanan)
Date: Tue, 16 Aug 2022 16:27:01 +0000
Subject: [petsc-users] Scaling of PETSc example ex2f.F90
Message-ID: <BY5PR03MB5250C9FC58B74B2296441FCC916B9@BY5PR03MB5250.namprd03.prod.outlook.com>

Hello PETSc team,

I am currently using PETSc to improve the performance of a CFD solver by using it's parallel linear solver. While doing so I ran across some scaling issues where performance increase ( measured as KSPSolve time using MPI_Wtime() function )  in the linear algebra operation was only 2 times while using 4 processes. So I tried to check the scaling of the example problem ex2f.F90 provided as a part of the PETSc package and ran across a similar issue there. Here is the issue, If I run the problem on 1000 x 1000 2-D, five-point stencil with the exact solution set to 1.0, the scaling is perfect. But as soon as I change the exact solution to random values by using the flag -random_exact_sol (generally between 0 and 1), the scaling takes a  major hit. I have not changed the example code apart from increasing problem size and adding the time stamps before and after KSPSolve. Could you please let me know what I am missing (or) doing wrong here ? And I was also wondering if PETSc had an option to print out the pre-conditioned matrix as changing the exact solution changes the RHS and hence might change the pre-conditioned matrix. I have provided below the exact code and the results from the 4 runs including the compilation part.

Source Code: (ex2f.F90 with time stamps and increased problem size)
!
!  Description: Solves a linear system in parallel with KSP (Fortran code).
!               Also shows how to set a user-defined monitoring routine.
!
!
!
! -----------------------------------------------------------------------

      program main
#include <petsc/finclude/petscksp.h>
      use petscksp
      implicit none
!
! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
!                   Variable declarations
! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
!
!  Variables:
!     ksp     - linear solver context
!     ksp      - Krylov subspace method context
!     pc       - preconditioner context
!     x, b, u  - approx solution, right-hand-side, exact solution vectors
!     A        - matrix that defines linear system
!     its      - iterations for convergence
!     norm     - norm of error in solution
!     rctx     - random number generator context
!
!  Note that vectors are declared as PETSc "Vec" objects.  These vectors
!  are mathematical objects that contain more than just an array of
!  double precision numbers. I.e., vectors in PETSc are not just
!        double precision x(*).
!  However, local vector data can be easily accessed via VecGetArray().
!  See the Fortran section of the PETSc users manual for details.
!
      PetscReal  norm,t1,t2
      PetscInt  i,j,II,JJ,m,n,its
      PetscInt  Istart,Iend,ione
      PetscErrorCode ierr
      PetscMPIInt     rank,size
      PetscBool   flg
      PetscScalar v,one,neg_one
      Vec         x,b,u
      Mat         A
      KSP         ksp
      PetscRandom rctx
      PetscViewerAndFormat vf,vzero

!  These variables are not currently used.
!      PC          pc
!      PCType      ptype
!      PetscReal tol

!  Note: Any user-defined Fortran routines (such as MyKSPMonitor)
!  MUST be declared as external.

      external MyKSPMonitor,MyKSPConverged

! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
!                 Beginning of program
! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

      call PetscInitialize(PETSC_NULL_CHARACTER,ierr)
      if (ierr .ne. 0) then
        print*,'Unable to initialize PETSc'
        stop
      endif
      m = 1000
      n = 1000
      one  = 1.0
      neg_one = -1.0
      ione    = 1
      call PetscOptionsGetInt(PETSC_NULL_OPTIONS,PETSC_NULL_CHARACTER,'-m',m,flg,ierr)
      call PetscOptionsGetInt(PETSC_NULL_OPTIONS,PETSC_NULL_CHARACTER,'-n',n,flg,ierr)
      call MPI_Comm_rank(PETSC_COMM_WORLD,rank,ierr)
      call MPI_Comm_size(PETSC_COMM_WORLD,size,ierr)

! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
!      Compute the matrix and right-hand-side vector that define
!      the linear system, Ax = b.
! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

!  Create parallel matrix, specifying only its global dimensions.
!  When using MatCreate(), the matrix format can be specified at
!  runtime. Also, the parallel partitioning of the matrix is
!  determined by PETSc at runtime.

      call MatCreate(PETSC_COMM_WORLD,A,ierr)
      call MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,m*n,m*n,ierr)
      call MatSetFromOptions(A,ierr)
      call MatSetUp(A,ierr)

!  Currently, all PETSc parallel matrix formats are partitioned by
!  contiguous chunks of rows across the processors.  Determine which
!  rows of the matrix are locally owned.

      call MatGetOwnershipRange(A,Istart,Iend,ierr)

!  Set matrix elements for the 2-D, five-point stencil in parallel.
!   - Each processor needs to insert only elements that it owns
!     locally (but any non-local elements will be sent to the
!     appropriate processor during matrix assembly).
!   - Always specify global row and columns of matrix entries.
!   - Note that MatSetValues() uses 0-based row and column numbers
!     in Fortran as well as in C.

!     Note: this uses the less common natural ordering that orders first
!     all the unknowns for x = h then for x = 2h etc; Hence you see JH = II +- n
!     instead of JJ = II +- m as you might expect. The more standard ordering
!     would first do all variables for y = h, then y = 2h etc.

      do 10, II=Istart,Iend-1
        v = -1.0
        i = II/n
        j = II - i*n
        if (i.gt.0) then
          JJ = II - n
          call MatSetValues(A,ione,II,ione,JJ,v,INSERT_VALUES,ierr)
        endif
        if (i.lt.m-1) then
          JJ = II + n
          call MatSetValues(A,ione,II,ione,JJ,v,INSERT_VALUES,ierr)
        endif
        if (j.gt.0) then
          JJ = II - 1
          call MatSetValues(A,ione,II,ione,JJ,v,INSERT_VALUES,ierr)
        endif
        if (j.lt.n-1) then
          JJ = II + 1
          call MatSetValues(A,ione,II,ione,JJ,v,INSERT_VALUES,ierr)
        endif
        v = 4.0
        call  MatSetValues(A,ione,II,ione,II,v,INSERT_VALUES,ierr)
10   continue

!  Assemble matrix, using the 2-step process:
!       MatAssemblyBegin(), MatAssemblyEnd()
!  Computations can be done while messages are in transition,
!  by placing code between these two statements.

      call MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY,ierr)
      call MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY,ierr)

!  Create parallel vectors.
!   - Here, the parallel partitioning of the vector is determined by
!     PETSc at runtime.  We could also specify the local dimensions
!     if desired -- or use the more general routine VecCreate().
!   - When solving a linear system, the vectors and matrices MUST
!     be partitioned accordingly.  PETSc automatically generates
!     appropriately partitioned matrices and vectors when MatCreate()
!     and VecCreate() are used with the same communicator.
!   - Note: We form 1 vector from scratch and then duplicate as needed.

      call VecCreateMPI(PETSC_COMM_WORLD,PETSC_DECIDE,m*n,u,ierr)
      call VecSetFromOptions(u,ierr)
      call VecDuplicate(u,b,ierr)
      call VecDuplicate(b,x,ierr)

!  Set exact solution; then compute right-hand-side vector.
!  By default we use an exact solution of a vector with all
!  elements of 1.0;  Alternatively, using the runtime option
!  -random_sol forms a solution vector with random components.

      call PetscOptionsHasName(PETSC_NULL_OPTIONS,PETSC_NULL_CHARACTER,'-random_exact_sol',flg,ierr)
      if (flg) then
         call PetscRandomCreate(PETSC_COMM_WORLD,rctx,ierr)
         call PetscRandomSetFromOptions(rctx,ierr)
         call VecSetRandom(u,rctx,ierr)
         call PetscRandomDestroy(rctx,ierr)
      else
         call VecSet(u,one,ierr)
      endif
      call MatMult(A,u,b,ierr)

!  View the exact solution vector if desired

      call PetscOptionsHasName(PETSC_NULL_OPTIONS,PETSC_NULL_CHARACTER,'-view_exact_sol',flg,ierr)
      if (flg) then
         call VecView(u,PETSC_VIEWER_STDOUT_WORLD,ierr)
      endif

! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
!         Create the linear solver and set various options
! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

!  Create linear solver context

      call KSPCreate(PETSC_COMM_WORLD,ksp,ierr)

!  Set operators. Here the matrix that defines the linear system
!  also serves as the preconditioning matrix.

      call KSPSetOperators(ksp,A,A,ierr)

!  Set linear solver defaults for this problem (optional).
!   - By extracting the KSP and PC contexts from the KSP context,
!     we can then directly directly call any KSP and PC routines
!     to set various options.
!   - The following four statements are optional; all of these
!     parameters could alternatively be specified at runtime via
!     KSPSetFromOptions(). All of these defaults can be
!     overridden at runtime, as indicated below.

!     We comment out this section of code since the Jacobi
!     preconditioner is not a good general default.

!      call KSPGetPC(ksp,pc,ierr)
!      ptype = PCJACOBI
!      call PCSetType(pc,ptype,ierr)
!      tol = 1.e-7
!      call KSPSetTolerances(ksp,tol,PETSC_DEFAULT_REAL,PETSC_DEFAULT_REAL,PETSC_DEFAULT_INTEGER,ierr)

!  Set user-defined monitoring routine if desired

      call PetscOptionsHasName(PETSC_NULL_OPTIONS,PETSC_NULL_CHARACTER,'-my_ksp_monitor',flg,ierr)
      if (flg) then
        vzero = 0
        call KSPMonitorSet(ksp,MyKSPMonitor,vzero,PETSC_NULL_FUNCTION,ierr)
!
!     Also use the default KSP monitor routine showing how it may be used from Fortran
!
        call PetscViewerAndFormatCreate(PETSC_VIEWER_STDOUT_WORLD,PETSC_VIEWER_DEFAULT,vf,ierr)
        call KSPMonitorSet(ksp,KSPMonitorResidual,vf,PetscViewerAndFormatDestroy,ierr)
      endif

!  Set runtime options, e.g.,
!      -ksp_type <type> -pc_type <type> -ksp_monitor -ksp_rtol <rtol>
!  These options will override those specified above as long as
!  KSPSetFromOptions() is called _after_ any other customization
!  routines.

      call KSPSetFromOptions(ksp,ierr)

!  Set convergence test routine if desired

      call PetscOptionsHasName(PETSC_NULL_OPTIONS,PETSC_NULL_CHARACTER,'-my_ksp_convergence',flg,ierr)
      if (flg) then
        call KSPSetConvergenceTest(ksp,MyKSPConverged,0,PETSC_NULL_FUNCTION,ierr)
      endif
!
! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
!                      Solve the linear system
! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
                 t1 = MPI_Wtime()
      call KSPSolve(ksp,b,x,ierr)
                 t2 = MPI_Wtime()
                 !x =A-1b
! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
!                     Check solution and clean up
! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

!  Check the error
      call VecAXPY(x,neg_one,u,ierr)
      call VecNorm(x,NORM_2,norm,ierr)
      call KSPGetIterationNumber(ksp,its,ierr)
      if (rank .eq. 0) then
        if (norm .gt. 1.e-12) then
           write(6,100) norm,its
        else
           write(6,110) its
        endif
                              write(6,111) t2 - t1
      endif
  100 format('Norm of error ',e11.4,' iterations ',i5)
  110 format('Norm of error < 1.e-12 iterations ',i5)
  111 format('KSPSolve time: ',f10.3)

!  Free work space.  All PETSc objects should be destroyed when they
!  are no longer needed.

      call KSPDestroy(ksp,ierr)
      call VecDestroy(u,ierr)
      call VecDestroy(x,ierr)
      call VecDestroy(b,ierr)
      call MatDestroy(A,ierr)

!  Always call PetscFinalize() before exiting a program.  This routine
!    - finalizes the PETSc libraries as well as MPI
!    - provides summary and diagnostic information if certain runtime
!      options are chosen (e.g., -log_view).  See PetscFinalize()
!      manpage for more information.

      call PetscFinalize(ierr)
      end

! --------------------------------------------------------------
!
!  MyKSPMonitor - This is a user-defined routine for monitoring
!  the KSP iterative solvers.
!
!  Input Parameters:
!    ksp   - iterative context
!    n     - iteration number
!    rnorm - 2-norm (preconditioned) residual value (may be estimated)
!    dummy - optional user-defined monitor context (unused here)
!
      subroutine MyKSPMonitor(ksp,n,rnorm,dummy,ierr)
      use petscksp
      implicit none

      KSP              ksp
      Vec              x
      PetscErrorCode ierr
      PetscInt n,dummy
      PetscMPIInt rank
      PetscReal rnorm

!  Build the solution vector
      call KSPBuildSolution(ksp,PETSC_NULL_VEC,x,ierr)

!  Write the solution vector and residual norm to stdout
!   - Note that the parallel viewer PETSC_VIEWER_STDOUT_WORLD
!     handles data from multiple processors so that the
!     output is not jumbled.

      call MPI_Comm_rank(PETSC_COMM_WORLD,rank,ierr)
      if (rank .eq. 0) write(6,100) n
      call VecView(x,PETSC_VIEWER_STDOUT_WORLD,ierr)
      if (rank .eq. 0) write(6,200) n,rnorm

100  format('iteration ',i5,' solution vector:')
200  format('iteration ',i5,' residual norm ',e11.4)
      ierr = 0
      end

! --------------------------------------------------------------
!
!  MyKSPConverged - This is a user-defined routine for testing
!  convergence of the KSP iterative solvers.
!
!  Input Parameters:
!    ksp   - iterative context
!    n     - iteration number
!    rnorm - 2-norm (preconditioned) residual value (may be estimated)
!    dummy - optional user-defined monitor context (unused here)
!
      subroutine MyKSPConverged(ksp,n,rnorm,flag,dummy,ierr)
      use petscksp
      implicit none

      KSP              ksp
      PetscErrorCode ierr
      PetscInt n,dummy
      KSPConvergedReason flag
      PetscReal rnorm

      if (rnorm .le. .05) then
        flag = 1
      else
        flag = 0
      endif
      ierr = 0

      end

!/*TEST
!
!   test:
!      nsize: 2
!      args: -pc_type jacobi -ksp_monitor_short -ksp_gmres_cgs_refinement_type refine_always
!
!   test:
!      suffix: 2
!      nsize: 2
!      args: -pc_type jacobi -my_ksp_monitor -ksp_gmres_cgs_refinement_type refine_always
!
!TEST*/

Output:
snarayanan1 at USLN38 /cygdrive/c/sidarth/petsc-test/scaling_test
$ make ex2f
/home/snarayanan1/petsc-release/lib/petsc/bin/win32fe/win32fe ifort -MT -O3 -fpp  -MT -O3 -fpp   -I/home/snarayanan1/petsc-release/include -I/home/snarayanan1/petsc-release/arch-ci-mswin-opt-impi/include -I/cygdrive/c/PROGRA~2/Intel/oneAPI/mpi/2021.5.0/include     ex2f.F90  -L/cygdrive/c/cygwin64/home/snarayanan1/petsc-release/arch-ci-mswin-opt-impi/lib -L/cygdrive/c/PROGRA~2/Intel/oneAPI/mkl/2022.0.0/lib/intel64 -lpetsc mkl_intel_lp64_dll.lib mkl_sequential_dll.lib mkl_core_dll.lib /cygdrive/c/PROGRA~2/Intel/oneAPI/mpi/2021.5.0/lib/release/impi.lib Gdi32.lib User32.lib Advapi32.lib Kernel32.lib Ws2_32.lib -o ex2f

snarayanan1 at USLN38 /cygdrive/c/sidarth/petsc-test/scaling_test
$ $PETSC_DIR/lib/petsc/bin/petscmpiexec -n 1 ex2f.exe
Norm of error  0.1214E+02 iterations  2571
KSPSolve time:    104.949

snarayanan1 at USLN38 /cygdrive/c/sidarth/petsc-test/scaling_test
$ $PETSC_DIR/lib/petsc/bin/petscmpiexec -n 2 ex2f.exe
Norm of error  0.1221E+02 iterations  1821
KSPSolve time:     47.139

snarayanan1 at USLN38 /cygdrive/c/sidarth/petsc-test/scaling_test
$ $PETSC_DIR/lib/petsc/bin/petscmpiexec -n 1 ex2f.exe -random_exact_sol
Norm of error  0.8541E+02 iterations  1003
KSPSolve time:     40.853

snarayanan1 at USLN38 /cygdrive/c/sidarth/petsc-test/scaling_test
$ $PETSC_DIR/lib/petsc/bin/petscmpiexec -n 2 ex2f.exe -random_exact_sol
Norm of error  0.8562E+02 iterations  1191
KSPSolve time:     31.745


Thank You,
Sidarth Narayanan
Electronics Thermal Management Intern
Altair Engineering Inc.
Troy, MI
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220816/39a9f47c/attachment-0001.html>

From Jiannan_Tu at uml.edu  Tue Aug 16 11:31:35 2022
From: Jiannan_Tu at uml.edu (Tu, Jiannan)
Date: Tue, 16 Aug 2022 16:31:35 +0000
Subject: [petsc-users] Using matrix-free with KSP
In-Reply-To: <A6629DE1-5A7F-414E-B164-A69877158675@petsc.dev>
References: <BY5PR02MB6471969F80FA5F732FC74D87E7D79@BY5PR02MB6471.namprd02.prod.outlook.com>
	<B55CFC2C-6364-4B78-8E99-9D53700743B8@petsc.dev>
	<BY5PR02MB6471BDF90783FE3A0EA176CBE7809@BY5PR02MB6471.namprd02.prod.outlook.com>
	<87iloa49kt.fsf@jedbrown.org>
	<BY5PR02MB6471C0ED04769D02046F68ACE7809@BY5PR02MB6471.namprd02.prod.outlook.com>
	<00EBE4B8-22E5-408F-B56D-14226A278D7E@petsc.dev>
	<BY5PR02MB6471E386057DDDE98AD05DFAE7809@BY5PR02MB6471.namprd02.prod.outlook.com>
	<BY5PR02MB647140084C7E7CDDB6BC2A11E76B9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<A6629DE1-5A7F-414E-B164-A69877158675@petsc.dev>
Message-ID: <BY5PR02MB6471AB1C004920DA1FC9B43EE76B9@BY5PR02MB6471.namprd02.prod.outlook.com>

Barry,

Thank you very much for your instructions. I am sorry I may not ask clearer questions.

I don't use SNES. What I am trying to solve is a very large linear equation system with dense and ill-conditioned matrix. Selected parts of the matrix can be pre-calculated and stored to serve as the preconditioner. My question is how I can apply this matrix as the preconditioner.

Here is the part of code from my linear equations solver. MatrixFreePreconditioner is supposed to be the routine to provide the preconditioner. But I don't understand how to use it. Should I use SNESSetJacobian() even if this is a linear problem?

    MatCreateMFFD(MPI_COMM_WORLD, PETSC_DECIDE, PETSC_DECIDE, N, N, &A);
    MatMFFDSetFunction(A, formfunction, &params);
    MatSetFromOptions(A);
    MatMFFDSetBase(A, X, NULL);

    KSPCreate(MPI_COMM_WORLD, &ksp);
    KSPSetOperators(ksp, A, A);
    KSPSetFromOptions(ksp);

    PC pc;
    KSPGetPC(ksp,&pc);
    PCSetType(pc,PCSHELL);
    PCShellSetApply(pc,MatrixFreePreconditioner);

Thank you,
Jiannan

________________________________
From: Barry Smith <bsmith at petsc.dev>
Sent: Tuesday, August 16, 2022 10:10 AM
To: Tu, Jiannan <Jiannan_Tu at uml.edu>
Cc: petsc-users at mcs.anl.gov <petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] Using matrix-free with KSP


CAUTION: This email was sent from outside the UMass Lowell network.


  I don't fully understand your question so I will answer questions that I do know the answer to :-)

  The function you provide to PCShellSetApply() applies the preconditioner to the input vector and puts the result in the output vector.  This input vector changes at every application of the preconditioner so should not be used in constructing the preconditioner.

  If you are using the standard PETSc matrix-free matrix-vector product via finite difference, -snes_mf_operator or via MatCreateSNESMF() or you are using a MatShell, the matrix-vector product routines input changes for each multiply and should not be used in constructing the preconditioner.

  The function you provide with SNESSetJacobian() is where you can compute anything you need to help build your preconditioner. The input vector to that function is the location at which the Jacobian is needed and is what you should use to build your approximate/part of Jacobian. You can store your approximate/part of Jacobian that you need to compute in the pmat and then in a PCShellSetUp() function that you provide you will compute what you will need latter to apply the preconditioner in your PCShellSetApply() function. Note that the values you put in the pmat do not need to be the full Jacobian; they can be anything you want since this pmat is not used to apply the matrix vector product.

  If this is unclear, feel free to ask additional questions.

  Barry


On Aug 16, 2022, at 9:40 AM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

I am trying to apply a user-defined preconditioner to speed up the convergency of matrix-free KSP solver. The user's manual provides an example of how to call the user-defined preconditioner routine. The routine has PC, input Vec and output Vec as arguments. I want to use part of the Jacobian as preconditioner matrix, the elements of which can be calculated using input Vec. My question is what the role of output Vec from the routine is in matrix-vector product? Or I just have the preconditioner matrix elements calculated in the routine and then Petsc handles applying the preconditioner to matrix-vector product? I used PCSetType(pc, PCSHELL) and PCShellSetApply.

Thank you and appreciate your help!

Jiannan
________________________________
From: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Sent: Wednesday, July 6, 2022 2:34 PM
To: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Cc: Jed Brown <jed at jedbrown.org<mailto:jed at jedbrown.org>>; petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] Using matrix-free with KSP

Barry,

VecScatterCreateToAll solves the problem! The code now gives the correct answer with multiple processes.

Thank you all so much!

Jiannan
________________________________
From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Wednesday, July 6, 2022 2:08 PM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Cc: Jed Brown <jed at jedbrown.org<mailto:jed at jedbrown.org>>; petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] Using matrix-free with KSP


  So your operator is a "dense" operator in that its matrix representation would be essentially dense. In that case, you can use specialized routines to do this efficiently. You can use VecScatterCreateToAll() and then VecScatterBegin/End to do the communication.

  Barry



On Jul 6, 2022, at 1:39 PM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

Jed, thank you very much for the reply.

I'm not sure GlobaltoLocal will work. Say the solution vector is of length 10. Two processes then each process can see 5 elements of the vector. For A x = b (10 equations), each process performs matrix-vector product for five equations. And for each equation i , sum_j A_ij *  x_j = b_i requires the process has access to all 10 unknows x_i (i=0,1,...,9).

Can VecScatter and PetscSF make the entire vector accessible to each process? I am reading the manual and trying to understand how VecScatter and PetscSF work.

Jiannan
________________________________
From: Jed Brown <jed at jedbrown.org<mailto:jed at jedbrown.org>>
Sent: Wednesday, July 6, 2022 10:09 AM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>; Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] Using matrix-free with KSP

You'll usually have a GlobalToLocal operation for each rank to get the halo data it needs, then either a LocalToGlobal to collect the result (e.g., finite element methods) or the local compute will write directly into the owned portion of the global vector. If you're doing the communication "raw", then you may find VecScatter or PetscSF useful to perform the necessary communication.

"Tu, Jiannan" <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> writes:

> Hi Barry,
>
> Following your instructions I implemented the matrix-free method to solve a large linear equation system resulted from a surface integration equation. The KSP solver works fine for single process, but it is problematic with multiple processes. The problem is that each process only can access its own part of solution vector so that each process only conducts part of matrix-vector multiplication. MPI can be used to assemble these partial matrix-vector multiplication together.  Does Petsc provide any ways to implement multi-process matrix-free method?
>
> Thanks,
> Jiannan
> ________________________________
> From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
> Sent: Tuesday, May 24, 2022 2:12 PM
> To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
> Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
> Subject: Re: [petsc-users] Using matrix-free with KSP
>
> This e-mail originated from outside the UMass Lowell network.
> ________________________________
>
>    You can use MatCreateMFFD https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMatCreateMFFD%2F&amp;data=05%7C01%7CJiannan_Tu%40uml.edu%7Cab0859d8d154471590bc08da5f5918d6%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637927133525745809%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=A7wqegTfh94No5BpDiWLK3VxOuR44U2wlWHVm2k7l60%3D&amp;reserved=0<https://urldefense.com/v3/__https://petsc.org/main/docs/manualpages/Mat/MatCreateMFFD/__;!!PVKG_VDCxu5g!stxq2NWGVjUihvHBQC9Rlk0aHVoowcy0PTQYMcQxQ4DqOEC05KSw6TmstSXKckiUgelHzy4ue-d10-zlXkw$><https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMatCreateMFFD%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7C9cf9b4c72f7c4f0cd92a08da7f9120f4%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637962558562603998%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=BdPTAQcdmpnMYPbWECCabNQSzJfqpiH5G7iq1d%2FlY4k%3D&reserved=0> MatMFFDSetFunction MatSetFromOptions MatMFFDSetBase and provide the matrix to KSP. Note you will need to use -pc_type none or provide your own custom preconditioner withhttps://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FPC%2FPCSHELL%2F&amp;data=05%7C01%7CJiannan_Tu%40uml.edu%7Cab0859d8d154471590bc08da5f5918d6%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637927133525745809%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=Gb%2F9uFG3jp%2FbfWaSh2cSEQLItHS9CuLwarzN0KcNiJY%3D&amp;reserved=0<https://urldefense.com/v3/__https://petsc.org/main/docs/manualpages/PC/PCSHELL/__;!!PVKG_VDCxu5g!stxq2NWGVjUihvHBQC9Rlk0aHVoowcy0PTQYMcQxQ4DqOEC05KSw6TmstSXKckiUgelHzy4ue-d1cA4fKGw$><https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FPC%2FPCSHELL%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7C9cf9b4c72f7c4f0cd92a08da7f9120f4%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637962558562603998%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=6XJ3yp6S8IbTevj%2BeUOHDjgCbeTq346j%2Fini%2BLGmkag%3D&reserved=0>
>
>
>
> On May 24, 2022, at 1:21 PM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu><mailto:Jiannan_Tu at uml.edu>> wrote:
>
> I want to use a matrix-free matrix to solve a large linear equation system because the matrix is too large to be stored. Petsc user manual describes matrix-free method for SNES with examples. The matrix-free matrices section explains how to set up such a matrix, but I can't find any example of matrix-free method with KSP. I am wondering how to apply the method to KSP iterative solver in parallel.
>
> I greatly appreciate your help for me to understand this topic.
>
> Jiannan Tu

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220816/f7fe5259/attachment-0001.html>

From jed at jedbrown.org  Tue Aug 16 12:06:22 2022
From: jed at jedbrown.org (Jed Brown)
Date: Tue, 16 Aug 2022 11:06:22 -0600
Subject: [petsc-users] Scaling of PETSc example ex2f.F90
In-Reply-To: <BY5PR03MB5250C9FC58B74B2296441FCC916B9@BY5PR03MB5250.namprd03.prod.outlook.com>
References: <BY5PR03MB5250C9FC58B74B2296441FCC916B9@BY5PR03MB5250.namprd03.prod.outlook.com>
Message-ID: <87zgg4yv75.fsf@jedbrown.org>

Your observed performance is explained by the number of iterations, which varies based on a number of factors, plus imperfect memory bandwidth scaling (https://petsc.org/release/faq/#what-kind-of-parallel-computers-or-clusters-are-needed-to-use-petsc-or-why-do-i-get-little-speedup).

Problems like this (and harder) can be solved much faster using a method like -pc_type gamg or -pc_type hypre (if configured with --download-hypre). Also note that this example does an intentionally poor job of partitioning (so it's easy to understand what PETSc is doing) -- you should use a better partition and preallocation for a real problem.

Sidarth Narayanan <snarayanan1 at altair.com> writes:

> Hello PETSc team,
>
> I am currently using PETSc to improve the performance of a CFD solver by using it's parallel linear solver. While doing so I ran across some scaling issues where performance increase ( measured as KSPSolve time using MPI_Wtime() function )  in the linear algebra operation was only 2 times while using 4 processes. So I tried to check the scaling of the example problem ex2f.F90 provided as a part of the PETSc package and ran across a similar issue there. Here is the issue, If I run the problem on 1000 x 1000 2-D, five-point stencil with the exact solution set to 1.0, the scaling is perfect. But as soon as I change the exact solution to random values by using the flag -random_exact_sol (generally between 0 and 1), the scaling takes a  major hit. I have not changed the example code apart from increasing problem size and adding the time stamps before and after KSPSolve. Could you please let me know what I am missing (or) doing wrong here ? And I was also wondering if PETSc had an option to print out the pre-conditioned matrix as changing the exact solution changes the RHS and hence might change the pre-conditioned matrix. I have provided below the exact code and the results from the 4 runs including the compilation part.
>
> Source Code: (ex2f.F90 with time stamps and increased problem size)
> !
> !  Description: Solves a linear system in parallel with KSP (Fortran code).
> !               Also shows how to set a user-defined monitoring routine.
> !
> !
> !
> ! -----------------------------------------------------------------------
>
>       program main
> #include <petsc/finclude/petscksp.h>
>       use petscksp
>       implicit none
> !
> ! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
> !                   Variable declarations
> ! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
> !
> !  Variables:
> !     ksp     - linear solver context
> !     ksp      - Krylov subspace method context
> !     pc       - preconditioner context
> !     x, b, u  - approx solution, right-hand-side, exact solution vectors
> !     A        - matrix that defines linear system
> !     its      - iterations for convergence
> !     norm     - norm of error in solution
> !     rctx     - random number generator context
> !
> !  Note that vectors are declared as PETSc "Vec" objects.  These vectors
> !  are mathematical objects that contain more than just an array of
> !  double precision numbers. I.e., vectors in PETSc are not just
> !        double precision x(*).
> !  However, local vector data can be easily accessed via VecGetArray().
> !  See the Fortran section of the PETSc users manual for details.
> !
>       PetscReal  norm,t1,t2
>       PetscInt  i,j,II,JJ,m,n,its
>       PetscInt  Istart,Iend,ione
>       PetscErrorCode ierr
>       PetscMPIInt     rank,size
>       PetscBool   flg
>       PetscScalar v,one,neg_one
>       Vec         x,b,u
>       Mat         A
>       KSP         ksp
>       PetscRandom rctx
>       PetscViewerAndFormat vf,vzero
>
> !  These variables are not currently used.
> !      PC          pc
> !      PCType      ptype
> !      PetscReal tol
>
> !  Note: Any user-defined Fortran routines (such as MyKSPMonitor)
> !  MUST be declared as external.
>
>       external MyKSPMonitor,MyKSPConverged
>
> ! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
> !                 Beginning of program
> ! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
>
>       call PetscInitialize(PETSC_NULL_CHARACTER,ierr)
>       if (ierr .ne. 0) then
>         print*,'Unable to initialize PETSc'
>         stop
>       endif
>       m = 1000
>       n = 1000
>       one  = 1.0
>       neg_one = -1.0
>       ione    = 1
>       call PetscOptionsGetInt(PETSC_NULL_OPTIONS,PETSC_NULL_CHARACTER,'-m',m,flg,ierr)
>       call PetscOptionsGetInt(PETSC_NULL_OPTIONS,PETSC_NULL_CHARACTER,'-n',n,flg,ierr)
>       call MPI_Comm_rank(PETSC_COMM_WORLD,rank,ierr)
>       call MPI_Comm_size(PETSC_COMM_WORLD,size,ierr)
>
> ! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
> !      Compute the matrix and right-hand-side vector that define
> !      the linear system, Ax = b.
> ! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
>
> !  Create parallel matrix, specifying only its global dimensions.
> !  When using MatCreate(), the matrix format can be specified at
> !  runtime. Also, the parallel partitioning of the matrix is
> !  determined by PETSc at runtime.
>
>       call MatCreate(PETSC_COMM_WORLD,A,ierr)
>       call MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,m*n,m*n,ierr)
>       call MatSetFromOptions(A,ierr)
>       call MatSetUp(A,ierr)
>
> !  Currently, all PETSc parallel matrix formats are partitioned by
> !  contiguous chunks of rows across the processors.  Determine which
> !  rows of the matrix are locally owned.
>
>       call MatGetOwnershipRange(A,Istart,Iend,ierr)
>
> !  Set matrix elements for the 2-D, five-point stencil in parallel.
> !   - Each processor needs to insert only elements that it owns
> !     locally (but any non-local elements will be sent to the
> !     appropriate processor during matrix assembly).
> !   - Always specify global row and columns of matrix entries.
> !   - Note that MatSetValues() uses 0-based row and column numbers
> !     in Fortran as well as in C.
>
> !     Note: this uses the less common natural ordering that orders first
> !     all the unknowns for x = h then for x = 2h etc; Hence you see JH = II +- n
> !     instead of JJ = II +- m as you might expect. The more standard ordering
> !     would first do all variables for y = h, then y = 2h etc.
>
>       do 10, II=Istart,Iend-1
>         v = -1.0
>         i = II/n
>         j = II - i*n
>         if (i.gt.0) then
>           JJ = II - n
>           call MatSetValues(A,ione,II,ione,JJ,v,INSERT_VALUES,ierr)
>         endif
>         if (i.lt.m-1) then
>           JJ = II + n
>           call MatSetValues(A,ione,II,ione,JJ,v,INSERT_VALUES,ierr)
>         endif
>         if (j.gt.0) then
>           JJ = II - 1
>           call MatSetValues(A,ione,II,ione,JJ,v,INSERT_VALUES,ierr)
>         endif
>         if (j.lt.n-1) then
>           JJ = II + 1
>           call MatSetValues(A,ione,II,ione,JJ,v,INSERT_VALUES,ierr)
>         endif
>         v = 4.0
>         call  MatSetValues(A,ione,II,ione,II,v,INSERT_VALUES,ierr)
> 10   continue
>
> !  Assemble matrix, using the 2-step process:
> !       MatAssemblyBegin(), MatAssemblyEnd()
> !  Computations can be done while messages are in transition,
> !  by placing code between these two statements.
>
>       call MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY,ierr)
>       call MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY,ierr)
>
> !  Create parallel vectors.
> !   - Here, the parallel partitioning of the vector is determined by
> !     PETSc at runtime.  We could also specify the local dimensions
> !     if desired -- or use the more general routine VecCreate().
> !   - When solving a linear system, the vectors and matrices MUST
> !     be partitioned accordingly.  PETSc automatically generates
> !     appropriately partitioned matrices and vectors when MatCreate()
> !     and VecCreate() are used with the same communicator.
> !   - Note: We form 1 vector from scratch and then duplicate as needed.
>
>       call VecCreateMPI(PETSC_COMM_WORLD,PETSC_DECIDE,m*n,u,ierr)
>       call VecSetFromOptions(u,ierr)
>       call VecDuplicate(u,b,ierr)
>       call VecDuplicate(b,x,ierr)
>
> !  Set exact solution; then compute right-hand-side vector.
> !  By default we use an exact solution of a vector with all
> !  elements of 1.0;  Alternatively, using the runtime option
> !  -random_sol forms a solution vector with random components.
>
>       call PetscOptionsHasName(PETSC_NULL_OPTIONS,PETSC_NULL_CHARACTER,'-random_exact_sol',flg,ierr)
>       if (flg) then
>          call PetscRandomCreate(PETSC_COMM_WORLD,rctx,ierr)
>          call PetscRandomSetFromOptions(rctx,ierr)
>          call VecSetRandom(u,rctx,ierr)
>          call PetscRandomDestroy(rctx,ierr)
>       else
>          call VecSet(u,one,ierr)
>       endif
>       call MatMult(A,u,b,ierr)
>
> !  View the exact solution vector if desired
>
>       call PetscOptionsHasName(PETSC_NULL_OPTIONS,PETSC_NULL_CHARACTER,'-view_exact_sol',flg,ierr)
>       if (flg) then
>          call VecView(u,PETSC_VIEWER_STDOUT_WORLD,ierr)
>       endif
>
> ! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
> !         Create the linear solver and set various options
> ! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
>
> !  Create linear solver context
>
>       call KSPCreate(PETSC_COMM_WORLD,ksp,ierr)
>
> !  Set operators. Here the matrix that defines the linear system
> !  also serves as the preconditioning matrix.
>
>       call KSPSetOperators(ksp,A,A,ierr)
>
> !  Set linear solver defaults for this problem (optional).
> !   - By extracting the KSP and PC contexts from the KSP context,
> !     we can then directly directly call any KSP and PC routines
> !     to set various options.
> !   - The following four statements are optional; all of these
> !     parameters could alternatively be specified at runtime via
> !     KSPSetFromOptions(). All of these defaults can be
> !     overridden at runtime, as indicated below.
>
> !     We comment out this section of code since the Jacobi
> !     preconditioner is not a good general default.
>
> !      call KSPGetPC(ksp,pc,ierr)
> !      ptype = PCJACOBI
> !      call PCSetType(pc,ptype,ierr)
> !      tol = 1.e-7
> !      call KSPSetTolerances(ksp,tol,PETSC_DEFAULT_REAL,PETSC_DEFAULT_REAL,PETSC_DEFAULT_INTEGER,ierr)
>
> !  Set user-defined monitoring routine if desired
>
>       call PetscOptionsHasName(PETSC_NULL_OPTIONS,PETSC_NULL_CHARACTER,'-my_ksp_monitor',flg,ierr)
>       if (flg) then
>         vzero = 0
>         call KSPMonitorSet(ksp,MyKSPMonitor,vzero,PETSC_NULL_FUNCTION,ierr)
> !
> !     Also use the default KSP monitor routine showing how it may be used from Fortran
> !
>         call PetscViewerAndFormatCreate(PETSC_VIEWER_STDOUT_WORLD,PETSC_VIEWER_DEFAULT,vf,ierr)
>         call KSPMonitorSet(ksp,KSPMonitorResidual,vf,PetscViewerAndFormatDestroy,ierr)
>       endif
>
> !  Set runtime options, e.g.,
> !      -ksp_type <type> -pc_type <type> -ksp_monitor -ksp_rtol <rtol>
> !  These options will override those specified above as long as
> !  KSPSetFromOptions() is called _after_ any other customization
> !  routines.
>
>       call KSPSetFromOptions(ksp,ierr)
>
> !  Set convergence test routine if desired
>
>       call PetscOptionsHasName(PETSC_NULL_OPTIONS,PETSC_NULL_CHARACTER,'-my_ksp_convergence',flg,ierr)
>       if (flg) then
>         call KSPSetConvergenceTest(ksp,MyKSPConverged,0,PETSC_NULL_FUNCTION,ierr)
>       endif
> !
> ! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
> !                      Solve the linear system
> ! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
>                  t1 = MPI_Wtime()
>       call KSPSolve(ksp,b,x,ierr)
>                  t2 = MPI_Wtime()
>                  !x =A-1b
> ! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
> !                     Check solution and clean up
> ! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
>
> !  Check the error
>       call VecAXPY(x,neg_one,u,ierr)
>       call VecNorm(x,NORM_2,norm,ierr)
>       call KSPGetIterationNumber(ksp,its,ierr)
>       if (rank .eq. 0) then
>         if (norm .gt. 1.e-12) then
>            write(6,100) norm,its
>         else
>            write(6,110) its
>         endif
>                               write(6,111) t2 - t1
>       endif
>   100 format('Norm of error ',e11.4,' iterations ',i5)
>   110 format('Norm of error < 1.e-12 iterations ',i5)
>   111 format('KSPSolve time: ',f10.3)
>
> !  Free work space.  All PETSc objects should be destroyed when they
> !  are no longer needed.
>
>       call KSPDestroy(ksp,ierr)
>       call VecDestroy(u,ierr)
>       call VecDestroy(x,ierr)
>       call VecDestroy(b,ierr)
>       call MatDestroy(A,ierr)
>
> !  Always call PetscFinalize() before exiting a program.  This routine
> !    - finalizes the PETSc libraries as well as MPI
> !    - provides summary and diagnostic information if certain runtime
> !      options are chosen (e.g., -log_view).  See PetscFinalize()
> !      manpage for more information.
>
>       call PetscFinalize(ierr)
>       end
>
> ! --------------------------------------------------------------
> !
> !  MyKSPMonitor - This is a user-defined routine for monitoring
> !  the KSP iterative solvers.
> !
> !  Input Parameters:
> !    ksp   - iterative context
> !    n     - iteration number
> !    rnorm - 2-norm (preconditioned) residual value (may be estimated)
> !    dummy - optional user-defined monitor context (unused here)
> !
>       subroutine MyKSPMonitor(ksp,n,rnorm,dummy,ierr)
>       use petscksp
>       implicit none
>
>       KSP              ksp
>       Vec              x
>       PetscErrorCode ierr
>       PetscInt n,dummy
>       PetscMPIInt rank
>       PetscReal rnorm
>
> !  Build the solution vector
>       call KSPBuildSolution(ksp,PETSC_NULL_VEC,x,ierr)
>
> !  Write the solution vector and residual norm to stdout
> !   - Note that the parallel viewer PETSC_VIEWER_STDOUT_WORLD
> !     handles data from multiple processors so that the
> !     output is not jumbled.
>
>       call MPI_Comm_rank(PETSC_COMM_WORLD,rank,ierr)
>       if (rank .eq. 0) write(6,100) n
>       call VecView(x,PETSC_VIEWER_STDOUT_WORLD,ierr)
>       if (rank .eq. 0) write(6,200) n,rnorm
>
> 100  format('iteration ',i5,' solution vector:')
> 200  format('iteration ',i5,' residual norm ',e11.4)
>       ierr = 0
>       end
>
> ! --------------------------------------------------------------
> !
> !  MyKSPConverged - This is a user-defined routine for testing
> !  convergence of the KSP iterative solvers.
> !
> !  Input Parameters:
> !    ksp   - iterative context
> !    n     - iteration number
> !    rnorm - 2-norm (preconditioned) residual value (may be estimated)
> !    dummy - optional user-defined monitor context (unused here)
> !
>       subroutine MyKSPConverged(ksp,n,rnorm,flag,dummy,ierr)
>       use petscksp
>       implicit none
>
>       KSP              ksp
>       PetscErrorCode ierr
>       PetscInt n,dummy
>       KSPConvergedReason flag
>       PetscReal rnorm
>
>       if (rnorm .le. .05) then
>         flag = 1
>       else
>         flag = 0
>       endif
>       ierr = 0
>
>       end
>
> !/*TEST
> !
> !   test:
> !      nsize: 2
> !      args: -pc_type jacobi -ksp_monitor_short -ksp_gmres_cgs_refinement_type refine_always
> !
> !   test:
> !      suffix: 2
> !      nsize: 2
> !      args: -pc_type jacobi -my_ksp_monitor -ksp_gmres_cgs_refinement_type refine_always
> !
> !TEST*/
>
> Output:
> snarayanan1 at USLN38 /cygdrive/c/sidarth/petsc-test/scaling_test
> $ make ex2f
> /home/snarayanan1/petsc-release/lib/petsc/bin/win32fe/win32fe ifort -MT -O3 -fpp  -MT -O3 -fpp   -I/home/snarayanan1/petsc-release/include -I/home/snarayanan1/petsc-release/arch-ci-mswin-opt-impi/include -I/cygdrive/c/PROGRA~2/Intel/oneAPI/mpi/2021.5.0/include     ex2f.F90  -L/cygdrive/c/cygwin64/home/snarayanan1/petsc-release/arch-ci-mswin-opt-impi/lib -L/cygdrive/c/PROGRA~2/Intel/oneAPI/mkl/2022.0.0/lib/intel64 -lpetsc mkl_intel_lp64_dll.lib mkl_sequential_dll.lib mkl_core_dll.lib /cygdrive/c/PROGRA~2/Intel/oneAPI/mpi/2021.5.0/lib/release/impi.lib Gdi32.lib User32.lib Advapi32.lib Kernel32.lib Ws2_32.lib -o ex2f
>
> snarayanan1 at USLN38 /cygdrive/c/sidarth/petsc-test/scaling_test
> $ $PETSC_DIR/lib/petsc/bin/petscmpiexec -n 1 ex2f.exe
> Norm of error  0.1214E+02 iterations  2571
> KSPSolve time:    104.949
>
> snarayanan1 at USLN38 /cygdrive/c/sidarth/petsc-test/scaling_test
> $ $PETSC_DIR/lib/petsc/bin/petscmpiexec -n 2 ex2f.exe
> Norm of error  0.1221E+02 iterations  1821
> KSPSolve time:     47.139
>
> snarayanan1 at USLN38 /cygdrive/c/sidarth/petsc-test/scaling_test
> $ $PETSC_DIR/lib/petsc/bin/petscmpiexec -n 1 ex2f.exe -random_exact_sol
> Norm of error  0.8541E+02 iterations  1003
> KSPSolve time:     40.853
>
> snarayanan1 at USLN38 /cygdrive/c/sidarth/petsc-test/scaling_test
> $ $PETSC_DIR/lib/petsc/bin/petscmpiexec -n 2 ex2f.exe -random_exact_sol
> Norm of error  0.8562E+02 iterations  1191
> KSPSolve time:     31.745
>
>
> Thank You,
> Sidarth Narayanan
> Electronics Thermal Management Intern
> Altair Engineering Inc.
> Troy, MI

From patrick.alken at geomag.info  Tue Aug 16 12:45:52 2022
From: patrick.alken at geomag.info (Patrick Alken)
Date: Tue, 16 Aug 2022 11:45:52 -0600
Subject: [petsc-users] Parallel assembly of a matrix problem
In-Reply-To: <CADOhEh5PQ=G+xkkP9k11sHU7QQYWJhgMXkT4zvXM7u5-Oo7UKQ@mail.gmail.com>
References: <e8c23b6d-7c55-5a23-91c4-99cbb8356cc1@geomag.info>
	<CADOhEh5PQ=G+xkkP9k11sHU7QQYWJhgMXkT4zvXM7u5-Oo7UKQ@mail.gmail.com>
Message-ID: <d796357d-8bd6-15dc-cf5f-374c8c285495@geomag.info>

I have managed to simplify my example program, and removed the 
dependencies on slepc and gsl (attached).

The program defines a 800x800 matrix with 3 non-zero entries per row, 
which are all set to 1.0.

When I run the program on 1 processor, everything works correctly. When 
I run it on 2+ processors, I get the following error:

-----

first = 400 last = 800
[1]PETSC ERROR: --------------------- Error Message 
--------------------------------------------------------------
[1]PETSC ERROR: Argument out of range
[1]PETSC ERROR: New nonzero at (0,399) caused a malloc
Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to turn 
off this check
[1]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[1]PETSC ERROR: Petsc Release Version 3.17.4, unknown
[1]PETSC ERROR: Configure options --download-make --download-mpich 
--download-scalapack --download-cmake --download-mumps
[1]PETSC ERROR: #1 MatSetValues_MPIAIJ() at 
/data/palken/petsc/src/mat/impls/aij/mpi/mpiaij.c:506
[1]PETSC ERROR: #2 MatSetValues() at 
/data/palken/petsc/src/mat/interface/matrix.c:1343
first = 0 last = 400
[0]PETSC ERROR: --------------------- Error Message 
--------------------------------------------------------------
[0]PETSC ERROR: Argument out of range
[0]PETSC ERROR: New nonzero at (399,400) caused a malloc
Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to turn 
off this check
[0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.17.4, unknown
[0]PETSC ERROR: Configure options --download-make --download-mpich 
--download-scalapack --download-cmake --download-mumps
[0]PETSC ERROR: #1 MatSetValues_MPIAIJ() at 
/data/palken/petsc/src/mat/impls/aij/mpi/mpiaij.c:506
[0]PETSC ERROR: #2 MatSetValues() at 
/data/palken/petsc/src/mat/interface/matrix.c:1343

-----

As far as I understand, processor 1 will have rows 1-400, while 
processor 2 will have rows 401-800. From my reading of the 
documentation, each processor will have all columns (1-800) for their 
block of rows. So I am at a loss to understand why the program is 
generating these errors. Any help is appreciated.

Patrick

On 8/15/22 04:29, Mark Adams wrote:
> You might need to fix this:
>
> if (i - j < m)
> to
> if (i - j < m && i - j >= 0)
> Mark
>
> On Mon, Aug 15, 2022 at 12:12 AM Patrick Alken 
> <patrick.alken at geomag.info> wrote:
>
>     I am trying to assemble a matrix in parallel in Petsc, and it is
>     working
>     for 1 processor, but when I try running it with 2 processors I get
>     errors like the following:
>
>     ----
>
>     [1]PETSC ERROR: --------------------- Error Message
>     --------------------------------------------------------------
>     [1]PETSC ERROR: Argument out of range
>     [1]PETSC ERROR: New nonzero at (0,207) caused a malloc
>     Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE)
>     to turn
>     off this check
>     [1]PETSC ERROR: See https://petsc.org/release/faq/ for trouble
>     shooting.
>     [1]PETSC ERROR: Petsc Release Version 3.17.4, unknown
>     [1]PETSC ERROR: #1 MatSetValues_MPIAIJ() at
>     /data/palken/petsc/src/mat/impls/aij/mpi/mpiaij.c:506
>
>     ----
>
>     and then many MatSetValues errors follow.
>
>     I have attached a minimal working example and a Makefile. My code
>     does
>     depend on the GSL library also. I think there must be a problem in
>     how I
>     set up the d_nnz and o_nnz arrays for the MatMPIAIJSetPreallocation()
>     function (lines 211-235 of mwe.c).
>
>     I am building a matrix using radial basis functions with compact
>     support, so the matrix entries are zero when ||Xi - Xj|| >= c,
>     where c
>     is the RBF shape parameter. This is accounted for on line 224 and
>     254 of
>     the code. As far as I can tell I am consistent in the way I define
>     the
>     nnz arrays and the way I call MatSetValues, so I don't know why Petsc
>     would need to malloc additional memory during the assembly process.
>
>     Any pointers would be greatly appreciated.
>
>     Patrick
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220816/6b7e3a18/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mwe.c
Type: text/x-csrc
Size: 1738 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220816/6b7e3a18/attachment-0001.bin>
-------------- next part --------------
default: mwe

include ${PETSC_DIR}/lib/petsc/conf/variables

OBJECTS = mwe.o

mwe.o: mwe.c
	${CLINKER} -I${PETSC_DIR}/include -c $< -o $@

mwe: ${OBJECTS}
	${CLINKER} -o mwe ${OBJECTS} ${PETSC_KSP_LIB}

clean:
	rm -f mwe mwe.o

From bsmith at petsc.dev  Tue Aug 16 12:59:34 2022
From: bsmith at petsc.dev (Barry Smith)
Date: Tue, 16 Aug 2022 13:59:34 -0400
Subject: [petsc-users] Using matrix-free with KSP
In-Reply-To: <BY5PR02MB6471AB1C004920DA1FC9B43EE76B9@BY5PR02MB6471.namprd02.prod.outlook.com>
References: <BY5PR02MB6471969F80FA5F732FC74D87E7D79@BY5PR02MB6471.namprd02.prod.outlook.com>
	<B55CFC2C-6364-4B78-8E99-9D53700743B8@petsc.dev>
	<BY5PR02MB6471BDF90783FE3A0EA176CBE7809@BY5PR02MB6471.namprd02.prod.outlook.com>
	<87iloa49kt.fsf@jedbrown.org>
	<BY5PR02MB6471C0ED04769D02046F68ACE7809@BY5PR02MB6471.namprd02.prod.outlook.com>
	<00EBE4B8-22E5-408F-B56D-14226A278D7E@petsc.dev>
	<BY5PR02MB6471E386057DDDE98AD05DFAE7809@BY5PR02MB6471.namprd02.prod.outlook.com>
	<BY5PR02MB647140084C7E7CDDB6BC2A11E76B9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<A6629DE1-5A7F-414E-B164-A69877158675@petsc.dev>
	<BY5PR02MB6471AB1C004920DA1FC9B43EE76B9@BY5PR02MB6471.namprd02.prod.outlook.com>
Message-ID: <103146FD-FF28-47CE-B661-F2B521BF0E7A@petsc.dev>


   Create another matrix B that contains "Selected parts of the matrix can be pre-calculated and stored to serve as the preconditioner." When you call KSPSetOperators() pass in B as the second matrix argument.

   Now if literally application of B is the preconditioner you want to use, then simply use PCMAT as the preconditioner, and it will apply B each time it needs to apply the preconditioner. With this, you do not even need a PCShell. 

   If your preconditioner uses B to construct your own specific data structure needed to apply your preconditioner, then you can use PCShellSetSetUp() and have your routine call PCGetOperators() to pull out the B matrix and use it to construct your preconditioner. 

   If you want PETSc to construct a standard preconditioner from B, then you don't need the PCShell; you simply pass in the B, as above, and use -pc_type type to set the preconditioner you want to use. 

   Barry


> On Aug 16, 2022, at 12:31 PM, Tu, Jiannan <Jiannan_Tu at uml.edu> wrote:
> 
> Barry,
> 
> Thank you very much for your instructions. I am sorry I may not ask clearer questions.
> 
> I don't use SNES. What I am trying to solve is a very large linear equation system with dense and ill-conditioned matrix. Selected parts of the matrix can be pre-calculated and stored to serve as the preconditioner. My question is how I can apply this matrix as the preconditioner. 
> 
> Here is the part of code from my linear equations solver. MatrixFreePreconditioner is supposed to be the routine to provide the preconditioner. But I don't understand how to use it. Should I use SNESSetJacobian() even if this is a linear problem? 
> 
>     MatCreateMFFD(MPI_COMM_WORLD, PETSC_DECIDE, PETSC_DECIDE, N, N, &A);
>     MatMFFDSetFunction(A, formfunction, &params);
>     MatSetFromOptions(A);
>     MatMFFDSetBase(A, X, NULL);
> 
>     KSPCreate(MPI_COMM_WORLD, &ksp);
>     KSPSetOperators(ksp, A, A);
>     KSPSetFromOptions(ksp);
> 
>     PC pc;
>     KSPGetPC(ksp,&pc);
>     PCSetType(pc,PCSHELL);
>     PCShellSetApply(pc,MatrixFreePreconditioner);
> 
> Thank you,
> Jiannan
> 
> From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>>
> Sent: Tuesday, August 16, 2022 10:10 AM
> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
> Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>>
> Subject: Re: [petsc-users] Using matrix-free with KSP
>  
> CAUTION: This email was sent from outside the UMass Lowell network.
> 
> 
>   I don't fully understand your question so I will answer questions that I do know the answer to :-)
> 
>   The function you provide to PCShellSetApply() applies the preconditioner to the input vector and puts the result in the output vector.  This input vector changes at every application of the preconditioner so should not be used in constructing the preconditioner.
> 
>   If you are using the standard PETSc matrix-free matrix-vector product via finite difference, -snes_mf_operator or via MatCreateSNESMF() or you are using a MatShell, the matrix-vector product routines input changes for each multiply and should not be used in constructing the preconditioner.
> 
>   The function you provide with SNESSetJacobian() is where you can compute anything you need to help build your preconditioner. The input vector to that function is the location at which the Jacobian is needed and is what you should use to build your approximate/part of Jacobian. You can store your approximate/part of Jacobian that you need to compute in the pmat and then in a PCShellSetUp() function that you provide you will compute what you will need latter to apply the preconditioner in your PCShellSetApply() function. Note that the values you put in the pmat do not need to be the full Jacobian; they can be anything you want since this pmat is not used to apply the matrix vector product.
> 
>   If this is unclear, feel free to ask additional questions.
> 
>   Barry
> 
> 
>> On Aug 16, 2022, at 9:40 AM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> wrote:
>> 
>> I am trying to apply a user-defined preconditioner to speed up the convergency of matrix-free KSP solver. The user's manual provides an example of how to call the user-defined preconditioner routine. The routine has PC, input Vec and output Vec as arguments. I want to use part of the Jacobian as preconditioner matrix, the elements of which can be calculated using input Vec. My question is what the role of output Vec from the routine is in matrix-vector product? Or I just have the preconditioner matrix elements calculated in the routine and then Petsc handles applying the preconditioner to matrix-vector product? I used PCSetType(pc, PCSHELL) and PCShellSetApply.
>> 
>> Thank you and appreciate your help!
>> 
>> Jiannan 
>> From: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
>> Sent: Wednesday, July 6, 2022 2:34 PM
>> To: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>>
>> Cc: Jed Brown <jed at jedbrown.org <mailto:jed at jedbrown.org>>; petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>>
>> Subject: Re: [petsc-users] Using matrix-free with KSP
>>  
>> Barry,
>> 
>> VecScatterCreateToAll solves the problem! The code now gives the correct answer with multiple processes. 
>> 
>> Thank you all so much!
>> 
>> Jiannan
>> From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>>
>> Sent: Wednesday, July 6, 2022 2:08 PM
>> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
>> Cc: Jed Brown <jed at jedbrown.org <mailto:jed at jedbrown.org>>; petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>>
>> Subject: Re: [petsc-users] Using matrix-free with KSP
>>  
>> 
>>   So your operator is a "dense" operator in that its matrix representation would be essentially dense. In that case, you can use specialized routines to do this efficiently. You can use VecScatterCreateToAll() and then VecScatterBegin/End to do the communication.
>> 
>>   Barry
>> 
>> 
>> 
>>> On Jul 6, 2022, at 1:39 PM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> wrote:
>>> 
>>> Jed, thank you very much for the reply. 
>>> 
>>> I'm not sure GlobaltoLocal will work. Say the solution vector is of length 10. Two processes then each process can see 5 elements of the vector. For A x = b (10 equations), each process performs matrix-vector product for five equations. And for each equation i , sum_j A_ij *  x_j = b_i requires the process has access to all 10 unknows x_i (i=0,1,...,9). 
>>> 
>>> Can VecScatter and PetscSF make the entire vector accessible to each process? I am reading the manual and trying to understand how VecScatter and PetscSF work.
>>> 
>>> Jiannan
>>> From: Jed Brown <jed at jedbrown.org <mailto:jed at jedbrown.org>>
>>> Sent: Wednesday, July 6, 2022 10:09 AM
>>> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>; Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>>
>>> Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>>
>>> Subject: Re: [petsc-users] Using matrix-free with KSP
>>>  
>>> You'll usually have a GlobalToLocal operation for each rank to get the halo data it needs, then either a LocalToGlobal to collect the result (e.g., finite element methods) or the local compute will write directly into the owned portion of the global vector. If you're doing the communication "raw", then you may find VecScatter or PetscSF useful to perform the necessary communication.
>>> 
>>> "Tu, Jiannan" <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> writes:
>>> 
>>> > Hi Barry,
>>> >
>>> > Following your instructions I implemented the matrix-free method to solve a large linear equation system resulted from a surface integration equation. The KSP solver works fine for single process, but it is problematic with multiple processes. The problem is that each process only can access its own part of solution vector so that each process only conducts part of matrix-vector multiplication. MPI can be used to assemble these partial matrix-vector multiplication together.  Does Petsc provide any ways to implement multi-process matrix-free method?
>>> >
>>> > Thanks,
>>> > Jiannan
>>> > ________________________________
>>> > From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>>
>>> > Sent: Tuesday, May 24, 2022 2:12 PM
>>> > To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
>>> > Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>>
>>> > Subject: Re: [petsc-users] Using matrix-free with KSP
>>> >
>>> > This e-mail originated from outside the UMass Lowell network.
>>> > ________________________________
>>> >
>>> >    You can use MatCreateMFFD https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMatCreateMFFD%2F&amp;data=05%7C01%7CJiannan_Tu%40uml.edu%7Cab0859d8d154471590bc08da5f5918d6%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637927133525745809%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=A7wqegTfh94No5BpDiWLK3VxOuR44U2wlWHVm2k7l60%3D&amp;reserved=0<https://urldefense.com/v3/__https://petsc.org/main/docs/manualpages/Mat/MatCreateMFFD/__;!!PVKG_VDCxu5g!stxq2NWGVjUihvHBQC9Rlk0aHVoowcy0PTQYMcQxQ4DqOEC05KSw6TmstSXKckiUgelHzy4ue-d10-zlXkw$> <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMatCreateMFFD%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7C9cf9b4c72f7c4f0cd92a08da7f9120f4%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637962558562603998%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=BdPTAQcdmpnMYPbWECCabNQSzJfqpiH5G7iq1d%2FlY4k%3D&reserved=0> MatMFFDSetFunction MatSetFromOptions MatMFFDSetBase and provide the matrix to KSP. Note you will need to use -pc_type none or provide your own custom preconditioner withhttps://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FPC%2FPCSHELL%2F&amp;data=05%7C01%7CJiannan_Tu%40uml.edu%7Cab0859d8d154471590bc08da5f5918d6%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637927133525745809%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=Gb%2F9uFG3jp%2FbfWaSh2cSEQLItHS9CuLwarzN0KcNiJY%3D&amp;reserved=0<https://urldefense.com/v3/__https://petsc.org/main/docs/manualpages/PC/PCSHELL/__;!!PVKG_VDCxu5g!stxq2NWGVjUihvHBQC9Rlk0aHVoowcy0PTQYMcQxQ4DqOEC05KSw6TmstSXKckiUgelHzy4ue-d1cA4fKGw$> <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FPC%2FPCSHELL%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7C9cf9b4c72f7c4f0cd92a08da7f9120f4%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637962558562603998%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=6XJ3yp6S8IbTevj%2BeUOHDjgCbeTq346j%2Fini%2BLGmkag%3D&reserved=0>
>>> >
>>> >
>>> >
>>> > On May 24, 2022, at 1:21 PM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu><mailto:Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>> wrote:
>>> >
>>> > I want to use a matrix-free matrix to solve a large linear equation system because the matrix is too large to be stored. Petsc user manual describes matrix-free method for SNES with examples. The matrix-free matrices section explains how to set up such a matrix, but I can't find any example of matrix-free method with KSP. I am wondering how to apply the method to KSP iterative solver in parallel.
>>> >
>>> > I greatly appreciate your help for me to understand this topic.
>>> >
>>> > Jiannan Tu

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220816/c1492a26/attachment-0001.html>

From bsmith at petsc.dev  Tue Aug 16 13:03:50 2022
From: bsmith at petsc.dev (Barry Smith)
Date: Tue, 16 Aug 2022 14:03:50 -0400
Subject: [petsc-users] Parallel assembly of a matrix problem
In-Reply-To: <d796357d-8bd6-15dc-cf5f-374c8c285495@geomag.info>
References: <e8c23b6d-7c55-5a23-91c4-99cbb8356cc1@geomag.info>
	<CADOhEh5PQ=G+xkkP9k11sHU7QQYWJhgMXkT4zvXM7u5-Oo7UKQ@mail.gmail.com>
	<d796357d-8bd6-15dc-cf5f-374c8c285495@geomag.info>
Message-ID: <71E7A139-7F61-4022-B97D-23EBED9F43EE@petsc.dev>


MatMPIAIJSetPreallocation(A, 3, NULL, 0, NULL);

The 0 indicates you expect that there will be no matrix entries in the columns of the matrix that represent degrees of freedom that are not stored on the given rank. In your case this means you expect the matrix to be block diagonal with one block per MPI rank. You need to put an appropriate value instead of 1. If you matrix is Tri-diagonal then using 1 will be enough. But if your matrix has a more complicated structure (but with nonzeros on the diagonal) then you need to use 2.

  Barry


> On Aug 16, 2022, at 1:45 PM, Patrick Alken <patrick.alken at geomag.info> wrote:
> 
> I have managed to simplify my example program, and removed the dependencies on slepc and gsl (attached). 
> 
> The program defines a 800x800 matrix with 3 non-zero entries per row, which are all set to 1.0.
> 
> When I run the program on 1 processor, everything works correctly. When I run it on 2+ processors, I get the following error:
> 
> -----
> 
> first = 400 last = 800 
> [1]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- 
> [1]PETSC ERROR: Argument out of range 
> [1]PETSC ERROR: New nonzero at (0,399) caused a malloc 
> Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to turn off this check 
> [1]PETSC ERROR: See https://petsc.org/release/faq/ <https://petsc.org/release/faq/> for trouble shooting. 
> [1]PETSC ERROR: Petsc Release Version 3.17.4, unknown  
> [1]PETSC ERROR: Configure options --download-make --download-mpich --download-scalapack --download-cmake --download-mumps 
> [1]PETSC ERROR: #1 MatSetValues_MPIAIJ() at /data/palken/petsc/src/mat/impls/aij/mpi/mpiaij.c:506 
> [1]PETSC ERROR: #2 MatSetValues() at /data/palken/petsc/src/mat/interface/matrix.c:1343 
> first = 0 last = 400 
> [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- 
> [0]PETSC ERROR: Argument out of range 
> [0]PETSC ERROR: New nonzero at (399,400) caused a malloc 
> Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to turn off this check 
> [0]PETSC ERROR: See https://petsc.org/release/faq/ <https://petsc.org/release/faq/> for trouble shooting. 
> [0]PETSC ERROR: Petsc Release Version 3.17.4, unknown  
> [0]PETSC ERROR: Configure options --download-make --download-mpich --download-scalapack --download-cmake --download-mumps 
> [0]PETSC ERROR: #1 MatSetValues_MPIAIJ() at /data/palken/petsc/src/mat/impls/aij/mpi/mpiaij.c:506 
> [0]PETSC ERROR: #2 MatSetValues() at /data/palken/petsc/src/mat/interface/matrix.c:1343
> 
> -----
> 
> As far as I understand, processor 1 will have rows 1-400, while processor 2 will have rows 401-800. From my reading of the documentation, each processor will have all columns (1-800) for their block of rows. So I am at a loss to understand why the program is generating these errors. Any help is appreciated.
> 
> Patrick
> 
> On 8/15/22 04:29, Mark Adams wrote:
>> You might need to fix this:
>> 
>> if (i - j < m)
>> to 
>> if (i - j < m && i - j >= 0)
>> Mark
>> 
>> On Mon, Aug 15, 2022 at 12:12 AM Patrick Alken <patrick.alken at geomag.info <mailto:patrick.alken at geomag.info>> wrote:
>> I am trying to assemble a matrix in parallel in Petsc, and it is working 
>> for 1 processor, but when I try running it with 2 processors I get 
>> errors like the following:
>> 
>> ----
>> 
>> [1]PETSC ERROR: --------------------- Error Message 
>> --------------------------------------------------------------
>> [1]PETSC ERROR: Argument out of range
>> [1]PETSC ERROR: New nonzero at (0,207) caused a malloc
>> Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to turn 
>> off this check
>> [1]PETSC ERROR: See https://petsc.org/release/faq/ <https://petsc.org/release/faq/> for trouble shooting.
>> [1]PETSC ERROR: Petsc Release Version 3.17.4, unknown
>> [1]PETSC ERROR: #1 MatSetValues_MPIAIJ() at 
>> /data/palken/petsc/src/mat/impls/aij/mpi/mpiaij.c:506
>> 
>> ----
>> 
>> and then many MatSetValues errors follow.
>> 
>> I have attached a minimal working example and a Makefile. My code does 
>> depend on the GSL library also. I think there must be a problem in how I 
>> set up the d_nnz and o_nnz arrays for the MatMPIAIJSetPreallocation() 
>> function (lines 211-235 of mwe.c).
>> 
>> I am building a matrix using radial basis functions with compact 
>> support, so the matrix entries are zero when ||Xi - Xj|| >= c, where c 
>> is the RBF shape parameter. This is accounted for on line 224 and 254 of 
>> the code. As far as I can tell I am consistent in the way I define the 
>> nnz arrays and the way I call MatSetValues, so I don't know why Petsc 
>> would need to malloc additional memory during the assembly process.
>> 
>> Any pointers would be greatly appreciated.
>> 
>> Patrick
> <mwe.c><Makefile.txt>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220816/8778558a/attachment.html>

From Jiannan_Tu at uml.edu  Tue Aug 16 13:32:26 2022
From: Jiannan_Tu at uml.edu (Tu, Jiannan)
Date: Tue, 16 Aug 2022 18:32:26 +0000
Subject: [petsc-users] Using matrix-free with KSP
In-Reply-To: <103146FD-FF28-47CE-B661-F2B521BF0E7A@petsc.dev>
References: <BY5PR02MB6471969F80FA5F732FC74D87E7D79@BY5PR02MB6471.namprd02.prod.outlook.com>
	<B55CFC2C-6364-4B78-8E99-9D53700743B8@petsc.dev>
	<BY5PR02MB6471BDF90783FE3A0EA176CBE7809@BY5PR02MB6471.namprd02.prod.outlook.com>
	<87iloa49kt.fsf@jedbrown.org>
	<BY5PR02MB6471C0ED04769D02046F68ACE7809@BY5PR02MB6471.namprd02.prod.outlook.com>
	<00EBE4B8-22E5-408F-B56D-14226A278D7E@petsc.dev>
	<BY5PR02MB6471E386057DDDE98AD05DFAE7809@BY5PR02MB6471.namprd02.prod.outlook.com>
	<BY5PR02MB647140084C7E7CDDB6BC2A11E76B9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<A6629DE1-5A7F-414E-B164-A69877158675@petsc.dev>
	<BY5PR02MB6471AB1C004920DA1FC9B43EE76B9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<103146FD-FF28-47CE-B661-F2B521BF0E7A@petsc.dev>
Message-ID: <BY5PR02MB6471F93C494BC75FFB173F00E76B9@BY5PR02MB6471.namprd02.prod.outlook.com>

Barry,

Thank you so much. I'll try these options and see how they work.

Jiannan
________________________________
From: Barry Smith <bsmith at petsc.dev>
Sent: Tuesday, August 16, 2022 1:59 PM
To: Tu, Jiannan <Jiannan_Tu at uml.edu>
Cc: petsc-users at mcs.anl.gov <petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] Using matrix-free with KSP


CAUTION: This email was sent from outside the UMass Lowell network.


   Create another matrix B that contains "Selected parts of the matrix can be pre-calculated and stored to serve as the preconditioner." When you call KSPSetOperators() pass in B as the second matrix argument.

   Now if literally application of B is the preconditioner you want to use, then simply use PCMAT as the preconditioner, and it will apply B each time it needs to apply the preconditioner. With this, you do not even need a PCShell.

   If your preconditioner uses B to construct your own specific data structure needed to apply your preconditioner, then you can use PCShellSetSetUp() and have your routine call PCGetOperators() to pull out the B matrix and use it to construct your preconditioner.

   If you want PETSc to construct a standard preconditioner from B, then you don't need the PCShell; you simply pass in the B, as above, and use -pc_type type to set the preconditioner you want to use.

   Barry


On Aug 16, 2022, at 12:31 PM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

Barry,

Thank you very much for your instructions. I am sorry I may not ask clearer questions.

I don't use SNES. What I am trying to solve is a very large linear equation system with dense and ill-conditioned matrix. Selected parts of the matrix can be pre-calculated and stored to serve as the preconditioner. My question is how I can apply this matrix as the preconditioner.

Here is the part of code from my linear equations solver. MatrixFreePreconditioner is supposed to be the routine to provide the preconditioner. But I don't understand how to use it. Should I use SNESSetJacobian() even if this is a linear problem?

    MatCreateMFFD(MPI_COMM_WORLD, PETSC_DECIDE, PETSC_DECIDE, N, N, &A);
    MatMFFDSetFunction(A, formfunction, &params);
    MatSetFromOptions(A);
    MatMFFDSetBase(A, X, NULL);

    KSPCreate(MPI_COMM_WORLD, &ksp);
    KSPSetOperators(ksp, A, A);
    KSPSetFromOptions(ksp);

    PC pc;
    KSPGetPC(ksp,&pc);
    PCSetType(pc,PCSHELL);
    PCShellSetApply(pc,MatrixFreePreconditioner);

Thank you,
Jiannan

________________________________
From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Tuesday, August 16, 2022 10:10 AM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] Using matrix-free with KSP


CAUTION: This email was sent from outside the UMass Lowell network.


  I don't fully understand your question so I will answer questions that I do know the answer to :-)

  The function you provide to PCShellSetApply() applies the preconditioner to the input vector and puts the result in the output vector.  This input vector changes at every application of the preconditioner so should not be used in constructing the preconditioner.

  If you are using the standard PETSc matrix-free matrix-vector product via finite difference, -snes_mf_operator or via MatCreateSNESMF() or you are using a MatShell, the matrix-vector product routines input changes for each multiply and should not be used in constructing the preconditioner.

  The function you provide with SNESSetJacobian() is where you can compute anything you need to help build your preconditioner. The input vector to that function is the location at which the Jacobian is needed and is what you should use to build your approximate/part of Jacobian. You can store your approximate/part of Jacobian that you need to compute in the pmat and then in a PCShellSetUp() function that you provide you will compute what you will need latter to apply the preconditioner in your PCShellSetApply() function. Note that the values you put in the pmat do not need to be the full Jacobian; they can be anything you want since this pmat is not used to apply the matrix vector product.

  If this is unclear, feel free to ask additional questions.

  Barry


On Aug 16, 2022, at 9:40 AM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

I am trying to apply a user-defined preconditioner to speed up the convergency of matrix-free KSP solver. The user's manual provides an example of how to call the user-defined preconditioner routine. The routine has PC, input Vec and output Vec as arguments. I want to use part of the Jacobian as preconditioner matrix, the elements of which can be calculated using input Vec. My question is what the role of output Vec from the routine is in matrix-vector product? Or I just have the preconditioner matrix elements calculated in the routine and then Petsc handles applying the preconditioner to matrix-vector product? I used PCSetType(pc, PCSHELL) and PCShellSetApply.

Thank you and appreciate your help!

Jiannan
________________________________
From: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Sent: Wednesday, July 6, 2022 2:34 PM
To: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Cc: Jed Brown <jed at jedbrown.org<mailto:jed at jedbrown.org>>; petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] Using matrix-free with KSP

Barry,

VecScatterCreateToAll solves the problem! The code now gives the correct answer with multiple processes.

Thank you all so much!

Jiannan
________________________________
From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Wednesday, July 6, 2022 2:08 PM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Cc: Jed Brown <jed at jedbrown.org<mailto:jed at jedbrown.org>>; petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] Using matrix-free with KSP


  So your operator is a "dense" operator in that its matrix representation would be essentially dense. In that case, you can use specialized routines to do this efficiently. You can use VecScatterCreateToAll() and then VecScatterBegin/End to do the communication.

  Barry



On Jul 6, 2022, at 1:39 PM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

Jed, thank you very much for the reply.

I'm not sure GlobaltoLocal will work. Say the solution vector is of length 10. Two processes then each process can see 5 elements of the vector. For A x = b (10 equations), each process performs matrix-vector product for five equations. And for each equation i , sum_j A_ij *  x_j = b_i requires the process has access to all 10 unknows x_i (i=0,1,...,9).

Can VecScatter and PetscSF make the entire vector accessible to each process? I am reading the manual and trying to understand how VecScatter and PetscSF work.

Jiannan
________________________________
From: Jed Brown <jed at jedbrown.org<mailto:jed at jedbrown.org>>
Sent: Wednesday, July 6, 2022 10:09 AM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>; Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] Using matrix-free with KSP

You'll usually have a GlobalToLocal operation for each rank to get the halo data it needs, then either a LocalToGlobal to collect the result (e.g., finite element methods) or the local compute will write directly into the owned portion of the global vector. If you're doing the communication "raw", then you may find VecScatter or PetscSF useful to perform the necessary communication.

"Tu, Jiannan" <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> writes:

> Hi Barry,
>
> Following your instructions I implemented the matrix-free method to solve a large linear equation system resulted from a surface integration equation. The KSP solver works fine for single process, but it is problematic with multiple processes. The problem is that each process only can access its own part of solution vector so that each process only conducts part of matrix-vector multiplication. MPI can be used to assemble these partial matrix-vector multiplication together.  Does Petsc provide any ways to implement multi-process matrix-free method?
>
> Thanks,
> Jiannan
> ________________________________
> From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
> Sent: Tuesday, May 24, 2022 2:12 PM
> To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
> Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
> Subject: Re: [petsc-users] Using matrix-free with KSP
>
> This e-mail originated from outside the UMass Lowell network.
> ________________________________
>
>    You can use MatCreateMFFD https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMatCreateMFFD%2F&amp;data=05%7C01%7CJiannan_Tu%40uml.edu%7Cab0859d8d154471590bc08da5f5918d6%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637927133525745809%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=A7wqegTfh94No5BpDiWLK3VxOuR44U2wlWHVm2k7l60%3D&amp;reserved=0<https://urldefense.com/v3/__https://petsc.org/main/docs/manualpages/Mat/MatCreateMFFD/__;!!PVKG_VDCxu5g!stxq2NWGVjUihvHBQC9Rlk0aHVoowcy0PTQYMcQxQ4DqOEC05KSw6TmstSXKckiUgelHzy4ue-d10-zlXkw$><https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMatCreateMFFD%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7C454a1eb8de0745be453508da7fb114e4%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637962695800558712%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=%2FausyQFMwKFHd4FnNaKAOCGwskUc%2F8jIlSxtWMdIjC4%3D&reserved=0> MatMFFDSetFunction MatSetFromOptions MatMFFDSetBase and provide the matrix to KSP. Note you will need to use -pc_type none or provide your own custom preconditioner withhttps://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FPC%2FPCSHELL%2F&amp;data=05%7C01%7CJiannan_Tu%40uml.edu%7Cab0859d8d154471590bc08da5f5918d6%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637927133525745809%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=Gb%2F9uFG3jp%2FbfWaSh2cSEQLItHS9CuLwarzN0KcNiJY%3D&amp;reserved=0<https://urldefense.com/v3/__https://petsc.org/main/docs/manualpages/PC/PCSHELL/__;!!PVKG_VDCxu5g!stxq2NWGVjUihvHBQC9Rlk0aHVoowcy0PTQYMcQxQ4DqOEC05KSw6TmstSXKckiUgelHzy4ue-d1cA4fKGw$><https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FPC%2FPCSHELL%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7C454a1eb8de0745be453508da7fb114e4%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637962695800558712%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=%2F5%2BLn2AdyPMZYfBS11Dd87S5anXtl3QPPNEst%2BjDGE4%3D&reserved=0>
>
>
>
> On May 24, 2022, at 1:21 PM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu><mailto:Jiannan_Tu at uml.edu>> wrote:
>
> I want to use a matrix-free matrix to solve a large linear equation system because the matrix is too large to be stored. Petsc user manual describes matrix-free method for SNES with examples. The matrix-free matrices section explains how to set up such a matrix, but I can't find any example of matrix-free method with KSP. I am wondering how to apply the method to KSP iterative solver in parallel.
>
> I greatly appreciate your help for me to understand this topic.
>
> Jiannan Tu

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220816/c544ae08/attachment-0001.html>

From narnoldm at umich.edu  Wed Aug 17 10:30:30 2022
From: narnoldm at umich.edu (Nicholas Arnold-Medabalimi)
Date: Wed, 17 Aug 2022 11:30:30 -0400
Subject: [petsc-users] DMPlex Transitive closure
Message-ID: <CAB1dcV6ZYs1mD3L2We7FVukix86Qq5L43Th9SGcKAz5STZGSyQ@mail.gmail.com>

Good Morning

I have been working with the transitive closure to check to see if a node
point has a cell that is rooted on a process. However, I think I am having
a more fundamental issue. Even just calling the RestoreTransitiveClosure
immediately after getting it is throwing a PetscError.(that is removing all
the calls between the two closure functions. I'm not exactly sure what is
causing the issue. I have used this method previously, except on cell
points to get the constituent vertex points. Is there some nuance to using
the supports instead of the cones?



DMPlexGetDepthStratum(dm, 0, &pStart, &pEnd);
PetscPrintf(PETSC_COMM_WORLD, "Pstart: %d, Pend: %d\n", pStart, pEnd);
for (int p = pStart; p < pEnd; p++)
    {

        PetscInt closureSize;
        PetscInt *closure;

        ierr = DMPlexGetTransitiveClosure(dm, p, PETSC_FALSE, &closureSize,
&closure);
        CHKERRQ(ierr);
        PetscSynchronizedPrintf(PETSC_COMM_WORLD, "rank:%d\n", rank);
        for (int i = 0; i < (2 * closureSize); i = i + 2)
        {
            PetscSynchronizedPrintf(PETSC_COMM_WORLD, "%d\n", closure[i]);
        }
        PetscSynchronizedFlush(PETSC_COMM_WORLD, NULL);
        ierr = DMPlexRestoreTransitiveClosure(dm, p, PETSC_FALSE, &
closureSize, &closure);
        CHKERRQ(ierr);
    }

Results in an error when the RestoreTransitiveClosure is called. I have
checked and the resulting output for the closures are correct.

Pstart: 510, Pend: 1532
rank:0
510
1534
1535
0
rank:1
492
1478
1479
0
[1]PETSC ERROR: 2953
--------------------- Error Message
--------------------------------------------------------------
[1]PETSC ERROR: 491
Object is in wrong state
[1]PETSC ERROR: Array was not checked out
[1]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[1]PETSC ERROR: Petsc Release Version 3.17.4, Aug 01, 2022
[1]PETSC ERROR: /home/narnoldm/code/solver-playground/build/bin/pgrid_c on
a  by narnoldm Wed Aug 17 11:18:20 2022
[0]PETSC ERROR: [1]PETSC ERROR: Configure options
--prefix=/home/narnoldm/code/solver-playground/build/external/PETSc
--with-hdf5-dir=/home/narnoldm/code/solver-playground/build/external/HDF5
--with-cgns-dir=/home/narnoldm/code/solver-playground/build/external/CGNS
--download-triangle --download-parmetis --download-metis --with-debugging=1

Any assistance is greatly appreciated.

Sincerely
Nicholas

-- 
Nicholas Arnold-Medabalimi

Ph.D. Candidate
Computational Aeroscience Lab
University of Michigan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220817/bd4902fb/attachment.html>

From Jiannan_Tu at uml.edu  Wed Aug 17 23:50:58 2022
From: Jiannan_Tu at uml.edu (Tu, Jiannan)
Date: Thu, 18 Aug 2022 04:50:58 +0000
Subject: [petsc-users] Using matrix-free with KSP
In-Reply-To: <103146FD-FF28-47CE-B661-F2B521BF0E7A@petsc.dev>
References: <BY5PR02MB6471969F80FA5F732FC74D87E7D79@BY5PR02MB6471.namprd02.prod.outlook.com>
	<B55CFC2C-6364-4B78-8E99-9D53700743B8@petsc.dev>
	<BY5PR02MB6471BDF90783FE3A0EA176CBE7809@BY5PR02MB6471.namprd02.prod.outlook.com>
	<87iloa49kt.fsf@jedbrown.org>
	<BY5PR02MB6471C0ED04769D02046F68ACE7809@BY5PR02MB6471.namprd02.prod.outlook.com>
	<00EBE4B8-22E5-408F-B56D-14226A278D7E@petsc.dev>
	<BY5PR02MB6471E386057DDDE98AD05DFAE7809@BY5PR02MB6471.namprd02.prod.outlook.com>
	<BY5PR02MB647140084C7E7CDDB6BC2A11E76B9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<A6629DE1-5A7F-414E-B164-A69877158675@petsc.dev>
	<BY5PR02MB6471AB1C004920DA1FC9B43EE76B9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<103146FD-FF28-47CE-B661-F2B521BF0E7A@petsc.dev>
Message-ID: <DM6PR02MB6478B2FB8CB1BE757CE7B003E76D9@DM6PR02MB6478.namprd02.prod.outlook.com>

I implemented the preconditioner. The solver converges quickly when the problem size is small. But when the size increases, say to 100k unknowns, the code hangs at assembly of the preconditioner matrix. The function call sequence is like this

   MatCreate(MPI_COMM_WORLD, &B);
    MatSetType(B, MATMPIAIJ);
    MatSetSizes(B, PETSC_DECIDE, PETSC_DECIDE, N, N);
    MatSetFromOptions(B);

    // The number of diagonal and off diagonal non zeros only can be determined run-time.
    MatMPIAIJSetPreallocation(B, 0, d_nnz, 0, o_nnz);
    MatSetOption(B, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE);

    //here insert values row by row.
MatSetValues(B, 1, &m, nn, nCols, matCols, INSERT_VALUES);

MatAssemblyBegin(B, MAT_FINAL_ASSEMBLY);
MatAssemblyEnd(B, MAT_FINAL_ASSEMBLY);

Could you please tell me what I have done wrong?

Thank you,
Jiannan

From: Barry Smith <bsmith at petsc.dev>
Sent: Tuesday, August 16, 2022 2:00 PM
To: Tu, Jiannan <Jiannan_Tu at uml.edu>
Cc: petsc-users at mcs.anl.gov
Subject: Re: [petsc-users] Using matrix-free with KSP

CAUTION: This email was sent from outside the UMass Lowell network.


   Create another matrix B that contains "Selected parts of the matrix can be pre-calculated and stored to serve as the preconditioner." When you call KSPSetOperators() pass in B as the second matrix argument.

   Now if literally application of B is the preconditioner you want to use, then simply use PCMAT as the preconditioner, and it will apply B each time it needs to apply the preconditioner. With this, you do not even need a PCShell.

   If your preconditioner uses B to construct your own specific data structure needed to apply your preconditioner, then you can use PCShellSetSetUp() and have your routine call PCGetOperators() to pull out the B matrix and use it to construct your preconditioner.

   If you want PETSc to construct a standard preconditioner from B, then you don't need the PCShell; you simply pass in the B, as above, and use -pc_type type to set the preconditioner you want to use.

   Barry



On Aug 16, 2022, at 12:31 PM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

Barry,

Thank you very much for your instructions. I am sorry I may not ask clearer questions.

I don't use SNES. What I am trying to solve is a very large linear equation system with dense and ill-conditioned matrix. Selected parts of the matrix can be pre-calculated and stored to serve as the preconditioner. My question is how I can apply this matrix as the preconditioner.

Here is the part of code from my linear equations solver. MatrixFreePreconditioner is supposed to be the routine to provide the preconditioner. But I don't understand how to use it. Should I use SNESSetJacobian() even if this is a linear problem?

    MatCreateMFFD(MPI_COMM_WORLD, PETSC_DECIDE, PETSC_DECIDE, N, N, &A);
    MatMFFDSetFunction(A, formfunction, &params);
    MatSetFromOptions(A);
    MatMFFDSetBase(A, X, NULL);

    KSPCreate(MPI_COMM_WORLD, &ksp);
    KSPSetOperators(ksp, A, A);
    KSPSetFromOptions(ksp);

    PC pc;
    KSPGetPC(ksp,&pc);
    PCSetType(pc,PCSHELL);
    PCShellSetApply(pc,MatrixFreePreconditioner);

Thank you,
Jiannan

________________________________
From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Tuesday, August 16, 2022 10:10 AM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] Using matrix-free with KSP

CAUTION: This email was sent from outside the UMass Lowell network.


  I don't fully understand your question so I will answer questions that I do know the answer to :-)

  The function you provide to PCShellSetApply() applies the preconditioner to the input vector and puts the result in the output vector.  This input vector changes at every application of the preconditioner so should not be used in constructing the preconditioner.

  If you are using the standard PETSc matrix-free matrix-vector product via finite difference, -snes_mf_operator or via MatCreateSNESMF() or you are using a MatShell, the matrix-vector product routines input changes for each multiply and should not be used in constructing the preconditioner.

  The function you provide with SNESSetJacobian() is where you can compute anything you need to help build your preconditioner. The input vector to that function is the location at which the Jacobian is needed and is what you should use to build your approximate/part of Jacobian. You can store your approximate/part of Jacobian that you need to compute in the pmat and then in a PCShellSetUp() function that you provide you will compute what you will need latter to apply the preconditioner in your PCShellSetApply() function. Note that the values you put in the pmat do not need to be the full Jacobian; they can be anything you want since this pmat is not used to apply the matrix vector product.

  If this is unclear, feel free to ask additional questions.

  Barry



On Aug 16, 2022, at 9:40 AM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

I am trying to apply a user-defined preconditioner to speed up the convergency of matrix-free KSP solver. The user's manual provides an example of how to call the user-defined preconditioner routine. The routine has PC, input Vec and output Vec as arguments. I want to use part of the Jacobian as preconditioner matrix, the elements of which can be calculated using input Vec. My question is what the role of output Vec from the routine is in matrix-vector product? Or I just have the preconditioner matrix elements calculated in the routine and then Petsc handles applying the preconditioner to matrix-vector product? I used PCSetType(pc, PCSHELL) and PCShellSetApply.

Thank you and appreciate your help!

Jiannan
________________________________
From: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Sent: Wednesday, July 6, 2022 2:34 PM
To: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Cc: Jed Brown <jed at jedbrown.org<mailto:jed at jedbrown.org>>; petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] Using matrix-free with KSP

Barry,

VecScatterCreateToAll solves the problem! The code now gives the correct answer with multiple processes.

Thank you all so much!

Jiannan
________________________________
From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Wednesday, July 6, 2022 2:08 PM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Cc: Jed Brown <jed at jedbrown.org<mailto:jed at jedbrown.org>>; petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] Using matrix-free with KSP


  So your operator is a "dense" operator in that its matrix representation would be essentially dense. In that case, you can use specialized routines to do this efficiently. You can use VecScatterCreateToAll() and then VecScatterBegin/End to do the communication.

  Barry




On Jul 6, 2022, at 1:39 PM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

Jed, thank you very much for the reply.

I'm not sure GlobaltoLocal will work. Say the solution vector is of length 10. Two processes then each process can see 5 elements of the vector. For A x = b (10 equations), each process performs matrix-vector product for five equations. And for each equation i , sum_j A_ij *  x_j = b_i requires the process has access to all 10 unknows x_i (i=0,1,...,9).

Can VecScatter and PetscSF make the entire vector accessible to each process? I am reading the manual and trying to understand how VecScatter and PetscSF work.

Jiannan
________________________________
From: Jed Brown <jed at jedbrown.org<mailto:jed at jedbrown.org>>
Sent: Wednesday, July 6, 2022 10:09 AM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>; Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] Using matrix-free with KSP

You'll usually have a GlobalToLocal operation for each rank to get the halo data it needs, then either a LocalToGlobal to collect the result (e.g., finite element methods) or the local compute will write directly into the owned portion of the global vector. If you're doing the communication "raw", then you may find VecScatter or PetscSF useful to perform the necessary communication.

"Tu, Jiannan" <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> writes:

> Hi Barry,
>
> Following your instructions I implemented the matrix-free method to solve a large linear equation system resulted from a surface integration equation. The KSP solver works fine for single process, but it is problematic with multiple processes. The problem is that each process only can access its own part of solution vector so that each process only conducts part of matrix-vector multiplication. MPI can be used to assemble these partial matrix-vector multiplication together.  Does Petsc provide any ways to implement multi-process matrix-free method?
>
> Thanks,
> Jiannan
> ________________________________
> From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
> Sent: Tuesday, May 24, 2022 2:12 PM
> To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
> Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
> Subject: Re: [petsc-users] Using matrix-free with KSP
>
> This e-mail originated from outside the UMass Lowell network.
> ________________________________
>
>    You can use MatCreateMFFD https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMatCreateMFFD%2F&amp;data=05%7C01%7CJiannan_Tu%40uml.edu%7Cab0859d8d154471590bc08da5f5918d6%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637927133525745809%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=A7wqegTfh94No5BpDiWLK3VxOuR44U2wlWHVm2k7l60%3D&amp;reserved=0<https://urldefense.com/v3/__https://petsc.org/main/docs/manualpages/Mat/MatCreateMFFD/__;!!PVKG_VDCxu5g!stxq2NWGVjUihvHBQC9Rlk0aHVoowcy0PTQYMcQxQ4DqOEC05KSw6TmstSXKckiUgelHzy4ue-d10-zlXkw$><https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMatCreateMFFD%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7C454a1eb8de0745be453508da7fb114e4%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637962695800558712%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=%2FausyQFMwKFHd4FnNaKAOCGwskUc%2F8jIlSxtWMdIjC4%3D&reserved=0> MatMFFDSetFunction MatSetFromOptions MatMFFDSetBase and provide the matrix to KSP. Note you will need to use -pc_type none or provide your own custom preconditioner withhttps://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FPC%2FPCSHELL%2F&amp;data=05%7C01%7CJiannan_Tu%40uml.edu%7Cab0859d8d154471590bc08da5f5918d6%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637927133525745809%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=Gb%2F9uFG3jp%2FbfWaSh2cSEQLItHS9CuLwarzN0KcNiJY%3D&amp;reserved=0<https://urldefense.com/v3/__https://petsc.org/main/docs/manualpages/PC/PCSHELL/__;!!PVKG_VDCxu5g!stxq2NWGVjUihvHBQC9Rlk0aHVoowcy0PTQYMcQxQ4DqOEC05KSw6TmstSXKckiUgelHzy4ue-d1cA4fKGw$><https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FPC%2FPCSHELL%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7C454a1eb8de0745be453508da7fb114e4%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637962695800558712%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=%2F5%2BLn2AdyPMZYfBS11Dd87S5anXtl3QPPNEst%2BjDGE4%3D&reserved=0>
>
>
>
> On May 24, 2022, at 1:21 PM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu><mailto:Jiannan_Tu at uml.edu>> wrote:
>
> I want to use a matrix-free matrix to solve a large linear equation system because the matrix is too large to be stored. Petsc user manual describes matrix-free method for SNES with examples. The matrix-free matrices section explains how to set up such a matrix, but I can't find any example of matrix-free method with KSP. I am wondering how to apply the method to KSP iterative solver in parallel.
>
> I greatly appreciate your help for me to understand this topic.
>
> Jiannan Tu

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220818/0bded64b/attachment-0001.html>

From bsmith at petsc.dev  Thu Aug 18 07:35:03 2022
From: bsmith at petsc.dev (Barry Smith)
Date: Thu, 18 Aug 2022 08:35:03 -0400
Subject: [petsc-users] Using matrix-free with KSP
In-Reply-To: <DM6PR02MB6478B2FB8CB1BE757CE7B003E76D9@DM6PR02MB6478.namprd02.prod.outlook.com>
References: <BY5PR02MB6471969F80FA5F732FC74D87E7D79@BY5PR02MB6471.namprd02.prod.outlook.com>
	<B55CFC2C-6364-4B78-8E99-9D53700743B8@petsc.dev>
	<BY5PR02MB6471BDF90783FE3A0EA176CBE7809@BY5PR02MB6471.namprd02.prod.outlook.com>
	<87iloa49kt.fsf@jedbrown.org>
	<BY5PR02MB6471C0ED04769D02046F68ACE7809@BY5PR02MB6471.namprd02.prod.outlook.com>
	<00EBE4B8-22E5-408F-B56D-14226A278D7E@petsc.dev>
	<BY5PR02MB6471E386057DDDE98AD05DFAE7809@BY5PR02MB6471.namprd02.prod.outlook.com>
	<BY5PR02MB647140084C7E7CDDB6BC2A11E76B9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<A6629DE1-5A7F-414E-B164-A69877158675@petsc.dev>
	<BY5PR02MB6471AB1C004920DA1FC9B43EE76B9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<103146FD-FF28-47CE-B661-F2B521BF0E7A@petsc.dev>
	<DM6PR02MB6478B2FB8CB1BE757CE7B003E76D9@DM6PR02MB6478.namprd02.prod.outlook.com>
Message-ID: <4C00F4B8-BB42-4932-BCD7-527E5FB39BFF@petsc.dev>


  Slow assembly is usually due to under preallocation. You can change to using 

> MatSetOption(B, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_TRUE);

  to detect if you are under preallocating. See https://petsc.org/main/docs/manual/mat/#sec-matsparse <https://petsc.org/main/docs/manual/mat/#sec-matsparse>. That section clearly needs some sprucing up. If it is difficult to determine good preallocation values you can use https://petsc.org/main/docs/manualpages/Mat/MATPREALLOCATOR/ <https://petsc.org/main/docs/manualpages/Mat/MATPREALLOCATOR/> to compute the needed preallocation efficiently.

  Are you running in parallel? If so how are you determining which rows of entries to compute on each MPI rank? You will want most of the rows to be computed on the rank where the values are stored, you can determine this with https://petsc.org/main/docs/manualpages/Mat/MatGetOwnershipRange.html <https://petsc.org/main/docs/manualpages/Mat/MatGetOwnershipRange.html>

  Barry





> On Aug 18, 2022, at 12:50 AM, Tu, Jiannan <Jiannan_Tu at uml.edu> wrote:
> 
> I implemented the preconditioner. The solver converges quickly when the problem size is small. But when the size increases, say to 100k unknowns, the code hangs at assembly of the preconditioner matrix. The function call sequence is like this
>  
>    MatCreate(MPI_COMM_WORLD, &B);
>     MatSetType(B, MATMPIAIJ);
>     MatSetSizes(B, PETSC_DECIDE, PETSC_DECIDE, N, N);
>     MatSetFromOptions(B);
>  
>     // The number of diagonal and off diagonal non zeros only can be determined run-time.
>     MatMPIAIJSetPreallocation(B, 0, d_nnz, 0, o_nnz);
>     MatSetOption(B, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE);
>  
>     //here insert values row by row.
> MatSetValues(B, 1, &m, nn, nCols, matCols, INSERT_VALUES);
>  
> MatAssemblyBegin(B, MAT_FINAL_ASSEMBLY);
> MatAssemblyEnd(B, MAT_FINAL_ASSEMBLY);
>  
> Could you please tell me what I have done wrong?
>  
> Thank you,
> Jiannan
>    
> From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>> 
> Sent: Tuesday, August 16, 2022 2:00 PM
> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
> Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>
> Subject: Re: [petsc-users] Using matrix-free with KSP
>  
> CAUTION: This email was sent from outside the UMass Lowell network.
>  
>  
>    Create another matrix B that contains "Selected parts of the matrix can be pre-calculated and stored to serve as the preconditioner." When you call KSPSetOperators() pass in B as the second matrix argument.
>  
>    Now if literally application of B is the preconditioner you want to use, then simply use PCMAT as the preconditioner, and it will apply B each time it needs to apply the preconditioner. With this, you do not even need a PCShell. 
>  
>    If your preconditioner uses B to construct your own specific data structure needed to apply your preconditioner, then you can use PCShellSetSetUp() and have your routine call PCGetOperators() to pull out the B matrix and use it to construct your preconditioner. 
>  
>    If you want PETSc to construct a standard preconditioner from B, then you don't need the PCShell; you simply pass in the B, as above, and use -pc_type type to set the preconditioner you want to use. 
>  
>    Barry
>  
> 
> 
> On Aug 16, 2022, at 12:31 PM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> wrote:
>  
> Barry,
>  
> Thank you very much for your instructions. I am sorry I may not ask clearer questions.
>  
> I don't use SNES. What I am trying to solve is a very large linear equation system with dense and ill-conditioned matrix. Selected parts of the matrix can be pre-calculated and stored to serve as the preconditioner. My question is how I can apply this matrix as the preconditioner. 
>  
> Here is the part of code from my linear equations solver. MatrixFreePreconditioner is supposed to be the routine to provide the preconditioner. But I don't understand how to use it. Should I use SNESSetJacobian() even if this is a linear problem? 
>  
>     MatCreateMFFD(MPI_COMM_WORLD, PETSC_DECIDE, PETSC_DECIDE, N, N, &A);
>     MatMFFDSetFunction(A, formfunction, &params);
>     MatSetFromOptions(A);
>     MatMFFDSetBase(A, X, NULL);
>  
>     KSPCreate(MPI_COMM_WORLD, &ksp);
>     KSPSetOperators(ksp, A, A);
>     KSPSetFromOptions(ksp);
>  
>     PC pc;
>     KSPGetPC(ksp,&pc);
>     PCSetType(pc,PCSHELL);
>     PCShellSetApply(pc,MatrixFreePreconditioner);
>  
> Thank you,
> Jiannan
>  
> From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>>
> Sent: Tuesday, August 16, 2022 10:10 AM
> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
> Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>>
> Subject: Re: [petsc-users] Using matrix-free with KSP
>  
> CAUTION: This email was sent from outside the UMass Lowell network.
>  
>  
>   I don't fully understand your question so I will answer questions that I do know the answer to :-)
>  
>   The function you provide to PCShellSetApply() applies the preconditioner to the input vector and puts the result in the output vector.  This input vector changes at every application of the preconditioner so should not be used in constructing the preconditioner.
>  
>   If you are using the standard PETSc matrix-free matrix-vector product via finite difference, -snes_mf_operator or via MatCreateSNESMF() or you are using a MatShell, the matrix-vector product routines input changes for each multiply and should not be used in constructing the preconditioner.
>  
>   The function you provide with SNESSetJacobian() is where you can compute anything you need to help build your preconditioner. The input vector to that function is the location at which the Jacobian is needed and is what you should use to build your approximate/part of Jacobian. You can store your approximate/part of Jacobian that you need to compute in the pmat and then in a PCShellSetUp() function that you provide you will compute what you will need latter to apply the preconditioner in your PCShellSetApply() function. Note that the values you put in the pmat do not need to be the full Jacobian; they can be anything you want since this pmat is not used to apply the matrix vector product.
>  
>   If this is unclear, feel free to ask additional questions.
>  
>   Barry
>  
> 
> 
> On Aug 16, 2022, at 9:40 AM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> wrote:
>  
> I am trying to apply a user-defined preconditioner to speed up the convergency of matrix-free KSP solver. The user's manual provides an example of how to call the user-defined preconditioner routine. The routine has PC, input Vec and output Vec as arguments. I want to use part of the Jacobian as preconditioner matrix, the elements of which can be calculated using input Vec. My question is what the role of output Vec from the routine is in matrix-vector product? Or I just have the preconditioner matrix elements calculated in the routine and then Petsc handles applying the preconditioner to matrix-vector product? I used PCSetType(pc, PCSHELL) and PCShellSetApply.
>  
> Thank you and appreciate your help!
>  
> Jiannan 
> From: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
> Sent: Wednesday, July 6, 2022 2:34 PM
> To: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>>
> Cc: Jed Brown <jed at jedbrown.org <mailto:jed at jedbrown.org>>; petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>>
> Subject: Re: [petsc-users] Using matrix-free with KSP
>  
> Barry,
>  
> VecScatterCreateToAll solves the problem! The code now gives the correct answer with multiple processes. 
>  
> Thank you all so much!
>  
> Jiannan
> From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>>
> Sent: Wednesday, July 6, 2022 2:08 PM
> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
> Cc: Jed Brown <jed at jedbrown.org <mailto:jed at jedbrown.org>>; petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>>
> Subject: Re: [petsc-users] Using matrix-free with KSP
>  
>  
>   So your operator is a "dense" operator in that its matrix representation would be essentially dense. In that case, you can use specialized routines to do this efficiently. You can use VecScatterCreateToAll() and then VecScatterBegin/End to do the communication.
>  
>   Barry
>  
>  
> 
> 
> On Jul 6, 2022, at 1:39 PM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> wrote:
>  
> Jed, thank you very much for the reply. 
>  
> I'm not sure GlobaltoLocal will work. Say the solution vector is of length 10. Two processes then each process can see 5 elements of the vector. For A x = b (10 equations), each process performs matrix-vector product for five equations. And for each equation i , sum_j A_ij *  x_j = b_i requires the process has access to all 10 unknows x_i (i=0,1,...,9). 
>  
> Can VecScatter and PetscSF make the entire vector accessible to each process? I am reading the manual and trying to understand how VecScatter and PetscSF work.
>  
> Jiannan
> From: Jed Brown <jed at jedbrown.org <mailto:jed at jedbrown.org>>
> Sent: Wednesday, July 6, 2022 10:09 AM
> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>; Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>>
> Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>>
> Subject: Re: [petsc-users] Using matrix-free with KSP
>  
> You'll usually have a GlobalToLocal operation for each rank to get the halo data it needs, then either a LocalToGlobal to collect the result (e.g., finite element methods) or the local compute will write directly into the owned portion of the global vector. If you're doing the communication "raw", then you may find VecScatter or PetscSF useful to perform the necessary communication.
> 
> "Tu, Jiannan" <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> writes:
> 
> > Hi Barry,
> >
> > Following your instructions I implemented the matrix-free method to solve a large linear equation system resulted from a surface integration equation. The KSP solver works fine for single process, but it is problematic with multiple processes. The problem is that each process only can access its own part of solution vector so that each process only conducts part of matrix-vector multiplication. MPI can be used to assemble these partial matrix-vector multiplication together.  Does Petsc provide any ways to implement multi-process matrix-free method?
> >
> > Thanks,
> > Jiannan
> > ________________________________
> > From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>>
> > Sent: Tuesday, May 24, 2022 2:12 PM
> > To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
> > Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>>
> > Subject: Re: [petsc-users] Using matrix-free with KSP
> >
> > This e-mail originated from outside the UMass Lowell network.
> > ________________________________
> >
> >    You can use MatCreateMFFD https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMatCreateMFFD%2F&amp;data=05%7C01%7CJiannan_Tu%40uml.edu%7Cab0859d8d154471590bc08da5f5918d6%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637927133525745809%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=A7wqegTfh94No5BpDiWLK3VxOuR44U2wlWHVm2k7l60%3D&amp;reserved=0<https://urldefense.com/v3/__https://petsc.org/main/docs/manualpages/Mat/MatCreateMFFD/__;!!PVKG_VDCxu5g!stxq2NWGVjUihvHBQC9Rlk0aHVoowcy0PTQYMcQxQ4DqOEC05KSw6TmstSXKckiUgelHzy4ue-d10-zlXkw$> <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMatCreateMFFD%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7C454a1eb8de0745be453508da7fb114e4%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637962695800558712%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=%2FausyQFMwKFHd4FnNaKAOCGwskUc%2F8jIlSxtWMdIjC4%3D&reserved=0> MatMFFDSetFunction MatSetFromOptions MatMFFDSetBase and provide the matrix to KSP. Note you will need to use -pc_type none or provide your own custom preconditioner withhttps://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FPC%2FPCSHELL%2F&amp;data=05%7C01%7CJiannan_Tu%40uml.edu%7Cab0859d8d154471590bc08da5f5918d6%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637927133525745809%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=Gb%2F9uFG3jp%2FbfWaSh2cSEQLItHS9CuLwarzN0KcNiJY%3D&amp;reserved=0<https://urldefense.com/v3/__https://petsc.org/main/docs/manualpages/PC/PCSHELL/__;!!PVKG_VDCxu5g!stxq2NWGVjUihvHBQC9Rlk0aHVoowcy0PTQYMcQxQ4DqOEC05KSw6TmstSXKckiUgelHzy4ue-d1cA4fKGw$> <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FPC%2FPCSHELL%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7C454a1eb8de0745be453508da7fb114e4%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637962695800558712%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=%2F5%2BLn2AdyPMZYfBS11Dd87S5anXtl3QPPNEst%2BjDGE4%3D&reserved=0>
> >
> >
> >
> > On May 24, 2022, at 1:21 PM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu><mailto:Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>> wrote:
> >
> > I want to use a matrix-free matrix to solve a large linear equation system because the matrix is too large to be stored. Petsc user manual describes matrix-free method for SNES with examples. The matrix-free matrices section explains how to set up such a matrix, but I can't find any example of matrix-free method with KSP. I am wondering how to apply the method to KSP iterative solver in parallel.
> >
> > I greatly appreciate your help for me to understand this topic.
> >
> > Jiannan Tu

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220818/219649d1/attachment-0001.html>

From Jiannan_Tu at uml.edu  Thu Aug 18 10:30:39 2022
From: Jiannan_Tu at uml.edu (Tu, Jiannan)
Date: Thu, 18 Aug 2022 15:30:39 +0000
Subject: [petsc-users] Using matrix-free with KSP
In-Reply-To: <4C00F4B8-BB42-4932-BCD7-527E5FB39BFF@petsc.dev>
References: <BY5PR02MB6471969F80FA5F732FC74D87E7D79@BY5PR02MB6471.namprd02.prod.outlook.com>
	<B55CFC2C-6364-4B78-8E99-9D53700743B8@petsc.dev>
	<BY5PR02MB6471BDF90783FE3A0EA176CBE7809@BY5PR02MB6471.namprd02.prod.outlook.com>
	<87iloa49kt.fsf@jedbrown.org>
	<BY5PR02MB6471C0ED04769D02046F68ACE7809@BY5PR02MB6471.namprd02.prod.outlook.com>
	<00EBE4B8-22E5-408F-B56D-14226A278D7E@petsc.dev>
	<BY5PR02MB6471E386057DDDE98AD05DFAE7809@BY5PR02MB6471.namprd02.prod.outlook.com>
	<BY5PR02MB647140084C7E7CDDB6BC2A11E76B9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<A6629DE1-5A7F-414E-B164-A69877158675@petsc.dev>
	<BY5PR02MB6471AB1C004920DA1FC9B43EE76B9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<103146FD-FF28-47CE-B661-F2B521BF0E7A@petsc.dev>
	<DM6PR02MB6478B2FB8CB1BE757CE7B003E76D9@DM6PR02MB6478.namprd02.prod.outlook.com>
	<4C00F4B8-BB42-4932-BCD7-527E5FB39BFF@petsc.dev>
Message-ID: <BY5PR02MB64710E70758E431108D66956E76D9@BY5PR02MB6471.namprd02.prod.outlook.com>

Hi Barry,

The MATPREALLOCATOR solved problem of slow matrix assembly. But the solver failed because of "Matrix type preallocator does not have a multiply defined".

I guess this is because the matrix-free is used. I am wondering how a preconditioner is applied to the matrix-vector multiply in Petsc.

Thank you,
Jiannan

[0]PETSC ERROR: No support for this operation for this object type
[0]PETSC ERROR: Matrix type preallocator does not have a multiply defined
[0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.16.6, Mar 30, 2022
[0]PETSC ERROR: ./iesolver on a  named REXNET-WS4 by jiannantu Thu Aug 18 11:21:55 2022
[0]PETSC ERROR: Configure options --download-f2cblaslapack=yes --with-mpi-dir=/usr/local --with-fc=0 --prefix=/home/jiannantu/petsc --with-scalar-type=complex --with-64-bit-indices=1
[0]PETSC ERROR: #1 MatMult() at /home/jiannantu/petsc-3.16.6/src/mat/interface/matrix.c:2437
[0]PETSC ERROR: #2 PCApply_Mat() at /home/jiannantu/petsc-3.16.6/src/ksp/pc/impls/mat/pcmat.c:9
[0]PETSC ERROR: #3 PCApply() at /home/jiannantu/petsc-3.16.6/src/ksp/pc/interface/precon.c:445
[0]PETSC ERROR: #4 KSP_PCApply() at /home/jiannantu/petsc-3.16.6/include/petsc/private/kspimpl.h:382
[0]PETSC ERROR: #5 KSPInitialResidual() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itres.c:65
[0]PETSC ERROR: #6 KSPSolve_BCGS() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/impls/bcgs/bcgs.c:43
[0]PETSC ERROR: #7 KSPSolve_Private() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itfunc.c:914
[0]PETSC ERROR: #8 KSPSolve() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itfunc.c:1086

From: Barry Smith <bsmith at petsc.dev>
Sent: Thursday, August 18, 2022 8:35 AM
To: Tu, Jiannan <Jiannan_Tu at uml.edu>
Cc: petsc-users at mcs.anl.gov
Subject: Re: [petsc-users] Using matrix-free with KSP

CAUTION: This email was sent from outside the UMass Lowell network.


  Slow assembly is usually due to under preallocation. You can change to using

MatSetOption(B, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_TRUE);

  to detect if you are under preallocating. See https://petsc.org/main/docs/manual/mat/#sec-matsparse<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanual%2Fmat%2F%23sec-matsparse&data=05%7C01%7CJiannan_Tu%40uml.edu%7C14cbcc56f6304ae2ada708da811614b4%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637964229096924381%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=O7ArEs5EQFfQyu%2F3o0ohxPtKAjD7waJHJMj5Y0nAr3k%3D&reserved=0>. That section clearly needs some sprucing up. If it is difficult to determine good preallocation values you can use https://petsc.org/main/docs/manualpages/Mat/MATPREALLOCATOR/<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMATPREALLOCATOR%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7C14cbcc56f6304ae2ada708da811614b4%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637964229096924381%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=VdrClFQzWc6uWprRsn%2FwaSYpEhCoQBZJHzJT2btkh%2Fs%3D&reserved=0> to compute the needed preallocation efficiently.

  Are you running in parallel? If so how are you determining which rows of entries to compute on each MPI rank? You will want most of the rows to be computed on the rank where the values are stored, you can determine this with https://petsc.org/main/docs/manualpages/Mat/MatGetOwnershipRange.html<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMatGetOwnershipRange.html&data=05%7C01%7CJiannan_Tu%40uml.edu%7C14cbcc56f6304ae2ada708da811614b4%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637964229096924381%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=UwV2PapoVnLaqsE5plKYwjZ9oCHUvVwW%2B5ArwkXM5uQ%3D&reserved=0>

  Barry







On Aug 18, 2022, at 12:50 AM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

I implemented the preconditioner. The solver converges quickly when the problem size is small. But when the size increases, say to 100k unknowns, the code hangs at assembly of the preconditioner matrix. The function call sequence is like this

   MatCreate(MPI_COMM_WORLD, &B);
    MatSetType(B, MATMPIAIJ);
    MatSetSizes(B, PETSC_DECIDE, PETSC_DECIDE, N, N);
    MatSetFromOptions(B);

    // The number of diagonal and off diagonal non zeros only can be determined run-time.
    MatMPIAIJSetPreallocation(B, 0, d_nnz, 0, o_nnz);
    MatSetOption(B, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE);

    //here insert values row by row.
MatSetValues(B, 1, &m, nn, nCols, matCols, INSERT_VALUES);

MatAssemblyBegin(B, MAT_FINAL_ASSEMBLY);
MatAssemblyEnd(B, MAT_FINAL_ASSEMBLY);

Could you please tell me what I have done wrong?

Thank you,
Jiannan

From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Tuesday, August 16, 2022 2:00 PM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] Using matrix-free with KSP

CAUTION: This email was sent from outside the UMass Lowell network.


   Create another matrix B that contains "Selected parts of the matrix can be pre-calculated and stored to serve as the preconditioner." When you call KSPSetOperators() pass in B as the second matrix argument.

   Now if literally application of B is the preconditioner you want to use, then simply use PCMAT as the preconditioner, and it will apply B each time it needs to apply the preconditioner. With this, you do not even need a PCShell.

   If your preconditioner uses B to construct your own specific data structure needed to apply your preconditioner, then you can use PCShellSetSetUp() and have your routine call PCGetOperators() to pull out the B matrix and use it to construct your preconditioner.

   If you want PETSc to construct a standard preconditioner from B, then you don't need the PCShell; you simply pass in the B, as above, and use -pc_type type to set the preconditioner you want to use.

   Barry




On Aug 16, 2022, at 12:31 PM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

Barry,

Thank you very much for your instructions. I am sorry I may not ask clearer questions.

I don't use SNES. What I am trying to solve is a very large linear equation system with dense and ill-conditioned matrix. Selected parts of the matrix can be pre-calculated and stored to serve as the preconditioner. My question is how I can apply this matrix as the preconditioner.

Here is the part of code from my linear equations solver. MatrixFreePreconditioner is supposed to be the routine to provide the preconditioner. But I don't understand how to use it. Should I use SNESSetJacobian() even if this is a linear problem?

    MatCreateMFFD(MPI_COMM_WORLD, PETSC_DECIDE, PETSC_DECIDE, N, N, &A);
    MatMFFDSetFunction(A, formfunction, &params);
    MatSetFromOptions(A);
    MatMFFDSetBase(A, X, NULL);

    KSPCreate(MPI_COMM_WORLD, &ksp);
    KSPSetOperators(ksp, A, A);
    KSPSetFromOptions(ksp);

    PC pc;
    KSPGetPC(ksp,&pc);
    PCSetType(pc,PCSHELL);
    PCShellSetApply(pc,MatrixFreePreconditioner);

Thank you,
Jiannan

________________________________
From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Tuesday, August 16, 2022 10:10 AM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] Using matrix-free with KSP

CAUTION: This email was sent from outside the UMass Lowell network.


  I don't fully understand your question so I will answer questions that I do know the answer to :-)

  The function you provide to PCShellSetApply() applies the preconditioner to the input vector and puts the result in the output vector.  This input vector changes at every application of the preconditioner so should not be used in constructing the preconditioner.

  If you are using the standard PETSc matrix-free matrix-vector product via finite difference, -snes_mf_operator or via MatCreateSNESMF() or you are using a MatShell, the matrix-vector product routines input changes for each multiply and should not be used in constructing the preconditioner.

  The function you provide with SNESSetJacobian() is where you can compute anything you need to help build your preconditioner. The input vector to that function is the location at which the Jacobian is needed and is what you should use to build your approximate/part of Jacobian. You can store your approximate/part of Jacobian that you need to compute in the pmat and then in a PCShellSetUp() function that you provide you will compute what you will need latter to apply the preconditioner in your PCShellSetApply() function. Note that the values you put in the pmat do not need to be the full Jacobian; they can be anything you want since this pmat is not used to apply the matrix vector product.

  If this is unclear, feel free to ask additional questions.

  Barry




On Aug 16, 2022, at 9:40 AM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

I am trying to apply a user-defined preconditioner to speed up the convergency of matrix-free KSP solver. The user's manual provides an example of how to call the user-defined preconditioner routine. The routine has PC, input Vec and output Vec as arguments. I want to use part of the Jacobian as preconditioner matrix, the elements of which can be calculated using input Vec. My question is what the role of output Vec from the routine is in matrix-vector product? Or I just have the preconditioner matrix elements calculated in the routine and then Petsc handles applying the preconditioner to matrix-vector product? I used PCSetType(pc, PCSHELL) and PCShellSetApply.

Thank you and appreciate your help!

Jiannan
________________________________
From: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Sent: Wednesday, July 6, 2022 2:34 PM
To: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Cc: Jed Brown <jed at jedbrown.org<mailto:jed at jedbrown.org>>; petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] Using matrix-free with KSP

Barry,

VecScatterCreateToAll solves the problem! The code now gives the correct answer with multiple processes.

Thank you all so much!

Jiannan
________________________________
From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Wednesday, July 6, 2022 2:08 PM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Cc: Jed Brown <jed at jedbrown.org<mailto:jed at jedbrown.org>>; petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] Using matrix-free with KSP


  So your operator is a "dense" operator in that its matrix representation would be essentially dense. In that case, you can use specialized routines to do this efficiently. You can use VecScatterCreateToAll() and then VecScatterBegin/End to do the communication.

  Barry





On Jul 6, 2022, at 1:39 PM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

Jed, thank you very much for the reply.

I'm not sure GlobaltoLocal will work. Say the solution vector is of length 10. Two processes then each process can see 5 elements of the vector. For A x = b (10 equations), each process performs matrix-vector product for five equations. And for each equation i , sum_j A_ij *  x_j = b_i requires the process has access to all 10 unknows x_i (i=0,1,...,9).

Can VecScatter and PetscSF make the entire vector accessible to each process? I am reading the manual and trying to understand how VecScatter and PetscSF work.

Jiannan
________________________________
From: Jed Brown <jed at jedbrown.org<mailto:jed at jedbrown.org>>
Sent: Wednesday, July 6, 2022 10:09 AM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>; Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] Using matrix-free with KSP

You'll usually have a GlobalToLocal operation for each rank to get the halo data it needs, then either a LocalToGlobal to collect the result (e.g., finite element methods) or the local compute will write directly into the owned portion of the global vector. If you're doing the communication "raw", then you may find VecScatter or PetscSF useful to perform the necessary communication.

"Tu, Jiannan" <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> writes:

> Hi Barry,
>
> Following your instructions I implemented the matrix-free method to solve a large linear equation system resulted from a surface integration equation. The KSP solver works fine for single process, but it is problematic with multiple processes. The problem is that each process only can access its own part of solution vector so that each process only conducts part of matrix-vector multiplication. MPI can be used to assemble these partial matrix-vector multiplication together.  Does Petsc provide any ways to implement multi-process matrix-free method?
>
> Thanks,
> Jiannan
> ________________________________
> From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
> Sent: Tuesday, May 24, 2022 2:12 PM
> To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
> Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
> Subject: Re: [petsc-users] Using matrix-free with KSP
>
> This e-mail originated from outside the UMass Lowell network.
> ________________________________
>
>    You can use MatCreateMFFD https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMatCreateMFFD%2F&amp;data=05%7C01%7CJiannan_Tu%40uml.edu%7Cab0859d8d154471590bc08da5f5918d6%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637927133525745809%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=A7wqegTfh94No5BpDiWLK3VxOuR44U2wlWHVm2k7l60%3D&amp;reserved=0<https://urldefense.com/v3/__https://petsc.org/main/docs/manualpages/Mat/MatCreateMFFD/__;!!PVKG_VDCxu5g!stxq2NWGVjUihvHBQC9Rlk0aHVoowcy0PTQYMcQxQ4DqOEC05KSw6TmstSXKckiUgelHzy4ue-d10-zlXkw$><https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMatCreateMFFD%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7C14cbcc56f6304ae2ada708da811614b4%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637964229096924381%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=IJGp1oTI1GnuKoKqchT0r5pC2sfO1F9Os867kP8i%2BzA%3D&reserved=0> MatMFFDSetFunction MatSetFromOptions MatMFFDSetBase and provide the matrix to KSP. Note you will need to use -pc_type none or provide your own custom preconditioner withhttps://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FPC%2FPCSHELL%2F&amp;data=05%7C01%7CJiannan_Tu%40uml.edu%7Cab0859d8d154471590bc08da5f5918d6%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637927133525745809%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=Gb%2F9uFG3jp%2FbfWaSh2cSEQLItHS9CuLwarzN0KcNiJY%3D&amp;reserved=0<https://urldefense.com/v3/__https://petsc.org/main/docs/manualpages/PC/PCSHELL/__;!!PVKG_VDCxu5g!stxq2NWGVjUihvHBQC9Rlk0aHVoowcy0PTQYMcQxQ4DqOEC05KSw6TmstSXKckiUgelHzy4ue-d1cA4fKGw$><https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FPC%2FPCSHELL%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7C14cbcc56f6304ae2ada708da811614b4%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637964229096924381%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=%2Ben584fnKhucOyFOS9C28HP%2BS8AiaYo0GRrWoFuSNus%3D&reserved=0>
>
>
>
> On May 24, 2022, at 1:21 PM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu><mailto:Jiannan_Tu at uml.edu>> wrote:
>
> I want to use a matrix-free matrix to solve a large linear equation system because the matrix is too large to be stored. Petsc user manual describes matrix-free method for SNES with examples. The matrix-free matrices section explains how to set up such a matrix, but I can't find any example of matrix-free method with KSP. I am wondering how to apply the method to KSP iterative solver in parallel.
>
> I greatly appreciate your help for me to understand this topic.
>
> Jiannan Tu

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220818/98d75cd8/attachment-0001.html>

From bsmith at petsc.dev  Thu Aug 18 10:37:12 2022
From: bsmith at petsc.dev (Barry Smith)
Date: Thu, 18 Aug 2022 11:37:12 -0400
Subject: [petsc-users] Using matrix-free with KSP
In-Reply-To: <BY5PR02MB64710E70758E431108D66956E76D9@BY5PR02MB6471.namprd02.prod.outlook.com>
References: <BY5PR02MB6471969F80FA5F732FC74D87E7D79@BY5PR02MB6471.namprd02.prod.outlook.com>
	<B55CFC2C-6364-4B78-8E99-9D53700743B8@petsc.dev>
	<BY5PR02MB6471BDF90783FE3A0EA176CBE7809@BY5PR02MB6471.namprd02.prod.outlook.com>
	<87iloa49kt.fsf@jedbrown.org>
	<BY5PR02MB6471C0ED04769D02046F68ACE7809@BY5PR02MB6471.namprd02.prod.outlook.com>
	<00EBE4B8-22E5-408F-B56D-14226A278D7E@petsc.dev>
	<BY5PR02MB6471E386057DDDE98AD05DFAE7809@BY5PR02MB6471.namprd02.prod.outlook.com>
	<BY5PR02MB647140084C7E7CDDB6BC2A11E76B9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<A6629DE1-5A7F-414E-B164-A69877158675@petsc.dev>
	<BY5PR02MB6471AB1C004920DA1FC9B43EE76B9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<103146FD-FF28-47CE-B661-F2B521BF0E7A@petsc.dev>
	<DM6PR02MB6478B2FB8CB1BE757CE7B003E76D9@DM6PR02MB6478.namprd02.prod.outlook.com>
	<4C00F4B8-BB42-4932-BCD7-527E5FB39BFF@petsc.dev>
	<BY5PR02MB64710E70758E431108D66956E76D9@BY5PR02MB6471.namprd02.prod.outlook.com>
Message-ID: <D0EB13D8-C5A6-4AE7-AFDE-2CDCCACBAF73@petsc.dev>


   The preallocator MatType matrix cannot be passed to the KSPSetOperators(), you need to create an actual matrix, for example MATAIJ and use the preallocator to set its preallocation and then fill up the MATAIJ matrix with the usual MatSetValues and pass that matrix to the KSPSetOperators.

> On Aug 18, 2022, at 11:30 AM, Tu, Jiannan <Jiannan_Tu at uml.edu> wrote:
> 
> Hi Barry,
>  
> The MATPREALLOCATOR solved problem of slow matrix assembly. But the solver failed because of ?Matrix type preallocator does not have a multiply defined?.
>  
> I guess this is because the matrix-free is used. I am wondering how a preconditioner is applied to the matrix-vector multiply in Petsc.
>  
> Thank you,
> Jiannan
>  
> [0]PETSC ERROR: No support for this operation for this object type
> [0]PETSC ERROR: Matrix type preallocator does not have a multiply defined
> [0]PETSC ERROR: See https://petsc.org/release/faq/ <https://petsc.org/release/faq/> for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.16.6, Mar 30, 2022
> [0]PETSC ERROR: ./iesolver on a  named REXNET-WS4 by jiannantu Thu Aug 18 11:21:55 2022
> [0]PETSC ERROR: Configure options --download-f2cblaslapack=yes --with-mpi-dir=/usr/local --with-fc=0 --prefix=/home/jiannantu/petsc --with-scalar-type=complex --with-64-bit-indices=1
> [0]PETSC ERROR: #1 MatMult() at /home/jiannantu/petsc-3.16.6/src/mat/interface/matrix.c:2437
> [0]PETSC ERROR: #2 PCApply_Mat() at /home/jiannantu/petsc-3.16.6/src/ksp/pc/impls/mat/pcmat.c:9
> [0]PETSC ERROR: #3 PCApply() at /home/jiannantu/petsc-3.16.6/src/ksp/pc/interface/precon.c:445
> [0]PETSC ERROR: #4 KSP_PCApply() at /home/jiannantu/petsc-3.16.6/include/petsc/private/kspimpl.h:382
> [0]PETSC ERROR: #5 KSPInitialResidual() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itres.c:65
> [0]PETSC ERROR: #6 KSPSolve_BCGS() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/impls/bcgs/bcgs.c:43
> [0]PETSC ERROR: #7 KSPSolve_Private() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itfunc.c:914
> [0]PETSC ERROR: #8 KSPSolve() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itfunc.c:1086
>  
> From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>> 
> Sent: Thursday, August 18, 2022 8:35 AM
> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
> Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>
> Subject: Re: [petsc-users] Using matrix-free with KSP
>  
> CAUTION: This email was sent from outside the UMass Lowell network.
>  
>  
>   Slow assembly is usually due to under preallocation. You can change to using 
>  
> MatSetOption(B, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_TRUE);
>  
>   to detect if you are under preallocating. See https://petsc.org/main/docs/manual/mat/#sec-matsparse <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanual%2Fmat%2F%23sec-matsparse&data=05%7C01%7CJiannan_Tu%40uml.edu%7C14cbcc56f6304ae2ada708da811614b4%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637964229096924381%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=O7ArEs5EQFfQyu%2F3o0ohxPtKAjD7waJHJMj5Y0nAr3k%3D&reserved=0>. That section clearly needs some sprucing up. If it is difficult to determine good preallocation values you can use https://petsc.org/main/docs/manualpages/Mat/MATPREALLOCATOR/ <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMATPREALLOCATOR%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7C14cbcc56f6304ae2ada708da811614b4%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637964229096924381%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=VdrClFQzWc6uWprRsn%2FwaSYpEhCoQBZJHzJT2btkh%2Fs%3D&reserved=0> to compute the needed preallocation efficiently.
>  
>   Are you running in parallel? If so how are you determining which rows of entries to compute on each MPI rank? You will want most of the rows to be computed on the rank where the values are stored, you can determine this with https://petsc.org/main/docs/manualpages/Mat/MatGetOwnershipRange.html <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMatGetOwnershipRange.html&data=05%7C01%7CJiannan_Tu%40uml.edu%7C14cbcc56f6304ae2ada708da811614b4%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637964229096924381%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=UwV2PapoVnLaqsE5plKYwjZ9oCHUvVwW%2B5ArwkXM5uQ%3D&reserved=0>
>  
>   Barry
>  
>  
> 
> 
>  
> 
> 
> On Aug 18, 2022, at 12:50 AM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> wrote:
>  
> I implemented the preconditioner. The solver converges quickly when the problem size is small. But when the size increases, say to 100k unknowns, the code hangs at assembly of the preconditioner matrix. The function call sequence is like this
>  
>    MatCreate(MPI_COMM_WORLD, &B);
>     MatSetType(B, MATMPIAIJ);
>     MatSetSizes(B, PETSC_DECIDE, PETSC_DECIDE, N, N);
>     MatSetFromOptions(B);
>  
>     // The number of diagonal and off diagonal non zeros only can be determined run-time.
>     MatMPIAIJSetPreallocation(B, 0, d_nnz, 0, o_nnz);
>     MatSetOption(B, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE);
>  
>     //here insert values row by row.
> MatSetValues(B, 1, &m, nn, nCols, matCols, INSERT_VALUES);
>  
> MatAssemblyBegin(B, MAT_FINAL_ASSEMBLY);
> MatAssemblyEnd(B, MAT_FINAL_ASSEMBLY);
>  
> Could you please tell me what I have done wrong?
>  
> Thank you,
> Jiannan
>    
> From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>> 
> Sent: Tuesday, August 16, 2022 2:00 PM
> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
> Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>
> Subject: Re: [petsc-users] Using matrix-free with KSP
>  
> CAUTION: This email was sent from outside the UMass Lowell network.
>  
>  
>    Create another matrix B that contains "Selected parts of the matrix can be pre-calculated and stored to serve as the preconditioner." When you call KSPSetOperators() pass in B as the second matrix argument.
>  
>    Now if literally application of B is the preconditioner you want to use, then simply use PCMAT as the preconditioner, and it will apply B each time it needs to apply the preconditioner. With this, you do not even need a PCShell. 
>  
>    If your preconditioner uses B to construct your own specific data structure needed to apply your preconditioner, then you can use PCShellSetSetUp() and have your routine call PCGetOperators() to pull out the B matrix and use it to construct your preconditioner. 
>  
>    If you want PETSc to construct a standard preconditioner from B, then you don't need the PCShell; you simply pass in the B, as above, and use -pc_type type to set the preconditioner you want to use. 
>  
>    Barry
>  
> 
> 
> 
> On Aug 16, 2022, at 12:31 PM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> wrote:
>  
> Barry,
>  
> Thank you very much for your instructions. I am sorry I may not ask clearer questions.
>  
> I don't use SNES. What I am trying to solve is a very large linear equation system with dense and ill-conditioned matrix. Selected parts of the matrix can be pre-calculated and stored to serve as the preconditioner. My question is how I can apply this matrix as the preconditioner. 
>  
> Here is the part of code from my linear equations solver. MatrixFreePreconditioner is supposed to be the routine to provide the preconditioner. But I don't understand how to use it. Should I use SNESSetJacobian() even if this is a linear problem? 
>  
>     MatCreateMFFD(MPI_COMM_WORLD, PETSC_DECIDE, PETSC_DECIDE, N, N, &A);
>     MatMFFDSetFunction(A, formfunction, &params);
>     MatSetFromOptions(A);
>     MatMFFDSetBase(A, X, NULL);
>  
>     KSPCreate(MPI_COMM_WORLD, &ksp);
>     KSPSetOperators(ksp, A, A);
>     KSPSetFromOptions(ksp);
>  
>     PC pc;
>     KSPGetPC(ksp,&pc);
>     PCSetType(pc,PCSHELL);
>     PCShellSetApply(pc,MatrixFreePreconditioner);
>  
> Thank you,
> Jiannan
>  
> From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>>
> Sent: Tuesday, August 16, 2022 10:10 AM
> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
> Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>>
> Subject: Re: [petsc-users] Using matrix-free with KSP
>  
> CAUTION: This email was sent from outside the UMass Lowell network.
>  
>  
>   I don't fully understand your question so I will answer questions that I do know the answer to :-)
>  
>   The function you provide to PCShellSetApply() applies the preconditioner to the input vector and puts the result in the output vector.  This input vector changes at every application of the preconditioner so should not be used in constructing the preconditioner.
>  
>   If you are using the standard PETSc matrix-free matrix-vector product via finite difference, -snes_mf_operator or via MatCreateSNESMF() or you are using a MatShell, the matrix-vector product routines input changes for each multiply and should not be used in constructing the preconditioner.
>  
>   The function you provide with SNESSetJacobian() is where you can compute anything you need to help build your preconditioner. The input vector to that function is the location at which the Jacobian is needed and is what you should use to build your approximate/part of Jacobian. You can store your approximate/part of Jacobian that you need to compute in the pmat and then in a PCShellSetUp() function that you provide you will compute what you will need latter to apply the preconditioner in your PCShellSetApply() function. Note that the values you put in the pmat do not need to be the full Jacobian; they can be anything you want since this pmat is not used to apply the matrix vector product.
>  
>   If this is unclear, feel free to ask additional questions.
>  
>   Barry
>  
> 
> 
> 
> On Aug 16, 2022, at 9:40 AM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> wrote:
>  
> I am trying to apply a user-defined preconditioner to speed up the convergency of matrix-free KSP solver. The user's manual provides an example of how to call the user-defined preconditioner routine. The routine has PC, input Vec and output Vec as arguments. I want to use part of the Jacobian as preconditioner matrix, the elements of which can be calculated using input Vec. My question is what the role of output Vec from the routine is in matrix-vector product? Or I just have the preconditioner matrix elements calculated in the routine and then Petsc handles applying the preconditioner to matrix-vector product? I used PCSetType(pc, PCSHELL) and PCShellSetApply.
>  
> Thank you and appreciate your help!
>  
> Jiannan 
> From: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
> Sent: Wednesday, July 6, 2022 2:34 PM
> To: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>>
> Cc: Jed Brown <jed at jedbrown.org <mailto:jed at jedbrown.org>>; petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>>
> Subject: Re: [petsc-users] Using matrix-free with KSP
>  
> Barry,
>  
> VecScatterCreateToAll solves the problem! The code now gives the correct answer with multiple processes. 
>  
> Thank you all so much!
>  
> Jiannan
> From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>>
> Sent: Wednesday, July 6, 2022 2:08 PM
> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
> Cc: Jed Brown <jed at jedbrown.org <mailto:jed at jedbrown.org>>; petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>>
> Subject: Re: [petsc-users] Using matrix-free with KSP
>  
>  
>   So your operator is a "dense" operator in that its matrix representation would be essentially dense. In that case, you can use specialized routines to do this efficiently. You can use VecScatterCreateToAll() and then VecScatterBegin/End to do the communication.
>  
>   Barry
>  
>  
> 
> 
> 
> On Jul 6, 2022, at 1:39 PM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> wrote:
>  
> Jed, thank you very much for the reply. 
>  
> I'm not sure GlobaltoLocal will work. Say the solution vector is of length 10. Two processes then each process can see 5 elements of the vector. For A x = b (10 equations), each process performs matrix-vector product for five equations. And for each equation i , sum_j A_ij *  x_j = b_i requires the process has access to all 10 unknows x_i (i=0,1,...,9). 
>  
> Can VecScatter and PetscSF make the entire vector accessible to each process? I am reading the manual and trying to understand how VecScatter and PetscSF work.
>  
> Jiannan
> From: Jed Brown <jed at jedbrown.org <mailto:jed at jedbrown.org>>
> Sent: Wednesday, July 6, 2022 10:09 AM
> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>; Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>>
> Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>>
> Subject: Re: [petsc-users] Using matrix-free with KSP
>  
> You'll usually have a GlobalToLocal operation for each rank to get the halo data it needs, then either a LocalToGlobal to collect the result (e.g., finite element methods) or the local compute will write directly into the owned portion of the global vector. If you're doing the communication "raw", then you may find VecScatter or PetscSF useful to perform the necessary communication.
> 
> "Tu, Jiannan" <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> writes:
> 
> > Hi Barry,
> >
> > Following your instructions I implemented the matrix-free method to solve a large linear equation system resulted from a surface integration equation. The KSP solver works fine for single process, but it is problematic with multiple processes. The problem is that each process only can access its own part of solution vector so that each process only conducts part of matrix-vector multiplication. MPI can be used to assemble these partial matrix-vector multiplication together.  Does Petsc provide any ways to implement multi-process matrix-free method?
> >
> > Thanks,
> > Jiannan
> > ________________________________
> > From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>>
> > Sent: Tuesday, May 24, 2022 2:12 PM
> > To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
> > Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>>
> > Subject: Re: [petsc-users] Using matrix-free with KSP
> >
> > This e-mail originated from outside the UMass Lowell network.
> > ________________________________
> >
> >    You can use MatCreateMFFD https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMatCreateMFFD%2F&amp;data=05%7C01%7CJiannan_Tu%40uml.edu%7Cab0859d8d154471590bc08da5f5918d6%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637927133525745809%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=A7wqegTfh94No5BpDiWLK3VxOuR44U2wlWHVm2k7l60%3D&amp;reserved=0<https://urldefense.com/v3/__https://petsc.org/main/docs/manualpages/Mat/MatCreateMFFD/__;!!PVKG_VDCxu5g!stxq2NWGVjUihvHBQC9Rlk0aHVoowcy0PTQYMcQxQ4DqOEC05KSw6TmstSXKckiUgelHzy4ue-d10-zlXkw$> <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMatCreateMFFD%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7C14cbcc56f6304ae2ada708da811614b4%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637964229096924381%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=IJGp1oTI1GnuKoKqchT0r5pC2sfO1F9Os867kP8i%2BzA%3D&reserved=0> MatMFFDSetFunction MatSetFromOptions MatMFFDSetBase and provide the matrix to KSP. Note you will need to use -pc_type none or provide your own custom preconditioner withhttps://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FPC%2FPCSHELL%2F&amp;data=05%7C01%7CJiannan_Tu%40uml.edu%7Cab0859d8d154471590bc08da5f5918d6%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637927133525745809%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=Gb%2F9uFG3jp%2FbfWaSh2cSEQLItHS9CuLwarzN0KcNiJY%3D&amp;reserved=0<https://urldefense.com/v3/__https://petsc.org/main/docs/manualpages/PC/PCSHELL/__;!!PVKG_VDCxu5g!stxq2NWGVjUihvHBQC9Rlk0aHVoowcy0PTQYMcQxQ4DqOEC05KSw6TmstSXKckiUgelHzy4ue-d1cA4fKGw$> <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FPC%2FPCSHELL%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7C14cbcc56f6304ae2ada708da811614b4%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637964229096924381%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=%2Ben584fnKhucOyFOS9C28HP%2BS8AiaYo0GRrWoFuSNus%3D&reserved=0>
> >
> >
> >
> > On May 24, 2022, at 1:21 PM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu><mailto:Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>> wrote:
> >
> > I want to use a matrix-free matrix to solve a large linear equation system because the matrix is too large to be stored. Petsc user manual describes matrix-free method for SNES with examples. The matrix-free matrices section explains how to set up such a matrix, but I can't find any example of matrix-free method with KSP. I am wondering how to apply the method to KSP iterative solver in parallel.
> >
> > I greatly appreciate your help for me to understand this topic.
> >
> > Jiannan Tu

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220818/843e60b8/attachment-0001.html>

From Jiannan_Tu at uml.edu  Thu Aug 18 11:52:32 2022
From: Jiannan_Tu at uml.edu (Tu, Jiannan)
Date: Thu, 18 Aug 2022 16:52:32 +0000
Subject: [petsc-users] Using matrix-free with KSP
In-Reply-To: <D0EB13D8-C5A6-4AE7-AFDE-2CDCCACBAF73@petsc.dev>
References: <BY5PR02MB6471969F80FA5F732FC74D87E7D79@BY5PR02MB6471.namprd02.prod.outlook.com>
	<B55CFC2C-6364-4B78-8E99-9D53700743B8@petsc.dev>
	<BY5PR02MB6471BDF90783FE3A0EA176CBE7809@BY5PR02MB6471.namprd02.prod.outlook.com>
	<87iloa49kt.fsf@jedbrown.org>
	<BY5PR02MB6471C0ED04769D02046F68ACE7809@BY5PR02MB6471.namprd02.prod.outlook.com>
	<00EBE4B8-22E5-408F-B56D-14226A278D7E@petsc.dev>
	<BY5PR02MB6471E386057DDDE98AD05DFAE7809@BY5PR02MB6471.namprd02.prod.outlook.com>
	<BY5PR02MB647140084C7E7CDDB6BC2A11E76B9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<A6629DE1-5A7F-414E-B164-A69877158675@petsc.dev>
	<BY5PR02MB6471AB1C004920DA1FC9B43EE76B9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<103146FD-FF28-47CE-B661-F2B521BF0E7A@petsc.dev>
	<DM6PR02MB6478B2FB8CB1BE757CE7B003E76D9@DM6PR02MB6478.namprd02.prod.outlook.com>
	<4C00F4B8-BB42-4932-BCD7-527E5FB39BFF@petsc.dev>
	<BY5PR02MB64710E70758E431108D66956E76D9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<D0EB13D8-C5A6-4AE7-AFDE-2CDCCACBAF73@petsc.dev>
Message-ID: <BY5PR02MB6471D4B05874DECFDDA13FFCE76D9@BY5PR02MB6471.namprd02.prod.outlook.com>

Thanks, Barry.

So I need actually two matrixes, one for preallocator and one for actual matrix that can be passed to KSPSetOperators(). -mat_type preallocator option is used to speed up doing insertion into preallocator, then use MatPreallocatorPreallocate() to preallocate actual matrix and do actual insertion of values into it., right?

The code runs in parallel. Each process owns number of rows that equals to number of unknowns (that is, xm in 1D DM) it owns.

Jiannan

From: Barry Smith <bsmith at petsc.dev>
Sent: Thursday, August 18, 2022 11:37 AM
To: Tu, Jiannan <Jiannan_Tu at uml.edu>
Cc: petsc-users at mcs.anl.gov
Subject: Re: [petsc-users] Using matrix-free with KSP

CAUTION: This email was sent from outside the UMass Lowell network.


   The preallocator MatType matrix cannot be passed to the KSPSetOperators(), you need to create an actual matrix, for example MATAIJ and use the preallocator to set its preallocation and then fill up the MATAIJ matrix with the usual MatSetValues and pass that matrix to the KSPSetOperators.


On Aug 18, 2022, at 11:30 AM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

Hi Barry,

The MATPREALLOCATOR solved problem of slow matrix assembly. But the solver failed because of "Matrix type preallocator does not have a multiply defined".

I guess this is because the matrix-free is used. I am wondering how a preconditioner is applied to the matrix-vector multiply in Petsc.

Thank you,
Jiannan

[0]PETSC ERROR: No support for this operation for this object type
[0]PETSC ERROR: Matrix type preallocator does not have a multiply defined
[0]PETSC ERROR: See https://petsc.org/release/faq/<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Frelease%2Ffaq%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7C59dae21b58e64ed9a89108da812f8660%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637964338374493241%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=n2BHnbiXekFhF%2FfhzCBMrOhsMLMiHdBzY4eknlnqJi0%3D&reserved=0> for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.16.6, Mar 30, 2022
[0]PETSC ERROR: ./iesolver on a  named REXNET-WS4 by jiannantu Thu Aug 18 11:21:55 2022
[0]PETSC ERROR: Configure options --download-f2cblaslapack=yes --with-mpi-dir=/usr/local --with-fc=0 --prefix=/home/jiannantu/petsc --with-scalar-type=complex --with-64-bit-indices=1
[0]PETSC ERROR: #1 MatMult() at /home/jiannantu/petsc-3.16.6/src/mat/interface/matrix.c:2437
[0]PETSC ERROR: #2 PCApply_Mat() at /home/jiannantu/petsc-3.16.6/src/ksp/pc/impls/mat/pcmat.c:9
[0]PETSC ERROR: #3 PCApply() at /home/jiannantu/petsc-3.16.6/src/ksp/pc/interface/precon.c:445
[0]PETSC ERROR: #4 KSP_PCApply() at /home/jiannantu/petsc-3.16.6/include/petsc/private/kspimpl.h:382
[0]PETSC ERROR: #5 KSPInitialResidual() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itres.c:65
[0]PETSC ERROR: #6 KSPSolve_BCGS() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/impls/bcgs/bcgs.c:43
[0]PETSC ERROR: #7 KSPSolve_Private() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itfunc.c:914
[0]PETSC ERROR: #8 KSPSolve() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itfunc.c:1086

From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Thursday, August 18, 2022 8:35 AM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] Using matrix-free with KSP

CAUTION: This email was sent from outside the UMass Lowell network.


  Slow assembly is usually due to under preallocation. You can change to using

MatSetOption(B, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_TRUE);

  to detect if you are under preallocating. See https://petsc.org/main/docs/manual/mat/#sec-matsparse<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanual%2Fmat%2F%23sec-matsparse&data=05%7C01%7CJiannan_Tu%40uml.edu%7C59dae21b58e64ed9a89108da812f8660%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637964338374493241%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=ZYbqedJGTeexabiVb6cLro4wb4tl9GMLrOsDsqtXNNk%3D&reserved=0>. That section clearly needs some sprucing up. If it is difficult to determine good preallocation values you can use https://petsc.org/main/docs/manualpages/Mat/MATPREALLOCATOR/<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMATPREALLOCATOR%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7C59dae21b58e64ed9a89108da812f8660%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637964338374493241%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=3wlhpm2mV1T4m%2B7EOqKhJAMUxE%2F0mkAx%2B8F%2F47oaNVw%3D&reserved=0> to compute the needed preallocation efficiently.

  Are you running in parallel? If so how are you determining which rows of entries to compute on each MPI rank? You will want most of the rows to be computed on the rank where the values are stored, you can determine this with https://petsc.org/main/docs/manualpages/Mat/MatGetOwnershipRange.html<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMatGetOwnershipRange.html&data=05%7C01%7CJiannan_Tu%40uml.edu%7C59dae21b58e64ed9a89108da812f8660%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637964338374493241%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=bviDcY78U9cAT2dvLADU49yCivArb4K%2F4Og2bAGFCRg%3D&reserved=0>

  Barry









On Aug 18, 2022, at 12:50 AM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

I implemented the preconditioner. The solver converges quickly when the problem size is small. But when the size increases, say to 100k unknowns, the code hangs at assembly of the preconditioner matrix. The function call sequence is like this

   MatCreate(MPI_COMM_WORLD, &B);
    MatSetType(B, MATMPIAIJ);
    MatSetSizes(B, PETSC_DECIDE, PETSC_DECIDE, N, N);
    MatSetFromOptions(B);

    // The number of diagonal and off diagonal non zeros only can be determined run-time.
    MatMPIAIJSetPreallocation(B, 0, d_nnz, 0, o_nnz);
    MatSetOption(B, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE);

    //here insert values row by row.
MatSetValues(B, 1, &m, nn, nCols, matCols, INSERT_VALUES);

MatAssemblyBegin(B, MAT_FINAL_ASSEMBLY);
MatAssemblyEnd(B, MAT_FINAL_ASSEMBLY);

Could you please tell me what I have done wrong?

Thank you,
Jiannan

From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Tuesday, August 16, 2022 2:00 PM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] Using matrix-free with KSP

CAUTION: This email was sent from outside the UMass Lowell network.


   Create another matrix B that contains "Selected parts of the matrix can be pre-calculated and stored to serve as the preconditioner." When you call KSPSetOperators() pass in B as the second matrix argument.

   Now if literally application of B is the preconditioner you want to use, then simply use PCMAT as the preconditioner, and it will apply B each time it needs to apply the preconditioner. With this, you do not even need a PCShell.

   If your preconditioner uses B to construct your own specific data structure needed to apply your preconditioner, then you can use PCShellSetSetUp() and have your routine call PCGetOperators() to pull out the B matrix and use it to construct your preconditioner.

   If you want PETSc to construct a standard preconditioner from B, then you don't need the PCShell; you simply pass in the B, as above, and use -pc_type type to set the preconditioner you want to use.

   Barry





On Aug 16, 2022, at 12:31 PM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

Barry,

Thank you very much for your instructions. I am sorry I may not ask clearer questions.

I don't use SNES. What I am trying to solve is a very large linear equation system with dense and ill-conditioned matrix. Selected parts of the matrix can be pre-calculated and stored to serve as the preconditioner. My question is how I can apply this matrix as the preconditioner.

Here is the part of code from my linear equations solver. MatrixFreePreconditioner is supposed to be the routine to provide the preconditioner. But I don't understand how to use it. Should I use SNESSetJacobian() even if this is a linear problem?

    MatCreateMFFD(MPI_COMM_WORLD, PETSC_DECIDE, PETSC_DECIDE, N, N, &A);
    MatMFFDSetFunction(A, formfunction, &params);
    MatSetFromOptions(A);
    MatMFFDSetBase(A, X, NULL);

    KSPCreate(MPI_COMM_WORLD, &ksp);
    KSPSetOperators(ksp, A, A);
    KSPSetFromOptions(ksp);

    PC pc;
    KSPGetPC(ksp,&pc);
    PCSetType(pc,PCSHELL);
    PCShellSetApply(pc,MatrixFreePreconditioner);

Thank you,
Jiannan

________________________________
From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Tuesday, August 16, 2022 10:10 AM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] Using matrix-free with KSP

CAUTION: This email was sent from outside the UMass Lowell network.


  I don't fully understand your question so I will answer questions that I do know the answer to :-)

  The function you provide to PCShellSetApply() applies the preconditioner to the input vector and puts the result in the output vector.  This input vector changes at every application of the preconditioner so should not be used in constructing the preconditioner.

  If you are using the standard PETSc matrix-free matrix-vector product via finite difference, -snes_mf_operator or via MatCreateSNESMF() or you are using a MatShell, the matrix-vector product routines input changes for each multiply and should not be used in constructing the preconditioner.

  The function you provide with SNESSetJacobian() is where you can compute anything you need to help build your preconditioner. The input vector to that function is the location at which the Jacobian is needed and is what you should use to build your approximate/part of Jacobian. You can store your approximate/part of Jacobian that you need to compute in the pmat and then in a PCShellSetUp() function that you provide you will compute what you will need latter to apply the preconditioner in your PCShellSetApply() function. Note that the values you put in the pmat do not need to be the full Jacobian; they can be anything you want since this pmat is not used to apply the matrix vector product.

  If this is unclear, feel free to ask additional questions.

  Barry





On Aug 16, 2022, at 9:40 AM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

I am trying to apply a user-defined preconditioner to speed up the convergency of matrix-free KSP solver. The user's manual provides an example of how to call the user-defined preconditioner routine. The routine has PC, input Vec and output Vec as arguments. I want to use part of the Jacobian as preconditioner matrix, the elements of which can be calculated using input Vec. My question is what the role of output Vec from the routine is in matrix-vector product? Or I just have the preconditioner matrix elements calculated in the routine and then Petsc handles applying the preconditioner to matrix-vector product? I used PCSetType(pc, PCSHELL) and PCShellSetApply.

Thank you and appreciate your help!

Jiannan
________________________________
From: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Sent: Wednesday, July 6, 2022 2:34 PM
To: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Cc: Jed Brown <jed at jedbrown.org<mailto:jed at jedbrown.org>>; petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] Using matrix-free with KSP

Barry,

VecScatterCreateToAll solves the problem! The code now gives the correct answer with multiple processes.

Thank you all so much!

Jiannan
________________________________
From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Wednesday, July 6, 2022 2:08 PM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Cc: Jed Brown <jed at jedbrown.org<mailto:jed at jedbrown.org>>; petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] Using matrix-free with KSP


  So your operator is a "dense" operator in that its matrix representation would be essentially dense. In that case, you can use specialized routines to do this efficiently. You can use VecScatterCreateToAll() and then VecScatterBegin/End to do the communication.

  Barry






On Jul 6, 2022, at 1:39 PM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

Jed, thank you very much for the reply.

I'm not sure GlobaltoLocal will work. Say the solution vector is of length 10. Two processes then each process can see 5 elements of the vector. For A x = b (10 equations), each process performs matrix-vector product for five equations. And for each equation i , sum_j A_ij *  x_j = b_i requires the process has access to all 10 unknows x_i (i=0,1,...,9).

Can VecScatter and PetscSF make the entire vector accessible to each process? I am reading the manual and trying to understand how VecScatter and PetscSF work.

Jiannan
________________________________
From: Jed Brown <jed at jedbrown.org<mailto:jed at jedbrown.org>>
Sent: Wednesday, July 6, 2022 10:09 AM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>; Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] Using matrix-free with KSP

You'll usually have a GlobalToLocal operation for each rank to get the halo data it needs, then either a LocalToGlobal to collect the result (e.g., finite element methods) or the local compute will write directly into the owned portion of the global vector. If you're doing the communication "raw", then you may find VecScatter or PetscSF useful to perform the necessary communication.

"Tu, Jiannan" <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> writes:

> Hi Barry,
>
> Following your instructions I implemented the matrix-free method to solve a large linear equation system resulted from a surface integration equation. The KSP solver works fine for single process, but it is problematic with multiple processes. The problem is that each process only can access its own part of solution vector so that each process only conducts part of matrix-vector multiplication. MPI can be used to assemble these partial matrix-vector multiplication together.  Does Petsc provide any ways to implement multi-process matrix-free method?
>
> Thanks,
> Jiannan
> ________________________________
> From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
> Sent: Tuesday, May 24, 2022 2:12 PM
> To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
> Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
> Subject: Re: [petsc-users] Using matrix-free with KSP
>
> This e-mail originated from outside the UMass Lowell network.
> ________________________________
>
>    You can use MatCreateMFFD https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMatCreateMFFD%2F&amp;data=05%7C01%7CJiannan_Tu%40uml.edu%7Cab0859d8d154471590bc08da5f5918d6%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637927133525745809%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=A7wqegTfh94No5BpDiWLK3VxOuR44U2wlWHVm2k7l60%3D&amp;reserved=0<https://urldefense.com/v3/__https://petsc.org/main/docs/manualpages/Mat/MatCreateMFFD/__;!!PVKG_VDCxu5g!stxq2NWGVjUihvHBQC9Rlk0aHVoowcy0PTQYMcQxQ4DqOEC05KSw6TmstSXKckiUgelHzy4ue-d10-zlXkw$><https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMatCreateMFFD%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7C59dae21b58e64ed9a89108da812f8660%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637964338374493241%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=2aLlxTwp4PnIfLRef2P8X60XEYgLo1FvhHp13QmXT7E%3D&reserved=0> MatMFFDSetFunction MatSetFromOptions MatMFFDSetBase and provide the matrix to KSP. Note you will need to use -pc_type none or provide your own custom preconditioner withhttps://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FPC%2FPCSHELL%2F&amp;data=05%7C01%7CJiannan_Tu%40uml.edu%7Cab0859d8d154471590bc08da5f5918d6%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637927133525745809%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=Gb%2F9uFG3jp%2FbfWaSh2cSEQLItHS9CuLwarzN0KcNiJY%3D&amp;reserved=0<https://urldefense.com/v3/__https://petsc.org/main/docs/manualpages/PC/PCSHELL/__;!!PVKG_VDCxu5g!stxq2NWGVjUihvHBQC9Rlk0aHVoowcy0PTQYMcQxQ4DqOEC05KSw6TmstSXKckiUgelHzy4ue-d1cA4fKGw$><https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FPC%2FPCSHELL%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7C59dae21b58e64ed9a89108da812f8660%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637964338374493241%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=qPnmCge3WYNb0OIQZMDODm8MpMF9AQliMPlWCjyDjDE%3D&reserved=0>
>
>
>
> On May 24, 2022, at 1:21 PM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu><mailto:Jiannan_Tu at uml.edu>> wrote:
>
> I want to use a matrix-free matrix to solve a large linear equation system because the matrix is too large to be stored. Petsc user manual describes matrix-free method for SNES with examples. The matrix-free matrices section explains how to set up such a matrix, but I can't find any example of matrix-free method with KSP. I am wondering how to apply the method to KSP iterative solver in parallel.
>
> I greatly appreciate your help for me to understand this topic.
>
> Jiannan Tu

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220818/a6eae2c0/attachment-0001.html>

From bsmith at petsc.dev  Thu Aug 18 11:59:26 2022
From: bsmith at petsc.dev (Barry Smith)
Date: Thu, 18 Aug 2022 12:59:26 -0400
Subject: [petsc-users] Using matrix-free with KSP
In-Reply-To: <BY5PR02MB6471D4B05874DECFDDA13FFCE76D9@BY5PR02MB6471.namprd02.prod.outlook.com>
References: <BY5PR02MB6471969F80FA5F732FC74D87E7D79@BY5PR02MB6471.namprd02.prod.outlook.com>
	<B55CFC2C-6364-4B78-8E99-9D53700743B8@petsc.dev>
	<BY5PR02MB6471BDF90783FE3A0EA176CBE7809@BY5PR02MB6471.namprd02.prod.outlook.com>
	<87iloa49kt.fsf@jedbrown.org>
	<BY5PR02MB6471C0ED04769D02046F68ACE7809@BY5PR02MB6471.namprd02.prod.outlook.com>
	<00EBE4B8-22E5-408F-B56D-14226A278D7E@petsc.dev>
	<BY5PR02MB6471E386057DDDE98AD05DFAE7809@BY5PR02MB6471.namprd02.prod.outlook.com>
	<BY5PR02MB647140084C7E7CDDB6BC2A11E76B9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<A6629DE1-5A7F-414E-B164-A69877158675@petsc.dev>
	<BY5PR02MB6471AB1C004920DA1FC9B43EE76B9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<103146FD-FF28-47CE-B661-F2B521BF0E7A@petsc.dev>
	<DM6PR02MB6478B2FB8CB1BE757CE7B003E76D9@DM6PR02MB6478.namprd02.prod.outlook.com>
	<4C00F4B8-BB42-4932-BCD7-527E5FB39BFF@petsc.dev>
	<BY5PR02MB64710E70758E431108D66956E76D9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<D0EB13D8-C5A6-4AE7-AFDE-2CDCCACBAF73@petsc.dev>
	<BY5PR02MB6471D4B05874DECFDDA13FFCE76D9@BY5PR02MB6471.namprd02.prod.outlook.com>
Message-ID: <367B124C-88BB-4889-A4AF-45A1D473078D@petsc.dev>


  Yes, once you have preallocated the real matrix you can destroy the preallocation matrix whose only job is to gather the preallocation information

> On Aug 18, 2022, at 12:52 PM, Tu, Jiannan <Jiannan_Tu at uml.edu> wrote:
> 
> Thanks, Barry.
>  
> So I need actually two matrixes, one for preallocator and one for actual matrix that can be passed to KSPSetOperators(). -mat_type preallocator option is used to speed up doing insertion into preallocator, then use MatPreallocatorPreallocate() to preallocate actual matrix and do actual insertion of values into it., right?
>  
> The code runs in parallel. Each process owns number of rows that equals to number of unknowns (that is, xm in 1D DM) it owns.
>  
> Jiannan
>  
> From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>> 
> Sent: Thursday, August 18, 2022 11:37 AM
> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
> Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>
> Subject: Re: [petsc-users] Using matrix-free with KSP
>  
> CAUTION: This email was sent from outside the UMass Lowell network.
>  
>  
>    The preallocator MatType matrix cannot be passed to the KSPSetOperators(), you need to create an actual matrix, for example MATAIJ and use the preallocator to set its preallocation and then fill up the MATAIJ matrix with the usual MatSetValues and pass that matrix to the KSPSetOperators.
> 
> 
> On Aug 18, 2022, at 11:30 AM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> wrote:
>  
> Hi Barry,
>  
> The MATPREALLOCATOR solved problem of slow matrix assembly. But the solver failed because of ?Matrix type preallocator does not have a multiply defined?.
>  
> I guess this is because the matrix-free is used. I am wondering how a preconditioner is applied to the matrix-vector multiply in Petsc.
>  
> Thank you,
> Jiannan
>  
> [0]PETSC ERROR: No support for this operation for this object type
> [0]PETSC ERROR: Matrix type preallocator does not have a multiply defined
> [0]PETSC ERROR: See https://petsc.org/release/faq/ <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Frelease%2Ffaq%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7C59dae21b58e64ed9a89108da812f8660%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637964338374493241%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=n2BHnbiXekFhF%2FfhzCBMrOhsMLMiHdBzY4eknlnqJi0%3D&reserved=0> for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.16.6, Mar 30, 2022
> [0]PETSC ERROR: ./iesolver on a  named REXNET-WS4 by jiannantu Thu Aug 18 11:21:55 2022
> [0]PETSC ERROR: Configure options --download-f2cblaslapack=yes --with-mpi-dir=/usr/local --with-fc=0 --prefix=/home/jiannantu/petsc --with-scalar-type=complex --with-64-bit-indices=1
> [0]PETSC ERROR: #1 MatMult() at /home/jiannantu/petsc-3.16.6/src/mat/interface/matrix.c:2437
> [0]PETSC ERROR: #2 PCApply_Mat() at /home/jiannantu/petsc-3.16.6/src/ksp/pc/impls/mat/pcmat.c:9
> [0]PETSC ERROR: #3 PCApply() at /home/jiannantu/petsc-3.16.6/src/ksp/pc/interface/precon.c:445
> [0]PETSC ERROR: #4 KSP_PCApply() at /home/jiannantu/petsc-3.16.6/include/petsc/private/kspimpl.h:382
> [0]PETSC ERROR: #5 KSPInitialResidual() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itres.c:65
> [0]PETSC ERROR: #6 KSPSolve_BCGS() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/impls/bcgs/bcgs.c:43
> [0]PETSC ERROR: #7 KSPSolve_Private() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itfunc.c:914
> [0]PETSC ERROR: #8 KSPSolve() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itfunc.c:1086
>  
> From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>> 
> Sent: Thursday, August 18, 2022 8:35 AM
> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
> Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>
> Subject: Re: [petsc-users] Using matrix-free with KSP
>  
> CAUTION: This email was sent from outside the UMass Lowell network.
>  
>  
>   Slow assembly is usually due to under preallocation. You can change to using 
>  
> MatSetOption(B, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_TRUE);
>  
>   to detect if you are under preallocating. See https://petsc.org/main/docs/manual/mat/#sec-matsparse <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanual%2Fmat%2F%23sec-matsparse&data=05%7C01%7CJiannan_Tu%40uml.edu%7C59dae21b58e64ed9a89108da812f8660%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637964338374493241%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=ZYbqedJGTeexabiVb6cLro4wb4tl9GMLrOsDsqtXNNk%3D&reserved=0>. That section clearly needs some sprucing up. If it is difficult to determine good preallocation values you can use https://petsc.org/main/docs/manualpages/Mat/MATPREALLOCATOR/ <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMATPREALLOCATOR%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7C59dae21b58e64ed9a89108da812f8660%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637964338374493241%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=3wlhpm2mV1T4m%2B7EOqKhJAMUxE%2F0mkAx%2B8F%2F47oaNVw%3D&reserved=0> to compute the needed preallocation efficiently.
>  
>   Are you running in parallel? If so how are you determining which rows of entries to compute on each MPI rank? You will want most of the rows to be computed on the rank where the values are stored, you can determine this with https://petsc.org/main/docs/manualpages/Mat/MatGetOwnershipRange.html <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMatGetOwnershipRange.html&data=05%7C01%7CJiannan_Tu%40uml.edu%7C59dae21b58e64ed9a89108da812f8660%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637964338374493241%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=bviDcY78U9cAT2dvLADU49yCivArb4K%2F4Og2bAGFCRg%3D&reserved=0>
>  
>   Barry
>  
>  
> 
> 
> 
>  
> 
> 
> 
> On Aug 18, 2022, at 12:50 AM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> wrote:
>  
> I implemented the preconditioner. The solver converges quickly when the problem size is small. But when the size increases, say to 100k unknowns, the code hangs at assembly of the preconditioner matrix. The function call sequence is like this
>  
>    MatCreate(MPI_COMM_WORLD, &B);
>     MatSetType(B, MATMPIAIJ);
>     MatSetSizes(B, PETSC_DECIDE, PETSC_DECIDE, N, N);
>     MatSetFromOptions(B);
>  
>     // The number of diagonal and off diagonal non zeros only can be determined run-time.
>     MatMPIAIJSetPreallocation(B, 0, d_nnz, 0, o_nnz);
>     MatSetOption(B, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE);
>  
>     //here insert values row by row.
> MatSetValues(B, 1, &m, nn, nCols, matCols, INSERT_VALUES);
>  
> MatAssemblyBegin(B, MAT_FINAL_ASSEMBLY);
> MatAssemblyEnd(B, MAT_FINAL_ASSEMBLY);
>  
> Could you please tell me what I have done wrong?
>  
> Thank you,
> Jiannan
>    
> From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>> 
> Sent: Tuesday, August 16, 2022 2:00 PM
> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
> Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>
> Subject: Re: [petsc-users] Using matrix-free with KSP
>  
> CAUTION: This email was sent from outside the UMass Lowell network.
>  
>  
>    Create another matrix B that contains "Selected parts of the matrix can be pre-calculated and stored to serve as the preconditioner." When you call KSPSetOperators() pass in B as the second matrix argument.
>  
>    Now if literally application of B is the preconditioner you want to use, then simply use PCMAT as the preconditioner, and it will apply B each time it needs to apply the preconditioner. With this, you do not even need a PCShell. 
>  
>    If your preconditioner uses B to construct your own specific data structure needed to apply your preconditioner, then you can use PCShellSetSetUp() and have your routine call PCGetOperators() to pull out the B matrix and use it to construct your preconditioner. 
>  
>    If you want PETSc to construct a standard preconditioner from B, then you don't need the PCShell; you simply pass in the B, as above, and use -pc_type type to set the preconditioner you want to use. 
>  
>    Barry
>  
> 
> 
> 
> 
> On Aug 16, 2022, at 12:31 PM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> wrote:
>  
> Barry,
>  
> Thank you very much for your instructions. I am sorry I may not ask clearer questions.
>  
> I don't use SNES. What I am trying to solve is a very large linear equation system with dense and ill-conditioned matrix. Selected parts of the matrix can be pre-calculated and stored to serve as the preconditioner. My question is how I can apply this matrix as the preconditioner. 
>  
> Here is the part of code from my linear equations solver. MatrixFreePreconditioner is supposed to be the routine to provide the preconditioner. But I don't understand how to use it. Should I use SNESSetJacobian() even if this is a linear problem? 
>  
>     MatCreateMFFD(MPI_COMM_WORLD, PETSC_DECIDE, PETSC_DECIDE, N, N, &A);
>     MatMFFDSetFunction(A, formfunction, &params);
>     MatSetFromOptions(A);
>     MatMFFDSetBase(A, X, NULL);
>  
>     KSPCreate(MPI_COMM_WORLD, &ksp);
>     KSPSetOperators(ksp, A, A);
>     KSPSetFromOptions(ksp);
>  
>     PC pc;
>     KSPGetPC(ksp,&pc);
>     PCSetType(pc,PCSHELL);
>     PCShellSetApply(pc,MatrixFreePreconditioner);
>  
> Thank you,
> Jiannan
>  
> From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>>
> Sent: Tuesday, August 16, 2022 10:10 AM
> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
> Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>>
> Subject: Re: [petsc-users] Using matrix-free with KSP
>  
> CAUTION: This email was sent from outside the UMass Lowell network.
>  
>  
>   I don't fully understand your question so I will answer questions that I do know the answer to :-)
>  
>   The function you provide to PCShellSetApply() applies the preconditioner to the input vector and puts the result in the output vector.  This input vector changes at every application of the preconditioner so should not be used in constructing the preconditioner.
>  
>   If you are using the standard PETSc matrix-free matrix-vector product via finite difference, -snes_mf_operator or via MatCreateSNESMF() or you are using a MatShell, the matrix-vector product routines input changes for each multiply and should not be used in constructing the preconditioner.
>  
>   The function you provide with SNESSetJacobian() is where you can compute anything you need to help build your preconditioner. The input vector to that function is the location at which the Jacobian is needed and is what you should use to build your approximate/part of Jacobian. You can store your approximate/part of Jacobian that you need to compute in the pmat and then in a PCShellSetUp() function that you provide you will compute what you will need latter to apply the preconditioner in your PCShellSetApply() function. Note that the values you put in the pmat do not need to be the full Jacobian; they can be anything you want since this pmat is not used to apply the matrix vector product.
>  
>   If this is unclear, feel free to ask additional questions.
>  
>   Barry
>  
> 
> 
> 
> 
> On Aug 16, 2022, at 9:40 AM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> wrote:
>  
> I am trying to apply a user-defined preconditioner to speed up the convergency of matrix-free KSP solver. The user's manual provides an example of how to call the user-defined preconditioner routine. The routine has PC, input Vec and output Vec as arguments. I want to use part of the Jacobian as preconditioner matrix, the elements of which can be calculated using input Vec. My question is what the role of output Vec from the routine is in matrix-vector product? Or I just have the preconditioner matrix elements calculated in the routine and then Petsc handles applying the preconditioner to matrix-vector product? I used PCSetType(pc, PCSHELL) and PCShellSetApply.
>  
> Thank you and appreciate your help!
>  
> Jiannan 
> From: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
> Sent: Wednesday, July 6, 2022 2:34 PM
> To: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>>
> Cc: Jed Brown <jed at jedbrown.org <mailto:jed at jedbrown.org>>; petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>>
> Subject: Re: [petsc-users] Using matrix-free with KSP
>  
> Barry,
>  
> VecScatterCreateToAll solves the problem! The code now gives the correct answer with multiple processes. 
>  
> Thank you all so much!
>  
> Jiannan
> From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>>
> Sent: Wednesday, July 6, 2022 2:08 PM
> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
> Cc: Jed Brown <jed at jedbrown.org <mailto:jed at jedbrown.org>>; petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>>
> Subject: Re: [petsc-users] Using matrix-free with KSP
>  
>  
>   So your operator is a "dense" operator in that its matrix representation would be essentially dense. In that case, you can use specialized routines to do this efficiently. You can use VecScatterCreateToAll() and then VecScatterBegin/End to do the communication.
>  
>   Barry
>  
>  
> 
> 
> 
> 
> On Jul 6, 2022, at 1:39 PM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> wrote:
>  
> Jed, thank you very much for the reply. 
>  
> I'm not sure GlobaltoLocal will work. Say the solution vector is of length 10. Two processes then each process can see 5 elements of the vector. For A x = b (10 equations), each process performs matrix-vector product for five equations. And for each equation i , sum_j A_ij *  x_j = b_i requires the process has access to all 10 unknows x_i (i=0,1,...,9). 
>  
> Can VecScatter and PetscSF make the entire vector accessible to each process? I am reading the manual and trying to understand how VecScatter and PetscSF work.
>  
> Jiannan
> From: Jed Brown <jed at jedbrown.org <mailto:jed at jedbrown.org>>
> Sent: Wednesday, July 6, 2022 10:09 AM
> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>; Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>>
> Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>>
> Subject: Re: [petsc-users] Using matrix-free with KSP
>  
> You'll usually have a GlobalToLocal operation for each rank to get the halo data it needs, then either a LocalToGlobal to collect the result (e.g., finite element methods) or the local compute will write directly into the owned portion of the global vector. If you're doing the communication "raw", then you may find VecScatter or PetscSF useful to perform the necessary communication.
> 
> "Tu, Jiannan" <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> writes:
> 
> > Hi Barry,
> >
> > Following your instructions I implemented the matrix-free method to solve a large linear equation system resulted from a surface integration equation. The KSP solver works fine for single process, but it is problematic with multiple processes. The problem is that each process only can access its own part of solution vector so that each process only conducts part of matrix-vector multiplication. MPI can be used to assemble these partial matrix-vector multiplication together.  Does Petsc provide any ways to implement multi-process matrix-free method?
> >
> > Thanks,
> > Jiannan
> > ________________________________
> > From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>>
> > Sent: Tuesday, May 24, 2022 2:12 PM
> > To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
> > Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>>
> > Subject: Re: [petsc-users] Using matrix-free with KSP
> >
> > This e-mail originated from outside the UMass Lowell network.
> > ________________________________
> >
> >    You can use MatCreateMFFD https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMatCreateMFFD%2F&amp;data=05%7C01%7CJiannan_Tu%40uml.edu%7Cab0859d8d154471590bc08da5f5918d6%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637927133525745809%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=A7wqegTfh94No5BpDiWLK3VxOuR44U2wlWHVm2k7l60%3D&amp;reserved=0<https://urldefense.com/v3/__https://petsc.org/main/docs/manualpages/Mat/MatCreateMFFD/__;!!PVKG_VDCxu5g!stxq2NWGVjUihvHBQC9Rlk0aHVoowcy0PTQYMcQxQ4DqOEC05KSw6TmstSXKckiUgelHzy4ue-d10-zlXkw$> <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMatCreateMFFD%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7C59dae21b58e64ed9a89108da812f8660%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637964338374493241%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=2aLlxTwp4PnIfLRef2P8X60XEYgLo1FvhHp13QmXT7E%3D&reserved=0> MatMFFDSetFunction MatSetFromOptions MatMFFDSetBase and provide the matrix to KSP. Note you will need to use -pc_type none or provide your own custom preconditioner withhttps://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FPC%2FPCSHELL%2F&amp;data=05%7C01%7CJiannan_Tu%40uml.edu%7Cab0859d8d154471590bc08da5f5918d6%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637927133525745809%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=Gb%2F9uFG3jp%2FbfWaSh2cSEQLItHS9CuLwarzN0KcNiJY%3D&amp;reserved=0<https://urldefense.com/v3/__https://petsc.org/main/docs/manualpages/PC/PCSHELL/__;!!PVKG_VDCxu5g!stxq2NWGVjUihvHBQC9Rlk0aHVoowcy0PTQYMcQxQ4DqOEC05KSw6TmstSXKckiUgelHzy4ue-d1cA4fKGw$> <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FPC%2FPCSHELL%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7C59dae21b58e64ed9a89108da812f8660%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637964338374493241%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=qPnmCge3WYNb0OIQZMDODm8MpMF9AQliMPlWCjyDjDE%3D&reserved=0>
> >
> >
> >
> > On May 24, 2022, at 1:21 PM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu><mailto:Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>> wrote:
> >
> > I want to use a matrix-free matrix to solve a large linear equation system because the matrix is too large to be stored. Petsc user manual describes matrix-free method for SNES with examples. The matrix-free matrices section explains how to set up such a matrix, but I can't find any example of matrix-free method with KSP. I am wondering how to apply the method to KSP iterative solver in parallel.
> >
> > I greatly appreciate your help for me to understand this topic.
> >
> > Jiannan Tu

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220818/13458248/attachment-0001.html>

From Jiannan_Tu at uml.edu  Thu Aug 18 14:05:37 2022
From: Jiannan_Tu at uml.edu (Tu, Jiannan)
Date: Thu, 18 Aug 2022 19:05:37 +0000
Subject: [petsc-users] Using matrix-free with KSP
In-Reply-To: <367B124C-88BB-4889-A4AF-45A1D473078D@petsc.dev>
References: <BY5PR02MB6471969F80FA5F732FC74D87E7D79@BY5PR02MB6471.namprd02.prod.outlook.com>
	<B55CFC2C-6364-4B78-8E99-9D53700743B8@petsc.dev>
	<BY5PR02MB6471BDF90783FE3A0EA176CBE7809@BY5PR02MB6471.namprd02.prod.outlook.com>
	<87iloa49kt.fsf@jedbrown.org>
	<BY5PR02MB6471C0ED04769D02046F68ACE7809@BY5PR02MB6471.namprd02.prod.outlook.com>
	<00EBE4B8-22E5-408F-B56D-14226A278D7E@petsc.dev>
	<BY5PR02MB6471E386057DDDE98AD05DFAE7809@BY5PR02MB6471.namprd02.prod.outlook.com>
	<BY5PR02MB647140084C7E7CDDB6BC2A11E76B9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<A6629DE1-5A7F-414E-B164-A69877158675@petsc.dev>
	<BY5PR02MB6471AB1C004920DA1FC9B43EE76B9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<103146FD-FF28-47CE-B661-F2B521BF0E7A@petsc.dev>
	<DM6PR02MB6478B2FB8CB1BE757CE7B003E76D9@DM6PR02MB6478.namprd02.prod.outlook.com>
	<4C00F4B8-BB42-4932-BCD7-527E5FB39BFF@petsc.dev>
	<BY5PR02MB64710E70758E431108D66956E76D9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<D0EB13D8-C5A6-4AE7-AFDE-2CDCCACBAF73@petsc.dev>
	<BY5PR02MB6471D4B05874DECFDDA13FFCE76D9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<367B124C-88BB-4889-A4AF-45A1D473078D@petsc.dev>
Message-ID: <BY5PR02MB6471F2C058B84DE092567731E76D9@BY5PR02MB6471.namprd02.prod.outlook.com>

Barry,

Thanks. The method works. Assembly of actual matrix now completes quickly. The convergency is much better for the first iteration. The residual normal is in the order of 10^-10 instead of 10^-2. However, there is an error message after the first iteration: "Object is in wrong state. Not for unassembled matrix" even though both preallocator and actual matrix have been assembled (see the code snippet below). Could you please give me some hints what's going wrong? I use KSP type bcgs and PC type mat

Thank you,
Jiannan

-----------------------------------------------------
Solve linear equations ...
  0 KSP Residual norm 9.841179147519e-10
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Object is in wrong state
[0]PETSC ERROR: Not for unassembled matrix
[0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.16.6, Mar 30, 2022
[0]PETSC ERROR: ./iesolver on a  named REXNET-WS4 by jiannantu Thu Aug 18 14:48:43 2022
[0]PETSC ERROR: Configure options --download-f2cblaslapack=yes --with-mpi-dir=/usr/local --with-fc=0 --prefix=/home/jiannantu/petsc --with-scalar-type=complex --with-64-bit-indices=1
[0]PETSC ERROR: #1 MatMult() at /home/jiannantu/petsc-3.16.6/src/mat/interface/matrix.c:2425
[0]PETSC ERROR: #2 PCApplyBAorAB() at /home/jiannantu/petsc-3.16.6/src/ksp/pc/interface/precon.c:730
[0]PETSC ERROR: #3 KSP_PCApplyBAorAB() at /home/jiannantu/petsc-3.16.6/include/petsc/private/kspimpl.h:421
[0]PETSC ERROR:  Linear equations not solved. Diverged reason: 0
#4 KSPSolve_BCGS() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/impls/bcgs/bcgs.c:87
[0]PETSC ERROR: #5 KSPSolve_Private() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itfunc.c:914
[0]PETSC ERROR: #6 KSPSolve() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itfunc.c:1086

-----------------------------------------------
    MatCreate(MPI_COMM_WORLD, &B);
    MatSetType(B, MATMPIAIJ);
    MatSetSizes(B, PETSC_DECIDE, PETSC_DECIDE, N, N);

    Mat preall;
    MatCreate(MPI_COMM_WORLD, &preall);
    MatSetType(preall, MATPREALLOCATOR);
    MatSetSizes(preall, PETSC_DECIDE, PETSC_DECIDE, N, N);

    MatMPIAIJSetPreallocation(preall, 0, d_nnz, 0, o_nnz);
    MatSetUp(preall);
MatSetOption(preall, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE);

//set values row by row
MatSetValues(preall, 1, &m, 1, nCols, matCols, INSERT_VALUES);

MatAssemblyBegin(preall, MAT_FINAL_ASSEMBLY);
MatAssemblyEnd(preall, MAT_FINAL_ASSEMBLY);

//set values for actual matrix row by row
MatSetValues(B, 1, &m, 1, nCols, matCols, INSERT_VALUES);

MatAssemblyBegin(B, MAT_FINAL_ASSEMBLY);
MatAssemblyEnd(B, MAT_FINAL_ASSEMBLY);

    KSPCreate(MPI_COMM_WORLD, &ksp);
    KSPSetOperators(ksp, A, B);
    KSPSetFromOptions(ksp);

    PC pc;
    KSPGetPC(ksp, &pc);
    PCSetFromOptions(pc);

From: Barry Smith <bsmith at petsc.dev>
Sent: Thursday, August 18, 2022 12:59 PM
To: Tu, Jiannan <Jiannan_Tu at uml.edu>
Cc: petsc-users at mcs.anl.gov
Subject: Re: [petsc-users] Using matrix-free with KSP

CAUTION: This email was sent from outside the UMass Lowell network.


  Yes, once you have preallocated the real matrix you can destroy the preallocation matrix whose only job is to gather the preallocation information


On Aug 18, 2022, at 12:52 PM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

Thanks, Barry.

So I need actually two matrixes, one for preallocator and one for actual matrix that can be passed to KSPSetOperators(). -mat_type preallocator option is used to speed up doing insertion into preallocator, then use MatPreallocatorPreallocate() to preallocate actual matrix and do actual insertion of values into it., right?

The code runs in parallel. Each process owns number of rows that equals to number of unknowns (that is, xm in 1D DM) it owns.

Jiannan

From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Thursday, August 18, 2022 11:37 AM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] Using matrix-free with KSP

CAUTION: This email was sent from outside the UMass Lowell network.


   The preallocator MatType matrix cannot be passed to the KSPSetOperators(), you need to create an actual matrix, for example MATAIJ and use the preallocator to set its preallocation and then fill up the MATAIJ matrix with the usual MatSetValues and pass that matrix to the KSPSetOperators.



On Aug 18, 2022, at 11:30 AM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

Hi Barry,

The MATPREALLOCATOR solved problem of slow matrix assembly. But the solver failed because of "Matrix type preallocator does not have a multiply defined".

I guess this is because the matrix-free is used. I am wondering how a preconditioner is applied to the matrix-vector multiply in Petsc.

Thank you,
Jiannan

[0]PETSC ERROR: No support for this operation for this object type
[0]PETSC ERROR: Matrix type preallocator does not have a multiply defined
[0]PETSC ERROR: See https://petsc.org/release/faq/<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Frelease%2Ffaq%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7C92781c0483d44c98a24508da813b02f8%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637964387727402803%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=vfijXLof%2FmAMQ%2F6qysJ9Z%2Fx7bG7hwN7z7da6NHBncFE%3D&reserved=0> for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.16.6, Mar 30, 2022
[0]PETSC ERROR: ./iesolver on a  named REXNET-WS4 by jiannantu Thu Aug 18 11:21:55 2022
[0]PETSC ERROR: Configure options --download-f2cblaslapack=yes --with-mpi-dir=/usr/local --with-fc=0 --prefix=/home/jiannantu/petsc --with-scalar-type=complex --with-64-bit-indices=1
[0]PETSC ERROR: #1 MatMult() at /home/jiannantu/petsc-3.16.6/src/mat/interface/matrix.c:2437
[0]PETSC ERROR: #2 PCApply_Mat() at /home/jiannantu/petsc-3.16.6/src/ksp/pc/impls/mat/pcmat.c:9
[0]PETSC ERROR: #3 PCApply() at /home/jiannantu/petsc-3.16.6/src/ksp/pc/interface/precon.c:445
[0]PETSC ERROR: #4 KSP_PCApply() at /home/jiannantu/petsc-3.16.6/include/petsc/private/kspimpl.h:382
[0]PETSC ERROR: #5 KSPInitialResidual() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itres.c:65
[0]PETSC ERROR: #6 KSPSolve_BCGS() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/impls/bcgs/bcgs.c:43
[0]PETSC ERROR: #7 KSPSolve_Private() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itfunc.c:914
[0]PETSC ERROR: #8 KSPSolve() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itfunc.c:1086

From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Thursday, August 18, 2022 8:35 AM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] Using matrix-free with KSP

CAUTION: This email was sent from outside the UMass Lowell network.


  Slow assembly is usually due to under preallocation. You can change to using

MatSetOption(B, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_TRUE);

  to detect if you are under preallocating. See https://petsc.org/main/docs/manual/mat/#sec-matsparse<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanual%2Fmat%2F%23sec-matsparse&data=05%7C01%7CJiannan_Tu%40uml.edu%7C92781c0483d44c98a24508da813b02f8%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637964387727402803%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=YV7wu69YyMaCNJS8KxY4Klhg1zR928MQDOdIvj6cSz4%3D&reserved=0>. That section clearly needs some sprucing up. If it is difficult to determine good preallocation values you can use https://petsc.org/main/docs/manualpages/Mat/MATPREALLOCATOR/<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMATPREALLOCATOR%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7C92781c0483d44c98a24508da813b02f8%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637964387727402803%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=xAB2%2BlqbKiXrY0wKdTYJ72wI%2Bj1TrYpNCscaUTt166s%3D&reserved=0> to compute the needed preallocation efficiently.

  Are you running in parallel? If so how are you determining which rows of entries to compute on each MPI rank? You will want most of the rows to be computed on the rank where the values are stored, you can determine this with https://petsc.org/main/docs/manualpages/Mat/MatGetOwnershipRange.html<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMatGetOwnershipRange.html&data=05%7C01%7CJiannan_Tu%40uml.edu%7C92781c0483d44c98a24508da813b02f8%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637964387727402803%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=v8ueYRddZ0zw8oVm9OpseHsDcQnnGZOndwVF294v4Mc%3D&reserved=0>

  Barry











On Aug 18, 2022, at 12:50 AM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

I implemented the preconditioner. The solver converges quickly when the problem size is small. But when the size increases, say to 100k unknowns, the code hangs at assembly of the preconditioner matrix. The function call sequence is like this

   MatCreate(MPI_COMM_WORLD, &B);
    MatSetType(B, MATMPIAIJ);
    MatSetSizes(B, PETSC_DECIDE, PETSC_DECIDE, N, N);
    MatSetFromOptions(B);

    // The number of diagonal and off diagonal non zeros only can be determined run-time.
    MatMPIAIJSetPreallocation(B, 0, d_nnz, 0, o_nnz);
    MatSetOption(B, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE);

    //here insert values row by row.
MatSetValues(B, 1, &m, nn, nCols, matCols, INSERT_VALUES);

MatAssemblyBegin(B, MAT_FINAL_ASSEMBLY);
MatAssemblyEnd(B, MAT_FINAL_ASSEMBLY);

Could you please tell me what I have done wrong?

Thank you,
Jiannan

From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Tuesday, August 16, 2022 2:00 PM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] Using matrix-free with KSP

CAUTION: This email was sent from outside the UMass Lowell network.


   Create another matrix B that contains "Selected parts of the matrix can be pre-calculated and stored to serve as the preconditioner." When you call KSPSetOperators() pass in B as the second matrix argument.

   Now if literally application of B is the preconditioner you want to use, then simply use PCMAT as the preconditioner, and it will apply B each time it needs to apply the preconditioner. With this, you do not even need a PCShell.

   If your preconditioner uses B to construct your own specific data structure needed to apply your preconditioner, then you can use PCShellSetSetUp() and have your routine call PCGetOperators() to pull out the B matrix and use it to construct your preconditioner.

   If you want PETSc to construct a standard preconditioner from B, then you don't need the PCShell; you simply pass in the B, as above, and use -pc_type type to set the preconditioner you want to use.

   Barry






On Aug 16, 2022, at 12:31 PM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

Barry,

Thank you very much for your instructions. I am sorry I may not ask clearer questions.

I don't use SNES. What I am trying to solve is a very large linear equation system with dense and ill-conditioned matrix. Selected parts of the matrix can be pre-calculated and stored to serve as the preconditioner. My question is how I can apply this matrix as the preconditioner.

Here is the part of code from my linear equations solver. MatrixFreePreconditioner is supposed to be the routine to provide the preconditioner. But I don't understand how to use it. Should I use SNESSetJacobian() even if this is a linear problem?

    MatCreateMFFD(MPI_COMM_WORLD, PETSC_DECIDE, PETSC_DECIDE, N, N, &A);
    MatMFFDSetFunction(A, formfunction, &params);
    MatSetFromOptions(A);
    MatMFFDSetBase(A, X, NULL);

    KSPCreate(MPI_COMM_WORLD, &ksp);
    KSPSetOperators(ksp, A, A);
    KSPSetFromOptions(ksp);

    PC pc;
    KSPGetPC(ksp,&pc);
    PCSetType(pc,PCSHELL);
    PCShellSetApply(pc,MatrixFreePreconditioner);

Thank you,
Jiannan

________________________________
From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Tuesday, August 16, 2022 10:10 AM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] Using matrix-free with KSP

CAUTION: This email was sent from outside the UMass Lowell network.


  I don't fully understand your question so I will answer questions that I do know the answer to :-)

  The function you provide to PCShellSetApply() applies the preconditioner to the input vector and puts the result in the output vector.  This input vector changes at every application of the preconditioner so should not be used in constructing the preconditioner.

  If you are using the standard PETSc matrix-free matrix-vector product via finite difference, -snes_mf_operator or via MatCreateSNESMF() or you are using a MatShell, the matrix-vector product routines input changes for each multiply and should not be used in constructing the preconditioner.

  The function you provide with SNESSetJacobian() is where you can compute anything you need to help build your preconditioner. The input vector to that function is the location at which the Jacobian is needed and is what you should use to build your approximate/part of Jacobian. You can store your approximate/part of Jacobian that you need to compute in the pmat and then in a PCShellSetUp() function that you provide you will compute what you will need latter to apply the preconditioner in your PCShellSetApply() function. Note that the values you put in the pmat do not need to be the full Jacobian; they can be anything you want since this pmat is not used to apply the matrix vector product.

  If this is unclear, feel free to ask additional questions.

  Barry






On Aug 16, 2022, at 9:40 AM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

I am trying to apply a user-defined preconditioner to speed up the convergency of matrix-free KSP solver. The user's manual provides an example of how to call the user-defined preconditioner routine. The routine has PC, input Vec and output Vec as arguments. I want to use part of the Jacobian as preconditioner matrix, the elements of which can be calculated using input Vec. My question is what the role of output Vec from the routine is in matrix-vector product? Or I just have the preconditioner matrix elements calculated in the routine and then Petsc handles applying the preconditioner to matrix-vector product? I used PCSetType(pc, PCSHELL) and PCShellSetApply.

Thank you and appreciate your help!

Jiannan
________________________________
From: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Sent: Wednesday, July 6, 2022 2:34 PM
To: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Cc: Jed Brown <jed at jedbrown.org<mailto:jed at jedbrown.org>>; petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] Using matrix-free with KSP

Barry,

VecScatterCreateToAll solves the problem! The code now gives the correct answer with multiple processes.

Thank you all so much!

Jiannan
________________________________
From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Wednesday, July 6, 2022 2:08 PM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Cc: Jed Brown <jed at jedbrown.org<mailto:jed at jedbrown.org>>; petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] Using matrix-free with KSP


  So your operator is a "dense" operator in that its matrix representation would be essentially dense. In that case, you can use specialized routines to do this efficiently. You can use VecScatterCreateToAll() and then VecScatterBegin/End to do the communication.

  Barry







On Jul 6, 2022, at 1:39 PM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

Jed, thank you very much for the reply.

I'm not sure GlobaltoLocal will work. Say the solution vector is of length 10. Two processes then each process can see 5 elements of the vector. For A x = b (10 equations), each process performs matrix-vector product for five equations. And for each equation i , sum_j A_ij *  x_j = b_i requires the process has access to all 10 unknows x_i (i=0,1,...,9).

Can VecScatter and PetscSF make the entire vector accessible to each process? I am reading the manual and trying to understand how VecScatter and PetscSF work.

Jiannan
________________________________
From: Jed Brown <jed at jedbrown.org<mailto:jed at jedbrown.org>>
Sent: Wednesday, July 6, 2022 10:09 AM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>; Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] Using matrix-free with KSP

You'll usually have a GlobalToLocal operation for each rank to get the halo data it needs, then either a LocalToGlobal to collect the result (e.g., finite element methods) or the local compute will write directly into the owned portion of the global vector. If you're doing the communication "raw", then you may find VecScatter or PetscSF useful to perform the necessary communication.

"Tu, Jiannan" <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> writes:

> Hi Barry,
>
> Following your instructions I implemented the matrix-free method to solve a large linear equation system resulted from a surface integration equation. The KSP solver works fine for single process, but it is problematic with multiple processes. The problem is that each process only can access its own part of solution vector so that each process only conducts part of matrix-vector multiplication. MPI can be used to assemble these partial matrix-vector multiplication together.  Does Petsc provide any ways to implement multi-process matrix-free method?
>
> Thanks,
> Jiannan
> ________________________________
> From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
> Sent: Tuesday, May 24, 2022 2:12 PM
> To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
> Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
> Subject: Re: [petsc-users] Using matrix-free with KSP
>
> This e-mail originated from outside the UMass Lowell network.
> ________________________________
>
>    You can use MatCreateMFFD https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMatCreateMFFD%2F&amp;data=05%7C01%7CJiannan_Tu%40uml.edu%7Cab0859d8d154471590bc08da5f5918d6%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637927133525745809%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=A7wqegTfh94No5BpDiWLK3VxOuR44U2wlWHVm2k7l60%3D&amp;reserved=0<https://urldefense.com/v3/__https://petsc.org/main/docs/manualpages/Mat/MatCreateMFFD/__;!!PVKG_VDCxu5g!stxq2NWGVjUihvHBQC9Rlk0aHVoowcy0PTQYMcQxQ4DqOEC05KSw6TmstSXKckiUgelHzy4ue-d10-zlXkw$><https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMatCreateMFFD%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7C92781c0483d44c98a24508da813b02f8%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637964387727402803%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=JTSUN1mODwX3H7TTnT2QlNnz01QNs1GXMtj6BuzH1b8%3D&reserved=0> MatMFFDSetFunction MatSetFromOptions MatMFFDSetBase and provide the matrix to KSP. Note you will need to use -pc_type none or provide your own custom preconditioner withhttps://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FPC%2FPCSHELL%2F&amp;data=05%7C01%7CJiannan_Tu%40uml.edu%7Cab0859d8d154471590bc08da5f5918d6%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637927133525745809%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=Gb%2F9uFG3jp%2FbfWaSh2cSEQLItHS9CuLwarzN0KcNiJY%3D&amp;reserved=0<https://urldefense.com/v3/__https://petsc.org/main/docs/manualpages/PC/PCSHELL/__;!!PVKG_VDCxu5g!stxq2NWGVjUihvHBQC9Rlk0aHVoowcy0PTQYMcQxQ4DqOEC05KSw6TmstSXKckiUgelHzy4ue-d1cA4fKGw$><https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FPC%2FPCSHELL%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7C92781c0483d44c98a24508da813b02f8%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637964387727559059%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=hVqa56fqRpHKVHfb9nzD7deTWPOTrTOSSYHT59%2FsyVc%3D&reserved=0>
>
>
>
> On May 24, 2022, at 1:21 PM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu><mailto:Jiannan_Tu at uml.edu>> wrote:
>
> I want to use a matrix-free matrix to solve a large linear equation system because the matrix is too large to be stored. Petsc user manual describes matrix-free method for SNES with examples. The matrix-free matrices section explains how to set up such a matrix, but I can't find any example of matrix-free method with KSP. I am wondering how to apply the method to KSP iterative solver in parallel.
>
> I greatly appreciate your help for me to understand this topic.
>
> Jiannan Tu

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220818/5879741a/attachment-0001.html>

From rohany at alumni.cmu.edu  Thu Aug 18 14:55:18 2022
From: rohany at alumni.cmu.edu (Rohan Yadav)
Date: Thu, 18 Aug 2022 15:55:18 -0400
Subject: [petsc-users] `petsc4py` GPU support
Message-ID: <CAHN2_8wGxWv3Wge2zLv_Ee2jrYjvx0sS+soqXCUiW36CXYyT5g@mail.gmail.com>

Hi,

I'm trying to use `petsc4py` on a machine with CUDA GPUs. Is there a way to
download a version of `petsc4py` from pip or conda with GPU support? If
not, how do I go about installing `petsc4py` with GPU support from source?
I can't find much documentation around doing this.

Thanks,

Rohan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220818/e64581c1/attachment.html>

From balay at mcs.anl.gov  Thu Aug 18 15:31:17 2022
From: balay at mcs.anl.gov (Satish Balay)
Date: Thu, 18 Aug 2022 15:31:17 -0500 (CDT)
Subject: [petsc-users] `petsc4py` GPU support
In-Reply-To: <CAHN2_8wGxWv3Wge2zLv_Ee2jrYjvx0sS+soqXCUiW36CXYyT5g@mail.gmail.com>
References: <CAHN2_8wGxWv3Wge2zLv_Ee2jrYjvx0sS+soqXCUiW36CXYyT5g@mail.gmail.com>
Message-ID: <23c0b56f-d4f8-6dca-9886-297726853aa0@mcs.anl.gov>

You could look at PETSc install instructions and examples https://petsc.org/release/install/install/#common-example-usages

Assuming a linux box with  cuda at /usr/local/cuda, mpicc etc in PATH, and have system blas/lapack installed, the following build process should work

./configure --with-cuda=1 --with-petsc4py && make && make check

Satish

On Thu, 18 Aug 2022, Rohan Yadav wrote:

> Hi,
> 
> I'm trying to use `petsc4py` on a machine with CUDA GPUs. Is there a way to
> download a version of `petsc4py` from pip or conda with GPU support? If
> not, how do I go about installing `petsc4py` with GPU support from source?
> I can't find much documentation around doing this.
> 
> Thanks,
> 
> Rohan
> 


From rohany at alumni.cmu.edu  Thu Aug 18 15:36:20 2022
From: rohany at alumni.cmu.edu (Rohan Yadav)
Date: Thu, 18 Aug 2022 16:36:20 -0400
Subject: [petsc-users] `petsc4py` GPU support
In-Reply-To: <23c0b56f-d4f8-6dca-9886-297726853aa0@mcs.anl.gov>
References: <CAHN2_8wGxWv3Wge2zLv_Ee2jrYjvx0sS+soqXCUiW36CXYyT5g@mail.gmail.com>
	<23c0b56f-d4f8-6dca-9886-297726853aa0@mcs.anl.gov>
Message-ID: <CAHN2_8wvNH3NJd0WLQjFO_WQB7-MJjQGLtFLfxBRJt9h=qwOLg@mail.gmail.com>

I see. To confirm:
1) There are no installations of `petsc4py` with GPU support available
through a package manager
2) Building and configuring PETSc with GPU support will also build a
version of `petsc4py`.

Thanks,
Rohan

On Thu, Aug 18, 2022 at 4:31 PM Satish Balay <balay at mcs.anl.gov> wrote:

> You could look at PETSc install instructions and examples
> https://petsc.org/release/install/install/#common-example-usages
>
> Assuming a linux box with  cuda at /usr/local/cuda, mpicc etc in PATH, and
> have system blas/lapack installed, the following build process should work
>
> ./configure --with-cuda=1 --with-petsc4py && make && make check
>
> Satish
>
> On Thu, 18 Aug 2022, Rohan Yadav wrote:
>
> > Hi,
> >
> > I'm trying to use `petsc4py` on a machine with CUDA GPUs. Is there a way
> to
> > download a version of `petsc4py` from pip or conda with GPU support? If
> > not, how do I go about installing `petsc4py` with GPU support from
> source?
> > I can't find much documentation around doing this.
> >
> > Thanks,
> >
> > Rohan
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220818/83d4ca97/attachment.html>

From balay at mcs.anl.gov  Thu Aug 18 15:53:36 2022
From: balay at mcs.anl.gov (Satish Balay)
Date: Thu, 18 Aug 2022 15:53:36 -0500 (CDT)
Subject: [petsc-users] `petsc4py` GPU support
In-Reply-To: <CAHN2_8wvNH3NJd0WLQjFO_WQB7-MJjQGLtFLfxBRJt9h=qwOLg@mail.gmail.com>
References: <CAHN2_8wGxWv3Wge2zLv_Ee2jrYjvx0sS+soqXCUiW36CXYyT5g@mail.gmail.com>
	<23c0b56f-d4f8-6dca-9886-297726853aa0@mcs.anl.gov>
	<CAHN2_8wvNH3NJd0WLQjFO_WQB7-MJjQGLtFLfxBRJt9h=qwOLg@mail.gmail.com>
Message-ID: <dafd8fd4-5789-55d4-f772-d6fdeb12a95@mcs.anl.gov>

On Thu, 18 Aug 2022, Rohan Yadav wrote:

> I see. To confirm:
> 1) There are no installations of `petsc4py` with GPU support available
> through a package manager

You could try via pip - but its not any more simple. Here the model is - you install petsc via pip - and then petsc4py. Here you would specify appropriate build options to PETSc via PETSC_CONFIGURE_OPTIONS env variable [as MPI, BlasLapack, CUDA are specific to your machine - and pip doesn't really now about them] 


> 2) Building and configuring PETSc with GPU support will also build a
> version of `petsc4py`.

PETSc configure option --with-petsc4py=1 will also build PETSc4py [along with PETSc]

Satish


> 
> Thanks,
> Rohan
> 
> On Thu, Aug 18, 2022 at 4:31 PM Satish Balay <balay at mcs.anl.gov> wrote:
> 
> > You could look at PETSc install instructions and examples
> > https://petsc.org/release/install/install/#common-example-usages
> >
> > Assuming a linux box with  cuda at /usr/local/cuda, mpicc etc in PATH, and
> > have system blas/lapack installed, the following build process should work
> >
> > ./configure --with-cuda=1 --with-petsc4py && make && make check
> >
> > Satish
> >
> > On Thu, 18 Aug 2022, Rohan Yadav wrote:
> >
> > > Hi,
> > >
> > > I'm trying to use `petsc4py` on a machine with CUDA GPUs. Is there a way
> > to
> > > download a version of `petsc4py` from pip or conda with GPU support? If
> > > not, how do I go about installing `petsc4py` with GPU support from
> > source?
> > > I can't find much documentation around doing this.
> > >
> > > Thanks,
> > >
> > > Rohan
> > >
> >
> >
> 


From knepley at gmail.com  Thu Aug 18 16:14:03 2022
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 18 Aug 2022 14:14:03 -0700
Subject: [petsc-users] DMPlex Transitive closure
In-Reply-To: <CAB1dcV6ZYs1mD3L2We7FVukix86Qq5L43Th9SGcKAz5STZGSyQ@mail.gmail.com>
References: <CAB1dcV6ZYs1mD3L2We7FVukix86Qq5L43Th9SGcKAz5STZGSyQ@mail.gmail.com>
Message-ID: <CAMYG4GmHKnGmKHVH_LrcgnNfvMQHQo0eSE0+AZGfiz+1T3Q-Kw@mail.gmail.com>

On Wed, Aug 17, 2022 at 8:31 AM Nicholas Arnold-Medabalimi <
narnoldm at umich.edu> wrote:

> Good Morning
>
> I have been working with the transitive closure to check to see if a node
> point has a cell that is rooted on a process. However, I think I am having
> a more fundamental issue. Even just calling the RestoreTransitiveClosure
> immediately after getting it is throwing a PetscError.(that is removing all
> the calls between the two closure functions. I'm not exactly sure what is
> causing the issue. I have used this method previously, except on cell
> points to get the constituent vertex points. Is there some nuance to using
> the supports instead of the cones?
>

You should initialize closure = NULL in its declaration.

  Thanks,

     Matt


>
> DMPlexGetDepthStratum(dm, 0, &pStart, &pEnd);
> PetscPrintf(PETSC_COMM_WORLD, "Pstart: %d, Pend: %d\n", pStart, pEnd);
> for (int p = pStart; p < pEnd; p++)
>     {
>
>         PetscInt closureSize;
>         PetscInt *closure;
>
>         ierr = DMPlexGetTransitiveClosure(dm, p, PETSC_FALSE, &closureSize,
> &closure);
>         CHKERRQ(ierr);
>         PetscSynchronizedPrintf(PETSC_COMM_WORLD, "rank:%d\n", rank);
>         for (int i = 0; i < (2 * closureSize); i = i + 2)
>         {
>             PetscSynchronizedPrintf(PETSC_COMM_WORLD, "%d\n", closure[i]);
>         }
>         PetscSynchronizedFlush(PETSC_COMM_WORLD, NULL);
>         ierr = DMPlexRestoreTransitiveClosure(dm, p, PETSC_FALSE, &
> closureSize, &closure);
>         CHKERRQ(ierr);
>     }
>
> Results in an error when the RestoreTransitiveClosure is called. I have
> checked and the resulting output for the closures are correct.
>
> Pstart: 510, Pend: 1532
> rank:0
> 510
> 1534
> 1535
> 0
> rank:1
> 492
> 1478
> 1479
> 0
> [1]PETSC ERROR: 2953
> --------------------- Error Message
> --------------------------------------------------------------
> [1]PETSC ERROR: 491
> Object is in wrong state
> [1]PETSC ERROR: Array was not checked out
> [1]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
> [1]PETSC ERROR: Petsc Release Version 3.17.4, Aug 01, 2022
> [1]PETSC ERROR: /home/narnoldm/code/solver-playground/build/bin/pgrid_c on
> a  by narnoldm Wed Aug 17 11:18:20 2022
> [0]PETSC ERROR: [1]PETSC ERROR: Configure options
> --prefix=/home/narnoldm/code/solver-playground/build/external/PETSc
> --with-hdf5-dir=/home/narnoldm/code/solver-playground/build/external/HDF5
> --with-cgns-dir=/home/narnoldm/code/solver-playground/build/external/CGNS
> --download-triangle --download-parmetis --download-metis --with-debugging=1
>
> Any assistance is greatly appreciated.
>
> Sincerely
> Nicholas
>
> --
> Nicholas Arnold-Medabalimi
>
> Ph.D. Candidate
> Computational Aeroscience Lab
> University of Michigan
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220818/8a89d618/attachment.html>

From bsmith at petsc.dev  Thu Aug 18 18:43:51 2022
From: bsmith at petsc.dev (Barry Smith)
Date: Thu, 18 Aug 2022 19:43:51 -0400
Subject: [petsc-users] Using matrix-free with KSP
In-Reply-To: <BY5PR02MB6471F2C058B84DE092567731E76D9@BY5PR02MB6471.namprd02.prod.outlook.com>
References: <BY5PR02MB6471969F80FA5F732FC74D87E7D79@BY5PR02MB6471.namprd02.prod.outlook.com>
	<B55CFC2C-6364-4B78-8E99-9D53700743B8@petsc.dev>
	<BY5PR02MB6471BDF90783FE3A0EA176CBE7809@BY5PR02MB6471.namprd02.prod.outlook.com>
	<87iloa49kt.fsf@jedbrown.org>
	<BY5PR02MB6471C0ED04769D02046F68ACE7809@BY5PR02MB6471.namprd02.prod.outlook.com>
	<00EBE4B8-22E5-408F-B56D-14226A278D7E@petsc.dev>
	<BY5PR02MB6471E386057DDDE98AD05DFAE7809@BY5PR02MB6471.namprd02.prod.outlook.com>
	<BY5PR02MB647140084C7E7CDDB6BC2A11E76B9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<A6629DE1-5A7F-414E-B164-A69877158675@petsc.dev>
	<BY5PR02MB6471AB1C004920DA1FC9B43EE76B9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<103146FD-FF28-47CE-B661-F2B521BF0E7A@petsc.dev>
	<DM6PR02MB6478B2FB8CB1BE757CE7B003E76D9@DM6PR02MB6478.namprd02.prod.outlook.com>
	<4C00F4B8-BB42-4932-BCD7-527E5FB39BFF@petsc.dev>
	<BY5PR02MB64710E70758E431108D66956E76D9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<D0EB13D8-C5A6-4AE7-AFDE-2CDCCACBAF73@petsc.dev>
	<BY5PR02MB6471D4B05874DECFDDA13FFCE76D9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<367B124C-88BB-4889-A4AF-45A1D473078D@petsc.dev>
	<BY5PR02MB6471F2C058B84DE092567731E76D9@BY5PR02MB6471.namprd02.prod.outlook.com>
Message-ID: <944DDEED-016D-43B2-A5A6-7BF2E8E41FA5@petsc.dev>



   Is it in the first iteration of the first KSPSolve that you get this message? Or in the second call to KSPSolve? 

  The error comes from a matrix vector product with your A matrix, what is that and where did it come from. The error message is not from your B matrix, that is not used in the MatMult()


> On Aug 18, 2022, at 3:05 PM, Tu, Jiannan <Jiannan_Tu at uml.edu> wrote:
> 
> Barry, 
>  
> Thanks. The method works. Assembly of actual matrix now completes quickly. The convergency is much better for the first iteration. The residual normal is in the order of 10^-10 instead of 10^-2. However, there is an error message after the first iteration: ?Object is in wrong state. Not for unassembled matrix? even though both preallocator and actual matrix have been assembled (see the code snippet below). Could you please give me some hints what?s going wrong? I use KSP type bcgs and PC type mat
>  
> Thank you,
> Jiannan
>  
> -----------------------------------------------------
> Solve linear equations ...
>   0 KSP Residual norm 9.841179147519e-10
> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [0]PETSC ERROR: Object is in wrong state
> [0]PETSC ERROR: Not for unassembled matrix
> [0]PETSC ERROR: See https://petsc.org/release/faq/ <https://petsc.org/release/faq/> for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.16.6, Mar 30, 2022
> [0]PETSC ERROR: ./iesolver on a  named REXNET-WS4 by jiannantu Thu Aug 18 14:48:43 2022
> [0]PETSC ERROR: Configure options --download-f2cblaslapack=yes --with-mpi-dir=/usr/local --with-fc=0 --prefix=/home/jiannantu/petsc --with-scalar-type=complex --with-64-bit-indices=1
> [0]PETSC ERROR: #1 MatMult() at /home/jiannantu/petsc-3.16.6/src/mat/interface/matrix.c:2425
> [0]PETSC ERROR: #2 PCApplyBAorAB() at /home/jiannantu/petsc-3.16.6/src/ksp/pc/interface/precon.c:730
> [0]PETSC ERROR: #3 KSP_PCApplyBAorAB() at /home/jiannantu/petsc-3.16.6/include/petsc/private/kspimpl.h:421
> [0]PETSC ERROR:  Linear equations not solved. Diverged reason: 0
> #4 KSPSolve_BCGS() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/impls/bcgs/bcgs.c:87
> [0]PETSC ERROR: #5 KSPSolve_Private() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itfunc.c:914
> [0]PETSC ERROR: #6 KSPSolve() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itfunc.c:1086
>  
> -----------------------------------------------
>     MatCreate(MPI_COMM_WORLD, &B);
>     MatSetType(B, MATMPIAIJ);
>     MatSetSizes(B, PETSC_DECIDE, PETSC_DECIDE, N, N);
>  
>     Mat preall;
>     MatCreate(MPI_COMM_WORLD, &preall);
>     MatSetType(preall, MATPREALLOCATOR);
>     MatSetSizes(preall, PETSC_DECIDE, PETSC_DECIDE, N, N);
>  
>     MatMPIAIJSetPreallocation(preall, 0, d_nnz, 0, o_nnz);
>     MatSetUp(preall);
> MatSetOption(preall, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE);
>  
> //set values row by row
> MatSetValues(preall, 1, &m, 1, nCols, matCols, INSERT_VALUES);
>  
> MatAssemblyBegin(preall, MAT_FINAL_ASSEMBLY);
> MatAssemblyEnd(preall, MAT_FINAL_ASSEMBLY);
>  
> //set values for actual matrix row by row
> MatSetValues(B, 1, &m, 1, nCols, matCols, INSERT_VALUES);
>  
> MatAssemblyBegin(B, MAT_FINAL_ASSEMBLY);
> MatAssemblyEnd(B, MAT_FINAL_ASSEMBLY);
>  
>     KSPCreate(MPI_COMM_WORLD, &ksp);
>     KSPSetOperators(ksp, A, B);
>     KSPSetFromOptions(ksp);
>  
>     PC pc;
>     KSPGetPC(ksp, &pc);
>     PCSetFromOptions(pc);
>  
> From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>> 
> Sent: Thursday, August 18, 2022 12:59 PM
> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
> Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>
> Subject: Re: [petsc-users] Using matrix-free with KSP
>  
> CAUTION: This email was sent from outside the UMass Lowell network.
>  
>  
>   Yes, once you have preallocated the real matrix you can destroy the preallocation matrix whose only job is to gather the preallocation information
> 
> 
> On Aug 18, 2022, at 12:52 PM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> wrote:
>  
> Thanks, Barry.
>  
> So I need actually two matrixes, one for preallocator and one for actual matrix that can be passed to KSPSetOperators(). -mat_type preallocator option is used to speed up doing insertion into preallocator, then use MatPreallocatorPreallocate() to preallocate actual matrix and do actual insertion of values into it., right?
>  
> The code runs in parallel. Each process owns number of rows that equals to number of unknowns (that is, xm in 1D DM) it owns.
>  
> Jiannan
>  
> From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>> 
> Sent: Thursday, August 18, 2022 11:37 AM
> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
> Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>
> Subject: Re: [petsc-users] Using matrix-free with KSP
>  
> CAUTION: This email was sent from outside the UMass Lowell network.
>  
>  
>    The preallocator MatType matrix cannot be passed to the KSPSetOperators(), you need to create an actual matrix, for example MATAIJ and use the preallocator to set its preallocation and then fill up the MATAIJ matrix with the usual MatSetValues and pass that matrix to the KSPSetOperators.
> 
> 
> 
> On Aug 18, 2022, at 11:30 AM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> wrote:
>  
> Hi Barry,
>  
> The MATPREALLOCATOR solved problem of slow matrix assembly. But the solver failed because of ?Matrix type preallocator does not have a multiply defined?.
>  
> I guess this is because the matrix-free is used. I am wondering how a preconditioner is applied to the matrix-vector multiply in Petsc.
>  
> Thank you,
> Jiannan
>  
> [0]PETSC ERROR: No support for this operation for this object type
> [0]PETSC ERROR: Matrix type preallocator does not have a multiply defined
> [0]PETSC ERROR: See https://petsc.org/release/faq/ <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Frelease%2Ffaq%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7C92781c0483d44c98a24508da813b02f8%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637964387727402803%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=vfijXLof%2FmAMQ%2F6qysJ9Z%2Fx7bG7hwN7z7da6NHBncFE%3D&reserved=0> for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.16.6, Mar 30, 2022
> [0]PETSC ERROR: ./iesolver on a  named REXNET-WS4 by jiannantu Thu Aug 18 11:21:55 2022
> [0]PETSC ERROR: Configure options --download-f2cblaslapack=yes --with-mpi-dir=/usr/local --with-fc=0 --prefix=/home/jiannantu/petsc --with-scalar-type=complex --with-64-bit-indices=1
> [0]PETSC ERROR: #1 MatMult() at /home/jiannantu/petsc-3.16.6/src/mat/interface/matrix.c:2437
> [0]PETSC ERROR: #2 PCApply_Mat() at /home/jiannantu/petsc-3.16.6/src/ksp/pc/impls/mat/pcmat.c:9
> [0]PETSC ERROR: #3 PCApply() at /home/jiannantu/petsc-3.16.6/src/ksp/pc/interface/precon.c:445
> [0]PETSC ERROR: #4 KSP_PCApply() at /home/jiannantu/petsc-3.16.6/include/petsc/private/kspimpl.h:382
> [0]PETSC ERROR: #5 KSPInitialResidual() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itres.c:65
> [0]PETSC ERROR: #6 KSPSolve_BCGS() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/impls/bcgs/bcgs.c:43
> [0]PETSC ERROR: #7 KSPSolve_Private() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itfunc.c:914
> [0]PETSC ERROR: #8 KSPSolve() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itfunc.c:1086
>  
> From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>> 
> Sent: Thursday, August 18, 2022 8:35 AM
> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
> Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>
> Subject: Re: [petsc-users] Using matrix-free with KSP
>  
> CAUTION: This email was sent from outside the UMass Lowell network.
>  
>  
>   Slow assembly is usually due to under preallocation. You can change to using 
>  
> MatSetOption(B, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_TRUE);
>  
>   to detect if you are under preallocating. See https://petsc.org/main/docs/manual/mat/#sec-matsparse <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanual%2Fmat%2F%23sec-matsparse&data=05%7C01%7CJiannan_Tu%40uml.edu%7C92781c0483d44c98a24508da813b02f8%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637964387727402803%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=YV7wu69YyMaCNJS8KxY4Klhg1zR928MQDOdIvj6cSz4%3D&reserved=0>. That section clearly needs some sprucing up. If it is difficult to determine good preallocation values you can use https://petsc.org/main/docs/manualpages/Mat/MATPREALLOCATOR/ <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMATPREALLOCATOR%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7C92781c0483d44c98a24508da813b02f8%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637964387727402803%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=xAB2%2BlqbKiXrY0wKdTYJ72wI%2Bj1TrYpNCscaUTt166s%3D&reserved=0> to compute the needed preallocation efficiently.
>  
>   Are you running in parallel? If so how are you determining which rows of entries to compute on each MPI rank? You will want most of the rows to be computed on the rank where the values are stored, you can determine this with https://petsc.org/main/docs/manualpages/Mat/MatGetOwnershipRange.html <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMatGetOwnershipRange.html&data=05%7C01%7CJiannan_Tu%40uml.edu%7C92781c0483d44c98a24508da813b02f8%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637964387727402803%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=v8ueYRddZ0zw8oVm9OpseHsDcQnnGZOndwVF294v4Mc%3D&reserved=0>
>  
>   Barry
>  
>  
> 
> 
> 
> 
>  
> 
> 
> 
> 
> On Aug 18, 2022, at 12:50 AM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> wrote:
>  
> I implemented the preconditioner. The solver converges quickly when the problem size is small. But when the size increases, say to 100k unknowns, the code hangs at assembly of the preconditioner matrix. The function call sequence is like this
>  
>    MatCreate(MPI_COMM_WORLD, &B);
>     MatSetType(B, MATMPIAIJ);
>     MatSetSizes(B, PETSC_DECIDE, PETSC_DECIDE, N, N);
>     MatSetFromOptions(B);
>  
>     // The number of diagonal and off diagonal non zeros only can be determined run-time.
>     MatMPIAIJSetPreallocation(B, 0, d_nnz, 0, o_nnz);
>     MatSetOption(B, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE);
>  
>     //here insert values row by row.
> MatSetValues(B, 1, &m, nn, nCols, matCols, INSERT_VALUES);
>  
> MatAssemblyBegin(B, MAT_FINAL_ASSEMBLY);
> MatAssemblyEnd(B, MAT_FINAL_ASSEMBLY);
>  
> Could you please tell me what I have done wrong?
>  
> Thank you,
> Jiannan
>    
> From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>> 
> Sent: Tuesday, August 16, 2022 2:00 PM
> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
> Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>
> Subject: Re: [petsc-users] Using matrix-free with KSP
>  
> CAUTION: This email was sent from outside the UMass Lowell network.
>  
>  
>    Create another matrix B that contains "Selected parts of the matrix can be pre-calculated and stored to serve as the preconditioner." When you call KSPSetOperators() pass in B as the second matrix argument.
>  
>    Now if literally application of B is the preconditioner you want to use, then simply use PCMAT as the preconditioner, and it will apply B each time it needs to apply the preconditioner. With this, you do not even need a PCShell. 
>  
>    If your preconditioner uses B to construct your own specific data structure needed to apply your preconditioner, then you can use PCShellSetSetUp() and have your routine call PCGetOperators() to pull out the B matrix and use it to construct your preconditioner. 
>  
>    If you want PETSc to construct a standard preconditioner from B, then you don't need the PCShell; you simply pass in the B, as above, and use -pc_type type to set the preconditioner you want to use. 
>  
>    Barry
>  
> 
> 
> 
> 
> 
> On Aug 16, 2022, at 12:31 PM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> wrote:
>  
> Barry,
>  
> Thank you very much for your instructions. I am sorry I may not ask clearer questions.
>  
> I don't use SNES. What I am trying to solve is a very large linear equation system with dense and ill-conditioned matrix. Selected parts of the matrix can be pre-calculated and stored to serve as the preconditioner. My question is how I can apply this matrix as the preconditioner. 
>  
> Here is the part of code from my linear equations solver. MatrixFreePreconditioner is supposed to be the routine to provide the preconditioner. But I don't understand how to use it. Should I use SNESSetJacobian() even if this is a linear problem? 
>  
>     MatCreateMFFD(MPI_COMM_WORLD, PETSC_DECIDE, PETSC_DECIDE, N, N, &A);
>     MatMFFDSetFunction(A, formfunction, &params);
>     MatSetFromOptions(A);
>     MatMFFDSetBase(A, X, NULL);
>  
>     KSPCreate(MPI_COMM_WORLD, &ksp);
>     KSPSetOperators(ksp, A, A);
>     KSPSetFromOptions(ksp);
>  
>     PC pc;
>     KSPGetPC(ksp,&pc);
>     PCSetType(pc,PCSHELL);
>     PCShellSetApply(pc,MatrixFreePreconditioner);
>  
> Thank you,
> Jiannan
>  
> From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>>
> Sent: Tuesday, August 16, 2022 10:10 AM
> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
> Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>>
> Subject: Re: [petsc-users] Using matrix-free with KSP
>  
> CAUTION: This email was sent from outside the UMass Lowell network.
>  
>  
>   I don't fully understand your question so I will answer questions that I do know the answer to :-)
>  
>   The function you provide to PCShellSetApply() applies the preconditioner to the input vector and puts the result in the output vector.  This input vector changes at every application of the preconditioner so should not be used in constructing the preconditioner.
>  
>   If you are using the standard PETSc matrix-free matrix-vector product via finite difference, -snes_mf_operator or via MatCreateSNESMF() or you are using a MatShell, the matrix-vector product routines input changes for each multiply and should not be used in constructing the preconditioner.
>  
>   The function you provide with SNESSetJacobian() is where you can compute anything you need to help build your preconditioner. The input vector to that function is the location at which the Jacobian is needed and is what you should use to build your approximate/part of Jacobian. You can store your approximate/part of Jacobian that you need to compute in the pmat and then in a PCShellSetUp() function that you provide you will compute what you will need latter to apply the preconditioner in your PCShellSetApply() function. Note that the values you put in the pmat do not need to be the full Jacobian; they can be anything you want since this pmat is not used to apply the matrix vector product.
>  
>   If this is unclear, feel free to ask additional questions.
>  
>   Barry
>  
> 
> 
> 
> 
> 
> On Aug 16, 2022, at 9:40 AM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> wrote:
>  
> I am trying to apply a user-defined preconditioner to speed up the convergency of matrix-free KSP solver. The user's manual provides an example of how to call the user-defined preconditioner routine. The routine has PC, input Vec and output Vec as arguments. I want to use part of the Jacobian as preconditioner matrix, the elements of which can be calculated using input Vec. My question is what the role of output Vec from the routine is in matrix-vector product? Or I just have the preconditioner matrix elements calculated in the routine and then Petsc handles applying the preconditioner to matrix-vector product? I used PCSetType(pc, PCSHELL) and PCShellSetApply.
>  
> Thank you and appreciate your help!
>  
> Jiannan 
> From: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
> Sent: Wednesday, July 6, 2022 2:34 PM
> To: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>>
> Cc: Jed Brown <jed at jedbrown.org <mailto:jed at jedbrown.org>>; petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>>
> Subject: Re: [petsc-users] Using matrix-free with KSP
>  
> Barry,
>  
> VecScatterCreateToAll solves the problem! The code now gives the correct answer with multiple processes. 
>  
> Thank you all so much!
>  
> Jiannan
> From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>>
> Sent: Wednesday, July 6, 2022 2:08 PM
> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
> Cc: Jed Brown <jed at jedbrown.org <mailto:jed at jedbrown.org>>; petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>>
> Subject: Re: [petsc-users] Using matrix-free with KSP
>  
>  
>   So your operator is a "dense" operator in that its matrix representation would be essentially dense. In that case, you can use specialized routines to do this efficiently. You can use VecScatterCreateToAll() and then VecScatterBegin/End to do the communication.
>  
>   Barry
>  
>  
> 
> 
> 
> 
> 
> On Jul 6, 2022, at 1:39 PM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> wrote:
>  
> Jed, thank you very much for the reply. 
>  
> I'm not sure GlobaltoLocal will work. Say the solution vector is of length 10. Two processes then each process can see 5 elements of the vector. For A x = b (10 equations), each process performs matrix-vector product for five equations. And for each equation i , sum_j A_ij *  x_j = b_i requires the process has access to all 10 unknows x_i (i=0,1,...,9). 
>  
> Can VecScatter and PetscSF make the entire vector accessible to each process? I am reading the manual and trying to understand how VecScatter and PetscSF work.
>  
> Jiannan
> From: Jed Brown <jed at jedbrown.org <mailto:jed at jedbrown.org>>
> Sent: Wednesday, July 6, 2022 10:09 AM
> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>; Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>>
> Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>>
> Subject: Re: [petsc-users] Using matrix-free with KSP
>  
> You'll usually have a GlobalToLocal operation for each rank to get the halo data it needs, then either a LocalToGlobal to collect the result (e.g., finite element methods) or the local compute will write directly into the owned portion of the global vector. If you're doing the communication "raw", then you may find VecScatter or PetscSF useful to perform the necessary communication.
> 
> "Tu, Jiannan" <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> writes:
> 
> > Hi Barry,
> >
> > Following your instructions I implemented the matrix-free method to solve a large linear equation system resulted from a surface integration equation. The KSP solver works fine for single process, but it is problematic with multiple processes. The problem is that each process only can access its own part of solution vector so that each process only conducts part of matrix-vector multiplication. MPI can be used to assemble these partial matrix-vector multiplication together.  Does Petsc provide any ways to implement multi-process matrix-free method?
> >
> > Thanks,
> > Jiannan
> > ________________________________
> > From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>>
> > Sent: Tuesday, May 24, 2022 2:12 PM
> > To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
> > Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>>
> > Subject: Re: [petsc-users] Using matrix-free with KSP
> >
> > This e-mail originated from outside the UMass Lowell network.
> > ________________________________
> >
> >    You can use MatCreateMFFD https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMatCreateMFFD%2F&amp;data=05%7C01%7CJiannan_Tu%40uml.edu%7Cab0859d8d154471590bc08da5f5918d6%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637927133525745809%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=A7wqegTfh94No5BpDiWLK3VxOuR44U2wlWHVm2k7l60%3D&amp;reserved=0<https://urldefense.com/v3/__https://petsc.org/main/docs/manualpages/Mat/MatCreateMFFD/__;!!PVKG_VDCxu5g!stxq2NWGVjUihvHBQC9Rlk0aHVoowcy0PTQYMcQxQ4DqOEC05KSw6TmstSXKckiUgelHzy4ue-d10-zlXkw$> <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMatCreateMFFD%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7C92781c0483d44c98a24508da813b02f8%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637964387727402803%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=JTSUN1mODwX3H7TTnT2QlNnz01QNs1GXMtj6BuzH1b8%3D&reserved=0> MatMFFDSetFunction MatSetFromOptions MatMFFDSetBase and provide the matrix to KSP. Note you will need to use -pc_type none or provide your own custom preconditioner withhttps://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FPC%2FPCSHELL%2F&amp;data=05%7C01%7CJiannan_Tu%40uml.edu%7Cab0859d8d154471590bc08da5f5918d6%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637927133525745809%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=Gb%2F9uFG3jp%2FbfWaSh2cSEQLItHS9CuLwarzN0KcNiJY%3D&amp;reserved=0<https://urldefense.com/v3/__https://petsc.org/main/docs/manualpages/PC/PCSHELL/__;!!PVKG_VDCxu5g!stxq2NWGVjUihvHBQC9Rlk0aHVoowcy0PTQYMcQxQ4DqOEC05KSw6TmstSXKckiUgelHzy4ue-d1cA4fKGw$> <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FPC%2FPCSHELL%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7C92781c0483d44c98a24508da813b02f8%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637964387727559059%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=hVqa56fqRpHKVHfb9nzD7deTWPOTrTOSSYHT59%2FsyVc%3D&reserved=0>
> >
> >
> >
> > On May 24, 2022, at 1:21 PM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu><mailto:Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>> wrote:
> >
> > I want to use a matrix-free matrix to solve a large linear equation system because the matrix is too large to be stored. Petsc user manual describes matrix-free method for SNES with examples. The matrix-free matrices section explains how to set up such a matrix, but I can't find any example of matrix-free method with KSP. I am wondering how to apply the method to KSP iterative solver in parallel.
> >
> > I greatly appreciate your help for me to understand this topic.
> >
> > Jiannan Tu

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220818/3d686e2f/attachment-0001.html>

From Jiannan_Tu at uml.edu  Thu Aug 18 22:41:02 2022
From: Jiannan_Tu at uml.edu (Tu, Jiannan)
Date: Fri, 19 Aug 2022 03:41:02 +0000
Subject: [petsc-users] Using matrix-free with KSP
In-Reply-To: <944DDEED-016D-43B2-A5A6-7BF2E8E41FA5@petsc.dev>
References: <BY5PR02MB6471969F80FA5F732FC74D87E7D79@BY5PR02MB6471.namprd02.prod.outlook.com>
	<B55CFC2C-6364-4B78-8E99-9D53700743B8@petsc.dev>
	<BY5PR02MB6471BDF90783FE3A0EA176CBE7809@BY5PR02MB6471.namprd02.prod.outlook.com>
	<87iloa49kt.fsf@jedbrown.org>
	<BY5PR02MB6471C0ED04769D02046F68ACE7809@BY5PR02MB6471.namprd02.prod.outlook.com>
	<00EBE4B8-22E5-408F-B56D-14226A278D7E@petsc.dev>
	<BY5PR02MB6471E386057DDDE98AD05DFAE7809@BY5PR02MB6471.namprd02.prod.outlook.com>
	<BY5PR02MB647140084C7E7CDDB6BC2A11E76B9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<A6629DE1-5A7F-414E-B164-A69877158675@petsc.dev>
	<BY5PR02MB6471AB1C004920DA1FC9B43EE76B9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<103146FD-FF28-47CE-B661-F2B521BF0E7A@petsc.dev>
	<DM6PR02MB6478B2FB8CB1BE757CE7B003E76D9@DM6PR02MB6478.namprd02.prod.outlook.com>
	<4C00F4B8-BB42-4932-BCD7-527E5FB39BFF@petsc.dev>
	<BY5PR02MB64710E70758E431108D66956E76D9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<D0EB13D8-C5A6-4AE7-AFDE-2CDCCACBAF73@petsc.dev>
	<BY5PR02MB6471D4B05874DECFDDA13FFCE76D9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<367B124C-88BB-4889-A4AF-45A1D473078D@petsc.dev>
	<BY5PR02MB6471F2C058B84DE092567731E76D9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<944DDEED-016D-43B2-A5A6-7BF2E8E41FA5@petsc.dev>
Message-ID: <BY5PR02MB6471CD0236F03D20763A3B64E76C9@BY5PR02MB6471.namprd02.prod.outlook.com>

The program calls KSPSolve just once and the error occurs after the first iteration.

I realized the matrix-vector product in my case is different from that approximating matrix-vector multiply by finite difference of function vector. It consists of two parts. One is multiplication of part of the matrix with part of vector, and another part results from so called fast multipole method. I'm not clear how to relate such calculation of matrix-vector product to the finite differencing of function vector.



From: Barry Smith <bsmith at petsc.dev>
Sent: Thursday, August 18, 2022 7:44 PM
To: Tu, Jiannan <Jiannan_Tu at uml.edu>
Cc: petsc-users at mcs.anl.gov
Subject: Re: [petsc-users] Using matrix-free with KSP

CAUTION: This email was sent from outside the UMass Lowell network.



   Is it in the first iteration of the first KSPSolve that you get this message? Or in the second call to KSPSolve?

  The error comes from a matrix vector product with your A matrix, what is that and where did it come from. The error message is not from your B matrix, that is not used in the MatMult()



On Aug 18, 2022, at 3:05 PM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

Barry,

Thanks. The method works. Assembly of actual matrix now completes quickly. The convergency is much better for the first iteration. The residual normal is in the order of 10^-10 instead of 10^-2. However, there is an error message after the first iteration: "Object is in wrong state. Not for unassembled matrix" even though both preallocator and actual matrix have been assembled (see the code snippet below). Could you please give me some hints what's going wrong? I use KSP type bcgs and PC type mat

Thank you,
Jiannan

-----------------------------------------------------
Solve linear equations ...
  0 KSP Residual norm 9.841179147519e-10
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Object is in wrong state
[0]PETSC ERROR: Not for unassembled matrix
[0]PETSC ERROR: See https://petsc.org/release/faq/<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Frelease%2Ffaq%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7C73f4b5b4a7244d57063108da81738355%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637964630446347701%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=EvNUIKJN2gp1%2FeAXYIG%2BDbfsbqMWrgTHaeBh0iy5D8U%3D&reserved=0> for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.16.6, Mar 30, 2022
[0]PETSC ERROR: ./iesolver on a  named REXNET-WS4 by jiannantu Thu Aug 18 14:48:43 2022
[0]PETSC ERROR: Configure options --download-f2cblaslapack=yes --with-mpi-dir=/usr/local --with-fc=0 --prefix=/home/jiannantu/petsc --with-scalar-type=complex --with-64-bit-indices=1
[0]PETSC ERROR: #1 MatMult() at /home/jiannantu/petsc-3.16.6/src/mat/interface/matrix.c:2425
[0]PETSC ERROR: #2 PCApplyBAorAB() at /home/jiannantu/petsc-3.16.6/src/ksp/pc/interface/precon.c:730
[0]PETSC ERROR: #3 KSP_PCApplyBAorAB() at /home/jiannantu/petsc-3.16.6/include/petsc/private/kspimpl.h:421
[0]PETSC ERROR:  Linear equations not solved. Diverged reason: 0
#4 KSPSolve_BCGS() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/impls/bcgs/bcgs.c:87
[0]PETSC ERROR: #5 KSPSolve_Private() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itfunc.c:914
[0]PETSC ERROR: #6 KSPSolve() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itfunc.c:1086

-----------------------------------------------
    MatCreate(MPI_COMM_WORLD, &B);
    MatSetType(B, MATMPIAIJ);
    MatSetSizes(B, PETSC_DECIDE, PETSC_DECIDE, N, N);

    Mat preall;
    MatCreate(MPI_COMM_WORLD, &preall);
    MatSetType(preall, MATPREALLOCATOR);
    MatSetSizes(preall, PETSC_DECIDE, PETSC_DECIDE, N, N);

    MatMPIAIJSetPreallocation(preall, 0, d_nnz, 0, o_nnz);
    MatSetUp(preall);
MatSetOption(preall, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE);

//set values row by row
MatSetValues(preall, 1, &m, 1, nCols, matCols, INSERT_VALUES);

MatAssemblyBegin(preall, MAT_FINAL_ASSEMBLY);
MatAssemblyEnd(preall, MAT_FINAL_ASSEMBLY);

//set values for actual matrix row by row
MatSetValues(B, 1, &m, 1, nCols, matCols, INSERT_VALUES);

MatAssemblyBegin(B, MAT_FINAL_ASSEMBLY);
MatAssemblyEnd(B, MAT_FINAL_ASSEMBLY);

    KSPCreate(MPI_COMM_WORLD, &ksp);
    KSPSetOperators(ksp, A, B);
    KSPSetFromOptions(ksp);

    PC pc;
    KSPGetPC(ksp, &pc);
    PCSetFromOptions(pc);

From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Thursday, August 18, 2022 12:59 PM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] Using matrix-free with KSP

CAUTION: This email was sent from outside the UMass Lowell network.


  Yes, once you have preallocated the real matrix you can destroy the preallocation matrix whose only job is to gather the preallocation information



On Aug 18, 2022, at 12:52 PM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

Thanks, Barry.

So I need actually two matrixes, one for preallocator and one for actual matrix that can be passed to KSPSetOperators(). -mat_type preallocator option is used to speed up doing insertion into preallocator, then use MatPreallocatorPreallocate() to preallocate actual matrix and do actual insertion of values into it., right?

The code runs in parallel. Each process owns number of rows that equals to number of unknowns (that is, xm in 1D DM) it owns.

Jiannan

From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Thursday, August 18, 2022 11:37 AM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] Using matrix-free with KSP

CAUTION: This email was sent from outside the UMass Lowell network.


   The preallocator MatType matrix cannot be passed to the KSPSetOperators(), you need to create an actual matrix, for example MATAIJ and use the preallocator to set its preallocation and then fill up the MATAIJ matrix with the usual MatSetValues and pass that matrix to the KSPSetOperators.




On Aug 18, 2022, at 11:30 AM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

Hi Barry,

The MATPREALLOCATOR solved problem of slow matrix assembly. But the solver failed because of "Matrix type preallocator does not have a multiply defined".

I guess this is because the matrix-free is used. I am wondering how a preconditioner is applied to the matrix-vector multiply in Petsc.

Thank you,
Jiannan

[0]PETSC ERROR: No support for this operation for this object type
[0]PETSC ERROR: Matrix type preallocator does not have a multiply defined
[0]PETSC ERROR: See https://petsc.org/release/faq/<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Frelease%2Ffaq%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7C73f4b5b4a7244d57063108da81738355%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637964630446347701%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=EvNUIKJN2gp1%2FeAXYIG%2BDbfsbqMWrgTHaeBh0iy5D8U%3D&reserved=0> for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.16.6, Mar 30, 2022
[0]PETSC ERROR: ./iesolver on a  named REXNET-WS4 by jiannantu Thu Aug 18 11:21:55 2022
[0]PETSC ERROR: Configure options --download-f2cblaslapack=yes --with-mpi-dir=/usr/local --with-fc=0 --prefix=/home/jiannantu/petsc --with-scalar-type=complex --with-64-bit-indices=1
[0]PETSC ERROR: #1 MatMult() at /home/jiannantu/petsc-3.16.6/src/mat/interface/matrix.c:2437
[0]PETSC ERROR: #2 PCApply_Mat() at /home/jiannantu/petsc-3.16.6/src/ksp/pc/impls/mat/pcmat.c:9
[0]PETSC ERROR: #3 PCApply() at /home/jiannantu/petsc-3.16.6/src/ksp/pc/interface/precon.c:445
[0]PETSC ERROR: #4 KSP_PCApply() at /home/jiannantu/petsc-3.16.6/include/petsc/private/kspimpl.h:382
[0]PETSC ERROR: #5 KSPInitialResidual() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itres.c:65
[0]PETSC ERROR: #6 KSPSolve_BCGS() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/impls/bcgs/bcgs.c:43
[0]PETSC ERROR: #7 KSPSolve_Private() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itfunc.c:914
[0]PETSC ERROR: #8 KSPSolve() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itfunc.c:1086

From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Thursday, August 18, 2022 8:35 AM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] Using matrix-free with KSP

CAUTION: This email was sent from outside the UMass Lowell network.


  Slow assembly is usually due to under preallocation. You can change to using

MatSetOption(B, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_TRUE);

  to detect if you are under preallocating. See https://petsc.org/main/docs/manual/mat/#sec-matsparse<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanual%2Fmat%2F%23sec-matsparse&data=05%7C01%7CJiannan_Tu%40uml.edu%7C73f4b5b4a7244d57063108da81738355%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637964630446347701%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=GxQiHn3e3teGRmy93A3YfyEHm8KZXXAVwrSj%2Fhb7neI%3D&reserved=0>. That section clearly needs some sprucing up. If it is difficult to determine good preallocation values you can use https://petsc.org/main/docs/manualpages/Mat/MATPREALLOCATOR/<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMATPREALLOCATOR%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7C73f4b5b4a7244d57063108da81738355%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637964630446347701%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=l%2FIl2%2Fef0HdEcnshrdwFsgLznDsJN18tQ1Q%2BQZThPZw%3D&reserved=0> to compute the needed preallocation efficiently.

  Are you running in parallel? If so how are you determining which rows of entries to compute on each MPI rank? You will want most of the rows to be computed on the rank where the values are stored, you can determine this with https://petsc.org/main/docs/manualpages/Mat/MatGetOwnershipRange.html<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMatGetOwnershipRange.html&data=05%7C01%7CJiannan_Tu%40uml.edu%7C73f4b5b4a7244d57063108da81738355%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637964630446347701%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=iK3RGAEOBy93g0WF41%2Bd9zRSx3ZZwb9rptrvDTFnf8U%3D&reserved=0>

  Barry













On Aug 18, 2022, at 12:50 AM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

I implemented the preconditioner. The solver converges quickly when the problem size is small. But when the size increases, say to 100k unknowns, the code hangs at assembly of the preconditioner matrix. The function call sequence is like this

   MatCreate(MPI_COMM_WORLD, &B);
    MatSetType(B, MATMPIAIJ);
    MatSetSizes(B, PETSC_DECIDE, PETSC_DECIDE, N, N);
    MatSetFromOptions(B);

    // The number of diagonal and off diagonal non zeros only can be determined run-time.
    MatMPIAIJSetPreallocation(B, 0, d_nnz, 0, o_nnz);
    MatSetOption(B, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE);

    //here insert values row by row.
MatSetValues(B, 1, &m, nn, nCols, matCols, INSERT_VALUES);

MatAssemblyBegin(B, MAT_FINAL_ASSEMBLY);
MatAssemblyEnd(B, MAT_FINAL_ASSEMBLY);

Could you please tell me what I have done wrong?

Thank you,
Jiannan

From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Tuesday, August 16, 2022 2:00 PM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] Using matrix-free with KSP

CAUTION: This email was sent from outside the UMass Lowell network.


   Create another matrix B that contains "Selected parts of the matrix can be pre-calculated and stored to serve as the preconditioner." When you call KSPSetOperators() pass in B as the second matrix argument.

   Now if literally application of B is the preconditioner you want to use, then simply use PCMAT as the preconditioner, and it will apply B each time it needs to apply the preconditioner. With this, you do not even need a PCShell.

   If your preconditioner uses B to construct your own specific data structure needed to apply your preconditioner, then you can use PCShellSetSetUp() and have your routine call PCGetOperators() to pull out the B matrix and use it to construct your preconditioner.

   If you want PETSc to construct a standard preconditioner from B, then you don't need the PCShell; you simply pass in the B, as above, and use -pc_type type to set the preconditioner you want to use.

   Barry







On Aug 16, 2022, at 12:31 PM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

Barry,

Thank you very much for your instructions. I am sorry I may not ask clearer questions.

I don't use SNES. What I am trying to solve is a very large linear equation system with dense and ill-conditioned matrix. Selected parts of the matrix can be pre-calculated and stored to serve as the preconditioner. My question is how I can apply this matrix as the preconditioner.

Here is the part of code from my linear equations solver. MatrixFreePreconditioner is supposed to be the routine to provide the preconditioner. But I don't understand how to use it. Should I use SNESSetJacobian() even if this is a linear problem?

    MatCreateMFFD(MPI_COMM_WORLD, PETSC_DECIDE, PETSC_DECIDE, N, N, &A);
    MatMFFDSetFunction(A, formfunction, &params);
    MatSetFromOptions(A);
    MatMFFDSetBase(A, X, NULL);

    KSPCreate(MPI_COMM_WORLD, &ksp);
    KSPSetOperators(ksp, A, A);
    KSPSetFromOptions(ksp);

    PC pc;
    KSPGetPC(ksp,&pc);
    PCSetType(pc,PCSHELL);
    PCShellSetApply(pc,MatrixFreePreconditioner);

Thank you,
Jiannan

________________________________
From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Tuesday, August 16, 2022 10:10 AM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] Using matrix-free with KSP

CAUTION: This email was sent from outside the UMass Lowell network.


  I don't fully understand your question so I will answer questions that I do know the answer to :-)

  The function you provide to PCShellSetApply() applies the preconditioner to the input vector and puts the result in the output vector.  This input vector changes at every application of the preconditioner so should not be used in constructing the preconditioner.

  If you are using the standard PETSc matrix-free matrix-vector product via finite difference, -snes_mf_operator or via MatCreateSNESMF() or you are using a MatShell, the matrix-vector product routines input changes for each multiply and should not be used in constructing the preconditioner.

  The function you provide with SNESSetJacobian() is where you can compute anything you need to help build your preconditioner. The input vector to that function is the location at which the Jacobian is needed and is what you should use to build your approximate/part of Jacobian. You can store your approximate/part of Jacobian that you need to compute in the pmat and then in a PCShellSetUp() function that you provide you will compute what you will need latter to apply the preconditioner in your PCShellSetApply() function. Note that the values you put in the pmat do not need to be the full Jacobian; they can be anything you want since this pmat is not used to apply the matrix vector product.

  If this is unclear, feel free to ask additional questions.

  Barry







On Aug 16, 2022, at 9:40 AM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

I am trying to apply a user-defined preconditioner to speed up the convergency of matrix-free KSP solver. The user's manual provides an example of how to call the user-defined preconditioner routine. The routine has PC, input Vec and output Vec as arguments. I want to use part of the Jacobian as preconditioner matrix, the elements of which can be calculated using input Vec. My question is what the role of output Vec from the routine is in matrix-vector product? Or I just have the preconditioner matrix elements calculated in the routine and then Petsc handles applying the preconditioner to matrix-vector product? I used PCSetType(pc, PCSHELL) and PCShellSetApply.

Thank you and appreciate your help!

Jiannan
________________________________
From: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Sent: Wednesday, July 6, 2022 2:34 PM
To: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Cc: Jed Brown <jed at jedbrown.org<mailto:jed at jedbrown.org>>; petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] Using matrix-free with KSP

Barry,

VecScatterCreateToAll solves the problem! The code now gives the correct answer with multiple processes.

Thank you all so much!

Jiannan
________________________________
From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Wednesday, July 6, 2022 2:08 PM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Cc: Jed Brown <jed at jedbrown.org<mailto:jed at jedbrown.org>>; petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] Using matrix-free with KSP


  So your operator is a "dense" operator in that its matrix representation would be essentially dense. In that case, you can use specialized routines to do this efficiently. You can use VecScatterCreateToAll() and then VecScatterBegin/End to do the communication.

  Barry








On Jul 6, 2022, at 1:39 PM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

Jed, thank you very much for the reply.

I'm not sure GlobaltoLocal will work. Say the solution vector is of length 10. Two processes then each process can see 5 elements of the vector. For A x = b (10 equations), each process performs matrix-vector product for five equations. And for each equation i , sum_j A_ij *  x_j = b_i requires the process has access to all 10 unknows x_i (i=0,1,...,9).

Can VecScatter and PetscSF make the entire vector accessible to each process? I am reading the manual and trying to understand how VecScatter and PetscSF work.

Jiannan
________________________________
From: Jed Brown <jed at jedbrown.org<mailto:jed at jedbrown.org>>
Sent: Wednesday, July 6, 2022 10:09 AM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>; Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] Using matrix-free with KSP

You'll usually have a GlobalToLocal operation for each rank to get the halo data it needs, then either a LocalToGlobal to collect the result (e.g., finite element methods) or the local compute will write directly into the owned portion of the global vector. If you're doing the communication "raw", then you may find VecScatter or PetscSF useful to perform the necessary communication.

"Tu, Jiannan" <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> writes:

> Hi Barry,
>
> Following your instructions I implemented the matrix-free method to solve a large linear equation system resulted from a surface integration equation. The KSP solver works fine for single process, but it is problematic with multiple processes. The problem is that each process only can access its own part of solution vector so that each process only conducts part of matrix-vector multiplication. MPI can be used to assemble these partial matrix-vector multiplication together.  Does Petsc provide any ways to implement multi-process matrix-free method?
>
> Thanks,
> Jiannan
> ________________________________
> From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
> Sent: Tuesday, May 24, 2022 2:12 PM
> To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
> Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
> Subject: Re: [petsc-users] Using matrix-free with KSP
>
> This e-mail originated from outside the UMass Lowell network.
> ________________________________
>
>    You can use MatCreateMFFD https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMatCreateMFFD%2F&amp;data=05%7C01%7CJiannan_Tu%40uml.edu%7Cab0859d8d154471590bc08da5f5918d6%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637927133525745809%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=A7wqegTfh94No5BpDiWLK3VxOuR44U2wlWHVm2k7l60%3D&amp;reserved=0<https://urldefense.com/v3/__https://petsc.org/main/docs/manualpages/Mat/MatCreateMFFD/__;!!PVKG_VDCxu5g!stxq2NWGVjUihvHBQC9Rlk0aHVoowcy0PTQYMcQxQ4DqOEC05KSw6TmstSXKckiUgelHzy4ue-d10-zlXkw$><https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMatCreateMFFD%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7C73f4b5b4a7244d57063108da81738355%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637964630446347701%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=W8qthQi69IdpZPk9SFcVdDEbuimrPYBnbgq%2BwJWzdzU%3D&reserved=0> MatMFFDSetFunction MatSetFromOptions MatMFFDSetBase and provide the matrix to KSP. Note you will need to use -pc_type none or provide your own custom preconditioner withhttps://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FPC%2FPCSHELL%2F&amp;data=05%7C01%7CJiannan_Tu%40uml.edu%7Cab0859d8d154471590bc08da5f5918d6%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637927133525745809%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=Gb%2F9uFG3jp%2FbfWaSh2cSEQLItHS9CuLwarzN0KcNiJY%3D&amp;reserved=0<https://urldefense.com/v3/__https://petsc.org/main/docs/manualpages/PC/PCSHELL/__;!!PVKG_VDCxu5g!stxq2NWGVjUihvHBQC9Rlk0aHVoowcy0PTQYMcQxQ4DqOEC05KSw6TmstSXKckiUgelHzy4ue-d1cA4fKGw$><https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FPC%2FPCSHELL%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7C73f4b5b4a7244d57063108da81738355%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637964630446347701%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=wrjEwpTTlr1Bg%2FkgNPpmt4YyIaF8unpRn2mogMLSHOw%3D&reserved=0>
>
>
>
> On May 24, 2022, at 1:21 PM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu><mailto:Jiannan_Tu at uml.edu>> wrote:
>
> I want to use a matrix-free matrix to solve a large linear equation system because the matrix is too large to be stored. Petsc user manual describes matrix-free method for SNES with examples. The matrix-free matrices section explains how to set up such a matrix, but I can't find any example of matrix-free method with KSP. I am wondering how to apply the method to KSP iterative solver in parallel.
>
> I greatly appreciate your help for me to understand this topic.
>
> Jiannan Tu

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220819/eb3f8925/attachment-0001.html>

From bsmith at petsc.dev  Fri Aug 19 09:26:36 2022
From: bsmith at petsc.dev (Barry Smith)
Date: Fri, 19 Aug 2022 10:26:36 -0400
Subject: [petsc-users] Using matrix-free with KSP
In-Reply-To: <BY5PR02MB6471CD0236F03D20763A3B64E76C9@BY5PR02MB6471.namprd02.prod.outlook.com>
References: <BY5PR02MB6471969F80FA5F732FC74D87E7D79@BY5PR02MB6471.namprd02.prod.outlook.com>
	<B55CFC2C-6364-4B78-8E99-9D53700743B8@petsc.dev>
	<BY5PR02MB6471BDF90783FE3A0EA176CBE7809@BY5PR02MB6471.namprd02.prod.outlook.com>
	<87iloa49kt.fsf@jedbrown.org>
	<BY5PR02MB6471C0ED04769D02046F68ACE7809@BY5PR02MB6471.namprd02.prod.outlook.com>
	<00EBE4B8-22E5-408F-B56D-14226A278D7E@petsc.dev>
	<BY5PR02MB6471E386057DDDE98AD05DFAE7809@BY5PR02MB6471.namprd02.prod.outlook.com>
	<BY5PR02MB647140084C7E7CDDB6BC2A11E76B9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<A6629DE1-5A7F-414E-B164-A69877158675@petsc.dev>
	<BY5PR02MB6471AB1C004920DA1FC9B43EE76B9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<103146FD-FF28-47CE-B661-F2B521BF0E7A@petsc.dev>
	<DM6PR02MB6478B2FB8CB1BE757CE7B003E76D9@DM6PR02MB6478.namprd02.prod.outlook.com>
	<4C00F4B8-BB42-4932-BCD7-527E5FB39BFF@petsc.dev>
	<BY5PR02MB64710E70758E431108D66956E76D9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<D0EB13D8-C5A6-4AE7-AFDE-2CDCCACBAF73@petsc.dev>
	<BY5PR02MB6471D4B05874DECFDDA13FFCE76D9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<367B124C-88BB-4889-A4AF-45A1D473078D@petsc.dev>
	<BY5PR02MB6471F2C058B84DE092567731E76D9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<944DDEED-016D-43B2-A5A6-7BF2E8E41FA5@petsc.dev>
	<BY5PR02MB6471CD0236F03D20763A3B64E76C9@BY5PR02MB6471.namprd02.prod.outlook.com>
Message-ID: <B4CF5E5B-984D-4EC1-AB77-1FD08C3FBBCF@petsc.dev>



> On Aug 18, 2022, at 11:41 PM, Tu, Jiannan <Jiannan_Tu at uml.edu> wrote:
> 
> The program calls KSPSolve just once and the error occurs after the first iteration.
>  
> I realized the matrix-vector product in my case is different from that approximating matrix-vector multiply by finite difference of function vector. It consists of two parts. One is multiplication of part of the matrix with part of vector, and another part results from so called fast multipole method. I?m not clear how to relate such calculation of matrix-vector product to the finite differencing of function vector.

  Are you using finite differencing at all in a matrix-vector product, or do you have your own custom matrix-vector product code (with MatShell)? 

  Anyways the easiest thing to do is 

  use the run time option -on_error_attach_debugger noxterm and run the code, 

  when it stops in the debugger you can type bt (for backtrace)

  then do up a few times to move up the stack until you get to the PETSc function call that generates the error

  you can then do "call MatView(thematrixvarible,0) to have PETSc display what the matrix is and

  you can also do print *thematrixvariable to see the exact struct with all its fields

  If this doesn't tell you why the troublesome matrix is not assembled then cut and paste ALL of the output and email it

  Barry

>  
>  
>  
> From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>> 
> Sent: Thursday, August 18, 2022 7:44 PM
> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
> Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>
> Subject: Re: [petsc-users] Using matrix-free with KSP
>  
> CAUTION: This email was sent from outside the UMass Lowell network.
>  
>  
>  
>    Is it in the first iteration of the first KSPSolve that you get this message? Or in the second call to KSPSolve? 
>  
>   The error comes from a matrix vector product with your A matrix, what is that and where did it come from. The error message is not from your B matrix, that is not used in the MatMult()
>  
> 
> 
> On Aug 18, 2022, at 3:05 PM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> wrote:
>  
> Barry, 
>  
> Thanks. The method works. Assembly of actual matrix now completes quickly. The convergency is much better for the first iteration. The residual normal is in the order of 10^-10 instead of 10^-2. However, there is an error message after the first iteration: ?Object is in wrong state. Not for unassembled matrix? even though both preallocator and actual matrix have been assembled (see the code snippet below). Could you please give me some hints what?s going wrong? I use KSP type bcgs and PC type mat
>  
> Thank you,
> Jiannan
>  
> -----------------------------------------------------
> Solve linear equations ...
>   0 KSP Residual norm 9.841179147519e-10
> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [0]PETSC ERROR: Object is in wrong state
> [0]PETSC ERROR: Not for unassembled matrix
> [0]PETSC ERROR: See https://petsc.org/release/faq/ <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Frelease%2Ffaq%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7C73f4b5b4a7244d57063108da81738355%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637964630446347701%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=EvNUIKJN2gp1%2FeAXYIG%2BDbfsbqMWrgTHaeBh0iy5D8U%3D&reserved=0> for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.16.6, Mar 30, 2022
> [0]PETSC ERROR: ./iesolver on a  named REXNET-WS4 by jiannantu Thu Aug 18 14:48:43 2022
> [0]PETSC ERROR: Configure options --download-f2cblaslapack=yes --with-mpi-dir=/usr/local --with-fc=0 --prefix=/home/jiannantu/petsc --with-scalar-type=complex --with-64-bit-indices=1
> [0]PETSC ERROR: #1 MatMult() at /home/jiannantu/petsc-3.16.6/src/mat/interface/matrix.c:2425
> [0]PETSC ERROR: #2 PCApplyBAorAB() at /home/jiannantu/petsc-3.16.6/src/ksp/pc/interface/precon.c:730
> [0]PETSC ERROR: #3 KSP_PCApplyBAorAB() at /home/jiannantu/petsc-3.16.6/include/petsc/private/kspimpl.h:421
> [0]PETSC ERROR:  Linear equations not solved. Diverged reason: 0
> #4 KSPSolve_BCGS() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/impls/bcgs/bcgs.c:87
> [0]PETSC ERROR: #5 KSPSolve_Private() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itfunc.c:914
> [0]PETSC ERROR: #6 KSPSolve() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itfunc.c:1086
>  
> -----------------------------------------------
>     MatCreate(MPI_COMM_WORLD, &B);
>     MatSetType(B, MATMPIAIJ);
>     MatSetSizes(B, PETSC_DECIDE, PETSC_DECIDE, N, N);
>  
>     Mat preall;
>     MatCreate(MPI_COMM_WORLD, &preall);
>     MatSetType(preall, MATPREALLOCATOR);
>     MatSetSizes(preall, PETSC_DECIDE, PETSC_DECIDE, N, N);
>  
>     MatMPIAIJSetPreallocation(preall, 0, d_nnz, 0, o_nnz);
>     MatSetUp(preall);
> MatSetOption(preall, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE);
>  
> //set values row by row
> MatSetValues(preall, 1, &m, 1, nCols, matCols, INSERT_VALUES);
>  
> MatAssemblyBegin(preall, MAT_FINAL_ASSEMBLY);
> MatAssemblyEnd(preall, MAT_FINAL_ASSEMBLY);
>  
> //set values for actual matrix row by row
> MatSetValues(B, 1, &m, 1, nCols, matCols, INSERT_VALUES);
>  
> MatAssemblyBegin(B, MAT_FINAL_ASSEMBLY);
> MatAssemblyEnd(B, MAT_FINAL_ASSEMBLY);
>  
>     KSPCreate(MPI_COMM_WORLD, &ksp);
>     KSPSetOperators(ksp, A, B);
>     KSPSetFromOptions(ksp);
>  
>     PC pc;
>     KSPGetPC(ksp, &pc);
>     PCSetFromOptions(pc);
>  
> From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>> 
> Sent: Thursday, August 18, 2022 12:59 PM
> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
> Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>
> Subject: Re: [petsc-users] Using matrix-free with KSP
>  
> CAUTION: This email was sent from outside the UMass Lowell network.
>  
>  
>   Yes, once you have preallocated the real matrix you can destroy the preallocation matrix whose only job is to gather the preallocation information
> 
> 
> 
> On Aug 18, 2022, at 12:52 PM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> wrote:
>  
> Thanks, Barry.
>  
> So I need actually two matrixes, one for preallocator and one for actual matrix that can be passed to KSPSetOperators(). -mat_type preallocator option is used to speed up doing insertion into preallocator, then use MatPreallocatorPreallocate() to preallocate actual matrix and do actual insertion of values into it., right?
>  
> The code runs in parallel. Each process owns number of rows that equals to number of unknowns (that is, xm in 1D DM) it owns.
>  
> Jiannan
>  
> From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>> 
> Sent: Thursday, August 18, 2022 11:37 AM
> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
> Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>
> Subject: Re: [petsc-users] Using matrix-free with KSP
>  
> CAUTION: This email was sent from outside the UMass Lowell network.
>  
>  
>    The preallocator MatType matrix cannot be passed to the KSPSetOperators(), you need to create an actual matrix, for example MATAIJ and use the preallocator to set its preallocation and then fill up the MATAIJ matrix with the usual MatSetValues and pass that matrix to the KSPSetOperators.
> 
> 
> 
> 
> On Aug 18, 2022, at 11:30 AM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> wrote:
>  
> Hi Barry,
>  
> The MATPREALLOCATOR solved problem of slow matrix assembly. But the solver failed because of ?Matrix type preallocator does not have a multiply defined?.
>  
> I guess this is because the matrix-free is used. I am wondering how a preconditioner is applied to the matrix-vector multiply in Petsc.
>  
> Thank you,
> Jiannan
>  
> [0]PETSC ERROR: No support for this operation for this object type
> [0]PETSC ERROR: Matrix type preallocator does not have a multiply defined
> [0]PETSC ERROR: See https://petsc.org/release/faq/ <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Frelease%2Ffaq%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7C73f4b5b4a7244d57063108da81738355%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637964630446347701%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=EvNUIKJN2gp1%2FeAXYIG%2BDbfsbqMWrgTHaeBh0iy5D8U%3D&reserved=0> for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.16.6, Mar 30, 2022
> [0]PETSC ERROR: ./iesolver on a  named REXNET-WS4 by jiannantu Thu Aug 18 11:21:55 2022
> [0]PETSC ERROR: Configure options --download-f2cblaslapack=yes --with-mpi-dir=/usr/local --with-fc=0 --prefix=/home/jiannantu/petsc --with-scalar-type=complex --with-64-bit-indices=1
> [0]PETSC ERROR: #1 MatMult() at /home/jiannantu/petsc-3.16.6/src/mat/interface/matrix.c:2437
> [0]PETSC ERROR: #2 PCApply_Mat() at /home/jiannantu/petsc-3.16.6/src/ksp/pc/impls/mat/pcmat.c:9
> [0]PETSC ERROR: #3 PCApply() at /home/jiannantu/petsc-3.16.6/src/ksp/pc/interface/precon.c:445
> [0]PETSC ERROR: #4 KSP_PCApply() at /home/jiannantu/petsc-3.16.6/include/petsc/private/kspimpl.h:382
> [0]PETSC ERROR: #5 KSPInitialResidual() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itres.c:65
> [0]PETSC ERROR: #6 KSPSolve_BCGS() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/impls/bcgs/bcgs.c:43
> [0]PETSC ERROR: #7 KSPSolve_Private() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itfunc.c:914
> [0]PETSC ERROR: #8 KSPSolve() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itfunc.c:1086
>  
> From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>> 
> Sent: Thursday, August 18, 2022 8:35 AM
> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
> Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>
> Subject: Re: [petsc-users] Using matrix-free with KSP
>  
> CAUTION: This email was sent from outside the UMass Lowell network.
>  
>  
>   Slow assembly is usually due to under preallocation. You can change to using 
>  
> MatSetOption(B, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_TRUE);
>  
>   to detect if you are under preallocating. See https://petsc.org/main/docs/manual/mat/#sec-matsparse <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanual%2Fmat%2F%23sec-matsparse&data=05%7C01%7CJiannan_Tu%40uml.edu%7C73f4b5b4a7244d57063108da81738355%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637964630446347701%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=GxQiHn3e3teGRmy93A3YfyEHm8KZXXAVwrSj%2Fhb7neI%3D&reserved=0>. That section clearly needs some sprucing up. If it is difficult to determine good preallocation values you can use https://petsc.org/main/docs/manualpages/Mat/MATPREALLOCATOR/ <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMATPREALLOCATOR%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7C73f4b5b4a7244d57063108da81738355%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637964630446347701%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=l%2FIl2%2Fef0HdEcnshrdwFsgLznDsJN18tQ1Q%2BQZThPZw%3D&reserved=0> to compute the needed preallocation efficiently.
>  
>   Are you running in parallel? If so how are you determining which rows of entries to compute on each MPI rank? You will want most of the rows to be computed on the rank where the values are stored, you can determine this with https://petsc.org/main/docs/manualpages/Mat/MatGetOwnershipRange.html <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMatGetOwnershipRange.html&data=05%7C01%7CJiannan_Tu%40uml.edu%7C73f4b5b4a7244d57063108da81738355%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637964630446347701%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=iK3RGAEOBy93g0WF41%2Bd9zRSx3ZZwb9rptrvDTFnf8U%3D&reserved=0>
>  
>   Barry
>  
>  
> 
> 
> 
> 
> 
>  
> 
> 
> 
> 
> 
> On Aug 18, 2022, at 12:50 AM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> wrote:
>  
> I implemented the preconditioner. The solver converges quickly when the problem size is small. But when the size increases, say to 100k unknowns, the code hangs at assembly of the preconditioner matrix. The function call sequence is like this
>  
>    MatCreate(MPI_COMM_WORLD, &B);
>     MatSetType(B, MATMPIAIJ);
>     MatSetSizes(B, PETSC_DECIDE, PETSC_DECIDE, N, N);
>     MatSetFromOptions(B);
>  
>     // The number of diagonal and off diagonal non zeros only can be determined run-time.
>     MatMPIAIJSetPreallocation(B, 0, d_nnz, 0, o_nnz);
>     MatSetOption(B, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE);
>  
>     //here insert values row by row.
> MatSetValues(B, 1, &m, nn, nCols, matCols, INSERT_VALUES);
>  
> MatAssemblyBegin(B, MAT_FINAL_ASSEMBLY);
> MatAssemblyEnd(B, MAT_FINAL_ASSEMBLY);
>  
> Could you please tell me what I have done wrong?
>  
> Thank you,
> Jiannan
>    
> From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>> 
> Sent: Tuesday, August 16, 2022 2:00 PM
> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
> Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>
> Subject: Re: [petsc-users] Using matrix-free with KSP
>  
> CAUTION: This email was sent from outside the UMass Lowell network.
>  
>  
>    Create another matrix B that contains "Selected parts of the matrix can be pre-calculated and stored to serve as the preconditioner." When you call KSPSetOperators() pass in B as the second matrix argument.
>  
>    Now if literally application of B is the preconditioner you want to use, then simply use PCMAT as the preconditioner, and it will apply B each time it needs to apply the preconditioner. With this, you do not even need a PCShell. 
>  
>    If your preconditioner uses B to construct your own specific data structure needed to apply your preconditioner, then you can use PCShellSetSetUp() and have your routine call PCGetOperators() to pull out the B matrix and use it to construct your preconditioner. 
>  
>    If you want PETSc to construct a standard preconditioner from B, then you don't need the PCShell; you simply pass in the B, as above, and use -pc_type type to set the preconditioner you want to use. 
>  
>    Barry
>  
> 
> 
> 
> 
> 
> 
> On Aug 16, 2022, at 12:31 PM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> wrote:
>  
> Barry,
>  
> Thank you very much for your instructions. I am sorry I may not ask clearer questions.
>  
> I don't use SNES. What I am trying to solve is a very large linear equation system with dense and ill-conditioned matrix. Selected parts of the matrix can be pre-calculated and stored to serve as the preconditioner. My question is how I can apply this matrix as the preconditioner. 
>  
> Here is the part of code from my linear equations solver. MatrixFreePreconditioner is supposed to be the routine to provide the preconditioner. But I don't understand how to use it. Should I use SNESSetJacobian() even if this is a linear problem? 
>  
>     MatCreateMFFD(MPI_COMM_WORLD, PETSC_DECIDE, PETSC_DECIDE, N, N, &A);
>     MatMFFDSetFunction(A, formfunction, &params);
>     MatSetFromOptions(A);
>     MatMFFDSetBase(A, X, NULL);
>  
>     KSPCreate(MPI_COMM_WORLD, &ksp);
>     KSPSetOperators(ksp, A, A);
>     KSPSetFromOptions(ksp);
>  
>     PC pc;
>     KSPGetPC(ksp,&pc);
>     PCSetType(pc,PCSHELL);
>     PCShellSetApply(pc,MatrixFreePreconditioner);
>  
> Thank you,
> Jiannan
>  
> From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>>
> Sent: Tuesday, August 16, 2022 10:10 AM
> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
> Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>>
> Subject: Re: [petsc-users] Using matrix-free with KSP
>  
> CAUTION: This email was sent from outside the UMass Lowell network.
>  
>  
>   I don't fully understand your question so I will answer questions that I do know the answer to :-)
>  
>   The function you provide to PCShellSetApply() applies the preconditioner to the input vector and puts the result in the output vector.  This input vector changes at every application of the preconditioner so should not be used in constructing the preconditioner.
>  
>   If you are using the standard PETSc matrix-free matrix-vector product via finite difference, -snes_mf_operator or via MatCreateSNESMF() or you are using a MatShell, the matrix-vector product routines input changes for each multiply and should not be used in constructing the preconditioner.
>  
>   The function you provide with SNESSetJacobian() is where you can compute anything you need to help build your preconditioner. The input vector to that function is the location at which the Jacobian is needed and is what you should use to build your approximate/part of Jacobian. You can store your approximate/part of Jacobian that you need to compute in the pmat and then in a PCShellSetUp() function that you provide you will compute what you will need latter to apply the preconditioner in your PCShellSetApply() function. Note that the values you put in the pmat do not need to be the full Jacobian; they can be anything you want since this pmat is not used to apply the matrix vector product.
>  
>   If this is unclear, feel free to ask additional questions.
>  
>   Barry
>  
> 
> 
> 
> 
> 
> 
> On Aug 16, 2022, at 9:40 AM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> wrote:
>  
> I am trying to apply a user-defined preconditioner to speed up the convergency of matrix-free KSP solver. The user's manual provides an example of how to call the user-defined preconditioner routine. The routine has PC, input Vec and output Vec as arguments. I want to use part of the Jacobian as preconditioner matrix, the elements of which can be calculated using input Vec. My question is what the role of output Vec from the routine is in matrix-vector product? Or I just have the preconditioner matrix elements calculated in the routine and then Petsc handles applying the preconditioner to matrix-vector product? I used PCSetType(pc, PCSHELL) and PCShellSetApply.
>  
> Thank you and appreciate your help!
>  
> Jiannan 
> From: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
> Sent: Wednesday, July 6, 2022 2:34 PM
> To: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>>
> Cc: Jed Brown <jed at jedbrown.org <mailto:jed at jedbrown.org>>; petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>>
> Subject: Re: [petsc-users] Using matrix-free with KSP
>  
> Barry,
>  
> VecScatterCreateToAll solves the problem! The code now gives the correct answer with multiple processes. 
>  
> Thank you all so much!
>  
> Jiannan
> From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>>
> Sent: Wednesday, July 6, 2022 2:08 PM
> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
> Cc: Jed Brown <jed at jedbrown.org <mailto:jed at jedbrown.org>>; petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>>
> Subject: Re: [petsc-users] Using matrix-free with KSP
>  
>  
>   So your operator is a "dense" operator in that its matrix representation would be essentially dense. In that case, you can use specialized routines to do this efficiently. You can use VecScatterCreateToAll() and then VecScatterBegin/End to do the communication.
>  
>   Barry
>  
>  
> 
> 
> 
> 
> 
> 
> On Jul 6, 2022, at 1:39 PM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> wrote:
>  
> Jed, thank you very much for the reply. 
>  
> I'm not sure GlobaltoLocal will work. Say the solution vector is of length 10. Two processes then each process can see 5 elements of the vector. For A x = b (10 equations), each process performs matrix-vector product for five equations. And for each equation i , sum_j A_ij *  x_j = b_i requires the process has access to all 10 unknows x_i (i=0,1,...,9). 
>  
> Can VecScatter and PetscSF make the entire vector accessible to each process? I am reading the manual and trying to understand how VecScatter and PetscSF work.
>  
> Jiannan
> From: Jed Brown <jed at jedbrown.org <mailto:jed at jedbrown.org>>
> Sent: Wednesday, July 6, 2022 10:09 AM
> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>; Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>>
> Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>>
> Subject: Re: [petsc-users] Using matrix-free with KSP
>  
> You'll usually have a GlobalToLocal operation for each rank to get the halo data it needs, then either a LocalToGlobal to collect the result (e.g., finite element methods) or the local compute will write directly into the owned portion of the global vector. If you're doing the communication "raw", then you may find VecScatter or PetscSF useful to perform the necessary communication.
> 
> "Tu, Jiannan" <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> writes:
> 
> > Hi Barry,
> >
> > Following your instructions I implemented the matrix-free method to solve a large linear equation system resulted from a surface integration equation. The KSP solver works fine for single process, but it is problematic with multiple processes. The problem is that each process only can access its own part of solution vector so that each process only conducts part of matrix-vector multiplication. MPI can be used to assemble these partial matrix-vector multiplication together.  Does Petsc provide any ways to implement multi-process matrix-free method?
> >
> > Thanks,
> > Jiannan
> > ________________________________
> > From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>>
> > Sent: Tuesday, May 24, 2022 2:12 PM
> > To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
> > Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>>
> > Subject: Re: [petsc-users] Using matrix-free with KSP
> >
> > This e-mail originated from outside the UMass Lowell network.
> > ________________________________
> >
> >    You can use MatCreateMFFD https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMatCreateMFFD%2F&amp;data=05%7C01%7CJiannan_Tu%40uml.edu%7Cab0859d8d154471590bc08da5f5918d6%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637927133525745809%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=A7wqegTfh94No5BpDiWLK3VxOuR44U2wlWHVm2k7l60%3D&amp;reserved=0<https://urldefense.com/v3/__https://petsc.org/main/docs/manualpages/Mat/MatCreateMFFD/__;!!PVKG_VDCxu5g!stxq2NWGVjUihvHBQC9Rlk0aHVoowcy0PTQYMcQxQ4DqOEC05KSw6TmstSXKckiUgelHzy4ue-d10-zlXkw$> <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMatCreateMFFD%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7C73f4b5b4a7244d57063108da81738355%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637964630446347701%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=W8qthQi69IdpZPk9SFcVdDEbuimrPYBnbgq%2BwJWzdzU%3D&reserved=0> MatMFFDSetFunction MatSetFromOptions MatMFFDSetBase and provide the matrix to KSP. Note you will need to use -pc_type none or provide your own custom preconditioner withhttps://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FPC%2FPCSHELL%2F&amp;data=05%7C01%7CJiannan_Tu%40uml.edu%7Cab0859d8d154471590bc08da5f5918d6%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637927133525745809%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=Gb%2F9uFG3jp%2FbfWaSh2cSEQLItHS9CuLwarzN0KcNiJY%3D&amp;reserved=0<https://urldefense.com/v3/__https://petsc.org/main/docs/manualpages/PC/PCSHELL/__;!!PVKG_VDCxu5g!stxq2NWGVjUihvHBQC9Rlk0aHVoowcy0PTQYMcQxQ4DqOEC05KSw6TmstSXKckiUgelHzy4ue-d1cA4fKGw$> <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FPC%2FPCSHELL%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7C73f4b5b4a7244d57063108da81738355%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637964630446347701%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=wrjEwpTTlr1Bg%2FkgNPpmt4YyIaF8unpRn2mogMLSHOw%3D&reserved=0>
> >
> >
> >
> > On May 24, 2022, at 1:21 PM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu><mailto:Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>> wrote:
> >
> > I want to use a matrix-free matrix to solve a large linear equation system because the matrix is too large to be stored. Petsc user manual describes matrix-free method for SNES with examples. The matrix-free matrices section explains how to set up such a matrix, but I can't find any example of matrix-free method with KSP. I am wondering how to apply the method to KSP iterative solver in parallel.
> >
> > I greatly appreciate your help for me to understand this topic.
> >
> > Jiannan Tu

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220819/0a602da3/attachment-0001.html>

From Jiannan_Tu at uml.edu  Fri Aug 19 10:47:58 2022
From: Jiannan_Tu at uml.edu (Tu, Jiannan)
Date: Fri, 19 Aug 2022 15:47:58 +0000
Subject: [petsc-users] Using matrix-free with KSP
In-Reply-To: <B4CF5E5B-984D-4EC1-AB77-1FD08C3FBBCF@petsc.dev>
References: <BY5PR02MB6471969F80FA5F732FC74D87E7D79@BY5PR02MB6471.namprd02.prod.outlook.com>
	<B55CFC2C-6364-4B78-8E99-9D53700743B8@petsc.dev>
	<BY5PR02MB6471BDF90783FE3A0EA176CBE7809@BY5PR02MB6471.namprd02.prod.outlook.com>
	<87iloa49kt.fsf@jedbrown.org>
	<BY5PR02MB6471C0ED04769D02046F68ACE7809@BY5PR02MB6471.namprd02.prod.outlook.com>
	<00EBE4B8-22E5-408F-B56D-14226A278D7E@petsc.dev>
	<BY5PR02MB6471E386057DDDE98AD05DFAE7809@BY5PR02MB6471.namprd02.prod.outlook.com>
	<BY5PR02MB647140084C7E7CDDB6BC2A11E76B9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<A6629DE1-5A7F-414E-B164-A69877158675@petsc.dev>
	<BY5PR02MB6471AB1C004920DA1FC9B43EE76B9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<103146FD-FF28-47CE-B661-F2B521BF0E7A@petsc.dev>
	<DM6PR02MB6478B2FB8CB1BE757CE7B003E76D9@DM6PR02MB6478.namprd02.prod.outlook.com>
	<4C00F4B8-BB42-4932-BCD7-527E5FB39BFF@petsc.dev>
	<BY5PR02MB64710E70758E431108D66956E76D9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<D0EB13D8-C5A6-4AE7-AFDE-2CDCCACBAF73@petsc.dev>
	<BY5PR02MB6471D4B05874DECFDDA13FFCE76D9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<367B124C-88BB-4889-A4AF-45A1D473078D@petsc.dev>
	<BY5PR02MB6471F2C058B84DE092567731E76D9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<944DDEED-016D-43B2-A5A6-7BF2E8E41FA5@petsc.dev>
	<BY5PR02MB6471CD0236F03D20763A3B64E76C9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<B4CF5E5B-984D-4EC1-AB77-1FD08C3FBBCF@petsc.dev>
Message-ID: <BY5PR02MB6471BA72871419781B875967E76C9@BY5PR02MB6471.namprd02.prod.outlook.com>

Barry,

I have my own matrix-vector product but not use MatShell. I followed your previous instruction to use

    MatCreateMFFD(MPI_COMM_WORLD, PETSC_DECIDE, PETSC_DECIDE, N, N, &A);
    MatMFFDSetFunction(A, formfunction, &params);
    MatSetFromOptions(A);
    MatMFFDSetBase(A, X, NULL);

The customized matrix-vector product is implemented in formfunction() provided to MatMFFDSetFunction(). In this way, the matrix A is not used. I think that is why Petsc complains the matrix is not assembled. However, when PC type is none, the code runs without error although KSP solver doesn't converge.

The question then becomes how to link A to matrix-free matrix-vector multiplication. And where to put MatShell?

Thank you,
Jiannan

From: Barry Smith <bsmith at petsc.dev>
Sent: Friday, August 19, 2022 10:27 AM
To: Tu, Jiannan <Jiannan_Tu at uml.edu>
Cc: petsc-users at mcs.anl.gov
Subject: Re: [petsc-users] Using matrix-free with KSP

CAUTION: This email was sent from outside the UMass Lowell network.




On Aug 18, 2022, at 11:41 PM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

The program calls KSPSolve just once and the error occurs after the first iteration.

I realized the matrix-vector product in my case is different from that approximating matrix-vector multiply by finite difference of function vector. It consists of two parts. One is multiplication of part of the matrix with part of vector, and another part results from so called fast multipole method. I'm not clear how to relate such calculation of matrix-vector product to the finite differencing of function vector.

  Are you using finite differencing at all in a matrix-vector product, or do you have your own custom matrix-vector product code (with MatShell)?

  Anyways the easiest thing to do is

  use the run time option -on_error_attach_debugger noxterm and run the code,

  when it stops in the debugger you can type bt (for backtrace)

  then do up a few times to move up the stack until you get to the PETSc function call that generates the error

  you can then do "call MatView(thematrixvarible,0) to have PETSc display what the matrix is and

  you can also do print *thematrixvariable to see the exact struct with all its fields

  If this doesn't tell you why the troublesome matrix is not assembled then cut and paste ALL of the output and email it

  Barry





From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Thursday, August 18, 2022 7:44 PM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] Using matrix-free with KSP

CAUTION: This email was sent from outside the UMass Lowell network.



   Is it in the first iteration of the first KSPSolve that you get this message? Or in the second call to KSPSolve?

  The error comes from a matrix vector product with your A matrix, what is that and where did it come from. The error message is not from your B matrix, that is not used in the MatMult()




On Aug 18, 2022, at 3:05 PM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

Barry,

Thanks. The method works. Assembly of actual matrix now completes quickly. The convergency is much better for the first iteration. The residual normal is in the order of 10^-10 instead of 10^-2. However, there is an error message after the first iteration: "Object is in wrong state. Not for unassembled matrix" even though both preallocator and actual matrix have been assembled (see the code snippet below). Could you please give me some hints what's going wrong? I use KSP type bcgs and PC type mat

Thank you,
Jiannan

-----------------------------------------------------
Solve linear equations ...
  0 KSP Residual norm 9.841179147519e-10
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Object is in wrong state
[0]PETSC ERROR: Not for unassembled matrix
[0]PETSC ERROR: See https://petsc.org/release/faq/<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Frelease%2Ffaq%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7Cfbe0ea83a9834c312a3108da81eed560%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637965160051463154%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=xTipZKMBKvPiOJbq46z08835vo1pXko1P1lelR0wL7Y%3D&reserved=0> for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.16.6, Mar 30, 2022
[0]PETSC ERROR: ./iesolver on a  named REXNET-WS4 by jiannantu Thu Aug 18 14:48:43 2022
[0]PETSC ERROR: Configure options --download-f2cblaslapack=yes --with-mpi-dir=/usr/local --with-fc=0 --prefix=/home/jiannantu/petsc --with-scalar-type=complex --with-64-bit-indices=1
[0]PETSC ERROR: #1 MatMult() at /home/jiannantu/petsc-3.16.6/src/mat/interface/matrix.c:2425
[0]PETSC ERROR: #2 PCApplyBAorAB() at /home/jiannantu/petsc-3.16.6/src/ksp/pc/interface/precon.c:730
[0]PETSC ERROR: #3 KSP_PCApplyBAorAB() at /home/jiannantu/petsc-3.16.6/include/petsc/private/kspimpl.h:421
[0]PETSC ERROR:  Linear equations not solved. Diverged reason: 0
#4 KSPSolve_BCGS() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/impls/bcgs/bcgs.c:87
[0]PETSC ERROR: #5 KSPSolve_Private() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itfunc.c:914
[0]PETSC ERROR: #6 KSPSolve() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itfunc.c:1086

-----------------------------------------------
    MatCreate(MPI_COMM_WORLD, &B);
    MatSetType(B, MATMPIAIJ);
    MatSetSizes(B, PETSC_DECIDE, PETSC_DECIDE, N, N);

    Mat preall;
    MatCreate(MPI_COMM_WORLD, &preall);
    MatSetType(preall, MATPREALLOCATOR);
    MatSetSizes(preall, PETSC_DECIDE, PETSC_DECIDE, N, N);

    MatMPIAIJSetPreallocation(preall, 0, d_nnz, 0, o_nnz);
    MatSetUp(preall);
MatSetOption(preall, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE);

//set values row by row
MatSetValues(preall, 1, &m, 1, nCols, matCols, INSERT_VALUES);

MatAssemblyBegin(preall, MAT_FINAL_ASSEMBLY);
MatAssemblyEnd(preall, MAT_FINAL_ASSEMBLY);

//set values for actual matrix row by row
MatSetValues(B, 1, &m, 1, nCols, matCols, INSERT_VALUES);

MatAssemblyBegin(B, MAT_FINAL_ASSEMBLY);
MatAssemblyEnd(B, MAT_FINAL_ASSEMBLY);

    KSPCreate(MPI_COMM_WORLD, &ksp);
    KSPSetOperators(ksp, A, B);
    KSPSetFromOptions(ksp);

    PC pc;
    KSPGetPC(ksp, &pc);
    PCSetFromOptions(pc);

From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Thursday, August 18, 2022 12:59 PM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] Using matrix-free with KSP

CAUTION: This email was sent from outside the UMass Lowell network.


  Yes, once you have preallocated the real matrix you can destroy the preallocation matrix whose only job is to gather the preallocation information




On Aug 18, 2022, at 12:52 PM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

Thanks, Barry.

So I need actually two matrixes, one for preallocator and one for actual matrix that can be passed to KSPSetOperators(). -mat_type preallocator option is used to speed up doing insertion into preallocator, then use MatPreallocatorPreallocate() to preallocate actual matrix and do actual insertion of values into it., right?

The code runs in parallel. Each process owns number of rows that equals to number of unknowns (that is, xm in 1D DM) it owns.

Jiannan

From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Thursday, August 18, 2022 11:37 AM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] Using matrix-free with KSP

CAUTION: This email was sent from outside the UMass Lowell network.


   The preallocator MatType matrix cannot be passed to the KSPSetOperators(), you need to create an actual matrix, for example MATAIJ and use the preallocator to set its preallocation and then fill up the MATAIJ matrix with the usual MatSetValues and pass that matrix to the KSPSetOperators.





On Aug 18, 2022, at 11:30 AM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

Hi Barry,

The MATPREALLOCATOR solved problem of slow matrix assembly. But the solver failed because of "Matrix type preallocator does not have a multiply defined".

I guess this is because the matrix-free is used. I am wondering how a preconditioner is applied to the matrix-vector multiply in Petsc.

Thank you,
Jiannan

[0]PETSC ERROR: No support for this operation for this object type
[0]PETSC ERROR: Matrix type preallocator does not have a multiply defined
[0]PETSC ERROR: See https://petsc.org/release/faq/<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Frelease%2Ffaq%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7Cfbe0ea83a9834c312a3108da81eed560%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637965160051619369%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=YraVMtPwCNL9miYa6d55KjhtTyiROmHOn2MMnq3pqT4%3D&reserved=0> for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.16.6, Mar 30, 2022
[0]PETSC ERROR: ./iesolver on a  named REXNET-WS4 by jiannantu Thu Aug 18 11:21:55 2022
[0]PETSC ERROR: Configure options --download-f2cblaslapack=yes --with-mpi-dir=/usr/local --with-fc=0 --prefix=/home/jiannantu/petsc --with-scalar-type=complex --with-64-bit-indices=1
[0]PETSC ERROR: #1 MatMult() at /home/jiannantu/petsc-3.16.6/src/mat/interface/matrix.c:2437
[0]PETSC ERROR: #2 PCApply_Mat() at /home/jiannantu/petsc-3.16.6/src/ksp/pc/impls/mat/pcmat.c:9
[0]PETSC ERROR: #3 PCApply() at /home/jiannantu/petsc-3.16.6/src/ksp/pc/interface/precon.c:445
[0]PETSC ERROR: #4 KSP_PCApply() at /home/jiannantu/petsc-3.16.6/include/petsc/private/kspimpl.h:382
[0]PETSC ERROR: #5 KSPInitialResidual() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itres.c:65
[0]PETSC ERROR: #6 KSPSolve_BCGS() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/impls/bcgs/bcgs.c:43
[0]PETSC ERROR: #7 KSPSolve_Private() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itfunc.c:914
[0]PETSC ERROR: #8 KSPSolve() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itfunc.c:1086

From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Thursday, August 18, 2022 8:35 AM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] Using matrix-free with KSP

CAUTION: This email was sent from outside the UMass Lowell network.


  Slow assembly is usually due to under preallocation. You can change to using

MatSetOption(B, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_TRUE);

  to detect if you are under preallocating. See https://petsc.org/main/docs/manual/mat/#sec-matsparse<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanual%2Fmat%2F%23sec-matsparse&data=05%7C01%7CJiannan_Tu%40uml.edu%7Cfbe0ea83a9834c312a3108da81eed560%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637965160051619369%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=9c3JU05Mb41i%2BIwJlvz9LsTqh1UlbaTpe1wjEtW6HHQ%3D&reserved=0>. That section clearly needs some sprucing up. If it is difficult to determine good preallocation values you can use https://petsc.org/main/docs/manualpages/Mat/MATPREALLOCATOR/<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMATPREALLOCATOR%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7Cfbe0ea83a9834c312a3108da81eed560%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637965160051619369%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=dgs3ZWbWM6cjLVKgk2TZ%2F%2FitR7cwNAfUNMTxyq3Wdk8%3D&reserved=0> to compute the needed preallocation efficiently.

  Are you running in parallel? If so how are you determining which rows of entries to compute on each MPI rank? You will want most of the rows to be computed on the rank where the values are stored, you can determine this with https://petsc.org/main/docs/manualpages/Mat/MatGetOwnershipRange.html<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMatGetOwnershipRange.html&data=05%7C01%7CJiannan_Tu%40uml.edu%7Cfbe0ea83a9834c312a3108da81eed560%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637965160051619369%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=YpT%2B7zNxT50v9BvuTKckuv1eQGyQKx6s1oed%2Bgg8XdY%3D&reserved=0>

  Barry















On Aug 18, 2022, at 12:50 AM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

I implemented the preconditioner. The solver converges quickly when the problem size is small. But when the size increases, say to 100k unknowns, the code hangs at assembly of the preconditioner matrix. The function call sequence is like this

   MatCreate(MPI_COMM_WORLD, &B);
    MatSetType(B, MATMPIAIJ);
    MatSetSizes(B, PETSC_DECIDE, PETSC_DECIDE, N, N);
    MatSetFromOptions(B);

    // The number of diagonal and off diagonal non zeros only can be determined run-time.
    MatMPIAIJSetPreallocation(B, 0, d_nnz, 0, o_nnz);
    MatSetOption(B, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE);

    //here insert values row by row.
MatSetValues(B, 1, &m, nn, nCols, matCols, INSERT_VALUES);

MatAssemblyBegin(B, MAT_FINAL_ASSEMBLY);
MatAssemblyEnd(B, MAT_FINAL_ASSEMBLY);

Could you please tell me what I have done wrong?

Thank you,
Jiannan

From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Tuesday, August 16, 2022 2:00 PM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] Using matrix-free with KSP

CAUTION: This email was sent from outside the UMass Lowell network.


   Create another matrix B that contains "Selected parts of the matrix can be pre-calculated and stored to serve as the preconditioner." When you call KSPSetOperators() pass in B as the second matrix argument.

   Now if literally application of B is the preconditioner you want to use, then simply use PCMAT as the preconditioner, and it will apply B each time it needs to apply the preconditioner. With this, you do not even need a PCShell.

   If your preconditioner uses B to construct your own specific data structure needed to apply your preconditioner, then you can use PCShellSetSetUp() and have your routine call PCGetOperators() to pull out the B matrix and use it to construct your preconditioner.

   If you want PETSc to construct a standard preconditioner from B, then you don't need the PCShell; you simply pass in the B, as above, and use -pc_type type to set the preconditioner you want to use.

   Barry








On Aug 16, 2022, at 12:31 PM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

Barry,

Thank you very much for your instructions. I am sorry I may not ask clearer questions.

I don't use SNES. What I am trying to solve is a very large linear equation system with dense and ill-conditioned matrix. Selected parts of the matrix can be pre-calculated and stored to serve as the preconditioner. My question is how I can apply this matrix as the preconditioner.

Here is the part of code from my linear equations solver. MatrixFreePreconditioner is supposed to be the routine to provide the preconditioner. But I don't understand how to use it. Should I use SNESSetJacobian() even if this is a linear problem?

    MatCreateMFFD(MPI_COMM_WORLD, PETSC_DECIDE, PETSC_DECIDE, N, N, &A);
    MatMFFDSetFunction(A, formfunction, &params);
    MatSetFromOptions(A);
    MatMFFDSetBase(A, X, NULL);

    KSPCreate(MPI_COMM_WORLD, &ksp);
    KSPSetOperators(ksp, A, A);
    KSPSetFromOptions(ksp);

    PC pc;
    KSPGetPC(ksp,&pc);
    PCSetType(pc,PCSHELL);
    PCShellSetApply(pc,MatrixFreePreconditioner);

Thank you,
Jiannan

________________________________
From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Tuesday, August 16, 2022 10:10 AM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] Using matrix-free with KSP

CAUTION: This email was sent from outside the UMass Lowell network.


  I don't fully understand your question so I will answer questions that I do know the answer to :-)

  The function you provide to PCShellSetApply() applies the preconditioner to the input vector and puts the result in the output vector.  This input vector changes at every application of the preconditioner so should not be used in constructing the preconditioner.

  If you are using the standard PETSc matrix-free matrix-vector product via finite difference, -snes_mf_operator or via MatCreateSNESMF() or you are using a MatShell, the matrix-vector product routines input changes for each multiply and should not be used in constructing the preconditioner.

  The function you provide with SNESSetJacobian() is where you can compute anything you need to help build your preconditioner. The input vector to that function is the location at which the Jacobian is needed and is what you should use to build your approximate/part of Jacobian. You can store your approximate/part of Jacobian that you need to compute in the pmat and then in a PCShellSetUp() function that you provide you will compute what you will need latter to apply the preconditioner in your PCShellSetApply() function. Note that the values you put in the pmat do not need to be the full Jacobian; they can be anything you want since this pmat is not used to apply the matrix vector product.

  If this is unclear, feel free to ask additional questions.

  Barry








On Aug 16, 2022, at 9:40 AM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

I am trying to apply a user-defined preconditioner to speed up the convergency of matrix-free KSP solver. The user's manual provides an example of how to call the user-defined preconditioner routine. The routine has PC, input Vec and output Vec as arguments. I want to use part of the Jacobian as preconditioner matrix, the elements of which can be calculated using input Vec. My question is what the role of output Vec from the routine is in matrix-vector product? Or I just have the preconditioner matrix elements calculated in the routine and then Petsc handles applying the preconditioner to matrix-vector product? I used PCSetType(pc, PCSHELL) and PCShellSetApply.

Thank you and appreciate your help!

Jiannan
________________________________
From: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Sent: Wednesday, July 6, 2022 2:34 PM
To: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Cc: Jed Brown <jed at jedbrown.org<mailto:jed at jedbrown.org>>; petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] Using matrix-free with KSP

Barry,

VecScatterCreateToAll solves the problem! The code now gives the correct answer with multiple processes.

Thank you all so much!

Jiannan
________________________________
From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Wednesday, July 6, 2022 2:08 PM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Cc: Jed Brown <jed at jedbrown.org<mailto:jed at jedbrown.org>>; petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] Using matrix-free with KSP


  So your operator is a "dense" operator in that its matrix representation would be essentially dense. In that case, you can use specialized routines to do this efficiently. You can use VecScatterCreateToAll() and then VecScatterBegin/End to do the communication.

  Barry









On Jul 6, 2022, at 1:39 PM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

Jed, thank you very much for the reply.

I'm not sure GlobaltoLocal will work. Say the solution vector is of length 10. Two processes then each process can see 5 elements of the vector. For A x = b (10 equations), each process performs matrix-vector product for five equations. And for each equation i , sum_j A_ij *  x_j = b_i requires the process has access to all 10 unknows x_i (i=0,1,...,9).

Can VecScatter and PetscSF make the entire vector accessible to each process? I am reading the manual and trying to understand how VecScatter and PetscSF work.

Jiannan
________________________________
From: Jed Brown <jed at jedbrown.org<mailto:jed at jedbrown.org>>
Sent: Wednesday, July 6, 2022 10:09 AM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>; Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] Using matrix-free with KSP

You'll usually have a GlobalToLocal operation for each rank to get the halo data it needs, then either a LocalToGlobal to collect the result (e.g., finite element methods) or the local compute will write directly into the owned portion of the global vector. If you're doing the communication "raw", then you may find VecScatter or PetscSF useful to perform the necessary communication.

"Tu, Jiannan" <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> writes:

> Hi Barry,
>
> Following your instructions I implemented the matrix-free method to solve a large linear equation system resulted from a surface integration equation. The KSP solver works fine for single process, but it is problematic with multiple processes. The problem is that each process only can access its own part of solution vector so that each process only conducts part of matrix-vector multiplication. MPI can be used to assemble these partial matrix-vector multiplication together.  Does Petsc provide any ways to implement multi-process matrix-free method?
>
> Thanks,
> Jiannan
> ________________________________
> From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
> Sent: Tuesday, May 24, 2022 2:12 PM
> To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
> Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
> Subject: Re: [petsc-users] Using matrix-free with KSP
>
> This e-mail originated from outside the UMass Lowell network.
> ________________________________
>
>    You can use MatCreateMFFD https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMatCreateMFFD%2F&amp;data=05%7C01%7CJiannan_Tu%40uml.edu%7Cab0859d8d154471590bc08da5f5918d6%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637927133525745809%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=A7wqegTfh94No5BpDiWLK3VxOuR44U2wlWHVm2k7l60%3D&amp;reserved=0<https://urldefense.com/v3/__https://petsc.org/main/docs/manualpages/Mat/MatCreateMFFD/__;!!PVKG_VDCxu5g!stxq2NWGVjUihvHBQC9Rlk0aHVoowcy0PTQYMcQxQ4DqOEC05KSw6TmstSXKckiUgelHzy4ue-d10-zlXkw$><https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMatCreateMFFD%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7Cfbe0ea83a9834c312a3108da81eed560%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637965160051619369%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=RNxXJRZlWi%2FHRzYkDAxH7wmxw21aRcQer21A0im9Ous%3D&reserved=0> MatMFFDSetFunction MatSetFromOptions MatMFFDSetBase and provide the matrix to KSP. Note you will need to use -pc_type none or provide your own custom preconditioner withhttps://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FPC%2FPCSHELL%2F&amp;data=05%7C01%7CJiannan_Tu%40uml.edu%7Cab0859d8d154471590bc08da5f5918d6%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637927133525745809%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=Gb%2F9uFG3jp%2FbfWaSh2cSEQLItHS9CuLwarzN0KcNiJY%3D&amp;reserved=0<https://urldefense.com/v3/__https://petsc.org/main/docs/manualpages/PC/PCSHELL/__;!!PVKG_VDCxu5g!stxq2NWGVjUihvHBQC9Rlk0aHVoowcy0PTQYMcQxQ4DqOEC05KSw6TmstSXKckiUgelHzy4ue-d1cA4fKGw$><https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FPC%2FPCSHELL%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7Cfbe0ea83a9834c312a3108da81eed560%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637965160051619369%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=PfRts5oplL0VR8NNMOo6SOXDtJ71v%2Fi55osmN2wSx80%3D&reserved=0>
>
>
>
> On May 24, 2022, at 1:21 PM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu><mailto:Jiannan_Tu at uml.edu>> wrote:
>
> I want to use a matrix-free matrix to solve a large linear equation system because the matrix is too large to be stored. Petsc user manual describes matrix-free method for SNES with examples. The matrix-free matrices section explains how to set up such a matrix, but I can't find any example of matrix-free method with KSP. I am wondering how to apply the method to KSP iterative solver in parallel.
>
> I greatly appreciate your help for me to understand this topic.
>
> Jiannan Tu

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220819/3d13a0f6/attachment-0001.html>

From bsmith at petsc.dev  Fri Aug 19 15:01:58 2022
From: bsmith at petsc.dev (Barry Smith)
Date: Fri, 19 Aug 2022 16:01:58 -0400
Subject: [petsc-users] Using matrix-free with KSP
In-Reply-To: <BY5PR02MB6471BA72871419781B875967E76C9@BY5PR02MB6471.namprd02.prod.outlook.com>
References: <BY5PR02MB6471969F80FA5F732FC74D87E7D79@BY5PR02MB6471.namprd02.prod.outlook.com>
	<B55CFC2C-6364-4B78-8E99-9D53700743B8@petsc.dev>
	<BY5PR02MB6471BDF90783FE3A0EA176CBE7809@BY5PR02MB6471.namprd02.prod.outlook.com>
	<87iloa49kt.fsf@jedbrown.org>
	<BY5PR02MB6471C0ED04769D02046F68ACE7809@BY5PR02MB6471.namprd02.prod.outlook.com>
	<00EBE4B8-22E5-408F-B56D-14226A278D7E@petsc.dev>
	<BY5PR02MB6471E386057DDDE98AD05DFAE7809@BY5PR02MB6471.namprd02.prod.outlook.com>
	<BY5PR02MB647140084C7E7CDDB6BC2A11E76B9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<A6629DE1-5A7F-414E-B164-A69877158675@petsc.dev>
	<BY5PR02MB6471AB1C004920DA1FC9B43EE76B9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<103146FD-FF28-47CE-B661-F2B521BF0E7A@petsc.dev>
	<DM6PR02MB6478B2FB8CB1BE757CE7B003E76D9@DM6PR02MB6478.namprd02.prod.outlook.com>
	<4C00F4B8-BB42-4932-BCD7-527E5FB39BFF@petsc.dev>
	<BY5PR02MB64710E70758E431108D66956E76D9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<D0EB13D8-C5A6-4AE7-AFDE-2CDCCACBAF73@petsc.dev>
	<BY5PR02MB6471D4B05874DECFDDA13FFCE76D9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<367B124C-88BB-4889-A4AF-45A1D473078D@petsc.dev>
	<BY5PR02MB6471F2C058B84DE092567731E76D9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<944DDEED-016D-43B2-A5A6-7BF2E8E41FA5@petsc.dev>
	<BY5PR02MB6471CD0236F03D20763A3B64E76C9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<B4CF5E5B-984D-4EC1-AB77-1FD08C3FBBCF@petsc.dev>
	<BY5PR02MB6471BA72871419781B875967E76C9@BY5PR02MB6471.namprd02.prod.outlook.com>
Message-ID: <3D179AEA-AD05-4B46-8DDC-9E126741C247@petsc.dev>


  I don't know why changing the PCType could matter, can you please try with your new code but use the -pc_type none option to turn off the use of the preconditioner to see if the error still appears (I think it should). 

  Also if you have a custom function that can do the matrix-vector product (not using any differencing) then you should use MatShell() and provide the matrix-vector product operation with your code.

  Barry


> On Aug 19, 2022, at 11:47 AM, Tu, Jiannan <Jiannan_Tu at uml.edu> wrote:
> 
> Barry,
>  
> I have my own matrix-vector product but not use MatShell. I followed your previous instruction to use
>  
>     MatCreateMFFD(MPI_COMM_WORLD, PETSC_DECIDE, PETSC_DECIDE, N, N, &A);
>     MatMFFDSetFunction(A, formfunction, &params);
>     MatSetFromOptions(A);
>     MatMFFDSetBase(A, X, NULL);
>  
> The customized matrix-vector product is implemented in formfunction() provided to MatMFFDSetFunction(). In this way, the matrix A is not used. I think that is why Petsc complains the matrix is not assembled. However, when PC type is none, the code runs without error although KSP solver doesn?t converge.
>  
> The question then becomes how to link A to matrix-free matrix-vector multiplication. And where to put MatShell?
>  
> Thank you,
> Jiannan
>  
> From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>> 
> Sent: Friday, August 19, 2022 10:27 AM
> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
> Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>
> Subject: Re: [petsc-users] Using matrix-free with KSP
>  
> CAUTION: This email was sent from outside the UMass Lowell network.
>  
>  
> 
> 
> On Aug 18, 2022, at 11:41 PM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> wrote:
>  
> The program calls KSPSolve just once and the error occurs after the first iteration.
>  
> I realized the matrix-vector product in my case is different from that approximating matrix-vector multiply by finite difference of function vector. It consists of two parts. One is multiplication of part of the matrix with part of vector, and another part results from so called fast multipole method. I?m not clear how to relate such calculation of matrix-vector product to the finite differencing of function vector.
>  
>   Are you using finite differencing at all in a matrix-vector product, or do you have your own custom matrix-vector product code (with MatShell)? 
>  
>   Anyways the easiest thing to do is 
>  
>   use the run time option -on_error_attach_debugger noxterm and run the code, 
>  
>   when it stops in the debugger you can type bt (for backtrace)
>  
>   then do up a few times to move up the stack until you get to the PETSc function call that generates the error
>  
>   you can then do "call MatView(thematrixvarible,0) to have PETSc display what the matrix is and
>  
>   you can also do print *thematrixvariable to see the exact struct with all its fields
>  
>   If this doesn't tell you why the troublesome matrix is not assembled then cut and paste ALL of the output and email it
>  
>   Barry
> 
> 
>  
>  
>  
> From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>> 
> Sent: Thursday, August 18, 2022 7:44 PM
> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
> Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>
> Subject: Re: [petsc-users] Using matrix-free with KSP
>  
> CAUTION: This email was sent from outside the UMass Lowell network.
>  
>  
>  
>    Is it in the first iteration of the first KSPSolve that you get this message? Or in the second call to KSPSolve? 
>  
>   The error comes from a matrix vector product with your A matrix, what is that and where did it come from. The error message is not from your B matrix, that is not used in the MatMult()
>  
> 
> 
> 
> On Aug 18, 2022, at 3:05 PM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> wrote:
>  
> Barry, 
>  
> Thanks. The method works. Assembly of actual matrix now completes quickly. The convergency is much better for the first iteration. The residual normal is in the order of 10^-10 instead of 10^-2. However, there is an error message after the first iteration: ?Object is in wrong state. Not for unassembled matrix? even though both preallocator and actual matrix have been assembled (see the code snippet below). Could you please give me some hints what?s going wrong? I use KSP type bcgs and PC type mat
>  
> Thank you,
> Jiannan
>  
> -----------------------------------------------------
> Solve linear equations ...
>   0 KSP Residual norm 9.841179147519e-10
> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [0]PETSC ERROR: Object is in wrong state
> [0]PETSC ERROR: Not for unassembled matrix
> [0]PETSC ERROR: See https://petsc.org/release/faq/ <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Frelease%2Ffaq%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7Cfbe0ea83a9834c312a3108da81eed560%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637965160051463154%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=xTipZKMBKvPiOJbq46z08835vo1pXko1P1lelR0wL7Y%3D&reserved=0> for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.16.6, Mar 30, 2022
> [0]PETSC ERROR: ./iesolver on a  named REXNET-WS4 by jiannantu Thu Aug 18 14:48:43 2022
> [0]PETSC ERROR: Configure options --download-f2cblaslapack=yes --with-mpi-dir=/usr/local --with-fc=0 --prefix=/home/jiannantu/petsc --with-scalar-type=complex --with-64-bit-indices=1
> [0]PETSC ERROR: #1 MatMult() at /home/jiannantu/petsc-3.16.6/src/mat/interface/matrix.c:2425
> [0]PETSC ERROR: #2 PCApplyBAorAB() at /home/jiannantu/petsc-3.16.6/src/ksp/pc/interface/precon.c:730
> [0]PETSC ERROR: #3 KSP_PCApplyBAorAB() at /home/jiannantu/petsc-3.16.6/include/petsc/private/kspimpl.h:421
> [0]PETSC ERROR:  Linear equations not solved. Diverged reason: 0
> #4 KSPSolve_BCGS() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/impls/bcgs/bcgs.c:87
> [0]PETSC ERROR: #5 KSPSolve_Private() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itfunc.c:914
> [0]PETSC ERROR: #6 KSPSolve() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itfunc.c:1086
>  
> -----------------------------------------------
>     MatCreate(MPI_COMM_WORLD, &B);
>     MatSetType(B, MATMPIAIJ);
>     MatSetSizes(B, PETSC_DECIDE, PETSC_DECIDE, N, N);
>  
>     Mat preall;
>     MatCreate(MPI_COMM_WORLD, &preall);
>     MatSetType(preall, MATPREALLOCATOR);
>     MatSetSizes(preall, PETSC_DECIDE, PETSC_DECIDE, N, N);
>  
>     MatMPIAIJSetPreallocation(preall, 0, d_nnz, 0, o_nnz);
>     MatSetUp(preall);
> MatSetOption(preall, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE);
>  
> //set values row by row
> MatSetValues(preall, 1, &m, 1, nCols, matCols, INSERT_VALUES);
>  
> MatAssemblyBegin(preall, MAT_FINAL_ASSEMBLY);
> MatAssemblyEnd(preall, MAT_FINAL_ASSEMBLY);
>  
> //set values for actual matrix row by row
> MatSetValues(B, 1, &m, 1, nCols, matCols, INSERT_VALUES);
>  
> MatAssemblyBegin(B, MAT_FINAL_ASSEMBLY);
> MatAssemblyEnd(B, MAT_FINAL_ASSEMBLY);
>  
>     KSPCreate(MPI_COMM_WORLD, &ksp);
>     KSPSetOperators(ksp, A, B);
>     KSPSetFromOptions(ksp);
>  
>     PC pc;
>     KSPGetPC(ksp, &pc);
>     PCSetFromOptions(pc);
>  
> From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>> 
> Sent: Thursday, August 18, 2022 12:59 PM
> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
> Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>
> Subject: Re: [petsc-users] Using matrix-free with KSP
>  
> CAUTION: This email was sent from outside the UMass Lowell network.
>  
>  
>   Yes, once you have preallocated the real matrix you can destroy the preallocation matrix whose only job is to gather the preallocation information
> 
> 
> 
> 
> On Aug 18, 2022, at 12:52 PM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> wrote:
>  
> Thanks, Barry.
>  
> So I need actually two matrixes, one for preallocator and one for actual matrix that can be passed to KSPSetOperators(). -mat_type preallocator option is used to speed up doing insertion into preallocator, then use MatPreallocatorPreallocate() to preallocate actual matrix and do actual insertion of values into it., right?
>  
> The code runs in parallel. Each process owns number of rows that equals to number of unknowns (that is, xm in 1D DM) it owns.
>  
> Jiannan
>  
> From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>> 
> Sent: Thursday, August 18, 2022 11:37 AM
> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
> Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>
> Subject: Re: [petsc-users] Using matrix-free with KSP
>  
> CAUTION: This email was sent from outside the UMass Lowell network.
>  
>  
>    The preallocator MatType matrix cannot be passed to the KSPSetOperators(), you need to create an actual matrix, for example MATAIJ and use the preallocator to set its preallocation and then fill up the MATAIJ matrix with the usual MatSetValues and pass that matrix to the KSPSetOperators.
> 
> 
> 
> 
> 
> On Aug 18, 2022, at 11:30 AM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> wrote:
>  
> Hi Barry,
>  
> The MATPREALLOCATOR solved problem of slow matrix assembly. But the solver failed because of ?Matrix type preallocator does not have a multiply defined?.
>  
> I guess this is because the matrix-free is used. I am wondering how a preconditioner is applied to the matrix-vector multiply in Petsc.
>  
> Thank you,
> Jiannan
>  
> [0]PETSC ERROR: No support for this operation for this object type
> [0]PETSC ERROR: Matrix type preallocator does not have a multiply defined
> [0]PETSC ERROR: See https://petsc.org/release/faq/ <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Frelease%2Ffaq%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7Cfbe0ea83a9834c312a3108da81eed560%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637965160051619369%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=YraVMtPwCNL9miYa6d55KjhtTyiROmHOn2MMnq3pqT4%3D&reserved=0> for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.16.6, Mar 30, 2022
> [0]PETSC ERROR: ./iesolver on a  named REXNET-WS4 by jiannantu Thu Aug 18 11:21:55 2022
> [0]PETSC ERROR: Configure options --download-f2cblaslapack=yes --with-mpi-dir=/usr/local --with-fc=0 --prefix=/home/jiannantu/petsc --with-scalar-type=complex --with-64-bit-indices=1
> [0]PETSC ERROR: #1 MatMult() at /home/jiannantu/petsc-3.16.6/src/mat/interface/matrix.c:2437
> [0]PETSC ERROR: #2 PCApply_Mat() at /home/jiannantu/petsc-3.16.6/src/ksp/pc/impls/mat/pcmat.c:9
> [0]PETSC ERROR: #3 PCApply() at /home/jiannantu/petsc-3.16.6/src/ksp/pc/interface/precon.c:445
> [0]PETSC ERROR: #4 KSP_PCApply() at /home/jiannantu/petsc-3.16.6/include/petsc/private/kspimpl.h:382
> [0]PETSC ERROR: #5 KSPInitialResidual() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itres.c:65
> [0]PETSC ERROR: #6 KSPSolve_BCGS() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/impls/bcgs/bcgs.c:43
> [0]PETSC ERROR: #7 KSPSolve_Private() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itfunc.c:914
> [0]PETSC ERROR: #8 KSPSolve() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itfunc.c:1086
>  
> From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>> 
> Sent: Thursday, August 18, 2022 8:35 AM
> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
> Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>
> Subject: Re: [petsc-users] Using matrix-free with KSP
>  
> CAUTION: This email was sent from outside the UMass Lowell network.
>  
>  
>   Slow assembly is usually due to under preallocation. You can change to using 
>  
> MatSetOption(B, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_TRUE);
>  
>   to detect if you are under preallocating. See https://petsc.org/main/docs/manual/mat/#sec-matsparse <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanual%2Fmat%2F%23sec-matsparse&data=05%7C01%7CJiannan_Tu%40uml.edu%7Cfbe0ea83a9834c312a3108da81eed560%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637965160051619369%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=9c3JU05Mb41i%2BIwJlvz9LsTqh1UlbaTpe1wjEtW6HHQ%3D&reserved=0>. That section clearly needs some sprucing up. If it is difficult to determine good preallocation values you can use https://petsc.org/main/docs/manualpages/Mat/MATPREALLOCATOR/ <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMATPREALLOCATOR%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7Cfbe0ea83a9834c312a3108da81eed560%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637965160051619369%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=dgs3ZWbWM6cjLVKgk2TZ%2F%2FitR7cwNAfUNMTxyq3Wdk8%3D&reserved=0> to compute the needed preallocation efficiently.
>  
>   Are you running in parallel? If so how are you determining which rows of entries to compute on each MPI rank? You will want most of the rows to be computed on the rank where the values are stored, you can determine this with https://petsc.org/main/docs/manualpages/Mat/MatGetOwnershipRange.html <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMatGetOwnershipRange.html&data=05%7C01%7CJiannan_Tu%40uml.edu%7Cfbe0ea83a9834c312a3108da81eed560%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637965160051619369%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=YpT%2B7zNxT50v9BvuTKckuv1eQGyQKx6s1oed%2Bgg8XdY%3D&reserved=0>
>  
>   Barry
>  
>  
> 
> 
> 
> 
> 
> 
>  
> 
> 
> 
> 
> 
> 
> On Aug 18, 2022, at 12:50 AM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> wrote:
>  
> I implemented the preconditioner. The solver converges quickly when the problem size is small. But when the size increases, say to 100k unknowns, the code hangs at assembly of the preconditioner matrix. The function call sequence is like this
>  
>    MatCreate(MPI_COMM_WORLD, &B);
>     MatSetType(B, MATMPIAIJ);
>     MatSetSizes(B, PETSC_DECIDE, PETSC_DECIDE, N, N);
>     MatSetFromOptions(B);
>  
>     // The number of diagonal and off diagonal non zeros only can be determined run-time.
>     MatMPIAIJSetPreallocation(B, 0, d_nnz, 0, o_nnz);
>     MatSetOption(B, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE);
>  
>     //here insert values row by row.
> MatSetValues(B, 1, &m, nn, nCols, matCols, INSERT_VALUES);
>  
> MatAssemblyBegin(B, MAT_FINAL_ASSEMBLY);
> MatAssemblyEnd(B, MAT_FINAL_ASSEMBLY);
>  
> Could you please tell me what I have done wrong?
>  
> Thank you,
> Jiannan
>    
> From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>> 
> Sent: Tuesday, August 16, 2022 2:00 PM
> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
> Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>
> Subject: Re: [petsc-users] Using matrix-free with KSP
>  
> CAUTION: This email was sent from outside the UMass Lowell network.
>  
>  
>    Create another matrix B that contains "Selected parts of the matrix can be pre-calculated and stored to serve as the preconditioner." When you call KSPSetOperators() pass in B as the second matrix argument.
>  
>    Now if literally application of B is the preconditioner you want to use, then simply use PCMAT as the preconditioner, and it will apply B each time it needs to apply the preconditioner. With this, you do not even need a PCShell. 
>  
>    If your preconditioner uses B to construct your own specific data structure needed to apply your preconditioner, then you can use PCShellSetSetUp() and have your routine call PCGetOperators() to pull out the B matrix and use it to construct your preconditioner. 
>  
>    If you want PETSc to construct a standard preconditioner from B, then you don't need the PCShell; you simply pass in the B, as above, and use -pc_type type to set the preconditioner you want to use. 
>  
>    Barry
>  
> 
> 
> 
> 
> 
> 
> 
> On Aug 16, 2022, at 12:31 PM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> wrote:
>  
> Barry,
>  
> Thank you very much for your instructions. I am sorry I may not ask clearer questions.
>  
> I don't use SNES. What I am trying to solve is a very large linear equation system with dense and ill-conditioned matrix. Selected parts of the matrix can be pre-calculated and stored to serve as the preconditioner. My question is how I can apply this matrix as the preconditioner. 
>  
> Here is the part of code from my linear equations solver. MatrixFreePreconditioner is supposed to be the routine to provide the preconditioner. But I don't understand how to use it. Should I use SNESSetJacobian() even if this is a linear problem? 
>  
>     MatCreateMFFD(MPI_COMM_WORLD, PETSC_DECIDE, PETSC_DECIDE, N, N, &A);
>     MatMFFDSetFunction(A, formfunction, &params);
>     MatSetFromOptions(A);
>     MatMFFDSetBase(A, X, NULL);
>  
>     KSPCreate(MPI_COMM_WORLD, &ksp);
>     KSPSetOperators(ksp, A, A);
>     KSPSetFromOptions(ksp);
>  
>     PC pc;
>     KSPGetPC(ksp,&pc);
>     PCSetType(pc,PCSHELL);
>     PCShellSetApply(pc,MatrixFreePreconditioner);
>  
> Thank you,
> Jiannan
>  
> From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>>
> Sent: Tuesday, August 16, 2022 10:10 AM
> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
> Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>>
> Subject: Re: [petsc-users] Using matrix-free with KSP
>  
> CAUTION: This email was sent from outside the UMass Lowell network.
>  
>  
>   I don't fully understand your question so I will answer questions that I do know the answer to :-)
>  
>   The function you provide to PCShellSetApply() applies the preconditioner to the input vector and puts the result in the output vector.  This input vector changes at every application of the preconditioner so should not be used in constructing the preconditioner.
>  
>   If you are using the standard PETSc matrix-free matrix-vector product via finite difference, -snes_mf_operator or via MatCreateSNESMF() or you are using a MatShell, the matrix-vector product routines input changes for each multiply and should not be used in constructing the preconditioner.
>  
>   The function you provide with SNESSetJacobian() is where you can compute anything you need to help build your preconditioner. The input vector to that function is the location at which the Jacobian is needed and is what you should use to build your approximate/part of Jacobian. You can store your approximate/part of Jacobian that you need to compute in the pmat and then in a PCShellSetUp() function that you provide you will compute what you will need latter to apply the preconditioner in your PCShellSetApply() function. Note that the values you put in the pmat do not need to be the full Jacobian; they can be anything you want since this pmat is not used to apply the matrix vector product.
>  
>   If this is unclear, feel free to ask additional questions.
>  
>   Barry
>  
> 
> 
> 
> 
> 
> 
> 
> On Aug 16, 2022, at 9:40 AM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> wrote:
>  
> I am trying to apply a user-defined preconditioner to speed up the convergency of matrix-free KSP solver. The user's manual provides an example of how to call the user-defined preconditioner routine. The routine has PC, input Vec and output Vec as arguments. I want to use part of the Jacobian as preconditioner matrix, the elements of which can be calculated using input Vec. My question is what the role of output Vec from the routine is in matrix-vector product? Or I just have the preconditioner matrix elements calculated in the routine and then Petsc handles applying the preconditioner to matrix-vector product? I used PCSetType(pc, PCSHELL) and PCShellSetApply.
>  
> Thank you and appreciate your help!
>  
> Jiannan 
> From: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
> Sent: Wednesday, July 6, 2022 2:34 PM
> To: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>>
> Cc: Jed Brown <jed at jedbrown.org <mailto:jed at jedbrown.org>>; petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>>
> Subject: Re: [petsc-users] Using matrix-free with KSP
>  
> Barry,
>  
> VecScatterCreateToAll solves the problem! The code now gives the correct answer with multiple processes. 
>  
> Thank you all so much!
>  
> Jiannan
> From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>>
> Sent: Wednesday, July 6, 2022 2:08 PM
> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
> Cc: Jed Brown <jed at jedbrown.org <mailto:jed at jedbrown.org>>; petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>>
> Subject: Re: [petsc-users] Using matrix-free with KSP
>  
>  
>   So your operator is a "dense" operator in that its matrix representation would be essentially dense. In that case, you can use specialized routines to do this efficiently. You can use VecScatterCreateToAll() and then VecScatterBegin/End to do the communication.
>  
>   Barry
>  
>  
> 
> 
> 
> 
> 
> 
> 
> On Jul 6, 2022, at 1:39 PM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> wrote:
>  
> Jed, thank you very much for the reply. 
>  
> I'm not sure GlobaltoLocal will work. Say the solution vector is of length 10. Two processes then each process can see 5 elements of the vector. For A x = b (10 equations), each process performs matrix-vector product for five equations. And for each equation i , sum_j A_ij *  x_j = b_i requires the process has access to all 10 unknows x_i (i=0,1,...,9). 
>  
> Can VecScatter and PetscSF make the entire vector accessible to each process? I am reading the manual and trying to understand how VecScatter and PetscSF work.
>  
> Jiannan
> From: Jed Brown <jed at jedbrown.org <mailto:jed at jedbrown.org>>
> Sent: Wednesday, July 6, 2022 10:09 AM
> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>; Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>>
> Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>>
> Subject: Re: [petsc-users] Using matrix-free with KSP
>  
> You'll usually have a GlobalToLocal operation for each rank to get the halo data it needs, then either a LocalToGlobal to collect the result (e.g., finite element methods) or the local compute will write directly into the owned portion of the global vector. If you're doing the communication "raw", then you may find VecScatter or PetscSF useful to perform the necessary communication.
> 
> "Tu, Jiannan" <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> writes:
> 
> > Hi Barry,
> >
> > Following your instructions I implemented the matrix-free method to solve a large linear equation system resulted from a surface integration equation. The KSP solver works fine for single process, but it is problematic with multiple processes. The problem is that each process only can access its own part of solution vector so that each process only conducts part of matrix-vector multiplication. MPI can be used to assemble these partial matrix-vector multiplication together.  Does Petsc provide any ways to implement multi-process matrix-free method?
> >
> > Thanks,
> > Jiannan
> > ________________________________
> > From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>>
> > Sent: Tuesday, May 24, 2022 2:12 PM
> > To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
> > Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>>
> > Subject: Re: [petsc-users] Using matrix-free with KSP
> >
> > This e-mail originated from outside the UMass Lowell network.
> > ________________________________
> >
> >    You can use MatCreateMFFD https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMatCreateMFFD%2F&amp;data=05%7C01%7CJiannan_Tu%40uml.edu%7Cab0859d8d154471590bc08da5f5918d6%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637927133525745809%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=A7wqegTfh94No5BpDiWLK3VxOuR44U2wlWHVm2k7l60%3D&amp;reserved=0<https://urldefense.com/v3/__https://petsc.org/main/docs/manualpages/Mat/MatCreateMFFD/__;!!PVKG_VDCxu5g!stxq2NWGVjUihvHBQC9Rlk0aHVoowcy0PTQYMcQxQ4DqOEC05KSw6TmstSXKckiUgelHzy4ue-d10-zlXkw$> <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMatCreateMFFD%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7Cfbe0ea83a9834c312a3108da81eed560%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637965160051619369%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=RNxXJRZlWi%2FHRzYkDAxH7wmxw21aRcQer21A0im9Ous%3D&reserved=0> MatMFFDSetFunction MatSetFromOptions MatMFFDSetBase and provide the matrix to KSP. Note you will need to use -pc_type none or provide your own custom preconditioner withhttps://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FPC%2FPCSHELL%2F&amp;data=05%7C01%7CJiannan_Tu%40uml.edu%7Cab0859d8d154471590bc08da5f5918d6%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637927133525745809%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=Gb%2F9uFG3jp%2FbfWaSh2cSEQLItHS9CuLwarzN0KcNiJY%3D&amp;reserved=0<https://urldefense.com/v3/__https://petsc.org/main/docs/manualpages/PC/PCSHELL/__;!!PVKG_VDCxu5g!stxq2NWGVjUihvHBQC9Rlk0aHVoowcy0PTQYMcQxQ4DqOEC05KSw6TmstSXKckiUgelHzy4ue-d1cA4fKGw$> <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FPC%2FPCSHELL%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7Cfbe0ea83a9834c312a3108da81eed560%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637965160051619369%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=PfRts5oplL0VR8NNMOo6SOXDtJ71v%2Fi55osmN2wSx80%3D&reserved=0>
> >
> >
> >
> > On May 24, 2022, at 1:21 PM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu><mailto:Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>> wrote:
> >
> > I want to use a matrix-free matrix to solve a large linear equation system because the matrix is too large to be stored. Petsc user manual describes matrix-free method for SNES with examples. The matrix-free matrices section explains how to set up such a matrix, but I can't find any example of matrix-free method with KSP. I am wondering how to apply the method to KSP iterative solver in parallel.
> >
> > I greatly appreciate your help for me to understand this topic.
> >
> > Jiannan Tu

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220819/ea696ff3/attachment-0001.html>

From drwells at email.unc.edu  Fri Aug 19 16:21:32 2022
From: drwells at email.unc.edu (Wells, David)
Date: Fri, 19 Aug 2022 21:21:32 +0000
Subject: [petsc-users] Using VecGetType() and VecNest
Message-ID: <MN2PR03MB48791D953204760BD052F61EED6C9@MN2PR03MB4879.namprd03.prod.outlook.com>

Hello PETSc experts,

I am using VecNest to solve a Stokes problem and I ran into an issue using the POD KSPGuess routines:

[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Unknown type. Check for miss-spelling or missing package: https://petsc.org/release/install/install/#external-packages
[0]PETSC ERROR: Unknown vector type: nest
[0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.17.0, Mar 30, 2022
[0]PETSC ERROR: /[...]/step-32/step-32 on a  named mitral by drwells Fri Aug 19 16:33:08 2022
[0]PETSC ERROR: Configure options [...]
[0]PETSC ERROR: #1 VecSetType() at /afs/cas.unc.edu/users/d/r/drwells/Documents/Code/C/petsc-3.17.0/src/vec/vec/interface/vecreg.c:80
[0]PETSC ERROR: #2 KSPGuessSetUp_POD() at /afs/cas.unc.edu/users/d/r/drwells/Documents/Code/C/petsc-3.17.0/src/ksp/ksp/guess/impls/pod/pod.c:118
[0]PETSC ERROR: #3 KSPGuessSetUp() at /afs/cas.unc.edu/users/d/r/drwells/Documents/Code/C/petsc-3.17.0/src/ksp/ksp/interface/iguess.c:352
[0]PETSC ERROR: #4 KSPSolve_Private() at /afs/cas.unc.edu/users/d/r/drwells/Documents/Code/C/petsc-3.17.0/src/ksp/ksp/interface/itfunc.c:830
[0]PETSC ERROR: #5 KSPSolve() at /afs/cas.unc.edu/users/d/r/drwells/Documents/Code/C/petsc-3.17.0/src/ksp/ksp/interface/itfunc.c:1078


It looks like VecGetType() works with VecNest while VecSetType() does not. I also noticed that VecNest isn't listed in VecRegisterAll().

I think I can write a patch to make the POD KSPGuess work with VecNest, but: should VecSetType() work with VecNest? It seems like this is an oversight but I'd like to know if there is some fundamental reason why the sequence used there

    PetscCall(KSPCreateVecs(guess->ksp,1,&v,0,NULL));
    PetscCall(VecCreate(PETSC_COMM_SELF,&vseq));
    PetscCall(VecGetLocalSize(v[0],&n));
    PetscCall(VecSetSizes(vseq,n,n));
    PetscCall(VecGetType(v[0],&type));
    PetscCall(VecSetType(vseq,type));
    PetscCall(VecDestroyVecs(1,&v));
    PetscCall(VecDuplicateVecs(vseq,pod->maxn,&pod->xsnap));
    PetscCall(VecDestroy(&vseq));
    PetscCall(PetscLogObjectParents(guess,pod->maxn,pod->xsnap));

can't work.

Best,
David Wells

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220819/aff88d79/attachment.html>

From bsmith at petsc.dev  Fri Aug 19 22:29:59 2022
From: bsmith at petsc.dev (Barry Smith)
Date: Fri, 19 Aug 2022 23:29:59 -0400
Subject: [petsc-users] Using VecGetType() and VecNest
In-Reply-To: <MN2PR03MB48791D953204760BD052F61EED6C9@MN2PR03MB4879.namprd03.prod.outlook.com>
References: <MN2PR03MB48791D953204760BD052F61EED6C9@MN2PR03MB4879.namprd03.prod.outlook.com>
Message-ID: <3D1B7A44-4FE7-4EA6-872C-85B313038940@petsc.dev>


  I am not sure why the code works this way creating these sequential work vectors. Since VecNest needs some information about the subvectors I don't think just setting these work vectors to nest vectors will work properly. 

  I am cc:ing Stefano who wrote the code and can likely say immediately what the solution is.

  Barry


> On Aug 19, 2022, at 5:21 PM, Wells, David <drwells at email.unc.edu> wrote:
> 
> Hello PETSc experts,
> 
> I am using VecNest to solve a Stokes problem and I ran into an issue using the POD KSPGuess routines:
> 
> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [0]PETSC ERROR: Unknown type. Check for miss-spelling or missing package: https://petsc.org/release/install/install/#external-packages <https://petsc.org/release/install/install/#external-packages>
> [0]PETSC ERROR: Unknown vector type: nest
> [0]PETSC ERROR: See https://petsc.org/release/faq/ <https://petsc.org/release/faq/> for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.17.0, Mar 30, 2022 
> [0]PETSC ERROR: /[...]/step-32/step-32 on a  named mitral by drwells Fri Aug 19 16:33:08 2022
> [0]PETSC ERROR: Configure options [...]
> [0]PETSC ERROR: #1 VecSetType() at /afs/cas.unc.edu/users/d/r/drwells/Documents/Code/C/petsc-3.17.0/src/vec/vec/interface/vecreg.c:80 <http://cas.unc.edu/users/d/r/drwells/Documents/Code/C/petsc-3.17.0/src/vec/vec/interface/vecreg.c:80>
> [0]PETSC ERROR: #2 KSPGuessSetUp_POD() at /afs/cas.unc.edu/users/d/r/drwells/Documents/Code/C/petsc-3.17.0/src/ksp/ksp/guess/impls/pod/pod.c:118 <http://cas.unc.edu/users/d/r/drwells/Documents/Code/C/petsc-3.17.0/src/ksp/ksp/guess/impls/pod/pod.c:118>
> [0]PETSC ERROR: #3 KSPGuessSetUp() at /afs/cas.unc.edu/users/d/r/drwells/Documents/Code/C/petsc-3.17.0/src/ksp/ksp/interface/iguess.c:352 <http://cas.unc.edu/users/d/r/drwells/Documents/Code/C/petsc-3.17.0/src/ksp/ksp/interface/iguess.c:352>
> [0]PETSC ERROR: #4 KSPSolve_Private() at /afs/cas.unc.edu/users/d/r/drwells/Documents/Code/C/petsc-3.17.0/src/ksp/ksp/interface/itfunc.c:830 <http://cas.unc.edu/users/d/r/drwells/Documents/Code/C/petsc-3.17.0/src/ksp/ksp/interface/itfunc.c:830>
> [0]PETSC ERROR: #5 KSPSolve() at /afs/cas.unc.edu/users/d/r/drwells/Documents/Code/C/petsc-3.17.0/src/ksp/ksp/interface/itfunc.c:1078 <http://cas.unc.edu/users/d/r/drwells/Documents/Code/C/petsc-3.17.0/src/ksp/ksp/interface/itfunc.c:1078>
> 
> 
> It looks like VecGetType() works with VecNest while VecSetType() does not. I also noticed that VecNest isn't listed in VecRegisterAll().
> 
> I think I can write a patch to make the POD KSPGuess work with VecNest, but: should VecSetType() work with VecNest? It seems like this is an oversight but I'd like to know if there is some fundamental reason why the sequence used there
> 
>     PetscCall(KSPCreateVecs(guess->ksp,1,&v,0,NULL));
>     PetscCall(VecCreate(PETSC_COMM_SELF,&vseq));
>     PetscCall(VecGetLocalSize(v[0],&n));
>     PetscCall(VecSetSizes(vseq,n,n));
>     PetscCall(VecGetType(v[0],&type));
>     PetscCall(VecSetType(vseq,type));
>     PetscCall(VecDestroyVecs(1,&v));
>     PetscCall(VecDuplicateVecs(vseq,pod->maxn,&pod->xsnap));
>     PetscCall(VecDestroy(&vseq));
>     PetscCall(PetscLogObjectParents(guess,pod->maxn,pod->xsnap));
> 
> can't work.
> 
> Best,
> David Wells

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220819/863aea23/attachment.html>

From stefano.zampini at gmail.com  Sat Aug 20 03:08:00 2022
From: stefano.zampini at gmail.com (Stefano Zampini)
Date: Sat, 20 Aug 2022 10:08:00 +0200
Subject: [petsc-users] Using VecGetType() and VecNest
In-Reply-To: <3D1B7A44-4FE7-4EA6-872C-85B313038940@petsc.dev>
References: <MN2PR03MB48791D953204760BD052F61EED6C9@MN2PR03MB4879.namprd03.prod.outlook.com>
	<3D1B7A44-4FE7-4EA6-872C-85B313038940@petsc.dev>
Message-ID: <CAGPUisjXi3sPqysW8ey9KbgpdNzLKgS1akzobnu9wFtm_coE6Q@mail.gmail.com>

The idea of the pod code was to create local vectors to perform local
products and use mpi non blocking collectives . Not sure how to fix for the
nest case. I'm on vacation now, I can take a look in a couple of weeks or so

On Sat, Aug 20, 2022, 05:30 Barry Smith <bsmith at petsc.dev> wrote:

>
>   I am not sure why the code works this way creating these sequential work
> vectors. Since VecNest needs some information about the subvectors I don't
> think just setting these work vectors to nest vectors will work properly.
>
>   I am cc:ing Stefano who wrote the code and can likely say immediately
> what the solution is.
>
>   Barry
>
>
> On Aug 19, 2022, at 5:21 PM, Wells, David <drwells at email.unc.edu> wrote:
>
> Hello PETSc experts,
>
> I am using VecNest to solve a Stokes problem and I ran into an issue using
> the POD KSPGuess routines:
>
> [0]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [0]PETSC ERROR: Unknown type. Check for miss-spelling or missing package:
> https://petsc.org/release/install/install/#external-packages
> [0]PETSC ERROR: Unknown vector type: nest
> [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.17.0, Mar 30, 2022
> [0]PETSC ERROR: /[...]/step-32/step-32 on a  named mitral by drwells Fri
> Aug 19 16:33:08 2022
> [0]PETSC ERROR: Configure options [...]
> [0]PETSC ERROR: #1 VecSetType() at /afs/
> cas.unc.edu/users/d/r/drwells/Documents/Code/C/petsc-3.17.0/src/vec/vec/interface/vecreg.c:80
> [0]PETSC ERROR: #2 KSPGuessSetUp_POD() at /afs/
> cas.unc.edu/users/d/r/drwells/Documents/Code/C/petsc-3.17.0/src/ksp/ksp/guess/impls/pod/pod.c:118
> [0]PETSC ERROR: #3 KSPGuessSetUp() at /afs/
> cas.unc.edu/users/d/r/drwells/Documents/Code/C/petsc-3.17.0/src/ksp/ksp/interface/iguess.c:352
> [0]PETSC ERROR: #4 KSPSolve_Private() at /afs/
> cas.unc.edu/users/d/r/drwells/Documents/Code/C/petsc-3.17.0/src/ksp/ksp/interface/itfunc.c:830
> [0]PETSC ERROR: #5 KSPSolve() at /afs/
> cas.unc.edu/users/d/r/drwells/Documents/Code/C/petsc-3.17.0/src/ksp/ksp/interface/itfunc.c:1078
>
>
> It looks like VecGetType() works with VecNest while VecSetType() does not.
> I also noticed that VecNest isn't listed in VecRegisterAll().
>
> I think I can write a patch to make the POD KSPGuess work with VecNest,
> but: should VecSetType() work with VecNest? It seems like this is an
> oversight but I'd like to know if there is some fundamental reason why the
> sequence used there
>
>     PetscCall(KSPCreateVecs(guess->ksp,1,&v,0,NULL));
>     PetscCall(VecCreate(PETSC_COMM_SELF,&vseq));
>     PetscCall(VecGetLocalSize(v[0],&n));
>     PetscCall(VecSetSizes(vseq,n,n));
>     PetscCall(VecGetType(v[0],&type));
>     PetscCall(VecSetType(vseq,type));
>     PetscCall(VecDestroyVecs(1,&v));
>     PetscCall(VecDuplicateVecs(vseq,pod->maxn,&pod->xsnap));
>     PetscCall(VecDestroy(&vseq));
>     PetscCall(PetscLogObjectParents(guess,pod->maxn,pod->xsnap));
>
> can't work.
>
> Best,
> David Wells
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220820/744a2c0c/attachment-0001.html>

From Jiannan_Tu at uml.edu  Sun Aug 21 22:18:44 2022
From: Jiannan_Tu at uml.edu (Tu, Jiannan)
Date: Mon, 22 Aug 2022 03:18:44 +0000
Subject: [petsc-users] Using matrix-free with KSP
In-Reply-To: <3D179AEA-AD05-4B46-8DDC-9E126741C247@petsc.dev>
References: <BY5PR02MB6471969F80FA5F732FC74D87E7D79@BY5PR02MB6471.namprd02.prod.outlook.com>
	<B55CFC2C-6364-4B78-8E99-9D53700743B8@petsc.dev>
	<BY5PR02MB6471BDF90783FE3A0EA176CBE7809@BY5PR02MB6471.namprd02.prod.outlook.com>
	<87iloa49kt.fsf@jedbrown.org>
	<BY5PR02MB6471C0ED04769D02046F68ACE7809@BY5PR02MB6471.namprd02.prod.outlook.com>
	<00EBE4B8-22E5-408F-B56D-14226A278D7E@petsc.dev>
	<BY5PR02MB6471E386057DDDE98AD05DFAE7809@BY5PR02MB6471.namprd02.prod.outlook.com>
	<BY5PR02MB647140084C7E7CDDB6BC2A11E76B9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<A6629DE1-5A7F-414E-B164-A69877158675@petsc.dev>
	<BY5PR02MB6471AB1C004920DA1FC9B43EE76B9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<103146FD-FF28-47CE-B661-F2B521BF0E7A@petsc.dev>
	<DM6PR02MB6478B2FB8CB1BE757CE7B003E76D9@DM6PR02MB6478.namprd02.prod.outlook.com>
	<4C00F4B8-BB42-4932-BCD7-527E5FB39BFF@petsc.dev>
	<BY5PR02MB64710E70758E431108D66956E76D9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<D0EB13D8-C5A6-4AE7-AFDE-2CDCCACBAF73@petsc.dev>
	<BY5PR02MB6471D4B05874DECFDDA13FFCE76D9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<367B124C-88BB-4889-A4AF-45A1D473078D@petsc.dev>
	<BY5PR02MB6471F2C058B84DE092567731E76D9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<944DDEED-016D-43B2-A5A6-7BF2E8E41FA5@petsc.dev>
	<BY5PR02MB6471CD0236F03D20763A3B64E76C9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<B4CF5E5B-984D-4EC1-AB77-1FD08C3FBBCF@petsc.dev>
	<BY5PR02MB6471BA72871419781B875967E76C9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<3D179AEA-AD05-4B46-8DDC-9E126741C247@petsc.dev>
Message-ID: <BY5PR02MB6471560EA70C6FAE826DDF27E7719@BY5PR02MB6471.namprd02.prod.outlook.com>

Barry,

You are right. With PC type none, the same error appears.

Now I use MatShell and I literally have three matrixes. A is a shell matrix, B is a preconditioner, and C is a matrix just for preallocation purpose.

First C is used to determine non-zero structure. Then use MatPreallocatorPreallocation(C, PETSC_FALSE, B) to preallocate for B. After that do insertion and assembly for B. These steps work fine.

Now the question is how to insert and assemble shell matrix A. MatPreallocatorPreallocation doesn't work with shell matrix as indicated by an error message. MatSetValues or Mat assembly doesn't support shell matrix either.

Thank you,
Jiannan

From: Barry Smith <bsmith at petsc.dev>
Sent: Friday, August 19, 2022 4:02 PM
To: Tu, Jiannan <Jiannan_Tu at uml.edu>
Cc: petsc-users at mcs.anl.gov
Subject: Re: [petsc-users] Using matrix-free with KSP

CAUTION: This email was sent from outside the UMass Lowell network.


  I don't know why changing the PCType could matter, can you please try with your new code but use the -pc_type none option to turn off the use of the preconditioner to see if the error still appears (I think it should).

  Also if you have a custom function that can do the matrix-vector product (not using any differencing) then you should use MatShell() and provide the matrix-vector product operation with your code.

  Barry



On Aug 19, 2022, at 11:47 AM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

Barry,

I have my own matrix-vector product but not use MatShell. I followed your previous instruction to use

    MatCreateMFFD(MPI_COMM_WORLD, PETSC_DECIDE, PETSC_DECIDE, N, N, &A);
    MatMFFDSetFunction(A, formfunction, &params);
    MatSetFromOptions(A);
    MatMFFDSetBase(A, X, NULL);

The customized matrix-vector product is implemented in formfunction() provided to MatMFFDSetFunction(). In this way, the matrix A is not used. I think that is why Petsc complains the matrix is not assembled. However, when PC type is none, the code runs without error although KSP solver doesn't converge.

The question then becomes how to link A to matrix-free matrix-vector multiplication. And where to put MatShell?

Thank you,
Jiannan

From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Friday, August 19, 2022 10:27 AM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] Using matrix-free with KSP

CAUTION: This email was sent from outside the UMass Lowell network.





On Aug 18, 2022, at 11:41 PM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

The program calls KSPSolve just once and the error occurs after the first iteration.

I realized the matrix-vector product in my case is different from that approximating matrix-vector multiply by finite difference of function vector. It consists of two parts. One is multiplication of part of the matrix with part of vector, and another part results from so called fast multipole method. I'm not clear how to relate such calculation of matrix-vector product to the finite differencing of function vector.

  Are you using finite differencing at all in a matrix-vector product, or do you have your own custom matrix-vector product code (with MatShell)?

  Anyways the easiest thing to do is

  use the run time option -on_error_attach_debugger noxterm and run the code,

  when it stops in the debugger you can type bt (for backtrace)

  then do up a few times to move up the stack until you get to the PETSc function call that generates the error

  you can then do "call MatView(thematrixvarible,0) to have PETSc display what the matrix is and

  you can also do print *thematrixvariable to see the exact struct with all its fields

  If this doesn't tell you why the troublesome matrix is not assembled then cut and paste ALL of the output and email it

  Barry






From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Thursday, August 18, 2022 7:44 PM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] Using matrix-free with KSP

CAUTION: This email was sent from outside the UMass Lowell network.



   Is it in the first iteration of the first KSPSolve that you get this message? Or in the second call to KSPSolve?

  The error comes from a matrix vector product with your A matrix, what is that and where did it come from. The error message is not from your B matrix, that is not used in the MatMult()





On Aug 18, 2022, at 3:05 PM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

Barry,

Thanks. The method works. Assembly of actual matrix now completes quickly. The convergency is much better for the first iteration. The residual normal is in the order of 10^-10 instead of 10^-2. However, there is an error message after the first iteration: "Object is in wrong state. Not for unassembled matrix" even though both preallocator and actual matrix have been assembled (see the code snippet below). Could you please give me some hints what's going wrong? I use KSP type bcgs and PC type mat

Thank you,
Jiannan

-----------------------------------------------------
Solve linear equations ...
  0 KSP Residual norm 9.841179147519e-10
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Object is in wrong state
[0]PETSC ERROR: Not for unassembled matrix
[0]PETSC ERROR: See https://petsc.org/release/faq/<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Frelease%2Ffaq%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7Cec4aed5ceff5479ec1ce08da821daee5%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637965361283417283%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=ISQJqjmeBCfGq3Ufro5G1RhJiw%2BHVsnQXSSGxAu0mj8%3D&reserved=0> for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.16.6, Mar 30, 2022
[0]PETSC ERROR: ./iesolver on a  named REXNET-WS4 by jiannantu Thu Aug 18 14:48:43 2022
[0]PETSC ERROR: Configure options --download-f2cblaslapack=yes --with-mpi-dir=/usr/local --with-fc=0 --prefix=/home/jiannantu/petsc --with-scalar-type=complex --with-64-bit-indices=1
[0]PETSC ERROR: #1 MatMult() at /home/jiannantu/petsc-3.16.6/src/mat/interface/matrix.c:2425
[0]PETSC ERROR: #2 PCApplyBAorAB() at /home/jiannantu/petsc-3.16.6/src/ksp/pc/interface/precon.c:730
[0]PETSC ERROR: #3 KSP_PCApplyBAorAB() at /home/jiannantu/petsc-3.16.6/include/petsc/private/kspimpl.h:421
[0]PETSC ERROR:  Linear equations not solved. Diverged reason: 0
#4 KSPSolve_BCGS() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/impls/bcgs/bcgs.c:87
[0]PETSC ERROR: #5 KSPSolve_Private() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itfunc.c:914
[0]PETSC ERROR: #6 KSPSolve() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itfunc.c:1086

-----------------------------------------------
    MatCreate(MPI_COMM_WORLD, &B);
    MatSetType(B, MATMPIAIJ);
    MatSetSizes(B, PETSC_DECIDE, PETSC_DECIDE, N, N);

    Mat preall;
    MatCreate(MPI_COMM_WORLD, &preall);
    MatSetType(preall, MATPREALLOCATOR);
    MatSetSizes(preall, PETSC_DECIDE, PETSC_DECIDE, N, N);

    MatMPIAIJSetPreallocation(preall, 0, d_nnz, 0, o_nnz);
    MatSetUp(preall);
MatSetOption(preall, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE);

//set values row by row
MatSetValues(preall, 1, &m, 1, nCols, matCols, INSERT_VALUES);

MatAssemblyBegin(preall, MAT_FINAL_ASSEMBLY);
MatAssemblyEnd(preall, MAT_FINAL_ASSEMBLY);

//set values for actual matrix row by row
MatSetValues(B, 1, &m, 1, nCols, matCols, INSERT_VALUES);

MatAssemblyBegin(B, MAT_FINAL_ASSEMBLY);
MatAssemblyEnd(B, MAT_FINAL_ASSEMBLY);

    KSPCreate(MPI_COMM_WORLD, &ksp);
    KSPSetOperators(ksp, A, B);
    KSPSetFromOptions(ksp);

    PC pc;
    KSPGetPC(ksp, &pc);
    PCSetFromOptions(pc);

From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Thursday, August 18, 2022 12:59 PM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] Using matrix-free with KSP

CAUTION: This email was sent from outside the UMass Lowell network.


  Yes, once you have preallocated the real matrix you can destroy the preallocation matrix whose only job is to gather the preallocation information





On Aug 18, 2022, at 12:52 PM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

Thanks, Barry.

So I need actually two matrixes, one for preallocator and one for actual matrix that can be passed to KSPSetOperators(). -mat_type preallocator option is used to speed up doing insertion into preallocator, then use MatPreallocatorPreallocate() to preallocate actual matrix and do actual insertion of values into it., right?

The code runs in parallel. Each process owns number of rows that equals to number of unknowns (that is, xm in 1D DM) it owns.

Jiannan

From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Thursday, August 18, 2022 11:37 AM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] Using matrix-free with KSP

CAUTION: This email was sent from outside the UMass Lowell network.


   The preallocator MatType matrix cannot be passed to the KSPSetOperators(), you need to create an actual matrix, for example MATAIJ and use the preallocator to set its preallocation and then fill up the MATAIJ matrix with the usual MatSetValues and pass that matrix to the KSPSetOperators.






On Aug 18, 2022, at 11:30 AM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

Hi Barry,

The MATPREALLOCATOR solved problem of slow matrix assembly. But the solver failed because of "Matrix type preallocator does not have a multiply defined".

I guess this is because the matrix-free is used. I am wondering how a preconditioner is applied to the matrix-vector multiply in Petsc.

Thank you,
Jiannan

[0]PETSC ERROR: No support for this operation for this object type
[0]PETSC ERROR: Matrix type preallocator does not have a multiply defined
[0]PETSC ERROR: See https://petsc.org/release/faq/<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Frelease%2Ffaq%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7Cec4aed5ceff5479ec1ce08da821daee5%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637965361283417283%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=ISQJqjmeBCfGq3Ufro5G1RhJiw%2BHVsnQXSSGxAu0mj8%3D&reserved=0> for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.16.6, Mar 30, 2022
[0]PETSC ERROR: ./iesolver on a  named REXNET-WS4 by jiannantu Thu Aug 18 11:21:55 2022
[0]PETSC ERROR: Configure options --download-f2cblaslapack=yes --with-mpi-dir=/usr/local --with-fc=0 --prefix=/home/jiannantu/petsc --with-scalar-type=complex --with-64-bit-indices=1
[0]PETSC ERROR: #1 MatMult() at /home/jiannantu/petsc-3.16.6/src/mat/interface/matrix.c:2437
[0]PETSC ERROR: #2 PCApply_Mat() at /home/jiannantu/petsc-3.16.6/src/ksp/pc/impls/mat/pcmat.c:9
[0]PETSC ERROR: #3 PCApply() at /home/jiannantu/petsc-3.16.6/src/ksp/pc/interface/precon.c:445
[0]PETSC ERROR: #4 KSP_PCApply() at /home/jiannantu/petsc-3.16.6/include/petsc/private/kspimpl.h:382
[0]PETSC ERROR: #5 KSPInitialResidual() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itres.c:65
[0]PETSC ERROR: #6 KSPSolve_BCGS() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/impls/bcgs/bcgs.c:43
[0]PETSC ERROR: #7 KSPSolve_Private() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itfunc.c:914
[0]PETSC ERROR: #8 KSPSolve() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itfunc.c:1086

From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Thursday, August 18, 2022 8:35 AM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] Using matrix-free with KSP

CAUTION: This email was sent from outside the UMass Lowell network.


  Slow assembly is usually due to under preallocation. You can change to using

MatSetOption(B, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_TRUE);

  to detect if you are under preallocating. See https://petsc.org/main/docs/manual/mat/#sec-matsparse<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanual%2Fmat%2F%23sec-matsparse&data=05%7C01%7CJiannan_Tu%40uml.edu%7Cec4aed5ceff5479ec1ce08da821daee5%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637965361283417283%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=NWz%2F%2FOqxxf4KV8op1B6mcvUuvShtB%2F2LQ94HnlntDXw%3D&reserved=0>. That section clearly needs some sprucing up. If it is difficult to determine good preallocation values you can use https://petsc.org/main/docs/manualpages/Mat/MATPREALLOCATOR/<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMATPREALLOCATOR%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7Cec4aed5ceff5479ec1ce08da821daee5%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637965361283417283%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=xlFH%2BZG1SB64uGGh4Gp9qWSwFYfUidhUQexxdrkopxw%3D&reserved=0> to compute the needed preallocation efficiently.

  Are you running in parallel? If so how are you determining which rows of entries to compute on each MPI rank? You will want most of the rows to be computed on the rank where the values are stored, you can determine this with https://petsc.org/main/docs/manualpages/Mat/MatGetOwnershipRange.html<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMatGetOwnershipRange.html&data=05%7C01%7CJiannan_Tu%40uml.edu%7Cec4aed5ceff5479ec1ce08da821daee5%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637965361283573499%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=CMfhxPbHyjOjfWYYlTM0J9fJrI%2FVRavZFrz4Uf7qHAo%3D&reserved=0>

  Barry

















On Aug 18, 2022, at 12:50 AM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

I implemented the preconditioner. The solver converges quickly when the problem size is small. But when the size increases, say to 100k unknowns, the code hangs at assembly of the preconditioner matrix. The function call sequence is like this

   MatCreate(MPI_COMM_WORLD, &B);
    MatSetType(B, MATMPIAIJ);
    MatSetSizes(B, PETSC_DECIDE, PETSC_DECIDE, N, N);
    MatSetFromOptions(B);

    // The number of diagonal and off diagonal non zeros only can be determined run-time.
    MatMPIAIJSetPreallocation(B, 0, d_nnz, 0, o_nnz);
    MatSetOption(B, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE);

    //here insert values row by row.
MatSetValues(B, 1, &m, nn, nCols, matCols, INSERT_VALUES);

MatAssemblyBegin(B, MAT_FINAL_ASSEMBLY);
MatAssemblyEnd(B, MAT_FINAL_ASSEMBLY);

Could you please tell me what I have done wrong?

Thank you,
Jiannan

From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Tuesday, August 16, 2022 2:00 PM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] Using matrix-free with KSP

CAUTION: This email was sent from outside the UMass Lowell network.


   Create another matrix B that contains "Selected parts of the matrix can be pre-calculated and stored to serve as the preconditioner." When you call KSPSetOperators() pass in B as the second matrix argument.

   Now if literally application of B is the preconditioner you want to use, then simply use PCMAT as the preconditioner, and it will apply B each time it needs to apply the preconditioner. With this, you do not even need a PCShell.

   If your preconditioner uses B to construct your own specific data structure needed to apply your preconditioner, then you can use PCShellSetSetUp() and have your routine call PCGetOperators() to pull out the B matrix and use it to construct your preconditioner.

   If you want PETSc to construct a standard preconditioner from B, then you don't need the PCShell; you simply pass in the B, as above, and use -pc_type type to set the preconditioner you want to use.

   Barry









On Aug 16, 2022, at 12:31 PM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

Barry,

Thank you very much for your instructions. I am sorry I may not ask clearer questions.

I don't use SNES. What I am trying to solve is a very large linear equation system with dense and ill-conditioned matrix. Selected parts of the matrix can be pre-calculated and stored to serve as the preconditioner. My question is how I can apply this matrix as the preconditioner.

Here is the part of code from my linear equations solver. MatrixFreePreconditioner is supposed to be the routine to provide the preconditioner. But I don't understand how to use it. Should I use SNESSetJacobian() even if this is a linear problem?

    MatCreateMFFD(MPI_COMM_WORLD, PETSC_DECIDE, PETSC_DECIDE, N, N, &A);
    MatMFFDSetFunction(A, formfunction, &params);
    MatSetFromOptions(A);
    MatMFFDSetBase(A, X, NULL);

    KSPCreate(MPI_COMM_WORLD, &ksp);
    KSPSetOperators(ksp, A, A);
    KSPSetFromOptions(ksp);

    PC pc;
    KSPGetPC(ksp,&pc);
    PCSetType(pc,PCSHELL);
    PCShellSetApply(pc,MatrixFreePreconditioner);

Thank you,
Jiannan

________________________________
From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Tuesday, August 16, 2022 10:10 AM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] Using matrix-free with KSP

CAUTION: This email was sent from outside the UMass Lowell network.


  I don't fully understand your question so I will answer questions that I do know the answer to :-)

  The function you provide to PCShellSetApply() applies the preconditioner to the input vector and puts the result in the output vector.  This input vector changes at every application of the preconditioner so should not be used in constructing the preconditioner.

  If you are using the standard PETSc matrix-free matrix-vector product via finite difference, -snes_mf_operator or via MatCreateSNESMF() or you are using a MatShell, the matrix-vector product routines input changes for each multiply and should not be used in constructing the preconditioner.

  The function you provide with SNESSetJacobian() is where you can compute anything you need to help build your preconditioner. The input vector to that function is the location at which the Jacobian is needed and is what you should use to build your approximate/part of Jacobian. You can store your approximate/part of Jacobian that you need to compute in the pmat and then in a PCShellSetUp() function that you provide you will compute what you will need latter to apply the preconditioner in your PCShellSetApply() function. Note that the values you put in the pmat do not need to be the full Jacobian; they can be anything you want since this pmat is not used to apply the matrix vector product.

  If this is unclear, feel free to ask additional questions.

  Barry









On Aug 16, 2022, at 9:40 AM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

I am trying to apply a user-defined preconditioner to speed up the convergency of matrix-free KSP solver. The user's manual provides an example of how to call the user-defined preconditioner routine. The routine has PC, input Vec and output Vec as arguments. I want to use part of the Jacobian as preconditioner matrix, the elements of which can be calculated using input Vec. My question is what the role of output Vec from the routine is in matrix-vector product? Or I just have the preconditioner matrix elements calculated in the routine and then Petsc handles applying the preconditioner to matrix-vector product? I used PCSetType(pc, PCSHELL) and PCShellSetApply.

Thank you and appreciate your help!

Jiannan
________________________________
From: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Sent: Wednesday, July 6, 2022 2:34 PM
To: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Cc: Jed Brown <jed at jedbrown.org<mailto:jed at jedbrown.org>>; petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] Using matrix-free with KSP

Barry,

VecScatterCreateToAll solves the problem! The code now gives the correct answer with multiple processes.

Thank you all so much!

Jiannan
________________________________
From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Wednesday, July 6, 2022 2:08 PM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Cc: Jed Brown <jed at jedbrown.org<mailto:jed at jedbrown.org>>; petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] Using matrix-free with KSP


  So your operator is a "dense" operator in that its matrix representation would be essentially dense. In that case, you can use specialized routines to do this efficiently. You can use VecScatterCreateToAll() and then VecScatterBegin/End to do the communication.

  Barry










On Jul 6, 2022, at 1:39 PM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

Jed, thank you very much for the reply.

I'm not sure GlobaltoLocal will work. Say the solution vector is of length 10. Two processes then each process can see 5 elements of the vector. For A x = b (10 equations), each process performs matrix-vector product for five equations. And for each equation i , sum_j A_ij *  x_j = b_i requires the process has access to all 10 unknows x_i (i=0,1,...,9).

Can VecScatter and PetscSF make the entire vector accessible to each process? I am reading the manual and trying to understand how VecScatter and PetscSF work.

Jiannan
________________________________
From: Jed Brown <jed at jedbrown.org<mailto:jed at jedbrown.org>>
Sent: Wednesday, July 6, 2022 10:09 AM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>; Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] Using matrix-free with KSP

You'll usually have a GlobalToLocal operation for each rank to get the halo data it needs, then either a LocalToGlobal to collect the result (e.g., finite element methods) or the local compute will write directly into the owned portion of the global vector. If you're doing the communication "raw", then you may find VecScatter or PetscSF useful to perform the necessary communication.

"Tu, Jiannan" <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> writes:

> Hi Barry,
>
> Following your instructions I implemented the matrix-free method to solve a large linear equation system resulted from a surface integration equation. The KSP solver works fine for single process, but it is problematic with multiple processes. The problem is that each process only can access its own part of solution vector so that each process only conducts part of matrix-vector multiplication. MPI can be used to assemble these partial matrix-vector multiplication together.  Does Petsc provide any ways to implement multi-process matrix-free method?
>
> Thanks,
> Jiannan
> ________________________________
> From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
> Sent: Tuesday, May 24, 2022 2:12 PM
> To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
> Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
> Subject: Re: [petsc-users] Using matrix-free with KSP
>
> This e-mail originated from outside the UMass Lowell network.
> ________________________________
>
>    You can use MatCreateMFFD https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMatCreateMFFD%2F&amp;data=05%7C01%7CJiannan_Tu%40uml.edu%7Cab0859d8d154471590bc08da5f5918d6%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637927133525745809%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=A7wqegTfh94No5BpDiWLK3VxOuR44U2wlWHVm2k7l60%3D&amp;reserved=0<https://urldefense.com/v3/__https://petsc.org/main/docs/manualpages/Mat/MatCreateMFFD/__;!!PVKG_VDCxu5g!stxq2NWGVjUihvHBQC9Rlk0aHVoowcy0PTQYMcQxQ4DqOEC05KSw6TmstSXKckiUgelHzy4ue-d10-zlXkw$><https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMatCreateMFFD%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7Cec4aed5ceff5479ec1ce08da821daee5%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637965361283573499%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=KJB5uPx7Hj1rfzvCa8PxN4CCRKgkbyC%2BLOA427KDN3I%3D&reserved=0> MatMFFDSetFunction MatSetFromOptions MatMFFDSetBase and provide the matrix to KSP. Note you will need to use -pc_type none or provide your own custom preconditioner withhttps://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FPC%2FPCSHELL%2F&amp;data=05%7C01%7CJiannan_Tu%40uml.edu%7Cab0859d8d154471590bc08da5f5918d6%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637927133525745809%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=Gb%2F9uFG3jp%2FbfWaSh2cSEQLItHS9CuLwarzN0KcNiJY%3D&amp;reserved=0<https://urldefense.com/v3/__https://petsc.org/main/docs/manualpages/PC/PCSHELL/__;!!PVKG_VDCxu5g!stxq2NWGVjUihvHBQC9Rlk0aHVoowcy0PTQYMcQxQ4DqOEC05KSw6TmstSXKckiUgelHzy4ue-d1cA4fKGw$><https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FPC%2FPCSHELL%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7Cec4aed5ceff5479ec1ce08da821daee5%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637965361283573499%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=BRENmf2B%2Fih%2F9%2BVQ10vURJIlFqo0JbzSEimQTbFO%2FaM%3D&reserved=0>
>
>
>
> On May 24, 2022, at 1:21 PM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu><mailto:Jiannan_Tu at uml.edu>> wrote:
>
> I want to use a matrix-free matrix to solve a large linear equation system because the matrix is too large to be stored. Petsc user manual describes matrix-free method for SNES with examples. The matrix-free matrices section explains how to set up such a matrix, but I can't find any example of matrix-free method with KSP. I am wondering how to apply the method to KSP iterative solver in parallel.
>
> I greatly appreciate your help for me to understand this topic.
>
> Jiannan Tu

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220822/74972976/attachment-0001.html>

From bsmith at petsc.dev  Mon Aug 22 07:42:35 2022
From: bsmith at petsc.dev (Barry Smith)
Date: Mon, 22 Aug 2022 08:42:35 -0400
Subject: [petsc-users] Using matrix-free with KSP
In-Reply-To: <BY5PR02MB6471560EA70C6FAE826DDF27E7719@BY5PR02MB6471.namprd02.prod.outlook.com>
References: <BY5PR02MB6471969F80FA5F732FC74D87E7D79@BY5PR02MB6471.namprd02.prod.outlook.com>
	<B55CFC2C-6364-4B78-8E99-9D53700743B8@petsc.dev>
	<BY5PR02MB6471BDF90783FE3A0EA176CBE7809@BY5PR02MB6471.namprd02.prod.outlook.com>
	<87iloa49kt.fsf@jedbrown.org>
	<BY5PR02MB6471C0ED04769D02046F68ACE7809@BY5PR02MB6471.namprd02.prod.outlook.com>
	<00EBE4B8-22E5-408F-B56D-14226A278D7E@petsc.dev>
	<BY5PR02MB6471E386057DDDE98AD05DFAE7809@BY5PR02MB6471.namprd02.prod.outlook.com>
	<BY5PR02MB647140084C7E7CDDB6BC2A11E76B9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<A6629DE1-5A7F-414E-B164-A69877158675@petsc.dev>
	<BY5PR02MB6471AB1C004920DA1FC9B43EE76B9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<103146FD-FF28-47CE-B661-F2B521BF0E7A@petsc.dev>
	<DM6PR02MB6478B2FB8CB1BE757CE7B003E76D9@DM6PR02MB6478.namprd02.prod.outlook.com>
	<4C00F4B8-BB42-4932-BCD7-527E5FB39BFF@petsc.dev>
	<BY5PR02MB64710E70758E431108D66956E76D9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<D0EB13D8-C5A6-4AE7-AFDE-2CDCCACBAF73@petsc.dev>
	<BY5PR02MB6471D4B05874DECFDDA13FFCE76D9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<367B124C-88BB-4889-A4AF-45A1D473078D@petsc.dev>
	<BY5PR02MB6471F2C058B84DE092567731E76D9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<944DDEED-016D-43B2-A5A6-7BF2E8E41FA5@petsc.dev>
	<BY5PR02MB6471CD0236F03D20763A3B64E76C9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<B4CF5E5B-984D-4EC1-AB77-1FD08C3FBBCF@petsc.dev>
	<BY5PR02MB6471BA72871419781B875967E76C9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<3D179AEA-AD05-4B46-8DDC-9E126741C247@petsc.dev>
	<BY5PR02MB6471560EA70C6FAE826DDF27E7719@BY5PR02MB6471.namprd02.prod.outlook.com>
Message-ID: <1C67C8DB-0653-4588-8137-E0865E0F2098@petsc.dev>



> On Aug 21, 2022, at 11:18 PM, Tu, Jiannan <Jiannan_Tu at uml.edu> wrote:
> 
> Barry,
>  
> You are right. With PC type none, the same error appears.
>  
> Now I use MatShell and I literally have three matrixes. A is a shell matrix, B is a preconditioner, and C is a matrix just for preallocation purpose.
>  
> First C is used to determine non-zero structure. Then use MatPreallocatorPreallocation(C, PETSC_FALSE, B) to preallocate for B. After that do insertion and assembly for B. These steps work fine.
>  
> Now the question is how to insert and assemble shell matrix A. MatPreallocatorPreallocation doesn?t work with shell matrix as indicated by an error message. MatSetValues or Mat assembly doesn?t support shell matrix either. 

  A shell matrix is completely under your control. You decide on what data you want to store in it, and you provide the matrix-vector product routine.  MatPreallocatorPreallocation and MatSetValues don't work for shell matrix unless you provide code for them to work, which normally you would not do.

  I don't understand why you want to use MatSetValues for your shell matrix. If you can use MatSetValues() and just build an explicit matrix, then why would you use a shell matrix, just use MATAIJ matrix? 

  Or does part of the operation inside your MatShell require a "regular" PETSc matrix? This would be fine, but it is your responsibility to design your Mat shell context to contain the "regular matrix" and your responsibility to fill up that regular matrix inside your shell context, calling whatever code you choose to write. So for example your shell context would be a C struct and one of the fields would be a regular PETSc matrix, also part of your custom matrix-vector product routine would call a regular MatMult() on the regular PETSc matrix stored inside the C struct.

  Barry


>  
> Thank you,
> Jiannan
>  
> From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>> 
> Sent: Friday, August 19, 2022 4:02 PM
> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
> Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>
> Subject: Re: [petsc-users] Using matrix-free with KSP
>  
> CAUTION: This email was sent from outside the UMass Lowell network.
>  
>  
>   I don't know why changing the PCType could matter, can you please try with your new code but use the -pc_type none option to turn off the use of the preconditioner to see if the error still appears (I think it should). 
>  
>   Also if you have a custom function that can do the matrix-vector product (not using any differencing) then you should use MatShell() and provide the matrix-vector product operation with your code.
>  
>   Barry
>  
> 
> 
> On Aug 19, 2022, at 11:47 AM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> wrote:
>  
> Barry,
>  
> I have my own matrix-vector product but not use MatShell. I followed your previous instruction to use
>  
>     MatCreateMFFD(MPI_COMM_WORLD, PETSC_DECIDE, PETSC_DECIDE, N, N, &A);
>     MatMFFDSetFunction(A, formfunction, &params);
>     MatSetFromOptions(A);
>     MatMFFDSetBase(A, X, NULL);
>  
> The customized matrix-vector product is implemented in formfunction() provided to MatMFFDSetFunction(). In this way, the matrix A is not used. I think that is why Petsc complains the matrix is not assembled. However, when PC type is none, the code runs without error although KSP solver doesn?t converge.
>  
> The question then becomes how to link A to matrix-free matrix-vector multiplication. And where to put MatShell?
>  
> Thank you,
> Jiannan
>  
> From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>> 
> Sent: Friday, August 19, 2022 10:27 AM
> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
> Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>
> Subject: Re: [petsc-users] Using matrix-free with KSP
>  
> CAUTION: This email was sent from outside the UMass Lowell network.
>  
>  
> 
> 
> 
> On Aug 18, 2022, at 11:41 PM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> wrote:
>  
> The program calls KSPSolve just once and the error occurs after the first iteration.
>  
> I realized the matrix-vector product in my case is different from that approximating matrix-vector multiply by finite difference of function vector. It consists of two parts. One is multiplication of part of the matrix with part of vector, and another part results from so called fast multipole method. I?m not clear how to relate such calculation of matrix-vector product to the finite differencing of function vector.
>  
>   Are you using finite differencing at all in a matrix-vector product, or do you have your own custom matrix-vector product code (with MatShell)? 
>  
>   Anyways the easiest thing to do is 
>  
>   use the run time option -on_error_attach_debugger noxterm and run the code, 
>  
>   when it stops in the debugger you can type bt (for backtrace)
>  
>   then do up a few times to move up the stack until you get to the PETSc function call that generates the error
>  
>   you can then do "call MatView(thematrixvarible,0) to have PETSc display what the matrix is and
>  
>   you can also do print *thematrixvariable to see the exact struct with all its fields
>  
>   If this doesn't tell you why the troublesome matrix is not assembled then cut and paste ALL of the output and email it
>  
>   Barry
> 
> 
> 
>  
>  
>  
> From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>> 
> Sent: Thursday, August 18, 2022 7:44 PM
> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
> Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>
> Subject: Re: [petsc-users] Using matrix-free with KSP
>  
> CAUTION: This email was sent from outside the UMass Lowell network.
>  
>  
>  
>    Is it in the first iteration of the first KSPSolve that you get this message? Or in the second call to KSPSolve? 
>  
>   The error comes from a matrix vector product with your A matrix, what is that and where did it come from. The error message is not from your B matrix, that is not used in the MatMult()
>  
> 
> 
> 
> 
> On Aug 18, 2022, at 3:05 PM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> wrote:
>  
> Barry, 
>  
> Thanks. The method works. Assembly of actual matrix now completes quickly. The convergency is much better for the first iteration. The residual normal is in the order of 10^-10 instead of 10^-2. However, there is an error message after the first iteration: ?Object is in wrong state. Not for unassembled matrix? even though both preallocator and actual matrix have been assembled (see the code snippet below). Could you please give me some hints what?s going wrong? I use KSP type bcgs and PC type mat
>  
> Thank you,
> Jiannan
>  
> -----------------------------------------------------
> Solve linear equations ...
>   0 KSP Residual norm 9.841179147519e-10
> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [0]PETSC ERROR: Object is in wrong state
> [0]PETSC ERROR: Not for unassembled matrix
> [0]PETSC ERROR: See https://petsc.org/release/faq/ <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Frelease%2Ffaq%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7Cec4aed5ceff5479ec1ce08da821daee5%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637965361283417283%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=ISQJqjmeBCfGq3Ufro5G1RhJiw%2BHVsnQXSSGxAu0mj8%3D&reserved=0> for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.16.6, Mar 30, 2022
> [0]PETSC ERROR: ./iesolver on a  named REXNET-WS4 by jiannantu Thu Aug 18 14:48:43 2022
> [0]PETSC ERROR: Configure options --download-f2cblaslapack=yes --with-mpi-dir=/usr/local --with-fc=0 --prefix=/home/jiannantu/petsc --with-scalar-type=complex --with-64-bit-indices=1
> [0]PETSC ERROR: #1 MatMult() at /home/jiannantu/petsc-3.16.6/src/mat/interface/matrix.c:2425
> [0]PETSC ERROR: #2 PCApplyBAorAB() at /home/jiannantu/petsc-3.16.6/src/ksp/pc/interface/precon.c:730
> [0]PETSC ERROR: #3 KSP_PCApplyBAorAB() at /home/jiannantu/petsc-3.16.6/include/petsc/private/kspimpl.h:421
> [0]PETSC ERROR:  Linear equations not solved. Diverged reason: 0
> #4 KSPSolve_BCGS() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/impls/bcgs/bcgs.c:87
> [0]PETSC ERROR: #5 KSPSolve_Private() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itfunc.c:914
> [0]PETSC ERROR: #6 KSPSolve() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itfunc.c:1086
>  
> -----------------------------------------------
>     MatCreate(MPI_COMM_WORLD, &B);
>     MatSetType(B, MATMPIAIJ);
>     MatSetSizes(B, PETSC_DECIDE, PETSC_DECIDE, N, N);
>  
>     Mat preall;
>     MatCreate(MPI_COMM_WORLD, &preall);
>     MatSetType(preall, MATPREALLOCATOR);
>     MatSetSizes(preall, PETSC_DECIDE, PETSC_DECIDE, N, N);
>  
>     MatMPIAIJSetPreallocation(preall, 0, d_nnz, 0, o_nnz);
>     MatSetUp(preall);
> MatSetOption(preall, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE);
>  
> //set values row by row
> MatSetValues(preall, 1, &m, 1, nCols, matCols, INSERT_VALUES);
>  
> MatAssemblyBegin(preall, MAT_FINAL_ASSEMBLY);
> MatAssemblyEnd(preall, MAT_FINAL_ASSEMBLY);
>  
> //set values for actual matrix row by row
> MatSetValues(B, 1, &m, 1, nCols, matCols, INSERT_VALUES);
>  
> MatAssemblyBegin(B, MAT_FINAL_ASSEMBLY);
> MatAssemblyEnd(B, MAT_FINAL_ASSEMBLY);
>  
>     KSPCreate(MPI_COMM_WORLD, &ksp);
>     KSPSetOperators(ksp, A, B);
>     KSPSetFromOptions(ksp);
>  
>     PC pc;
>     KSPGetPC(ksp, &pc);
>     PCSetFromOptions(pc);
>  
> From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>> 
> Sent: Thursday, August 18, 2022 12:59 PM
> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
> Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>
> Subject: Re: [petsc-users] Using matrix-free with KSP
>  
> CAUTION: This email was sent from outside the UMass Lowell network.
>  
>  
>   Yes, once you have preallocated the real matrix you can destroy the preallocation matrix whose only job is to gather the preallocation information
> 
> 
> 
> 
> 
> On Aug 18, 2022, at 12:52 PM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> wrote:
>  
> Thanks, Barry.
>  
> So I need actually two matrixes, one for preallocator and one for actual matrix that can be passed to KSPSetOperators(). -mat_type preallocator option is used to speed up doing insertion into preallocator, then use MatPreallocatorPreallocate() to preallocate actual matrix and do actual insertion of values into it., right?
>  
> The code runs in parallel. Each process owns number of rows that equals to number of unknowns (that is, xm in 1D DM) it owns.
>  
> Jiannan
>  
> From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>> 
> Sent: Thursday, August 18, 2022 11:37 AM
> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
> Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>
> Subject: Re: [petsc-users] Using matrix-free with KSP
>  
> CAUTION: This email was sent from outside the UMass Lowell network.
>  
>  
>    The preallocator MatType matrix cannot be passed to the KSPSetOperators(), you need to create an actual matrix, for example MATAIJ and use the preallocator to set its preallocation and then fill up the MATAIJ matrix with the usual MatSetValues and pass that matrix to the KSPSetOperators.
> 
> 
> 
> 
> 
> 
> On Aug 18, 2022, at 11:30 AM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> wrote:
>  
> Hi Barry,
>  
> The MATPREALLOCATOR solved problem of slow matrix assembly. But the solver failed because of ?Matrix type preallocator does not have a multiply defined?.
>  
> I guess this is because the matrix-free is used. I am wondering how a preconditioner is applied to the matrix-vector multiply in Petsc.
>  
> Thank you,
> Jiannan
>  
> [0]PETSC ERROR: No support for this operation for this object type
> [0]PETSC ERROR: Matrix type preallocator does not have a multiply defined
> [0]PETSC ERROR: See https://petsc.org/release/faq/ <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Frelease%2Ffaq%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7Cec4aed5ceff5479ec1ce08da821daee5%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637965361283417283%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=ISQJqjmeBCfGq3Ufro5G1RhJiw%2BHVsnQXSSGxAu0mj8%3D&reserved=0> for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.16.6, Mar 30, 2022
> [0]PETSC ERROR: ./iesolver on a  named REXNET-WS4 by jiannantu Thu Aug 18 11:21:55 2022
> [0]PETSC ERROR: Configure options --download-f2cblaslapack=yes --with-mpi-dir=/usr/local --with-fc=0 --prefix=/home/jiannantu/petsc --with-scalar-type=complex --with-64-bit-indices=1
> [0]PETSC ERROR: #1 MatMult() at /home/jiannantu/petsc-3.16.6/src/mat/interface/matrix.c:2437
> [0]PETSC ERROR: #2 PCApply_Mat() at /home/jiannantu/petsc-3.16.6/src/ksp/pc/impls/mat/pcmat.c:9
> [0]PETSC ERROR: #3 PCApply() at /home/jiannantu/petsc-3.16.6/src/ksp/pc/interface/precon.c:445
> [0]PETSC ERROR: #4 KSP_PCApply() at /home/jiannantu/petsc-3.16.6/include/petsc/private/kspimpl.h:382
> [0]PETSC ERROR: #5 KSPInitialResidual() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itres.c:65
> [0]PETSC ERROR: #6 KSPSolve_BCGS() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/impls/bcgs/bcgs.c:43
> [0]PETSC ERROR: #7 KSPSolve_Private() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itfunc.c:914
> [0]PETSC ERROR: #8 KSPSolve() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itfunc.c:1086
>  
> From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>> 
> Sent: Thursday, August 18, 2022 8:35 AM
> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
> Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>
> Subject: Re: [petsc-users] Using matrix-free with KSP
>  
> CAUTION: This email was sent from outside the UMass Lowell network.
>  
>  
>   Slow assembly is usually due to under preallocation. You can change to using 
>  
> MatSetOption(B, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_TRUE);
>  
>   to detect if you are under preallocating. See https://petsc.org/main/docs/manual/mat/#sec-matsparse <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanual%2Fmat%2F%23sec-matsparse&data=05%7C01%7CJiannan_Tu%40uml.edu%7Cec4aed5ceff5479ec1ce08da821daee5%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637965361283417283%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=NWz%2F%2FOqxxf4KV8op1B6mcvUuvShtB%2F2LQ94HnlntDXw%3D&reserved=0>. That section clearly needs some sprucing up. If it is difficult to determine good preallocation values you can use https://petsc.org/main/docs/manualpages/Mat/MATPREALLOCATOR/ <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMATPREALLOCATOR%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7Cec4aed5ceff5479ec1ce08da821daee5%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637965361283417283%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=xlFH%2BZG1SB64uGGh4Gp9qWSwFYfUidhUQexxdrkopxw%3D&reserved=0> to compute the needed preallocation efficiently.
>  
>   Are you running in parallel? If so how are you determining which rows of entries to compute on each MPI rank? You will want most of the rows to be computed on the rank where the values are stored, you can determine this with https://petsc.org/main/docs/manualpages/Mat/MatGetOwnershipRange.html <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMatGetOwnershipRange.html&data=05%7C01%7CJiannan_Tu%40uml.edu%7Cec4aed5ceff5479ec1ce08da821daee5%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637965361283573499%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=CMfhxPbHyjOjfWYYlTM0J9fJrI%2FVRavZFrz4Uf7qHAo%3D&reserved=0>
>  
>   Barry
>  
>  
> 
> 
> 
> 
> 
> 
> 
>  
> 
> 
> 
> 
> 
> 
> 
> On Aug 18, 2022, at 12:50 AM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> wrote:
>  
> I implemented the preconditioner. The solver converges quickly when the problem size is small. But when the size increases, say to 100k unknowns, the code hangs at assembly of the preconditioner matrix. The function call sequence is like this
>  
>    MatCreate(MPI_COMM_WORLD, &B);
>     MatSetType(B, MATMPIAIJ);
>     MatSetSizes(B, PETSC_DECIDE, PETSC_DECIDE, N, N);
>     MatSetFromOptions(B);
>  
>     // The number of diagonal and off diagonal non zeros only can be determined run-time.
>     MatMPIAIJSetPreallocation(B, 0, d_nnz, 0, o_nnz);
>     MatSetOption(B, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE);
>  
>     //here insert values row by row.
> MatSetValues(B, 1, &m, nn, nCols, matCols, INSERT_VALUES);
>  
> MatAssemblyBegin(B, MAT_FINAL_ASSEMBLY);
> MatAssemblyEnd(B, MAT_FINAL_ASSEMBLY);
>  
> Could you please tell me what I have done wrong?
>  
> Thank you,
> Jiannan
>    
> From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>> 
> Sent: Tuesday, August 16, 2022 2:00 PM
> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
> Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>
> Subject: Re: [petsc-users] Using matrix-free with KSP
>  
> CAUTION: This email was sent from outside the UMass Lowell network.
>  
>  
>    Create another matrix B that contains "Selected parts of the matrix can be pre-calculated and stored to serve as the preconditioner." When you call KSPSetOperators() pass in B as the second matrix argument.
>  
>    Now if literally application of B is the preconditioner you want to use, then simply use PCMAT as the preconditioner, and it will apply B each time it needs to apply the preconditioner. With this, you do not even need a PCShell. 
>  
>    If your preconditioner uses B to construct your own specific data structure needed to apply your preconditioner, then you can use PCShellSetSetUp() and have your routine call PCGetOperators() to pull out the B matrix and use it to construct your preconditioner. 
>  
>    If you want PETSc to construct a standard preconditioner from B, then you don't need the PCShell; you simply pass in the B, as above, and use -pc_type type to set the preconditioner you want to use. 
>  
>    Barry
>  
> 
> 
> 
> 
> 
> 
> 
> 
> On Aug 16, 2022, at 12:31 PM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> wrote:
>  
> Barry,
>  
> Thank you very much for your instructions. I am sorry I may not ask clearer questions.
>  
> I don't use SNES. What I am trying to solve is a very large linear equation system with dense and ill-conditioned matrix. Selected parts of the matrix can be pre-calculated and stored to serve as the preconditioner. My question is how I can apply this matrix as the preconditioner. 
>  
> Here is the part of code from my linear equations solver. MatrixFreePreconditioner is supposed to be the routine to provide the preconditioner. But I don't understand how to use it. Should I use SNESSetJacobian() even if this is a linear problem? 
>  
>     MatCreateMFFD(MPI_COMM_WORLD, PETSC_DECIDE, PETSC_DECIDE, N, N, &A);
>     MatMFFDSetFunction(A, formfunction, &params);
>     MatSetFromOptions(A);
>     MatMFFDSetBase(A, X, NULL);
>  
>     KSPCreate(MPI_COMM_WORLD, &ksp);
>     KSPSetOperators(ksp, A, A);
>     KSPSetFromOptions(ksp);
>  
>     PC pc;
>     KSPGetPC(ksp,&pc);
>     PCSetType(pc,PCSHELL);
>     PCShellSetApply(pc,MatrixFreePreconditioner);
>  
> Thank you,
> Jiannan
>  
> From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>>
> Sent: Tuesday, August 16, 2022 10:10 AM
> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
> Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>>
> Subject: Re: [petsc-users] Using matrix-free with KSP
>  
> CAUTION: This email was sent from outside the UMass Lowell network.
>  
>  
>   I don't fully understand your question so I will answer questions that I do know the answer to :-)
>  
>   The function you provide to PCShellSetApply() applies the preconditioner to the input vector and puts the result in the output vector.  This input vector changes at every application of the preconditioner so should not be used in constructing the preconditioner.
>  
>   If you are using the standard PETSc matrix-free matrix-vector product via finite difference, -snes_mf_operator or via MatCreateSNESMF() or you are using a MatShell, the matrix-vector product routines input changes for each multiply and should not be used in constructing the preconditioner.
>  
>   The function you provide with SNESSetJacobian() is where you can compute anything you need to help build your preconditioner. The input vector to that function is the location at which the Jacobian is needed and is what you should use to build your approximate/part of Jacobian. You can store your approximate/part of Jacobian that you need to compute in the pmat and then in a PCShellSetUp() function that you provide you will compute what you will need latter to apply the preconditioner in your PCShellSetApply() function. Note that the values you put in the pmat do not need to be the full Jacobian; they can be anything you want since this pmat is not used to apply the matrix vector product.
>  
>   If this is unclear, feel free to ask additional questions.
>  
>   Barry
>  
> 
> 
> 
> 
> 
> 
> 
> 
> On Aug 16, 2022, at 9:40 AM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> wrote:
>  
> I am trying to apply a user-defined preconditioner to speed up the convergency of matrix-free KSP solver. The user's manual provides an example of how to call the user-defined preconditioner routine. The routine has PC, input Vec and output Vec as arguments. I want to use part of the Jacobian as preconditioner matrix, the elements of which can be calculated using input Vec. My question is what the role of output Vec from the routine is in matrix-vector product? Or I just have the preconditioner matrix elements calculated in the routine and then Petsc handles applying the preconditioner to matrix-vector product? I used PCSetType(pc, PCSHELL) and PCShellSetApply.
>  
> Thank you and appreciate your help!
>  
> Jiannan 
> From: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
> Sent: Wednesday, July 6, 2022 2:34 PM
> To: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>>
> Cc: Jed Brown <jed at jedbrown.org <mailto:jed at jedbrown.org>>; petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>>
> Subject: Re: [petsc-users] Using matrix-free with KSP
>  
> Barry,
>  
> VecScatterCreateToAll solves the problem! The code now gives the correct answer with multiple processes. 
>  
> Thank you all so much!
>  
> Jiannan
> From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>>
> Sent: Wednesday, July 6, 2022 2:08 PM
> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
> Cc: Jed Brown <jed at jedbrown.org <mailto:jed at jedbrown.org>>; petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>>
> Subject: Re: [petsc-users] Using matrix-free with KSP
>  
>  
>   So your operator is a "dense" operator in that its matrix representation would be essentially dense. In that case, you can use specialized routines to do this efficiently. You can use VecScatterCreateToAll() and then VecScatterBegin/End to do the communication.
>  
>   Barry
>  
>  
> 
> 
> 
> 
> 
> 
> 
> 
> On Jul 6, 2022, at 1:39 PM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> wrote:
>  
> Jed, thank you very much for the reply. 
>  
> I'm not sure GlobaltoLocal will work. Say the solution vector is of length 10. Two processes then each process can see 5 elements of the vector. For A x = b (10 equations), each process performs matrix-vector product for five equations. And for each equation i , sum_j A_ij *  x_j = b_i requires the process has access to all 10 unknows x_i (i=0,1,...,9). 
>  
> Can VecScatter and PetscSF make the entire vector accessible to each process? I am reading the manual and trying to understand how VecScatter and PetscSF work.
>  
> Jiannan
> From: Jed Brown <jed at jedbrown.org <mailto:jed at jedbrown.org>>
> Sent: Wednesday, July 6, 2022 10:09 AM
> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>; Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>>
> Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>>
> Subject: Re: [petsc-users] Using matrix-free with KSP
>  
> You'll usually have a GlobalToLocal operation for each rank to get the halo data it needs, then either a LocalToGlobal to collect the result (e.g., finite element methods) or the local compute will write directly into the owned portion of the global vector. If you're doing the communication "raw", then you may find VecScatter or PetscSF useful to perform the necessary communication.
> 
> "Tu, Jiannan" <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> writes:
> 
> > Hi Barry,
> >
> > Following your instructions I implemented the matrix-free method to solve a large linear equation system resulted from a surface integration equation. The KSP solver works fine for single process, but it is problematic with multiple processes. The problem is that each process only can access its own part of solution vector so that each process only conducts part of matrix-vector multiplication. MPI can be used to assemble these partial matrix-vector multiplication together.  Does Petsc provide any ways to implement multi-process matrix-free method?
> >
> > Thanks,
> > Jiannan
> > ________________________________
> > From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>>
> > Sent: Tuesday, May 24, 2022 2:12 PM
> > To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
> > Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>>
> > Subject: Re: [petsc-users] Using matrix-free with KSP
> >
> > This e-mail originated from outside the UMass Lowell network.
> > ________________________________
> >
> >    You can use MatCreateMFFD https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMatCreateMFFD%2F&amp;data=05%7C01%7CJiannan_Tu%40uml.edu%7Cab0859d8d154471590bc08da5f5918d6%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637927133525745809%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=A7wqegTfh94No5BpDiWLK3VxOuR44U2wlWHVm2k7l60%3D&amp;reserved=0<https://urldefense.com/v3/__https://petsc.org/main/docs/manualpages/Mat/MatCreateMFFD/__;!!PVKG_VDCxu5g!stxq2NWGVjUihvHBQC9Rlk0aHVoowcy0PTQYMcQxQ4DqOEC05KSw6TmstSXKckiUgelHzy4ue-d10-zlXkw$> <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMatCreateMFFD%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7Cec4aed5ceff5479ec1ce08da821daee5%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637965361283573499%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=KJB5uPx7Hj1rfzvCa8PxN4CCRKgkbyC%2BLOA427KDN3I%3D&reserved=0> MatMFFDSetFunction MatSetFromOptions MatMFFDSetBase and provide the matrix to KSP. Note you will need to use -pc_type none or provide your own custom preconditioner withhttps://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FPC%2FPCSHELL%2F&amp;data=05%7C01%7CJiannan_Tu%40uml.edu%7Cab0859d8d154471590bc08da5f5918d6%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637927133525745809%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=Gb%2F9uFG3jp%2FbfWaSh2cSEQLItHS9CuLwarzN0KcNiJY%3D&amp;reserved=0<https://urldefense.com/v3/__https://petsc.org/main/docs/manualpages/PC/PCSHELL/__;!!PVKG_VDCxu5g!stxq2NWGVjUihvHBQC9Rlk0aHVoowcy0PTQYMcQxQ4DqOEC05KSw6TmstSXKckiUgelHzy4ue-d1cA4fKGw$> <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FPC%2FPCSHELL%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7Cec4aed5ceff5479ec1ce08da821daee5%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637965361283573499%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=BRENmf2B%2Fih%2F9%2BVQ10vURJIlFqo0JbzSEimQTbFO%2FaM%3D&reserved=0>
> >
> >
> >
> > On May 24, 2022, at 1:21 PM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu><mailto:Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>> wrote:
> >
> > I want to use a matrix-free matrix to solve a large linear equation system because the matrix is too large to be stored. Petsc user manual describes matrix-free method for SNES with examples. The matrix-free matrices section explains how to set up such a matrix, but I can't find any example of matrix-free method with KSP. I am wondering how to apply the method to KSP iterative solver in parallel.
> >
> > I greatly appreciate your help for me to understand this topic.
> >
> > Jiannan Tu

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220822/363632e2/attachment-0001.html>

From patrick.alken at geomag.info  Mon Aug 22 10:56:34 2022
From: patrick.alken at geomag.info (Patrick Alken)
Date: Mon, 22 Aug 2022 09:56:34 -0600
Subject: [petsc-users] Preallocating a matrix with the d_nnz and o_nnz arrays
Message-ID: <47e773f6-f008-ec23-7554-ddefc30ddccf@geomag.info>

I am looping through all the non-zero elements of my matrix to count the 
diagonal and off-diagonal terms in order to pre-allocate the matrix. 
However, what is the criteria for knowing if a given element is in the 
diagonal portion or off-diagonal portion? It seems that it will depend 
on row, column, and also processor. I was doing something like this:

MatGetLocalSize(A, &m, &n);

MatGetOwnershipRange(A, &first, &last);

for (i = first; i < last; ++i) {

 ??? for (j = 0; j < N; ++j) {

 ??? ??? if (abs(i - j) < m)

 ??? ??? ??? ++d_nnz[i - first];

 ??? ??? else

 ??? ??? ??? ++o_nnz[i - first];

 ??? }

}

However, the criteria: abs(i - j) < m is not correct. I think the 
correct criteria will include the processor number also. Does anyone 
know the correct criteria to use?


From hzhang at mcs.anl.gov  Mon Aug 22 11:03:31 2022
From: hzhang at mcs.anl.gov (Zhang, Hong)
Date: Mon, 22 Aug 2022 16:03:31 +0000
Subject: [petsc-users] Preallocating a matrix with the d_nnz and o_nnz
 arrays
In-Reply-To: <47e773f6-f008-ec23-7554-ddefc30ddccf@geomag.info>
References: <47e773f6-f008-ec23-7554-ddefc30ddccf@geomag.info>
Message-ID: <SA1PR09MB8607667B9B068ADE59ACC19588719@SA1PR09MB8607.namprd09.prod.outlook.com>

See https://petsc.org/release/docs/manualpages/Mat/MatMPIAIJSetPreallocation.html#MatMPIAIJSetPreallocation
MatMPIAIJSetPreallocation - PETSc<https://petsc.org/release/docs/manualpages/Mat/MatMPIAIJSetPreallocation.html#MatMPIAIJSetPreallocation>
MatMPIAIJSetPreallocation Preallocates memory for a sparse parallel matrix in AIJ format (the default parallel PETSc format). For good matrix assembly performance the user should preallocate the matrix storage by setting the parameters d_nz (or d_nnz) and o_nz (or o_nnz).
petsc.org
if you use mpiaij matrix.
Hong
________________________________
From: petsc-users <petsc-users-bounces at mcs.anl.gov> on behalf of Patrick Alken <patrick.alken at geomag.info>
Sent: Monday, August 22, 2022 10:56 AM
To: petsc-users <petsc-users at mcs.anl.gov>
Subject: [petsc-users] Preallocating a matrix with the d_nnz and o_nnz arrays

I am looping through all the non-zero elements of my matrix to count the
diagonal and off-diagonal terms in order to pre-allocate the matrix.
However, what is the criteria for knowing if a given element is in the
diagonal portion or off-diagonal portion? It seems that it will depend
on row, column, and also processor. I was doing something like this:

MatGetLocalSize(A, &m, &n);

MatGetOwnershipRange(A, &first, &last);

for (i = first; i < last; ++i) {

     for (j = 0; j < N; ++j) {

         if (abs(i - j) < m)

             ++d_nnz[i - first];

         else

             ++o_nnz[i - first];

     }

}

However, the criteria: abs(i - j) < m is not correct. I think the
correct criteria will include the processor number also. Does anyone
know the correct criteria to use?

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220822/b356a284/attachment.html>

From patrick.alken at geomag.info  Mon Aug 22 11:12:34 2022
From: patrick.alken at geomag.info (Patrick Alken)
Date: Mon, 22 Aug 2022 10:12:34 -0600
Subject: [petsc-users] Preallocating a matrix with the d_nnz and o_nnz
 arrays
In-Reply-To: <SA1PR09MB8607667B9B068ADE59ACC19588719@SA1PR09MB8607.namprd09.prod.outlook.com>
References: <47e773f6-f008-ec23-7554-ddefc30ddccf@geomag.info>
	<SA1PR09MB8607667B9B068ADE59ACC19588719@SA1PR09MB8607.namprd09.prod.outlook.com>
Message-ID: <a6fbe8b2-0680-faf4-8b23-f248f7a7bad3@geomag.info>

Thank you, I have read that document. I have changed my criteria to:


if (j >= first && (j - first) < m) {

 ? /*diagonal*/

else

 ? /*off-diagonal*/


It seems to be working better. I don't know how to get the c1 and c2 
columns referred to by the documentation.


Patrick



On 8/22/22 10:03, Zhang, Hong wrote:
> See 
> https://petsc.org/release/docs/manualpages/Mat/MatMPIAIJSetPreallocation.html#MatMPIAIJSetPreallocation
> MatMPIAIJSetPreallocation - PETSc 
> <https://petsc.org/release/docs/manualpages/Mat/MatMPIAIJSetPreallocation.html#MatMPIAIJSetPreallocation>
> MatMPIAIJSetPreallocation Preallocates memory for a sparse parallel 
> matrix in AIJ format (the default parallel PETSc format). For good 
> matrix assembly performance the user should preallocate the matrix 
> storage by setting the parameters d_nz (or d_nnz) and o_nz (or o_nnz).
> petsc.org
>
> if you use mpiaij matrix.
> Hong
> ------------------------------------------------------------------------
> *From:* petsc-users <petsc-users-bounces at mcs.anl.gov> on behalf of 
> Patrick Alken <patrick.alken at geomag.info>
> *Sent:* Monday, August 22, 2022 10:56 AM
> *To:* petsc-users <petsc-users at mcs.anl.gov>
> *Subject:* [petsc-users] Preallocating a matrix with the d_nnz and 
> o_nnz arrays
> I am looping through all the non-zero elements of my matrix to count the
> diagonal and off-diagonal terms in order to pre-allocate the matrix.
> However, what is the criteria for knowing if a given element is in the
> diagonal portion or off-diagonal portion? It seems that it will depend
> on row, column, and also processor. I was doing something like this:
>
> MatGetLocalSize(A, &m, &n);
>
> MatGetOwnershipRange(A, &first, &last);
>
> for (i = first; i < last; ++i) {
>
> ???? for (j = 0; j < N; ++j) {
>
> ???? ??? if (abs(i - j) < m)
>
> ???? ??? ??? ++d_nnz[i - first];
>
> ???? ??? else
>
> ???? ??? ??? ++o_nnz[i - first];
>
> ???? }
>
> }
>
> However, the criteria: abs(i - j) < m is not correct. I think the
> correct criteria will include the processor number also. Does anyone
> know the correct criteria to use?
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220822/a8c13014/attachment-0001.html>

From Jiannan_Tu at uml.edu  Mon Aug 22 11:25:04 2022
From: Jiannan_Tu at uml.edu (Tu, Jiannan)
Date: Mon, 22 Aug 2022 16:25:04 +0000
Subject: [petsc-users] Using matrix-free with KSP
In-Reply-To: <1C67C8DB-0653-4588-8137-E0865E0F2098@petsc.dev>
References: <BY5PR02MB6471969F80FA5F732FC74D87E7D79@BY5PR02MB6471.namprd02.prod.outlook.com>
	<B55CFC2C-6364-4B78-8E99-9D53700743B8@petsc.dev>
	<BY5PR02MB6471BDF90783FE3A0EA176CBE7809@BY5PR02MB6471.namprd02.prod.outlook.com>
	<87iloa49kt.fsf@jedbrown.org>
	<BY5PR02MB6471C0ED04769D02046F68ACE7809@BY5PR02MB6471.namprd02.prod.outlook.com>
	<00EBE4B8-22E5-408F-B56D-14226A278D7E@petsc.dev>
	<BY5PR02MB6471E386057DDDE98AD05DFAE7809@BY5PR02MB6471.namprd02.prod.outlook.com>
	<BY5PR02MB647140084C7E7CDDB6BC2A11E76B9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<A6629DE1-5A7F-414E-B164-A69877158675@petsc.dev>
	<BY5PR02MB6471AB1C004920DA1FC9B43EE76B9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<103146FD-FF28-47CE-B661-F2B521BF0E7A@petsc.dev>
	<DM6PR02MB6478B2FB8CB1BE757CE7B003E76D9@DM6PR02MB6478.namprd02.prod.outlook.com>
	<4C00F4B8-BB42-4932-BCD7-527E5FB39BFF@petsc.dev>
	<BY5PR02MB64710E70758E431108D66956E76D9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<D0EB13D8-C5A6-4AE7-AFDE-2CDCCACBAF73@petsc.dev>
	<BY5PR02MB6471D4B05874DECFDDA13FFCE76D9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<367B124C-88BB-4889-A4AF-45A1D473078D@petsc.dev>
	<BY5PR02MB6471F2C058B84DE092567731E76D9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<944DDEED-016D-43B2-A5A6-7BF2E8E41FA5@petsc.dev>
	<BY5PR02MB6471CD0236F03D20763A3B64E76C9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<B4CF5E5B-984D-4EC1-AB77-1FD08C3FBBCF@petsc.dev>
	<BY5PR02MB6471BA72871419781B875967E76C9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<3D179AEA-AD05-4B46-8DDC-9E126741C247@petsc.dev>
	<BY5PR02MB6471560EA70C6FAE826DDF27E7719@BY5PR02MB6471.namprd02.prod.outlook.com>
	<1C67C8DB-0653-4588-8137-E0865E0F2098@petsc.dev>
Message-ID: <BY5PR02MB6471C5EAE18AB9B114E76232E7719@BY5PR02MB6471.namprd02.prod.outlook.com>

The matrix-vector product involves part of matrix elements explicitly as shown below so that I need to insert values into this part of the matrix.
[cid:07fd6bc2-973b-4485-8416-e22ba5302a62]
Initially I wrote my own code for calculating and storing matrix A^near_mi without using any Petsc matrix and then performed matrix-vector product using above equation. The code runs but converges very slow. Apparently a preconditioned is needed.

I am not clear how Petsc can provide a preconditioner for my case.
________________________________
From: Barry Smith <bsmith at petsc.dev>
Sent: Monday, August 22, 2022 8:42 AM
To: Tu, Jiannan <Jiannan_Tu at uml.edu>
Cc: petsc-users at mcs.anl.gov <petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] Using matrix-free with KSP


CAUTION: This email was sent from outside the UMass Lowell network.



On Aug 21, 2022, at 11:18 PM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

Barry,

You are right. With PC type none, the same error appears.

Now I use MatShell and I literally have three matrixes. A is a shell matrix, B is a preconditioner, and C is a matrix just for preallocation purpose.

First C is used to determine non-zero structure. Then use MatPreallocatorPreallocation(C, PETSC_FALSE, B) to preallocate for B. After that do insertion and assembly for B. These steps work fine.

Now the question is how to insert and assemble shell matrix A. MatPreallocatorPreallocation doesn?t work with shell matrix as indicated by an error message. MatSetValues or Mat assembly doesn?t support shell matrix either.

  A shell matrix is completely under your control. You decide on what data you want to store in it, and you provide the matrix-vector product routine.  MatPreallocatorPreallocation and MatSetValues don't work for shell matrix unless you provide code for them to work, which normally you would not do.

  I don't understand why you want to use MatSetValues for your shell matrix. If you can use MatSetValues() and just build an explicit matrix, then why would you use a shell matrix, just use MATAIJ matrix?

  Or does part of the operation inside your MatShell require a "regular" PETSc matrix? This would be fine, but it is your responsibility to design your Mat shell context to contain the "regular matrix" and your responsibility to fill up that regular matrix inside your shell context, calling whatever code you choose to write. So for example your shell context would be a C struct and one of the fields would be a regular PETSc matrix, also part of your custom matrix-vector product routine would call a regular MatMult() on the regular PETSc matrix stored inside the C struct.

  Barry



Thank you,
Jiannan

From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Friday, August 19, 2022 4:02 PM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] Using matrix-free with KSP

CAUTION: This email was sent from outside the UMass Lowell network.


  I don't know why changing the PCType could matter, can you please try with your new code but use the -pc_type none option to turn off the use of the preconditioner to see if the error still appears (I think it should).

  Also if you have a custom function that can do the matrix-vector product (not using any differencing) then you should use MatShell() and provide the matrix-vector product operation with your code.

  Barry



On Aug 19, 2022, at 11:47 AM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

Barry,

I have my own matrix-vector product but not use MatShell. I followed your previous instruction to use

    MatCreateMFFD(MPI_COMM_WORLD, PETSC_DECIDE, PETSC_DECIDE, N, N, &A);
    MatMFFDSetFunction(A, formfunction, &params);
    MatSetFromOptions(A);
    MatMFFDSetBase(A, X, NULL);

The customized matrix-vector product is implemented in formfunction() provided to MatMFFDSetFunction(). In this way, the matrix A is not used. I think that is why Petsc complains the matrix is not assembled. However, when PC type is none, the code runs without error although KSP solver doesn?t converge.

The question then becomes how to link A to matrix-free matrix-vector multiplication. And where to put MatShell?

Thank you,
Jiannan

From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Friday, August 19, 2022 10:27 AM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] Using matrix-free with KSP

CAUTION: This email was sent from outside the UMass Lowell network.





On Aug 18, 2022, at 11:41 PM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

The program calls KSPSolve just once and the error occurs after the first iteration.

I realized the matrix-vector product in my case is different from that approximating matrix-vector multiply by finite difference of function vector. It consists of two parts. One is multiplication of part of the matrix with part of vector, and another part results from so called fast multipole method. I?m not clear how to relate such calculation of matrix-vector product to the finite differencing of function vector.

  Are you using finite differencing at all in a matrix-vector product, or do you have your own custom matrix-vector product code (with MatShell)?

  Anyways the easiest thing to do is

  use the run time option -on_error_attach_debugger noxterm and run the code,

  when it stops in the debugger you can type bt (for backtrace)

  then do up a few times to move up the stack until you get to the PETSc function call that generates the error

  you can then do "call MatView(thematrixvarible,0) to have PETSc display what the matrix is and

  you can also do print *thematrixvariable to see the exact struct with all its fields

  If this doesn't tell you why the troublesome matrix is not assembled then cut and paste ALL of the output and email it

  Barry






From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Thursday, August 18, 2022 7:44 PM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] Using matrix-free with KSP

CAUTION: This email was sent from outside the UMass Lowell network.



   Is it in the first iteration of the first KSPSolve that you get this message? Or in the second call to KSPSolve?

  The error comes from a matrix vector product with your A matrix, what is that and where did it come from. The error message is not from your B matrix, that is not used in the MatMult()





On Aug 18, 2022, at 3:05 PM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

Barry,

Thanks. The method works. Assembly of actual matrix now completes quickly. The convergency is much better for the first iteration. The residual normal is in the order of 10^-10 instead of 10^-2. However, there is an error message after the first iteration: ?Object is in wrong state. Not for unassembled matrix? even though both preallocator and actual matrix have been assembled (see the code snippet below). Could you please give me some hints what?s going wrong? I use KSP type bcgs and PC type mat

Thank you,
Jiannan

-----------------------------------------------------
Solve linear equations ...
  0 KSP Residual norm 9.841179147519e-10
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Object is in wrong state
[0]PETSC ERROR: Not for unassembled matrix
[0]PETSC ERROR: See https://petsc.org/release/faq/<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Frelease%2Ffaq%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7C76c4d9eda7804ab02ddb08da843bcb86%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637967689641631746%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=yku5%2F0k1g7PMAwuI4WGQ5MmM4g7OUh0C9EvjaWAJAAE%3D&reserved=0> for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.16.6, Mar 30, 2022
[0]PETSC ERROR: ./iesolver on a  named REXNET-WS4 by jiannantu Thu Aug 18 14:48:43 2022
[0]PETSC ERROR: Configure options --download-f2cblaslapack=yes --with-mpi-dir=/usr/local --with-fc=0 --prefix=/home/jiannantu/petsc --with-scalar-type=complex --with-64-bit-indices=1
[0]PETSC ERROR: #1 MatMult() at /home/jiannantu/petsc-3.16.6/src/mat/interface/matrix.c:2425
[0]PETSC ERROR: #2 PCApplyBAorAB() at /home/jiannantu/petsc-3.16.6/src/ksp/pc/interface/precon.c:730
[0]PETSC ERROR: #3 KSP_PCApplyBAorAB() at /home/jiannantu/petsc-3.16.6/include/petsc/private/kspimpl.h:421
[0]PETSC ERROR:  Linear equations not solved. Diverged reason: 0
#4 KSPSolve_BCGS() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/impls/bcgs/bcgs.c:87
[0]PETSC ERROR: #5 KSPSolve_Private() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itfunc.c:914
[0]PETSC ERROR: #6 KSPSolve() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itfunc.c:1086

-----------------------------------------------
    MatCreate(MPI_COMM_WORLD, &B);
    MatSetType(B, MATMPIAIJ);
    MatSetSizes(B, PETSC_DECIDE, PETSC_DECIDE, N, N);

    Mat preall;
    MatCreate(MPI_COMM_WORLD, &preall);
    MatSetType(preall, MATPREALLOCATOR);
    MatSetSizes(preall, PETSC_DECIDE, PETSC_DECIDE, N, N);

    MatMPIAIJSetPreallocation(preall, 0, d_nnz, 0, o_nnz);
    MatSetUp(preall);
MatSetOption(preall, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE);

//set values row by row
MatSetValues(preall, 1, &m, 1, nCols, matCols, INSERT_VALUES);

MatAssemblyBegin(preall, MAT_FINAL_ASSEMBLY);
MatAssemblyEnd(preall, MAT_FINAL_ASSEMBLY);

//set values for actual matrix row by row
MatSetValues(B, 1, &m, 1, nCols, matCols, INSERT_VALUES);

MatAssemblyBegin(B, MAT_FINAL_ASSEMBLY);
MatAssemblyEnd(B, MAT_FINAL_ASSEMBLY);

    KSPCreate(MPI_COMM_WORLD, &ksp);
    KSPSetOperators(ksp, A, B);
    KSPSetFromOptions(ksp);

    PC pc;
    KSPGetPC(ksp, &pc);
    PCSetFromOptions(pc);

From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Thursday, August 18, 2022 12:59 PM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] Using matrix-free with KSP

CAUTION: This email was sent from outside the UMass Lowell network.


  Yes, once you have preallocated the real matrix you can destroy the preallocation matrix whose only job is to gather the preallocation information





On Aug 18, 2022, at 12:52 PM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

Thanks, Barry.

So I need actually two matrixes, one for preallocator and one for actual matrix that can be passed to KSPSetOperators(). -mat_type preallocator option is used to speed up doing insertion into preallocator, then use MatPreallocatorPreallocate() to preallocate actual matrix and do actual insertion of values into it., right?

The code runs in parallel. Each process owns number of rows that equals to number of unknowns (that is, xm in 1D DM) it owns.

Jiannan

From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Thursday, August 18, 2022 11:37 AM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] Using matrix-free with KSP

CAUTION: This email was sent from outside the UMass Lowell network.


   The preallocator MatType matrix cannot be passed to the KSPSetOperators(), you need to create an actual matrix, for example MATAIJ and use the preallocator to set its preallocation and then fill up the MATAIJ matrix with the usual MatSetValues and pass that matrix to the KSPSetOperators.






On Aug 18, 2022, at 11:30 AM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

Hi Barry,

The MATPREALLOCATOR solved problem of slow matrix assembly. But the solver failed because of ?Matrix type preallocator does not have a multiply defined?.

I guess this is because the matrix-free is used. I am wondering how a preconditioner is applied to the matrix-vector multiply in Petsc.

Thank you,
Jiannan

[0]PETSC ERROR: No support for this operation for this object type
[0]PETSC ERROR: Matrix type preallocator does not have a multiply defined
[0]PETSC ERROR: See https://petsc.org/release/faq/<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Frelease%2Ffaq%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7C76c4d9eda7804ab02ddb08da843bcb86%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637967689641631746%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=yku5%2F0k1g7PMAwuI4WGQ5MmM4g7OUh0C9EvjaWAJAAE%3D&reserved=0> for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.16.6, Mar 30, 2022
[0]PETSC ERROR: ./iesolver on a  named REXNET-WS4 by jiannantu Thu Aug 18 11:21:55 2022
[0]PETSC ERROR: Configure options --download-f2cblaslapack=yes --with-mpi-dir=/usr/local --with-fc=0 --prefix=/home/jiannantu/petsc --with-scalar-type=complex --with-64-bit-indices=1
[0]PETSC ERROR: #1 MatMult() at /home/jiannantu/petsc-3.16.6/src/mat/interface/matrix.c:2437
[0]PETSC ERROR: #2 PCApply_Mat() at /home/jiannantu/petsc-3.16.6/src/ksp/pc/impls/mat/pcmat.c:9
[0]PETSC ERROR: #3 PCApply() at /home/jiannantu/petsc-3.16.6/src/ksp/pc/interface/precon.c:445
[0]PETSC ERROR: #4 KSP_PCApply() at /home/jiannantu/petsc-3.16.6/include/petsc/private/kspimpl.h:382
[0]PETSC ERROR: #5 KSPInitialResidual() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itres.c:65
[0]PETSC ERROR: #6 KSPSolve_BCGS() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/impls/bcgs/bcgs.c:43
[0]PETSC ERROR: #7 KSPSolve_Private() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itfunc.c:914
[0]PETSC ERROR: #8 KSPSolve() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itfunc.c:1086

From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Thursday, August 18, 2022 8:35 AM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] Using matrix-free with KSP

CAUTION: This email was sent from outside the UMass Lowell network.


  Slow assembly is usually due to under preallocation. You can change to using

MatSetOption(B, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_TRUE);

  to detect if you are under preallocating. See https://petsc.org/main/docs/manual/mat/#sec-matsparse<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanual%2Fmat%2F%23sec-matsparse&data=05%7C01%7CJiannan_Tu%40uml.edu%7C76c4d9eda7804ab02ddb08da843bcb86%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637967689641631746%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=R%2F49HLXjwq%2FdFzTGL8mpz8X7YKi0iH0%2FWDz4FANCoD8%3D&reserved=0>. That section clearly needs some sprucing up. If it is difficult to determine good preallocation values you can use https://petsc.org/main/docs/manualpages/Mat/MATPREALLOCATOR/<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMATPREALLOCATOR%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7C76c4d9eda7804ab02ddb08da843bcb86%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637967689641631746%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=yiRv7HiBgqJxwszz%2FdV2EsO7iJ5rgVIOMeW1cyNzScI%3D&reserved=0> to compute the needed preallocation efficiently.

  Are you running in parallel? If so how are you determining which rows of entries to compute on each MPI rank? You will want most of the rows to be computed on the rank where the values are stored, you can determine this with https://petsc.org/main/docs/manualpages/Mat/MatGetOwnershipRange.html<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMatGetOwnershipRange.html&data=05%7C01%7CJiannan_Tu%40uml.edu%7C76c4d9eda7804ab02ddb08da843bcb86%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637967689641631746%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=W5lM%2B6UCP3%2BT5zT2uZPObqt0sZ8u841Jn79fBdikXWY%3D&reserved=0>

  Barry

















On Aug 18, 2022, at 12:50 AM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

I implemented the preconditioner. The solver converges quickly when the problem size is small. But when the size increases, say to 100k unknowns, the code hangs at assembly of the preconditioner matrix. The function call sequence is like this

   MatCreate(MPI_COMM_WORLD, &B);
    MatSetType(B, MATMPIAIJ);
    MatSetSizes(B, PETSC_DECIDE, PETSC_DECIDE, N, N);
    MatSetFromOptions(B);

    // The number of diagonal and off diagonal non zeros only can be determined run-time.
    MatMPIAIJSetPreallocation(B, 0, d_nnz, 0, o_nnz);
    MatSetOption(B, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE);

    //here insert values row by row.
MatSetValues(B, 1, &m, nn, nCols, matCols, INSERT_VALUES);

MatAssemblyBegin(B, MAT_FINAL_ASSEMBLY);
MatAssemblyEnd(B, MAT_FINAL_ASSEMBLY);

Could you please tell me what I have done wrong?

Thank you,
Jiannan

From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Tuesday, August 16, 2022 2:00 PM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] Using matrix-free with KSP

CAUTION: This email was sent from outside the UMass Lowell network.


   Create another matrix B that contains "Selected parts of the matrix can be pre-calculated and stored to serve as the preconditioner." When you call KSPSetOperators() pass in B as the second matrix argument.

   Now if literally application of B is the preconditioner you want to use, then simply use PCMAT as the preconditioner, and it will apply B each time it needs to apply the preconditioner. With this, you do not even need a PCShell.

   If your preconditioner uses B to construct your own specific data structure needed to apply your preconditioner, then you can use PCShellSetSetUp() and have your routine call PCGetOperators() to pull out the B matrix and use it to construct your preconditioner.

   If you want PETSc to construct a standard preconditioner from B, then you don't need the PCShell; you simply pass in the B, as above, and use -pc_type type to set the preconditioner you want to use.

   Barry









On Aug 16, 2022, at 12:31 PM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

Barry,

Thank you very much for your instructions. I am sorry I may not ask clearer questions.

I don't use SNES. What I am trying to solve is a very large linear equation system with dense and ill-conditioned matrix. Selected parts of the matrix can be pre-calculated and stored to serve as the preconditioner. My question is how I can apply this matrix as the preconditioner.

Here is the part of code from my linear equations solver. MatrixFreePreconditioner is supposed to be the routine to provide the preconditioner. But I don't understand how to use it. Should I use SNESSetJacobian() even if this is a linear problem?

    MatCreateMFFD(MPI_COMM_WORLD, PETSC_DECIDE, PETSC_DECIDE, N, N, &A);
    MatMFFDSetFunction(A, formfunction, &params);
    MatSetFromOptions(A);
    MatMFFDSetBase(A, X, NULL);

    KSPCreate(MPI_COMM_WORLD, &ksp);
    KSPSetOperators(ksp, A, A);
    KSPSetFromOptions(ksp);

    PC pc;
    KSPGetPC(ksp,&pc);
    PCSetType(pc,PCSHELL);
    PCShellSetApply(pc,MatrixFreePreconditioner);

Thank you,
Jiannan

________________________________
From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Tuesday, August 16, 2022 10:10 AM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] Using matrix-free with KSP

CAUTION: This email was sent from outside the UMass Lowell network.


  I don't fully understand your question so I will answer questions that I do know the answer to :-)

  The function you provide to PCShellSetApply() applies the preconditioner to the input vector and puts the result in the output vector.  This input vector changes at every application of the preconditioner so should not be used in constructing the preconditioner.

  If you are using the standard PETSc matrix-free matrix-vector product via finite difference, -snes_mf_operator or via MatCreateSNESMF() or you are using a MatShell, the matrix-vector product routines input changes for each multiply and should not be used in constructing the preconditioner.

  The function you provide with SNESSetJacobian() is where you can compute anything you need to help build your preconditioner. The input vector to that function is the location at which the Jacobian is needed and is what you should use to build your approximate/part of Jacobian. You can store your approximate/part of Jacobian that you need to compute in the pmat and then in a PCShellSetUp() function that you provide you will compute what you will need latter to apply the preconditioner in your PCShellSetApply() function. Note that the values you put in the pmat do not need to be the full Jacobian; they can be anything you want since this pmat is not used to apply the matrix vector product.

  If this is unclear, feel free to ask additional questions.

  Barry









On Aug 16, 2022, at 9:40 AM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

I am trying to apply a user-defined preconditioner to speed up the convergency of matrix-free KSP solver. The user's manual provides an example of how to call the user-defined preconditioner routine. The routine has PC, input Vec and output Vec as arguments. I want to use part of the Jacobian as preconditioner matrix, the elements of which can be calculated using input Vec. My question is what the role of output Vec from the routine is in matrix-vector product? Or I just have the preconditioner matrix elements calculated in the routine and then Petsc handles applying the preconditioner to matrix-vector product? I used PCSetType(pc, PCSHELL) and PCShellSetApply.

Thank you and appreciate your help!

Jiannan
________________________________
From: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Sent: Wednesday, July 6, 2022 2:34 PM
To: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Cc: Jed Brown <jed at jedbrown.org<mailto:jed at jedbrown.org>>; petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] Using matrix-free with KSP

Barry,

VecScatterCreateToAll solves the problem! The code now gives the correct answer with multiple processes.

Thank you all so much!

Jiannan
________________________________
From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Wednesday, July 6, 2022 2:08 PM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Cc: Jed Brown <jed at jedbrown.org<mailto:jed at jedbrown.org>>; petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] Using matrix-free with KSP


  So your operator is a "dense" operator in that its matrix representation would be essentially dense. In that case, you can use specialized routines to do this efficiently. You can use VecScatterCreateToAll() and then VecScatterBegin/End to do the communication.

  Barry










On Jul 6, 2022, at 1:39 PM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

Jed, thank you very much for the reply.

I'm not sure GlobaltoLocal will work. Say the solution vector is of length 10. Two processes then each process can see 5 elements of the vector. For A x = b (10 equations), each process performs matrix-vector product for five equations. And for each equation i , sum_j A_ij *  x_j = b_i requires the process has access to all 10 unknows x_i (i=0,1,...,9).

Can VecScatter and PetscSF make the entire vector accessible to each process? I am reading the manual and trying to understand how VecScatter and PetscSF work.

Jiannan
________________________________
From: Jed Brown <jed at jedbrown.org<mailto:jed at jedbrown.org>>
Sent: Wednesday, July 6, 2022 10:09 AM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>; Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] Using matrix-free with KSP

You'll usually have a GlobalToLocal operation for each rank to get the halo data it needs, then either a LocalToGlobal to collect the result (e.g., finite element methods) or the local compute will write directly into the owned portion of the global vector. If you're doing the communication "raw", then you may find VecScatter or PetscSF useful to perform the necessary communication.

"Tu, Jiannan" <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> writes:

> Hi Barry,
>
> Following your instructions I implemented the matrix-free method to solve a large linear equation system resulted from a surface integration equation. The KSP solver works fine for single process, but it is problematic with multiple processes. The problem is that each process only can access its own part of solution vector so that each process only conducts part of matrix-vector multiplication. MPI can be used to assemble these partial matrix-vector multiplication together.  Does Petsc provide any ways to implement multi-process matrix-free method?
>
> Thanks,
> Jiannan
> ________________________________
> From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
> Sent: Tuesday, May 24, 2022 2:12 PM
> To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
> Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
> Subject: Re: [petsc-users] Using matrix-free with KSP
>
> This e-mail originated from outside the UMass Lowell network.
> ________________________________
>
>    You can use MatCreateMFFD https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMatCreateMFFD%2F&amp;data=05%7C01%7CJiannan_Tu%40uml.edu%7Cab0859d8d154471590bc08da5f5918d6%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637927133525745809%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=A7wqegTfh94No5BpDiWLK3VxOuR44U2wlWHVm2k7l60%3D&amp;reserved=0<https://urldefense.com/v3/__https://petsc.org/main/docs/manualpages/Mat/MatCreateMFFD/__;!!PVKG_VDCxu5g!stxq2NWGVjUihvHBQC9Rlk0aHVoowcy0PTQYMcQxQ4DqOEC05KSw6TmstSXKckiUgelHzy4ue-d10-zlXkw$><https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMatCreateMFFD%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7C76c4d9eda7804ab02ddb08da843bcb86%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637967689641631746%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=psLU6qTaDTxSsrgXMdgpyjfdc0Pc1ZD%2BrzxDa8ThXHk%3D&reserved=0> MatMFFDSetFunction MatSetFromOptions MatMFFDSetBase and provide the matrix to KSP. Note you will need to use -pc_type none or provide your own custom preconditioner withhttps://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FPC%2FPCSHELL%2F&amp;data=05%7C01%7CJiannan_Tu%40uml.edu%7Cab0859d8d154471590bc08da5f5918d6%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637927133525745809%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=Gb%2F9uFG3jp%2FbfWaSh2cSEQLItHS9CuLwarzN0KcNiJY%3D&amp;reserved=0<https://urldefense.com/v3/__https://petsc.org/main/docs/manualpages/PC/PCSHELL/__;!!PVKG_VDCxu5g!stxq2NWGVjUihvHBQC9Rlk0aHVoowcy0PTQYMcQxQ4DqOEC05KSw6TmstSXKckiUgelHzy4ue-d1cA4fKGw$><https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FPC%2FPCSHELL%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7C76c4d9eda7804ab02ddb08da843bcb86%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637967689641631746%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=CWffbXJW4UGJifTbRNGmI2IpLYVVglfeLyzLxprZLyY%3D&reserved=0>
>
>
>
> On May 24, 2022, at 1:21 PM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu><mailto:Jiannan_Tu at uml.edu>> wrote:
>
> I want to use a matrix-free matrix to solve a large linear equation system because the matrix is too large to be stored. Petsc user manual describes matrix-free method for SNES with examples. The matrix-free matrices section explains how to set up such a matrix, but I can't find any example of matrix-free method with KSP. I am wondering how to apply the method to KSP iterative solver in parallel.
>
> I greatly appreciate your help for me to understand this topic.
>
> Jiannan Tu

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220822/3fcd3f9b/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 19906 bytes
Desc: image.png
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220822/3fcd3f9b/attachment-0001.png>

From bsmith at petsc.dev  Mon Aug 22 11:39:15 2022
From: bsmith at petsc.dev (Barry Smith)
Date: Mon, 22 Aug 2022 12:39:15 -0400
Subject: [petsc-users] Preallocating a matrix with the d_nnz and o_nnz
 arrays
In-Reply-To: <a6fbe8b2-0680-faf4-8b23-f248f7a7bad3@geomag.info>
References: <47e773f6-f008-ec23-7554-ddefc30ddccf@geomag.info>
	<SA1PR09MB8607667B9B068ADE59ACC19588719@SA1PR09MB8607.namprd09.prod.outlook.com>
	<a6fbe8b2-0680-faf4-8b23-f248f7a7bad3@geomag.info>
Message-ID: <18A38FB2-EF56-45EA-AE4E-DB814BDDC805@petsc.dev>


 MatGetOwnershipRangeColumn() tells you the start and end+1 of the columns that are in the diagonal block. For square matrices they are the same as rstart and rend.

  Barry


> On Aug 22, 2022, at 12:12 PM, Patrick Alken <patrick.alken at geomag.info> wrote:
> 
> Thank you, I have read that document. I have changed my criteria to:
> 
> if (j >= first && (j - first) < m) {
>   /*diagonal*/
> else
>   /*off-diagonal*/
> 
> It seems to be working better. I don't know how to get the c1 and c2 columns referred to by the documentation.
> 
> Patrick
> 
> 
> On 8/22/22 10:03, Zhang, Hong wrote:
>> See https://petsc.org/release/docs/manualpages/Mat/MatMPIAIJSetPreallocation.html#MatMPIAIJSetPreallocation <https://petsc.org/release/docs/manualpages/Mat/MatMPIAIJSetPreallocation.html#MatMPIAIJSetPreallocation>
>> MatMPIAIJSetPreallocation - PETSc <https://petsc.org/release/docs/manualpages/Mat/MatMPIAIJSetPreallocation.html#MatMPIAIJSetPreallocation>
>> MatMPIAIJSetPreallocation Preallocates memory for a sparse parallel matrix in AIJ format (the default parallel PETSc format). For good matrix assembly performance the user should preallocate the matrix storage by setting the parameters d_nz (or d_nnz) and o_nz (or o_nnz).
>> petsc.org <http://petsc.org/>
>> if you use mpiaij matrix.
>> Hong
>> From: petsc-users <petsc-users-bounces at mcs.anl.gov> <mailto:petsc-users-bounces at mcs.anl.gov> on behalf of Patrick Alken <patrick.alken at geomag.info> <mailto:patrick.alken at geomag.info>
>> Sent: Monday, August 22, 2022 10:56 AM
>> To: petsc-users <petsc-users at mcs.anl.gov> <mailto:petsc-users at mcs.anl.gov>
>> Subject: [petsc-users] Preallocating a matrix with the d_nnz and o_nnz arrays
>>  
>> I am looping through all the non-zero elements of my matrix to count the 
>> diagonal and off-diagonal terms in order to pre-allocate the matrix. 
>> However, what is the criteria for knowing if a given element is in the 
>> diagonal portion or off-diagonal portion? It seems that it will depend 
>> on row, column, and also processor. I was doing something like this:
>> 
>> MatGetLocalSize(A, &m, &n);
>> 
>> MatGetOwnershipRange(A, &first, &last);
>> 
>> for (i = first; i < last; ++i) {
>> 
>>      for (j = 0; j < N; ++j) {
>> 
>>          if (abs(i - j) < m)
>> 
>>              ++d_nnz[i - first];
>> 
>>          else
>> 
>>              ++o_nnz[i - first];
>> 
>>      }
>> 
>> }
>> 
>> However, the criteria: abs(i - j) < m is not correct. I think the 
>> correct criteria will include the processor number also. Does anyone 
>> know the correct criteria to use?

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220822/f9247c8a/attachment.html>

From bsmith at petsc.dev  Mon Aug 22 11:42:54 2022
From: bsmith at petsc.dev (Barry Smith)
Date: Mon, 22 Aug 2022 12:42:54 -0400
Subject: [petsc-users] Using matrix-free with KSP
In-Reply-To: <BY5PR02MB6471C5EAE18AB9B114E76232E7719@BY5PR02MB6471.namprd02.prod.outlook.com>
References: <BY5PR02MB6471969F80FA5F732FC74D87E7D79@BY5PR02MB6471.namprd02.prod.outlook.com>
	<B55CFC2C-6364-4B78-8E99-9D53700743B8@petsc.dev>
	<BY5PR02MB6471BDF90783FE3A0EA176CBE7809@BY5PR02MB6471.namprd02.prod.outlook.com>
	<87iloa49kt.fsf@jedbrown.org>
	<BY5PR02MB6471C0ED04769D02046F68ACE7809@BY5PR02MB6471.namprd02.prod.outlook.com>
	<00EBE4B8-22E5-408F-B56D-14226A278D7E@petsc.dev>
	<BY5PR02MB6471E386057DDDE98AD05DFAE7809@BY5PR02MB6471.namprd02.prod.outlook.com>
	<BY5PR02MB647140084C7E7CDDB6BC2A11E76B9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<A6629DE1-5A7F-414E-B164-A69877158675@petsc.dev>
	<BY5PR02MB6471AB1C004920DA1FC9B43EE76B9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<103146FD-FF28-47CE-B661-F2B521BF0E7A@petsc.dev>
	<DM6PR02MB6478B2FB8CB1BE757CE7B003E76D9@DM6PR02MB6478.namprd02.prod.outlook.com>
	<4C00F4B8-BB42-4932-BCD7-527E5FB39BFF@petsc.dev>
	<BY5PR02MB64710E70758E431108D66956E76D9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<D0EB13D8-C5A6-4AE7-AFDE-2CDCCACBAF73@petsc.dev>
	<BY5PR02MB6471D4B05874DECFDDA13FFCE76D9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<367B124C-88BB-4889-A4AF-45A1D473078D@petsc.dev>
	<BY5PR02MB6471F2C058B84DE092567731E76D9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<944DDEED-016D-43B2-A5A6-7BF2E8E41FA5@petsc.dev>
	<BY5PR02MB6471CD0236F03D20763A3B64E76C9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<B4CF5E5B-984D-4EC1-AB77-1FD08C3FBBCF@petsc.dev>
	<BY5PR02MB6471BA72871419781B875967E76C9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<3D179AEA-AD05-4B46-8DDC-9E126741C247@petsc.dev>
	<BY5PR02MB6471560EA70C6FAE826DDF27E7719@BY5PR02MB6471.namprd02.prod.outlook.com>
	<1C67C8DB-0653-4588-8137-E0865E0F2098@petsc.dev>
	<BY5PR02MB6471C5EAE18AB9B114E76232E7719@BY5PR02MB6471.namprd02.prod.outlook.com>
Message-ID: <FAA1668F-08C1-4BAD-89B4-784D8AD99C36@petsc.dev>



> On Aug 22, 2022, at 12:25 PM, Tu, Jiannan <Jiannan_Tu at uml.edu> wrote:
> 
> The matrix-vector product involves part of matrix elements explicitly as shown below so that I need to insert values into this part of the matrix.
> <image.png>
> Initially I wrote my own code for calculating and storing matrix A^near_mi without using any Petsc matrix and then performed matrix-vector product using above equation. The code runs but converges very slow. Apparently a preconditioned is needed. 
> 
> I am not clear how Petsc can provide a preconditioner for my case.

  I am now confused. I thought you were forming Anear as a PETSc matrix and passing that to KSPSetOperators as the second matrix and so the preconditioner was being built with Anear?

  For your shell matrix vector product you should call MatMult(Anear,x,y) inside your custom routine and then add the rest of the matrix-vector product using your fast multipole products. What am I missing?

  Barry

> 
> From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>>
> Sent: Monday, August 22, 2022 8:42 AM
> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
> Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>>
> Subject: Re: [petsc-users] Using matrix-free with KSP
>  
> CAUTION: This email was sent from outside the UMass Lowell network.
> 
> 
> 
>> On Aug 21, 2022, at 11:18 PM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> wrote:
>> 
>> Barry,
>>  
>> You are right. With PC type none, the same error appears.
>>  
>> Now I use MatShell and I literally have three matrixes. A is a shell matrix, B is a preconditioner, and C is a matrix just for preallocation purpose.
>>  
>> First C is used to determine non-zero structure. Then use MatPreallocatorPreallocation(C, PETSC_FALSE, B) to preallocate for B. After that do insertion and assembly for B. These steps work fine.
>>  
>> Now the question is how to insert and assemble shell matrix A. MatPreallocatorPreallocation doesn?t work with shell matrix as indicated by an error message. MatSetValues or Mat assembly doesn?t support shell matrix either. 
> 
>   A shell matrix is completely under your control. You decide on what data you want to store in it, and you provide the matrix-vector product routine.  MatPreallocatorPreallocation and MatSetValues don't work for shell matrix unless you provide code for them to work, which normally you would not do.
> 
>   I don't understand why you want to use MatSetValues for your shell matrix. If you can use MatSetValues() and just build an explicit matrix, then why would you use a shell matrix, just use MATAIJ matrix? 
> 
>   Or does part of the operation inside your MatShell require a "regular" PETSc matrix? This would be fine, but it is your responsibility to design your Mat shell context to contain the "regular matrix" and your responsibility to fill up that regular matrix inside your shell context, calling whatever code you choose to write. So for example your shell context would be a C struct and one of the fields would be a regular PETSc matrix, also part of your custom matrix-vector product routine would call a regular MatMult() on the regular PETSc matrix stored inside the C struct.
> 
>   Barry
> 
> 
>>  
>> Thank you,
>> Jiannan
>>  
>> From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>> 
>> Sent: Friday, August 19, 2022 4:02 PM
>> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
>> Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>
>> Subject: Re: [petsc-users] Using matrix-free with KSP
>>  
>> CAUTION: This email was sent from outside the UMass Lowell network.
>>  
>>  
>>   I don't know why changing the PCType could matter, can you please try with your new code but use the -pc_type none option to turn off the use of the preconditioner to see if the error still appears (I think it should). 
>>  
>>   Also if you have a custom function that can do the matrix-vector product (not using any differencing) then you should use MatShell() and provide the matrix-vector product operation with your code.
>>  
>>   Barry
>>  
>> 
>> 
>> On Aug 19, 2022, at 11:47 AM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> wrote:
>>  
>> Barry,
>>  
>> I have my own matrix-vector product but not use MatShell. I followed your previous instruction to use
>>  
>>     MatCreateMFFD(MPI_COMM_WORLD, PETSC_DECIDE, PETSC_DECIDE, N, N, &A);
>>     MatMFFDSetFunction(A, formfunction, &params);
>>     MatSetFromOptions(A);
>>     MatMFFDSetBase(A, X, NULL);
>>  
>> The customized matrix-vector product is implemented in formfunction() provided to MatMFFDSetFunction(). In this way, the matrix A is not used. I think that is why Petsc complains the matrix is not assembled. However, when PC type is none, the code runs without error although KSP solver doesn?t converge.
>>  
>> The question then becomes how to link A to matrix-free matrix-vector multiplication. And where to put MatShell?
>>  
>> Thank you,
>> Jiannan
>>  
>> From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>> 
>> Sent: Friday, August 19, 2022 10:27 AM
>> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
>> Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>
>> Subject: Re: [petsc-users] Using matrix-free with KSP
>>  
>> CAUTION: This email was sent from outside the UMass Lowell network.
>>  
>>  
>> 
>> 
>> 
>> On Aug 18, 2022, at 11:41 PM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> wrote:
>>  
>> The program calls KSPSolve just once and the error occurs after the first iteration.
>>  
>> I realized the matrix-vector product in my case is different from that approximating matrix-vector multiply by finite difference of function vector. It consists of two parts. One is multiplication of part of the matrix with part of vector, and another part results from so called fast multipole method. I?m not clear how to relate such calculation of matrix-vector product to the finite differencing of function vector.
>>  
>>   Are you using finite differencing at all in a matrix-vector product, or do you have your own custom matrix-vector product code (with MatShell)? 
>>  
>>   Anyways the easiest thing to do is 
>>  
>>   use the run time option -on_error_attach_debugger noxterm and run the code, 
>>  
>>   when it stops in the debugger you can type bt (for backtrace)
>>  
>>   then do up a few times to move up the stack until you get to the PETSc function call that generates the error
>>  
>>   you can then do "call MatView(thematrixvarible,0) to have PETSc display what the matrix is and
>>  
>>   you can also do print *thematrixvariable to see the exact struct with all its fields
>>  
>>   If this doesn't tell you why the troublesome matrix is not assembled then cut and paste ALL of the output and email it
>>  
>>   Barry
>> 
>> 
>> 
>>  
>>  
>>  
>> From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>> 
>> Sent: Thursday, August 18, 2022 7:44 PM
>> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
>> Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>
>> Subject: Re: [petsc-users] Using matrix-free with KSP
>>  
>> CAUTION: This email was sent from outside the UMass Lowell network.
>>  
>>  
>>  
>>    Is it in the first iteration of the first KSPSolve that you get this message? Or in the second call to KSPSolve? 
>>  
>>   The error comes from a matrix vector product with your A matrix, what is that and where did it come from. The error message is not from your B matrix, that is not used in the MatMult()
>>  
>> 
>> 
>> 
>> 
>> On Aug 18, 2022, at 3:05 PM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> wrote:
>>  
>> Barry, 
>>  
>> Thanks. The method works. Assembly of actual matrix now completes quickly. The convergency is much better for the first iteration. The residual normal is in the order of 10^-10 instead of 10^-2. However, there is an error message after the first iteration: ?Object is in wrong state. Not for unassembled matrix? even though both preallocator and actual matrix have been assembled (see the code snippet below). Could you please give me some hints what?s going wrong? I use KSP type bcgs and PC type mat
>>  
>> Thank you,
>> Jiannan
>>  
>> -----------------------------------------------------
>> Solve linear equations ...
>>   0 KSP Residual norm 9.841179147519e-10
>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>> [0]PETSC ERROR: Object is in wrong state
>> [0]PETSC ERROR: Not for unassembled matrix
>> [0]PETSC ERROR: See https://petsc.org/release/faq/ <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Frelease%2Ffaq%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7C76c4d9eda7804ab02ddb08da843bcb86%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637967689641631746%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=yku5%2F0k1g7PMAwuI4WGQ5MmM4g7OUh0C9EvjaWAJAAE%3D&reserved=0> for trouble shooting.
>> [0]PETSC ERROR: Petsc Release Version 3.16.6, Mar 30, 2022
>> [0]PETSC ERROR: ./iesolver on a  named REXNET-WS4 by jiannantu Thu Aug 18 14:48:43 2022
>> [0]PETSC ERROR: Configure options --download-f2cblaslapack=yes --with-mpi-dir=/usr/local --with-fc=0 --prefix=/home/jiannantu/petsc --with-scalar-type=complex --with-64-bit-indices=1
>> [0]PETSC ERROR: #1 MatMult() at /home/jiannantu/petsc-3.16.6/src/mat/interface/matrix.c:2425
>> [0]PETSC ERROR: #2 PCApplyBAorAB() at /home/jiannantu/petsc-3.16.6/src/ksp/pc/interface/precon.c:730
>> [0]PETSC ERROR: #3 KSP_PCApplyBAorAB() at /home/jiannantu/petsc-3.16.6/include/petsc/private/kspimpl.h:421
>> [0]PETSC ERROR:  Linear equations not solved. Diverged reason: 0
>> #4 KSPSolve_BCGS() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/impls/bcgs/bcgs.c:87
>> [0]PETSC ERROR: #5 KSPSolve_Private() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itfunc.c:914
>> [0]PETSC ERROR: #6 KSPSolve() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itfunc.c:1086
>>  
>> -----------------------------------------------
>>     MatCreate(MPI_COMM_WORLD, &B);
>>     MatSetType(B, MATMPIAIJ);
>>     MatSetSizes(B, PETSC_DECIDE, PETSC_DECIDE, N, N);
>>  
>>     Mat preall;
>>     MatCreate(MPI_COMM_WORLD, &preall);
>>     MatSetType(preall, MATPREALLOCATOR);
>>     MatSetSizes(preall, PETSC_DECIDE, PETSC_DECIDE, N, N);
>>  
>>     MatMPIAIJSetPreallocation(preall, 0, d_nnz, 0, o_nnz);
>>     MatSetUp(preall);
>> MatSetOption(preall, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE);
>>  
>> //set values row by row
>> MatSetValues(preall, 1, &m, 1, nCols, matCols, INSERT_VALUES);
>>  
>> MatAssemblyBegin(preall, MAT_FINAL_ASSEMBLY);
>> MatAssemblyEnd(preall, MAT_FINAL_ASSEMBLY);
>>  
>> //set values for actual matrix row by row
>> MatSetValues(B, 1, &m, 1, nCols, matCols, INSERT_VALUES);
>>  
>> MatAssemblyBegin(B, MAT_FINAL_ASSEMBLY);
>> MatAssemblyEnd(B, MAT_FINAL_ASSEMBLY);
>>  
>>     KSPCreate(MPI_COMM_WORLD, &ksp);
>>     KSPSetOperators(ksp, A, B);
>>     KSPSetFromOptions(ksp);
>>  
>>     PC pc;
>>     KSPGetPC(ksp, &pc);
>>     PCSetFromOptions(pc);
>>  
>> From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>> 
>> Sent: Thursday, August 18, 2022 12:59 PM
>> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
>> Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>
>> Subject: Re: [petsc-users] Using matrix-free with KSP
>>  
>> CAUTION: This email was sent from outside the UMass Lowell network.
>>  
>>  
>>   Yes, once you have preallocated the real matrix you can destroy the preallocation matrix whose only job is to gather the preallocation information
>> 
>> 
>> 
>> 
>> 
>> On Aug 18, 2022, at 12:52 PM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> wrote:
>>  
>> Thanks, Barry.
>>  
>> So I need actually two matrixes, one for preallocator and one for actual matrix that can be passed to KSPSetOperators(). -mat_type preallocator option is used to speed up doing insertion into preallocator, then use MatPreallocatorPreallocate() to preallocate actual matrix and do actual insertion of values into it., right?
>>  
>> The code runs in parallel. Each process owns number of rows that equals to number of unknowns (that is, xm in 1D DM) it owns.
>>  
>> Jiannan
>>  
>> From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>> 
>> Sent: Thursday, August 18, 2022 11:37 AM
>> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
>> Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>
>> Subject: Re: [petsc-users] Using matrix-free with KSP
>>  
>> CAUTION: This email was sent from outside the UMass Lowell network.
>>  
>>  
>>    The preallocator MatType matrix cannot be passed to the KSPSetOperators(), you need to create an actual matrix, for example MATAIJ and use the preallocator to set its preallocation and then fill up the MATAIJ matrix with the usual MatSetValues and pass that matrix to the KSPSetOperators.
>> 
>> 
>> 
>> 
>> 
>> 
>> On Aug 18, 2022, at 11:30 AM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> wrote:
>>  
>> Hi Barry,
>>  
>> The MATPREALLOCATOR solved problem of slow matrix assembly. But the solver failed because of ?Matrix type preallocator does not have a multiply defined?.
>>  
>> I guess this is because the matrix-free is used. I am wondering how a preconditioner is applied to the matrix-vector multiply in Petsc.
>>  
>> Thank you,
>> Jiannan
>>  
>> [0]PETSC ERROR: No support for this operation for this object type
>> [0]PETSC ERROR: Matrix type preallocator does not have a multiply defined
>> [0]PETSC ERROR: See https://petsc.org/release/faq/ <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Frelease%2Ffaq%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7C76c4d9eda7804ab02ddb08da843bcb86%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637967689641631746%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=yku5%2F0k1g7PMAwuI4WGQ5MmM4g7OUh0C9EvjaWAJAAE%3D&reserved=0> for trouble shooting.
>> [0]PETSC ERROR: Petsc Release Version 3.16.6, Mar 30, 2022
>> [0]PETSC ERROR: ./iesolver on a  named REXNET-WS4 by jiannantu Thu Aug 18 11:21:55 2022
>> [0]PETSC ERROR: Configure options --download-f2cblaslapack=yes --with-mpi-dir=/usr/local --with-fc=0 --prefix=/home/jiannantu/petsc --with-scalar-type=complex --with-64-bit-indices=1
>> [0]PETSC ERROR: #1 MatMult() at /home/jiannantu/petsc-3.16.6/src/mat/interface/matrix.c:2437
>> [0]PETSC ERROR: #2 PCApply_Mat() at /home/jiannantu/petsc-3.16.6/src/ksp/pc/impls/mat/pcmat.c:9
>> [0]PETSC ERROR: #3 PCApply() at /home/jiannantu/petsc-3.16.6/src/ksp/pc/interface/precon.c:445
>> [0]PETSC ERROR: #4 KSP_PCApply() at /home/jiannantu/petsc-3.16.6/include/petsc/private/kspimpl.h:382
>> [0]PETSC ERROR: #5 KSPInitialResidual() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itres.c:65
>> [0]PETSC ERROR: #6 KSPSolve_BCGS() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/impls/bcgs/bcgs.c:43
>> [0]PETSC ERROR: #7 KSPSolve_Private() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itfunc.c:914
>> [0]PETSC ERROR: #8 KSPSolve() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itfunc.c:1086
>>  
>> From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>> 
>> Sent: Thursday, August 18, 2022 8:35 AM
>> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
>> Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>
>> Subject: Re: [petsc-users] Using matrix-free with KSP
>>  
>> CAUTION: This email was sent from outside the UMass Lowell network.
>>  
>>  
>>   Slow assembly is usually due to under preallocation. You can change to using 
>>  
>> MatSetOption(B, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_TRUE);
>>  
>>   to detect if you are under preallocating. See https://petsc.org/main/docs/manual/mat/#sec-matsparse <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanual%2Fmat%2F%23sec-matsparse&data=05%7C01%7CJiannan_Tu%40uml.edu%7C76c4d9eda7804ab02ddb08da843bcb86%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637967689641631746%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=R%2F49HLXjwq%2FdFzTGL8mpz8X7YKi0iH0%2FWDz4FANCoD8%3D&reserved=0>. That section clearly needs some sprucing up. If it is difficult to determine good preallocation values you can use https://petsc.org/main/docs/manualpages/Mat/MATPREALLOCATOR/ <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMATPREALLOCATOR%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7C76c4d9eda7804ab02ddb08da843bcb86%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637967689641631746%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=yiRv7HiBgqJxwszz%2FdV2EsO7iJ5rgVIOMeW1cyNzScI%3D&reserved=0> to compute the needed preallocation efficiently.
>>  
>>   Are you running in parallel? If so how are you determining which rows of entries to compute on each MPI rank? You will want most of the rows to be computed on the rank where the values are stored, you can determine this with https://petsc.org/main/docs/manualpages/Mat/MatGetOwnershipRange.html <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMatGetOwnershipRange.html&data=05%7C01%7CJiannan_Tu%40uml.edu%7C76c4d9eda7804ab02ddb08da843bcb86%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637967689641631746%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=W5lM%2B6UCP3%2BT5zT2uZPObqt0sZ8u841Jn79fBdikXWY%3D&reserved=0>
>>  
>>   Barry
>>  
>>  
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>>  
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> On Aug 18, 2022, at 12:50 AM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> wrote:
>>  
>> I implemented the preconditioner. The solver converges quickly when the problem size is small. But when the size increases, say to 100k unknowns, the code hangs at assembly of the preconditioner matrix. The function call sequence is like this
>>  
>>    MatCreate(MPI_COMM_WORLD, &B);
>>     MatSetType(B, MATMPIAIJ);
>>     MatSetSizes(B, PETSC_DECIDE, PETSC_DECIDE, N, N);
>>     MatSetFromOptions(B);
>>  
>>     // The number of diagonal and off diagonal non zeros only can be determined run-time.
>>     MatMPIAIJSetPreallocation(B, 0, d_nnz, 0, o_nnz);
>>     MatSetOption(B, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE);
>>  
>>     //here insert values row by row.
>> MatSetValues(B, 1, &m, nn, nCols, matCols, INSERT_VALUES);
>>  
>> MatAssemblyBegin(B, MAT_FINAL_ASSEMBLY);
>> MatAssemblyEnd(B, MAT_FINAL_ASSEMBLY);
>>  
>> Could you please tell me what I have done wrong?
>>  
>> Thank you,
>> Jiannan
>>    
>> From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>> 
>> Sent: Tuesday, August 16, 2022 2:00 PM
>> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
>> Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>
>> Subject: Re: [petsc-users] Using matrix-free with KSP
>>  
>> CAUTION: This email was sent from outside the UMass Lowell network.
>>  
>>  
>>    Create another matrix B that contains "Selected parts of the matrix can be pre-calculated and stored to serve as the preconditioner." When you call KSPSetOperators() pass in B as the second matrix argument.
>>  
>>    Now if literally application of B is the preconditioner you want to use, then simply use PCMAT as the preconditioner, and it will apply B each time it needs to apply the preconditioner. With this, you do not even need a PCShell. 
>>  
>>    If your preconditioner uses B to construct your own specific data structure needed to apply your preconditioner, then you can use PCShellSetSetUp() and have your routine call PCGetOperators() to pull out the B matrix and use it to construct your preconditioner. 
>>  
>>    If you want PETSc to construct a standard preconditioner from B, then you don't need the PCShell; you simply pass in the B, as above, and use -pc_type type to set the preconditioner you want to use. 
>>  
>>    Barry
>>  
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> On Aug 16, 2022, at 12:31 PM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> wrote:
>>  
>> Barry,
>>  
>> Thank you very much for your instructions. I am sorry I may not ask clearer questions.
>>  
>> I don't use SNES. What I am trying to solve is a very large linear equation system with dense and ill-conditioned matrix. Selected parts of the matrix can be pre-calculated and stored to serve as the preconditioner. My question is how I can apply this matrix as the preconditioner. 
>>  
>> Here is the part of code from my linear equations solver. MatrixFreePreconditioner is supposed to be the routine to provide the preconditioner. But I don't understand how to use it. Should I use SNESSetJacobian() even if this is a linear problem? 
>>  
>>     MatCreateMFFD(MPI_COMM_WORLD, PETSC_DECIDE, PETSC_DECIDE, N, N, &A);
>>     MatMFFDSetFunction(A, formfunction, &params);
>>     MatSetFromOptions(A);
>>     MatMFFDSetBase(A, X, NULL);
>>  
>>     KSPCreate(MPI_COMM_WORLD, &ksp);
>>     KSPSetOperators(ksp, A, A);
>>     KSPSetFromOptions(ksp);
>>  
>>     PC pc;
>>     KSPGetPC(ksp,&pc);
>>     PCSetType(pc,PCSHELL);
>>     PCShellSetApply(pc,MatrixFreePreconditioner);
>>  
>> Thank you,
>> Jiannan
>>  
>> From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>>
>> Sent: Tuesday, August 16, 2022 10:10 AM
>> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
>> Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>>
>> Subject: Re: [petsc-users] Using matrix-free with KSP
>>  
>> CAUTION: This email was sent from outside the UMass Lowell network.
>>  
>>  
>>   I don't fully understand your question so I will answer questions that I do know the answer to :-)
>>  
>>   The function you provide to PCShellSetApply() applies the preconditioner to the input vector and puts the result in the output vector.  This input vector changes at every application of the preconditioner so should not be used in constructing the preconditioner.
>>  
>>   If you are using the standard PETSc matrix-free matrix-vector product via finite difference, -snes_mf_operator or via MatCreateSNESMF() or you are using a MatShell, the matrix-vector product routines input changes for each multiply and should not be used in constructing the preconditioner.
>>  
>>   The function you provide with SNESSetJacobian() is where you can compute anything you need to help build your preconditioner. The input vector to that function is the location at which the Jacobian is needed and is what you should use to build your approximate/part of Jacobian. You can store your approximate/part of Jacobian that you need to compute in the pmat and then in a PCShellSetUp() function that you provide you will compute what you will need latter to apply the preconditioner in your PCShellSetApply() function. Note that the values you put in the pmat do not need to be the full Jacobian; they can be anything you want since this pmat is not used to apply the matrix vector product.
>>  
>>   If this is unclear, feel free to ask additional questions.
>>  
>>   Barry
>>  
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> On Aug 16, 2022, at 9:40 AM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> wrote:
>>  
>> I am trying to apply a user-defined preconditioner to speed up the convergency of matrix-free KSP solver. The user's manual provides an example of how to call the user-defined preconditioner routine. The routine has PC, input Vec and output Vec as arguments. I want to use part of the Jacobian as preconditioner matrix, the elements of which can be calculated using input Vec. My question is what the role of output Vec from the routine is in matrix-vector product? Or I just have the preconditioner matrix elements calculated in the routine and then Petsc handles applying the preconditioner to matrix-vector product? I used PCSetType(pc, PCSHELL) and PCShellSetApply.
>>  
>> Thank you and appreciate your help!
>>  
>> Jiannan 
>> From: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
>> Sent: Wednesday, July 6, 2022 2:34 PM
>> To: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>>
>> Cc: Jed Brown <jed at jedbrown.org <mailto:jed at jedbrown.org>>; petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>>
>> Subject: Re: [petsc-users] Using matrix-free with KSP
>>  
>> Barry,
>>  
>> VecScatterCreateToAll solves the problem! The code now gives the correct answer with multiple processes. 
>>  
>> Thank you all so much!
>>  
>> Jiannan
>> From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>>
>> Sent: Wednesday, July 6, 2022 2:08 PM
>> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
>> Cc: Jed Brown <jed at jedbrown.org <mailto:jed at jedbrown.org>>; petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>>
>> Subject: Re: [petsc-users] Using matrix-free with KSP
>>  
>>  
>>   So your operator is a "dense" operator in that its matrix representation would be essentially dense. In that case, you can use specialized routines to do this efficiently. You can use VecScatterCreateToAll() and then VecScatterBegin/End to do the communication.
>>  
>>   Barry
>>  
>>  
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> On Jul 6, 2022, at 1:39 PM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>  wrote:
>>  
>> Jed, thank you very much for the reply. 
>>  
>> I'm not sure GlobaltoLocal will work. Say the solution vector is of length 10. Two processes then each process can see 5 elements of the vector. For A x = b (10 equations), each process performs matrix-vector product for five equations. And for each equation i , sum_j A_ij *  x_j = b_i requires the process has access to all 10 unknows x_i (i=0,1,...,9). 
>>  
>> Can VecScatter and PetscSF make the entire vector accessible to each process? I am reading the manual and trying to understand how VecScatter and PetscSF work.
>>  
>> Jiannan
>> From: Jed Brown <jed at jedbrown.org <mailto:jed at jedbrown.org>>
>> Sent: Wednesday, July 6, 2022 10:09 AM
>> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>; Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>>
>> Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>>
>> Subject: Re: [petsc-users] Using matrix-free with KSP
>>  
>> You'll usually have a GlobalToLocal operation for each rank to get the halo data it needs, then either a LocalToGlobal to collect the result (e.g., finite element methods) or the local compute will write directly into the owned portion of the global vector. If you're doing the communication "raw", then you may find VecScatter or PetscSF useful to perform the necessary communication.
>> 
>> "Tu, Jiannan" <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> writes:
>> 
>> > Hi Barry,
>> >
>> > Following your instructions I implemented the matrix-free method to solve a large linear equation system resulted from a surface integration equation. The KSP solver works fine for single process, but it is problematic with multiple processes. The problem is that each process only can access its own part of solution vector so that each process only conducts part of matrix-vector multiplication. MPI can be used to assemble these partial matrix-vector multiplication together.  Does Petsc provide any ways to implement multi-process matrix-free method?
>> >
>> > Thanks,
>> > Jiannan
>> > ________________________________
>> > From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>>
>> > Sent: Tuesday, May 24, 2022 2:12 PM
>> > To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
>> > Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>>
>> > Subject: Re: [petsc-users] Using matrix-free with KSP
>> >
>> > This e-mail originated from outside the UMass Lowell network.
>> > ________________________________
>> >
>> >    You can use MatCreateMFFD https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMatCreateMFFD%2F&amp;data=05%7C01%7CJiannan_Tu%40uml.edu%7Cab0859d8d154471590bc08da5f5918d6%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637927133525745809%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=A7wqegTfh94No5BpDiWLK3VxOuR44U2wlWHVm2k7l60%3D&amp;reserved=0<https://urldefense.com/v3/__https://petsc.org/main/docs/manualpages/Mat/MatCreateMFFD/__;!!PVKG_VDCxu5g!stxq2NWGVjUihvHBQC9Rlk0aHVoowcy0PTQYMcQxQ4DqOEC05KSw6TmstSXKckiUgelHzy4ue-d10-zlXkw$> <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMatCreateMFFD%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7C76c4d9eda7804ab02ddb08da843bcb86%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637967689641631746%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=psLU6qTaDTxSsrgXMdgpyjfdc0Pc1ZD%2BrzxDa8ThXHk%3D&reserved=0> MatMFFDSetFunction MatSetFromOptions MatMFFDSetBase and provide the matrix to KSP. Note you will need to use -pc_type none or provide your own custom preconditioner withhttps://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FPC%2FPCSHELL%2F&amp;data=05%7C01%7CJiannan_Tu%40uml.edu%7Cab0859d8d154471590bc08da5f5918d6%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637927133525745809%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=Gb%2F9uFG3jp%2FbfWaSh2cSEQLItHS9CuLwarzN0KcNiJY%3D&amp;reserved=0<https://urldefense.com/v3/__https://petsc.org/main/docs/manualpages/PC/PCSHELL/__;!!PVKG_VDCxu5g!stxq2NWGVjUihvHBQC9Rlk0aHVoowcy0PTQYMcQxQ4DqOEC05KSw6TmstSXKckiUgelHzy4ue-d1cA4fKGw$> <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FPC%2FPCSHELL%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7C76c4d9eda7804ab02ddb08da843bcb86%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637967689641631746%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=CWffbXJW4UGJifTbRNGmI2IpLYVVglfeLyzLxprZLyY%3D&reserved=0>
>> >
>> >
>> >
>> > On May 24, 2022, at 1:21 PM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu><mailto:Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>> wrote:
>> >
>> > I want to use a matrix-free matrix to solve a large linear equation system because the matrix is too large to be stored. Petsc user manual describes matrix-free method for SNES with examples. The matrix-free matrices section explains how to set up such a matrix, but I can't find any example of matrix-free method with KSP. I am wondering how to apply the method to KSP iterative solver in parallel.
>> >
>> > I greatly appreciate your help for me to understand this topic.
>> >
>> > Jiannan Tu

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220822/ea4b667b/attachment-0001.html>

From Jiannan_Tu at uml.edu  Mon Aug 22 12:37:40 2022
From: Jiannan_Tu at uml.edu (Tu, Jiannan)
Date: Mon, 22 Aug 2022 17:37:40 +0000
Subject: [petsc-users] Using matrix-free with KSP
In-Reply-To: <FAA1668F-08C1-4BAD-89B4-784D8AD99C36@petsc.dev>
References: <BY5PR02MB6471969F80FA5F732FC74D87E7D79@BY5PR02MB6471.namprd02.prod.outlook.com>
	<B55CFC2C-6364-4B78-8E99-9D53700743B8@petsc.dev>
	<BY5PR02MB6471BDF90783FE3A0EA176CBE7809@BY5PR02MB6471.namprd02.prod.outlook.com>
	<87iloa49kt.fsf@jedbrown.org>
	<BY5PR02MB6471C0ED04769D02046F68ACE7809@BY5PR02MB6471.namprd02.prod.outlook.com>
	<00EBE4B8-22E5-408F-B56D-14226A278D7E@petsc.dev>
	<BY5PR02MB6471E386057DDDE98AD05DFAE7809@BY5PR02MB6471.namprd02.prod.outlook.com>
	<BY5PR02MB647140084C7E7CDDB6BC2A11E76B9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<A6629DE1-5A7F-414E-B164-A69877158675@petsc.dev>
	<BY5PR02MB6471AB1C004920DA1FC9B43EE76B9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<103146FD-FF28-47CE-B661-F2B521BF0E7A@petsc.dev>
	<DM6PR02MB6478B2FB8CB1BE757CE7B003E76D9@DM6PR02MB6478.namprd02.prod.outlook.com>
	<4C00F4B8-BB42-4932-BCD7-527E5FB39BFF@petsc.dev>
	<BY5PR02MB64710E70758E431108D66956E76D9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<D0EB13D8-C5A6-4AE7-AFDE-2CDCCACBAF73@petsc.dev>
	<BY5PR02MB6471D4B05874DECFDDA13FFCE76D9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<367B124C-88BB-4889-A4AF-45A1D473078D@petsc.dev>
	<BY5PR02MB6471F2C058B84DE092567731E76D9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<944DDEED-016D-43B2-A5A6-7BF2E8E41FA5@petsc.dev>
	<BY5PR02MB6471CD0236F03D20763A3B64E76C9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<B4CF5E5B-984D-4EC1-AB77-1FD08C3FBBCF@petsc.dev>
	<BY5PR02MB6471BA72871419781B875967E76C9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<3D179AEA-AD05-4B46-8DDC-9E126741C247@petsc.dev>
	<BY5PR02MB6471560EA70C6FAE826DDF27E7719@BY5PR02MB6471.namprd02.prod.outlook.com>
	<1C67C8DB-0653-4588-8137-E0865E0F2098@petsc.dev>
	<BY5PR02MB6471C5EAE18AB9B114E76232E7719@BY5PR02MB6471.namprd02.prod.outlook.com>
	<FAA1668F-08C1-4BAD-89B4-784D8AD99C36@petsc.dev>
Message-ID: <BY5PR02MB64715D7FE585A7F88092E04DE7719@BY5PR02MB6471.namprd02.prod.outlook.com>

Barry,

Yes, you are right, Anear is the preconditioner that needs to be passed to KSPSetOperators() as the second matrix.

I think I have figured out how to provide a preconditioner for matrix-free method. I was confused by matrix A created by MatCreateMFFD(), which is type MFFD. I thought I have to use it as the first matrix passed to KSPSetOperators(). Since it was not assembled (and can't) so that Petsc gave error message "matrix unassembled". In fact, I should ignore A, but use other two matrixes AA and B, which are type MPIAIJ, assemble them, pass them to KSPSetOperators() as first and second matrix. Since function passed into MatMFFDSetFunction() does not take a matrix as an argument, matrix AA needs to be declared in user context so that it can be used in that function to perform matrix-vector product (partially) using MatMult() as you suggested.

I have used a small test program to solve an 8x8 linear equation system and the code produces the same answers with/without the preconditioner. The iteration for this small problem, of course no improvement with the preconditioner but the residual norms with/without the preconditioner are slightly different. I suppose the preconditioner takes effect.

Now I need to think about if the applied preconditioner does the right job on the other part of matrix-vector product (non near part) for the problem I am working on.

Thank you very very much and greatly appreciate your help!

Jiannan
________________________________
From: Barry Smith <bsmith at petsc.dev>
Sent: Monday, August 22, 2022 12:42 PM
To: Tu, Jiannan <Jiannan_Tu at uml.edu>
Cc: petsc-users at mcs.anl.gov <petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] Using matrix-free with KSP


CAUTION: This email was sent from outside the UMass Lowell network.



On Aug 22, 2022, at 12:25 PM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

The matrix-vector product involves part of matrix elements explicitly as shown below so that I need to insert values into this part of the matrix.
<image.png>
Initially I wrote my own code for calculating and storing matrix A^near_mi without using any Petsc matrix and then performed matrix-vector product using above equation. The code runs but converges very slow. Apparently a preconditioned is needed.

I am not clear how Petsc can provide a preconditioner for my case.

  I am now confused. I thought you were forming Anear as a PETSc matrix and passing that to KSPSetOperators as the second matrix and so the preconditioner was being built with Anear?

  For your shell matrix vector product you should call MatMult(Anear,x,y) inside your custom routine and then add the rest of the matrix-vector product using your fast multipole products. What am I missing?

  Barry

________________________________
From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Monday, August 22, 2022 8:42 AM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] Using matrix-free with KSP


CAUTION: This email was sent from outside the UMass Lowell network.



On Aug 21, 2022, at 11:18 PM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

Barry,

You are right. With PC type none, the same error appears.

Now I use MatShell and I literally have three matrixes. A is a shell matrix, B is a preconditioner, and C is a matrix just for preallocation purpose.

First C is used to determine non-zero structure. Then use MatPreallocatorPreallocation(C, PETSC_FALSE, B) to preallocate for B. After that do insertion and assembly for B. These steps work fine.

Now the question is how to insert and assemble shell matrix A. MatPreallocatorPreallocation doesn?t work with shell matrix as indicated by an error message. MatSetValues or Mat assembly doesn?t support shell matrix either.

  A shell matrix is completely under your control. You decide on what data you want to store in it, and you provide the matrix-vector product routine.  MatPreallocatorPreallocation and MatSetValues don't work for shell matrix unless you provide code for them to work, which normally you would not do.

  I don't understand why you want to use MatSetValues for your shell matrix. If you can use MatSetValues() and just build an explicit matrix, then why would you use a shell matrix, just use MATAIJ matrix?

  Or does part of the operation inside your MatShell require a "regular" PETSc matrix? This would be fine, but it is your responsibility to design your Mat shell context to contain the "regular matrix" and your responsibility to fill up that regular matrix inside your shell context, calling whatever code you choose to write. So for example your shell context would be a C struct and one of the fields would be a regular PETSc matrix, also part of your custom matrix-vector product routine would call a regular MatMult() on the regular PETSc matrix stored inside the C struct.

  Barry



Thank you,
Jiannan

From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Friday, August 19, 2022 4:02 PM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] Using matrix-free with KSP

CAUTION: This email was sent from outside the UMass Lowell network.


  I don't know why changing the PCType could matter, can you please try with your new code but use the -pc_type none option to turn off the use of the preconditioner to see if the error still appears (I think it should).

  Also if you have a custom function that can do the matrix-vector product (not using any differencing) then you should use MatShell() and provide the matrix-vector product operation with your code.

  Barry



On Aug 19, 2022, at 11:47 AM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

Barry,

I have my own matrix-vector product but not use MatShell. I followed your previous instruction to use

    MatCreateMFFD(MPI_COMM_WORLD, PETSC_DECIDE, PETSC_DECIDE, N, N, &A);
    MatMFFDSetFunction(A, formfunction, &params);
    MatSetFromOptions(A);
    MatMFFDSetBase(A, X, NULL);

The customized matrix-vector product is implemented in formfunction() provided to MatMFFDSetFunction(). In this way, the matrix A is not used. I think that is why Petsc complains the matrix is not assembled. However, when PC type is none, the code runs without error although KSP solver doesn?t converge.

The question then becomes how to link A to matrix-free matrix-vector multiplication. And where to put MatShell?

Thank you,
Jiannan

From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Friday, August 19, 2022 10:27 AM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] Using matrix-free with KSP

CAUTION: This email was sent from outside the UMass Lowell network.





On Aug 18, 2022, at 11:41 PM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

The program calls KSPSolve just once and the error occurs after the first iteration.

I realized the matrix-vector product in my case is different from that approximating matrix-vector multiply by finite difference of function vector. It consists of two parts. One is multiplication of part of the matrix with part of vector, and another part results from so called fast multipole method. I?m not clear how to relate such calculation of matrix-vector product to the finite differencing of function vector.

  Are you using finite differencing at all in a matrix-vector product, or do you have your own custom matrix-vector product code (with MatShell)?

  Anyways the easiest thing to do is

  use the run time option -on_error_attach_debugger noxterm and run the code,

  when it stops in the debugger you can type bt (for backtrace)

  then do up a few times to move up the stack until you get to the PETSc function call that generates the error

  you can then do "call MatView(thematrixvarible,0) to have PETSc display what the matrix is and

  you can also do print *thematrixvariable to see the exact struct with all its fields

  If this doesn't tell you why the troublesome matrix is not assembled then cut and paste ALL of the output and email it

  Barry






From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Thursday, August 18, 2022 7:44 PM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] Using matrix-free with KSP

CAUTION: This email was sent from outside the UMass Lowell network.



   Is it in the first iteration of the first KSPSolve that you get this message? Or in the second call to KSPSolve?

  The error comes from a matrix vector product with your A matrix, what is that and where did it come from. The error message is not from your B matrix, that is not used in the MatMult()





On Aug 18, 2022, at 3:05 PM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

Barry,

Thanks. The method works. Assembly of actual matrix now completes quickly. The convergency is much better for the first iteration. The residual normal is in the order of 10^-10 instead of 10^-2. However, there is an error message after the first iteration: ?Object is in wrong state. Not for unassembled matrix? even though both preallocator and actual matrix have been assembled (see the code snippet below). Could you please give me some hints what?s going wrong? I use KSP type bcgs and PC type mat

Thank you,
Jiannan

-----------------------------------------------------
Solve linear equations ...
  0 KSP Residual norm 9.841179147519e-10
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Object is in wrong state
[0]PETSC ERROR: Not for unassembled matrix
[0]PETSC ERROR: See https://petsc.org/release/faq/<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Frelease%2Ffaq%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7Cc8226b4a8af040e8ca0208da845d5e8b%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637967833829887546%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=PFsyWAtTw7CvGpV9csURQXWxsSFCbZ3xGUPkE3cARgo%3D&reserved=0> for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.16.6, Mar 30, 2022
[0]PETSC ERROR: ./iesolver on a  named REXNET-WS4 by jiannantu Thu Aug 18 14:48:43 2022
[0]PETSC ERROR: Configure options --download-f2cblaslapack=yes --with-mpi-dir=/usr/local --with-fc=0 --prefix=/home/jiannantu/petsc --with-scalar-type=complex --with-64-bit-indices=1
[0]PETSC ERROR: #1 MatMult() at /home/jiannantu/petsc-3.16.6/src/mat/interface/matrix.c:2425
[0]PETSC ERROR: #2 PCApplyBAorAB() at /home/jiannantu/petsc-3.16.6/src/ksp/pc/interface/precon.c:730
[0]PETSC ERROR: #3 KSP_PCApplyBAorAB() at /home/jiannantu/petsc-3.16.6/include/petsc/private/kspimpl.h:421
[0]PETSC ERROR:  Linear equations not solved. Diverged reason: 0
#4 KSPSolve_BCGS() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/impls/bcgs/bcgs.c:87
[0]PETSC ERROR: #5 KSPSolve_Private() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itfunc.c:914
[0]PETSC ERROR: #6 KSPSolve() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itfunc.c:1086

-----------------------------------------------
    MatCreate(MPI_COMM_WORLD, &B);
    MatSetType(B, MATMPIAIJ);
    MatSetSizes(B, PETSC_DECIDE, PETSC_DECIDE, N, N);

    Mat preall;
    MatCreate(MPI_COMM_WORLD, &preall);
    MatSetType(preall, MATPREALLOCATOR);
    MatSetSizes(preall, PETSC_DECIDE, PETSC_DECIDE, N, N);

    MatMPIAIJSetPreallocation(preall, 0, d_nnz, 0, o_nnz);
    MatSetUp(preall);
MatSetOption(preall, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE);

//set values row by row
MatSetValues(preall, 1, &m, 1, nCols, matCols, INSERT_VALUES);

MatAssemblyBegin(preall, MAT_FINAL_ASSEMBLY);
MatAssemblyEnd(preall, MAT_FINAL_ASSEMBLY);

//set values for actual matrix row by row
MatSetValues(B, 1, &m, 1, nCols, matCols, INSERT_VALUES);

MatAssemblyBegin(B, MAT_FINAL_ASSEMBLY);
MatAssemblyEnd(B, MAT_FINAL_ASSEMBLY);

    KSPCreate(MPI_COMM_WORLD, &ksp);
    KSPSetOperators(ksp, A, B);
    KSPSetFromOptions(ksp);

    PC pc;
    KSPGetPC(ksp, &pc);
    PCSetFromOptions(pc);

From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Thursday, August 18, 2022 12:59 PM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] Using matrix-free with KSP

CAUTION: This email was sent from outside the UMass Lowell network.


  Yes, once you have preallocated the real matrix you can destroy the preallocation matrix whose only job is to gather the preallocation information





On Aug 18, 2022, at 12:52 PM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

Thanks, Barry.

So I need actually two matrixes, one for preallocator and one for actual matrix that can be passed to KSPSetOperators(). -mat_type preallocator option is used to speed up doing insertion into preallocator, then use MatPreallocatorPreallocate() to preallocate actual matrix and do actual insertion of values into it., right?

The code runs in parallel. Each process owns number of rows that equals to number of unknowns (that is, xm in 1D DM) it owns.

Jiannan

From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Thursday, August 18, 2022 11:37 AM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] Using matrix-free with KSP

CAUTION: This email was sent from outside the UMass Lowell network.


   The preallocator MatType matrix cannot be passed to the KSPSetOperators(), you need to create an actual matrix, for example MATAIJ and use the preallocator to set its preallocation and then fill up the MATAIJ matrix with the usual MatSetValues and pass that matrix to the KSPSetOperators.






On Aug 18, 2022, at 11:30 AM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

Hi Barry,

The MATPREALLOCATOR solved problem of slow matrix assembly. But the solver failed because of ?Matrix type preallocator does not have a multiply defined?.

I guess this is because the matrix-free is used. I am wondering how a preconditioner is applied to the matrix-vector multiply in Petsc.

Thank you,
Jiannan

[0]PETSC ERROR: No support for this operation for this object type
[0]PETSC ERROR: Matrix type preallocator does not have a multiply defined
[0]PETSC ERROR: See https://petsc.org/release/faq/<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Frelease%2Ffaq%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7Cc8226b4a8af040e8ca0208da845d5e8b%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637967833829887546%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=PFsyWAtTw7CvGpV9csURQXWxsSFCbZ3xGUPkE3cARgo%3D&reserved=0> for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.16.6, Mar 30, 2022
[0]PETSC ERROR: ./iesolver on a  named REXNET-WS4 by jiannantu Thu Aug 18 11:21:55 2022
[0]PETSC ERROR: Configure options --download-f2cblaslapack=yes --with-mpi-dir=/usr/local --with-fc=0 --prefix=/home/jiannantu/petsc --with-scalar-type=complex --with-64-bit-indices=1
[0]PETSC ERROR: #1 MatMult() at /home/jiannantu/petsc-3.16.6/src/mat/interface/matrix.c:2437
[0]PETSC ERROR: #2 PCApply_Mat() at /home/jiannantu/petsc-3.16.6/src/ksp/pc/impls/mat/pcmat.c:9
[0]PETSC ERROR: #3 PCApply() at /home/jiannantu/petsc-3.16.6/src/ksp/pc/interface/precon.c:445
[0]PETSC ERROR: #4 KSP_PCApply() at /home/jiannantu/petsc-3.16.6/include/petsc/private/kspimpl.h:382
[0]PETSC ERROR: #5 KSPInitialResidual() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itres.c:65
[0]PETSC ERROR: #6 KSPSolve_BCGS() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/impls/bcgs/bcgs.c:43
[0]PETSC ERROR: #7 KSPSolve_Private() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itfunc.c:914
[0]PETSC ERROR: #8 KSPSolve() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itfunc.c:1086

From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Thursday, August 18, 2022 8:35 AM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] Using matrix-free with KSP

CAUTION: This email was sent from outside the UMass Lowell network.


  Slow assembly is usually due to under preallocation. You can change to using

MatSetOption(B, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_TRUE);

  to detect if you are under preallocating. See https://petsc.org/main/docs/manual/mat/#sec-matsparse<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanual%2Fmat%2F%23sec-matsparse&data=05%7C01%7CJiannan_Tu%40uml.edu%7Cc8226b4a8af040e8ca0208da845d5e8b%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637967833829887546%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=xdOGjLTnr9umc%2FOmbWXPu3BFQd7kjxeYJF2RS%2F1vGMg%3D&reserved=0>. That section clearly needs some sprucing up. If it is difficult to determine good preallocation values you can use https://petsc.org/main/docs/manualpages/Mat/MATPREALLOCATOR/<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMATPREALLOCATOR%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7Cc8226b4a8af040e8ca0208da845d5e8b%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637967833829887546%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=D%2BPftcvN3AjHrPPNNiZP%2B0GiF2DJobBtqO3IIBmRDf8%3D&reserved=0> to compute the needed preallocation efficiently.

  Are you running in parallel? If so how are you determining which rows of entries to compute on each MPI rank? You will want most of the rows to be computed on the rank where the values are stored, you can determine this with https://petsc.org/main/docs/manualpages/Mat/MatGetOwnershipRange.html<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMatGetOwnershipRange.html&data=05%7C01%7CJiannan_Tu%40uml.edu%7Cc8226b4a8af040e8ca0208da845d5e8b%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637967833829887546%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=dmqAfAFRE%2FTvZ4b2yb4btNZe48pZlL0DbLpOLLRqOd8%3D&reserved=0>

  Barry

















On Aug 18, 2022, at 12:50 AM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

I implemented the preconditioner. The solver converges quickly when the problem size is small. But when the size increases, say to 100k unknowns, the code hangs at assembly of the preconditioner matrix. The function call sequence is like this

   MatCreate(MPI_COMM_WORLD, &B);
    MatSetType(B, MATMPIAIJ);
    MatSetSizes(B, PETSC_DECIDE, PETSC_DECIDE, N, N);
    MatSetFromOptions(B);

    // The number of diagonal and off diagonal non zeros only can be determined run-time.
    MatMPIAIJSetPreallocation(B, 0, d_nnz, 0, o_nnz);
    MatSetOption(B, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE);

    //here insert values row by row.
MatSetValues(B, 1, &m, nn, nCols, matCols, INSERT_VALUES);

MatAssemblyBegin(B, MAT_FINAL_ASSEMBLY);
MatAssemblyEnd(B, MAT_FINAL_ASSEMBLY);

Could you please tell me what I have done wrong?

Thank you,
Jiannan

From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Tuesday, August 16, 2022 2:00 PM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] Using matrix-free with KSP

CAUTION: This email was sent from outside the UMass Lowell network.


   Create another matrix B that contains "Selected parts of the matrix can be pre-calculated and stored to serve as the preconditioner." When you call KSPSetOperators() pass in B as the second matrix argument.

   Now if literally application of B is the preconditioner you want to use, then simply use PCMAT as the preconditioner, and it will apply B each time it needs to apply the preconditioner. With this, you do not even need a PCShell.

   If your preconditioner uses B to construct your own specific data structure needed to apply your preconditioner, then you can use PCShellSetSetUp() and have your routine call PCGetOperators() to pull out the B matrix and use it to construct your preconditioner.

   If you want PETSc to construct a standard preconditioner from B, then you don't need the PCShell; you simply pass in the B, as above, and use -pc_type type to set the preconditioner you want to use.

   Barry









On Aug 16, 2022, at 12:31 PM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

Barry,

Thank you very much for your instructions. I am sorry I may not ask clearer questions.

I don't use SNES. What I am trying to solve is a very large linear equation system with dense and ill-conditioned matrix. Selected parts of the matrix can be pre-calculated and stored to serve as the preconditioner. My question is how I can apply this matrix as the preconditioner.

Here is the part of code from my linear equations solver. MatrixFreePreconditioner is supposed to be the routine to provide the preconditioner. But I don't understand how to use it. Should I use SNESSetJacobian() even if this is a linear problem?

    MatCreateMFFD(MPI_COMM_WORLD, PETSC_DECIDE, PETSC_DECIDE, N, N, &A);
    MatMFFDSetFunction(A, formfunction, &params);
    MatSetFromOptions(A);
    MatMFFDSetBase(A, X, NULL);

    KSPCreate(MPI_COMM_WORLD, &ksp);
    KSPSetOperators(ksp, A, A);
    KSPSetFromOptions(ksp);

    PC pc;
    KSPGetPC(ksp,&pc);
    PCSetType(pc,PCSHELL);
    PCShellSetApply(pc,MatrixFreePreconditioner);

Thank you,
Jiannan

________________________________
From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Tuesday, August 16, 2022 10:10 AM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] Using matrix-free with KSP

CAUTION: This email was sent from outside the UMass Lowell network.


  I don't fully understand your question so I will answer questions that I do know the answer to :-)

  The function you provide to PCShellSetApply() applies the preconditioner to the input vector and puts the result in the output vector.  This input vector changes at every application of the preconditioner so should not be used in constructing the preconditioner.

  If you are using the standard PETSc matrix-free matrix-vector product via finite difference, -snes_mf_operator or via MatCreateSNESMF() or you are using a MatShell, the matrix-vector product routines input changes for each multiply and should not be used in constructing the preconditioner.

  The function you provide with SNESSetJacobian() is where you can compute anything you need to help build your preconditioner. The input vector to that function is the location at which the Jacobian is needed and is what you should use to build your approximate/part of Jacobian. You can store your approximate/part of Jacobian that you need to compute in the pmat and then in a PCShellSetUp() function that you provide you will compute what you will need latter to apply the preconditioner in your PCShellSetApply() function. Note that the values you put in the pmat do not need to be the full Jacobian; they can be anything you want since this pmat is not used to apply the matrix vector product.

  If this is unclear, feel free to ask additional questions.

  Barry









On Aug 16, 2022, at 9:40 AM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

I am trying to apply a user-defined preconditioner to speed up the convergency of matrix-free KSP solver. The user's manual provides an example of how to call the user-defined preconditioner routine. The routine has PC, input Vec and output Vec as arguments. I want to use part of the Jacobian as preconditioner matrix, the elements of which can be calculated using input Vec. My question is what the role of output Vec from the routine is in matrix-vector product? Or I just have the preconditioner matrix elements calculated in the routine and then Petsc handles applying the preconditioner to matrix-vector product? I used PCSetType(pc, PCSHELL) and PCShellSetApply.

Thank you and appreciate your help!

Jiannan
________________________________
From: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Sent: Wednesday, July 6, 2022 2:34 PM
To: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Cc: Jed Brown <jed at jedbrown.org<mailto:jed at jedbrown.org>>; petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] Using matrix-free with KSP

Barry,

VecScatterCreateToAll solves the problem! The code now gives the correct answer with multiple processes.

Thank you all so much!

Jiannan
________________________________
From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Wednesday, July 6, 2022 2:08 PM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Cc: Jed Brown <jed at jedbrown.org<mailto:jed at jedbrown.org>>; petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] Using matrix-free with KSP


  So your operator is a "dense" operator in that its matrix representation would be essentially dense. In that case, you can use specialized routines to do this efficiently. You can use VecScatterCreateToAll() and then VecScatterBegin/End to do the communication.

  Barry










On Jul 6, 2022, at 1:39 PM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>  wrote:

Jed, thank you very much for the reply.

I'm not sure GlobaltoLocal will work. Say the solution vector is of length 10. Two processes then each process can see 5 elements of the vector. For A x = b (10 equations), each process performs matrix-vector product for five equations. And for each equation i , sum_j A_ij *  x_j = b_i requires the process has access to all 10 unknows x_i (i=0,1,...,9).

Can VecScatter and PetscSF make the entire vector accessible to each process? I am reading the manual and trying to understand how VecScatter and PetscSF work.

Jiannan
________________________________
From: Jed Brown <jed at jedbrown.org<mailto:jed at jedbrown.org>>
Sent: Wednesday, July 6, 2022 10:09 AM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>; Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] Using matrix-free with KSP

You'll usually have a GlobalToLocal operation for each rank to get the halo data it needs, then either a LocalToGlobal to collect the result (e.g., finite element methods) or the local compute will write directly into the owned portion of the global vector. If you're doing the communication "raw", then you may find VecScatter or PetscSF useful to perform the necessary communication.

"Tu, Jiannan" <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> writes:

> Hi Barry,
>
> Following your instructions I implemented the matrix-free method to solve a large linear equation system resulted from a surface integration equation. The KSP solver works fine for single process, but it is problematic with multiple processes. The problem is that each process only can access its own part of solution vector so that each process only conducts part of matrix-vector multiplication. MPI can be used to assemble these partial matrix-vector multiplication together.  Does Petsc provide any ways to implement multi-process matrix-free method?
>
> Thanks,
> Jiannan
> ________________________________
> From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
> Sent: Tuesday, May 24, 2022 2:12 PM
> To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
> Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
> Subject: Re: [petsc-users] Using matrix-free with KSP
>
> This e-mail originated from outside the UMass Lowell network.
> ________________________________
>
>    You can use MatCreateMFFD https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMatCreateMFFD%2F&amp;data=05%7C01%7CJiannan_Tu%40uml.edu%7Cab0859d8d154471590bc08da5f5918d6%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637927133525745809%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=A7wqegTfh94No5BpDiWLK3VxOuR44U2wlWHVm2k7l60%3D&amp;reserved=0<https://urldefense.com/v3/__https://petsc.org/main/docs/manualpages/Mat/MatCreateMFFD/__;!!PVKG_VDCxu5g!stxq2NWGVjUihvHBQC9Rlk0aHVoowcy0PTQYMcQxQ4DqOEC05KSw6TmstSXKckiUgelHzy4ue-d10-zlXkw$><https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMatCreateMFFD%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7Cc8226b4a8af040e8ca0208da845d5e8b%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637967833829887546%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=pQ3eVZvOw0WEJdHYm%2FP3leoX3tBNKpgLDQo5Xm10akw%3D&reserved=0> MatMFFDSetFunction MatSetFromOptions MatMFFDSetBase and provide the matrix to KSP. Note you will need to use -pc_type none or provide your own custom preconditioner withhttps://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FPC%2FPCSHELL%2F&amp;data=05%7C01%7CJiannan_Tu%40uml.edu%7Cab0859d8d154471590bc08da5f5918d6%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637927133525745809%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=Gb%2F9uFG3jp%2FbfWaSh2cSEQLItHS9CuLwarzN0KcNiJY%3D&amp;reserved=0<https://urldefense.com/v3/__https://petsc.org/main/docs/manualpages/PC/PCSHELL/__;!!PVKG_VDCxu5g!stxq2NWGVjUihvHBQC9Rlk0aHVoowcy0PTQYMcQxQ4DqOEC05KSw6TmstSXKckiUgelHzy4ue-d1cA4fKGw$><https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FPC%2FPCSHELL%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7Cc8226b4a8af040e8ca0208da845d5e8b%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637967833829887546%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=OwryOVq7EBof9oYaYJmN3klaLNwxtAxcZBgj44BF%2BJo%3D&reserved=0>
>
>
>
> On May 24, 2022, at 1:21 PM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu><mailto:Jiannan_Tu at uml.edu>> wrote:
>
> I want to use a matrix-free matrix to solve a large linear equation system because the matrix is too large to be stored. Petsc user manual describes matrix-free method for SNES with examples. The matrix-free matrices section explains how to set up such a matrix, but I can't find any example of matrix-free method with KSP. I am wondering how to apply the method to KSP iterative solver in parallel.
>
> I greatly appreciate your help for me to understand this topic.
>
> Jiannan Tu

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220822/b39f8c85/attachment-0001.html>

From rafel.amer at upc.edu  Mon Aug 22 13:31:57 2022
From: rafel.amer at upc.edu (Rafel Amer Ramon)
Date: Mon, 22 Aug 2022 20:31:57 +0200
Subject: [petsc-users] PETSc on fedora 36
Message-ID: <aa3cbfe2-2ccc-93cc-6004-5c2ae6429047@upc.edu>

An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220822/d6a1deb1/attachment.html>

From bsmith at petsc.dev  Mon Aug 22 13:36:43 2022
From: bsmith at petsc.dev (Barry Smith)
Date: Mon, 22 Aug 2022 14:36:43 -0400
Subject: [petsc-users] PETSc on fedora 36
In-Reply-To: <aa3cbfe2-2ccc-93cc-6004-5c2ae6429047@upc.edu>
References: <aa3cbfe2-2ccc-93cc-6004-5c2ae6429047@upc.edu>
Message-ID: <358BFF34-D941-42AB-8648-871186EF64DB@petsc.dev>


  Are you sure the mpirun you are using matches the mpi that PETSc was built with? 

> On Aug 22, 2022, at 2:31 PM, Rafel Amer Ramon <rafel.amer at upc.edu> wrote:
> 
> Hi,
> 
> I have installed the following packages on fedora 36
> 
> ~# rpm -qa | grep petsc
> petsc64-3.16.4-3.fc36.x86_64
> petsc-3.16.4-3.fc36.x86_64
> petsc-openmpi-3.16.4-3.fc36.x86_64
> petsc-openmpi-devel-3.16.4-3.fc36.x86_64
> python3-petsc-openmpi-3.16.4-3.fc36.x86_64
> petsc-devel-3.16.4-3.fc36.x86_64
> petsc64-devel-3.16.4-3.fc36.x86_64
> 
> ~# rpm -qa | grep openmpi
> openmpi-4.1.2-3.fc36.x86_64
> ptscotch-openmpi-6.1.2-2.fc36.x86_64
> scalapack-openmpi-2.1.0-11.fc36.x86_64
> openmpi-devel-4.1.2-3.fc36.x86_64
> MUMPS-openmpi-5.4.1-2.fc36.x86_64
> superlu_dist-openmpi-6.1.1-9.fc36.x86_64
> hdf5-openmpi-1.12.1-5.fc36.x86_64
> hypre-openmpi-2.18.2-6.fc36.x86_64
> petsc-openmpi-3.16.4-3.fc36.x86_64
> fftw-openmpi-libs-double-3.3.10-2.fc36.x86_64
> fftw-openmpi-libs-long-3.3.10-2.fc36.x86_64
> fftw-openmpi-libs-single-3.3.10-2.fc36.x86_64
> fftw-openmpi-libs-3.3.10-2.fc36.x86_64
> fftw-openmpi-devel-3.3.10-2.fc36.x86_64
> petsc-openmpi-devel-3.16.4-3.fc36.x86_64
> python3-petsc-openmpi-3.16.4-3.fc36.x86_64
> scalapack-openmpi-devel-2.1.0-11.fc36.x86_64
> python3-openmpi-4.1.2-3.fc36.x86_64
> hdf5-openmpi-devel-1.12.1-5.fc36.x86_64
> cgnslib-openmpi-4.2.0-6.fc36.x86_64
> cgnslib-openmpi-devel-4.2.0-6.fc36.x86_64
> 
> and I try to compile and run the program written in the file e.c
> 
> ~# cat e.c
> #include <petsc.h>
> 
> int main(int argc, char **argv) {
>   PetscErrorCode ierr;
>   PetscMPIInt    rank;
> 
>   ierr = PetscInitialize(&argc,&argv,NULL,
>       "Compute e in parallel with PETSc.\n\n"); if (ierr) return ierr;
>   ierr = MPI_Comm_rank(PETSC_COMM_WORLD,&rank); CHKERRQ(ierr);
>   ierr = PetscPrintf(PETSC_COMM_WORLD,"My rank is %d\n",rank); CHKERRQ(ierr);
> 
>   
>   return PetscFinalize();
> }
> 
> I compile it with the command
> 
> ~# mpicc -o e e.c -I/usr/include/petsc64 -lpetsc64
> 
> and I run it with 
> 
> ~# mpirun -np 8 ./e
> My rank is 0
> My rank is 0
> My rank is 0
> My rank is 0
> My rank is 0
> My rank is 0
> My rank is 0
> My rank is 0
> 
> but all my process get rank 0.
> 
> Do you know what I'm doing wrong?
> 
> Thank you!!
> 
> Best regards,
> 
> Rafel Amer

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220822/28a12c95/attachment.html>

From bsmith at petsc.dev  Mon Aug 22 13:39:42 2022
From: bsmith at petsc.dev (Barry Smith)
Date: Mon, 22 Aug 2022 14:39:42 -0400
Subject: [petsc-users] Using matrix-free with KSP
In-Reply-To: <BY5PR02MB64715D7FE585A7F88092E04DE7719@BY5PR02MB6471.namprd02.prod.outlook.com>
References: <BY5PR02MB6471969F80FA5F732FC74D87E7D79@BY5PR02MB6471.namprd02.prod.outlook.com>
	<B55CFC2C-6364-4B78-8E99-9D53700743B8@petsc.dev>
	<BY5PR02MB6471BDF90783FE3A0EA176CBE7809@BY5PR02MB6471.namprd02.prod.outlook.com>
	<87iloa49kt.fsf@jedbrown.org>
	<BY5PR02MB6471C0ED04769D02046F68ACE7809@BY5PR02MB6471.namprd02.prod.outlook.com>
	<00EBE4B8-22E5-408F-B56D-14226A278D7E@petsc.dev>
	<BY5PR02MB6471E386057DDDE98AD05DFAE7809@BY5PR02MB6471.namprd02.prod.outlook.com>
	<BY5PR02MB647140084C7E7CDDB6BC2A11E76B9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<A6629DE1-5A7F-414E-B164-A69877158675@petsc.dev>
	<BY5PR02MB6471AB1C004920DA1FC9B43EE76B9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<103146FD-FF28-47CE-B661-F2B521BF0E7A@petsc.dev>
	<DM6PR02MB6478B2FB8CB1BE757CE7B003E76D9@DM6PR02MB6478.namprd02.prod.outlook.com>
	<4C00F4B8-BB42-4932-BCD7-527E5FB39BFF@petsc.dev>
	<BY5PR02MB64710E70758E431108D66956E76D9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<D0EB13D8-C5A6-4AE7-AFDE-2CDCCACBAF73@petsc.dev>
	<BY5PR02MB6471D4B05874DECFDDA13FFCE76D9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<367B124C-88BB-4889-A4AF-45A1D473078D@petsc.dev>
	<BY5PR02MB6471F2C058B84DE092567731E76D9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<944DDEED-016D-43B2-A5A6-7BF2E8E41FA5@petsc.dev>
	<BY5PR02MB6471CD0236F03D20763A3B64E76C9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<B4CF5E5B-984D-4EC1-AB77-1FD08C3FBBCF@petsc.dev>
	<BY5PR02MB6471BA72871419781B875967E76C9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<3D179AEA-AD05-4B46-8DDC-9E126741C247@petsc.dev>
	<BY5PR02MB6471560EA70C6FAE826DDF27E7719@BY5PR02MB6471.namprd02.prod.outlook.com>
	<1C67C8DB-0653-4588-8137-E0865E0F2098@petsc.dev>
	<BY5PR02MB6471C5EAE18AB9B114E76232E7719@BY5PR02MB6471.namprd02.prod.outlook.com>
	<FAA1668F-08C1-4BAD-89B4-784D8AD99C36@petsc.dev>
	<BY5PR02MB64715D7FE585A7F88092E04DE7719@BY5PR02MB6471.namprd02.prod.outlook.com>
Message-ID: <F4C76705-B402-4F8B-A42C-DDBE95574065@petsc.dev>



> On Aug 22, 2022, at 1:37 PM, Tu, Jiannan <Jiannan_Tu at uml.edu> wrote:
> 
> Barry,
> 
> Yes, you are right, Anear is the preconditioner that needs to be passed to KSPSetOperators() as the second matrix. 
> 
> I think I have figured out how to provide a preconditioner for matrix-free method. I was confused by matrix A created by MatCreateMFFD(), which is type MFFD. I thought I have to use it as the first matrix passed to KSPSetOperators(). Since it was not assembled (and can't) so that Petsc gave error message "matrix unassembled". In fact, I should ignore A, but use other two matrixes AA and B, which are type MPIAIJ, assemble them, pass them to KSPSetOperators() as first and second matrix. Since function passed into MatMFFDSetFunction() does not take a matrix as an argument, matrix AA needs to be declared in user context so that it can be used in that function to perform matrix-vector product (partially) using MatMult() as you suggested. 
> 
   I think your AA should be the shell matrix, and the B should be Anear. There should be no MatMFFD. Inside the AA shell matrix mat-mult you call MatMult( on the Anear put Anear in the Shell matrix context also) and then add the part of the multiply coming from the far field.

 
> I have used a small test program to solve an 8x8 linear equation system and the code produces the same answers with/without the preconditioner. The iteration for this small problem, of course no improvement with the preconditioner but the residual norms with/without the preconditioner are slightly different. I suppose the preconditioner takes effect.
> 
> Now I need to think about if the applied preconditioner does the right job on the other part of matrix-vector product (non near part) for the problem I am working on.
> 
> Thank you very very much and greatly appreciate your help!
> 
> Jiannan
> From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>>
> Sent: Monday, August 22, 2022 12:42 PM
> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
> Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>>
> Subject: Re: [petsc-users] Using matrix-free with KSP
>  
> CAUTION: This email was sent from outside the UMass Lowell network.
> 
> 
> 
>> On Aug 22, 2022, at 12:25 PM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> wrote:
>> 
>> The matrix-vector product involves part of matrix elements explicitly as shown below so that I need to insert values into this part of the matrix.
>> <image.png>
>> Initially I wrote my own code for calculating and storing matrix A^near_mi without using any Petsc matrix and then performed matrix-vector product using above equation. The code runs but converges very slow. Apparently a preconditioned is needed. 
>> 
>> I am not clear how Petsc can provide a preconditioner for my case.
> 
>   I am now confused. I thought you were forming Anear as a PETSc matrix and passing that to KSPSetOperators as the second matrix and so the preconditioner was being built with Anear?
> 
>   For your shell matrix vector product you should call MatMult(Anear,x,y) inside your custom routine and then add the rest of the matrix-vector product using your fast multipole products. What am I missing?
> 
>   Barry
> 
>> 
>> From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>>
>> Sent: Monday, August 22, 2022 8:42 AM
>> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
>> Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>>
>> Subject: Re: [petsc-users] Using matrix-free with KSP
>>  
>> CAUTION: This email was sent from outside the UMass Lowell network.
>> 
>> 
>> 
>>> On Aug 21, 2022, at 11:18 PM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> wrote:
>>> 
>>> Barry,
>>>  
>>> You are right. With PC type none, the same error appears.
>>>  
>>> Now I use MatShell and I literally have three matrixes. A is a shell matrix, B is a preconditioner, and C is a matrix just for preallocation purpose.
>>>  
>>> First C is used to determine non-zero structure. Then use MatPreallocatorPreallocation(C, PETSC_FALSE, B) to preallocate for B. After that do insertion and assembly for B. These steps work fine.
>>>  
>>> Now the question is how to insert and assemble shell matrix A. MatPreallocatorPreallocation doesn?t work with shell matrix as indicated by an error message. MatSetValues or Mat assembly doesn?t support shell matrix either. 
>> 
>>   A shell matrix is completely under your control. You decide on what data you want to store in it, and you provide the matrix-vector product routine.  MatPreallocatorPreallocation and MatSetValues don't work for shell matrix unless you provide code for them to work, which normally you would not do.
>> 
>>   I don't understand why you want to use MatSetValues for your shell matrix. If you can use MatSetValues() and just build an explicit matrix, then why would you use a shell matrix, just use MATAIJ matrix? 
>> 
>>   Or does part of the operation inside your MatShell require a "regular" PETSc matrix? This would be fine, but it is your responsibility to design your Mat shell context to contain the "regular matrix" and your responsibility to fill up that regular matrix inside your shell context, calling whatever code you choose to write. So for example your shell context would be a C struct and one of the fields would be a regular PETSc matrix, also part of your custom matrix-vector product routine would call a regular MatMult() on the regular PETSc matrix stored inside the C struct.
>> 
>>   Barry
>> 
>> 
>>>  
>>> Thank you,
>>> Jiannan
>>>  
>>> From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>> 
>>> Sent: Friday, August 19, 2022 4:02 PM
>>> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
>>> Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>
>>> Subject: Re: [petsc-users] Using matrix-free with KSP
>>>  
>>> CAUTION: This email was sent from outside the UMass Lowell network.
>>>  
>>>  
>>>   I don't know why changing the PCType could matter, can you please try with your new code but use the -pc_type none option to turn off the use of the preconditioner to see if the error still appears (I think it should). 
>>>  
>>>   Also if you have a custom function that can do the matrix-vector product (not using any differencing) then you should use MatShell() and provide the matrix-vector product operation with your code.
>>>  
>>>   Barry
>>>  
>>> 
>>> 
>>> On Aug 19, 2022, at 11:47 AM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> wrote:
>>>  
>>> Barry,
>>>  
>>> I have my own matrix-vector product but not use MatShell. I followed your previous instruction to use
>>>  
>>>     MatCreateMFFD(MPI_COMM_WORLD, PETSC_DECIDE, PETSC_DECIDE, N, N, &A);
>>>     MatMFFDSetFunction(A, formfunction, &params);
>>>     MatSetFromOptions(A);
>>>     MatMFFDSetBase(A, X, NULL);
>>>  
>>> The customized matrix-vector product is implemented in formfunction() provided to MatMFFDSetFunction(). In this way, the matrix A is not used. I think that is why Petsc complains the matrix is not assembled. However, when PC type is none, the code runs without error although KSP solver doesn?t converge.
>>>  
>>> The question then becomes how to link A to matrix-free matrix-vector multiplication. And where to put MatShell?
>>>  
>>> Thank you,
>>> Jiannan
>>>  
>>> From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>> 
>>> Sent: Friday, August 19, 2022 10:27 AM
>>> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
>>> Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>
>>> Subject: Re: [petsc-users] Using matrix-free with KSP
>>>  
>>> CAUTION: This email was sent from outside the UMass Lowell network.
>>>  
>>>  
>>> 
>>> 
>>> 
>>> On Aug 18, 2022, at 11:41 PM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> wrote:
>>>  
>>> The program calls KSPSolve just once and the error occurs after the first iteration.
>>>  
>>> I realized the matrix-vector product in my case is different from that approximating matrix-vector multiply by finite difference of function vector. It consists of two parts. One is multiplication of part of the matrix with part of vector, and another part results from so called fast multipole method. I?m not clear how to relate such calculation of matrix-vector product to the finite differencing of function vector.
>>>  
>>>   Are you using finite differencing at all in a matrix-vector product, or do you have your own custom matrix-vector product code (with MatShell)? 
>>>  
>>>   Anyways the easiest thing to do is 
>>>  
>>>   use the run time option -on_error_attach_debugger noxterm and run the code, 
>>>  
>>>   when it stops in the debugger you can type bt (for backtrace)
>>>  
>>>   then do up a few times to move up the stack until you get to the PETSc function call that generates the error
>>>  
>>>   you can then do "call MatView(thematrixvarible,0) to have PETSc display what the matrix is and
>>>  
>>>   you can also do print *thematrixvariable to see the exact struct with all its fields
>>>  
>>>   If this doesn't tell you why the troublesome matrix is not assembled then cut and paste ALL of the output and email it
>>>  
>>>   Barry
>>> 
>>> 
>>> 
>>>  
>>>  
>>>  
>>> From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>> 
>>> Sent: Thursday, August 18, 2022 7:44 PM
>>> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
>>> Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>
>>> Subject: Re: [petsc-users] Using matrix-free with KSP
>>>  
>>> CAUTION: This email was sent from outside the UMass Lowell network.
>>>  
>>>  
>>>  
>>>    Is it in the first iteration of the first KSPSolve that you get this message? Or in the second call to KSPSolve? 
>>>  
>>>   The error comes from a matrix vector product with your A matrix, what is that and where did it come from. The error message is not from your B matrix, that is not used in the MatMult()
>>>  
>>> 
>>> 
>>> 
>>> 
>>> On Aug 18, 2022, at 3:05 PM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> wrote:
>>>  
>>> Barry, 
>>>  
>>> Thanks. The method works. Assembly of actual matrix now completes quickly. The convergency is much better for the first iteration. The residual normal is in the order of 10^-10 instead of 10^-2. However, there is an error message after the first iteration: ?Object is in wrong state. Not for unassembled matrix? even though both preallocator and actual matrix have been assembled (see the code snippet below). Could you please give me some hints what?s going wrong? I use KSP type bcgs and PC type mat
>>>  
>>> Thank you,
>>> Jiannan
>>>  
>>> -----------------------------------------------------
>>> Solve linear equations ...
>>>   0 KSP Residual norm 9.841179147519e-10
>>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>>> [0]PETSC ERROR: Object is in wrong state
>>> [0]PETSC ERROR: Not for unassembled matrix
>>> [0]PETSC ERROR: See https://petsc.org/release/faq/ <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Frelease%2Ffaq%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7Cc8226b4a8af040e8ca0208da845d5e8b%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637967833829887546%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=PFsyWAtTw7CvGpV9csURQXWxsSFCbZ3xGUPkE3cARgo%3D&reserved=0> for trouble shooting.
>>> [0]PETSC ERROR: Petsc Release Version 3.16.6, Mar 30, 2022
>>> [0]PETSC ERROR: ./iesolver on a  named REXNET-WS4 by jiannantu Thu Aug 18 14:48:43 2022
>>> [0]PETSC ERROR: Configure options --download-f2cblaslapack=yes --with-mpi-dir=/usr/local --with-fc=0 --prefix=/home/jiannantu/petsc --with-scalar-type=complex --with-64-bit-indices=1
>>> [0]PETSC ERROR: #1 MatMult() at /home/jiannantu/petsc-3.16.6/src/mat/interface/matrix.c:2425
>>> [0]PETSC ERROR: #2 PCApplyBAorAB() at /home/jiannantu/petsc-3.16.6/src/ksp/pc/interface/precon.c:730
>>> [0]PETSC ERROR: #3 KSP_PCApplyBAorAB() at /home/jiannantu/petsc-3.16.6/include/petsc/private/kspimpl.h:421
>>> [0]PETSC ERROR:  Linear equations not solved. Diverged reason: 0
>>> #4 KSPSolve_BCGS() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/impls/bcgs/bcgs.c:87
>>> [0]PETSC ERROR: #5 KSPSolve_Private() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itfunc.c:914
>>> [0]PETSC ERROR: #6 KSPSolve() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itfunc.c:1086
>>>  
>>> -----------------------------------------------
>>>     MatCreate(MPI_COMM_WORLD, &B);
>>>     MatSetType(B, MATMPIAIJ);
>>>     MatSetSizes(B, PETSC_DECIDE, PETSC_DECIDE, N, N);
>>>  
>>>     Mat preall;
>>>     MatCreate(MPI_COMM_WORLD, &preall);
>>>     MatSetType(preall, MATPREALLOCATOR);
>>>     MatSetSizes(preall, PETSC_DECIDE, PETSC_DECIDE, N, N);
>>>  
>>>     MatMPIAIJSetPreallocation(preall, 0, d_nnz, 0, o_nnz);
>>>     MatSetUp(preall);
>>> MatSetOption(preall, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE);
>>>  
>>> //set values row by row
>>> MatSetValues(preall, 1, &m, 1, nCols, matCols, INSERT_VALUES);
>>>  
>>> MatAssemblyBegin(preall, MAT_FINAL_ASSEMBLY);
>>> MatAssemblyEnd(preall, MAT_FINAL_ASSEMBLY);
>>>  
>>> //set values for actual matrix row by row
>>> MatSetValues(B, 1, &m, 1, nCols, matCols, INSERT_VALUES);
>>>  
>>> MatAssemblyBegin(B, MAT_FINAL_ASSEMBLY);
>>> MatAssemblyEnd(B, MAT_FINAL_ASSEMBLY);
>>>  
>>>     KSPCreate(MPI_COMM_WORLD, &ksp);
>>>     KSPSetOperators(ksp, A, B);
>>>     KSPSetFromOptions(ksp);
>>>  
>>>     PC pc;
>>>     KSPGetPC(ksp, &pc);
>>>     PCSetFromOptions(pc);
>>>  
>>> From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>> 
>>> Sent: Thursday, August 18, 2022 12:59 PM
>>> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
>>> Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>
>>> Subject: Re: [petsc-users] Using matrix-free with KSP
>>>  
>>> CAUTION: This email was sent from outside the UMass Lowell network.
>>>  
>>>  
>>>   Yes, once you have preallocated the real matrix you can destroy the preallocation matrix whose only job is to gather the preallocation information
>>> 
>>> 
>>> 
>>> 
>>> 
>>> On Aug 18, 2022, at 12:52 PM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> wrote:
>>>  
>>> Thanks, Barry.
>>>  
>>> So I need actually two matrixes, one for preallocator and one for actual matrix that can be passed to KSPSetOperators(). -mat_type preallocator option is used to speed up doing insertion into preallocator, then use MatPreallocatorPreallocate() to preallocate actual matrix and do actual insertion of values into it., right?
>>>  
>>> The code runs in parallel. Each process owns number of rows that equals to number of unknowns (that is, xm in 1D DM) it owns.
>>>  
>>> Jiannan
>>>  
>>> From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>> 
>>> Sent: Thursday, August 18, 2022 11:37 AM
>>> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
>>> Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>
>>> Subject: Re: [petsc-users] Using matrix-free with KSP
>>>  
>>> CAUTION: This email was sent from outside the UMass Lowell network.
>>>  
>>>  
>>>    The preallocator MatType matrix cannot be passed to the KSPSetOperators(), you need to create an actual matrix, for example MATAIJ and use the preallocator to set its preallocation and then fill up the MATAIJ matrix with the usual MatSetValues and pass that matrix to the KSPSetOperators.
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> On Aug 18, 2022, at 11:30 AM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> wrote:
>>>  
>>> Hi Barry,
>>>  
>>> The MATPREALLOCATOR solved problem of slow matrix assembly. But the solver failed because of ?Matrix type preallocator does not have a multiply defined?.
>>>  
>>> I guess this is because the matrix-free is used. I am wondering how a preconditioner is applied to the matrix-vector multiply in Petsc.
>>>  
>>> Thank you,
>>> Jiannan
>>>  
>>> [0]PETSC ERROR: No support for this operation for this object type
>>> [0]PETSC ERROR: Matrix type preallocator does not have a multiply defined
>>> [0]PETSC ERROR: See https://petsc.org/release/faq/ <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Frelease%2Ffaq%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7Cc8226b4a8af040e8ca0208da845d5e8b%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637967833829887546%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=PFsyWAtTw7CvGpV9csURQXWxsSFCbZ3xGUPkE3cARgo%3D&reserved=0> for trouble shooting.
>>> [0]PETSC ERROR: Petsc Release Version 3.16.6, Mar 30, 2022
>>> [0]PETSC ERROR: ./iesolver on a  named REXNET-WS4 by jiannantu Thu Aug 18 11:21:55 2022
>>> [0]PETSC ERROR: Configure options --download-f2cblaslapack=yes --with-mpi-dir=/usr/local --with-fc=0 --prefix=/home/jiannantu/petsc --with-scalar-type=complex --with-64-bit-indices=1
>>> [0]PETSC ERROR: #1 MatMult() at /home/jiannantu/petsc-3.16.6/src/mat/interface/matrix.c:2437
>>> [0]PETSC ERROR: #2 PCApply_Mat() at /home/jiannantu/petsc-3.16.6/src/ksp/pc/impls/mat/pcmat.c:9
>>> [0]PETSC ERROR: #3 PCApply() at /home/jiannantu/petsc-3.16.6/src/ksp/pc/interface/precon.c:445
>>> [0]PETSC ERROR: #4 KSP_PCApply() at /home/jiannantu/petsc-3.16.6/include/petsc/private/kspimpl.h:382
>>> [0]PETSC ERROR: #5 KSPInitialResidual() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itres.c:65
>>> [0]PETSC ERROR: #6 KSPSolve_BCGS() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/impls/bcgs/bcgs.c:43
>>> [0]PETSC ERROR: #7 KSPSolve_Private() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itfunc.c:914
>>> [0]PETSC ERROR: #8 KSPSolve() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itfunc.c:1086
>>>  
>>> From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>> 
>>> Sent: Thursday, August 18, 2022 8:35 AM
>>> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
>>> Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>
>>> Subject: Re: [petsc-users] Using matrix-free with KSP
>>>  
>>> CAUTION: This email was sent from outside the UMass Lowell network.
>>>  
>>>  
>>>   Slow assembly is usually due to under preallocation. You can change to using 
>>>  
>>> MatSetOption(B, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_TRUE);
>>>  
>>>   to detect if you are under preallocating. See https://petsc.org/main/docs/manual/mat/#sec-matsparse <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanual%2Fmat%2F%23sec-matsparse&data=05%7C01%7CJiannan_Tu%40uml.edu%7Cc8226b4a8af040e8ca0208da845d5e8b%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637967833829887546%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=xdOGjLTnr9umc%2FOmbWXPu3BFQd7kjxeYJF2RS%2F1vGMg%3D&reserved=0>. That section clearly needs some sprucing up. If it is difficult to determine good preallocation values you can use https://petsc.org/main/docs/manualpages/Mat/MATPREALLOCATOR/ <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMATPREALLOCATOR%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7Cc8226b4a8af040e8ca0208da845d5e8b%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637967833829887546%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=D%2BPftcvN3AjHrPPNNiZP%2B0GiF2DJobBtqO3IIBmRDf8%3D&reserved=0> to compute the needed preallocation efficiently.
>>>  
>>>   Are you running in parallel? If so how are you determining which rows of entries to compute on each MPI rank? You will want most of the rows to be computed on the rank where the values are stored, you can determine this with https://petsc.org/main/docs/manualpages/Mat/MatGetOwnershipRange.html <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMatGetOwnershipRange.html&data=05%7C01%7CJiannan_Tu%40uml.edu%7Cc8226b4a8af040e8ca0208da845d5e8b%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637967833829887546%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=dmqAfAFRE%2FTvZ4b2yb4btNZe48pZlL0DbLpOLLRqOd8%3D&reserved=0>
>>>  
>>>   Barry
>>>  
>>>  
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>>  
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> On Aug 18, 2022, at 12:50 AM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> wrote:
>>>  
>>> I implemented the preconditioner. The solver converges quickly when the problem size is small. But when the size increases, say to 100k unknowns, the code hangs at assembly of the preconditioner matrix. The function call sequence is like this
>>>  
>>>    MatCreate(MPI_COMM_WORLD, &B);
>>>     MatSetType(B, MATMPIAIJ);
>>>     MatSetSizes(B, PETSC_DECIDE, PETSC_DECIDE, N, N);
>>>     MatSetFromOptions(B);
>>>  
>>>     // The number of diagonal and off diagonal non zeros only can be determined run-time.
>>>     MatMPIAIJSetPreallocation(B, 0, d_nnz, 0, o_nnz);
>>>     MatSetOption(B, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE);
>>>  
>>>     //here insert values row by row.
>>> MatSetValues(B, 1, &m, nn, nCols, matCols, INSERT_VALUES);
>>>  
>>> MatAssemblyBegin(B, MAT_FINAL_ASSEMBLY);
>>> MatAssemblyEnd(B, MAT_FINAL_ASSEMBLY);
>>>  
>>> Could you please tell me what I have done wrong?
>>>  
>>> Thank you,
>>> Jiannan
>>>    
>>> From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>> 
>>> Sent: Tuesday, August 16, 2022 2:00 PM
>>> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
>>> Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>
>>> Subject: Re: [petsc-users] Using matrix-free with KSP
>>>  
>>> CAUTION: This email was sent from outside the UMass Lowell network.
>>>  
>>>  
>>>    Create another matrix B that contains "Selected parts of the matrix can be pre-calculated and stored to serve as the preconditioner." When you call KSPSetOperators() pass in B as the second matrix argument.
>>>  
>>>    Now if literally application of B is the preconditioner you want to use, then simply use PCMAT as the preconditioner, and it will apply B each time it needs to apply the preconditioner. With this, you do not even need a PCShell. 
>>>  
>>>    If your preconditioner uses B to construct your own specific data structure needed to apply your preconditioner, then you can use PCShellSetSetUp() and have your routine call PCGetOperators() to pull out the B matrix and use it to construct your preconditioner. 
>>>  
>>>    If you want PETSc to construct a standard preconditioner from B, then you don't need the PCShell; you simply pass in the B, as above, and use -pc_type type to set the preconditioner you want to use. 
>>>  
>>>    Barry
>>>  
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> On Aug 16, 2022, at 12:31 PM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> wrote:
>>>  
>>> Barry,
>>>  
>>> Thank you very much for your instructions. I am sorry I may not ask clearer questions.
>>>  
>>> I don't use SNES. What I am trying to solve is a very large linear equation system with dense and ill-conditioned matrix. Selected parts of the matrix can be pre-calculated and stored to serve as the preconditioner. My question is how I can apply this matrix as the preconditioner. 
>>>  
>>> Here is the part of code from my linear equations solver. MatrixFreePreconditioner is supposed to be the routine to provide the preconditioner. But I don't understand how to use it. Should I use SNESSetJacobian() even if this is a linear problem? 
>>>  
>>>     MatCreateMFFD(MPI_COMM_WORLD, PETSC_DECIDE, PETSC_DECIDE, N, N, &A);
>>>     MatMFFDSetFunction(A, formfunction, &params);
>>>     MatSetFromOptions(A);
>>>     MatMFFDSetBase(A, X, NULL);
>>>  
>>>     KSPCreate(MPI_COMM_WORLD, &ksp);
>>>     KSPSetOperators(ksp, A, A);
>>>     KSPSetFromOptions(ksp);
>>>  
>>>     PC pc;
>>>     KSPGetPC(ksp,&pc);
>>>     PCSetType(pc,PCSHELL);
>>>     PCShellSetApply(pc,MatrixFreePreconditioner);
>>>  
>>> Thank you,
>>> Jiannan
>>>  
>>> From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>>
>>> Sent: Tuesday, August 16, 2022 10:10 AM
>>> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
>>> Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>>
>>> Subject: Re: [petsc-users] Using matrix-free with KSP
>>>  
>>> CAUTION: This email was sent from outside the UMass Lowell network.
>>>  
>>>  
>>>   I don't fully understand your question so I will answer questions that I do know the answer to :-)
>>>  
>>>   The function you provide to PCShellSetApply() applies the preconditioner to the input vector and puts the result in the output vector.  This input vector changes at every application of the preconditioner so should not be used in constructing the preconditioner.
>>>  
>>>   If you are using the standard PETSc matrix-free matrix-vector product via finite difference, -snes_mf_operator or via MatCreateSNESMF() or you are using a MatShell, the matrix-vector product routines input changes for each multiply and should not be used in constructing the preconditioner.
>>>  
>>>   The function you provide with SNESSetJacobian() is where you can compute anything you need to help build your preconditioner. The input vector to that function is the location at which the Jacobian is needed and is what you should use to build your approximate/part of Jacobian. You can store your approximate/part of Jacobian that you need to compute in the pmat and then in a PCShellSetUp() function that you provide you will compute what you will need latter to apply the preconditioner in your PCShellSetApply() function. Note that the values you put in the pmat do not need to be the full Jacobian; they can be anything you want since this pmat is not used to apply the matrix vector product.
>>>  
>>>   If this is unclear, feel free to ask additional questions.
>>>  
>>>   Barry
>>>  
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> On Aug 16, 2022, at 9:40 AM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> wrote:
>>>  
>>> I am trying to apply a user-defined preconditioner to speed up the convergency of matrix-free KSP solver. The user's manual provides an example of how to call the user-defined preconditioner routine. The routine has PC, input Vec and output Vec as arguments. I want to use part of the Jacobian as preconditioner matrix, the elements of which can be calculated using input Vec. My question is what the role of output Vec from the routine is in matrix-vector product? Or I just have the preconditioner matrix elements calculated in the routine and then Petsc handles applying the preconditioner to matrix-vector product? I used PCSetType(pc, PCSHELL) and PCShellSetApply.
>>>  
>>> Thank you and appreciate your help!
>>>  
>>> Jiannan 
>>> From: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
>>> Sent: Wednesday, July 6, 2022 2:34 PM
>>> To: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>>
>>> Cc: Jed Brown <jed at jedbrown.org <mailto:jed at jedbrown.org>>; petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>>
>>> Subject: Re: [petsc-users] Using matrix-free with KSP
>>>  
>>> Barry,
>>>  
>>> VecScatterCreateToAll solves the problem! The code now gives the correct answer with multiple processes. 
>>>  
>>> Thank you all so much!
>>>  
>>> Jiannan
>>> From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>>
>>> Sent: Wednesday, July 6, 2022 2:08 PM
>>> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
>>> Cc: Jed Brown <jed at jedbrown.org <mailto:jed at jedbrown.org>>; petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>>
>>> Subject: Re: [petsc-users] Using matrix-free with KSP
>>>  
>>>  
>>>   So your operator is a "dense" operator in that its matrix representation would be essentially dense. In that case, you can use specialized routines to do this efficiently. You can use VecScatterCreateToAll() and then VecScatterBegin/End to do the communication.
>>>  
>>>   Barry
>>>  
>>>  
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> On Jul 6, 2022, at 1:39 PM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>  wrote:
>>>  
>>> Jed, thank you very much for the reply. 
>>>  
>>> I'm not sure GlobaltoLocal will work. Say the solution vector is of length 10. Two processes then each process can see 5 elements of the vector. For A x = b (10 equations), each process performs matrix-vector product for five equations. And for each equation i , sum_j A_ij *  x_j = b_i requires the process has access to all 10 unknows x_i (i=0,1,...,9). 
>>>  
>>> Can VecScatter and PetscSF make the entire vector accessible to each process? I am reading the manual and trying to understand how VecScatter and PetscSF work.
>>>  
>>> Jiannan
>>> From: Jed Brown <jed at jedbrown.org <mailto:jed at jedbrown.org>>
>>> Sent: Wednesday, July 6, 2022 10:09 AM
>>> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>; Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>>
>>> Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>>
>>> Subject: Re: [petsc-users] Using matrix-free with KSP
>>>  
>>> You'll usually have a GlobalToLocal operation for each rank to get the halo data it needs, then either a LocalToGlobal to collect the result (e.g., finite element methods) or the local compute will write directly into the owned portion of the global vector. If you're doing the communication "raw", then you may find VecScatter or PetscSF useful to perform the necessary communication.
>>> 
>>> "Tu, Jiannan" <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>  writes:
>>> 
>>> > Hi Barry,
>>> >
>>> > Following your instructions I implemented the matrix-free method to solve a large linear equation system resulted from a surface integration equation. The KSP solver works fine for single process, but it is problematic with multiple processes. The problem is that each process only can access its own part of solution vector so that each process only conducts part of matrix-vector multiplication. MPI can be used to assemble these partial matrix-vector multiplication together.  Does Petsc provide any ways to implement multi-process matrix-free method?
>>> >
>>> > Thanks,
>>> > Jiannan
>>> > ________________________________
>>> > From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>>
>>> > Sent: Tuesday, May 24, 2022 2:12 PM
>>> > To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
>>> > Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>>
>>> > Subject: Re: [petsc-users] Using matrix-free with KSP
>>> >
>>> > This e-mail originated from outside the UMass Lowell network.
>>> > ________________________________
>>> >
>>> >    You can use MatCreateMFFD https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMatCreateMFFD%2F&amp;data=05%7C01%7CJiannan_Tu%40uml.edu%7Cab0859d8d154471590bc08da5f5918d6%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637927133525745809%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=A7wqegTfh94No5BpDiWLK3VxOuR44U2wlWHVm2k7l60%3D&amp;reserved=0<https://urldefense.com/v3/__https://petsc.org/main/docs/manualpages/Mat/MatCreateMFFD/__;!!PVKG_VDCxu5g!stxq2NWGVjUihvHBQC9Rlk0aHVoowcy0PTQYMcQxQ4DqOEC05KSw6TmstSXKckiUgelHzy4ue-d10-zlXkw$> <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMatCreateMFFD%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7Cc8226b4a8af040e8ca0208da845d5e8b%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637967833829887546%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=pQ3eVZvOw0WEJdHYm%2FP3leoX3tBNKpgLDQo5Xm10akw%3D&reserved=0> MatMFFDSetFunction MatSetFromOptions MatMFFDSetBase and provide the matrix to KSP. Note you will need to use -pc_type none or provide your own custom preconditioner withhttps://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FPC%2FPCSHELL%2F&amp;data=05%7C01%7CJiannan_Tu%40uml.edu%7Cab0859d8d154471590bc08da5f5918d6%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637927133525745809%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=Gb%2F9uFG3jp%2FbfWaSh2cSEQLItHS9CuLwarzN0KcNiJY%3D&amp;reserved=0<https://urldefense.com/v3/__https://petsc.org/main/docs/manualpages/PC/PCSHELL/__;!!PVKG_VDCxu5g!stxq2NWGVjUihvHBQC9Rlk0aHVoowcy0PTQYMcQxQ4DqOEC05KSw6TmstSXKckiUgelHzy4ue-d1cA4fKGw$> <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FPC%2FPCSHELL%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7Cc8226b4a8af040e8ca0208da845d5e8b%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637967833829887546%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=OwryOVq7EBof9oYaYJmN3klaLNwxtAxcZBgj44BF%2BJo%3D&reserved=0>
>>> >
>>> >
>>> >
>>> > On May 24, 2022, at 1:21 PM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu><mailto:Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>> wrote:
>>> >
>>> > I want to use a matrix-free matrix to solve a large linear equation system because the matrix is too large to be stored. Petsc user manual describes matrix-free method for SNES with examples. The matrix-free matrices section explains how to set up such a matrix, but I can't find any example of matrix-free method with KSP. I am wondering how to apply the method to KSP iterative solver in parallel.
>>> >
>>> > I greatly appreciate your help for me to understand this topic.
>>> >
>>> > Jiannan Tu

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220822/b26dd569/attachment-0001.html>

From snailsoar at hotmail.com  Mon Aug 22 13:45:37 2022
From: snailsoar at hotmail.com (feng wang)
Date: Mon, 22 Aug 2022 18:45:37 +0000
Subject: [petsc-users] Slepc Question, shell-matrix
Message-ID: <ME3PR01MB7624DF748CD2568A0BAAC5B6B9719@ME3PR01MB7624.ausprd01.prod.outlook.com>

Hello,

I am new to Slepc and trying to work out the eigenvalues and eigenvectors of my Jacobian matrix. I am using a shell matrix to work out the matrix-vector product and I am using the default Krylov-schur method.

My first attempt was not successful and I got the following errors:

[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Missing or incorrect user input
[0]PETSC ERROR: The inner product is not well defined: indefinite matrix
[0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.17.4, unknown
[0]PETSC ERROR: cfdtest on a arch-debug named ming by feng Mon Aug 22 19:21:41 2022
[0]PETSC ERROR: Configure options --with-cc=mpicc --with-cxx=mpicxx --with-fc=0 PETSC_ARCH=arch-debug
[0]PETSC ERROR: #1 BV_SafeSqrt() at /home/feng/cfd/slepc-3.17.1/include/slepc/private/bvimpl.h:130
[0]PETSC ERROR: #2 BV_SquareRoot_Default() at /home/feng/cfd/slepc-3.17.1/include/slepc/private/bvimpl.h:365
[0]PETSC ERROR: #3 BVOrthogonalizeCGS1() at /home/feng/cfd/slepc-3.17.1/src/sys/classes/bv/interface/bvorthog.c:101
[0]PETSC ERROR: #4 BVOrthogonalizeGS() at /home/feng/cfd/slepc-3.17.1/src/sys/classes/bv/interface/bvorthog.c:177
[0]PETSC ERROR: #5 BVOrthonormalizeColumn() at /home/feng/cfd/slepc-3.17.1/src/sys/classes/bv/interface/bvorthog.c:402
[0]PETSC ERROR: #6 BVMatArnoldi() at /home/feng/cfd/slepc-3.17.1/src/sys/classes/bv/interface/bvkrylov.c:91
[0]PETSC ERROR: #7 EPSSolve_KrylovSchur_Default() at /home/feng/cfd/slepc-3.17.1/src/eps/impls/krylov/krylovschur/krylovschur.c:261
[0]PETSC ERROR: #8 EPSSolve() at /home/feng/cfd/slepc-3.17.1/src/eps/interface/epssolve.c:147
[0]PETSC ERROR: #9 slepc_eigen_comp() at domain/cfd/slepc_eigen_solve.cpp:77

Could someone please shine some light on this? I have also attached my code. The code is part of my big library and I cannot attach the whole code, sorry about this. but I am happy to provide more information. The attached code has some arrangements for halo exchange, but for the moment, it assumes it is a serial run.

Many thanks,
Feng
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220822/6142820b/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: slepc_eigen_solve.cpp
Type: text/x-c++src
Size: 5623 bytes
Desc: slepc_eigen_solve.cpp
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220822/6142820b/attachment.bin>

From jroman at dsic.upv.es  Mon Aug 22 14:32:34 2022
From: jroman at dsic.upv.es (Jose E. Roman)
Date: Mon, 22 Aug 2022 21:32:34 +0200
Subject: [petsc-users] Slepc Question, shell-matrix
In-Reply-To: <ME3PR01MB7624DF748CD2568A0BAAC5B6B9719@ME3PR01MB7624.ausprd01.prod.outlook.com>
References: <ME3PR01MB7624DF748CD2568A0BAAC5B6B9719@ME3PR01MB7624.ausprd01.prod.outlook.com>
Message-ID: <C1E1C97D-7F02-4764-9732-9330A6076438@dsic.upv.es>

This is very strange. This error appears when the solver employs the B-inner product, but in your case you don't have a B matrix, so you should never see that error. Try running under valgrind to see if it gives more hints.

Jose


> El 22 ago 2022, a las 20:45, feng wang <snailsoar at hotmail.com> escribi?:
> 
> Hello,
> 
> I am new to Slepc and trying to work out the eigenvalues and eigenvectors of my Jacobian matrix. I am using a shell matrix to work out the matrix-vector product and I am using the default Krylov-schur method.  
> 
> My first attempt was not successful and I got the following errors:
> 
> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [0]PETSC ERROR: Missing or incorrect user input 
> [0]PETSC ERROR: The inner product is not well defined: indefinite matrix
> [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.17.4, unknown 
> [0]PETSC ERROR: cfdtest on a arch-debug named ming by feng Mon Aug 22 19:21:41 2022
> [0]PETSC ERROR: Configure options --with-cc=mpicc --with-cxx=mpicxx --with-fc=0 PETSC_ARCH=arch-debug
> [0]PETSC ERROR: #1 BV_SafeSqrt() at /home/feng/cfd/slepc-3.17.1/include/slepc/private/bvimpl.h:130
> [0]PETSC ERROR: #2 BV_SquareRoot_Default() at /home/feng/cfd/slepc-3.17.1/include/slepc/private/bvimpl.h:365
> [0]PETSC ERROR: #3 BVOrthogonalizeCGS1() at /home/feng/cfd/slepc-3.17.1/src/sys/classes/bv/interface/bvorthog.c:101
> [0]PETSC ERROR: #4 BVOrthogonalizeGS() at /home/feng/cfd/slepc-3.17.1/src/sys/classes/bv/interface/bvorthog.c:177
> [0]PETSC ERROR: #5 BVOrthonormalizeColumn() at /home/feng/cfd/slepc-3.17.1/src/sys/classes/bv/interface/bvorthog.c:402
> [0]PETSC ERROR: #6 BVMatArnoldi() at /home/feng/cfd/slepc-3.17.1/src/sys/classes/bv/interface/bvkrylov.c:91
> [0]PETSC ERROR: #7 EPSSolve_KrylovSchur_Default() at /home/feng/cfd/slepc-3.17.1/src/eps/impls/krylov/krylovschur/krylovschur.c:261
> [0]PETSC ERROR: #8 EPSSolve() at /home/feng/cfd/slepc-3.17.1/src/eps/interface/epssolve.c:147
> [0]PETSC ERROR: #9 slepc_eigen_comp() at domain/cfd/slepc_eigen_solve.cpp:77
> 
> Could someone please shine some light on this? I have also attached my code. The code is part of my big library and I cannot attach the whole code, sorry about this. but I am happy to provide more information. The attached code has some arrangements for halo exchange, but for the moment, it assumes it is a serial run.
> 
> Many thanks,
> Feng
> <slepc_eigen_solve.cpp>


From snailsoar at hotmail.com  Mon Aug 22 15:12:02 2022
From: snailsoar at hotmail.com (feng wang)
Date: Mon, 22 Aug 2022 20:12:02 +0000
Subject: [petsc-users] Slepc Question, shell-matrix
In-Reply-To: <C1E1C97D-7F02-4764-9732-9330A6076438@dsic.upv.es>
References: <ME3PR01MB7624DF748CD2568A0BAAC5B6B9719@ME3PR01MB7624.ausprd01.prod.outlook.com>
	<C1E1C97D-7F02-4764-9732-9330A6076438@dsic.upv.es>
Message-ID: <SYZPR01MB7627B4AE35964AB699B4DCAFB9719@SYZPR01MB7627.ausprd01.prod.outlook.com>

Hi Jose,

Thanks for your reply.

The test cases from slepc runs with no problem. so the library should be installed correctly. I have tried valgrind, it does not show any useful information. what else could I check?

Best regards,
Feng
________________________________
From: Jose E. Roman <jroman at dsic.upv.es>
Sent: 22 August 2022 19:32
To: feng wang <snailsoar at hotmail.com>
Cc: petsc-users at mcs.anl.gov <petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] Slepc Question, shell-matrix

This is very strange. This error appears when the solver employs the B-inner product, but in your case you don't have a B matrix, so you should never see that error. Try running under valgrind to see if it gives more hints.

Jose


> El 22 ago 2022, a las 20:45, feng wang <snailsoar at hotmail.com> escribi?:
>
> Hello,
>
> I am new to Slepc and trying to work out the eigenvalues and eigenvectors of my Jacobian matrix. I am using a shell matrix to work out the matrix-vector product and I am using the default Krylov-schur method.
>
> My first attempt was not successful and I got the following errors:
>
> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [0]PETSC ERROR: Missing or incorrect user input
> [0]PETSC ERROR: The inner product is not well defined: indefinite matrix
> [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.17.4, unknown
> [0]PETSC ERROR: cfdtest on a arch-debug named ming by feng Mon Aug 22 19:21:41 2022
> [0]PETSC ERROR: Configure options --with-cc=mpicc --with-cxx=mpicxx --with-fc=0 PETSC_ARCH=arch-debug
> [0]PETSC ERROR: #1 BV_SafeSqrt() at /home/feng/cfd/slepc-3.17.1/include/slepc/private/bvimpl.h:130
> [0]PETSC ERROR: #2 BV_SquareRoot_Default() at /home/feng/cfd/slepc-3.17.1/include/slepc/private/bvimpl.h:365
> [0]PETSC ERROR: #3 BVOrthogonalizeCGS1() at /home/feng/cfd/slepc-3.17.1/src/sys/classes/bv/interface/bvorthog.c:101
> [0]PETSC ERROR: #4 BVOrthogonalizeGS() at /home/feng/cfd/slepc-3.17.1/src/sys/classes/bv/interface/bvorthog.c:177
> [0]PETSC ERROR: #5 BVOrthonormalizeColumn() at /home/feng/cfd/slepc-3.17.1/src/sys/classes/bv/interface/bvorthog.c:402
> [0]PETSC ERROR: #6 BVMatArnoldi() at /home/feng/cfd/slepc-3.17.1/src/sys/classes/bv/interface/bvkrylov.c:91
> [0]PETSC ERROR: #7 EPSSolve_KrylovSchur_Default() at /home/feng/cfd/slepc-3.17.1/src/eps/impls/krylov/krylovschur/krylovschur.c:261
> [0]PETSC ERROR: #8 EPSSolve() at /home/feng/cfd/slepc-3.17.1/src/eps/interface/epssolve.c:147
> [0]PETSC ERROR: #9 slepc_eigen_comp() at domain/cfd/slepc_eigen_solve.cpp:77
>
> Could someone please shine some light on this? I have also attached my code. The code is part of my big library and I cannot attach the whole code, sorry about this. but I am happy to provide more information. The attached code has some arrangements for halo exchange, but for the moment, it assumes it is a serial run.
>
> Many thanks,
> Feng
> <slepc_eigen_solve.cpp>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220822/c5643272/attachment-0001.html>

From patrick.alken at geomag.info  Mon Aug 22 19:11:32 2022
From: patrick.alken at geomag.info (Patrick Alken)
Date: Mon, 22 Aug 2022 18:11:32 -0600
Subject: [petsc-users] [Slepc] eigensolver not running in parallel
Message-ID: <718c6e23-63ff-8061-e8da-8c0abaf44538@geomag.info>

I am trying to solve a symmetric generalized eigenproblem with slepc (n 
= 40500) using 8 processors. The command I use is:

mpiexec -n 8 ./main -eps_nev 10 -eps_target 0 -st_type sinvert

The program assembles the two matrices (A,B), sets up the eigensolver 
and then calls EPSSolve. The issue is that the matrix assembly runs fine 
in parallel. And for about 1 minute or so, the EPSSolve call uses all 8 
processors (according to the top program). But then after around 1 
minute or so, the program collapses to just 1 thread and just runs that 
way for a long time (still in the EPSSolve call).

Is there a way to diagnose what is going on, and how to make the program 
use all 8 processors for the eigensolver?




From jroman at dsic.upv.es  Tue Aug 23 00:24:29 2022
From: jroman at dsic.upv.es (Jose E. Roman)
Date: Tue, 23 Aug 2022 07:24:29 +0200
Subject: [petsc-users] Slepc Question, shell-matrix
In-Reply-To: <ME3PR01MB7624ADA914B555F17A490226B9719@ME3PR01MB7624.ausprd01.prod.outlook.com>
References: <ME3PR01MB7624DF748CD2568A0BAAC5B6B9719@ME3PR01MB7624.ausprd01.prod.outlook.com>
	<C1E1C97D-7F02-4764-9732-9330A6076438@dsic.upv.es>
	<ME3PR01MB7624ADA914B555F17A490226B9719@ME3PR01MB7624.ausprd01.prod.outlook.com>
Message-ID: <8B78A886-8CB5-49A9-AFED-E67C22696400@dsic.upv.es>

Please always respond to the list, otherwise the thread appears as unresolved in the archives of the mailing list.


> El 22 ago 2022, a las 22:45, feng wang <snailsoar at hotmail.com> escribi?:
> 
> Hi Jose,
> 
> I think I might have solved my problem. I have some uninitialized values in my part of code to compute the right hand side. so it produces a NAN when it tries to compute the matrix-vector product. 
> 
> Many thanks for your help!
> 
> Best regards,
> Feng
> From: Jose E. Roman <jroman at dsic.upv.es>
> Sent: 22 August 2022 19:32
> To: feng wang <snailsoar at hotmail.com>
> Cc: petsc-users at mcs.anl.gov <petsc-users at mcs.anl.gov>
> Subject: Re: [petsc-users] Slepc Question, shell-matrix
>  
> This is very strange. This error appears when the solver employs the B-inner product, but in your case you don't have a B matrix, so you should never see that error. Try running under valgrind to see if it gives more hints.
> 
> Jose
> 
> 
> > El 22 ago 2022, a las 20:45, feng wang <snailsoar at hotmail.com> escribi?:
> > 
> > Hello,
> > 
> > I am new to Slepc and trying to work out the eigenvalues and eigenvectors of my Jacobian matrix. I am using a shell matrix to work out the matrix-vector product and I am using the default Krylov-schur method.  
> > 
> > My first attempt was not successful and I got the following errors:
> > 
> > [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> > [0]PETSC ERROR: Missing or incorrect user input 
> > [0]PETSC ERROR: The inner product is not well defined: indefinite matrix
> > [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
> > [0]PETSC ERROR: Petsc Release Version 3.17.4, unknown 
> > [0]PETSC ERROR: cfdtest on a arch-debug named ming by feng Mon Aug 22 19:21:41 2022
> > [0]PETSC ERROR: Configure options --with-cc=mpicc --with-cxx=mpicxx --with-fc=0 PETSC_ARCH=arch-debug
> > [0]PETSC ERROR: #1 BV_SafeSqrt() at /home/feng/cfd/slepc-3.17.1/include/slepc/private/bvimpl.h:130
> > [0]PETSC ERROR: #2 BV_SquareRoot_Default() at /home/feng/cfd/slepc-3.17.1/include/slepc/private/bvimpl.h:365
> > [0]PETSC ERROR: #3 BVOrthogonalizeCGS1() at /home/feng/cfd/slepc-3.17.1/src/sys/classes/bv/interface/bvorthog.c:101
> > [0]PETSC ERROR: #4 BVOrthogonalizeGS() at /home/feng/cfd/slepc-3.17.1/src/sys/classes/bv/interface/bvorthog.c:177
> > [0]PETSC ERROR: #5 BVOrthonormalizeColumn() at /home/feng/cfd/slepc-3.17.1/src/sys/classes/bv/interface/bvorthog.c:402
> > [0]PETSC ERROR: #6 BVMatArnoldi() at /home/feng/cfd/slepc-3.17.1/src/sys/classes/bv/interface/bvkrylov.c:91
> > [0]PETSC ERROR: #7 EPSSolve_KrylovSchur_Default() at /home/feng/cfd/slepc-3.17.1/src/eps/impls/krylov/krylovschur/krylovschur.c:261
> > [0]PETSC ERROR: #8 EPSSolve() at /home/feng/cfd/slepc-3.17.1/src/eps/interface/epssolve.c:147
> > [0]PETSC ERROR: #9 slepc_eigen_comp() at domain/cfd/slepc_eigen_solve.cpp:77
> > 
> > Could someone please shine some light on this? I have also attached my code. The code is part of my big library and I cannot attach the whole code, sorry about this. but I am happy to provide more information. The attached code has some arrangements for halo exchange, but for the moment, it assumes it is a serial run.
> > 
> > Many thanks,
> > Feng
> > <slepc_eigen_solve.cpp>


From jroman at dsic.upv.es  Tue Aug 23 00:32:42 2022
From: jroman at dsic.upv.es (Jose E. Roman)
Date: Tue, 23 Aug 2022 07:32:42 +0200
Subject: [petsc-users] [Slepc] eigensolver not running in parallel
In-Reply-To: <718c6e23-63ff-8061-e8da-8c0abaf44538@geomag.info>
References: <718c6e23-63ff-8061-e8da-8c0abaf44538@geomag.info>
Message-ID: <3FD1D9E5-1300-491D-9AED-3DB18906B20A@dsic.upv.es>

Which version of SLEPc are you using?

You should get an error unless you have configure PETSc with a parallel direct linear solver. Which option did you use to configure PETSc?

Send the output when using the option -eps_view_pre

Jose


> El 23 ago 2022, a las 2:11, Patrick Alken <patrick.alken at geomag.info> escribi?:
> 
> I am trying to solve a symmetric generalized eigenproblem with slepc (n = 40500) using 8 processors. The command I use is:
> 
> mpiexec -n 8 ./main -eps_nev 10 -eps_target 0 -st_type sinvert
> 
> The program assembles the two matrices (A,B), sets up the eigensolver and then calls EPSSolve. The issue is that the matrix assembly runs fine in parallel. And for about 1 minute or so, the EPSSolve call uses all 8 processors (according to the top program). But then after around 1 minute or so, the program collapses to just 1 thread and just runs that way for a long time (still in the EPSSolve call).
> 
> Is there a way to diagnose what is going on, and how to make the program use all 8 processors for the eigensolver?
> 
> 
> 


From snailsoar at hotmail.com  Tue Aug 23 05:14:14 2022
From: snailsoar at hotmail.com (feng wang)
Date: Tue, 23 Aug 2022 10:14:14 +0000
Subject: [petsc-users] Slepc Question, shell-matrix
In-Reply-To: <8B78A886-8CB5-49A9-AFED-E67C22696400@dsic.upv.es>
References: <ME3PR01MB7624DF748CD2568A0BAAC5B6B9719@ME3PR01MB7624.ausprd01.prod.outlook.com>
	<C1E1C97D-7F02-4764-9732-9330A6076438@dsic.upv.es>
	<ME3PR01MB7624ADA914B555F17A490226B9719@ME3PR01MB7624.ausprd01.prod.outlook.com>
	<8B78A886-8CB5-49A9-AFED-E67C22696400@dsic.upv.es>
Message-ID: <SYZPR01MB76275849E71CDD1083D79937B9709@SYZPR01MB7627.ausprd01.prod.outlook.com>

Hi Jose,

I think the previous problem comes from my side. I have some uninitialized values in my part of code to compute the non-linear residuals. so, it produces a NAN when it tries to compute the matrix-vector product using finite difference.  This might make the slepc/pestc do unexpected things.

Now It seems I've got slepc running. eps_nev is set to 3 and I am trying to compute the ones with the largest amplitudes. Below is the slepc output.

 14 EPS converged value (error) #0 -0.000164282 (5.36813206e-09)
 16 EPS converged value (error) #1 -0.000160691+2.17113e-05i (3.37429620e-09)
 16 EPS converged value (error) #2 -0.000160691-2.17113e-05i (3.37429620e-09)
 Solution method: krylovschur

 Number of requested eigenvalues: 2
 Linear eigensolve converged (3 eigenpairs) due to CONVERGED_TOL; iterations 16
 ---------------------- --------------------
            k             ||Ax-kx||/||kx||
 ---------------------- --------------------
       -0.000164              0.0613788
  -0.000161+0.000022i         0.0773339
  -0.000161-0.000022i         0.0774536
 ---------------------- --------------------

The values in the brackets are the absolute error (I believe) and they seem very low. The relative error seems quite large.  Could you please comment on this?


Best regards,
Feng

________________________________
From: Jose E. Roman <jroman at dsic.upv.es>
Sent: 23 August 2022 5:24
To: feng wang <snailsoar at hotmail.com>
Cc: petsc-users <petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] Slepc Question, shell-matrix

Please always respond to the list, otherwise the thread appears as unresolved in the archives of the mailing list.


> El 22 ago 2022, a las 22:45, feng wang <snailsoar at hotmail.com> escribi?:
>
> Hi Jose,
>
> I think I might have solved my problem. I have some uninitialized values in my part of code to compute the right hand side. so it produces a NAN when it tries to compute the matrix-vector product.
>
> Many thanks for your help!
>
> Best regards,
> Feng
> From: Jose E. Roman <jroman at dsic.upv.es>
> Sent: 22 August 2022 19:32
> To: feng wang <snailsoar at hotmail.com>
> Cc: petsc-users at mcs.anl.gov <petsc-users at mcs.anl.gov>
> Subject: Re: [petsc-users] Slepc Question, shell-matrix
>
> This is very strange. This error appears when the solver employs the B-inner product, but in your case you don't have a B matrix, so you should never see that error. Try running under valgrind to see if it gives more hints.
>
> Jose
>
>
> > El 22 ago 2022, a las 20:45, feng wang <snailsoar at hotmail.com> escribi?:
> >
> > Hello,
> >
> > I am new to Slepc and trying to work out the eigenvalues and eigenvectors of my Jacobian matrix. I am using a shell matrix to work out the matrix-vector product and I am using the default Krylov-schur method.
> >
> > My first attempt was not successful and I got the following errors:
> >
> > [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> > [0]PETSC ERROR: Missing or incorrect user input
> > [0]PETSC ERROR: The inner product is not well defined: indefinite matrix
> > [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
> > [0]PETSC ERROR: Petsc Release Version 3.17.4, unknown
> > [0]PETSC ERROR: cfdtest on a arch-debug named ming by feng Mon Aug 22 19:21:41 2022
> > [0]PETSC ERROR: Configure options --with-cc=mpicc --with-cxx=mpicxx --with-fc=0 PETSC_ARCH=arch-debug
> > [0]PETSC ERROR: #1 BV_SafeSqrt() at /home/feng/cfd/slepc-3.17.1/include/slepc/private/bvimpl.h:130
> > [0]PETSC ERROR: #2 BV_SquareRoot_Default() at /home/feng/cfd/slepc-3.17.1/include/slepc/private/bvimpl.h:365
> > [0]PETSC ERROR: #3 BVOrthogonalizeCGS1() at /home/feng/cfd/slepc-3.17.1/src/sys/classes/bv/interface/bvorthog.c:101
> > [0]PETSC ERROR: #4 BVOrthogonalizeGS() at /home/feng/cfd/slepc-3.17.1/src/sys/classes/bv/interface/bvorthog.c:177
> > [0]PETSC ERROR: #5 BVOrthonormalizeColumn() at /home/feng/cfd/slepc-3.17.1/src/sys/classes/bv/interface/bvorthog.c:402
> > [0]PETSC ERROR: #6 BVMatArnoldi() at /home/feng/cfd/slepc-3.17.1/src/sys/classes/bv/interface/bvkrylov.c:91
> > [0]PETSC ERROR: #7 EPSSolve_KrylovSchur_Default() at /home/feng/cfd/slepc-3.17.1/src/eps/impls/krylov/krylovschur/krylovschur.c:261
> > [0]PETSC ERROR: #8 EPSSolve() at /home/feng/cfd/slepc-3.17.1/src/eps/interface/epssolve.c:147
> > [0]PETSC ERROR: #9 slepc_eigen_comp() at domain/cfd/slepc_eigen_solve.cpp:77
> >
> > Could someone please shine some light on this? I have also attached my code. The code is part of my big library and I cannot attach the whole code, sorry about this. but I am happy to provide more information. The attached code has some arrangements for halo exchange, but for the moment, it assumes it is a serial run.
> >
> > Many thanks,
> > Feng
> > <slepc_eigen_solve.cpp>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220823/a190e2a6/attachment-0001.html>

From jroman at dsic.upv.es  Tue Aug 23 05:32:26 2022
From: jroman at dsic.upv.es (Jose E. Roman)
Date: Tue, 23 Aug 2022 12:32:26 +0200
Subject: [petsc-users] Slepc Question, shell-matrix
In-Reply-To: <SYZPR01MB76275849E71CDD1083D79937B9709@SYZPR01MB7627.ausprd01.prod.outlook.com>
References: <ME3PR01MB7624DF748CD2568A0BAAC5B6B9719@ME3PR01MB7624.ausprd01.prod.outlook.com>
	<C1E1C97D-7F02-4764-9732-9330A6076438@dsic.upv.es>
	<ME3PR01MB7624ADA914B555F17A490226B9719@ME3PR01MB7624.ausprd01.prod.outlook.com>
	<8B78A886-8CB5-49A9-AFED-E67C22696400@dsic.upv.es>
	<SYZPR01MB76275849E71CDD1083D79937B9709@SYZPR01MB7627.ausprd01.prod.outlook.com>
Message-ID: <17F56A61-2DB1-416B-85CE-8E585051F1EB@dsic.upv.es>

The relative residual norms that are printed at the end are too large. For NHEP problems, they should be below the tolerance. Don't know what is happening. Does your shell matrix represent a linear (constant) operator? Or does it change slightly depending on the input vector? 

> El 23 ago 2022, a las 12:14, feng wang <snailsoar at hotmail.com> escribi?:
> 
> Hi Jose,
> 
> I think the previous problem comes from my side. I have some uninitialized values in my part of code to compute the non-linear residuals. so, it produces a NAN when it tries to compute the matrix-vector product using finite difference.  This might make the slepc/pestc do unexpected things.
> 
> Now It seems I've got slepc running. eps_nev is set to 3 and I am trying to compute the ones with the largest amplitudes. Below is the slepc output. 
> 
>  14 EPS converged value (error) #0 -0.000164282 (5.36813206e-09)
>  16 EPS converged value (error) #1 -0.000160691+2.17113e-05i (3.37429620e-09)
>  16 EPS converged value (error) #2 -0.000160691-2.17113e-05i (3.37429620e-09)
>  Solution method: krylovschur
> 
>  Number of requested eigenvalues: 2
>  Linear eigensolve converged (3 eigenpairs) due to CONVERGED_TOL; iterations 16
>  ---------------------- --------------------
>             k             ||Ax-kx||/||kx||
>  ---------------------- --------------------
>        -0.000164              0.0613788
>   -0.000161+0.000022i         0.0773339
>   -0.000161-0.000022i         0.0774536
>  ---------------------- --------------------
> 
> The values in the brackets are the absolute error (I believe) and they seem very low. The relative error seems quite large.  Could you please comment on this? 
> 
> 
> Best regards,
> Feng
> 
> From: Jose E. Roman <jroman at dsic.upv.es>
> Sent: 23 August 2022 5:24
> To: feng wang <snailsoar at hotmail.com>
> Cc: petsc-users <petsc-users at mcs.anl.gov>
> Subject: Re: [petsc-users] Slepc Question, shell-matrix
>  
> Please always respond to the list, otherwise the thread appears as unresolved in the archives of the mailing list.
> 
> 
> > El 22 ago 2022, a las 22:45, feng wang <snailsoar at hotmail.com> escribi?:
> > 
> > Hi Jose,
> > 
> > I think I might have solved my problem. I have some uninitialized values in my part of code to compute the right hand side. so it produces a NAN when it tries to compute the matrix-vector product. 
> > 
> > Many thanks for your help!
> > 
> > Best regards,
> > Feng
> > From: Jose E. Roman <jroman at dsic.upv.es>
> > Sent: 22 August 2022 19:32
> > To: feng wang <snailsoar at hotmail.com>
> > Cc: petsc-users at mcs.anl.gov <petsc-users at mcs.anl.gov>
> > Subject: Re: [petsc-users] Slepc Question, shell-matrix
> >  
> > This is very strange. This error appears when the solver employs the B-inner product, but in your case you don't have a B matrix, so you should never see that error. Try running under valgrind to see if it gives more hints.
> > 
> > Jose
> > 
> > 
> > > El 22 ago 2022, a las 20:45, feng wang <snailsoar at hotmail.com> escribi?:
> > > 
> > > Hello,
> > > 
> > > I am new to Slepc and trying to work out the eigenvalues and eigenvectors of my Jacobian matrix. I am using a shell matrix to work out the matrix-vector product and I am using the default Krylov-schur method.  
> > > 
> > > My first attempt was not successful and I got the following errors:
> > > 
> > > [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> > > [0]PETSC ERROR: Missing or incorrect user input 
> > > [0]PETSC ERROR: The inner product is not well defined: indefinite matrix
> > > [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
> > > [0]PETSC ERROR: Petsc Release Version 3.17.4, unknown 
> > > [0]PETSC ERROR: cfdtest on a arch-debug named ming by feng Mon Aug 22 19:21:41 2022
> > > [0]PETSC ERROR: Configure options --with-cc=mpicc --with-cxx=mpicxx --with-fc=0 PETSC_ARCH=arch-debug
> > > [0]PETSC ERROR: #1 BV_SafeSqrt() at /home/feng/cfd/slepc-3.17.1/include/slepc/private/bvimpl.h:130
> > > [0]PETSC ERROR: #2 BV_SquareRoot_Default() at /home/feng/cfd/slepc-3.17.1/include/slepc/private/bvimpl.h:365
> > > [0]PETSC ERROR: #3 BVOrthogonalizeCGS1() at /home/feng/cfd/slepc-3.17.1/src/sys/classes/bv/interface/bvorthog.c:101
> > > [0]PETSC ERROR: #4 BVOrthogonalizeGS() at /home/feng/cfd/slepc-3.17.1/src/sys/classes/bv/interface/bvorthog.c:177
> > > [0]PETSC ERROR: #5 BVOrthonormalizeColumn() at /home/feng/cfd/slepc-3.17.1/src/sys/classes/bv/interface/bvorthog.c:402
> > > [0]PETSC ERROR: #6 BVMatArnoldi() at /home/feng/cfd/slepc-3.17.1/src/sys/classes/bv/interface/bvkrylov.c:91
> > > [0]PETSC ERROR: #7 EPSSolve_KrylovSchur_Default() at /home/feng/cfd/slepc-3.17.1/src/eps/impls/krylov/krylovschur/krylovschur.c:261
> > > [0]PETSC ERROR: #8 EPSSolve() at /home/feng/cfd/slepc-3.17.1/src/eps/interface/epssolve.c:147
> > > [0]PETSC ERROR: #9 slepc_eigen_comp() at domain/cfd/slepc_eigen_solve.cpp:77
> > > 
> > > Could someone please shine some light on this? I have also attached my code. The code is part of my big library and I cannot attach the whole code, sorry about this. but I am happy to provide more information. The attached code has some arrangements for halo exchange, but for the moment, it assumes it is a serial run.
> > > 
> > > Many thanks,
> > > Feng
> > > <slepc_eigen_solve.cpp>


From rafel.amer at upc.edu  Tue Aug 23 05:34:28 2022
From: rafel.amer at upc.edu (Rafel Amer Ramon)
Date: Tue, 23 Aug 2022 12:34:28 +0200
Subject: [petsc-users] PETSc on fedora 36
In-Reply-To: <358BFF34-D941-42AB-8648-871186EF64DB@petsc.dev>
References: <aa3cbfe2-2ccc-93cc-6004-5c2ae6429047@upc.edu>
	<358BFF34-D941-42AB-8648-871186EF64DB@petsc.dev>
Message-ID: <65d7f5ad-fcd5-2cc9-0f74-8c9c8f5eaeb8@upc.edu>

An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220823/54dc612a/attachment.html>

From snailsoar at hotmail.com  Tue Aug 23 05:57:31 2022
From: snailsoar at hotmail.com (feng wang)
Date: Tue, 23 Aug 2022 10:57:31 +0000
Subject: [petsc-users] Slepc Question, shell-matrix
In-Reply-To: <17F56A61-2DB1-416B-85CE-8E585051F1EB@dsic.upv.es>
References: <ME3PR01MB7624DF748CD2568A0BAAC5B6B9719@ME3PR01MB7624.ausprd01.prod.outlook.com>
	<C1E1C97D-7F02-4764-9732-9330A6076438@dsic.upv.es>
	<ME3PR01MB7624ADA914B555F17A490226B9719@ME3PR01MB7624.ausprd01.prod.outlook.com>
	<8B78A886-8CB5-49A9-AFED-E67C22696400@dsic.upv.es>
	<SYZPR01MB76275849E71CDD1083D79937B9709@SYZPR01MB7627.ausprd01.prod.outlook.com>
	<17F56A61-2DB1-416B-85CE-8E585051F1EB@dsic.upv.es>
Message-ID: <ME3PR01MB762414A8D756C6C68BC95E92B9709@ME3PR01MB7624.ausprd01.prod.outlook.com>

Hi Jose,

Thanks for your reply.

It represents a linear operator. In my shell matrix, I am computing the non-linear residuals twice with perturbed flow variables. The matrix-vector product is computed as:

A*v = (R(q+eps*v) - R(q-eps*v))/(2*eps)

R is the non-linear residual. q is my flow variable and it does not change. eps is the perturbation. A is my Jacobian matrix. Besides, for some background, I am computing a steady RANS flow with finite volume method and trying to do a global stability analysis by looking at the Jacobian matrix.

Thanks,
Feng

________________________________
From: Jose E. Roman <jroman at dsic.upv.es>
Sent: 23 August 2022 10:32
To: feng wang <snailsoar at hotmail.com>
Cc: petsc-users <petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] Slepc Question, shell-matrix

The relative residual norms that are printed at the end are too large. For NHEP problems, they should be below the tolerance. Don't know what is happening. Does your shell matrix represent a linear (constant) operator? Or does it change slightly depending on the input vector?

> El 23 ago 2022, a las 12:14, feng wang <snailsoar at hotmail.com> escribi?:
>
> Hi Jose,
>
> I think the previous problem comes from my side. I have some uninitialized values in my part of code to compute the non-linear residuals. so, it produces a NAN when it tries to compute the matrix-vector product using finite difference.  This might make the slepc/pestc do unexpected things.
>
> Now It seems I've got slepc running. eps_nev is set to 3 and I am trying to compute the ones with the largest amplitudes. Below is the slepc output.
>
>  14 EPS converged value (error) #0 -0.000164282 (5.36813206e-09)
>  16 EPS converged value (error) #1 -0.000160691+2.17113e-05i (3.37429620e-09)
>  16 EPS converged value (error) #2 -0.000160691-2.17113e-05i (3.37429620e-09)
>  Solution method: krylovschur
>
>  Number of requested eigenvalues: 2
>  Linear eigensolve converged (3 eigenpairs) due to CONVERGED_TOL; iterations 16
>  ---------------------- --------------------
>             k             ||Ax-kx||/||kx||
>  ---------------------- --------------------
>        -0.000164              0.0613788
>   -0.000161+0.000022i         0.0773339
>   -0.000161-0.000022i         0.0774536
>  ---------------------- --------------------
>
> The values in the brackets are the absolute error (I believe) and they seem very low. The relative error seems quite large.  Could you please comment on this?
>
>
> Best regards,
> Feng
>
> From: Jose E. Roman <jroman at dsic.upv.es>
> Sent: 23 August 2022 5:24
> To: feng wang <snailsoar at hotmail.com>
> Cc: petsc-users <petsc-users at mcs.anl.gov>
> Subject: Re: [petsc-users] Slepc Question, shell-matrix
>
> Please always respond to the list, otherwise the thread appears as unresolved in the archives of the mailing list.
>
>
> > El 22 ago 2022, a las 22:45, feng wang <snailsoar at hotmail.com> escribi?:
> >
> > Hi Jose,
> >
> > I think I might have solved my problem. I have some uninitialized values in my part of code to compute the right hand side. so it produces a NAN when it tries to compute the matrix-vector product.
> >
> > Many thanks for your help!
> >
> > Best regards,
> > Feng
> > From: Jose E. Roman <jroman at dsic.upv.es>
> > Sent: 22 August 2022 19:32
> > To: feng wang <snailsoar at hotmail.com>
> > Cc: petsc-users at mcs.anl.gov <petsc-users at mcs.anl.gov>
> > Subject: Re: [petsc-users] Slepc Question, shell-matrix
> >
> > This is very strange. This error appears when the solver employs the B-inner product, but in your case you don't have a B matrix, so you should never see that error. Try running under valgrind to see if it gives more hints.
> >
> > Jose
> >
> >
> > > El 22 ago 2022, a las 20:45, feng wang <snailsoar at hotmail.com> escribi?:
> > >
> > > Hello,
> > >
> > > I am new to Slepc and trying to work out the eigenvalues and eigenvectors of my Jacobian matrix. I am using a shell matrix to work out the matrix-vector product and I am using the default Krylov-schur method.
> > >
> > > My first attempt was not successful and I got the following errors:
> > >
> > > [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> > > [0]PETSC ERROR: Missing or incorrect user input
> > > [0]PETSC ERROR: The inner product is not well defined: indefinite matrix
> > > [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
> > > [0]PETSC ERROR: Petsc Release Version 3.17.4, unknown
> > > [0]PETSC ERROR: cfdtest on a arch-debug named ming by feng Mon Aug 22 19:21:41 2022
> > > [0]PETSC ERROR: Configure options --with-cc=mpicc --with-cxx=mpicxx --with-fc=0 PETSC_ARCH=arch-debug
> > > [0]PETSC ERROR: #1 BV_SafeSqrt() at /home/feng/cfd/slepc-3.17.1/include/slepc/private/bvimpl.h:130
> > > [0]PETSC ERROR: #2 BV_SquareRoot_Default() at /home/feng/cfd/slepc-3.17.1/include/slepc/private/bvimpl.h:365
> > > [0]PETSC ERROR: #3 BVOrthogonalizeCGS1() at /home/feng/cfd/slepc-3.17.1/src/sys/classes/bv/interface/bvorthog.c:101
> > > [0]PETSC ERROR: #4 BVOrthogonalizeGS() at /home/feng/cfd/slepc-3.17.1/src/sys/classes/bv/interface/bvorthog.c:177
> > > [0]PETSC ERROR: #5 BVOrthonormalizeColumn() at /home/feng/cfd/slepc-3.17.1/src/sys/classes/bv/interface/bvorthog.c:402
> > > [0]PETSC ERROR: #6 BVMatArnoldi() at /home/feng/cfd/slepc-3.17.1/src/sys/classes/bv/interface/bvkrylov.c:91
> > > [0]PETSC ERROR: #7 EPSSolve_KrylovSchur_Default() at /home/feng/cfd/slepc-3.17.1/src/eps/impls/krylov/krylovschur/krylovschur.c:261
> > > [0]PETSC ERROR: #8 EPSSolve() at /home/feng/cfd/slepc-3.17.1/src/eps/interface/epssolve.c:147
> > > [0]PETSC ERROR: #9 slepc_eigen_comp() at domain/cfd/slepc_eigen_solve.cpp:77
> > >
> > > Could someone please shine some light on this? I have also attached my code. The code is part of my big library and I cannot attach the whole code, sorry about this. but I am happy to provide more information. The attached code has some arrangements for halo exchange, but for the moment, it assumes it is a serial run.
> > >
> > > Many thanks,
> > > Feng
> > > <slepc_eigen_solve.cpp>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220823/454b5a12/attachment-0001.html>

From jroman at dsic.upv.es  Tue Aug 23 06:06:08 2022
From: jroman at dsic.upv.es (Jose E. Roman)
Date: Tue, 23 Aug 2022 13:06:08 +0200
Subject: [petsc-users] Slepc Question, shell-matrix
In-Reply-To: <ME3PR01MB762414A8D756C6C68BC95E92B9709@ME3PR01MB7624.ausprd01.prod.outlook.com>
References: <ME3PR01MB7624DF748CD2568A0BAAC5B6B9719@ME3PR01MB7624.ausprd01.prod.outlook.com>
	<C1E1C97D-7F02-4764-9732-9330A6076438@dsic.upv.es>
	<ME3PR01MB7624ADA914B555F17A490226B9719@ME3PR01MB7624.ausprd01.prod.outlook.com>
	<8B78A886-8CB5-49A9-AFED-E67C22696400@dsic.upv.es>
	<SYZPR01MB76275849E71CDD1083D79937B9709@SYZPR01MB7627.ausprd01.prod.outlook.com>
	<17F56A61-2DB1-416B-85CE-8E585051F1EB@dsic.upv.es>
	<ME3PR01MB762414A8D756C6C68BC95E92B9709@ME3PR01MB7624.ausprd01.prod.outlook.com>
Message-ID: <3E25E7D6-C81A-4A75-A215-A5FCFB7F3CB6@dsic.upv.es>

You can try the following. Save your shell matrix to disk with the option
-eps_view_mat0 binary:amatrix.bin
then repeat the computation with ex4.c loading this file.

Note that this should be done for a small problem size, because the conversion from a shell matrix implies a matrix-vector product per each column.

Jose

> El 23 ago 2022, a las 12:57, feng wang <snailsoar at hotmail.com> escribi?:
> 
> Hi Jose,
> 
> Thanks for your reply.
> 
> It represents a linear operator. In my shell matrix, I am computing the non-linear residuals twice with perturbed flow variables. The matrix-vector product is computed as:
> 
> A*v = (R(q+eps*v) - R(q-eps*v))/(2*eps)
> 
> R is the non-linear residual. q is my flow variable and it does not change. eps is the perturbation. A is my Jacobian matrix. Besides, for some background, I am computing a steady RANS flow with finite volume method and trying to do a global stability analysis by looking at the Jacobian matrix. 
> 
> Thanks,
> Feng
> 
> From: Jose E. Roman <jroman at dsic.upv.es>
> Sent: 23 August 2022 10:32
> To: feng wang <snailsoar at hotmail.com>
> Cc: petsc-users <petsc-users at mcs.anl.gov>
> Subject: Re: [petsc-users] Slepc Question, shell-matrix
>  
> The relative residual norms that are printed at the end are too large. For NHEP problems, they should be below the tolerance. Don't know what is happening. Does your shell matrix represent a linear (constant) operator? Or does it change slightly depending on the input vector? 
> 
> > El 23 ago 2022, a las 12:14, feng wang <snailsoar at hotmail.com> escribi?:
> > 
> > Hi Jose,
> > 
> > I think the previous problem comes from my side. I have some uninitialized values in my part of code to compute the non-linear residuals. so, it produces a NAN when it tries to compute the matrix-vector product using finite difference.  This might make the slepc/pestc do unexpected things.
> > 
> > Now It seems I've got slepc running. eps_nev is set to 3 and I am trying to compute the ones with the largest amplitudes. Below is the slepc output. 
> > 
> >  14 EPS converged value (error) #0 -0.000164282 (5.36813206e-09)
> >  16 EPS converged value (error) #1 -0.000160691+2.17113e-05i (3.37429620e-09)
> >  16 EPS converged value (error) #2 -0.000160691-2.17113e-05i (3.37429620e-09)
> >  Solution method: krylovschur
> > 
> >  Number of requested eigenvalues: 2
> >  Linear eigensolve converged (3 eigenpairs) due to CONVERGED_TOL; iterations 16
> >  ---------------------- --------------------
> >             k             ||Ax-kx||/||kx||
> >  ---------------------- --------------------
> >        -0.000164              0.0613788
> >   -0.000161+0.000022i         0.0773339
> >   -0.000161-0.000022i         0.0774536
> >  ---------------------- --------------------
> > 
> > The values in the brackets are the absolute error (I believe) and they seem very low. The relative error seems quite large.  Could you please comment on this? 
> > 
> > 
> > Best regards,
> > Feng
> > 
> > From: Jose E. Roman <jroman at dsic.upv.es>
> > Sent: 23 August 2022 5:24
> > To: feng wang <snailsoar at hotmail.com>
> > Cc: petsc-users <petsc-users at mcs.anl.gov>
> > Subject: Re: [petsc-users] Slepc Question, shell-matrix
> >  
> > Please always respond to the list, otherwise the thread appears as unresolved in the archives of the mailing list.
> > 
> > 
> > > El 22 ago 2022, a las 22:45, feng wang <snailsoar at hotmail.com> escribi?:
> > > 
> > > Hi Jose,
> > > 
> > > I think I might have solved my problem. I have some uninitialized values in my part of code to compute the right hand side. so it produces a NAN when it tries to compute the matrix-vector product. 
> > > 
> > > Many thanks for your help!
> > > 
> > > Best regards,
> > > Feng
> > > From: Jose E. Roman <jroman at dsic.upv.es>
> > > Sent: 22 August 2022 19:32
> > > To: feng wang <snailsoar at hotmail.com>
> > > Cc: petsc-users at mcs.anl.gov <petsc-users at mcs.anl.gov>
> > > Subject: Re: [petsc-users] Slepc Question, shell-matrix
> > >  
> > > This is very strange. This error appears when the solver employs the B-inner product, but in your case you don't have a B matrix, so you should never see that error. Try running under valgrind to see if it gives more hints.
> > > 
> > > Jose
> > > 
> > > 
> > > > El 22 ago 2022, a las 20:45, feng wang <snailsoar at hotmail.com> escribi?:
> > > > 
> > > > Hello,
> > > > 
> > > > I am new to Slepc and trying to work out the eigenvalues and eigenvectors of my Jacobian matrix. I am using a shell matrix to work out the matrix-vector product and I am using the default Krylov-schur method.  
> > > > 
> > > > My first attempt was not successful and I got the following errors:
> > > > 
> > > > [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> > > > [0]PETSC ERROR: Missing or incorrect user input 
> > > > [0]PETSC ERROR: The inner product is not well defined: indefinite matrix
> > > > [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
> > > > [0]PETSC ERROR: Petsc Release Version 3.17.4, unknown 
> > > > [0]PETSC ERROR: cfdtest on a arch-debug named ming by feng Mon Aug 22 19:21:41 2022
> > > > [0]PETSC ERROR: Configure options --with-cc=mpicc --with-cxx=mpicxx --with-fc=0 PETSC_ARCH=arch-debug
> > > > [0]PETSC ERROR: #1 BV_SafeSqrt() at /home/feng/cfd/slepc-3.17.1/include/slepc/private/bvimpl.h:130
> > > > [0]PETSC ERROR: #2 BV_SquareRoot_Default() at /home/feng/cfd/slepc-3.17.1/include/slepc/private/bvimpl.h:365
> > > > [0]PETSC ERROR: #3 BVOrthogonalizeCGS1() at /home/feng/cfd/slepc-3.17.1/src/sys/classes/bv/interface/bvorthog.c:101
> > > > [0]PETSC ERROR: #4 BVOrthogonalizeGS() at /home/feng/cfd/slepc-3.17.1/src/sys/classes/bv/interface/bvorthog.c:177
> > > > [0]PETSC ERROR: #5 BVOrthonormalizeColumn() at /home/feng/cfd/slepc-3.17.1/src/sys/classes/bv/interface/bvorthog.c:402
> > > > [0]PETSC ERROR: #6 BVMatArnoldi() at /home/feng/cfd/slepc-3.17.1/src/sys/classes/bv/interface/bvkrylov.c:91
> > > > [0]PETSC ERROR: #7 EPSSolve_KrylovSchur_Default() at /home/feng/cfd/slepc-3.17.1/src/eps/impls/krylov/krylovschur/krylovschur.c:261
> > > > [0]PETSC ERROR: #8 EPSSolve() at /home/feng/cfd/slepc-3.17.1/src/eps/interface/epssolve.c:147
> > > > [0]PETSC ERROR: #9 slepc_eigen_comp() at domain/cfd/slepc_eigen_solve.cpp:77
> > > > 
> > > > Could someone please shine some light on this? I have also attached my code. The code is part of my big library and I cannot attach the whole code, sorry about this. but I am happy to provide more information. The attached code has some arrangements for halo exchange, but for the moment, it assumes it is a serial run.
> > > > 
> > > > Many thanks,
> > > > Feng
> > > > <slepc_eigen_solve.cpp>


From knepley at gmail.com  Tue Aug 23 07:41:47 2022
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 23 Aug 2022 08:41:47 -0400
Subject: [petsc-users] Preallocating a matrix with the d_nnz and o_nnz
 arrays
In-Reply-To: <a6fbe8b2-0680-faf4-8b23-f248f7a7bad3@geomag.info>
References: <47e773f6-f008-ec23-7554-ddefc30ddccf@geomag.info>
	<SA1PR09MB8607667B9B068ADE59ACC19588719@SA1PR09MB8607.namprd09.prod.outlook.com>
	<a6fbe8b2-0680-faf4-8b23-f248f7a7bad3@geomag.info>
Message-ID: <CAMYG4GnMEzaKR386cHfeJzdf3739rSxMfFpMz8yvt9xLVV+_yA@mail.gmail.com>

On Mon, Aug 22, 2022 at 12:12 PM Patrick Alken <patrick.alken at geomag.info>
wrote:

> Thank you, I have read that document. I have changed my criteria to:
>
>
> if (j >= first && (j - first) < m) {
>
As Barry says, this should be

if ((j >= first) && (j < last)) {


>   /*diagonal*/
>
> else
>
>   /*off-diagonal*/
>
>
> It seems to be working better. I don't know how to get the c1 and c2
> columns referred to by the documentation.
>
Thanks

    Matt


> Patrick
>
>
>
> On 8/22/22 10:03, Zhang, Hong wrote:
>
> See
> https://petsc.org/release/docs/manualpages/Mat/MatMPIAIJSetPreallocation.html#MatMPIAIJSetPreallocation
> MatMPIAIJSetPreallocation - PETSc
> <https://petsc.org/release/docs/manualpages/Mat/MatMPIAIJSetPreallocation.html#MatMPIAIJSetPreallocation>
> MatMPIAIJSetPreallocation Preallocates memory for a sparse parallel matrix
> in AIJ format (the default parallel PETSc format). For good matrix assembly
> performance the user should preallocate the matrix storage by setting the
> parameters d_nz (or d_nnz) and o_nz (or o_nnz).
> petsc.org
> if you use mpiaij matrix.
> Hong
> ------------------------------
> *From:* petsc-users <petsc-users-bounces at mcs.anl.gov>
> <petsc-users-bounces at mcs.anl.gov> on behalf of Patrick Alken
> <patrick.alken at geomag.info> <patrick.alken at geomag.info>
> *Sent:* Monday, August 22, 2022 10:56 AM
> *To:* petsc-users <petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov>
> *Subject:* [petsc-users] Preallocating a matrix with the d_nnz and o_nnz
> arrays
>
> I am looping through all the non-zero elements of my matrix to count the
> diagonal and off-diagonal terms in order to pre-allocate the matrix.
> However, what is the criteria for knowing if a given element is in the
> diagonal portion or off-diagonal portion? It seems that it will depend
> on row, column, and also processor. I was doing something like this:
>
> MatGetLocalSize(A, &m, &n);
>
> MatGetOwnershipRange(A, &first, &last);
>
> for (i = first; i < last; ++i) {
>
>      for (j = 0; j < N; ++j) {
>
>          if (abs(i - j) < m)
>
>              ++d_nnz[i - first];
>
>          else
>
>              ++o_nnz[i - first];
>
>      }
>
> }
>
> However, the criteria: abs(i - j) < m is not correct. I think the
> correct criteria will include the processor number also. Does anyone
> know the correct criteria to use?
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220823/65f9f714/attachment.html>

From knepley at gmail.com  Tue Aug 23 07:46:58 2022
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 23 Aug 2022 08:46:58 -0400
Subject: [petsc-users] PETSc on fedora 36
In-Reply-To: <65d7f5ad-fcd5-2cc9-0f74-8c9c8f5eaeb8@upc.edu>
References: <aa3cbfe2-2ccc-93cc-6004-5c2ae6429047@upc.edu>
	<358BFF34-D941-42AB-8648-871186EF64DB@petsc.dev>
	<65d7f5ad-fcd5-2cc9-0f74-8c9c8f5eaeb8@upc.edu>
Message-ID: <CAMYG4GkBXMO3t7sMgOXS-Qwns6Uy1Eg3hYg6rref+Hre-u0hxA@mail.gmail.com>

Can you run anything in parallel? Say the small sample code that calculates
pi?


https://www.cs.usfca.edu/~mmalensek/cs220/schedule/code/week09/pi-mpi.c.html

  Thanks,

      Matt

On Tue, Aug 23, 2022 at 6:34 AM Rafel Amer Ramon <rafel.amer at upc.edu> wrote:

>
> Hi,
>
> mpicc and mpirun are from the package openmpi-devel-4.1.2-3.fc36.x86_64
> and the petsc64 library is linked with
> /usr/lib64/openmpi/lib/libmpi.so.40
>
> ~# ldd /lib64/libpetsc64.so.3.16.4
> 	linux-vdso.so.1 (0x00007fff5becc000)
> 	libflexiblas64.so.3 => /lib64/libflexiblas64.so.3 (0x00007f3030e30000)
> 	libcgns.so.4.2 => /usr/lib64/openmpi/lib/libcgns.so.4.2 (0x00007f3030d46000)
> 	libhdf5.so.200 => /usr/lib64/openmpi/lib/libhdf5.so.200 (0x00007f303091a000)
> 	libm.so.6 => /lib64/libm.so.6 (0x00007f303083c000)
> 	libX11.so.6 => /lib64/libX11.so.6 (0x00007f30306f4000)
> 	libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f30304c0000)
> 	libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f303049e000)
> 	libc.so.6 => /lib64/libc.so.6 (0x00007f303029c000)
> 	libgfortran.so.5 => /lib64/libgfortran.so.5 (0x00007f302ffcf000)
> 	libquadmath.so.0 => /lib64/libquadmath.so.0 (0x00007f302ff87000)
> 	/lib64/ld-linux-x86-64.so.2 (0x00007f30327bb000)
> 	libmpi.so.40 => /usr/lib64/openmpi/lib/libmpi.so.40 (0x00007f302fe59000)
> 	libsz.so.2 => /lib64/libsz.so.2 (0x00007f302fe50000)
> 	libz.so.1 => /lib64/libz.so.1 (0x00007f302fe34000)
> 	libxcb.so.1 => /lib64/libxcb.so.1 (0x00007f302fe08000)
> 	libopen-rte.so.40 => /usr/lib64/openmpi/lib/libopen-rte.so.40 (0x00007f302fd4d000)
> 	libopen-pal.so.40 => /usr/lib64/openmpi/lib/libopen-pal.so.40 (0x00007f302fc9f000)
> 	libhwloc.so.15 => /lib64/libhwloc.so.15 (0x00007f302fc42000)
> 	libevent_core-2.1.so.7 => /lib64/libevent_core-2.1.so.7 (0x00007f302fc07000)
> 	libevent_pthreads-2.1.so.7 => /lib64/libevent_pthreads-2.1.so.7 (0x00007f302fc02000)
> 	libXau.so.6 => /lib64/libXau.so.6 (0x00007f302fbfc000)
>
>
> I think that it's correct, but it don't work.
>
> Best regards,
>
> Rafel Amer
>
> El 22/8/22 a les 20:36, Barry Smith ha escrit:
>
>
>   Are you sure the mpirun you are using matches the mpi that PETSc was
> built with?
>
> On Aug 22, 2022, at 2:31 PM, Rafel Amer Ramon <rafel.amer at upc.edu> wrote:
>
> Hi,
>
> I have installed the following packages on fedora 36
>
> ~# rpm -qa | grep petsc
> petsc64-3.16.4-3.fc36.x86_64
> petsc-3.16.4-3.fc36.x86_64
> petsc-openmpi-3.16.4-3.fc36.x86_64
> petsc-openmpi-devel-3.16.4-3.fc36.x86_64
> python3-petsc-openmpi-3.16.4-3.fc36.x86_64
> petsc-devel-3.16.4-3.fc36.x86_64
> petsc64-devel-3.16.4-3.fc36.x86_64
> ~# rpm -qa | grep openmpi
> openmpi-4.1.2-3.fc36.x86_64
> ptscotch-openmpi-6.1.2-2.fc36.x86_64
> scalapack-openmpi-2.1.0-11.fc36.x86_64
> openmpi-devel-4.1.2-3.fc36.x86_64
> MUMPS-openmpi-5.4.1-2.fc36.x86_64
> superlu_dist-openmpi-6.1.1-9.fc36.x86_64
> hdf5-openmpi-1.12.1-5.fc36.x86_64
> hypre-openmpi-2.18.2-6.fc36.x86_64
> petsc-openmpi-3.16.4-3.fc36.x86_64
> fftw-openmpi-libs-double-3.3.10-2.fc36.x86_64
> fftw-openmpi-libs-long-3.3.10-2.fc36.x86_64
> fftw-openmpi-libs-single-3.3.10-2.fc36.x86_64
> fftw-openmpi-libs-3.3.10-2.fc36.x86_64
> fftw-openmpi-devel-3.3.10-2.fc36.x86_64
> petsc-openmpi-devel-3.16.4-3.fc36.x86_64
> python3-petsc-openmpi-3.16.4-3.fc36.x86_64
> scalapack-openmpi-devel-2.1.0-11.fc36.x86_64
> python3-openmpi-4.1.2-3.fc36.x86_64
> hdf5-openmpi-devel-1.12.1-5.fc36.x86_64
> cgnslib-openmpi-4.2.0-6.fc36.x86_64
> cgnslib-openmpi-devel-4.2.0-6.fc36.x86_64
>
> and I try to compile and run the program written in the file e.c
>
> ~# cat e.c
> #include <petsc.h>
>
> int main(int argc, char **argv) {
>   PetscErrorCode ierr;
>   PetscMPIInt    rank;
>
>   ierr = PetscInitialize(&argc,&argv,NULL,
>       "Compute e in parallel with PETSc.\n\n"); if (ierr) return ierr;
>   ierr = MPI_Comm_rank(PETSC_COMM_WORLD,&rank); CHKERRQ(ierr);
>   ierr = PetscPrintf(PETSC_COMM_WORLD,"My rank is %d\n",rank); CHKERRQ(ierr);
>
>
>   return PetscFinalize();
> }
>
> I compile it with the command
>
> ~# mpicc -o e e.c -I/usr/include/petsc64 -lpetsc64
>
> and I run it with
>
> ~# mpirun -np 8 ./eMy rank is 0
> My rank is 0
> My rank is 0
> My rank is 0
> My rank is 0
> My rank is 0
> My rank is 0
> My rank is 0
>
> but all my process get rank 0.
>
> Do you know what I'm doing wrong?
>
> Thank you!!
>
> Best regards,
>
> Rafel Amer
>
>
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220823/9626f401/attachment-0001.html>

From rafel.amer at upc.edu  Tue Aug 23 08:48:20 2022
From: rafel.amer at upc.edu (Rafel Amer Ramon)
Date: Tue, 23 Aug 2022 15:48:20 +0200
Subject: [petsc-users] PETSc on fedora 36
In-Reply-To: <CAMYG4GkBXMO3t7sMgOXS-Qwns6Uy1Eg3hYg6rref+Hre-u0hxA@mail.gmail.com>
References: <aa3cbfe2-2ccc-93cc-6004-5c2ae6429047@upc.edu>
	<358BFF34-D941-42AB-8648-871186EF64DB@petsc.dev>
	<65d7f5ad-fcd5-2cc9-0f74-8c9c8f5eaeb8@upc.edu>
	<CAMYG4GkBXMO3t7sMgOXS-Qwns6Uy1Eg3hYg6rref+Hre-u0hxA@mail.gmail.com>
Message-ID: <acda5081-735d-dc66-ca08-91105d5585e1@upc.edu>

An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220823/e25c2ad0/attachment.html>

From Jiannan_Tu at uml.edu  Tue Aug 23 08:51:39 2022
From: Jiannan_Tu at uml.edu (Tu, Jiannan)
Date: Tue, 23 Aug 2022 13:51:39 +0000
Subject: [petsc-users] Using matrix-free with KSP
In-Reply-To: <F4C76705-B402-4F8B-A42C-DDBE95574065@petsc.dev>
References: <BY5PR02MB6471969F80FA5F732FC74D87E7D79@BY5PR02MB6471.namprd02.prod.outlook.com>
	<B55CFC2C-6364-4B78-8E99-9D53700743B8@petsc.dev>
	<BY5PR02MB6471BDF90783FE3A0EA176CBE7809@BY5PR02MB6471.namprd02.prod.outlook.com>
	<87iloa49kt.fsf@jedbrown.org>
	<BY5PR02MB6471C0ED04769D02046F68ACE7809@BY5PR02MB6471.namprd02.prod.outlook.com>
	<00EBE4B8-22E5-408F-B56D-14226A278D7E@petsc.dev>
	<BY5PR02MB6471E386057DDDE98AD05DFAE7809@BY5PR02MB6471.namprd02.prod.outlook.com>
	<BY5PR02MB647140084C7E7CDDB6BC2A11E76B9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<A6629DE1-5A7F-414E-B164-A69877158675@petsc.dev>
	<BY5PR02MB6471AB1C004920DA1FC9B43EE76B9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<103146FD-FF28-47CE-B661-F2B521BF0E7A@petsc.dev>
	<DM6PR02MB6478B2FB8CB1BE757CE7B003E76D9@DM6PR02MB6478.namprd02.prod.outlook.com>
	<4C00F4B8-BB42-4932-BCD7-527E5FB39BFF@petsc.dev>
	<BY5PR02MB64710E70758E431108D66956E76D9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<D0EB13D8-C5A6-4AE7-AFDE-2CDCCACBAF73@petsc.dev>
	<BY5PR02MB6471D4B05874DECFDDA13FFCE76D9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<367B124C-88BB-4889-A4AF-45A1D473078D@petsc.dev>
	<BY5PR02MB6471F2C058B84DE092567731E76D9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<944DDEED-016D-43B2-A5A6-7BF2E8E41FA5@petsc.dev>
	<BY5PR02MB6471CD0236F03D20763A3B64E76C9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<B4CF5E5B-984D-4EC1-AB77-1FD08C3FBBCF@petsc.dev>
	<BY5PR02MB6471BA72871419781B875967E76C9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<3D179AEA-AD05-4B46-8DDC-9E126741C247@petsc.dev>
	<BY5PR02MB6471560EA70C6FAE826DDF27E7719@BY5PR02MB6471.namprd02.prod.outlook.com>
	<1C67C8DB-0653-4588-8137-E0865E0F2098@petsc.dev>
	<BY5PR02MB6471C5EAE18AB9B114E76232E7719@BY5PR02MB6471.namprd02.prod.outlook.com>
	<FAA1668F-08C1-4BAD-89B4-784D8AD99C36@petsc.dev>
	<BY5PR02MB64715D7FE585A7F88092E04DE7719@BY5PR02MB6471.namprd02.prod.outlook.com>
	<F4C76705-B402-4F8B-A42C-DDBE95574065@petsc.dev>
Message-ID: <BY5PR02MB647100464DF06C7A06F15862E7709@BY5PR02MB6471.namprd02.prod.outlook.com>

Barry,

Indeed, MATFFD is not necessary. MATSHELL works the same but better code.

Thank you again for helping me to solve the problem.

Best,
Jiannan

________________________________
From: Barry Smith <bsmith at petsc.dev>
Sent: Monday, August 22, 2022 2:39 PM
To: Tu, Jiannan <Jiannan_Tu at uml.edu>
Cc: petsc-users at mcs.anl.gov <petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] Using matrix-free with KSP


CAUTION: This email was sent from outside the UMass Lowell network.



On Aug 22, 2022, at 1:37 PM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

Barry,

Yes, you are right, Anear is the preconditioner that needs to be passed to KSPSetOperators() as the second matrix.

I think I have figured out how to provide a preconditioner for matrix-free method. I was confused by matrix A created by MatCreateMFFD(), which is type MFFD. I thought I have to use it as the first matrix passed to KSPSetOperators(). Since it was not assembled (and can't) so that Petsc gave error message "matrix unassembled". In fact, I should ignore A, but use other two matrixes AA and B, which are type MPIAIJ, assemble them, pass them to KSPSetOperators() as first and second matrix. Since function passed into MatMFFDSetFunction() does not take a matrix as an argument, matrix AA needs to be declared in user context so that it can be used in that function to perform matrix-vector product (partially) using MatMult() as you suggested.

   I think your AA should be the shell matrix, and the B should be Anear. There should be no MatMFFD. Inside the AA shell matrix mat-mult you call MatMult( on the Anear put Anear in the Shell matrix context also) and then add the part of the multiply coming from the far field.


I have used a small test program to solve an 8x8 linear equation system and the code produces the same answers with/without the preconditioner. The iteration for this small problem, of course no improvement with the preconditioner but the residual norms with/without the preconditioner are slightly different. I suppose the preconditioner takes effect.

Now I need to think about if the applied preconditioner does the right job on the other part of matrix-vector product (non near part) for the problem I am working on.

Thank you very very much and greatly appreciate your help!

Jiannan
________________________________
From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Monday, August 22, 2022 12:42 PM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] Using matrix-free with KSP


CAUTION: This email was sent from outside the UMass Lowell network.



On Aug 22, 2022, at 12:25 PM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

The matrix-vector product involves part of matrix elements explicitly as shown below so that I need to insert values into this part of the matrix.
<image.png>
Initially I wrote my own code for calculating and storing matrix A^near_mi without using any Petsc matrix and then performed matrix-vector product using above equation. The code runs but converges very slow. Apparently a preconditioned is needed.

I am not clear how Petsc can provide a preconditioner for my case.

  I am now confused. I thought you were forming Anear as a PETSc matrix and passing that to KSPSetOperators as the second matrix and so the preconditioner was being built with Anear?

  For your shell matrix vector product you should call MatMult(Anear,x,y) inside your custom routine and then add the rest of the matrix-vector product using your fast multipole products. What am I missing?

  Barry

________________________________
From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Monday, August 22, 2022 8:42 AM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] Using matrix-free with KSP


CAUTION: This email was sent from outside the UMass Lowell network.



On Aug 21, 2022, at 11:18 PM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

Barry,

You are right. With PC type none, the same error appears.

Now I use MatShell and I literally have three matrixes. A is a shell matrix, B is a preconditioner, and C is a matrix just for preallocation purpose.

First C is used to determine non-zero structure. Then use MatPreallocatorPreallocation(C, PETSC_FALSE, B) to preallocate for B. After that do insertion and assembly for B. These steps work fine.

Now the question is how to insert and assemble shell matrix A. MatPreallocatorPreallocation doesn?t work with shell matrix as indicated by an error message. MatSetValues or Mat assembly doesn?t support shell matrix either.

  A shell matrix is completely under your control. You decide on what data you want to store in it, and you provide the matrix-vector product routine.  MatPreallocatorPreallocation and MatSetValues don't work for shell matrix unless you provide code for them to work, which normally you would not do.

  I don't understand why you want to use MatSetValues for your shell matrix. If you can use MatSetValues() and just build an explicit matrix, then why would you use a shell matrix, just use MATAIJ matrix?

  Or does part of the operation inside your MatShell require a "regular" PETSc matrix? This would be fine, but it is your responsibility to design your Mat shell context to contain the "regular matrix" and your responsibility to fill up that regular matrix inside your shell context, calling whatever code you choose to write. So for example your shell context would be a C struct and one of the fields would be a regular PETSc matrix, also part of your custom matrix-vector product routine would call a regular MatMult() on the regular PETSc matrix stored inside the C struct.

  Barry



Thank you,
Jiannan

From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Friday, August 19, 2022 4:02 PM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] Using matrix-free with KSP

CAUTION: This email was sent from outside the UMass Lowell network.


  I don't know why changing the PCType could matter, can you please try with your new code but use the -pc_type none option to turn off the use of the preconditioner to see if the error still appears (I think it should).

  Also if you have a custom function that can do the matrix-vector product (not using any differencing) then you should use MatShell() and provide the matrix-vector product operation with your code.

  Barry



On Aug 19, 2022, at 11:47 AM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

Barry,

I have my own matrix-vector product but not use MatShell. I followed your previous instruction to use

    MatCreateMFFD(MPI_COMM_WORLD, PETSC_DECIDE, PETSC_DECIDE, N, N, &A);
    MatMFFDSetFunction(A, formfunction, &params);
    MatSetFromOptions(A);
    MatMFFDSetBase(A, X, NULL);

The customized matrix-vector product is implemented in formfunction() provided to MatMFFDSetFunction(). In this way, the matrix A is not used. I think that is why Petsc complains the matrix is not assembled. However, when PC type is none, the code runs without error although KSP solver doesn?t converge.

The question then becomes how to link A to matrix-free matrix-vector multiplication. And where to put MatShell?

Thank you,
Jiannan

From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Friday, August 19, 2022 10:27 AM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] Using matrix-free with KSP

CAUTION: This email was sent from outside the UMass Lowell network.





On Aug 18, 2022, at 11:41 PM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

The program calls KSPSolve just once and the error occurs after the first iteration.

I realized the matrix-vector product in my case is different from that approximating matrix-vector multiply by finite difference of function vector. It consists of two parts. One is multiplication of part of the matrix with part of vector, and another part results from so called fast multipole method. I?m not clear how to relate such calculation of matrix-vector product to the finite differencing of function vector.

  Are you using finite differencing at all in a matrix-vector product, or do you have your own custom matrix-vector product code (with MatShell)?

  Anyways the easiest thing to do is

  use the run time option -on_error_attach_debugger noxterm and run the code,

  when it stops in the debugger you can type bt (for backtrace)

  then do up a few times to move up the stack until you get to the PETSc function call that generates the error

  you can then do "call MatView(thematrixvarible,0) to have PETSc display what the matrix is and

  you can also do print *thematrixvariable to see the exact struct with all its fields

  If this doesn't tell you why the troublesome matrix is not assembled then cut and paste ALL of the output and email it

  Barry






From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Thursday, August 18, 2022 7:44 PM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] Using matrix-free with KSP

CAUTION: This email was sent from outside the UMass Lowell network.



   Is it in the first iteration of the first KSPSolve that you get this message? Or in the second call to KSPSolve?

  The error comes from a matrix vector product with your A matrix, what is that and where did it come from. The error message is not from your B matrix, that is not used in the MatMult()





On Aug 18, 2022, at 3:05 PM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

Barry,

Thanks. The method works. Assembly of actual matrix now completes quickly. The convergency is much better for the first iteration. The residual normal is in the order of 10^-10 instead of 10^-2. However, there is an error message after the first iteration: ?Object is in wrong state. Not for unassembled matrix? even though both preallocator and actual matrix have been assembled (see the code snippet below). Could you please give me some hints what?s going wrong? I use KSP type bcgs and PC type mat

Thank you,
Jiannan

-----------------------------------------------------
Solve linear equations ...
  0 KSP Residual norm 9.841179147519e-10
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Object is in wrong state
[0]PETSC ERROR: Not for unassembled matrix
[0]PETSC ERROR: See https://petsc.org/release/faq/<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Frelease%2Ffaq%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7Cc1e9f88adcda42f7c8b408da846dae97%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637967903891169434%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=aQgMnGt515eFE4s6OnTPPOylUFUaLPCpO678MJNvBQs%3D&reserved=0> for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.16.6, Mar 30, 2022
[0]PETSC ERROR: ./iesolver on a  named REXNET-WS4 by jiannantu Thu Aug 18 14:48:43 2022
[0]PETSC ERROR: Configure options --download-f2cblaslapack=yes --with-mpi-dir=/usr/local --with-fc=0 --prefix=/home/jiannantu/petsc --with-scalar-type=complex --with-64-bit-indices=1
[0]PETSC ERROR: #1 MatMult() at /home/jiannantu/petsc-3.16.6/src/mat/interface/matrix.c:2425
[0]PETSC ERROR: #2 PCApplyBAorAB() at /home/jiannantu/petsc-3.16.6/src/ksp/pc/interface/precon.c:730
[0]PETSC ERROR: #3 KSP_PCApplyBAorAB() at /home/jiannantu/petsc-3.16.6/include/petsc/private/kspimpl.h:421
[0]PETSC ERROR:  Linear equations not solved. Diverged reason: 0
#4 KSPSolve_BCGS() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/impls/bcgs/bcgs.c:87
[0]PETSC ERROR: #5 KSPSolve_Private() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itfunc.c:914
[0]PETSC ERROR: #6 KSPSolve() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itfunc.c:1086

-----------------------------------------------
    MatCreate(MPI_COMM_WORLD, &B);
    MatSetType(B, MATMPIAIJ);
    MatSetSizes(B, PETSC_DECIDE, PETSC_DECIDE, N, N);

    Mat preall;
    MatCreate(MPI_COMM_WORLD, &preall);
    MatSetType(preall, MATPREALLOCATOR);
    MatSetSizes(preall, PETSC_DECIDE, PETSC_DECIDE, N, N);

    MatMPIAIJSetPreallocation(preall, 0, d_nnz, 0, o_nnz);
    MatSetUp(preall);
MatSetOption(preall, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE);

//set values row by row
MatSetValues(preall, 1, &m, 1, nCols, matCols, INSERT_VALUES);

MatAssemblyBegin(preall, MAT_FINAL_ASSEMBLY);
MatAssemblyEnd(preall, MAT_FINAL_ASSEMBLY);

//set values for actual matrix row by row
MatSetValues(B, 1, &m, 1, nCols, matCols, INSERT_VALUES);

MatAssemblyBegin(B, MAT_FINAL_ASSEMBLY);
MatAssemblyEnd(B, MAT_FINAL_ASSEMBLY);

    KSPCreate(MPI_COMM_WORLD, &ksp);
    KSPSetOperators(ksp, A, B);
    KSPSetFromOptions(ksp);

    PC pc;
    KSPGetPC(ksp, &pc);
    PCSetFromOptions(pc);

From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Thursday, August 18, 2022 12:59 PM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] Using matrix-free with KSP

CAUTION: This email was sent from outside the UMass Lowell network.


  Yes, once you have preallocated the real matrix you can destroy the preallocation matrix whose only job is to gather the preallocation information





On Aug 18, 2022, at 12:52 PM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

Thanks, Barry.

So I need actually two matrixes, one for preallocator and one for actual matrix that can be passed to KSPSetOperators(). -mat_type preallocator option is used to speed up doing insertion into preallocator, then use MatPreallocatorPreallocate() to preallocate actual matrix and do actual insertion of values into it., right?

The code runs in parallel. Each process owns number of rows that equals to number of unknowns (that is, xm in 1D DM) it owns.

Jiannan

From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Thursday, August 18, 2022 11:37 AM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] Using matrix-free with KSP

CAUTION: This email was sent from outside the UMass Lowell network.


   The preallocator MatType matrix cannot be passed to the KSPSetOperators(), you need to create an actual matrix, for example MATAIJ and use the preallocator to set its preallocation and then fill up the MATAIJ matrix with the usual MatSetValues and pass that matrix to the KSPSetOperators.






On Aug 18, 2022, at 11:30 AM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

Hi Barry,

The MATPREALLOCATOR solved problem of slow matrix assembly. But the solver failed because of ?Matrix type preallocator does not have a multiply defined?.

I guess this is because the matrix-free is used. I am wondering how a preconditioner is applied to the matrix-vector multiply in Petsc.

Thank you,
Jiannan

[0]PETSC ERROR: No support for this operation for this object type
[0]PETSC ERROR: Matrix type preallocator does not have a multiply defined
[0]PETSC ERROR: See https://petsc.org/release/faq/<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Frelease%2Ffaq%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7Cc1e9f88adcda42f7c8b408da846dae97%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637967903891169434%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=aQgMnGt515eFE4s6OnTPPOylUFUaLPCpO678MJNvBQs%3D&reserved=0> for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.16.6, Mar 30, 2022
[0]PETSC ERROR: ./iesolver on a  named REXNET-WS4 by jiannantu Thu Aug 18 11:21:55 2022
[0]PETSC ERROR: Configure options --download-f2cblaslapack=yes --with-mpi-dir=/usr/local --with-fc=0 --prefix=/home/jiannantu/petsc --with-scalar-type=complex --with-64-bit-indices=1
[0]PETSC ERROR: #1 MatMult() at /home/jiannantu/petsc-3.16.6/src/mat/interface/matrix.c:2437
[0]PETSC ERROR: #2 PCApply_Mat() at /home/jiannantu/petsc-3.16.6/src/ksp/pc/impls/mat/pcmat.c:9
[0]PETSC ERROR: #3 PCApply() at /home/jiannantu/petsc-3.16.6/src/ksp/pc/interface/precon.c:445
[0]PETSC ERROR: #4 KSP_PCApply() at /home/jiannantu/petsc-3.16.6/include/petsc/private/kspimpl.h:382
[0]PETSC ERROR: #5 KSPInitialResidual() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itres.c:65
[0]PETSC ERROR: #6 KSPSolve_BCGS() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/impls/bcgs/bcgs.c:43
[0]PETSC ERROR: #7 KSPSolve_Private() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itfunc.c:914
[0]PETSC ERROR: #8 KSPSolve() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itfunc.c:1086

From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Thursday, August 18, 2022 8:35 AM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] Using matrix-free with KSP

CAUTION: This email was sent from outside the UMass Lowell network.


  Slow assembly is usually due to under preallocation. You can change to using

MatSetOption(B, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_TRUE);

  to detect if you are under preallocating. See https://petsc.org/main/docs/manual/mat/#sec-matsparse<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanual%2Fmat%2F%23sec-matsparse&data=05%7C01%7CJiannan_Tu%40uml.edu%7Cc1e9f88adcda42f7c8b408da846dae97%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637967903891169434%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=p1XPEBBZycyyYnbb2U3ESQGlQV44ZfqzwYL857jOz5c%3D&reserved=0>. That section clearly needs some sprucing up. If it is difficult to determine good preallocation values you can use https://petsc.org/main/docs/manualpages/Mat/MATPREALLOCATOR/<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMATPREALLOCATOR%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7Cc1e9f88adcda42f7c8b408da846dae97%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637967903891169434%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=YE9ST8%2B1vojZ1NlVkzQm26VjaiGZ4RJOya7NGe7avJs%3D&reserved=0> to compute the needed preallocation efficiently.

  Are you running in parallel? If so how are you determining which rows of entries to compute on each MPI rank? You will want most of the rows to be computed on the rank where the values are stored, you can determine this with https://petsc.org/main/docs/manualpages/Mat/MatGetOwnershipRange.html<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMatGetOwnershipRange.html&data=05%7C01%7CJiannan_Tu%40uml.edu%7Cc1e9f88adcda42f7c8b408da846dae97%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637967903891169434%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=%2B0kDehpllbZjZGY5HvxmY%2BQ2cmKO4AX5DkNCQlbllsg%3D&reserved=0>

  Barry

















On Aug 18, 2022, at 12:50 AM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

I implemented the preconditioner. The solver converges quickly when the problem size is small. But when the size increases, say to 100k unknowns, the code hangs at assembly of the preconditioner matrix. The function call sequence is like this

   MatCreate(MPI_COMM_WORLD, &B);
    MatSetType(B, MATMPIAIJ);
    MatSetSizes(B, PETSC_DECIDE, PETSC_DECIDE, N, N);
    MatSetFromOptions(B);

    // The number of diagonal and off diagonal non zeros only can be determined run-time.
    MatMPIAIJSetPreallocation(B, 0, d_nnz, 0, o_nnz);
    MatSetOption(B, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE);

    //here insert values row by row.
MatSetValues(B, 1, &m, nn, nCols, matCols, INSERT_VALUES);

MatAssemblyBegin(B, MAT_FINAL_ASSEMBLY);
MatAssemblyEnd(B, MAT_FINAL_ASSEMBLY);

Could you please tell me what I have done wrong?

Thank you,
Jiannan

From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Tuesday, August 16, 2022 2:00 PM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] Using matrix-free with KSP

CAUTION: This email was sent from outside the UMass Lowell network.


   Create another matrix B that contains "Selected parts of the matrix can be pre-calculated and stored to serve as the preconditioner." When you call KSPSetOperators() pass in B as the second matrix argument.

   Now if literally application of B is the preconditioner you want to use, then simply use PCMAT as the preconditioner, and it will apply B each time it needs to apply the preconditioner. With this, you do not even need a PCShell.

   If your preconditioner uses B to construct your own specific data structure needed to apply your preconditioner, then you can use PCShellSetSetUp() and have your routine call PCGetOperators() to pull out the B matrix and use it to construct your preconditioner.

   If you want PETSc to construct a standard preconditioner from B, then you don't need the PCShell; you simply pass in the B, as above, and use -pc_type type to set the preconditioner you want to use.

   Barry









On Aug 16, 2022, at 12:31 PM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

Barry,

Thank you very much for your instructions. I am sorry I may not ask clearer questions.

I don't use SNES. What I am trying to solve is a very large linear equation system with dense and ill-conditioned matrix. Selected parts of the matrix can be pre-calculated and stored to serve as the preconditioner. My question is how I can apply this matrix as the preconditioner.

Here is the part of code from my linear equations solver. MatrixFreePreconditioner is supposed to be the routine to provide the preconditioner. But I don't understand how to use it. Should I use SNESSetJacobian() even if this is a linear problem?

    MatCreateMFFD(MPI_COMM_WORLD, PETSC_DECIDE, PETSC_DECIDE, N, N, &A);
    MatMFFDSetFunction(A, formfunction, &params);
    MatSetFromOptions(A);
    MatMFFDSetBase(A, X, NULL);

    KSPCreate(MPI_COMM_WORLD, &ksp);
    KSPSetOperators(ksp, A, A);
    KSPSetFromOptions(ksp);

    PC pc;
    KSPGetPC(ksp,&pc);
    PCSetType(pc,PCSHELL);
    PCShellSetApply(pc,MatrixFreePreconditioner);

Thank you,
Jiannan

________________________________
From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Tuesday, August 16, 2022 10:10 AM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] Using matrix-free with KSP

CAUTION: This email was sent from outside the UMass Lowell network.


  I don't fully understand your question so I will answer questions that I do know the answer to :-)

  The function you provide to PCShellSetApply() applies the preconditioner to the input vector and puts the result in the output vector.  This input vector changes at every application of the preconditioner so should not be used in constructing the preconditioner.

  If you are using the standard PETSc matrix-free matrix-vector product via finite difference, -snes_mf_operator or via MatCreateSNESMF() or you are using a MatShell, the matrix-vector product routines input changes for each multiply and should not be used in constructing the preconditioner.

  The function you provide with SNESSetJacobian() is where you can compute anything you need to help build your preconditioner. The input vector to that function is the location at which the Jacobian is needed and is what you should use to build your approximate/part of Jacobian. You can store your approximate/part of Jacobian that you need to compute in the pmat and then in a PCShellSetUp() function that you provide you will compute what you will need latter to apply the preconditioner in your PCShellSetApply() function. Note that the values you put in the pmat do not need to be the full Jacobian; they can be anything you want since this pmat is not used to apply the matrix vector product.

  If this is unclear, feel free to ask additional questions.

  Barry









On Aug 16, 2022, at 9:40 AM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

I am trying to apply a user-defined preconditioner to speed up the convergency of matrix-free KSP solver. The user's manual provides an example of how to call the user-defined preconditioner routine. The routine has PC, input Vec and output Vec as arguments. I want to use part of the Jacobian as preconditioner matrix, the elements of which can be calculated using input Vec. My question is what the role of output Vec from the routine is in matrix-vector product? Or I just have the preconditioner matrix elements calculated in the routine and then Petsc handles applying the preconditioner to matrix-vector product? I used PCSetType(pc, PCSHELL) and PCShellSetApply.

Thank you and appreciate your help!

Jiannan
________________________________
From: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Sent: Wednesday, July 6, 2022 2:34 PM
To: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Cc: Jed Brown <jed at jedbrown.org<mailto:jed at jedbrown.org>>; petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] Using matrix-free with KSP

Barry,

VecScatterCreateToAll solves the problem! The code now gives the correct answer with multiple processes.

Thank you all so much!

Jiannan
________________________________
From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Wednesday, July 6, 2022 2:08 PM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Cc: Jed Brown <jed at jedbrown.org<mailto:jed at jedbrown.org>>; petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] Using matrix-free with KSP


  So your operator is a "dense" operator in that its matrix representation would be essentially dense. In that case, you can use specialized routines to do this efficiently. You can use VecScatterCreateToAll() and then VecScatterBegin/End to do the communication.

  Barry










On Jul 6, 2022, at 1:39 PM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>  wrote:

Jed, thank you very much for the reply.

I'm not sure GlobaltoLocal will work. Say the solution vector is of length 10. Two processes then each process can see 5 elements of the vector. For A x = b (10 equations), each process performs matrix-vector product for five equations. And for each equation i , sum_j A_ij *  x_j = b_i requires the process has access to all 10 unknows x_i (i=0,1,...,9).

Can VecScatter and PetscSF make the entire vector accessible to each process? I am reading the manual and trying to understand how VecScatter and PetscSF work.

Jiannan
________________________________
From: Jed Brown <jed at jedbrown.org<mailto:jed at jedbrown.org>>
Sent: Wednesday, July 6, 2022 10:09 AM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>; Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] Using matrix-free with KSP

You'll usually have a GlobalToLocal operation for each rank to get the halo data it needs, then either a LocalToGlobal to collect the result (e.g., finite element methods) or the local compute will write directly into the owned portion of the global vector. If you're doing the communication "raw", then you may find VecScatter or PetscSF useful to perform the necessary communication.

"Tu, Jiannan" <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>  writes:

> Hi Barry,
>
> Following your instructions I implemented the matrix-free method to solve a large linear equation system resulted from a surface integration equation. The KSP solver works fine for single process, but it is problematic with multiple processes. The problem is that each process only can access its own part of solution vector so that each process only conducts part of matrix-vector multiplication. MPI can be used to assemble these partial matrix-vector multiplication together.  Does Petsc provide any ways to implement multi-process matrix-free method?
>
> Thanks,
> Jiannan
> ________________________________
> From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
> Sent: Tuesday, May 24, 2022 2:12 PM
> To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
> Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
> Subject: Re: [petsc-users] Using matrix-free with KSP
>
> This e-mail originated from outside the UMass Lowell network.
> ________________________________
>
>    You can use MatCreateMFFD https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMatCreateMFFD%2F&amp;data=05%7C01%7CJiannan_Tu%40uml.edu%7Cab0859d8d154471590bc08da5f5918d6%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637927133525745809%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=A7wqegTfh94No5BpDiWLK3VxOuR44U2wlWHVm2k7l60%3D&amp;reserved=0<https://urldefense.com/v3/__https://petsc.org/main/docs/manualpages/Mat/MatCreateMFFD/__;!!PVKG_VDCxu5g!stxq2NWGVjUihvHBQC9Rlk0aHVoowcy0PTQYMcQxQ4DqOEC05KSw6TmstSXKckiUgelHzy4ue-d10-zlXkw$><https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMatCreateMFFD%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7Cc1e9f88adcda42f7c8b408da846dae97%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637967903891325655%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=OH0Bs%2BaniUQaAhidHWg0JNSsakEXfTaddvArBg%2BNbwU%3D&reserved=0> MatMFFDSetFunction MatSetFromOptions MatMFFDSetBase and provide the matrix to KSP. Note you will need to use -pc_type none or provide your own custom preconditioner withhttps://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FPC%2FPCSHELL%2F&amp;data=05%7C01%7CJiannan_Tu%40uml.edu%7Cab0859d8d154471590bc08da5f5918d6%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637927133525745809%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=Gb%2F9uFG3jp%2FbfWaSh2cSEQLItHS9CuLwarzN0KcNiJY%3D&amp;reserved=0<https://urldefense.com/v3/__https://petsc.org/main/docs/manualpages/PC/PCSHELL/__;!!PVKG_VDCxu5g!stxq2NWGVjUihvHBQC9Rlk0aHVoowcy0PTQYMcQxQ4DqOEC05KSw6TmstSXKckiUgelHzy4ue-d1cA4fKGw$><https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FPC%2FPCSHELL%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7Cc1e9f88adcda42f7c8b408da846dae97%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637967903891325655%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=jyMwuz6XPcmbNa9KliKnH0qQo0wo6qZvdrZtZRvXuxA%3D&reserved=0>
>
>
>
> On May 24, 2022, at 1:21 PM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu><mailto:Jiannan_Tu at uml.edu>> wrote:
>
> I want to use a matrix-free matrix to solve a large linear equation system because the matrix is too large to be stored. Petsc user manual describes matrix-free method for SNES with examples. The matrix-free matrices section explains how to set up such a matrix, but I can't find any example of matrix-free method with KSP. I am wondering how to apply the method to KSP iterative solver in parallel.
>
> I greatly appreciate your help for me to understand this topic.
>
> Jiannan Tu

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220823/44d6ca4e/attachment-0001.html>

From knepley at gmail.com  Tue Aug 23 09:04:27 2022
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 23 Aug 2022 10:04:27 -0400
Subject: [petsc-users] PETSc on fedora 36
In-Reply-To: <acda5081-735d-dc66-ca08-91105d5585e1@upc.edu>
References: <aa3cbfe2-2ccc-93cc-6004-5c2ae6429047@upc.edu>
	<358BFF34-D941-42AB-8648-871186EF64DB@petsc.dev>
	<65d7f5ad-fcd5-2cc9-0f74-8c9c8f5eaeb8@upc.edu>
	<CAMYG4GkBXMO3t7sMgOXS-Qwns6Uy1Eg3hYg6rref+Hre-u0hxA@mail.gmail.com>
	<acda5081-735d-dc66-ca08-91105d5585e1@upc.edu>
Message-ID: <CAMYG4GmN_+VRt_cMf+KROyHDMe6SMyycwEeBtGJCfw3qC-0fFA@mail.gmail.com>

On Tue, Aug 23, 2022 at 9:48 AM Rafel Amer Ramon <rafel.amer at upc.edu> wrote:

>
> Hi,
>
> yes, I can compile and run it:
>
> ~$ mpicc -o pi-mpi pi-mpi.c -lm
> ~$ mpirun -np 16 --hostfile ~/hosts ./pi-mpi
> Number of intervals: 2048
> Result:   3.1411043724
> Accuracy: 0.0004882812
> Time:     0.0314383930
> ~$
>
>
> PETSc does not control the launch, so you must be using a different MPI.
For building the PETSc executable, try instead
the minimum makefile

e: e.o
      ${CLINKER} -o e e.o ${PETSC_LIB}

include ${PETSC_DIR}/lib/petsc/conf/variables
include ${PETSC_DIR}/lib/petsc/conf/rules

  Thanks,

     Matt


> Best regards,
>
> Rafel Amer
>
>
>
>
> El 23/8/22 a les 14:46, Matthew Knepley ha escrit:
>
> Can you run anything in parallel? Say the small sample code that
> calculates pi?
>
>
> https://www.cs.usfca.edu/~mmalensek/cs220/schedule/code/week09/pi-mpi.c.html
>
>   Thanks,
>
>       Matt
>
> On Tue, Aug 23, 2022 at 6:34 AM Rafel Amer Ramon <rafel.amer at upc.edu>
> wrote:
>
>>
>> Hi,
>>
>> mpicc and mpirun are from the package openmpi-devel-4.1.2-3.fc36.x86_64
>> and the petsc64 library is linked with
>> /usr/lib64/openmpi/lib/libmpi.so.40
>>
>> ~# ldd /lib64/libpetsc64.so.3.16.4
>> 	linux-vdso.so.1 (0x00007fff5becc000)
>> 	libflexiblas64.so.3 => /lib64/libflexiblas64.so.3 (0x00007f3030e30000)
>> 	libcgns.so.4.2 => /usr/lib64/openmpi/lib/libcgns.so.4.2 (0x00007f3030d46000)
>> 	libhdf5.so.200 => /usr/lib64/openmpi/lib/libhdf5.so.200 (0x00007f303091a000)
>> 	libm.so.6 => /lib64/libm.so.6 (0x00007f303083c000)
>> 	libX11.so.6 => /lib64/libX11.so.6 (0x00007f30306f4000)
>> 	libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f30304c0000)
>> 	libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f303049e000)
>> 	libc.so.6 => /lib64/libc.so.6 (0x00007f303029c000)
>> 	libgfortran.so.5 => /lib64/libgfortran.so.5 (0x00007f302ffcf000)
>> 	libquadmath.so.0 => /lib64/libquadmath.so.0 (0x00007f302ff87000)
>> 	/lib64/ld-linux-x86-64.so.2 (0x00007f30327bb000)
>> 	libmpi.so.40 => /usr/lib64/openmpi/lib/libmpi.so.40 (0x00007f302fe59000)
>> 	libsz.so.2 => /lib64/libsz.so.2 (0x00007f302fe50000)
>> 	libz.so.1 => /lib64/libz.so.1 (0x00007f302fe34000)
>> 	libxcb.so.1 => /lib64/libxcb.so.1 (0x00007f302fe08000)
>> 	libopen-rte.so.40 => /usr/lib64/openmpi/lib/libopen-rte.so.40 (0x00007f302fd4d000)
>> 	libopen-pal.so.40 => /usr/lib64/openmpi/lib/libopen-pal.so.40 (0x00007f302fc9f000)
>> 	libhwloc.so.15 => /lib64/libhwloc.so.15 (0x00007f302fc42000)
>> 	libevent_core-2.1.so.7 => /lib64/libevent_core-2.1.so.7 (0x00007f302fc07000)
>> 	libevent_pthreads-2.1.so.7 => /lib64/libevent_pthreads-2.1.so.7 (0x00007f302fc02000)
>> 	libXau.so.6 => /lib64/libXau.so.6 (0x00007f302fbfc000)
>>
>>
>> I think that it's correct, but it don't work.
>>
>> Best regards,
>>
>> Rafel Amer
>>
>> El 22/8/22 a les 20:36, Barry Smith ha escrit:
>>
>>
>>   Are you sure the mpirun you are using matches the mpi that PETSc was
>> built with?
>>
>> On Aug 22, 2022, at 2:31 PM, Rafel Amer Ramon <rafel.amer at upc.edu> wrote:
>>
>> Hi,
>>
>> I have installed the following packages on fedora 36
>>
>> ~# rpm -qa | grep petsc
>> petsc64-3.16.4-3.fc36.x86_64
>> petsc-3.16.4-3.fc36.x86_64
>> petsc-openmpi-3.16.4-3.fc36.x86_64
>> petsc-openmpi-devel-3.16.4-3.fc36.x86_64
>> python3-petsc-openmpi-3.16.4-3.fc36.x86_64
>> petsc-devel-3.16.4-3.fc36.x86_64
>> petsc64-devel-3.16.4-3.fc36.x86_64
>> ~# rpm -qa | grep openmpi
>> openmpi-4.1.2-3.fc36.x86_64
>> ptscotch-openmpi-6.1.2-2.fc36.x86_64
>> scalapack-openmpi-2.1.0-11.fc36.x86_64
>> openmpi-devel-4.1.2-3.fc36.x86_64
>> MUMPS-openmpi-5.4.1-2.fc36.x86_64
>> superlu_dist-openmpi-6.1.1-9.fc36.x86_64
>> hdf5-openmpi-1.12.1-5.fc36.x86_64
>> hypre-openmpi-2.18.2-6.fc36.x86_64
>> petsc-openmpi-3.16.4-3.fc36.x86_64
>> fftw-openmpi-libs-double-3.3.10-2.fc36.x86_64
>> fftw-openmpi-libs-long-3.3.10-2.fc36.x86_64
>> fftw-openmpi-libs-single-3.3.10-2.fc36.x86_64
>> fftw-openmpi-libs-3.3.10-2.fc36.x86_64
>> fftw-openmpi-devel-3.3.10-2.fc36.x86_64
>> petsc-openmpi-devel-3.16.4-3.fc36.x86_64
>> python3-petsc-openmpi-3.16.4-3.fc36.x86_64
>> scalapack-openmpi-devel-2.1.0-11.fc36.x86_64
>> python3-openmpi-4.1.2-3.fc36.x86_64
>> hdf5-openmpi-devel-1.12.1-5.fc36.x86_64
>> cgnslib-openmpi-4.2.0-6.fc36.x86_64
>> cgnslib-openmpi-devel-4.2.0-6.fc36.x86_64
>>
>> and I try to compile and run the program written in the file e.c
>>
>> ~# cat e.c
>> #include <petsc.h>
>>
>> int main(int argc, char **argv) {
>>   PetscErrorCode ierr;
>>   PetscMPIInt    rank;
>>
>>   ierr = PetscInitialize(&argc,&argv,NULL,
>>       "Compute e in parallel with PETSc.\n\n"); if (ierr) return ierr;
>>   ierr = MPI_Comm_rank(PETSC_COMM_WORLD,&rank); CHKERRQ(ierr);
>>   ierr = PetscPrintf(PETSC_COMM_WORLD,"My rank is %d\n",rank); CHKERRQ(ierr);
>>
>>
>>   return PetscFinalize();
>> }
>>
>> I compile it with the command
>>
>> ~# mpicc -o e e.c -I/usr/include/petsc64 -lpetsc64
>>
>> and I run it with
>>
>> ~# mpirun -np 8 ./eMy rank is 0
>> My rank is 0
>> My rank is 0
>> My rank is 0
>> My rank is 0
>> My rank is 0
>> My rank is 0
>> My rank is 0
>>
>> but all my process get rank 0.
>>
>> Do you know what I'm doing wrong?
>>
>> Thank you!!
>>
>> Best regards,
>>
>> Rafel Amer
>>
>>
>>
>>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220823/29a45c60/attachment.html>

From rafel.amer at upc.edu  Tue Aug 23 09:33:16 2022
From: rafel.amer at upc.edu (Rafel Amer Ramon)
Date: Tue, 23 Aug 2022 16:33:16 +0200
Subject: [petsc-users] PETSc on fedora 36
In-Reply-To: <CAMYG4GmN_+VRt_cMf+KROyHDMe6SMyycwEeBtGJCfw3qC-0fFA@mail.gmail.com>
References: <aa3cbfe2-2ccc-93cc-6004-5c2ae6429047@upc.edu>
	<358BFF34-D941-42AB-8648-871186EF64DB@petsc.dev>
	<65d7f5ad-fcd5-2cc9-0f74-8c9c8f5eaeb8@upc.edu>
	<CAMYG4GkBXMO3t7sMgOXS-Qwns6Uy1Eg3hYg6rref+Hre-u0hxA@mail.gmail.com>
	<acda5081-735d-dc66-ca08-91105d5585e1@upc.edu>
	<CAMYG4GmN_+VRt_cMf+KROyHDMe6SMyycwEeBtGJCfw3qC-0fFA@mail.gmail.com>
Message-ID: <8848fba2-d1fb-f5e6-da12-f5ad9985aeae@upc.edu>

An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220823/0d33e5f8/attachment-0001.html>

From knepley at gmail.com  Tue Aug 23 10:21:17 2022
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 23 Aug 2022 11:21:17 -0400
Subject: [petsc-users] PETSc on fedora 36
In-Reply-To: <8848fba2-d1fb-f5e6-da12-f5ad9985aeae@upc.edu>
References: <aa3cbfe2-2ccc-93cc-6004-5c2ae6429047@upc.edu>
	<358BFF34-D941-42AB-8648-871186EF64DB@petsc.dev>
	<65d7f5ad-fcd5-2cc9-0f74-8c9c8f5eaeb8@upc.edu>
	<CAMYG4GkBXMO3t7sMgOXS-Qwns6Uy1Eg3hYg6rref+Hre-u0hxA@mail.gmail.com>
	<acda5081-735d-dc66-ca08-91105d5585e1@upc.edu>
	<CAMYG4GmN_+VRt_cMf+KROyHDMe6SMyycwEeBtGJCfw3qC-0fFA@mail.gmail.com>
	<8848fba2-d1fb-f5e6-da12-f5ad9985aeae@upc.edu>
Message-ID: <CAMYG4Gn8q-BKATqoGGjduL+vk4wp-t09A+LAGB7-HwJkir01qg@mail.gmail.com>

On Tue, Aug 23, 2022 at 10:33 AM Rafel Amer Ramon <rafel.amer at upc.edu>
wrote:

> Hi,
>
> the files rules and variables are in /usr/lib64/petsc/conf, so I have set
>
> ~$ export PETSC_DIR=/usr
>
> ~$ cat Makefile
>
> e: e.o
>     ${CLINKER} -o e e.o ${PETSC_LIB}
>
> include ${PETSC_DIR}/lib64/petsc/conf/variables
> include ${PETSC_DIR}/lib64/petsc/conf/rules
>
> ~$
>
> but, when I try to compile the program, I get the error
>
> ~$ make e
> gcc -Wl,-z,relro -Wl,--as-needed  -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -Wl,--build-id=sha1 -Wl,-dT,/builddir/build/BUILD/petsc-3.16.4/.package_note-petsc-3.16.4-3.fc36.x86_64.ld -fPIC -Wl,-z,relro -Wl,--as-needed  -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -Wl,--build-id=sha1 -Wl,-dT,/builddir/build/BUILD/petsc-3.16.4/.package_note-petsc-3.16.4-3.fc36.x86_64.ld -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64  -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -O3 -fPIC -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64  -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -O3  -o e e.o -L/usr/lib64 -L/usr//lib -lpetsc -lsuperlu -lflexiblas -lcgns -lhdf5 -lhdf5_hl -lmetis -lm -lX11 -lquadmath -lstdc++ -ldl
> /usr/bin/ld: cannot open linker script file /builddir/build/BUILD/petsc-3.16.4/.package_note-petsc-3.16.4-3.fc36.x86_64.ld: El fitxer o directori no existeix
> collect2: error: ld returned 1 exit status
> make: *** [Makefile:2: e] Error 1
>
>
> In the file /usr/lib64/petsc/conf/petscvariables, there are some
> references to the inexistent file
> /builddir/build/BUILD/petsc-3.16.4/.package_note-petsc-3.16.4-3.fc36.x86_64.ld:
>
> ~$ grep /builddir/build/BUILD/petsc-3.16.4/.package_note-petsc-3.16.4-3.fc36.x86_64.ld /usr/lib64/petsc/conf/petscvariables
> CXX_LINKER_FLAGS = -Wl,-z,relro -Wl,--as-needed  -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -Wl,--build-id=sha1 -Wl,-dT,/builddir/build/BUILD/petsc-3.16.4/.package_note-petsc-3.16.4-3.fc36.x86_64.ld -fPIC  -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64  -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -O3 -fPIC -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64  -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -O3
> PCC_LINKER_FLAGS = -Wl,-z,relro -Wl,--as-needed  -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -Wl,--build-id=sha1 -Wl,-dT,/builddir/build/BUILD/petsc-3.16.4/.package_note-petsc-3.16.4-3.fc36.x86_64.ld -fPIC -Wl,-z,relro -Wl,--as-needed  -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -Wl,--build-id=sha1 -Wl,-dT,/builddir/build/BUILD/petsc-3.16.4/.package_note-petsc-3.16.4-3.fc36.x86_64.ld -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64  -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -O3 -fPIC -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64  -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -O3
> FC_LINKER_FLAGS = -Wl,-z,relro -Wl,--as-needed  -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -Wl,--build-id=sha1 -Wl,-dT,/builddir/build/BUILD/petsc-3.16.4/.package_note-petsc-3.16.4-3.fc36.x86_64.ld -fPIC -Wl,-z,relro -Wl,--as-needed  -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -Wl,--build-id=sha1 -Wl,-dT,/builddir/build/BUILD/petsc-3.16.4/.package_note-petsc-3.16.4-3.fc36.x86_64.ld -lgfortran -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64  -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -I/usr/lib64/gfortran/modules -O3 -fPIC -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64  -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -I/usr/lib64/gfortran/modules -O3
> CONFIGURE_OPTIONS = --build=x86_64-redhat-linux-gnu --host=x86_64-redhat-linux-gnu --program-prefix= --with-dependency-tracking=0 --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc --datadir=/usr/share --includedir=/usr/include --libdir=/usr/lib64 --libexecdir=/usr/libexec --localstatedir=/var --sharedstatedir=/var/lib --mandir=/usr/share/man --infodir=/usr/share/info --with-cc=gcc --with-cxx=g++ --with-fc=gfortran CFLAGS=\"-O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64  -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -O3 -fPIC\" CXXFLAGS=\"-O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64  -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -O3 -fPIC\" FFLAGS=\"-O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64  -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -I/usr/lib64/gfortran/modules -O3 -fPIC\" LDFLAGS=\"-Wl,-z,relro -Wl,--as-needed  -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -Wl,--build-id=sha1 -Wl,-dT,/builddir/build/BUILD/petsc-3.16.4/.package_note-petsc-3.16.4-3.fc36.x86_64.ld -fPIC\" COPTFLAGS=\"-O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64  -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -O3\" CXXOPTFLAGS=\"-O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64  -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -O3\" FOPTFLAGS=\"-O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64  -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -I/usr/lib64/gfortran/modules -O3\" FCFLAGS=\"-O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64  -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -I/usr/lib64/gfortran/modules -O3 -fPIC\" --CC_LINKER_FLAGS=\"-Wl,-z,relro -Wl,--as-needed  -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -Wl,--build-id=sha1 -Wl,-dT,/builddir/build/BUILD/petsc-3.16.4/.package_note-petsc-3.16.4-3.fc36.x86_64.ld\" --FC_LINKER_FLAGS=\"-Wl,-z,relro -Wl,--as-needed  -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -Wl,--build-id=sha1 -Wl,-dT,/builddir/build/BUILD/petsc-3.16.4/.package_note-petsc-3.16.4-3.fc36.x86_64.ld -lgfortran\" --with-default-arch=0 --with-make=1 --with-cmake-exec=/usr/bin/cmake3 --with-ctest-exec=/usr/bin/ctest3 --with-single-library=1 --with-precision=double --with-petsc-arch=x86_64 --with-clanguage=C --with-shared-libraries=1 --with-fortran-interfaces=1 --with-windows-graphics=0 --CC=gcc --FC=gfortran --CXX=g++ --with-shared-ld=ld --with-pic=1 --with-clib-autodetect=0 --with-fortranlib-autodetect=0 --with-threadsafety=0 --with-log=1 --with-mkl_sparse=0 --with-mkl_sparse_optimize=0 --with-mkl_cpardiso=0 --with-mkl_pardiso=0 --with-python=0 --with-cxxlib-autodetect=1 --with-debugging=0 --with-mumps-serial=1 --with-mpi=0 --with-hdf5=1 --with-hdf5-include= --with-hdf5-lib=\"-lhdf5 -lhdf5_hl\" --with-cgns=1 --with-cgns-include= --with-cgns-lib=-lcgns --with-x=1 --with-openmp=0 --with-hwloc=0 --with-ssl=0 --with-pthread=1 --with-valgrind=1 --with-64-bit-indices=0 --with-blaslapack=1 --with-blaslapack-lib=-lflexiblas --with-blaslapack-include=/usr/include/flexiblas --with-metis=1 --with-superlu=1 --with-superlu-include=/usr/include/SuperLU --with-superlu-lib=-lsuperlu
> SL_LINKER_FUNCTION = -Wl,-z,relro -Wl,--as-needed  -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -Wl,--build-id=sha1 -Wl,-dT,/builddir/build/BUILD/petsc-3.16.4/.package_note-petsc-3.16.4-3.fc36.x86_64.ld -fPIC -shared -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64  -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -O3 -fPIC -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64  -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -O3 -Wl,-soname,$(call SONAME_FUNCTION,$(notdir $(1)),$(2))
> ~$
>
>
> then, I get the error
>
> /usr/bin/ld: cannot open linker script file /builddir/build/BUILD/petsc-3.16.4/.package_note-petsc-3.16.4-3.fc36.x86_64.ld: the file or folder doesn't exist
>
> I do not know who maintains these PETSc packages for Fedora, but they seem
hopelessly broken.

I would recommend just installing yourself.

  Thanks,

     Matt

> Best regards,
>
> Rafel Amer
>
>
>
>
>
> El 23/8/22 a les 16:04, Matthew Knepley ha escrit:
>
> On Tue, Aug 23, 2022 at 9:48 AM Rafel Amer Ramon <rafel.amer at upc.edu>
> wrote:
>
>>
>> Hi,
>>
>> yes, I can compile and run it:
>>
>> ~$ mpicc -o pi-mpi pi-mpi.c -lm
>> ~$ mpirun -np 16 --hostfile ~/hosts ./pi-mpi
>> Number of intervals: 2048
>> Result:   3.1411043724
>> Accuracy: 0.0004882812
>> Time:     0.0314383930
>> ~$
>>
>>
>> PETSc does not control the launch, so you must be using a different MPI.
> For building the PETSc executable, try instead
> the minimum makefile
>
> e: e.o
>       ${CLINKER} -o e e.o ${PETSC_LIB}
>
> include ${PETSC_DIR}/lib/petsc/conf/variables
> include ${PETSC_DIR}/lib/petsc/conf/rules
>
>   Thanks,
>
>      Matt
>
>
>> Best regards,
>>
>> Rafel Amer
>>
>>
>>
>>
>> El 23/8/22 a les 14:46, Matthew Knepley ha escrit:
>>
>> Can you run anything in parallel? Say the small sample code that
>> calculates pi?
>>
>>
>> https://www.cs.usfca.edu/~mmalensek/cs220/schedule/code/week09/pi-mpi.c.html
>>
>>   Thanks,
>>
>>       Matt
>>
>> On Tue, Aug 23, 2022 at 6:34 AM Rafel Amer Ramon <rafel.amer at upc.edu>
>> wrote:
>>
>>>
>>> Hi,
>>>
>>> mpicc and mpirun are from the package openmpi-devel-4.1.2-3.fc36.x86_64
>>> and the petsc64 library is linked with
>>> /usr/lib64/openmpi/lib/libmpi.so.40
>>>
>>> ~# ldd /lib64/libpetsc64.so.3.16.4
>>> 	linux-vdso.so.1 (0x00007fff5becc000)
>>> 	libflexiblas64.so.3 => /lib64/libflexiblas64.so.3 (0x00007f3030e30000)
>>> 	libcgns.so.4.2 => /usr/lib64/openmpi/lib/libcgns.so.4.2 (0x00007f3030d46000)
>>> 	libhdf5.so.200 => /usr/lib64/openmpi/lib/libhdf5.so.200 (0x00007f303091a000)
>>> 	libm.so.6 => /lib64/libm.so.6 (0x00007f303083c000)
>>> 	libX11.so.6 => /lib64/libX11.so.6 (0x00007f30306f4000)
>>> 	libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f30304c0000)
>>> 	libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f303049e000)
>>> 	libc.so.6 => /lib64/libc.so.6 (0x00007f303029c000)
>>> 	libgfortran.so.5 => /lib64/libgfortran.so.5 (0x00007f302ffcf000)
>>> 	libquadmath.so.0 => /lib64/libquadmath.so.0 (0x00007f302ff87000)
>>> 	/lib64/ld-linux-x86-64.so.2 (0x00007f30327bb000)
>>> 	libmpi.so.40 => /usr/lib64/openmpi/lib/libmpi.so.40 (0x00007f302fe59000)
>>> 	libsz.so.2 => /lib64/libsz.so.2 (0x00007f302fe50000)
>>> 	libz.so.1 => /lib64/libz.so.1 (0x00007f302fe34000)
>>> 	libxcb.so.1 => /lib64/libxcb.so.1 (0x00007f302fe08000)
>>> 	libopen-rte.so.40 => /usr/lib64/openmpi/lib/libopen-rte.so.40 (0x00007f302fd4d000)
>>> 	libopen-pal.so.40 => /usr/lib64/openmpi/lib/libopen-pal.so.40 (0x00007f302fc9f000)
>>> 	libhwloc.so.15 => /lib64/libhwloc.so.15 (0x00007f302fc42000)
>>> 	libevent_core-2.1.so.7 => /lib64/libevent_core-2.1.so.7 (0x00007f302fc07000)
>>> 	libevent_pthreads-2.1.so.7 => /lib64/libevent_pthreads-2.1.so.7 (0x00007f302fc02000)
>>> 	libXau.so.6 => /lib64/libXau.so.6 (0x00007f302fbfc000)
>>>
>>>
>>> I think that it's correct, but it don't work.
>>>
>>> Best regards,
>>>
>>> Rafel Amer
>>>
>>> El 22/8/22 a les 20:36, Barry Smith ha escrit:
>>>
>>>
>>>   Are you sure the mpirun you are using matches the mpi that PETSc was
>>> built with?
>>>
>>> On Aug 22, 2022, at 2:31 PM, Rafel Amer Ramon <rafel.amer at upc.edu>
>>> wrote:
>>>
>>> Hi,
>>>
>>> I have installed the following packages on fedora 36
>>>
>>> ~# rpm -qa | grep petsc
>>> petsc64-3.16.4-3.fc36.x86_64
>>> petsc-3.16.4-3.fc36.x86_64
>>> petsc-openmpi-3.16.4-3.fc36.x86_64
>>> petsc-openmpi-devel-3.16.4-3.fc36.x86_64
>>> python3-petsc-openmpi-3.16.4-3.fc36.x86_64
>>> petsc-devel-3.16.4-3.fc36.x86_64
>>> petsc64-devel-3.16.4-3.fc36.x86_64
>>> ~# rpm -qa | grep openmpi
>>> openmpi-4.1.2-3.fc36.x86_64
>>> ptscotch-openmpi-6.1.2-2.fc36.x86_64
>>> scalapack-openmpi-2.1.0-11.fc36.x86_64
>>> openmpi-devel-4.1.2-3.fc36.x86_64
>>> MUMPS-openmpi-5.4.1-2.fc36.x86_64
>>> superlu_dist-openmpi-6.1.1-9.fc36.x86_64
>>> hdf5-openmpi-1.12.1-5.fc36.x86_64
>>> hypre-openmpi-2.18.2-6.fc36.x86_64
>>> petsc-openmpi-3.16.4-3.fc36.x86_64
>>> fftw-openmpi-libs-double-3.3.10-2.fc36.x86_64
>>> fftw-openmpi-libs-long-3.3.10-2.fc36.x86_64
>>> fftw-openmpi-libs-single-3.3.10-2.fc36.x86_64
>>> fftw-openmpi-libs-3.3.10-2.fc36.x86_64
>>> fftw-openmpi-devel-3.3.10-2.fc36.x86_64
>>> petsc-openmpi-devel-3.16.4-3.fc36.x86_64
>>> python3-petsc-openmpi-3.16.4-3.fc36.x86_64
>>> scalapack-openmpi-devel-2.1.0-11.fc36.x86_64
>>> python3-openmpi-4.1.2-3.fc36.x86_64
>>> hdf5-openmpi-devel-1.12.1-5.fc36.x86_64
>>> cgnslib-openmpi-4.2.0-6.fc36.x86_64
>>> cgnslib-openmpi-devel-4.2.0-6.fc36.x86_64
>>>
>>> and I try to compile and run the program written in the file e.c
>>>
>>> ~# cat e.c
>>> #include <petsc.h>
>>>
>>> int main(int argc, char **argv) {
>>>   PetscErrorCode ierr;
>>>   PetscMPIInt    rank;
>>>
>>>   ierr = PetscInitialize(&argc,&argv,NULL,
>>>       "Compute e in parallel with PETSc.\n\n"); if (ierr) return ierr;
>>>   ierr = MPI_Comm_rank(PETSC_COMM_WORLD,&rank); CHKERRQ(ierr);
>>>   ierr = PetscPrintf(PETSC_COMM_WORLD,"My rank is %d\n",rank); CHKERRQ(ierr);
>>>
>>>
>>>   return PetscFinalize();
>>> }
>>>
>>> I compile it with the command
>>>
>>> ~# mpicc -o e e.c -I/usr/include/petsc64 -lpetsc64
>>>
>>> and I run it with
>>>
>>> ~# mpirun -np 8 ./eMy rank is 0
>>> My rank is 0
>>> My rank is 0
>>> My rank is 0
>>> My rank is 0
>>> My rank is 0
>>> My rank is 0
>>> My rank is 0
>>>
>>> but all my process get rank 0.
>>>
>>> Do you know what I'm doing wrong?
>>>
>>> Thank you!!
>>>
>>> Best regards,
>>>
>>> Rafel Amer
>>>
>>>
>>>
>>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/~knepley/>
>>
>>
>>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220823/2b073c00/attachment-0001.html>

From bsmith at petsc.dev  Tue Aug 23 10:21:31 2022
From: bsmith at petsc.dev (Barry Smith)
Date: Tue, 23 Aug 2022 11:21:31 -0400
Subject: [petsc-users] PETSc on fedora 36
In-Reply-To: <65d7f5ad-fcd5-2cc9-0f74-8c9c8f5eaeb8@upc.edu>
References: <aa3cbfe2-2ccc-93cc-6004-5c2ae6429047@upc.edu>
	<358BFF34-D941-42AB-8648-871186EF64DB@petsc.dev>
	<65d7f5ad-fcd5-2cc9-0f74-8c9c8f5eaeb8@upc.edu>
Message-ID: <FE4FA4D4-28C0-48B3-9948-E65A261EE17B@petsc.dev>


  1) Run the trivial PETSc program with 2 MPI ranks and -info (send the output)

  2) do a ldd on the PETSc program and the pure MPI program to see if they are all using the same libraries

  3) add a MPI_Initialize() immediately before the PetscInitialize() and immediately after the PetscFinalize(), does the program think it is parallel?

  4) is PETSC_HAVE_MPI_INIT_THREAD defined in $PETSC_ARCH/include/petscconf.h ?  If so this means MPI_Init_thread() is used to start up MPI instead of MPI_Init(). If you change MPI_Init() in your MPI standalone code to MPI_Init_thread() does it still run properly in parallel?





  Barry


> On Aug 23, 2022, at 6:34 AM, Rafel Amer Ramon <rafel.amer at upc.edu> wrote:
> 
> 
> Hi,
> 
> mpicc and mpirun are from the package openmpi-devel-4.1.2-3.fc36.x86_64
> and the petsc64 library is linked with  /usr/lib64/openmpi/lib/libmpi.so.40
> ~# ldd /lib64/libpetsc64.so.3.16.4 
> 	linux-vdso.so.1 (0x00007fff5becc000)
> 	libflexiblas64.so.3 => /lib64/libflexiblas64.so.3 (0x00007f3030e30000)
> 	libcgns.so.4.2 => /usr/lib64/openmpi/lib/libcgns.so.4.2 (0x00007f3030d46000)
> 	libhdf5.so.200 => /usr/lib64/openmpi/lib/libhdf5.so.200 (0x00007f303091a000)
> 	libm.so.6 => /lib64/libm.so.6 (0x00007f303083c000)
> 	libX11.so.6 => /lib64/libX11.so.6 (0x00007f30306f4000)
> 	libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f30304c0000)
> 	libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f303049e000)
> 	libc.so.6 => /lib64/libc.so.6 (0x00007f303029c000)
> 	libgfortran.so.5 => /lib64/libgfortran.so.5 (0x00007f302ffcf000)
> 	libquadmath.so.0 => /lib64/libquadmath.so.0 (0x00007f302ff87000)
> 	/lib64/ld-linux-x86-64.so.2 (0x00007f30327bb000)
> 	libmpi.so.40 => /usr/lib64/openmpi/lib/libmpi.so.40 (0x00007f302fe59000)
> 	libsz.so.2 => /lib64/libsz.so.2 (0x00007f302fe50000)
> 	libz.so.1 => /lib64/libz.so.1 (0x00007f302fe34000)
> 	libxcb.so.1 => /lib64/libxcb.so.1 (0x00007f302fe08000)
> 	libopen-rte.so.40 => /usr/lib64/openmpi/lib/libopen-rte.so.40 (0x00007f302fd4d000)
> 	libopen-pal.so.40 => /usr/lib64/openmpi/lib/libopen-pal.so.40 (0x00007f302fc9f000)
> 	libhwloc.so.15 => /lib64/libhwloc.so.15 (0x00007f302fc42000)
> 	libevent_core-2.1.so.7 => /lib64/libevent_core-2.1.so.7 (0x00007f302fc07000)
> 	libevent_pthreads-2.1.so.7 => /lib64/libevent_pthreads-2.1.so.7 (0x00007f302fc02000)
> 	libXau.so.6 => /lib64/libXau.so.6 (0x00007f302fbfc000)
> 
> I think that it's correct, but it don't work.
> 
> Best regards,
> 
> Rafel Amer
> 
> El 22/8/22 a les 20:36, Barry Smith ha escrit:
>> 
>>   Are you sure the mpirun you are using matches the mpi that PETSc was built with? 
>> 
>>> On Aug 22, 2022, at 2:31 PM, Rafel Amer Ramon <rafel.amer at upc.edu <mailto:rafel.amer at upc.edu>> wrote:
>>> 
>>> Hi,
>>> 
>>> I have installed the following packages on fedora 36
>>> 
>>> ~# rpm -qa | grep petsc
>>> petsc64-3.16.4-3.fc36.x86_64
>>> petsc-3.16.4-3.fc36.x86_64
>>> petsc-openmpi-3.16.4-3.fc36.x86_64
>>> petsc-openmpi-devel-3.16.4-3.fc36.x86_64
>>> python3-petsc-openmpi-3.16.4-3.fc36.x86_64
>>> petsc-devel-3.16.4-3.fc36.x86_64
>>> petsc64-devel-3.16.4-3.fc36.x86_64
>>> 
>>> ~# rpm -qa | grep openmpi
>>> openmpi-4.1.2-3.fc36.x86_64
>>> ptscotch-openmpi-6.1.2-2.fc36.x86_64
>>> scalapack-openmpi-2.1.0-11.fc36.x86_64
>>> openmpi-devel-4.1.2-3.fc36.x86_64
>>> MUMPS-openmpi-5.4.1-2.fc36.x86_64
>>> superlu_dist-openmpi-6.1.1-9.fc36.x86_64
>>> hdf5-openmpi-1.12.1-5.fc36.x86_64
>>> hypre-openmpi-2.18.2-6.fc36.x86_64
>>> petsc-openmpi-3.16.4-3.fc36.x86_64
>>> fftw-openmpi-libs-double-3.3.10-2.fc36.x86_64
>>> fftw-openmpi-libs-long-3.3.10-2.fc36.x86_64
>>> fftw-openmpi-libs-single-3.3.10-2.fc36.x86_64
>>> fftw-openmpi-libs-3.3.10-2.fc36.x86_64
>>> fftw-openmpi-devel-3.3.10-2.fc36.x86_64
>>> petsc-openmpi-devel-3.16.4-3.fc36.x86_64
>>> python3-petsc-openmpi-3.16.4-3.fc36.x86_64
>>> scalapack-openmpi-devel-2.1.0-11.fc36.x86_64
>>> python3-openmpi-4.1.2-3.fc36.x86_64
>>> hdf5-openmpi-devel-1.12.1-5.fc36.x86_64
>>> cgnslib-openmpi-4.2.0-6.fc36.x86_64
>>> cgnslib-openmpi-devel-4.2.0-6.fc36.x86_64
>>> 
>>> and I try to compile and run the program written in the file e.c
>>> 
>>> ~# cat e.c
>>> #include <petsc.h>
>>> 
>>> int main(int argc, char **argv) {
>>>   PetscErrorCode ierr;
>>>   PetscMPIInt    rank;
>>> 
>>>   ierr = PetscInitialize(&argc,&argv,NULL,
>>>       "Compute e in parallel with PETSc.\n\n"); if (ierr) return ierr;
>>>   ierr = MPI_Comm_rank(PETSC_COMM_WORLD,&rank); CHKERRQ(ierr);
>>>   ierr = PetscPrintf(PETSC_COMM_WORLD,"My rank is %d\n",rank); CHKERRQ(ierr);
>>> 
>>>   
>>>   return PetscFinalize();
>>> }
>>> 
>>> I compile it with the command
>>> 
>>> ~# mpicc -o e e.c -I/usr/include/petsc64 -lpetsc64
>>> 
>>> and I run it with 
>>> 
>>> ~# mpirun -np 8 ./e
>>> My rank is 0
>>> My rank is 0
>>> My rank is 0
>>> My rank is 0
>>> My rank is 0
>>> My rank is 0
>>> My rank is 0
>>> My rank is 0
>>> 
>>> but all my process get rank 0.
>>> 
>>> Do you know what I'm doing wrong?
>>> 
>>> Thank you!!
>>> 
>>> Best regards,
>>> 
>>> Rafel Amer
>> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220823/b0406d36/attachment.html>

From balay at mcs.anl.gov  Tue Aug 23 10:27:26 2022
From: balay at mcs.anl.gov (Satish Balay)
Date: Tue, 23 Aug 2022 10:27:26 -0500 (CDT)
Subject: [petsc-users] PETSc on fedora 36
In-Reply-To: <aa3cbfe2-2ccc-93cc-6004-5c2ae6429047@upc.edu>
References: <aa3cbfe2-2ccc-93cc-6004-5c2ae6429047@upc.edu>
Message-ID: <3b34b37e-f6b-2675-3886-8c5d6fed42c@mcs.anl.gov>

[root at pj01 ~]# yum search petsc |grep devel
Last metadata expiration check: 0:01:57 ago on Tue 23 Aug 2022 10:23:51 AM CDT.
petsc-devel.i686 : Portable Extensible Toolkit for Scientific Computation (developer files)
petsc-devel.x86_64 : Portable Extensible Toolkit for Scientific Computation (developer files)
petsc-mpich-devel.i686 : Portable Extensible Toolkit for Scientific Computation (MPICH)
petsc-mpich-devel.x86_64 : Portable Extensible Toolkit for Scientific Computation (MPICH)
petsc-openmpi-devel.i686 : Portable Extensible Toolkit for Scientific Computation (OpenMPI)
petsc-openmpi-devel.x86_64 : Portable Extensible Toolkit for Scientific Computation (OpenMPI)
petsc64-devel.x86_64 : Portable Extensible Toolkit for Scientific Computation (64bit INTEGER)


Likely 'petsc64' is a sequential-only build - so you would need to switch to 'petsc-mpich-devel' or 'petsc-openmpi-devel'

And if you need a parallel 64-bit-indices mode - then suggest installing PETSc from source.

Satish


On Mon, 22 Aug 2022, Rafel Amer Ramon wrote:

> Hi,
> 
> I have installed the following packages on fedora 36
> 
> ~# rpm -qa | grep petsc
> petsc64-3.16.4-3.fc36.x86_64
> petsc-3.16.4-3.fc36.x86_64
> petsc-openmpi-3.16.4-3.fc36.x86_64
> petsc-openmpi-devel-3.16.4-3.fc36.x86_64
> python3-petsc-openmpi-3.16.4-3.fc36.x86_64
> petsc-devel-3.16.4-3.fc36.x86_64
> petsc64-devel-3.16.4-3.fc36.x86_64
> 
> ~# rpm -qa | grep openmpi
> openmpi-4.1.2-3.fc36.x86_64
> ptscotch-openmpi-6.1.2-2.fc36.x86_64
> scalapack-openmpi-2.1.0-11.fc36.x86_64
> openmpi-devel-4.1.2-3.fc36.x86_64
> MUMPS-openmpi-5.4.1-2.fc36.x86_64
> superlu_dist-openmpi-6.1.1-9.fc36.x86_64
> hdf5-openmpi-1.12.1-5.fc36.x86_64
> hypre-openmpi-2.18.2-6.fc36.x86_64
> petsc-openmpi-3.16.4-3.fc36.x86_64
> fftw-openmpi-libs-double-3.3.10-2.fc36.x86_64
> fftw-openmpi-libs-long-3.3.10-2.fc36.x86_64
> fftw-openmpi-libs-single-3.3.10-2.fc36.x86_64
> fftw-openmpi-libs-3.3.10-2.fc36.x86_64
> fftw-openmpi-devel-3.3.10-2.fc36.x86_64
> petsc-openmpi-devel-3.16.4-3.fc36.x86_64
> python3-petsc-openmpi-3.16.4-3.fc36.x86_64
> scalapack-openmpi-devel-2.1.0-11.fc36.x86_64
> python3-openmpi-4.1.2-3.fc36.x86_64
> hdf5-openmpi-devel-1.12.1-5.fc36.x86_64
> cgnslib-openmpi-4.2.0-6.fc36.x86_64
> cgnslib-openmpi-devel-4.2.0-6.fc36.x86_64
> 
> and I try to compile and run the program written in the file e.c
> 
> ~# cat e.c
> #include <petsc.h>
> 
> int main(int argc, char **argv) {
>   PetscErrorCode ierr;
>   PetscMPIInt    rank;
> 
>   ierr = PetscInitialize(&argc,&argv,NULL,
>       "Compute e in parallel with PETSc.\n\n"); if (ierr) return ierr;
>   ierr = MPI_Comm_rank(PETSC_COMM_WORLD,&rank); CHKERRQ(ierr);
>   ierr = PetscPrintf(PETSC_COMM_WORLD,"My rank is %d\n",rank); CHKERRQ(ierr);
> 
>   
>   return PetscFinalize();
> }
> 
> I compile it with the command
> 
> ~# mpicc -o e e.c -I/usr/include/petsc64 -lpetsc64
> 
> and I run it with
> 
> ~# mpirun -np 8 ./e
> My rank is 0
> My rank is 0
> My rank is 0
> My rank is 0
> My rank is 0
> My rank is 0
> My rank is 0
> My rank is 0
> 
> but all my process get rank 0.
> 
> Do you know what I'm doing wrong?
> 
> Thank you!!
> 
> Best regards,
> 
> Rafel Amer
> 


From balay at mcs.anl.gov  Tue Aug 23 10:33:58 2022
From: balay at mcs.anl.gov (Satish Balay)
Date: Tue, 23 Aug 2022 10:33:58 -0500 (CDT)
Subject: [petsc-users] PETSc on fedora 36
In-Reply-To: <65d7f5ad-fcd5-2cc9-0f74-8c9c8f5eaeb8@upc.edu>
References: <aa3cbfe2-2ccc-93cc-6004-5c2ae6429047@upc.edu>
	<358BFF34-D941-42AB-8648-871186EF64DB@petsc.dev>
	<65d7f5ad-fcd5-2cc9-0f74-8c9c8f5eaeb8@upc.edu>
Message-ID: <5f57fbe-c49a-baad-faa-24e49ed6ed45@mcs.anl.gov>

Hm - I don't see this.

[root at pj01 ~]# ldd /usr/lib64/libpetsc64.so
	linux-vdso.so.1 => linux-vdso.so.1 (0x00007ffd10f66000)
	libflexiblas64.so.3 => /lib64/libflexiblas64.so.3 (0x00007f9e1fe00000)
	libcgns.so.4.3 => /lib64/libcgns.so.4.3 (0x00007f9e1fd19000)
	libhdf5.so.200 => /lib64/libhdf5.so.200 (0x00007f9e1f800000)
	libmetis64.so.0 => /lib64/libmetis64.so.0 (0x00007f9e219a8000)
	libm.so.6 => /lib64/libm.so.6 (0x00007f9e1fc3b000)
	libX11.so.6 => /lib64/libX11.so.6 (0x00007f9e1f6b9000)
	libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f9e1f400000)
	libc.so.6 => /lib64/libc.so.6 (0x00007f9e1f228000)
	libgfortran.so.5 => /lib64/libgfortran.so.5 (0x00007f9e1ee00000)
	libquadmath.so.0 => /lib64/libquadmath.so.0 (0x00007f9e2195e000)
	/lib64/ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2 (0x00007f9e21a24000)
	libsz.so.2 => /lib64/libsz.so.2 (0x00007f9e21955000)
	libz.so.1 => /lib64/libz.so.1 (0x00007f9e2193b000)
	libpcre2-posix.so.3 => /lib64/libpcre2-posix.so.3 (0x00007f9e21934000)
	libgomp.so.1 => /lib64/libgomp.so.1 (0x00007f9e201b6000)
	libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f9e1fc1b000)
	libxcb.so.1 => /lib64/libxcb.so.1 (0x00007f9e1f68e000)
	libpcre2-8.so.0 => /lib64/libpcre2-8.so.0 (0x00007f9e1f18b000)
	libXau.so.6 => /lib64/libXau.so.6 (0x00007f9e1fc15000)

[root at pj01 ~]# rpm -ql petsc64-devel |grep petscconf.h
/usr/include/petsc64/petscconf.h
[root at pj01 ~]# grep MPIUNI /usr/include/petsc64/petscconf.h
#define PETSC_HAVE_MPIUNI 1

So /usr/lib64/libpetsc64.so appears to be a sequential build [for me]

Satish


On Tue, 23 Aug 2022, Rafel Amer Ramon wrote:

> 
> Hi,
> 
> mpicc and mpirun are from the package openmpi-devel-4.1.2-3.fc36.x86_64
> and the petsc64 library is linked with? /usr/lib64/openmpi/lib/libmpi.so.40
> 
> ~# ldd /lib64/libpetsc64.so.3.16.4 
> 	linux-vdso.so.1 (0x00007fff5becc000)
> 	libflexiblas64.so.3 => /lib64/libflexiblas64.so.3 (0x00007f3030e30000)
> 	libcgns.so.4.2 => /usr/lib64/openmpi/lib/libcgns.so.4.2 (0x00007f3030d46000)
> 	libhdf5.so.200 => /usr/lib64/openmpi/lib/libhdf5.so.200 (0x00007f303091a000)
> 	libm.so.6 => /lib64/libm.so.6 (0x00007f303083c000)
> 	libX11.so.6 => /lib64/libX11.so.6 (0x00007f30306f4000)
> 	libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f30304c0000)
> 	libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f303049e000)
> 	libc.so.6 => /lib64/libc.so.6 (0x00007f303029c000)
> 	libgfortran.so.5 => /lib64/libgfortran.so.5 (0x00007f302ffcf000)
> 	libquadmath.so.0 => /lib64/libquadmath.so.0 (0x00007f302ff87000)
> 	/lib64/ld-linux-x86-64.so.2 (0x00007f30327bb000)
> 	libmpi.so.40 => /usr/lib64/openmpi/lib/libmpi.so.40 (0x00007f302fe59000)
> 	libsz.so.2 => /lib64/libsz.so.2 (0x00007f302fe50000)
> 	libz.so.1 => /lib64/libz.so.1 (0x00007f302fe34000)
> 	libxcb.so.1 => /lib64/libxcb.so.1 (0x00007f302fe08000)
> 	libopen-rte.so.40 => /usr/lib64/openmpi/lib/libopen-rte.so.40 (0x00007f302fd4d000)
> 	libopen-pal.so.40 => /usr/lib64/openmpi/lib/libopen-pal.so.40 (0x00007f302fc9f000)
> 	libhwloc.so.15 => /lib64/libhwloc.so.15 (0x00007f302fc42000)
> 	libevent_core-2.1.so.7 => /lib64/libevent_core-2.1.so.7 (0x00007f302fc07000)
> 	libevent_pthreads-2.1.so.7 => /lib64/libevent_pthreads-2.1.so.7 (0x00007f302fc02000)
> 	libXau.so.6 => /lib64/libXau.so.6 (0x00007f302fbfc000)
> 
> I think that it's correct, but it don't work.
> 
> Best regards,
> 
> Rafel Amer
> 
> El 22/8/22 a les 20:36, Barry Smith ha escrit:
> 
> ? Are you sure the mpirun you are using matches the mpi that PETSc was built with??
> 
>       On Aug 22, 2022, at 2:31 PM, Rafel Amer Ramon <rafel.amer at upc.edu> wrote:
> 
> Hi,
> 
> I have installed the following packages on fedora 36
> 
> ~# rpm -qa | grep petsc
> petsc64-3.16.4-3.fc36.x86_64
> petsc-3.16.4-3.fc36.x86_64
> petsc-openmpi-3.16.4-3.fc36.x86_64
> petsc-openmpi-devel-3.16.4-3.fc36.x86_64
> python3-petsc-openmpi-3.16.4-3.fc36.x86_64
> petsc-devel-3.16.4-3.fc36.x86_64
> petsc64-devel-3.16.4-3.fc36.x86_64
> 
> ~# rpm -qa | grep openmpi
> openmpi-4.1.2-3.fc36.x86_64
> ptscotch-openmpi-6.1.2-2.fc36.x86_64
> scalapack-openmpi-2.1.0-11.fc36.x86_64
> openmpi-devel-4.1.2-3.fc36.x86_64
> MUMPS-openmpi-5.4.1-2.fc36.x86_64
> superlu_dist-openmpi-6.1.1-9.fc36.x86_64
> hdf5-openmpi-1.12.1-5.fc36.x86_64
> hypre-openmpi-2.18.2-6.fc36.x86_64
> petsc-openmpi-3.16.4-3.fc36.x86_64
> fftw-openmpi-libs-double-3.3.10-2.fc36.x86_64
> fftw-openmpi-libs-long-3.3.10-2.fc36.x86_64
> fftw-openmpi-libs-single-3.3.10-2.fc36.x86_64
> fftw-openmpi-libs-3.3.10-2.fc36.x86_64
> fftw-openmpi-devel-3.3.10-2.fc36.x86_64
> petsc-openmpi-devel-3.16.4-3.fc36.x86_64
> python3-petsc-openmpi-3.16.4-3.fc36.x86_64
> scalapack-openmpi-devel-2.1.0-11.fc36.x86_64
> python3-openmpi-4.1.2-3.fc36.x86_64
> hdf5-openmpi-devel-1.12.1-5.fc36.x86_64
> cgnslib-openmpi-4.2.0-6.fc36.x86_64
> cgnslib-openmpi-devel-4.2.0-6.fc36.x86_64
> 
> and I try to compile and run the program written in the file e.c
> 
> ~# cat e.c
> #include <petsc.h>
> 
> int main(int argc, char **argv) {
>   PetscErrorCode ierr;
>   PetscMPIInt    rank;
> 
>   ierr = PetscInitialize(&argc,&argv,NULL,
>       "Compute e in parallel with PETSc.\n\n"); if (ierr) return ierr;
>   ierr = MPI_Comm_rank(PETSC_COMM_WORLD,&rank); CHKERRQ(ierr);
>   ierr = PetscPrintf(PETSC_COMM_WORLD,"My rank is %d\n",rank); CHKERRQ(ierr);
> 
>   
>   return PetscFinalize();
> }
> 
> I compile it with the command
> 
> ~# mpicc -o e e.c -I/usr/include/petsc64 -lpetsc64
> 
> and I run it with
> 
> ~# mpirun -np 8 ./e
> My rank is 0
> My rank is 0
> My rank is 0
> My rank is 0
> My rank is 0
> My rank is 0
> My rank is 0
> My rank is 0
> 
> but all my process get rank 0.
> 
> Do you know what I'm doing wrong?
> 
> Thank you!!
> 
> Best regards,
> 
> Rafel Amer
> 
> 
> 
> 
> 

From Jiannan_Tu at uml.edu  Tue Aug 23 11:00:51 2022
From: Jiannan_Tu at uml.edu (Tu, Jiannan)
Date: Tue, 23 Aug 2022 16:00:51 +0000
Subject: [petsc-users] Using matrix-free with KSP
In-Reply-To: <F4C76705-B402-4F8B-A42C-DDBE95574065@petsc.dev>
References: <BY5PR02MB6471969F80FA5F732FC74D87E7D79@BY5PR02MB6471.namprd02.prod.outlook.com>
	<B55CFC2C-6364-4B78-8E99-9D53700743B8@petsc.dev>
	<BY5PR02MB6471BDF90783FE3A0EA176CBE7809@BY5PR02MB6471.namprd02.prod.outlook.com>
	<87iloa49kt.fsf@jedbrown.org>
	<BY5PR02MB6471C0ED04769D02046F68ACE7809@BY5PR02MB6471.namprd02.prod.outlook.com>
	<00EBE4B8-22E5-408F-B56D-14226A278D7E@petsc.dev>
	<BY5PR02MB6471E386057DDDE98AD05DFAE7809@BY5PR02MB6471.namprd02.prod.outlook.com>
	<BY5PR02MB647140084C7E7CDDB6BC2A11E76B9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<A6629DE1-5A7F-414E-B164-A69877158675@petsc.dev>
	<BY5PR02MB6471AB1C004920DA1FC9B43EE76B9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<103146FD-FF28-47CE-B661-F2B521BF0E7A@petsc.dev>
	<DM6PR02MB6478B2FB8CB1BE757CE7B003E76D9@DM6PR02MB6478.namprd02.prod.outlook.com>
	<4C00F4B8-BB42-4932-BCD7-527E5FB39BFF@petsc.dev>
	<BY5PR02MB64710E70758E431108D66956E76D9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<D0EB13D8-C5A6-4AE7-AFDE-2CDCCACBAF73@petsc.dev>
	<BY5PR02MB6471D4B05874DECFDDA13FFCE76D9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<367B124C-88BB-4889-A4AF-45A1D473078D@petsc.dev>
	<BY5PR02MB6471F2C058B84DE092567731E76D9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<944DDEED-016D-43B2-A5A6-7BF2E8E41FA5@petsc.dev>
	<BY5PR02MB6471CD0236F03D20763A3B64E76C9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<B4CF5E5B-984D-4EC1-AB77-1FD08C3FBBCF@petsc.dev>
	<BY5PR02MB6471BA72871419781B875967E76C9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<3D179AEA-AD05-4B46-8DDC-9E126741C247@petsc.dev>
	<BY5PR02MB6471560EA70C6FAE826DDF27E7719@BY5PR02MB6471.namprd02.prod.outlook.com>
	<1C67C8DB-0653-4588-8137-E0865E0F2098@petsc.dev>
	<BY5PR02MB6471C5EAE18AB9B114E76232E7719@BY5PR02MB6471.namprd02.prod.outlook.com>
	<FAA1668F-08C1-4BAD-89B4-784D8AD99C36@petsc.dev>
	<BY5PR02MB64715D7FE585A7F88092E04DE7719@BY5PR02MB6471.namprd02.prod.outlook.com>
	<F4C76705-B402-4F8B-A42C-DDBE95574065@petsc.dev>
Message-ID: <BY5PR02MB64712BB4AF9E7CD2A66FE2F3E7709@BY5PR02MB6471.namprd02.prod.outlook.com>

Hi Barry,

I celebrated too early. Just found that the matrix-vector multiply function put into MatShellSetOperation() is not called. I set a printf() in it no effect. Then I comment out all execution statements int it, KSP converges the same way and the solution is the same. Apparently, that mat-vec multiply function doesn't take effect.

Following is complete sequency of KSP calls. Could you please help me identify what the problem is?

Thank you very much.

Jiannan

------------------------------------
    DMDACreate1d(MPI_COMM_WORLD, DM_BOUNDARY_GHOSTED, N, 1, 1, NULL, &da);
    DMSetFromOptions(da);
    DMSetUp(da);
    params.da = da;

    DMDAGetCorners(da, &xs, NULL, NULL, &xm, NULL, NULL);

    DMCreateGlobalVector(da, &X);
    VecDuplicate(X, &Rm);

    MatCreate(MPI_COMM_WORLD, &params.A);
    MatSetType(params.A, MATMPIAIJ);
    MatSetSizes(params.A, PETSC_DECIDE, PETSC_DECIDE, N, N);

    MatCreateShell(MPI_COMM_WORLD, xm, N, PETSC_DETERMINE, PETSC_DETERMINE, &params, &B);
    MatShellSetOperation(B, MATOP_MULT, (void(*)(void))matvecmult);

    Mat C;
    MatCreate(MPI_COMM_WORLD, &C);
    MatSetType(C, MATPREALLOCATOR);
    MatSetSizes(C, PETSC_DECIDE, PETSC_DECIDE, N, N);

    //calculate matrix elements from nearby interactions
    matrix_near_elements(&params, params.A, C);
    MatDestroy(&C);

    KSPCreate(MPI_COMM_WORLD, &ksp);
    KSPSetOperators(ksp, params.A, params.A);
    KSPSetFromOptions(ksp);

    PC pc;
    KSPGetPC(ksp, &pc);
    PCSetFromOptions(pc);

    //quantities needed to calculate interactions with far groups
    fmm_formular(&params);

    //compute right hand-side vector
    computeRhs(&params, Rm);

    //solve matrix equation
    KSPSolve(ksp, Rm, X);

________________________________
From: Barry Smith <bsmith at petsc.dev>
Sent: Monday, August 22, 2022 2:39 PM
To: Tu, Jiannan <Jiannan_Tu at uml.edu>
Cc: petsc-users at mcs.anl.gov <petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] Using matrix-free with KSP


CAUTION: This email was sent from outside the UMass Lowell network.



On Aug 22, 2022, at 1:37 PM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

Barry,

Yes, you are right, Anear is the preconditioner that needs to be passed to KSPSetOperators() as the second matrix.

I think I have figured out how to provide a preconditioner for matrix-free method. I was confused by matrix A created by MatCreateMFFD(), which is type MFFD. I thought I have to use it as the first matrix passed to KSPSetOperators(). Since it was not assembled (and can't) so that Petsc gave error message "matrix unassembled". In fact, I should ignore A, but use other two matrixes AA and B, which are type MPIAIJ, assemble them, pass them to KSPSetOperators() as first and second matrix. Since function passed into MatMFFDSetFunction() does not take a matrix as an argument, matrix AA needs to be declared in user context so that it can be used in that function to perform matrix-vector product (partially) using MatMult() as you suggested.

   I think your AA should be the shell matrix, and the B should be Anear. There should be no MatMFFD. Inside the AA shell matrix mat-mult you call MatMult( on the Anear put Anear in the Shell matrix context also) and then add the part of the multiply coming from the far field.


I have used a small test program to solve an 8x8 linear equation system and the code produces the same answers with/without the preconditioner. The iteration for this small problem, of course no improvement with the preconditioner but the residual norms with/without the preconditioner are slightly different. I suppose the preconditioner takes effect.

Now I need to think about if the applied preconditioner does the right job on the other part of matrix-vector product (non near part) for the problem I am working on.

Thank you very very much and greatly appreciate your help!

Jiannan
________________________________
From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Monday, August 22, 2022 12:42 PM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] Using matrix-free with KSP


CAUTION: This email was sent from outside the UMass Lowell network.



On Aug 22, 2022, at 12:25 PM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

The matrix-vector product involves part of matrix elements explicitly as shown below so that I need to insert values into this part of the matrix.
<image.png>
Initially I wrote my own code for calculating and storing matrix A^near_mi without using any Petsc matrix and then performed matrix-vector product using above equation. The code runs but converges very slow. Apparently a preconditioned is needed.

I am not clear how Petsc can provide a preconditioner for my case.

  I am now confused. I thought you were forming Anear as a PETSc matrix and passing that to KSPSetOperators as the second matrix and so the preconditioner was being built with Anear?

  For your shell matrix vector product you should call MatMult(Anear,x,y) inside your custom routine and then add the rest of the matrix-vector product using your fast multipole products. What am I missing?

  Barry

________________________________
From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Monday, August 22, 2022 8:42 AM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] Using matrix-free with KSP


CAUTION: This email was sent from outside the UMass Lowell network.



On Aug 21, 2022, at 11:18 PM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

Barry,

You are right. With PC type none, the same error appears.

Now I use MatShell and I literally have three matrixes. A is a shell matrix, B is a preconditioner, and C is a matrix just for preallocation purpose.

First C is used to determine non-zero structure. Then use MatPreallocatorPreallocation(C, PETSC_FALSE, B) to preallocate for B. After that do insertion and assembly for B. These steps work fine.

Now the question is how to insert and assemble shell matrix A. MatPreallocatorPreallocation doesn?t work with shell matrix as indicated by an error message. MatSetValues or Mat assembly doesn?t support shell matrix either.

  A shell matrix is completely under your control. You decide on what data you want to store in it, and you provide the matrix-vector product routine.  MatPreallocatorPreallocation and MatSetValues don't work for shell matrix unless you provide code for them to work, which normally you would not do.

  I don't understand why you want to use MatSetValues for your shell matrix. If you can use MatSetValues() and just build an explicit matrix, then why would you use a shell matrix, just use MATAIJ matrix?

  Or does part of the operation inside your MatShell require a "regular" PETSc matrix? This would be fine, but it is your responsibility to design your Mat shell context to contain the "regular matrix" and your responsibility to fill up that regular matrix inside your shell context, calling whatever code you choose to write. So for example your shell context would be a C struct and one of the fields would be a regular PETSc matrix, also part of your custom matrix-vector product routine would call a regular MatMult() on the regular PETSc matrix stored inside the C struct.

  Barry



Thank you,
Jiannan

From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Friday, August 19, 2022 4:02 PM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] Using matrix-free with KSP

CAUTION: This email was sent from outside the UMass Lowell network.


  I don't know why changing the PCType could matter, can you please try with your new code but use the -pc_type none option to turn off the use of the preconditioner to see if the error still appears (I think it should).

  Also if you have a custom function that can do the matrix-vector product (not using any differencing) then you should use MatShell() and provide the matrix-vector product operation with your code.

  Barry



On Aug 19, 2022, at 11:47 AM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

Barry,

I have my own matrix-vector product but not use MatShell. I followed your previous instruction to use

    MatCreateMFFD(MPI_COMM_WORLD, PETSC_DECIDE, PETSC_DECIDE, N, N, &A);
    MatMFFDSetFunction(A, formfunction, &params);
    MatSetFromOptions(A);
    MatMFFDSetBase(A, X, NULL);

The customized matrix-vector product is implemented in formfunction() provided to MatMFFDSetFunction(). In this way, the matrix A is not used. I think that is why Petsc complains the matrix is not assembled. However, when PC type is none, the code runs without error although KSP solver doesn?t converge.

The question then becomes how to link A to matrix-free matrix-vector multiplication. And where to put MatShell?

Thank you,
Jiannan

From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Friday, August 19, 2022 10:27 AM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] Using matrix-free with KSP

CAUTION: This email was sent from outside the UMass Lowell network.





On Aug 18, 2022, at 11:41 PM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

The program calls KSPSolve just once and the error occurs after the first iteration.

I realized the matrix-vector product in my case is different from that approximating matrix-vector multiply by finite difference of function vector. It consists of two parts. One is multiplication of part of the matrix with part of vector, and another part results from so called fast multipole method. I?m not clear how to relate such calculation of matrix-vector product to the finite differencing of function vector.

  Are you using finite differencing at all in a matrix-vector product, or do you have your own custom matrix-vector product code (with MatShell)?

  Anyways the easiest thing to do is

  use the run time option -on_error_attach_debugger noxterm and run the code,

  when it stops in the debugger you can type bt (for backtrace)

  then do up a few times to move up the stack until you get to the PETSc function call that generates the error

  you can then do "call MatView(thematrixvarible,0) to have PETSc display what the matrix is and

  you can also do print *thematrixvariable to see the exact struct with all its fields

  If this doesn't tell you why the troublesome matrix is not assembled then cut and paste ALL of the output and email it

  Barry






From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Thursday, August 18, 2022 7:44 PM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] Using matrix-free with KSP

CAUTION: This email was sent from outside the UMass Lowell network.



   Is it in the first iteration of the first KSPSolve that you get this message? Or in the second call to KSPSolve?

  The error comes from a matrix vector product with your A matrix, what is that and where did it come from. The error message is not from your B matrix, that is not used in the MatMult()





On Aug 18, 2022, at 3:05 PM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

Barry,

Thanks. The method works. Assembly of actual matrix now completes quickly. The convergency is much better for the first iteration. The residual normal is in the order of 10^-10 instead of 10^-2. However, there is an error message after the first iteration: ?Object is in wrong state. Not for unassembled matrix? even though both preallocator and actual matrix have been assembled (see the code snippet below). Could you please give me some hints what?s going wrong? I use KSP type bcgs and PC type mat

Thank you,
Jiannan

-----------------------------------------------------
Solve linear equations ...
  0 KSP Residual norm 9.841179147519e-10
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Object is in wrong state
[0]PETSC ERROR: Not for unassembled matrix
[0]PETSC ERROR: See https://petsc.org/release/faq/<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Frelease%2Ffaq%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7Cc1e9f88adcda42f7c8b408da846dae97%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637967903891169434%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=aQgMnGt515eFE4s6OnTPPOylUFUaLPCpO678MJNvBQs%3D&reserved=0> for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.16.6, Mar 30, 2022
[0]PETSC ERROR: ./iesolver on a  named REXNET-WS4 by jiannantu Thu Aug 18 14:48:43 2022
[0]PETSC ERROR: Configure options --download-f2cblaslapack=yes --with-mpi-dir=/usr/local --with-fc=0 --prefix=/home/jiannantu/petsc --with-scalar-type=complex --with-64-bit-indices=1
[0]PETSC ERROR: #1 MatMult() at /home/jiannantu/petsc-3.16.6/src/mat/interface/matrix.c:2425
[0]PETSC ERROR: #2 PCApplyBAorAB() at /home/jiannantu/petsc-3.16.6/src/ksp/pc/interface/precon.c:730
[0]PETSC ERROR: #3 KSP_PCApplyBAorAB() at /home/jiannantu/petsc-3.16.6/include/petsc/private/kspimpl.h:421
[0]PETSC ERROR:  Linear equations not solved. Diverged reason: 0
#4 KSPSolve_BCGS() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/impls/bcgs/bcgs.c:87
[0]PETSC ERROR: #5 KSPSolve_Private() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itfunc.c:914
[0]PETSC ERROR: #6 KSPSolve() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itfunc.c:1086

-----------------------------------------------
    MatCreate(MPI_COMM_WORLD, &B);
    MatSetType(B, MATMPIAIJ);
    MatSetSizes(B, PETSC_DECIDE, PETSC_DECIDE, N, N);

    Mat preall;
    MatCreate(MPI_COMM_WORLD, &preall);
    MatSetType(preall, MATPREALLOCATOR);
    MatSetSizes(preall, PETSC_DECIDE, PETSC_DECIDE, N, N);

    MatMPIAIJSetPreallocation(preall, 0, d_nnz, 0, o_nnz);
    MatSetUp(preall);
MatSetOption(preall, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE);

//set values row by row
MatSetValues(preall, 1, &m, 1, nCols, matCols, INSERT_VALUES);

MatAssemblyBegin(preall, MAT_FINAL_ASSEMBLY);
MatAssemblyEnd(preall, MAT_FINAL_ASSEMBLY);

//set values for actual matrix row by row
MatSetValues(B, 1, &m, 1, nCols, matCols, INSERT_VALUES);

MatAssemblyBegin(B, MAT_FINAL_ASSEMBLY);
MatAssemblyEnd(B, MAT_FINAL_ASSEMBLY);

    KSPCreate(MPI_COMM_WORLD, &ksp);
    KSPSetOperators(ksp, A, B);
    KSPSetFromOptions(ksp);

    PC pc;
    KSPGetPC(ksp, &pc);
    PCSetFromOptions(pc);

From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Thursday, August 18, 2022 12:59 PM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] Using matrix-free with KSP

CAUTION: This email was sent from outside the UMass Lowell network.


  Yes, once you have preallocated the real matrix you can destroy the preallocation matrix whose only job is to gather the preallocation information





On Aug 18, 2022, at 12:52 PM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

Thanks, Barry.

So I need actually two matrixes, one for preallocator and one for actual matrix that can be passed to KSPSetOperators(). -mat_type preallocator option is used to speed up doing insertion into preallocator, then use MatPreallocatorPreallocate() to preallocate actual matrix and do actual insertion of values into it., right?

The code runs in parallel. Each process owns number of rows that equals to number of unknowns (that is, xm in 1D DM) it owns.

Jiannan

From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Thursday, August 18, 2022 11:37 AM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] Using matrix-free with KSP

CAUTION: This email was sent from outside the UMass Lowell network.


   The preallocator MatType matrix cannot be passed to the KSPSetOperators(), you need to create an actual matrix, for example MATAIJ and use the preallocator to set its preallocation and then fill up the MATAIJ matrix with the usual MatSetValues and pass that matrix to the KSPSetOperators.






On Aug 18, 2022, at 11:30 AM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

Hi Barry,

The MATPREALLOCATOR solved problem of slow matrix assembly. But the solver failed because of ?Matrix type preallocator does not have a multiply defined?.

I guess this is because the matrix-free is used. I am wondering how a preconditioner is applied to the matrix-vector multiply in Petsc.

Thank you,
Jiannan

[0]PETSC ERROR: No support for this operation for this object type
[0]PETSC ERROR: Matrix type preallocator does not have a multiply defined
[0]PETSC ERROR: See https://petsc.org/release/faq/<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Frelease%2Ffaq%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7Cc1e9f88adcda42f7c8b408da846dae97%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637967903891169434%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=aQgMnGt515eFE4s6OnTPPOylUFUaLPCpO678MJNvBQs%3D&reserved=0> for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.16.6, Mar 30, 2022
[0]PETSC ERROR: ./iesolver on a  named REXNET-WS4 by jiannantu Thu Aug 18 11:21:55 2022
[0]PETSC ERROR: Configure options --download-f2cblaslapack=yes --with-mpi-dir=/usr/local --with-fc=0 --prefix=/home/jiannantu/petsc --with-scalar-type=complex --with-64-bit-indices=1
[0]PETSC ERROR: #1 MatMult() at /home/jiannantu/petsc-3.16.6/src/mat/interface/matrix.c:2437
[0]PETSC ERROR: #2 PCApply_Mat() at /home/jiannantu/petsc-3.16.6/src/ksp/pc/impls/mat/pcmat.c:9
[0]PETSC ERROR: #3 PCApply() at /home/jiannantu/petsc-3.16.6/src/ksp/pc/interface/precon.c:445
[0]PETSC ERROR: #4 KSP_PCApply() at /home/jiannantu/petsc-3.16.6/include/petsc/private/kspimpl.h:382
[0]PETSC ERROR: #5 KSPInitialResidual() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itres.c:65
[0]PETSC ERROR: #6 KSPSolve_BCGS() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/impls/bcgs/bcgs.c:43
[0]PETSC ERROR: #7 KSPSolve_Private() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itfunc.c:914
[0]PETSC ERROR: #8 KSPSolve() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itfunc.c:1086

From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Thursday, August 18, 2022 8:35 AM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] Using matrix-free with KSP

CAUTION: This email was sent from outside the UMass Lowell network.


  Slow assembly is usually due to under preallocation. You can change to using

MatSetOption(B, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_TRUE);

  to detect if you are under preallocating. See https://petsc.org/main/docs/manual/mat/#sec-matsparse<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanual%2Fmat%2F%23sec-matsparse&data=05%7C01%7CJiannan_Tu%40uml.edu%7Cc1e9f88adcda42f7c8b408da846dae97%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637967903891169434%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=p1XPEBBZycyyYnbb2U3ESQGlQV44ZfqzwYL857jOz5c%3D&reserved=0>. That section clearly needs some sprucing up. If it is difficult to determine good preallocation values you can use https://petsc.org/main/docs/manualpages/Mat/MATPREALLOCATOR/<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMATPREALLOCATOR%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7Cc1e9f88adcda42f7c8b408da846dae97%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637967903891169434%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=YE9ST8%2B1vojZ1NlVkzQm26VjaiGZ4RJOya7NGe7avJs%3D&reserved=0> to compute the needed preallocation efficiently.

  Are you running in parallel? If so how are you determining which rows of entries to compute on each MPI rank? You will want most of the rows to be computed on the rank where the values are stored, you can determine this with https://petsc.org/main/docs/manualpages/Mat/MatGetOwnershipRange.html<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMatGetOwnershipRange.html&data=05%7C01%7CJiannan_Tu%40uml.edu%7Cc1e9f88adcda42f7c8b408da846dae97%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637967903891169434%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=%2B0kDehpllbZjZGY5HvxmY%2BQ2cmKO4AX5DkNCQlbllsg%3D&reserved=0>

  Barry

















On Aug 18, 2022, at 12:50 AM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

I implemented the preconditioner. The solver converges quickly when the problem size is small. But when the size increases, say to 100k unknowns, the code hangs at assembly of the preconditioner matrix. The function call sequence is like this

   MatCreate(MPI_COMM_WORLD, &B);
    MatSetType(B, MATMPIAIJ);
    MatSetSizes(B, PETSC_DECIDE, PETSC_DECIDE, N, N);
    MatSetFromOptions(B);

    // The number of diagonal and off diagonal non zeros only can be determined run-time.
    MatMPIAIJSetPreallocation(B, 0, d_nnz, 0, o_nnz);
    MatSetOption(B, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE);

    //here insert values row by row.
MatSetValues(B, 1, &m, nn, nCols, matCols, INSERT_VALUES);

MatAssemblyBegin(B, MAT_FINAL_ASSEMBLY);
MatAssemblyEnd(B, MAT_FINAL_ASSEMBLY);

Could you please tell me what I have done wrong?

Thank you,
Jiannan

From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Tuesday, August 16, 2022 2:00 PM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] Using matrix-free with KSP

CAUTION: This email was sent from outside the UMass Lowell network.


   Create another matrix B that contains "Selected parts of the matrix can be pre-calculated and stored to serve as the preconditioner." When you call KSPSetOperators() pass in B as the second matrix argument.

   Now if literally application of B is the preconditioner you want to use, then simply use PCMAT as the preconditioner, and it will apply B each time it needs to apply the preconditioner. With this, you do not even need a PCShell.

   If your preconditioner uses B to construct your own specific data structure needed to apply your preconditioner, then you can use PCShellSetSetUp() and have your routine call PCGetOperators() to pull out the B matrix and use it to construct your preconditioner.

   If you want PETSc to construct a standard preconditioner from B, then you don't need the PCShell; you simply pass in the B, as above, and use -pc_type type to set the preconditioner you want to use.

   Barry









On Aug 16, 2022, at 12:31 PM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

Barry,

Thank you very much for your instructions. I am sorry I may not ask clearer questions.

I don't use SNES. What I am trying to solve is a very large linear equation system with dense and ill-conditioned matrix. Selected parts of the matrix can be pre-calculated and stored to serve as the preconditioner. My question is how I can apply this matrix as the preconditioner.

Here is the part of code from my linear equations solver. MatrixFreePreconditioner is supposed to be the routine to provide the preconditioner. But I don't understand how to use it. Should I use SNESSetJacobian() even if this is a linear problem?

    MatCreateMFFD(MPI_COMM_WORLD, PETSC_DECIDE, PETSC_DECIDE, N, N, &A);
    MatMFFDSetFunction(A, formfunction, &params);
    MatSetFromOptions(A);
    MatMFFDSetBase(A, X, NULL);

    KSPCreate(MPI_COMM_WORLD, &ksp);
    KSPSetOperators(ksp, A, A);
    KSPSetFromOptions(ksp);

    PC pc;
    KSPGetPC(ksp,&pc);
    PCSetType(pc,PCSHELL);
    PCShellSetApply(pc,MatrixFreePreconditioner);

Thank you,
Jiannan

________________________________
From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Tuesday, August 16, 2022 10:10 AM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] Using matrix-free with KSP

CAUTION: This email was sent from outside the UMass Lowell network.


  I don't fully understand your question so I will answer questions that I do know the answer to :-)

  The function you provide to PCShellSetApply() applies the preconditioner to the input vector and puts the result in the output vector.  This input vector changes at every application of the preconditioner so should not be used in constructing the preconditioner.

  If you are using the standard PETSc matrix-free matrix-vector product via finite difference, -snes_mf_operator or via MatCreateSNESMF() or you are using a MatShell, the matrix-vector product routines input changes for each multiply and should not be used in constructing the preconditioner.

  The function you provide with SNESSetJacobian() is where you can compute anything you need to help build your preconditioner. The input vector to that function is the location at which the Jacobian is needed and is what you should use to build your approximate/part of Jacobian. You can store your approximate/part of Jacobian that you need to compute in the pmat and then in a PCShellSetUp() function that you provide you will compute what you will need latter to apply the preconditioner in your PCShellSetApply() function. Note that the values you put in the pmat do not need to be the full Jacobian; they can be anything you want since this pmat is not used to apply the matrix vector product.

  If this is unclear, feel free to ask additional questions.

  Barry









On Aug 16, 2022, at 9:40 AM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

I am trying to apply a user-defined preconditioner to speed up the convergency of matrix-free KSP solver. The user's manual provides an example of how to call the user-defined preconditioner routine. The routine has PC, input Vec and output Vec as arguments. I want to use part of the Jacobian as preconditioner matrix, the elements of which can be calculated using input Vec. My question is what the role of output Vec from the routine is in matrix-vector product? Or I just have the preconditioner matrix elements calculated in the routine and then Petsc handles applying the preconditioner to matrix-vector product? I used PCSetType(pc, PCSHELL) and PCShellSetApply.

Thank you and appreciate your help!

Jiannan
________________________________
From: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Sent: Wednesday, July 6, 2022 2:34 PM
To: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Cc: Jed Brown <jed at jedbrown.org<mailto:jed at jedbrown.org>>; petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] Using matrix-free with KSP

Barry,

VecScatterCreateToAll solves the problem! The code now gives the correct answer with multiple processes.

Thank you all so much!

Jiannan
________________________________
From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Wednesday, July 6, 2022 2:08 PM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Cc: Jed Brown <jed at jedbrown.org<mailto:jed at jedbrown.org>>; petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] Using matrix-free with KSP


  So your operator is a "dense" operator in that its matrix representation would be essentially dense. In that case, you can use specialized routines to do this efficiently. You can use VecScatterCreateToAll() and then VecScatterBegin/End to do the communication.

  Barry










On Jul 6, 2022, at 1:39 PM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>  wrote:

Jed, thank you very much for the reply.

I'm not sure GlobaltoLocal will work. Say the solution vector is of length 10. Two processes then each process can see 5 elements of the vector. For A x = b (10 equations), each process performs matrix-vector product for five equations. And for each equation i , sum_j A_ij *  x_j = b_i requires the process has access to all 10 unknows x_i (i=0,1,...,9).

Can VecScatter and PetscSF make the entire vector accessible to each process? I am reading the manual and trying to understand how VecScatter and PetscSF work.

Jiannan
________________________________
From: Jed Brown <jed at jedbrown.org<mailto:jed at jedbrown.org>>
Sent: Wednesday, July 6, 2022 10:09 AM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>; Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] Using matrix-free with KSP

You'll usually have a GlobalToLocal operation for each rank to get the halo data it needs, then either a LocalToGlobal to collect the result (e.g., finite element methods) or the local compute will write directly into the owned portion of the global vector. If you're doing the communication "raw", then you may find VecScatter or PetscSF useful to perform the necessary communication.

"Tu, Jiannan" <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>  writes:

> Hi Barry,
>
> Following your instructions I implemented the matrix-free method to solve a large linear equation system resulted from a surface integration equation. The KSP solver works fine for single process, but it is problematic with multiple processes. The problem is that each process only can access its own part of solution vector so that each process only conducts part of matrix-vector multiplication. MPI can be used to assemble these partial matrix-vector multiplication together.  Does Petsc provide any ways to implement multi-process matrix-free method?
>
> Thanks,
> Jiannan
> ________________________________
> From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
> Sent: Tuesday, May 24, 2022 2:12 PM
> To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
> Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
> Subject: Re: [petsc-users] Using matrix-free with KSP
>
> This e-mail originated from outside the UMass Lowell network.
> ________________________________
>
>    You can use MatCreateMFFD https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMatCreateMFFD%2F&amp;data=05%7C01%7CJiannan_Tu%40uml.edu%7Cab0859d8d154471590bc08da5f5918d6%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637927133525745809%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=A7wqegTfh94No5BpDiWLK3VxOuR44U2wlWHVm2k7l60%3D&amp;reserved=0<https://urldefense.com/v3/__https://petsc.org/main/docs/manualpages/Mat/MatCreateMFFD/__;!!PVKG_VDCxu5g!stxq2NWGVjUihvHBQC9Rlk0aHVoowcy0PTQYMcQxQ4DqOEC05KSw6TmstSXKckiUgelHzy4ue-d10-zlXkw$><https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMatCreateMFFD%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7Cc1e9f88adcda42f7c8b408da846dae97%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637967903891325655%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=OH0Bs%2BaniUQaAhidHWg0JNSsakEXfTaddvArBg%2BNbwU%3D&reserved=0> MatMFFDSetFunction MatSetFromOptions MatMFFDSetBase and provide the matrix to KSP. Note you will need to use -pc_type none or provide your own custom preconditioner withhttps://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FPC%2FPCSHELL%2F&amp;data=05%7C01%7CJiannan_Tu%40uml.edu%7Cab0859d8d154471590bc08da5f5918d6%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637927133525745809%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=Gb%2F9uFG3jp%2FbfWaSh2cSEQLItHS9CuLwarzN0KcNiJY%3D&amp;reserved=0<https://urldefense.com/v3/__https://petsc.org/main/docs/manualpages/PC/PCSHELL/__;!!PVKG_VDCxu5g!stxq2NWGVjUihvHBQC9Rlk0aHVoowcy0PTQYMcQxQ4DqOEC05KSw6TmstSXKckiUgelHzy4ue-d1cA4fKGw$><https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FPC%2FPCSHELL%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7Cc1e9f88adcda42f7c8b408da846dae97%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637967903891325655%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=jyMwuz6XPcmbNa9KliKnH0qQo0wo6qZvdrZtZRvXuxA%3D&reserved=0>
>
>
>
> On May 24, 2022, at 1:21 PM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu><mailto:Jiannan_Tu at uml.edu>> wrote:
>
> I want to use a matrix-free matrix to solve a large linear equation system because the matrix is too large to be stored. Petsc user manual describes matrix-free method for SNES with examples. The matrix-free matrices section explains how to set up such a matrix, but I can't find any example of matrix-free method with KSP. I am wondering how to apply the method to KSP iterative solver in parallel.
>
> I greatly appreciate your help for me to understand this topic.
>
> Jiannan Tu

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220823/9c11de1b/attachment-0001.html>

From sagitter at fedoraproject.org  Tue Aug 23 11:02:20 2022
From: sagitter at fedoraproject.org (Antonio T. sagitter)
Date: Tue, 23 Aug 2022 18:02:20 +0200
Subject: [petsc-users] PETSc on fedora 36
In-Reply-To: <8848fba2-d1fb-f5e6-da12-f5ad9985aeae@upc.edu>
References: <aa3cbfe2-2ccc-93cc-6004-5c2ae6429047@upc.edu>
	<358BFF34-D941-42AB-8648-871186EF64DB@petsc.dev>
	<65d7f5ad-fcd5-2cc9-0f74-8c9c8f5eaeb8@upc.edu>
	<CAMYG4GkBXMO3t7sMgOXS-Qwns6Uy1Eg3hYg6rref+Hre-u0hxA@mail.gmail.com>
	<acda5081-735d-dc66-ca08-91105d5585e1@upc.edu>
	<CAMYG4GmN_+VRt_cMf+KROyHDMe6SMyycwEeBtGJCfw3qC-0fFA@mail.gmail.com>
	<8848fba2-d1fb-f5e6-da12-f5ad9985aeae@upc.edu>
Message-ID: <bcbae348-2315-3108-4016-b931681ae4f5@fedoraproject.org>

Hi all.

On 23/08/22 16:33, Rafel Amer Ramon wrote:
> Hi,
> 
> the files rules and variables are in /usr/lib64/petsc/conf, so I have set
> 
> ~$ export PETSC_DIR=/usr
> 
> ~$ cat Makefile
> 
> e: e.o
>  ?? ?${CLINKER} -o e e.o ${PETSC_LIB}
> 
> include ${PETSC_DIR}/lib64/petsc/conf/variables
> include ${PETSC_DIR}/lib64/petsc/conf/rules
> 
> ~$
> 
> but, when I try to compile the program, I get the error
> 
> ~$ make e
> gcc -Wl,-z,relro -Wl,--as-needed  -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -Wl,--build-id=sha1 -Wl,-dT,/builddir/build/BUILD/petsc-3.16.4/.package_note-petsc-3.16.4-3.fc36.x86_64.ld -fPIC -Wl,-z,relro -Wl,--as-needed  -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -Wl,--build-id=sha1 -Wl,-dT,/builddir/build/BUILD/petsc-3.16.4/.package_note-petsc-3.16.4-3.fc36.x86_64.ld -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64  -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -O3 -fPIC -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64  -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -O3  -o e e.o -L/usr/lib64 -L/usr//lib -lpetsc -lsuperlu -lflexiblas -lcgns -lhdf5 -lhdf5_hl -lmetis -lm -lX11 -lquadmath -lstdc++ -ldl
> /usr/bin/ld: cannot open linker script file /builddir/build/BUILD/petsc-3.16.4/.package_note-petsc-3.16.4-3.fc36.x86_64.ld: El fitxer o directori no existeix
> collect2: error: ld returned 1 exit status
> make: *** [Makefile:2: e] Error 1
> 
> In the file /usr/lib64/petsc/conf/petscvariables, there are some 
> references to the inexistent file 
> /builddir/build/BUILD/petsc-3.16.4/.package_note-petsc-3.16.4-3.fc36.x86_64.ld:
> 
> ~$ grep /builddir/build/BUILD/petsc-3.16.4/.package_note-petsc-3.16.4-3.fc36.x86_64.ld /usr/lib64/petsc/conf/petscvariables
> CXX_LINKER_FLAGS = -Wl,-z,relro -Wl,--as-needed  -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -Wl,--build-id=sha1 -Wl,-dT,/builddir/build/BUILD/petsc-3.16.4/.package_note-petsc-3.16.4-3.fc36.x86_64.ld -fPIC  -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64  -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -O3 -fPIC -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64  -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -O3
> PCC_LINKER_FLAGS = -Wl,-z,relro -Wl,--as-needed  -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -Wl,--build-id=sha1 -Wl,-dT,/builddir/build/BUILD/petsc-3.16.4/.package_note-petsc-3.16.4-3.fc36.x86_64.ld -fPIC -Wl,-z,relro -Wl,--as-needed  -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -Wl,--build-id=sha1 -Wl,-dT,/builddir/build/BUILD/petsc-3.16.4/.package_note-petsc-3.16.4-3.fc36.x86_64.ld -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64  -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -O3 -fPIC -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64  -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -O3
> FC_LINKER_FLAGS = -Wl,-z,relro -Wl,--as-needed  -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -Wl,--build-id=sha1 -Wl,-dT,/builddir/build/BUILD/petsc-3.16.4/.package_note-petsc-3.16.4-3.fc36.x86_64.ld -fPIC -Wl,-z,relro -Wl,--as-needed  -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -Wl,--build-id=sha1 -Wl,-dT,/builddir/build/BUILD/petsc-3.16.4/.package_note-petsc-3.16.4-3.fc36.x86_64.ld -lgfortran -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64  -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -I/usr/lib64/gfortran/modules -O3 -fPIC -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64  -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -I/usr/lib64/gfortran/modules -O3
> CONFIGURE_OPTIONS = --build=x86_64-redhat-linux-gnu --host=x86_64-redhat-linux-gnu --program-prefix= --with-dependency-tracking=0 --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc --datadir=/usr/share --includedir=/usr/include --libdir=/usr/lib64 --libexecdir=/usr/libexec --localstatedir=/var --sharedstatedir=/var/lib --mandir=/usr/share/man --infodir=/usr/share/info --with-cc=gcc --with-cxx=g++ --with-fc=gfortran CFLAGS=\"-O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64  -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -O3 -fPIC\" CXXFLAGS=\"-O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64  -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -O3 -fPIC\" FFLAGS=\"-O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64  -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -I/usr/lib64/gfortran/modules -O3 -fPIC\" LDFLAGS=\"-Wl,-z,relro -Wl,--as-needed  -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -Wl,--build-id=sha1 -Wl,-dT,/builddir/build/BUILD/petsc-3.16.4/.package_note-petsc-3.16.4-3.fc36.x86_64.ld -fPIC\" COPTFLAGS=\"-O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64  -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -O3\" CXXOPTFLAGS=\"-O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64  -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -O3\" FOPTFLAGS=\"-O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64  -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -I/usr/lib64/gfortran/modules -O3\" FCFLAGS=\"-O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64  -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -I/usr/lib64/gfortran/modules -O3 -fPIC\" --CC_LINKER_FLAGS=\"-Wl,-z,relro -Wl,--as-needed  -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -Wl,--build-id=sha1 -Wl,-dT,/builddir/build/BUILD/petsc-3.16.4/.package_note-petsc-3.16.4-3.fc36.x86_64.ld\" --FC_LINKER_FLAGS=\"-Wl,-z,relro -Wl,--as-needed  -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -Wl,--build-id=sha1 -Wl,-dT,/builddir/build/BUILD/petsc-3.16.4/.package_note-petsc-3.16.4-3.fc36.x86_64.ld -lgfortran\" --with-default-arch=0 --with-make=1 --with-cmake-exec=/usr/bin/cmake3 --with-ctest-exec=/usr/bin/ctest3 --with-single-library=1 --with-precision=double --with-petsc-arch=x86_64 --with-clanguage=C --with-shared-libraries=1 --with-fortran-interfaces=1 --with-windows-graphics=0 --CC=gcc --FC=gfortran --CXX=g++ --with-shared-ld=ld --with-pic=1 --with-clib-autodetect=0 --with-fortranlib-autodetect=0 --with-threadsafety=0 --with-log=1 --with-mkl_sparse=0 --with-mkl_sparse_optimize=0 --with-mkl_cpardiso=0 --with-mkl_pardiso=0 --with-python=0 --with-cxxlib-autodetect=1 --with-debugging=0 --with-mumps-serial=1 --with-mpi=0 --with-hdf5=1 --with-hdf5-include= --with-hdf5-lib=\"-lhdf5 -lhdf5_hl\" --with-cgns=1 --with-cgns-include= --with-cgns-lib=-lcgns --with-x=1 --with-openmp=0 --with-hwloc=0 --with-ssl=0 --with-pthread=1 --with-valgrind=1 --with-64-bit-indices=0 --with-blaslapack=1 --with-blaslapack-lib=-lflexiblas --with-blaslapack-include=/usr/include/flexiblas --with-metis=1 --with-superlu=1 --with-superlu-include=/usr/include/SuperLU --with-superlu-lib=-lsuperlu
> SL_LINKER_FUNCTION = -Wl,-z,relro -Wl,--as-needed  -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -Wl,--build-id=sha1 -Wl,-dT,/builddir/build/BUILD/petsc-3.16.4/.package_note-petsc-3.16.4-3.fc36.x86_64.ld -fPIC -shared -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64  -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -O3 -fPIC -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64  -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -O3 -Wl,-soname,$(call SONAME_FUNCTION,$(notdir $(1)),$(2))
> ~$
> 
> 
> then, I get the error
> 
> /usr/bin/ld: cannot open linker script file /builddir/build/BUILD/petsc-3.16.4/.package_note-petsc-3.16.4-3.fc36.x86_64.ld: the file or folder doesn't exist
> 
> Best regards,
> 
> Rafel Amer
> 
> 
> 
> 

Links to unknown folders are generated by a new functionality of RPM 
packaging in Fedora 
(https://fedoraproject.org/wiki/Changes/Package_information_on_ELF_objects#Package_information_on_ELF_objects), 

the files rules and variables acquire these broken links during RPM 
building.

I'm disabling "package-note" links in the PETSc rpms in next build 
release. Please, let me know if they're still there by opening a 
bugzilla ticket (https://bugzilla.redhat.com/) if you still want to use 
PETSc from Fedora repositories.

-- 
---
Antonio Trande
Fedora Project
mailto: sagitter at fedoraproject.org
GPG key: 0xCC1CFEF30920C8AE
GPG key server: https://keyserver1.pgp.com/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_0xCC1CFEF30920C8AE.asc
Type: application/pgp-keys
Size: 1798 bytes
Desc: OpenPGP public key
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220823/b5081cbd/attachment.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature
Type: application/pgp-signature
Size: 495 bytes
Desc: OpenPGP digital signature
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220823/b5081cbd/attachment.sig>

From rafel.amer at upc.edu  Tue Aug 23 11:03:48 2022
From: rafel.amer at upc.edu (Rafel Amer Ramon)
Date: Tue, 23 Aug 2022 18:03:48 +0200
Subject: [petsc-users] PETSc on fedora 36. Solved
In-Reply-To: <5f57fbe-c49a-baad-faa-24e49ed6ed45@mcs.anl.gov>
References: <aa3cbfe2-2ccc-93cc-6004-5c2ae6429047@upc.edu>
	<358BFF34-D941-42AB-8648-871186EF64DB@petsc.dev>
	<65d7f5ad-fcd5-2cc9-0f74-8c9c8f5eaeb8@upc.edu>
	<5f57fbe-c49a-baad-faa-24e49ed6ed45@mcs.anl.gov>
Message-ID: <af11ac6f-a545-0927-7053-4e56e9b5e994@upc.edu>

An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220823/6fce8f41/attachment-0001.html>

From bsmith at petsc.dev  Tue Aug 23 11:08:08 2022
From: bsmith at petsc.dev (Barry Smith)
Date: Tue, 23 Aug 2022 12:08:08 -0400
Subject: [petsc-users] Using matrix-free with KSP
In-Reply-To: <BY5PR02MB64712BB4AF9E7CD2A66FE2F3E7709@BY5PR02MB6471.namprd02.prod.outlook.com>
References: <BY5PR02MB6471969F80FA5F732FC74D87E7D79@BY5PR02MB6471.namprd02.prod.outlook.com>
	<B55CFC2C-6364-4B78-8E99-9D53700743B8@petsc.dev>
	<BY5PR02MB6471BDF90783FE3A0EA176CBE7809@BY5PR02MB6471.namprd02.prod.outlook.com>
	<87iloa49kt.fsf@jedbrown.org>
	<BY5PR02MB6471C0ED04769D02046F68ACE7809@BY5PR02MB6471.namprd02.prod.outlook.com>
	<00EBE4B8-22E5-408F-B56D-14226A278D7E@petsc.dev>
	<BY5PR02MB6471E386057DDDE98AD05DFAE7809@BY5PR02MB6471.namprd02.prod.outlook.com>
	<BY5PR02MB647140084C7E7CDDB6BC2A11E76B9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<A6629DE1-5A7F-414E-B164-A69877158675@petsc.dev>
	<BY5PR02MB6471AB1C004920DA1FC9B43EE76B9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<103146FD-FF28-47CE-B661-F2B521BF0E7A@petsc.dev>
	<DM6PR02MB6478B2FB8CB1BE757CE7B003E76D9@DM6PR02MB6478.namprd02.prod.outlook.com>
	<4C00F4B8-BB42-4932-BCD7-527E5FB39BFF@petsc.dev>
	<BY5PR02MB64710E70758E431108D66956E76D9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<D0EB13D8-C5A6-4AE7-AFDE-2CDCCACBAF73@petsc.dev>
	<BY5PR02MB6471D4B05874DECFDDA13FFCE76D9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<367B124C-88BB-4889-A4AF-45A1D473078D@petsc.dev>
	<BY5PR02MB6471F2C058B84DE092567731E76D9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<944DDEED-016D-43B2-A5A6-7BF2E8E41FA5@petsc.dev>
	<BY5PR02MB6471CD0236F03D20763A3B64E76C9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<B4CF5E5B-984D-4EC1-AB77-1FD08C3FBBCF@petsc.dev>
	<BY5PR02MB6471BA72871419781B875967E76C9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<3D179AEA-AD05-4B46-8DDC-9E126741C247@petsc.dev>
	<BY5PR02MB6471560EA70C6FAE826DDF27E7719@BY5PR02MB6471.namprd02.prod.outlook.com>
	<1C67C8DB-0653-4588-8137-E0865E0F2098@petsc.dev>
	<BY5PR02MB6471C5EAE18AB9B114E76232E7719@BY5PR02MB6471.namprd02.prod.outlook.com>
	<FAA1668F-08C1-4BAD-89B4-784D8AD99C36@petsc.dev>
	<BY5PR02MB64715D7FE585A7F88092E04DE7719@BY5PR02MB6471.namprd02.prod.outlook.com>
	<F4C76705-B402-4F8B-A42C-DDBE95574065@petsc.dev>
	<BY5PR02MB64712BB4AF9E7CD2A66FE2F3E7709@BY5PR02MB6471.namprd02.prod.outlook.com>
Message-ID: <D2A1B111-0AC4-441B-AA81-9011967F7641@petsc.dev>


 KSPSetOperators(ksp, params.A, params.A); should be  KSPSetOperators(B, params.A);



> On Aug 23, 2022, at 12:00 PM, Tu, Jiannan <Jiannan_Tu at uml.edu> wrote:
> 
> Hi Barry,
> 
> I celebrated too early. Just found that the matrix-vector multiply function put into MatShellSetOperation() is not called. I set a printf() in it no effect. Then I comment out all execution statements int it, KSP converges the same way and the solution is the same. Apparently, that mat-vec multiply function doesn't take effect.  
> 
> Following is complete sequency of KSP calls. Could you please help me identify what the problem is? 
> 
> Thank you very much.
> 
> Jiannan
> 
> ------------------------------------
>     DMDACreate1d(MPI_COMM_WORLD, DM_BOUNDARY_GHOSTED, N, 1, 1, NULL, &da);
>     DMSetFromOptions(da);
>     DMSetUp(da);
>     params.da = da;
> 
>     DMDAGetCorners(da, &xs, NULL, NULL, &xm, NULL, NULL);
> 
>     DMCreateGlobalVector(da, &X);
>     VecDuplicate(X, &Rm);
> 
>     MatCreate(MPI_COMM_WORLD, &params.A);
>     MatSetType(params.A, MATMPIAIJ);
>     MatSetSizes(params.A, PETSC_DECIDE, PETSC_DECIDE, N, N);
> 
>     MatCreateShell(MPI_COMM_WORLD, xm, N, PETSC_DETERMINE, PETSC_DETERMINE, &params, &B);
>     MatShellSetOperation(B, MATOP_MULT, (void(*)(void))matvecmult);
> 
>     Mat C;
>     MatCreate(MPI_COMM_WORLD, &C);
>     MatSetType(C, MATPREALLOCATOR);
>     MatSetSizes(C, PETSC_DECIDE, PETSC_DECIDE, N, N);
> 
>     //calculate matrix elements from nearby interactions
>     matrix_near_elements(&params, params.A, C);
>     MatDestroy(&C);
> 
>     KSPCreate(MPI_COMM_WORLD, &ksp);
>     KSPSetOperators(ksp, params.A, params.A);
>     KSPSetFromOptions(ksp);
>     
>     PC pc;
>     KSPGetPC(ksp, &pc);
>     PCSetFromOptions(pc);
> 
>     //quantities needed to calculate interactions with far groups
>     fmm_formular(&params);
> 
>     //compute right hand-side vector
>     computeRhs(&params, Rm);
> 
>     //solve matrix equation
>     KSPSolve(ksp, Rm, X);
> 
> From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>>
> Sent: Monday, August 22, 2022 2:39 PM
> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
> Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>>
> Subject: Re: [petsc-users] Using matrix-free with KSP
>  
> CAUTION: This email was sent from outside the UMass Lowell network.
> 
> 
> 
>> On Aug 22, 2022, at 1:37 PM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> wrote:
>> 
>> Barry,
>> 
>> Yes, you are right, Anear is the preconditioner that needs to be passed to KSPSetOperators() as the second matrix. 
>> 
>> I think I have figured out how to provide a preconditioner for matrix-free method. I was confused by matrix A created by MatCreateMFFD(), which is type MFFD. I thought I have to use it as the first matrix passed to KSPSetOperators(). Since it was not assembled (and can't) so that Petsc gave error message "matrix unassembled". In fact, I should ignore A, but use other two matrixes AA and B, which are type MPIAIJ, assemble them, pass them to KSPSetOperators() as first and second matrix. Since function passed into MatMFFDSetFunction() does not take a matrix as an argument, matrix AA needs to be declared in user context so that it can be used in that function to perform matrix-vector product (partially) using MatMult() as you suggested. 
>> 
>    I think your AA should be the shell matrix, and the B should be Anear. There should be no MatMFFD. Inside the AA shell matrix mat-mult you call MatMult( on the Anear put Anear in the Shell matrix context also) and then add the part of the multiply coming from the far field.
> 
>  
>> I have used a small test program to solve an 8x8 linear equation system and the code produces the same answers with/without the preconditioner. The iteration for this small problem, of course no improvement with the preconditioner but the residual norms with/without the preconditioner are slightly different. I suppose the preconditioner takes effect.
>> 
>> Now I need to think about if the applied preconditioner does the right job on the other part of matrix-vector product (non near part) for the problem I am working on.
>> 
>> Thank you very very much and greatly appreciate your help!
>> 
>> Jiannan
>> From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>>
>> Sent: Monday, August 22, 2022 12:42 PM
>> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
>> Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>>
>> Subject: Re: [petsc-users] Using matrix-free with KSP
>>  
>> CAUTION: This email was sent from outside the UMass Lowell network.
>> 
>> 
>> 
>>> On Aug 22, 2022, at 12:25 PM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> wrote:
>>> 
>>> The matrix-vector product involves part of matrix elements explicitly as shown below so that I need to insert values into this part of the matrix.
>>> <image.png>
>>> Initially I wrote my own code for calculating and storing matrix A^near_mi without using any Petsc matrix and then performed matrix-vector product using above equation. The code runs but converges very slow. Apparently a preconditioned is needed. 
>>> 
>>> I am not clear how Petsc can provide a preconditioner for my case.
>> 
>>   I am now confused. I thought you were forming Anear as a PETSc matrix and passing that to KSPSetOperators as the second matrix and so the preconditioner was being built with Anear?
>> 
>>   For your shell matrix vector product you should call MatMult(Anear,x,y) inside your custom routine and then add the rest of the matrix-vector product using your fast multipole products. What am I missing?
>> 
>>   Barry
>> 
>>> 
>>> From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>>
>>> Sent: Monday, August 22, 2022 8:42 AM
>>> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
>>> Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>>
>>> Subject: Re: [petsc-users] Using matrix-free with KSP
>>>  
>>> CAUTION: This email was sent from outside the UMass Lowell network.
>>> 
>>> 
>>> 
>>>> On Aug 21, 2022, at 11:18 PM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> wrote:
>>>> 
>>>> Barry,
>>>>  
>>>> You are right. With PC type none, the same error appears.
>>>>  
>>>> Now I use MatShell and I literally have three matrixes. A is a shell matrix, B is a preconditioner, and C is a matrix just for preallocation purpose.
>>>>  
>>>> First C is used to determine non-zero structure. Then use MatPreallocatorPreallocation(C, PETSC_FALSE, B) to preallocate for B. After that do insertion and assembly for B. These steps work fine.
>>>>  
>>>> Now the question is how to insert and assemble shell matrix A. MatPreallocatorPreallocation doesn?t work with shell matrix as indicated by an error message. MatSetValues or Mat assembly doesn?t support shell matrix either. 
>>> 
>>>   A shell matrix is completely under your control. You decide on what data you want to store in it, and you provide the matrix-vector product routine.  MatPreallocatorPreallocation and MatSetValues don't work for shell matrix unless you provide code for them to work, which normally you would not do.
>>> 
>>>   I don't understand why you want to use MatSetValues for your shell matrix. If you can use MatSetValues() and just build an explicit matrix, then why would you use a shell matrix, just use MATAIJ matrix? 
>>> 
>>>   Or does part of the operation inside your MatShell require a "regular" PETSc matrix? This would be fine, but it is your responsibility to design your Mat shell context to contain the "regular matrix" and your responsibility to fill up that regular matrix inside your shell context, calling whatever code you choose to write. So for example your shell context would be a C struct and one of the fields would be a regular PETSc matrix, also part of your custom matrix-vector product routine would call a regular MatMult() on the regular PETSc matrix stored inside the C struct.
>>> 
>>>   Barry
>>> 
>>> 
>>>>  
>>>> Thank you,
>>>> Jiannan
>>>>  
>>>> From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>> 
>>>> Sent: Friday, August 19, 2022 4:02 PM
>>>> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
>>>> Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>
>>>> Subject: Re: [petsc-users] Using matrix-free with KSP
>>>>  
>>>> CAUTION: This email was sent from outside the UMass Lowell network.
>>>>  
>>>>  
>>>>   I don't know why changing the PCType could matter, can you please try with your new code but use the -pc_type none option to turn off the use of the preconditioner to see if the error still appears (I think it should). 
>>>>  
>>>>   Also if you have a custom function that can do the matrix-vector product (not using any differencing) then you should use MatShell() and provide the matrix-vector product operation with your code.
>>>>  
>>>>   Barry
>>>>  
>>>> 
>>>> 
>>>> On Aug 19, 2022, at 11:47 AM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> wrote:
>>>>  
>>>> Barry,
>>>>  
>>>> I have my own matrix-vector product but not use MatShell. I followed your previous instruction to use
>>>>  
>>>>     MatCreateMFFD(MPI_COMM_WORLD, PETSC_DECIDE, PETSC_DECIDE, N, N, &A);
>>>>     MatMFFDSetFunction(A, formfunction, &params);
>>>>     MatSetFromOptions(A);
>>>>     MatMFFDSetBase(A, X, NULL);
>>>>  
>>>> The customized matrix-vector product is implemented in formfunction() provided to MatMFFDSetFunction(). In this way, the matrix A is not used. I think that is why Petsc complains the matrix is not assembled. However, when PC type is none, the code runs without error although KSP solver doesn?t converge.
>>>>  
>>>> The question then becomes how to link A to matrix-free matrix-vector multiplication. And where to put MatShell?
>>>>  
>>>> Thank you,
>>>> Jiannan
>>>>  
>>>> From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>> 
>>>> Sent: Friday, August 19, 2022 10:27 AM
>>>> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
>>>> Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>
>>>> Subject: Re: [petsc-users] Using matrix-free with KSP
>>>>  
>>>> CAUTION: This email was sent from outside the UMass Lowell network.
>>>>  
>>>>  
>>>> 
>>>> 
>>>> 
>>>> On Aug 18, 2022, at 11:41 PM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> wrote:
>>>>  
>>>> The program calls KSPSolve just once and the error occurs after the first iteration.
>>>>  
>>>> I realized the matrix-vector product in my case is different from that approximating matrix-vector multiply by finite difference of function vector. It consists of two parts. One is multiplication of part of the matrix with part of vector, and another part results from so called fast multipole method. I?m not clear how to relate such calculation of matrix-vector product to the finite differencing of function vector.
>>>>  
>>>>   Are you using finite differencing at all in a matrix-vector product, or do you have your own custom matrix-vector product code (with MatShell)? 
>>>>  
>>>>   Anyways the easiest thing to do is 
>>>>  
>>>>   use the run time option -on_error_attach_debugger noxterm and run the code, 
>>>>  
>>>>   when it stops in the debugger you can type bt (for backtrace)
>>>>  
>>>>   then do up a few times to move up the stack until you get to the PETSc function call that generates the error
>>>>  
>>>>   you can then do "call MatView(thematrixvarible,0) to have PETSc display what the matrix is and
>>>>  
>>>>   you can also do print *thematrixvariable to see the exact struct with all its fields
>>>>  
>>>>   If this doesn't tell you why the troublesome matrix is not assembled then cut and paste ALL of the output and email it
>>>>  
>>>>   Barry
>>>> 
>>>> 
>>>> 
>>>>  
>>>>  
>>>>  
>>>> From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>> 
>>>> Sent: Thursday, August 18, 2022 7:44 PM
>>>> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
>>>> Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>
>>>> Subject: Re: [petsc-users] Using matrix-free with KSP
>>>>  
>>>> CAUTION: This email was sent from outside the UMass Lowell network.
>>>>  
>>>>  
>>>>  
>>>>    Is it in the first iteration of the first KSPSolve that you get this message? Or in the second call to KSPSolve? 
>>>>  
>>>>   The error comes from a matrix vector product with your A matrix, what is that and where did it come from. The error message is not from your B matrix, that is not used in the MatMult()
>>>>  
>>>> 
>>>> 
>>>> 
>>>> 
>>>> On Aug 18, 2022, at 3:05 PM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> wrote:
>>>>  
>>>> Barry, 
>>>>  
>>>> Thanks. The method works. Assembly of actual matrix now completes quickly. The convergency is much better for the first iteration. The residual normal is in the order of 10^-10 instead of 10^-2. However, there is an error message after the first iteration: ?Object is in wrong state. Not for unassembled matrix? even though both preallocator and actual matrix have been assembled (see the code snippet below). Could you please give me some hints what?s going wrong? I use KSP type bcgs and PC type mat
>>>>  
>>>> Thank you,
>>>> Jiannan
>>>>  
>>>> -----------------------------------------------------
>>>> Solve linear equations ...
>>>>   0 KSP Residual norm 9.841179147519e-10
>>>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>>>> [0]PETSC ERROR: Object is in wrong state
>>>> [0]PETSC ERROR: Not for unassembled matrix
>>>> [0]PETSC ERROR: See https://petsc.org/release/faq/ <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Frelease%2Ffaq%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7Cc1e9f88adcda42f7c8b408da846dae97%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637967903891169434%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=aQgMnGt515eFE4s6OnTPPOylUFUaLPCpO678MJNvBQs%3D&reserved=0> for trouble shooting.
>>>> [0]PETSC ERROR: Petsc Release Version 3.16.6, Mar 30, 2022
>>>> [0]PETSC ERROR: ./iesolver on a  named REXNET-WS4 by jiannantu Thu Aug 18 14:48:43 2022
>>>> [0]PETSC ERROR: Configure options --download-f2cblaslapack=yes --with-mpi-dir=/usr/local --with-fc=0 --prefix=/home/jiannantu/petsc --with-scalar-type=complex --with-64-bit-indices=1
>>>> [0]PETSC ERROR: #1 MatMult() at /home/jiannantu/petsc-3.16.6/src/mat/interface/matrix.c:2425
>>>> [0]PETSC ERROR: #2 PCApplyBAorAB() at /home/jiannantu/petsc-3.16.6/src/ksp/pc/interface/precon.c:730
>>>> [0]PETSC ERROR: #3 KSP_PCApplyBAorAB() at /home/jiannantu/petsc-3.16.6/include/petsc/private/kspimpl.h:421
>>>> [0]PETSC ERROR:  Linear equations not solved. Diverged reason: 0
>>>> #4 KSPSolve_BCGS() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/impls/bcgs/bcgs.c:87
>>>> [0]PETSC ERROR: #5 KSPSolve_Private() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itfunc.c:914
>>>> [0]PETSC ERROR: #6 KSPSolve() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itfunc.c:1086
>>>>  
>>>> -----------------------------------------------
>>>>     MatCreate(MPI_COMM_WORLD, &B);
>>>>     MatSetType(B, MATMPIAIJ);
>>>>     MatSetSizes(B, PETSC_DECIDE, PETSC_DECIDE, N, N);
>>>>  
>>>>     Mat preall;
>>>>     MatCreate(MPI_COMM_WORLD, &preall);
>>>>     MatSetType(preall, MATPREALLOCATOR);
>>>>     MatSetSizes(preall, PETSC_DECIDE, PETSC_DECIDE, N, N);
>>>>  
>>>>     MatMPIAIJSetPreallocation(preall, 0, d_nnz, 0, o_nnz);
>>>>     MatSetUp(preall);
>>>> MatSetOption(preall, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE);
>>>>  
>>>> //set values row by row
>>>> MatSetValues(preall, 1, &m, 1, nCols, matCols, INSERT_VALUES);
>>>>  
>>>> MatAssemblyBegin(preall, MAT_FINAL_ASSEMBLY);
>>>> MatAssemblyEnd(preall, MAT_FINAL_ASSEMBLY);
>>>>  
>>>> //set values for actual matrix row by row
>>>> MatSetValues(B, 1, &m, 1, nCols, matCols, INSERT_VALUES);
>>>>  
>>>> MatAssemblyBegin(B, MAT_FINAL_ASSEMBLY);
>>>> MatAssemblyEnd(B, MAT_FINAL_ASSEMBLY);
>>>>  
>>>>     KSPCreate(MPI_COMM_WORLD, &ksp);
>>>>     KSPSetOperators(ksp, A, B);
>>>>     KSPSetFromOptions(ksp);
>>>>  
>>>>     PC pc;
>>>>     KSPGetPC(ksp, &pc);
>>>>     PCSetFromOptions(pc);
>>>>  
>>>> From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>> 
>>>> Sent: Thursday, August 18, 2022 12:59 PM
>>>> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
>>>> Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>
>>>> Subject: Re: [petsc-users] Using matrix-free with KSP
>>>>  
>>>> CAUTION: This email was sent from outside the UMass Lowell network.
>>>>  
>>>>  
>>>>   Yes, once you have preallocated the real matrix you can destroy the preallocation matrix whose only job is to gather the preallocation information
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> On Aug 18, 2022, at 12:52 PM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> wrote:
>>>>  
>>>> Thanks, Barry.
>>>>  
>>>> So I need actually two matrixes, one for preallocator and one for actual matrix that can be passed to KSPSetOperators(). -mat_type preallocator option is used to speed up doing insertion into preallocator, then use MatPreallocatorPreallocate() to preallocate actual matrix and do actual insertion of values into it., right?
>>>>  
>>>> The code runs in parallel. Each process owns number of rows that equals to number of unknowns (that is, xm in 1D DM) it owns.
>>>>  
>>>> Jiannan
>>>>  
>>>> From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>> 
>>>> Sent: Thursday, August 18, 2022 11:37 AM
>>>> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
>>>> Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>
>>>> Subject: Re: [petsc-users] Using matrix-free with KSP
>>>>  
>>>> CAUTION: This email was sent from outside the UMass Lowell network.
>>>>  
>>>>  
>>>>    The preallocator MatType matrix cannot be passed to the KSPSetOperators(), you need to create an actual matrix, for example MATAIJ and use the preallocator to set its preallocation and then fill up the MATAIJ matrix with the usual MatSetValues and pass that matrix to the KSPSetOperators.
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> On Aug 18, 2022, at 11:30 AM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> wrote:
>>>>  
>>>> Hi Barry,
>>>>  
>>>> The MATPREALLOCATOR solved problem of slow matrix assembly. But the solver failed because of ?Matrix type preallocator does not have a multiply defined?.
>>>>  
>>>> I guess this is because the matrix-free is used. I am wondering how a preconditioner is applied to the matrix-vector multiply in Petsc.
>>>>  
>>>> Thank you,
>>>> Jiannan
>>>>  
>>>> [0]PETSC ERROR: No support for this operation for this object type
>>>> [0]PETSC ERROR: Matrix type preallocator does not have a multiply defined
>>>> [0]PETSC ERROR: See https://petsc.org/release/faq/ <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Frelease%2Ffaq%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7Cc1e9f88adcda42f7c8b408da846dae97%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637967903891169434%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=aQgMnGt515eFE4s6OnTPPOylUFUaLPCpO678MJNvBQs%3D&reserved=0> for trouble shooting.
>>>> [0]PETSC ERROR: Petsc Release Version 3.16.6, Mar 30, 2022
>>>> [0]PETSC ERROR: ./iesolver on a  named REXNET-WS4 by jiannantu Thu Aug 18 11:21:55 2022
>>>> [0]PETSC ERROR: Configure options --download-f2cblaslapack=yes --with-mpi-dir=/usr/local --with-fc=0 --prefix=/home/jiannantu/petsc --with-scalar-type=complex --with-64-bit-indices=1
>>>> [0]PETSC ERROR: #1 MatMult() at /home/jiannantu/petsc-3.16.6/src/mat/interface/matrix.c:2437
>>>> [0]PETSC ERROR: #2 PCApply_Mat() at /home/jiannantu/petsc-3.16.6/src/ksp/pc/impls/mat/pcmat.c:9
>>>> [0]PETSC ERROR: #3 PCApply() at /home/jiannantu/petsc-3.16.6/src/ksp/pc/interface/precon.c:445
>>>> [0]PETSC ERROR: #4 KSP_PCApply() at /home/jiannantu/petsc-3.16.6/include/petsc/private/kspimpl.h:382
>>>> [0]PETSC ERROR: #5 KSPInitialResidual() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itres.c:65
>>>> [0]PETSC ERROR: #6 KSPSolve_BCGS() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/impls/bcgs/bcgs.c:43
>>>> [0]PETSC ERROR: #7 KSPSolve_Private() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itfunc.c:914
>>>> [0]PETSC ERROR: #8 KSPSolve() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itfunc.c:1086
>>>>  
>>>> From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>> 
>>>> Sent: Thursday, August 18, 2022 8:35 AM
>>>> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
>>>> Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>
>>>> Subject: Re: [petsc-users] Using matrix-free with KSP
>>>>  
>>>> CAUTION: This email was sent from outside the UMass Lowell network.
>>>>  
>>>>  
>>>>   Slow assembly is usually due to under preallocation. You can change to using 
>>>>  
>>>> MatSetOption(B, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_TRUE);
>>>>  
>>>>   to detect if you are under preallocating. See https://petsc.org/main/docs/manual/mat/#sec-matsparse <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanual%2Fmat%2F%23sec-matsparse&data=05%7C01%7CJiannan_Tu%40uml.edu%7Cc1e9f88adcda42f7c8b408da846dae97%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637967903891169434%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=p1XPEBBZycyyYnbb2U3ESQGlQV44ZfqzwYL857jOz5c%3D&reserved=0>. That section clearly needs some sprucing up. If it is difficult to determine good preallocation values you can use https://petsc.org/main/docs/manualpages/Mat/MATPREALLOCATOR/ <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMATPREALLOCATOR%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7Cc1e9f88adcda42f7c8b408da846dae97%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637967903891169434%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=YE9ST8%2B1vojZ1NlVkzQm26VjaiGZ4RJOya7NGe7avJs%3D&reserved=0> to compute the needed preallocation efficiently.
>>>>  
>>>>   Are you running in parallel? If so how are you determining which rows of entries to compute on each MPI rank? You will want most of the rows to be computed on the rank where the values are stored, you can determine this with https://petsc.org/main/docs/manualpages/Mat/MatGetOwnershipRange.html <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMatGetOwnershipRange.html&data=05%7C01%7CJiannan_Tu%40uml.edu%7Cc1e9f88adcda42f7c8b408da846dae97%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637967903891169434%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=%2B0kDehpllbZjZGY5HvxmY%2BQ2cmKO4AX5DkNCQlbllsg%3D&reserved=0>
>>>>  
>>>>   Barry
>>>>  
>>>>  
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>>  
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> On Aug 18, 2022, at 12:50 AM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> wrote:
>>>>  
>>>> I implemented the preconditioner. The solver converges quickly when the problem size is small. But when the size increases, say to 100k unknowns, the code hangs at assembly of the preconditioner matrix. The function call sequence is like this
>>>>  
>>>>    MatCreate(MPI_COMM_WORLD, &B);
>>>>     MatSetType(B, MATMPIAIJ);
>>>>     MatSetSizes(B, PETSC_DECIDE, PETSC_DECIDE, N, N);
>>>>     MatSetFromOptions(B);
>>>>  
>>>>     // The number of diagonal and off diagonal non zeros only can be determined run-time.
>>>>     MatMPIAIJSetPreallocation(B, 0, d_nnz, 0, o_nnz);
>>>>     MatSetOption(B, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE);
>>>>  
>>>>     //here insert values row by row.
>>>> MatSetValues(B, 1, &m, nn, nCols, matCols, INSERT_VALUES);
>>>>  
>>>> MatAssemblyBegin(B, MAT_FINAL_ASSEMBLY);
>>>> MatAssemblyEnd(B, MAT_FINAL_ASSEMBLY);
>>>>  
>>>> Could you please tell me what I have done wrong?
>>>>  
>>>> Thank you,
>>>> Jiannan
>>>>    
>>>> From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>> 
>>>> Sent: Tuesday, August 16, 2022 2:00 PM
>>>> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
>>>> Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>
>>>> Subject: Re: [petsc-users] Using matrix-free with KSP
>>>>  
>>>> CAUTION: This email was sent from outside the UMass Lowell network.
>>>>  
>>>>  
>>>>    Create another matrix B that contains "Selected parts of the matrix can be pre-calculated and stored to serve as the preconditioner." When you call KSPSetOperators() pass in B as the second matrix argument.
>>>>  
>>>>    Now if literally application of B is the preconditioner you want to use, then simply use PCMAT as the preconditioner, and it will apply B each time it needs to apply the preconditioner. With this, you do not even need a PCShell. 
>>>>  
>>>>    If your preconditioner uses B to construct your own specific data structure needed to apply your preconditioner, then you can use PCShellSetSetUp() and have your routine call PCGetOperators() to pull out the B matrix and use it to construct your preconditioner. 
>>>>  
>>>>    If you want PETSc to construct a standard preconditioner from B, then you don't need the PCShell; you simply pass in the B, as above, and use -pc_type type to set the preconditioner you want to use. 
>>>>  
>>>>    Barry
>>>>  
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> On Aug 16, 2022, at 12:31 PM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> wrote:
>>>>  
>>>> Barry,
>>>>  
>>>> Thank you very much for your instructions. I am sorry I may not ask clearer questions.
>>>>  
>>>> I don't use SNES. What I am trying to solve is a very large linear equation system with dense and ill-conditioned matrix. Selected parts of the matrix can be pre-calculated and stored to serve as the preconditioner. My question is how I can apply this matrix as the preconditioner. 
>>>>  
>>>> Here is the part of code from my linear equations solver. MatrixFreePreconditioner is supposed to be the routine to provide the preconditioner. But I don't understand how to use it. Should I use SNESSetJacobian() even if this is a linear problem? 
>>>>  
>>>>     MatCreateMFFD(MPI_COMM_WORLD, PETSC_DECIDE, PETSC_DECIDE, N, N, &A);
>>>>     MatMFFDSetFunction(A, formfunction, &params);
>>>>     MatSetFromOptions(A);
>>>>     MatMFFDSetBase(A, X, NULL);
>>>>  
>>>>     KSPCreate(MPI_COMM_WORLD, &ksp);
>>>>     KSPSetOperators(ksp, A, A);
>>>>     KSPSetFromOptions(ksp);
>>>>  
>>>>     PC pc;
>>>>     KSPGetPC(ksp,&pc);
>>>>     PCSetType(pc,PCSHELL);
>>>>     PCShellSetApply(pc,MatrixFreePreconditioner);
>>>>  
>>>> Thank you,
>>>> Jiannan
>>>>  
>>>> From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>>
>>>> Sent: Tuesday, August 16, 2022 10:10 AM
>>>> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
>>>> Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>>
>>>> Subject: Re: [petsc-users] Using matrix-free with KSP
>>>>  
>>>> CAUTION: This email was sent from outside the UMass Lowell network.
>>>>  
>>>>  
>>>>   I don't fully understand your question so I will answer questions that I do know the answer to :-)
>>>>  
>>>>   The function you provide to PCShellSetApply() applies the preconditioner to the input vector and puts the result in the output vector.  This input vector changes at every application of the preconditioner so should not be used in constructing the preconditioner.
>>>>  
>>>>   If you are using the standard PETSc matrix-free matrix-vector product via finite difference, -snes_mf_operator or via MatCreateSNESMF() or you are using a MatShell, the matrix-vector product routines input changes for each multiply and should not be used in constructing the preconditioner.
>>>>  
>>>>   The function you provide with SNESSetJacobian() is where you can compute anything you need to help build your preconditioner. The input vector to that function is the location at which the Jacobian is needed and is what you should use to build your approximate/part of Jacobian. You can store your approximate/part of Jacobian that you need to compute in the pmat and then in a PCShellSetUp() function that you provide you will compute what you will need latter to apply the preconditioner in your PCShellSetApply() function. Note that the values you put in the pmat do not need to be the full Jacobian; they can be anything you want since this pmat is not used to apply the matrix vector product.
>>>>  
>>>>   If this is unclear, feel free to ask additional questions.
>>>>  
>>>>   Barry
>>>>  
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> On Aug 16, 2022, at 9:40 AM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> wrote:
>>>>  
>>>> I am trying to apply a user-defined preconditioner to speed up the convergency of matrix-free KSP solver. The user's manual provides an example of how to call the user-defined preconditioner routine. The routine has PC, input Vec and output Vec as arguments. I want to use part of the Jacobian as preconditioner matrix, the elements of which can be calculated using input Vec. My question is what the role of output Vec from the routine is in matrix-vector product? Or I just have the preconditioner matrix elements calculated in the routine and then Petsc handles applying the preconditioner to matrix-vector product? I used PCSetType(pc, PCSHELL) and PCShellSetApply.
>>>>  
>>>> Thank you and appreciate your help!
>>>>  
>>>> Jiannan 
>>>> From: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
>>>> Sent: Wednesday, July 6, 2022 2:34 PM
>>>> To: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>>
>>>> Cc: Jed Brown <jed at jedbrown.org <mailto:jed at jedbrown.org>>; petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>>
>>>> Subject: Re: [petsc-users] Using matrix-free with KSP
>>>>  
>>>> Barry,
>>>>  
>>>> VecScatterCreateToAll solves the problem! The code now gives the correct answer with multiple processes. 
>>>>  
>>>> Thank you all so much!
>>>>  
>>>> Jiannan
>>>> From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>>
>>>> Sent: Wednesday, July 6, 2022 2:08 PM
>>>> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
>>>> Cc: Jed Brown <jed at jedbrown.org <mailto:jed at jedbrown.org>>; petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>>
>>>> Subject: Re: [petsc-users] Using matrix-free with KSP
>>>>  
>>>>  
>>>>   So your operator is a "dense" operator in that its matrix representation would be essentially dense. In that case, you can use specialized routines to do this efficiently. You can use VecScatterCreateToAll() and then VecScatterBegin/End to do the communication.
>>>>  
>>>>   Barry
>>>>  
>>>>  
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> On Jul 6, 2022, at 1:39 PM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>  wrote:
>>>>  
>>>> Jed, thank you very much for the reply. 
>>>>  
>>>> I'm not sure GlobaltoLocal will work. Say the solution vector is of length 10. Two processes then each process can see 5 elements of the vector. For A x = b (10 equations), each process performs matrix-vector product for five equations. And for each equation i , sum_j A_ij *  x_j = b_i requires the process has access to all 10 unknows x_i (i=0,1,...,9). 
>>>>  
>>>> Can VecScatter and PetscSF make the entire vector accessible to each process? I am reading the manual and trying to understand how VecScatter and PetscSF work.
>>>>  
>>>> Jiannan
>>>> From: Jed Brown <jed at jedbrown.org <mailto:jed at jedbrown.org>>
>>>> Sent: Wednesday, July 6, 2022 10:09 AM
>>>> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>; Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>>
>>>> Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>>
>>>> Subject: Re: [petsc-users] Using matrix-free with KSP
>>>>  
>>>> You'll usually have a GlobalToLocal operation for each rank to get the halo data it needs, then either a LocalToGlobal to collect the result (e.g., finite element methods) or the local compute will write directly into the owned portion of the global vector. If you're doing the communication "raw", then you may find VecScatter or PetscSF useful to perform the necessary communication.
>>>> 
>>>> "Tu, Jiannan" <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>  writes:
>>>> 
>>>> > Hi Barry,
>>>> >
>>>> > Following your instructions I implemented the matrix-free method to solve a large linear equation system resulted from a surface integration equation. The KSP solver works fine for single process, but it is problematic with multiple processes. The problem is that each process only can access its own part of solution vector so that each process only conducts part of matrix-vector multiplication. MPI can be used to assemble these partial matrix-vector multiplication together.  Does Petsc provide any ways to implement multi-process matrix-free method?
>>>> >
>>>> > Thanks,
>>>> > Jiannan
>>>> > ________________________________
>>>> > From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>>
>>>> > Sent: Tuesday, May 24, 2022 2:12 PM
>>>> > To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
>>>> > Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>>
>>>> > Subject: Re: [petsc-users] Using matrix-free with KSP
>>>> >
>>>> > This e-mail originated from outside the UMass Lowell network.
>>>> > ________________________________
>>>> >
>>>> >    You can use MatCreateMFFD https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMatCreateMFFD%2F&amp;data=05%7C01%7CJiannan_Tu%40uml.edu%7Cab0859d8d154471590bc08da5f5918d6%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637927133525745809%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=A7wqegTfh94No5BpDiWLK3VxOuR44U2wlWHVm2k7l60%3D&amp;reserved=0<https://urldefense.com/v3/__https://petsc.org/main/docs/manualpages/Mat/MatCreateMFFD/__;!!PVKG_VDCxu5g!stxq2NWGVjUihvHBQC9Rlk0aHVoowcy0PTQYMcQxQ4DqOEC05KSw6TmstSXKckiUgelHzy4ue-d10-zlXkw$> <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMatCreateMFFD%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7Cc1e9f88adcda42f7c8b408da846dae97%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637967903891325655%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=OH0Bs%2BaniUQaAhidHWg0JNSsakEXfTaddvArBg%2BNbwU%3D&reserved=0> MatMFFDSetFunction MatSetFromOptions MatMFFDSetBase and provide the matrix to KSP. Note you will need to use -pc_type none or provide your own custom preconditioner withhttps://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FPC%2FPCSHELL%2F&amp;data=05%7C01%7CJiannan_Tu%40uml.edu%7Cab0859d8d154471590bc08da5f5918d6%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637927133525745809%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=Gb%2F9uFG3jp%2FbfWaSh2cSEQLItHS9CuLwarzN0KcNiJY%3D&amp;reserved=0<https://urldefense.com/v3/__https://petsc.org/main/docs/manualpages/PC/PCSHELL/__;!!PVKG_VDCxu5g!stxq2NWGVjUihvHBQC9Rlk0aHVoowcy0PTQYMcQxQ4DqOEC05KSw6TmstSXKckiUgelHzy4ue-d1cA4fKGw$> <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FPC%2FPCSHELL%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7Cc1e9f88adcda42f7c8b408da846dae97%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637967903891325655%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=jyMwuz6XPcmbNa9KliKnH0qQo0wo6qZvdrZtZRvXuxA%3D&reserved=0>
>>>> >
>>>> >
>>>> >
>>>> > On May 24, 2022, at 1:21 PM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu><mailto:Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>> wrote:
>>>> >
>>>> > I want to use a matrix-free matrix to solve a large linear equation system because the matrix is too large to be stored. Petsc user manual describes matrix-free method for SNES with examples. The matrix-free matrices section explains how to set up such a matrix, but I can't find any example of matrix-free method with KSP. I am wondering how to apply the method to KSP iterative solver in parallel.
>>>> >
>>>> > I greatly appreciate your help for me to understand this topic.
>>>> >
>>>> > Jiannan Tu

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220823/49f7274f/attachment-0001.html>

From Jiannan_Tu at uml.edu  Tue Aug 23 11:42:15 2022
From: Jiannan_Tu at uml.edu (Tu, Jiannan)
Date: Tue, 23 Aug 2022 16:42:15 +0000
Subject: [petsc-users] Using matrix-free with KSP
In-Reply-To: <D2A1B111-0AC4-441B-AA81-9011967F7641@petsc.dev>
References: <BY5PR02MB6471969F80FA5F732FC74D87E7D79@BY5PR02MB6471.namprd02.prod.outlook.com>
	<B55CFC2C-6364-4B78-8E99-9D53700743B8@petsc.dev>
	<BY5PR02MB6471BDF90783FE3A0EA176CBE7809@BY5PR02MB6471.namprd02.prod.outlook.com>
	<87iloa49kt.fsf@jedbrown.org>
	<BY5PR02MB6471C0ED04769D02046F68ACE7809@BY5PR02MB6471.namprd02.prod.outlook.com>
	<00EBE4B8-22E5-408F-B56D-14226A278D7E@petsc.dev>
	<BY5PR02MB6471E386057DDDE98AD05DFAE7809@BY5PR02MB6471.namprd02.prod.outlook.com>
	<BY5PR02MB647140084C7E7CDDB6BC2A11E76B9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<A6629DE1-5A7F-414E-B164-A69877158675@petsc.dev>
	<BY5PR02MB6471AB1C004920DA1FC9B43EE76B9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<103146FD-FF28-47CE-B661-F2B521BF0E7A@petsc.dev>
	<DM6PR02MB6478B2FB8CB1BE757CE7B003E76D9@DM6PR02MB6478.namprd02.prod.outlook.com>
	<4C00F4B8-BB42-4932-BCD7-527E5FB39BFF@petsc.dev>
	<BY5PR02MB64710E70758E431108D66956E76D9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<D0EB13D8-C5A6-4AE7-AFDE-2CDCCACBAF73@petsc.dev>
	<BY5PR02MB6471D4B05874DECFDDA13FFCE76D9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<367B124C-88BB-4889-A4AF-45A1D473078D@petsc.dev>
	<BY5PR02MB6471F2C058B84DE092567731E76D9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<944DDEED-016D-43B2-A5A6-7BF2E8E41FA5@petsc.dev>
	<BY5PR02MB6471CD0236F03D20763A3B64E76C9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<B4CF5E5B-984D-4EC1-AB77-1FD08C3FBBCF@petsc.dev>
	<BY5PR02MB6471BA72871419781B875967E76C9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<3D179AEA-AD05-4B46-8DDC-9E126741C247@petsc.dev>
	<BY5PR02MB6471560EA70C6FAE826DDF27E7719@BY5PR02MB6471.namprd02.prod.outlook.com>
	<1C67C8DB-0653-4588-8137-E0865E0F2098@petsc.dev>
	<BY5PR02MB6471C5EAE18AB9B114E76232E7719@BY5PR02MB6471.namprd02.prod.outlook.com>
	<FAA1668F-08C1-4BAD-89B4-784D8AD99C36@petsc.dev>
	<BY5PR02MB64715D7FE585A7F88092E04DE7719@BY5PR02MB6471.namprd02.prod.outlook.com>
	<F4C76705-B402-4F8B-A42C-DDBE95574065@petsc.dev>
	<BY5PR02MB64712BB4AF9E7CD2A66FE2F3E7709@BY5PR02MB6471.namprd02.prod.outlook.com>
	<D2A1B111-0AC4-441B-AA81-9011967F7641@petsc.dev>
Message-ID: <BY5PR02MB6471300031FBDA77EDABA2FCE7709@BY5PR02MB6471.namprd02.prod.outlook.com>

I misunderstood the usage. Inserting params.A in the first place lets KSP use it as linear equation matrix and ignores mat-vec multiply function.

Thank you very much,
Jiannan
________________________________
From: Barry Smith <bsmith at petsc.dev>
Sent: Tuesday, August 23, 2022 12:08 PM
To: Tu, Jiannan <Jiannan_Tu at uml.edu>
Cc: petsc-users at mcs.anl.gov <petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] Using matrix-free with KSP


CAUTION: This email was sent from outside the UMass Lowell network.


 KSPSetOperators(ksp, params.A, params.A); should be  KSPSetOperators(B, params.A);



On Aug 23, 2022, at 12:00 PM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

Hi Barry,

I celebrated too early. Just found that the matrix-vector multiply function put into MatShellSetOperation() is not called. I set a printf() in it no effect. Then I comment out all execution statements int it, KSP converges the same way and the solution is the same. Apparently, that mat-vec multiply function doesn't take effect.

Following is complete sequency of KSP calls. Could you please help me identify what the problem is?

Thank you very much.

Jiannan

------------------------------------
    DMDACreate1d(MPI_COMM_WORLD, DM_BOUNDARY_GHOSTED, N, 1, 1, NULL, &da);
    DMSetFromOptions(da);
    DMSetUp(da);
    params.da = da;

    DMDAGetCorners(da, &xs, NULL, NULL, &xm, NULL, NULL);

    DMCreateGlobalVector(da, &X);
    VecDuplicate(X, &Rm);

    MatCreate(MPI_COMM_WORLD, &params.A);
    MatSetType(params.A, MATMPIAIJ);
    MatSetSizes(params.A, PETSC_DECIDE, PETSC_DECIDE, N, N);

    MatCreateShell(MPI_COMM_WORLD, xm, N, PETSC_DETERMINE, PETSC_DETERMINE, &params, &B);
    MatShellSetOperation(B, MATOP_MULT, (void(*)(void))matvecmult);

    Mat C;
    MatCreate(MPI_COMM_WORLD, &C);
    MatSetType(C, MATPREALLOCATOR);
    MatSetSizes(C, PETSC_DECIDE, PETSC_DECIDE, N, N);

    //calculate matrix elements from nearby interactions
    matrix_near_elements(&params, params.A, C);
    MatDestroy(&C);

    KSPCreate(MPI_COMM_WORLD, &ksp);
    KSPSetOperators(ksp, params.A, params.A);
    KSPSetFromOptions(ksp);

    PC pc;
    KSPGetPC(ksp, &pc);
    PCSetFromOptions(pc);

    //quantities needed to calculate interactions with far groups
    fmm_formular(&params);

    //compute right hand-side vector
    computeRhs(&params, Rm);

    //solve matrix equation
    KSPSolve(ksp, Rm, X);

________________________________
From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Monday, August 22, 2022 2:39 PM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] Using matrix-free with KSP


CAUTION: This email was sent from outside the UMass Lowell network.



On Aug 22, 2022, at 1:37 PM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

Barry,

Yes, you are right, Anear is the preconditioner that needs to be passed to KSPSetOperators() as the second matrix.

I think I have figured out how to provide a preconditioner for matrix-free method. I was confused by matrix A created by MatCreateMFFD(), which is type MFFD. I thought I have to use it as the first matrix passed to KSPSetOperators(). Since it was not assembled (and can't) so that Petsc gave error message "matrix unassembled". In fact, I should ignore A, but use other two matrixes AA and B, which are type MPIAIJ, assemble them, pass them to KSPSetOperators() as first and second matrix. Since function passed into MatMFFDSetFunction() does not take a matrix as an argument, matrix AA needs to be declared in user context so that it can be used in that function to perform matrix-vector product (partially) using MatMult() as you suggested.

   I think your AA should be the shell matrix, and the B should be Anear. There should be no MatMFFD. Inside the AA shell matrix mat-mult you call MatMult( on the Anear put Anear in the Shell matrix context also) and then add the part of the multiply coming from the far field.


I have used a small test program to solve an 8x8 linear equation system and the code produces the same answers with/without the preconditioner. The iteration for this small problem, of course no improvement with the preconditioner but the residual norms with/without the preconditioner are slightly different. I suppose the preconditioner takes effect.

Now I need to think about if the applied preconditioner does the right job on the other part of matrix-vector product (non near part) for the problem I am working on.

Thank you very very much and greatly appreciate your help!

Jiannan
________________________________
From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Monday, August 22, 2022 12:42 PM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] Using matrix-free with KSP


CAUTION: This email was sent from outside the UMass Lowell network.



On Aug 22, 2022, at 12:25 PM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

The matrix-vector product involves part of matrix elements explicitly as shown below so that I need to insert values into this part of the matrix.
<image.png>
Initially I wrote my own code for calculating and storing matrix A^near_mi without using any Petsc matrix and then performed matrix-vector product using above equation. The code runs but converges very slow. Apparently a preconditioned is needed.

I am not clear how Petsc can provide a preconditioner for my case.

  I am now confused. I thought you were forming Anear as a PETSc matrix and passing that to KSPSetOperators as the second matrix and so the preconditioner was being built with Anear?

  For your shell matrix vector product you should call MatMult(Anear,x,y) inside your custom routine and then add the rest of the matrix-vector product using your fast multipole products. What am I missing?

  Barry

________________________________
From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Monday, August 22, 2022 8:42 AM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] Using matrix-free with KSP


CAUTION: This email was sent from outside the UMass Lowell network.



On Aug 21, 2022, at 11:18 PM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

Barry,

You are right. With PC type none, the same error appears.

Now I use MatShell and I literally have three matrixes. A is a shell matrix, B is a preconditioner, and C is a matrix just for preallocation purpose.

First C is used to determine non-zero structure. Then use MatPreallocatorPreallocation(C, PETSC_FALSE, B) to preallocate for B. After that do insertion and assembly for B. These steps work fine.

Now the question is how to insert and assemble shell matrix A. MatPreallocatorPreallocation doesn?t work with shell matrix as indicated by an error message. MatSetValues or Mat assembly doesn?t support shell matrix either.

  A shell matrix is completely under your control. You decide on what data you want to store in it, and you provide the matrix-vector product routine.  MatPreallocatorPreallocation and MatSetValues don't work for shell matrix unless you provide code for them to work, which normally you would not do.

  I don't understand why you want to use MatSetValues for your shell matrix. If you can use MatSetValues() and just build an explicit matrix, then why would you use a shell matrix, just use MATAIJ matrix?

  Or does part of the operation inside your MatShell require a "regular" PETSc matrix? This would be fine, but it is your responsibility to design your Mat shell context to contain the "regular matrix" and your responsibility to fill up that regular matrix inside your shell context, calling whatever code you choose to write. So for example your shell context would be a C struct and one of the fields would be a regular PETSc matrix, also part of your custom matrix-vector product routine would call a regular MatMult() on the regular PETSc matrix stored inside the C struct.

  Barry



Thank you,
Jiannan

From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Friday, August 19, 2022 4:02 PM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] Using matrix-free with KSP

CAUTION: This email was sent from outside the UMass Lowell network.


  I don't know why changing the PCType could matter, can you please try with your new code but use the -pc_type none option to turn off the use of the preconditioner to see if the error still appears (I think it should).

  Also if you have a custom function that can do the matrix-vector product (not using any differencing) then you should use MatShell() and provide the matrix-vector product operation with your code.

  Barry



On Aug 19, 2022, at 11:47 AM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

Barry,

I have my own matrix-vector product but not use MatShell. I followed your previous instruction to use

    MatCreateMFFD(MPI_COMM_WORLD, PETSC_DECIDE, PETSC_DECIDE, N, N, &A);
    MatMFFDSetFunction(A, formfunction, &params);
    MatSetFromOptions(A);
    MatMFFDSetBase(A, X, NULL);

The customized matrix-vector product is implemented in formfunction() provided to MatMFFDSetFunction(). In this way, the matrix A is not used. I think that is why Petsc complains the matrix is not assembled. However, when PC type is none, the code runs without error although KSP solver doesn?t converge.

The question then becomes how to link A to matrix-free matrix-vector multiplication. And where to put MatShell?

Thank you,
Jiannan

From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Friday, August 19, 2022 10:27 AM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] Using matrix-free with KSP

CAUTION: This email was sent from outside the UMass Lowell network.





On Aug 18, 2022, at 11:41 PM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

The program calls KSPSolve just once and the error occurs after the first iteration.

I realized the matrix-vector product in my case is different from that approximating matrix-vector multiply by finite difference of function vector. It consists of two parts. One is multiplication of part of the matrix with part of vector, and another part results from so called fast multipole method. I?m not clear how to relate such calculation of matrix-vector product to the finite differencing of function vector.

  Are you using finite differencing at all in a matrix-vector product, or do you have your own custom matrix-vector product code (with MatShell)?

  Anyways the easiest thing to do is

  use the run time option -on_error_attach_debugger noxterm and run the code,

  when it stops in the debugger you can type bt (for backtrace)

  then do up a few times to move up the stack until you get to the PETSc function call that generates the error

  you can then do "call MatView(thematrixvarible,0) to have PETSc display what the matrix is and

  you can also do print *thematrixvariable to see the exact struct with all its fields

  If this doesn't tell you why the troublesome matrix is not assembled then cut and paste ALL of the output and email it

  Barry






From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Thursday, August 18, 2022 7:44 PM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] Using matrix-free with KSP

CAUTION: This email was sent from outside the UMass Lowell network.



   Is it in the first iteration of the first KSPSolve that you get this message? Or in the second call to KSPSolve?

  The error comes from a matrix vector product with your A matrix, what is that and where did it come from. The error message is not from your B matrix, that is not used in the MatMult()





On Aug 18, 2022, at 3:05 PM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

Barry,

Thanks. The method works. Assembly of actual matrix now completes quickly. The convergency is much better for the first iteration. The residual normal is in the order of 10^-10 instead of 10^-2. However, there is an error message after the first iteration: ?Object is in wrong state. Not for unassembled matrix? even though both preallocator and actual matrix have been assembled (see the code snippet below). Could you please give me some hints what?s going wrong? I use KSP type bcgs and PC type mat

Thank you,
Jiannan

-----------------------------------------------------
Solve linear equations ...
  0 KSP Residual norm 9.841179147519e-10
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Object is in wrong state
[0]PETSC ERROR: Not for unassembled matrix
[0]PETSC ERROR: See https://petsc.org/release/faq/<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Frelease%2Ffaq%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7Cce0f40b04bda4d3bb36608da8521acb8%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637968676966261576%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=%2FF%2FegTjfpbU0iS7vGRDAhBF7x%2FbgTMplabEpa7Ued4o%3D&reserved=0> for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.16.6, Mar 30, 2022
[0]PETSC ERROR: ./iesolver on a  named REXNET-WS4 by jiannantu Thu Aug 18 14:48:43 2022
[0]PETSC ERROR: Configure options --download-f2cblaslapack=yes --with-mpi-dir=/usr/local --with-fc=0 --prefix=/home/jiannantu/petsc --with-scalar-type=complex --with-64-bit-indices=1
[0]PETSC ERROR: #1 MatMult() at /home/jiannantu/petsc-3.16.6/src/mat/interface/matrix.c:2425
[0]PETSC ERROR: #2 PCApplyBAorAB() at /home/jiannantu/petsc-3.16.6/src/ksp/pc/interface/precon.c:730
[0]PETSC ERROR: #3 KSP_PCApplyBAorAB() at /home/jiannantu/petsc-3.16.6/include/petsc/private/kspimpl.h:421
[0]PETSC ERROR:  Linear equations not solved. Diverged reason: 0
#4 KSPSolve_BCGS() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/impls/bcgs/bcgs.c:87
[0]PETSC ERROR: #5 KSPSolve_Private() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itfunc.c:914
[0]PETSC ERROR: #6 KSPSolve() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itfunc.c:1086

-----------------------------------------------
    MatCreate(MPI_COMM_WORLD, &B);
    MatSetType(B, MATMPIAIJ);
    MatSetSizes(B, PETSC_DECIDE, PETSC_DECIDE, N, N);

    Mat preall;
    MatCreate(MPI_COMM_WORLD, &preall);
    MatSetType(preall, MATPREALLOCATOR);
    MatSetSizes(preall, PETSC_DECIDE, PETSC_DECIDE, N, N);

    MatMPIAIJSetPreallocation(preall, 0, d_nnz, 0, o_nnz);
    MatSetUp(preall);
MatSetOption(preall, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE);

//set values row by row
MatSetValues(preall, 1, &m, 1, nCols, matCols, INSERT_VALUES);

MatAssemblyBegin(preall, MAT_FINAL_ASSEMBLY);
MatAssemblyEnd(preall, MAT_FINAL_ASSEMBLY);

//set values for actual matrix row by row
MatSetValues(B, 1, &m, 1, nCols, matCols, INSERT_VALUES);

MatAssemblyBegin(B, MAT_FINAL_ASSEMBLY);
MatAssemblyEnd(B, MAT_FINAL_ASSEMBLY);

    KSPCreate(MPI_COMM_WORLD, &ksp);
    KSPSetOperators(ksp, A, B);
    KSPSetFromOptions(ksp);

    PC pc;
    KSPGetPC(ksp, &pc);
    PCSetFromOptions(pc);

From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Thursday, August 18, 2022 12:59 PM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] Using matrix-free with KSP

CAUTION: This email was sent from outside the UMass Lowell network.


  Yes, once you have preallocated the real matrix you can destroy the preallocation matrix whose only job is to gather the preallocation information





On Aug 18, 2022, at 12:52 PM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

Thanks, Barry.

So I need actually two matrixes, one for preallocator and one for actual matrix that can be passed to KSPSetOperators(). -mat_type preallocator option is used to speed up doing insertion into preallocator, then use MatPreallocatorPreallocate() to preallocate actual matrix and do actual insertion of values into it., right?

The code runs in parallel. Each process owns number of rows that equals to number of unknowns (that is, xm in 1D DM) it owns.

Jiannan

From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Thursday, August 18, 2022 11:37 AM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] Using matrix-free with KSP

CAUTION: This email was sent from outside the UMass Lowell network.


   The preallocator MatType matrix cannot be passed to the KSPSetOperators(), you need to create an actual matrix, for example MATAIJ and use the preallocator to set its preallocation and then fill up the MATAIJ matrix with the usual MatSetValues and pass that matrix to the KSPSetOperators.






On Aug 18, 2022, at 11:30 AM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

Hi Barry,

The MATPREALLOCATOR solved problem of slow matrix assembly. But the solver failed because of ?Matrix type preallocator does not have a multiply defined?.

I guess this is because the matrix-free is used. I am wondering how a preconditioner is applied to the matrix-vector multiply in Petsc.

Thank you,
Jiannan

[0]PETSC ERROR: No support for this operation for this object type
[0]PETSC ERROR: Matrix type preallocator does not have a multiply defined
[0]PETSC ERROR: See https://petsc.org/release/faq/<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Frelease%2Ffaq%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7Cce0f40b04bda4d3bb36608da8521acb8%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637968676966261576%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=%2FF%2FegTjfpbU0iS7vGRDAhBF7x%2FbgTMplabEpa7Ued4o%3D&reserved=0> for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.16.6, Mar 30, 2022
[0]PETSC ERROR: ./iesolver on a  named REXNET-WS4 by jiannantu Thu Aug 18 11:21:55 2022
[0]PETSC ERROR: Configure options --download-f2cblaslapack=yes --with-mpi-dir=/usr/local --with-fc=0 --prefix=/home/jiannantu/petsc --with-scalar-type=complex --with-64-bit-indices=1
[0]PETSC ERROR: #1 MatMult() at /home/jiannantu/petsc-3.16.6/src/mat/interface/matrix.c:2437
[0]PETSC ERROR: #2 PCApply_Mat() at /home/jiannantu/petsc-3.16.6/src/ksp/pc/impls/mat/pcmat.c:9
[0]PETSC ERROR: #3 PCApply() at /home/jiannantu/petsc-3.16.6/src/ksp/pc/interface/precon.c:445
[0]PETSC ERROR: #4 KSP_PCApply() at /home/jiannantu/petsc-3.16.6/include/petsc/private/kspimpl.h:382
[0]PETSC ERROR: #5 KSPInitialResidual() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itres.c:65
[0]PETSC ERROR: #6 KSPSolve_BCGS() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/impls/bcgs/bcgs.c:43
[0]PETSC ERROR: #7 KSPSolve_Private() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itfunc.c:914
[0]PETSC ERROR: #8 KSPSolve() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itfunc.c:1086

From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Thursday, August 18, 2022 8:35 AM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] Using matrix-free with KSP

CAUTION: This email was sent from outside the UMass Lowell network.


  Slow assembly is usually due to under preallocation. You can change to using

MatSetOption(B, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_TRUE);

  to detect if you are under preallocating. See https://petsc.org/main/docs/manual/mat/#sec-matsparse<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanual%2Fmat%2F%23sec-matsparse&data=05%7C01%7CJiannan_Tu%40uml.edu%7Cce0f40b04bda4d3bb36608da8521acb8%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637968676966261576%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=W4yc%2BKlgsE6%2FAs4CVIEk3sg9FikUlfVdZRB2PMQO4Mc%3D&reserved=0>. That section clearly needs some sprucing up. If it is difficult to determine good preallocation values you can use https://petsc.org/main/docs/manualpages/Mat/MATPREALLOCATOR/<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMATPREALLOCATOR%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7Cce0f40b04bda4d3bb36608da8521acb8%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637968676966261576%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=pR8QnfL6k4hHgV%2BbuByyRdachoOAtAhoNhpzu3pQC%2FQ%3D&reserved=0> to compute the needed preallocation efficiently.

  Are you running in parallel? If so how are you determining which rows of entries to compute on each MPI rank? You will want most of the rows to be computed on the rank where the values are stored, you can determine this with https://petsc.org/main/docs/manualpages/Mat/MatGetOwnershipRange.html<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMatGetOwnershipRange.html&data=05%7C01%7CJiannan_Tu%40uml.edu%7Cce0f40b04bda4d3bb36608da8521acb8%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637968676966261576%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=izhtvCMmQIy59Eisqgz%2F8hUhk5NgVR6z%2B5%2Bc6yzE7G0%3D&reserved=0>

  Barry

















On Aug 18, 2022, at 12:50 AM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

I implemented the preconditioner. The solver converges quickly when the problem size is small. But when the size increases, say to 100k unknowns, the code hangs at assembly of the preconditioner matrix. The function call sequence is like this

   MatCreate(MPI_COMM_WORLD, &B);
    MatSetType(B, MATMPIAIJ);
    MatSetSizes(B, PETSC_DECIDE, PETSC_DECIDE, N, N);
    MatSetFromOptions(B);

    // The number of diagonal and off diagonal non zeros only can be determined run-time.
    MatMPIAIJSetPreallocation(B, 0, d_nnz, 0, o_nnz);
    MatSetOption(B, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE);

    //here insert values row by row.
MatSetValues(B, 1, &m, nn, nCols, matCols, INSERT_VALUES);

MatAssemblyBegin(B, MAT_FINAL_ASSEMBLY);
MatAssemblyEnd(B, MAT_FINAL_ASSEMBLY);

Could you please tell me what I have done wrong?

Thank you,
Jiannan

From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Tuesday, August 16, 2022 2:00 PM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] Using matrix-free with KSP

CAUTION: This email was sent from outside the UMass Lowell network.


   Create another matrix B that contains "Selected parts of the matrix can be pre-calculated and stored to serve as the preconditioner." When you call KSPSetOperators() pass in B as the second matrix argument.

   Now if literally application of B is the preconditioner you want to use, then simply use PCMAT as the preconditioner, and it will apply B each time it needs to apply the preconditioner. With this, you do not even need a PCShell.

   If your preconditioner uses B to construct your own specific data structure needed to apply your preconditioner, then you can use PCShellSetSetUp() and have your routine call PCGetOperators() to pull out the B matrix and use it to construct your preconditioner.

   If you want PETSc to construct a standard preconditioner from B, then you don't need the PCShell; you simply pass in the B, as above, and use -pc_type type to set the preconditioner you want to use.

   Barry









On Aug 16, 2022, at 12:31 PM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

Barry,

Thank you very much for your instructions. I am sorry I may not ask clearer questions.

I don't use SNES. What I am trying to solve is a very large linear equation system with dense and ill-conditioned matrix. Selected parts of the matrix can be pre-calculated and stored to serve as the preconditioner. My question is how I can apply this matrix as the preconditioner.

Here is the part of code from my linear equations solver. MatrixFreePreconditioner is supposed to be the routine to provide the preconditioner. But I don't understand how to use it. Should I use SNESSetJacobian() even if this is a linear problem?

    MatCreateMFFD(MPI_COMM_WORLD, PETSC_DECIDE, PETSC_DECIDE, N, N, &A);
    MatMFFDSetFunction(A, formfunction, &params);
    MatSetFromOptions(A);
    MatMFFDSetBase(A, X, NULL);

    KSPCreate(MPI_COMM_WORLD, &ksp);
    KSPSetOperators(ksp, A, A);
    KSPSetFromOptions(ksp);

    PC pc;
    KSPGetPC(ksp,&pc);
    PCSetType(pc,PCSHELL);
    PCShellSetApply(pc,MatrixFreePreconditioner);

Thank you,
Jiannan

________________________________
From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Tuesday, August 16, 2022 10:10 AM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] Using matrix-free with KSP

CAUTION: This email was sent from outside the UMass Lowell network.


  I don't fully understand your question so I will answer questions that I do know the answer to :-)

  The function you provide to PCShellSetApply() applies the preconditioner to the input vector and puts the result in the output vector.  This input vector changes at every application of the preconditioner so should not be used in constructing the preconditioner.

  If you are using the standard PETSc matrix-free matrix-vector product via finite difference, -snes_mf_operator or via MatCreateSNESMF() or you are using a MatShell, the matrix-vector product routines input changes for each multiply and should not be used in constructing the preconditioner.

  The function you provide with SNESSetJacobian() is where you can compute anything you need to help build your preconditioner. The input vector to that function is the location at which the Jacobian is needed and is what you should use to build your approximate/part of Jacobian. You can store your approximate/part of Jacobian that you need to compute in the pmat and then in a PCShellSetUp() function that you provide you will compute what you will need latter to apply the preconditioner in your PCShellSetApply() function. Note that the values you put in the pmat do not need to be the full Jacobian; they can be anything you want since this pmat is not used to apply the matrix vector product.

  If this is unclear, feel free to ask additional questions.

  Barry









On Aug 16, 2022, at 9:40 AM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>> wrote:

I am trying to apply a user-defined preconditioner to speed up the convergency of matrix-free KSP solver. The user's manual provides an example of how to call the user-defined preconditioner routine. The routine has PC, input Vec and output Vec as arguments. I want to use part of the Jacobian as preconditioner matrix, the elements of which can be calculated using input Vec. My question is what the role of output Vec from the routine is in matrix-vector product? Or I just have the preconditioner matrix elements calculated in the routine and then Petsc handles applying the preconditioner to matrix-vector product? I used PCSetType(pc, PCSHELL) and PCShellSetApply.

Thank you and appreciate your help!

Jiannan
________________________________
From: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Sent: Wednesday, July 6, 2022 2:34 PM
To: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Cc: Jed Brown <jed at jedbrown.org<mailto:jed at jedbrown.org>>; petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] Using matrix-free with KSP

Barry,

VecScatterCreateToAll solves the problem! The code now gives the correct answer with multiple processes.

Thank you all so much!

Jiannan
________________________________
From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Wednesday, July 6, 2022 2:08 PM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
Cc: Jed Brown <jed at jedbrown.org<mailto:jed at jedbrown.org>>; petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] Using matrix-free with KSP


  So your operator is a "dense" operator in that its matrix representation would be essentially dense. In that case, you can use specialized routines to do this efficiently. You can use VecScatterCreateToAll() and then VecScatterBegin/End to do the communication.

  Barry










On Jul 6, 2022, at 1:39 PM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>  wrote:

Jed, thank you very much for the reply.

I'm not sure GlobaltoLocal will work. Say the solution vector is of length 10. Two processes then each process can see 5 elements of the vector. For A x = b (10 equations), each process performs matrix-vector product for five equations. And for each equation i , sum_j A_ij *  x_j = b_i requires the process has access to all 10 unknows x_i (i=0,1,...,9).

Can VecScatter and PetscSF make the entire vector accessible to each process? I am reading the manual and trying to understand how VecScatter and PetscSF work.

Jiannan
________________________________
From: Jed Brown <jed at jedbrown.org<mailto:jed at jedbrown.org>>
Sent: Wednesday, July 6, 2022 10:09 AM
To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>; Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] Using matrix-free with KSP

You'll usually have a GlobalToLocal operation for each rank to get the halo data it needs, then either a LocalToGlobal to collect the result (e.g., finite element methods) or the local compute will write directly into the owned portion of the global vector. If you're doing the communication "raw", then you may find VecScatter or PetscSF useful to perform the necessary communication.

"Tu, Jiannan" <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>  writes:

> Hi Barry,
>
> Following your instructions I implemented the matrix-free method to solve a large linear equation system resulted from a surface integration equation. The KSP solver works fine for single process, but it is problematic with multiple processes. The problem is that each process only can access its own part of solution vector so that each process only conducts part of matrix-vector multiplication. MPI can be used to assemble these partial matrix-vector multiplication together.  Does Petsc provide any ways to implement multi-process matrix-free method?
>
> Thanks,
> Jiannan
> ________________________________
> From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
> Sent: Tuesday, May 24, 2022 2:12 PM
> To: Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu>>
> Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
> Subject: Re: [petsc-users] Using matrix-free with KSP
>
> This e-mail originated from outside the UMass Lowell network.
> ________________________________
>
>    You can use MatCreateMFFD https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMatCreateMFFD%2F&amp;data=05%7C01%7CJiannan_Tu%40uml.edu%7Cab0859d8d154471590bc08da5f5918d6%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637927133525745809%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=A7wqegTfh94No5BpDiWLK3VxOuR44U2wlWHVm2k7l60%3D&amp;reserved=0<https://urldefense.com/v3/__https://petsc.org/main/docs/manualpages/Mat/MatCreateMFFD/__;!!PVKG_VDCxu5g!stxq2NWGVjUihvHBQC9Rlk0aHVoowcy0PTQYMcQxQ4DqOEC05KSw6TmstSXKckiUgelHzy4ue-d10-zlXkw$><https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMatCreateMFFD%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7Cce0f40b04bda4d3bb36608da8521acb8%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637968676966261576%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=9eaJu5qbYwAEIvoafGEZya7Bn5diQYlBLzmv2BNS88M%3D&reserved=0> MatMFFDSetFunction MatSetFromOptions MatMFFDSetBase and provide the matrix to KSP. Note you will need to use -pc_type none or provide your own custom preconditioner withhttps://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FPC%2FPCSHELL%2F&amp;data=05%7C01%7CJiannan_Tu%40uml.edu%7Cab0859d8d154471590bc08da5f5918d6%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637927133525745809%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=Gb%2F9uFG3jp%2FbfWaSh2cSEQLItHS9CuLwarzN0KcNiJY%3D&amp;reserved=0<https://urldefense.com/v3/__https://petsc.org/main/docs/manualpages/PC/PCSHELL/__;!!PVKG_VDCxu5g!stxq2NWGVjUihvHBQC9Rlk0aHVoowcy0PTQYMcQxQ4DqOEC05KSw6TmstSXKckiUgelHzy4ue-d1cA4fKGw$><https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FPC%2FPCSHELL%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7Cce0f40b04bda4d3bb36608da8521acb8%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637968676966261576%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=bHW7pDGZ7qunbuUwGLdAi4kZLdCVnu0K3DCW%2B0nyRiM%3D&reserved=0>
>
>
>
> On May 24, 2022, at 1:21 PM, Tu, Jiannan <Jiannan_Tu at uml.edu<mailto:Jiannan_Tu at uml.edu><mailto:Jiannan_Tu at uml.edu>> wrote:
>
> I want to use a matrix-free matrix to solve a large linear equation system because the matrix is too large to be stored. Petsc user manual describes matrix-free method for SNES with examples. The matrix-free matrices section explains how to set up such a matrix, but I can't find any example of matrix-free method with KSP. I am wondering how to apply the method to KSP iterative solver in parallel.
>
> I greatly appreciate your help for me to understand this topic.
>
> Jiannan Tu

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220823/88d9e406/attachment-0001.html>

From bsmith at petsc.dev  Tue Aug 23 14:05:29 2022
From: bsmith at petsc.dev (Barry Smith)
Date: Tue, 23 Aug 2022 15:05:29 -0400
Subject: [petsc-users] Using matrix-free with KSP
In-Reply-To: <BY5PR02MB6471300031FBDA77EDABA2FCE7709@BY5PR02MB6471.namprd02.prod.outlook.com>
References: <BY5PR02MB6471969F80FA5F732FC74D87E7D79@BY5PR02MB6471.namprd02.prod.outlook.com>
	<B55CFC2C-6364-4B78-8E99-9D53700743B8@petsc.dev>
	<BY5PR02MB6471BDF90783FE3A0EA176CBE7809@BY5PR02MB6471.namprd02.prod.outlook.com>
	<87iloa49kt.fsf@jedbrown.org>
	<BY5PR02MB6471C0ED04769D02046F68ACE7809@BY5PR02MB6471.namprd02.prod.outlook.com>
	<00EBE4B8-22E5-408F-B56D-14226A278D7E@petsc.dev>
	<BY5PR02MB6471E386057DDDE98AD05DFAE7809@BY5PR02MB6471.namprd02.prod.outlook.com>
	<BY5PR02MB647140084C7E7CDDB6BC2A11E76B9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<A6629DE1-5A7F-414E-B164-A69877158675@petsc.dev>
	<BY5PR02MB6471AB1C004920DA1FC9B43EE76B9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<103146FD-FF28-47CE-B661-F2B521BF0E7A@petsc.dev>
	<DM6PR02MB6478B2FB8CB1BE757CE7B003E76D9@DM6PR02MB6478.namprd02.prod.outlook.com>
	<4C00F4B8-BB42-4932-BCD7-527E5FB39BFF@petsc.dev>
	<BY5PR02MB64710E70758E431108D66956E76D9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<D0EB13D8-C5A6-4AE7-AFDE-2CDCCACBAF73@petsc.dev>
	<BY5PR02MB6471D4B05874DECFDDA13FFCE76D9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<367B124C-88BB-4889-A4AF-45A1D473078D@petsc.dev>
	<BY5PR02MB6471F2C058B84DE092567731E76D9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<944DDEED-016D-43B2-A5A6-7BF2E8E41FA5@petsc.dev>
	<BY5PR02MB6471CD0236F03D20763A3B64E76C9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<B4CF5E5B-984D-4EC1-AB77-1FD08C3FBBCF@petsc.dev>
	<BY5PR02MB6471BA72871419781B875967E76C9@BY5PR02MB6471.namprd02.prod.outlook.com>
	<3D179AEA-AD05-4B46-8DDC-9E126741C247@petsc.dev>
	<BY5PR02MB6471560EA70C6FAE826DDF27E7719@BY5PR02MB6471.namprd02.prod.outlook.com>
	<1C67C8DB-0653-4588-8137-E0865E0F2098@petsc.dev>
	<BY5PR02MB6471C5EAE18AB9B114E76232E7719@BY5PR02MB6471.namprd02.prod.outlook.com>
	<FAA1668F-08C1-4BAD-89B4-784D8AD99C36@petsc.dev>
	<BY5PR02MB64715D7FE585A7F88092E04DE7719@BY5PR02MB6471.namprd02.prod.outlook.com>
	<F4C76705-B402-4F8B-A42C-DDBE95574065@petsc.dev>
	<BY5PR02MB64712BB4AF9E7CD2A66FE2F3E7709@BY5PR02MB6471.namprd02.prod.outlook.com>
	<D2A1B111-0AC4-441B-AA81-9011967F7641@petsc.dev>
	<BY5PR02MB6471300031FBDA77EDABA2FCE7709@BY5PR02MB6471.namprd02.prod.outlook.com>
Message-ID: <E10757A9-410E-4368-AFC3-F102F9995D56@petsc.dev>



> On Aug 23, 2022, at 12:42 PM, Tu, Jiannan <Jiannan_Tu at uml.edu> wrote:
> 
> I misunderstood the usage. Inserting params.A in the first place lets KSP use it as linear equation matrix and ignores mat-vec multiply function.

  To be clear, in normal usage the matmult provide by the first matrix defines the linear system to be solved, the second matrix defines the beast from which to construct the preconditioner (that matmult in the second matrix is not used except inside certain types of fancy preconditions). So the first has to be accurate, the second has to be efficient (for example fewer non-zeros in the operator).


> 
> Thank you very much,
> Jiannan
> From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>>
> Sent: Tuesday, August 23, 2022 12:08 PM
> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
> Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>>
> Subject: Re: [petsc-users] Using matrix-free with KSP
>  
> CAUTION: This email was sent from outside the UMass Lowell network.
> 
> 
>  KSPSetOperators(ksp, params.A, params.A); should be  KSPSetOperators(B, params.A);
> 
> 
> 
>> On Aug 23, 2022, at 12:00 PM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> wrote:
>> 
>> Hi Barry,
>> 
>> I celebrated too early. Just found that the matrix-vector multiply function put into MatShellSetOperation() is not called. I set a printf() in it no effect. Then I comment out all execution statements int it, KSP converges the same way and the solution is the same. Apparently, that mat-vec multiply function doesn't take effect.  
>> 
>> Following is complete sequency of KSP calls. Could you please help me identify what the problem is? 
>> 
>> Thank you very much.
>> 
>> Jiannan
>> 
>> ------------------------------------
>>     DMDACreate1d(MPI_COMM_WORLD, DM_BOUNDARY_GHOSTED, N, 1, 1, NULL, &da);
>>     DMSetFromOptions(da);
>>     DMSetUp(da);
>>     params.da = da;
>> 
>>     DMDAGetCorners(da, &xs, NULL, NULL, &xm, NULL, NULL);
>> 
>>     DMCreateGlobalVector(da, &X);
>>     VecDuplicate(X, &Rm);
>> 
>>     MatCreate(MPI_COMM_WORLD, &params.A);
>>     MatSetType(params.A, MATMPIAIJ);
>>     MatSetSizes(params.A, PETSC_DECIDE, PETSC_DECIDE, N, N);
>> 
>>     MatCreateShell(MPI_COMM_WORLD, xm, N, PETSC_DETERMINE, PETSC_DETERMINE, &params, &B);
>>     MatShellSetOperation(B, MATOP_MULT, (void(*)(void))matvecmult);
>> 
>>     Mat C;
>>     MatCreate(MPI_COMM_WORLD, &C);
>>     MatSetType(C, MATPREALLOCATOR);
>>     MatSetSizes(C, PETSC_DECIDE, PETSC_DECIDE, N, N);
>> 
>>     //calculate matrix elements from nearby interactions
>>     matrix_near_elements(&params, params.A, C);
>>     MatDestroy(&C);
>> 
>>     KSPCreate(MPI_COMM_WORLD, &ksp);
>>     KSPSetOperators(ksp, params.A, params.A);
>>     KSPSetFromOptions(ksp);
>>     
>>     PC pc;
>>     KSPGetPC(ksp, &pc);
>>     PCSetFromOptions(pc);
>> 
>>     //quantities needed to calculate interactions with far groups
>>     fmm_formular(&params);
>> 
>>     //compute right hand-side vector
>>     computeRhs(&params, Rm);
>> 
>>     //solve matrix equation
>>     KSPSolve(ksp, Rm, X);
>> 
>> From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>>
>> Sent: Monday, August 22, 2022 2:39 PM
>> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
>> Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>>
>> Subject: Re: [petsc-users] Using matrix-free with KSP
>>  
>> CAUTION: This email was sent from outside the UMass Lowell network.
>> 
>> 
>> 
>>> On Aug 22, 2022, at 1:37 PM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> wrote:
>>> 
>>> Barry,
>>> 
>>> Yes, you are right, Anear is the preconditioner that needs to be passed to KSPSetOperators() as the second matrix. 
>>> 
>>> I think I have figured out how to provide a preconditioner for matrix-free method. I was confused by matrix A created by MatCreateMFFD(), which is type MFFD. I thought I have to use it as the first matrix passed to KSPSetOperators(). Since it was not assembled (and can't) so that Petsc gave error message "matrix unassembled". In fact, I should ignore A, but use other two matrixes AA and B, which are type MPIAIJ, assemble them, pass them to KSPSetOperators() as first and second matrix. Since function passed into MatMFFDSetFunction() does not take a matrix as an argument, matrix AA needs to be declared in user context so that it can be used in that function to perform matrix-vector product (partially) using MatMult() as you suggested. 
>>> 
>>    I think your AA should be the shell matrix, and the B should be Anear. There should be no MatMFFD. Inside the AA shell matrix mat-mult you call MatMult( on the Anear put Anear in the Shell matrix context also) and then add the part of the multiply coming from the far field.
>> 
>>  
>>> I have used a small test program to solve an 8x8 linear equation system and the code produces the same answers with/without the preconditioner. The iteration for this small problem, of course no improvement with the preconditioner but the residual norms with/without the preconditioner are slightly different. I suppose the preconditioner takes effect.
>>> 
>>> Now I need to think about if the applied preconditioner does the right job on the other part of matrix-vector product (non near part) for the problem I am working on.
>>> 
>>> Thank you very very much and greatly appreciate your help!
>>> 
>>> Jiannan
>>> From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>>
>>> Sent: Monday, August 22, 2022 12:42 PM
>>> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
>>> Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>>
>>> Subject: Re: [petsc-users] Using matrix-free with KSP
>>>  
>>> CAUTION: This email was sent from outside the UMass Lowell network.
>>> 
>>> 
>>> 
>>>> On Aug 22, 2022, at 12:25 PM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> wrote:
>>>> 
>>>> The matrix-vector product involves part of matrix elements explicitly as shown below so that I need to insert values into this part of the matrix.
>>>> <image.png>
>>>> Initially I wrote my own code for calculating and storing matrix A^near_mi without using any Petsc matrix and then performed matrix-vector product using above equation. The code runs but converges very slow. Apparently a preconditioned is needed. 
>>>> 
>>>> I am not clear how Petsc can provide a preconditioner for my case.
>>> 
>>>   I am now confused. I thought you were forming Anear as a PETSc matrix and passing that to KSPSetOperators as the second matrix and so the preconditioner was being built with Anear?
>>> 
>>>   For your shell matrix vector product you should call MatMult(Anear,x,y) inside your custom routine and then add the rest of the matrix-vector product using your fast multipole products. What am I missing?
>>> 
>>>   Barry
>>> 
>>>> 
>>>> From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>>
>>>> Sent: Monday, August 22, 2022 8:42 AM
>>>> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
>>>> Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>>
>>>> Subject: Re: [petsc-users] Using matrix-free with KSP
>>>>  
>>>> CAUTION: This email was sent from outside the UMass Lowell network.
>>>> 
>>>> 
>>>> 
>>>>> On Aug 21, 2022, at 11:18 PM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> wrote:
>>>>> 
>>>>> Barry,
>>>>>  
>>>>> You are right. With PC type none, the same error appears.
>>>>>  
>>>>> Now I use MatShell and I literally have three matrixes. A is a shell matrix, B is a preconditioner, and C is a matrix just for preallocation purpose.
>>>>>  
>>>>> First C is used to determine non-zero structure. Then use MatPreallocatorPreallocation(C, PETSC_FALSE, B) to preallocate for B. After that do insertion and assembly for B. These steps work fine.
>>>>>  
>>>>> Now the question is how to insert and assemble shell matrix A. MatPreallocatorPreallocation doesn?t work with shell matrix as indicated by an error message. MatSetValues or Mat assembly doesn?t support shell matrix either. 
>>>> 
>>>>   A shell matrix is completely under your control. You decide on what data you want to store in it, and you provide the matrix-vector product routine.  MatPreallocatorPreallocation and MatSetValues don't work for shell matrix unless you provide code for them to work, which normally you would not do.
>>>> 
>>>>   I don't understand why you want to use MatSetValues for your shell matrix. If you can use MatSetValues() and just build an explicit matrix, then why would you use a shell matrix, just use MATAIJ matrix? 
>>>> 
>>>>   Or does part of the operation inside your MatShell require a "regular" PETSc matrix? This would be fine, but it is your responsibility to design your Mat shell context to contain the "regular matrix" and your responsibility to fill up that regular matrix inside your shell context, calling whatever code you choose to write. So for example your shell context would be a C struct and one of the fields would be a regular PETSc matrix, also part of your custom matrix-vector product routine would call a regular MatMult() on the regular PETSc matrix stored inside the C struct.
>>>> 
>>>>   Barry
>>>> 
>>>> 
>>>>>  
>>>>> Thank you,
>>>>> Jiannan
>>>>>  
>>>>> From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>> 
>>>>> Sent: Friday, August 19, 2022 4:02 PM
>>>>> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
>>>>> Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>
>>>>> Subject: Re: [petsc-users] Using matrix-free with KSP
>>>>>  
>>>>> CAUTION: This email was sent from outside the UMass Lowell network.
>>>>>  
>>>>>  
>>>>>   I don't know why changing the PCType could matter, can you please try with your new code but use the -pc_type none option to turn off the use of the preconditioner to see if the error still appears (I think it should). 
>>>>>  
>>>>>   Also if you have a custom function that can do the matrix-vector product (not using any differencing) then you should use MatShell() and provide the matrix-vector product operation with your code.
>>>>>  
>>>>>   Barry
>>>>>  
>>>>> 
>>>>> 
>>>>> On Aug 19, 2022, at 11:47 AM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> wrote:
>>>>>  
>>>>> Barry,
>>>>>  
>>>>> I have my own matrix-vector product but not use MatShell. I followed your previous instruction to use
>>>>>  
>>>>>     MatCreateMFFD(MPI_COMM_WORLD, PETSC_DECIDE, PETSC_DECIDE, N, N, &A);
>>>>>     MatMFFDSetFunction(A, formfunction, &params);
>>>>>     MatSetFromOptions(A);
>>>>>     MatMFFDSetBase(A, X, NULL);
>>>>>  
>>>>> The customized matrix-vector product is implemented in formfunction() provided to MatMFFDSetFunction(). In this way, the matrix A is not used. I think that is why Petsc complains the matrix is not assembled. However, when PC type is none, the code runs without error although KSP solver doesn?t converge.
>>>>>  
>>>>> The question then becomes how to link A to matrix-free matrix-vector multiplication. And where to put MatShell?
>>>>>  
>>>>> Thank you,
>>>>> Jiannan
>>>>>  
>>>>> From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>> 
>>>>> Sent: Friday, August 19, 2022 10:27 AM
>>>>> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
>>>>> Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>
>>>>> Subject: Re: [petsc-users] Using matrix-free with KSP
>>>>>  
>>>>> CAUTION: This email was sent from outside the UMass Lowell network.
>>>>>  
>>>>>  
>>>>> 
>>>>> 
>>>>> 
>>>>> On Aug 18, 2022, at 11:41 PM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> wrote:
>>>>>  
>>>>> The program calls KSPSolve just once and the error occurs after the first iteration.
>>>>>  
>>>>> I realized the matrix-vector product in my case is different from that approximating matrix-vector multiply by finite difference of function vector. It consists of two parts. One is multiplication of part of the matrix with part of vector, and another part results from so called fast multipole method. I?m not clear how to relate such calculation of matrix-vector product to the finite differencing of function vector.
>>>>>  
>>>>>   Are you using finite differencing at all in a matrix-vector product, or do you have your own custom matrix-vector product code (with MatShell)? 
>>>>>  
>>>>>   Anyways the easiest thing to do is 
>>>>>  
>>>>>   use the run time option -on_error_attach_debugger noxterm and run the code, 
>>>>>  
>>>>>   when it stops in the debugger you can type bt (for backtrace)
>>>>>  
>>>>>   then do up a few times to move up the stack until you get to the PETSc function call that generates the error
>>>>>  
>>>>>   you can then do "call MatView(thematrixvarible,0) to have PETSc display what the matrix is and
>>>>>  
>>>>>   you can also do print *thematrixvariable to see the exact struct with all its fields
>>>>>  
>>>>>   If this doesn't tell you why the troublesome matrix is not assembled then cut and paste ALL of the output and email it
>>>>>  
>>>>>   Barry
>>>>> 
>>>>> 
>>>>> 
>>>>>  
>>>>>  
>>>>>  
>>>>> From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>> 
>>>>> Sent: Thursday, August 18, 2022 7:44 PM
>>>>> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
>>>>> Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>
>>>>> Subject: Re: [petsc-users] Using matrix-free with KSP
>>>>>  
>>>>> CAUTION: This email was sent from outside the UMass Lowell network.
>>>>>  
>>>>>  
>>>>>  
>>>>>    Is it in the first iteration of the first KSPSolve that you get this message? Or in the second call to KSPSolve? 
>>>>>  
>>>>>   The error comes from a matrix vector product with your A matrix, what is that and where did it come from. The error message is not from your B matrix, that is not used in the MatMult()
>>>>>  
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> On Aug 18, 2022, at 3:05 PM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> wrote:
>>>>>  
>>>>> Barry, 
>>>>>  
>>>>> Thanks. The method works. Assembly of actual matrix now completes quickly. The convergency is much better for the first iteration. The residual normal is in the order of 10^-10 instead of 10^-2. However, there is an error message after the first iteration: ?Object is in wrong state. Not for unassembled matrix? even though both preallocator and actual matrix have been assembled (see the code snippet below). Could you please give me some hints what?s going wrong? I use KSP type bcgs and PC type mat
>>>>>  
>>>>> Thank you,
>>>>> Jiannan
>>>>>  
>>>>> -----------------------------------------------------
>>>>> Solve linear equations ...
>>>>>   0 KSP Residual norm 9.841179147519e-10
>>>>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>>>>> [0]PETSC ERROR: Object is in wrong state
>>>>> [0]PETSC ERROR: Not for unassembled matrix
>>>>> [0]PETSC ERROR: See https://petsc.org/release/faq/ <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Frelease%2Ffaq%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7Cce0f40b04bda4d3bb36608da8521acb8%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637968676966261576%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=%2FF%2FegTjfpbU0iS7vGRDAhBF7x%2FbgTMplabEpa7Ued4o%3D&reserved=0> for trouble shooting.
>>>>> [0]PETSC ERROR: Petsc Release Version 3.16.6, Mar 30, 2022
>>>>> [0]PETSC ERROR: ./iesolver on a  named REXNET-WS4 by jiannantu Thu Aug 18 14:48:43 2022
>>>>> [0]PETSC ERROR: Configure options --download-f2cblaslapack=yes --with-mpi-dir=/usr/local --with-fc=0 --prefix=/home/jiannantu/petsc --with-scalar-type=complex --with-64-bit-indices=1
>>>>> [0]PETSC ERROR: #1 MatMult() at /home/jiannantu/petsc-3.16.6/src/mat/interface/matrix.c:2425
>>>>> [0]PETSC ERROR: #2 PCApplyBAorAB() at /home/jiannantu/petsc-3.16.6/src/ksp/pc/interface/precon.c:730
>>>>> [0]PETSC ERROR: #3 KSP_PCApplyBAorAB() at /home/jiannantu/petsc-3.16.6/include/petsc/private/kspimpl.h:421
>>>>> [0]PETSC ERROR:  Linear equations not solved. Diverged reason: 0
>>>>> #4 KSPSolve_BCGS() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/impls/bcgs/bcgs.c:87
>>>>> [0]PETSC ERROR: #5 KSPSolve_Private() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itfunc.c:914
>>>>> [0]PETSC ERROR: #6 KSPSolve() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itfunc.c:1086
>>>>>  
>>>>> -----------------------------------------------
>>>>>     MatCreate(MPI_COMM_WORLD, &B);
>>>>>     MatSetType(B, MATMPIAIJ);
>>>>>     MatSetSizes(B, PETSC_DECIDE, PETSC_DECIDE, N, N);
>>>>>  
>>>>>     Mat preall;
>>>>>     MatCreate(MPI_COMM_WORLD, &preall);
>>>>>     MatSetType(preall, MATPREALLOCATOR);
>>>>>     MatSetSizes(preall, PETSC_DECIDE, PETSC_DECIDE, N, N);
>>>>>  
>>>>>     MatMPIAIJSetPreallocation(preall, 0, d_nnz, 0, o_nnz);
>>>>>     MatSetUp(preall);
>>>>> MatSetOption(preall, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE);
>>>>>  
>>>>> //set values row by row
>>>>> MatSetValues(preall, 1, &m, 1, nCols, matCols, INSERT_VALUES);
>>>>>  
>>>>> MatAssemblyBegin(preall, MAT_FINAL_ASSEMBLY);
>>>>> MatAssemblyEnd(preall, MAT_FINAL_ASSEMBLY);
>>>>>  
>>>>> //set values for actual matrix row by row
>>>>> MatSetValues(B, 1, &m, 1, nCols, matCols, INSERT_VALUES);
>>>>>  
>>>>> MatAssemblyBegin(B, MAT_FINAL_ASSEMBLY);
>>>>> MatAssemblyEnd(B, MAT_FINAL_ASSEMBLY);
>>>>>  
>>>>>     KSPCreate(MPI_COMM_WORLD, &ksp);
>>>>>     KSPSetOperators(ksp, A, B);
>>>>>     KSPSetFromOptions(ksp);
>>>>>  
>>>>>     PC pc;
>>>>>     KSPGetPC(ksp, &pc);
>>>>>     PCSetFromOptions(pc);
>>>>>  
>>>>> From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>> 
>>>>> Sent: Thursday, August 18, 2022 12:59 PM
>>>>> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
>>>>> Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>
>>>>> Subject: Re: [petsc-users] Using matrix-free with KSP
>>>>>  
>>>>> CAUTION: This email was sent from outside the UMass Lowell network.
>>>>>  
>>>>>  
>>>>>   Yes, once you have preallocated the real matrix you can destroy the preallocation matrix whose only job is to gather the preallocation information
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> On Aug 18, 2022, at 12:52 PM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> wrote:
>>>>>  
>>>>> Thanks, Barry.
>>>>>  
>>>>> So I need actually two matrixes, one for preallocator and one for actual matrix that can be passed to KSPSetOperators(). -mat_type preallocator option is used to speed up doing insertion into preallocator, then use MatPreallocatorPreallocate() to preallocate actual matrix and do actual insertion of values into it., right?
>>>>>  
>>>>> The code runs in parallel. Each process owns number of rows that equals to number of unknowns (that is, xm in 1D DM) it owns.
>>>>>  
>>>>> Jiannan
>>>>>  
>>>>> From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>> 
>>>>> Sent: Thursday, August 18, 2022 11:37 AM
>>>>> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
>>>>> Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>
>>>>> Subject: Re: [petsc-users] Using matrix-free with KSP
>>>>>  
>>>>> CAUTION: This email was sent from outside the UMass Lowell network.
>>>>>  
>>>>>  
>>>>>    The preallocator MatType matrix cannot be passed to the KSPSetOperators(), you need to create an actual matrix, for example MATAIJ and use the preallocator to set its preallocation and then fill up the MATAIJ matrix with the usual MatSetValues and pass that matrix to the KSPSetOperators.
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> On Aug 18, 2022, at 11:30 AM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> wrote:
>>>>>  
>>>>> Hi Barry,
>>>>>  
>>>>> The MATPREALLOCATOR solved problem of slow matrix assembly. But the solver failed because of ?Matrix type preallocator does not have a multiply defined?.
>>>>>  
>>>>> I guess this is because the matrix-free is used. I am wondering how a preconditioner is applied to the matrix-vector multiply in Petsc.
>>>>>  
>>>>> Thank you,
>>>>> Jiannan
>>>>>  
>>>>> [0]PETSC ERROR: No support for this operation for this object type
>>>>> [0]PETSC ERROR: Matrix type preallocator does not have a multiply defined
>>>>> [0]PETSC ERROR: See https://petsc.org/release/faq/ <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Frelease%2Ffaq%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7Cce0f40b04bda4d3bb36608da8521acb8%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637968676966261576%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=%2FF%2FegTjfpbU0iS7vGRDAhBF7x%2FbgTMplabEpa7Ued4o%3D&reserved=0> for trouble shooting.
>>>>> [0]PETSC ERROR: Petsc Release Version 3.16.6, Mar 30, 2022
>>>>> [0]PETSC ERROR: ./iesolver on a  named REXNET-WS4 by jiannantu Thu Aug 18 11:21:55 2022
>>>>> [0]PETSC ERROR: Configure options --download-f2cblaslapack=yes --with-mpi-dir=/usr/local --with-fc=0 --prefix=/home/jiannantu/petsc --with-scalar-type=complex --with-64-bit-indices=1
>>>>> [0]PETSC ERROR: #1 MatMult() at /home/jiannantu/petsc-3.16.6/src/mat/interface/matrix.c:2437
>>>>> [0]PETSC ERROR: #2 PCApply_Mat() at /home/jiannantu/petsc-3.16.6/src/ksp/pc/impls/mat/pcmat.c:9
>>>>> [0]PETSC ERROR: #3 PCApply() at /home/jiannantu/petsc-3.16.6/src/ksp/pc/interface/precon.c:445
>>>>> [0]PETSC ERROR: #4 KSP_PCApply() at /home/jiannantu/petsc-3.16.6/include/petsc/private/kspimpl.h:382
>>>>> [0]PETSC ERROR: #5 KSPInitialResidual() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itres.c:65
>>>>> [0]PETSC ERROR: #6 KSPSolve_BCGS() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/impls/bcgs/bcgs.c:43
>>>>> [0]PETSC ERROR: #7 KSPSolve_Private() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itfunc.c:914
>>>>> [0]PETSC ERROR: #8 KSPSolve() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itfunc.c:1086
>>>>>  
>>>>> From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>> 
>>>>> Sent: Thursday, August 18, 2022 8:35 AM
>>>>> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
>>>>> Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>
>>>>> Subject: Re: [petsc-users] Using matrix-free with KSP
>>>>>  
>>>>> CAUTION: This email was sent from outside the UMass Lowell network.
>>>>>  
>>>>>  
>>>>>   Slow assembly is usually due to under preallocation. You can change to using 
>>>>>  
>>>>> MatSetOption(B, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_TRUE);
>>>>>  
>>>>>   to detect if you are under preallocating. See https://petsc.org/main/docs/manual/mat/#sec-matsparse <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanual%2Fmat%2F%23sec-matsparse&data=05%7C01%7CJiannan_Tu%40uml.edu%7Cce0f40b04bda4d3bb36608da8521acb8%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637968676966261576%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=W4yc%2BKlgsE6%2FAs4CVIEk3sg9FikUlfVdZRB2PMQO4Mc%3D&reserved=0>. That section clearly needs some sprucing up. If it is difficult to determine good preallocation values you can use https://petsc.org/main/docs/manualpages/Mat/MATPREALLOCATOR/ <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMATPREALLOCATOR%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7Cce0f40b04bda4d3bb36608da8521acb8%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637968676966261576%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=pR8QnfL6k4hHgV%2BbuByyRdachoOAtAhoNhpzu3pQC%2FQ%3D&reserved=0> to compute the needed preallocation efficiently.
>>>>>  
>>>>>   Are you running in parallel? If so how are you determining which rows of entries to compute on each MPI rank? You will want most of the rows to be computed on the rank where the values are stored, you can determine this with https://petsc.org/main/docs/manualpages/Mat/MatGetOwnershipRange.html <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMatGetOwnershipRange.html&data=05%7C01%7CJiannan_Tu%40uml.edu%7Cce0f40b04bda4d3bb36608da8521acb8%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637968676966261576%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=izhtvCMmQIy59Eisqgz%2F8hUhk5NgVR6z%2B5%2Bc6yzE7G0%3D&reserved=0>
>>>>>  
>>>>>   Barry
>>>>>  
>>>>>  
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>>  
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> On Aug 18, 2022, at 12:50 AM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> wrote:
>>>>>  
>>>>> I implemented the preconditioner. The solver converges quickly when the problem size is small. But when the size increases, say to 100k unknowns, the code hangs at assembly of the preconditioner matrix. The function call sequence is like this
>>>>>  
>>>>>    MatCreate(MPI_COMM_WORLD, &B);
>>>>>     MatSetType(B, MATMPIAIJ);
>>>>>     MatSetSizes(B, PETSC_DECIDE, PETSC_DECIDE, N, N);
>>>>>     MatSetFromOptions(B);
>>>>>  
>>>>>     // The number of diagonal and off diagonal non zeros only can be determined run-time.
>>>>>     MatMPIAIJSetPreallocation(B, 0, d_nnz, 0, o_nnz);
>>>>>     MatSetOption(B, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE);
>>>>>  
>>>>>     //here insert values row by row.
>>>>> MatSetValues(B, 1, &m, nn, nCols, matCols, INSERT_VALUES);
>>>>>  
>>>>> MatAssemblyBegin(B, MAT_FINAL_ASSEMBLY);
>>>>> MatAssemblyEnd(B, MAT_FINAL_ASSEMBLY);
>>>>>  
>>>>> Could you please tell me what I have done wrong?
>>>>>  
>>>>> Thank you,
>>>>> Jiannan
>>>>>    
>>>>> From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>> 
>>>>> Sent: Tuesday, August 16, 2022 2:00 PM
>>>>> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
>>>>> Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>
>>>>> Subject: Re: [petsc-users] Using matrix-free with KSP
>>>>>  
>>>>> CAUTION: This email was sent from outside the UMass Lowell network.
>>>>>  
>>>>>  
>>>>>    Create another matrix B that contains "Selected parts of the matrix can be pre-calculated and stored to serve as the preconditioner." When you call KSPSetOperators() pass in B as the second matrix argument.
>>>>>  
>>>>>    Now if literally application of B is the preconditioner you want to use, then simply use PCMAT as the preconditioner, and it will apply B each time it needs to apply the preconditioner. With this, you do not even need a PCShell. 
>>>>>  
>>>>>    If your preconditioner uses B to construct your own specific data structure needed to apply your preconditioner, then you can use PCShellSetSetUp() and have your routine call PCGetOperators() to pull out the B matrix and use it to construct your preconditioner. 
>>>>>  
>>>>>    If you want PETSc to construct a standard preconditioner from B, then you don't need the PCShell; you simply pass in the B, as above, and use -pc_type type to set the preconditioner you want to use. 
>>>>>  
>>>>>    Barry
>>>>>  
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> On Aug 16, 2022, at 12:31 PM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> wrote:
>>>>>  
>>>>> Barry,
>>>>>  
>>>>> Thank you very much for your instructions. I am sorry I may not ask clearer questions.
>>>>>  
>>>>> I don't use SNES. What I am trying to solve is a very large linear equation system with dense and ill-conditioned matrix. Selected parts of the matrix can be pre-calculated and stored to serve as the preconditioner. My question is how I can apply this matrix as the preconditioner. 
>>>>>  
>>>>> Here is the part of code from my linear equations solver. MatrixFreePreconditioner is supposed to be the routine to provide the preconditioner. But I don't understand how to use it. Should I use SNESSetJacobian() even if this is a linear problem? 
>>>>>  
>>>>>     MatCreateMFFD(MPI_COMM_WORLD, PETSC_DECIDE, PETSC_DECIDE, N, N, &A);
>>>>>     MatMFFDSetFunction(A, formfunction, &params);
>>>>>     MatSetFromOptions(A);
>>>>>     MatMFFDSetBase(A, X, NULL);
>>>>>  
>>>>>     KSPCreate(MPI_COMM_WORLD, &ksp);
>>>>>     KSPSetOperators(ksp, A, A);
>>>>>     KSPSetFromOptions(ksp);
>>>>>  
>>>>>     PC pc;
>>>>>     KSPGetPC(ksp,&pc);
>>>>>     PCSetType(pc,PCSHELL);
>>>>>     PCShellSetApply(pc,MatrixFreePreconditioner);
>>>>>  
>>>>> Thank you,
>>>>> Jiannan
>>>>>  
>>>>> From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>>
>>>>> Sent: Tuesday, August 16, 2022 10:10 AM
>>>>> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
>>>>> Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>>
>>>>> Subject: Re: [petsc-users] Using matrix-free with KSP
>>>>>  
>>>>> CAUTION: This email was sent from outside the UMass Lowell network.
>>>>>  
>>>>>  
>>>>>   I don't fully understand your question so I will answer questions that I do know the answer to :-)
>>>>>  
>>>>>   The function you provide to PCShellSetApply() applies the preconditioner to the input vector and puts the result in the output vector.  This input vector changes at every application of the preconditioner so should not be used in constructing the preconditioner.
>>>>>  
>>>>>   If you are using the standard PETSc matrix-free matrix-vector product via finite difference, -snes_mf_operator or via MatCreateSNESMF() or you are using a MatShell, the matrix-vector product routines input changes for each multiply and should not be used in constructing the preconditioner.
>>>>>  
>>>>>   The function you provide with SNESSetJacobian() is where you can compute anything you need to help build your preconditioner. The input vector to that function is the location at which the Jacobian is needed and is what you should use to build your approximate/part of Jacobian. You can store your approximate/part of Jacobian that you need to compute in the pmat and then in a PCShellSetUp() function that you provide you will compute what you will need latter to apply the preconditioner in your PCShellSetApply() function. Note that the values you put in the pmat do not need to be the full Jacobian; they can be anything you want since this pmat is not used to apply the matrix vector product.
>>>>>  
>>>>>   If this is unclear, feel free to ask additional questions.
>>>>>  
>>>>>   Barry
>>>>>  
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> On Aug 16, 2022, at 9:40 AM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>> wrote:
>>>>>  
>>>>> I am trying to apply a user-defined preconditioner to speed up the convergency of matrix-free KSP solver. The user's manual provides an example of how to call the user-defined preconditioner routine. The routine has PC, input Vec and output Vec as arguments. I want to use part of the Jacobian as preconditioner matrix, the elements of which can be calculated using input Vec. My question is what the role of output Vec from the routine is in matrix-vector product? Or I just have the preconditioner matrix elements calculated in the routine and then Petsc handles applying the preconditioner to matrix-vector product? I used PCSetType(pc, PCSHELL) and PCShellSetApply.
>>>>>  
>>>>> Thank you and appreciate your help!
>>>>>  
>>>>> Jiannan 
>>>>> From: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
>>>>> Sent: Wednesday, July 6, 2022 2:34 PM
>>>>> To: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>>
>>>>> Cc: Jed Brown <jed at jedbrown.org <mailto:jed at jedbrown.org>>; petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>>
>>>>> Subject: Re: [petsc-users] Using matrix-free with KSP
>>>>>  
>>>>> Barry,
>>>>>  
>>>>> VecScatterCreateToAll solves the problem! The code now gives the correct answer with multiple processes. 
>>>>>  
>>>>> Thank you all so much!
>>>>>  
>>>>> Jiannan
>>>>> From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>>
>>>>> Sent: Wednesday, July 6, 2022 2:08 PM
>>>>> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
>>>>> Cc: Jed Brown <jed at jedbrown.org <mailto:jed at jedbrown.org>>; petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>>
>>>>> Subject: Re: [petsc-users] Using matrix-free with KSP
>>>>>  
>>>>>  
>>>>>   So your operator is a "dense" operator in that its matrix representation would be essentially dense. In that case, you can use specialized routines to do this efficiently. You can use VecScatterCreateToAll() and then VecScatterBegin/End to do the communication.
>>>>>  
>>>>>   Barry
>>>>>  
>>>>>  
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> On Jul 6, 2022, at 1:39 PM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>  wrote:
>>>>>  
>>>>> Jed, thank you very much for the reply. 
>>>>>  
>>>>> I'm not sure GlobaltoLocal will work. Say the solution vector is of length 10. Two processes then each process can see 5 elements of the vector. For A x = b (10 equations), each process performs matrix-vector product for five equations. And for each equation i , sum_j A_ij *  x_j = b_i requires the process has access to all 10 unknows x_i (i=0,1,...,9). 
>>>>>  
>>>>> Can VecScatter and PetscSF make the entire vector accessible to each process? I am reading the manual and trying to understand how VecScatter and PetscSF work.
>>>>>  
>>>>> Jiannan
>>>>> From: Jed Brown <jed at jedbrown.org <mailto:jed at jedbrown.org>>
>>>>> Sent: Wednesday, July 6, 2022 10:09 AM
>>>>> To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>; Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>>
>>>>> Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>>
>>>>> Subject: Re: [petsc-users] Using matrix-free with KSP
>>>>>  
>>>>> You'll usually have a GlobalToLocal operation for each rank to get the halo data it needs, then either a LocalToGlobal to collect the result (e.g., finite element methods) or the local compute will write directly into the owned portion of the global vector. If you're doing the communication "raw", then you may find VecScatter or PetscSF useful to perform the necessary communication.
>>>>> 
>>>>> "Tu, Jiannan" <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>  writes:
>>>>> 
>>>>> > Hi Barry,
>>>>> >
>>>>> > Following your instructions I implemented the matrix-free method to solve a large linear equation system resulted from a surface integration equation. The KSP solver works fine for single process, but it is problematic with multiple processes. The problem is that each process only can access its own part of solution vector so that each process only conducts part of matrix-vector multiplication. MPI can be used to assemble these partial matrix-vector multiplication together.  Does Petsc provide any ways to implement multi-process matrix-free method?
>>>>> >
>>>>> > Thanks,
>>>>> > Jiannan
>>>>> > ________________________________
>>>>> > From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>>
>>>>> > Sent: Tuesday, May 24, 2022 2:12 PM
>>>>> > To: Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>
>>>>> > Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>>
>>>>> > Subject: Re: [petsc-users] Using matrix-free with KSP
>>>>> >
>>>>> > This e-mail originated from outside the UMass Lowell network.
>>>>> > ________________________________
>>>>> >
>>>>> >    You can use MatCreateMFFD https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMatCreateMFFD%2F&amp;data=05%7C01%7CJiannan_Tu%40uml.edu%7Cab0859d8d154471590bc08da5f5918d6%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637927133525745809%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=A7wqegTfh94No5BpDiWLK3VxOuR44U2wlWHVm2k7l60%3D&amp;reserved=0<https://urldefense.com/v3/__https://petsc.org/main/docs/manualpages/Mat/MatCreateMFFD/__;!!PVKG_VDCxu5g!stxq2NWGVjUihvHBQC9Rlk0aHVoowcy0PTQYMcQxQ4DqOEC05KSw6TmstSXKckiUgelHzy4ue-d10-zlXkw$> <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMatCreateMFFD%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7Cce0f40b04bda4d3bb36608da8521acb8%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637968676966261576%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=9eaJu5qbYwAEIvoafGEZya7Bn5diQYlBLzmv2BNS88M%3D&reserved=0> MatMFFDSetFunction MatSetFromOptions MatMFFDSetBase and provide the matrix to KSP. Note you will need to use -pc_type none or provide your own custom preconditioner withhttps://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FPC%2FPCSHELL%2F&amp;data=05%7C01%7CJiannan_Tu%40uml.edu%7Cab0859d8d154471590bc08da5f5918d6%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637927133525745809%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=Gb%2F9uFG3jp%2FbfWaSh2cSEQLItHS9CuLwarzN0KcNiJY%3D&amp;reserved=0<https://urldefense.com/v3/__https://petsc.org/main/docs/manualpages/PC/PCSHELL/__;!!PVKG_VDCxu5g!stxq2NWGVjUihvHBQC9Rlk0aHVoowcy0PTQYMcQxQ4DqOEC05KSw6TmstSXKckiUgelHzy4ue-d1cA4fKGw$> <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FPC%2FPCSHELL%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7Cce0f40b04bda4d3bb36608da8521acb8%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637968676966261576%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=bHW7pDGZ7qunbuUwGLdAi4kZLdCVnu0K3DCW%2B0nyRiM%3D&reserved=0>
>>>>> >
>>>>> >
>>>>> >
>>>>> > On May 24, 2022, at 1:21 PM, Tu, Jiannan <Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu><mailto:Jiannan_Tu at uml.edu <mailto:Jiannan_Tu at uml.edu>>> wrote:
>>>>> >
>>>>> > I want to use a matrix-free matrix to solve a large linear equation system because the matrix is too large to be stored. Petsc user manual describes matrix-free method for SNES with examples. The matrix-free matrices section explains how to set up such a matrix, but I can't find any example of matrix-free method with KSP. I am wondering how to apply the method to KSP iterative solver in parallel.
>>>>> >
>>>>> > I greatly appreciate your help for me to understand this topic.
>>>>> >
>>>>> > Jiannan Tu

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220823/116c97c2/attachment-0001.html>

From snailsoar at hotmail.com  Wed Aug 24 10:03:07 2022
From: snailsoar at hotmail.com (feng wang)
Date: Wed, 24 Aug 2022 15:03:07 +0000
Subject: [petsc-users] Slepc Question, shell-matrix
In-Reply-To: <3E25E7D6-C81A-4A75-A215-A5FCFB7F3CB6@dsic.upv.es>
References: <ME3PR01MB7624DF748CD2568A0BAAC5B6B9719@ME3PR01MB7624.ausprd01.prod.outlook.com>
	<C1E1C97D-7F02-4764-9732-9330A6076438@dsic.upv.es>
	<ME3PR01MB7624ADA914B555F17A490226B9719@ME3PR01MB7624.ausprd01.prod.outlook.com>
	<8B78A886-8CB5-49A9-AFED-E67C22696400@dsic.upv.es>
	<SYZPR01MB76275849E71CDD1083D79937B9709@SYZPR01MB7627.ausprd01.prod.outlook.com>
	<17F56A61-2DB1-416B-85CE-8E585051F1EB@dsic.upv.es>
	<ME3PR01MB762414A8D756C6C68BC95E92B9709@ME3PR01MB7624.ausprd01.prod.outlook.com>
	<3E25E7D6-C81A-4A75-A215-A5FCFB7F3CB6@dsic.upv.es>
Message-ID: <ME3PR01MB762495A98C93E2D6EA7E30ABB9739@ME3PR01MB7624.ausprd01.prod.outlook.com>

Hi Jose,

Thanks for your reply.

I have tried -eps_view_mat0 binary:amatrix.bin to save my shell matrix. It seems not saving the shell matrix and just created an empty file.

Besides, I am trying to understand the output of slepc.  I can set  -eps_tol to a low value, but the final relative residual norm is still high , what does this tell me?

Best regards,
Feng

________________________________
From: Jose E. Roman <jroman at dsic.upv.es>
Sent: 23 August 2022 11:06
To: feng wang <snailsoar at hotmail.com>
Cc: petsc-users <petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] Slepc Question, shell-matrix

You can try the following. Save your shell matrix to disk with the option
-eps_view_mat0 binary:amatrix.bin
then repeat the computation with ex4.c loading this file.

Note that this should be done for a small problem size, because the conversion from a shell matrix implies a matrix-vector product per each column.

Jose

> El 23 ago 2022, a las 12:57, feng wang <snailsoar at hotmail.com> escribi?:
>
> Hi Jose,
>
> Thanks for your reply.
>
> It represents a linear operator. In my shell matrix, I am computing the non-linear residuals twice with perturbed flow variables. The matrix-vector product is computed as:
>
> A*v = (R(q+eps*v) - R(q-eps*v))/(2*eps)
>
> R is the non-linear residual. q is my flow variable and it does not change. eps is the perturbation. A is my Jacobian matrix. Besides, for some background, I am computing a steady RANS flow with finite volume method and trying to do a global stability analysis by looking at the Jacobian matrix.
>
> Thanks,
> Feng
>
> From: Jose E. Roman <jroman at dsic.upv.es>
> Sent: 23 August 2022 10:32
> To: feng wang <snailsoar at hotmail.com>
> Cc: petsc-users <petsc-users at mcs.anl.gov>
> Subject: Re: [petsc-users] Slepc Question, shell-matrix
>
> The relative residual norms that are printed at the end are too large. For NHEP problems, they should be below the tolerance. Don't know what is happening. Does your shell matrix represent a linear (constant) operator? Or does it change slightly depending on the input vector?
>
> > El 23 ago 2022, a las 12:14, feng wang <snailsoar at hotmail.com> escribi?:
> >
> > Hi Jose,
> >
> > I think the previous problem comes from my side. I have some uninitialized values in my part of code to compute the non-linear residuals. so, it produces a NAN when it tries to compute the matrix-vector product using finite difference.  This might make the slepc/pestc do unexpected things.
> >
> > Now It seems I've got slepc running. eps_nev is set to 3 and I am trying to compute the ones with the largest amplitudes. Below is the slepc output.
> >
> >  14 EPS converged value (error) #0 -0.000164282 (5.36813206e-09)
> >  16 EPS converged value (error) #1 -0.000160691+2.17113e-05i (3.37429620e-09)
> >  16 EPS converged value (error) #2 -0.000160691-2.17113e-05i (3.37429620e-09)
> >  Solution method: krylovschur
> >
> >  Number of requested eigenvalues: 2
> >  Linear eigensolve converged (3 eigenpairs) due to CONVERGED_TOL; iterations 16
> >  ---------------------- --------------------
> >             k             ||Ax-kx||/||kx||
> >  ---------------------- --------------------
> >        -0.000164              0.0613788
> >   -0.000161+0.000022i         0.0773339
> >   -0.000161-0.000022i         0.0774536
> >  ---------------------- --------------------
> >
> > The values in the brackets are the absolute error (I believe) and they seem very low. The relative error seems quite large.  Could you please comment on this?
> >
> >
> > Best regards,
> > Feng
> >
> > From: Jose E. Roman <jroman at dsic.upv.es>
> > Sent: 23 August 2022 5:24
> > To: feng wang <snailsoar at hotmail.com>
> > Cc: petsc-users <petsc-users at mcs.anl.gov>
> > Subject: Re: [petsc-users] Slepc Question, shell-matrix
> >
> > Please always respond to the list, otherwise the thread appears as unresolved in the archives of the mailing list.
> >
> >
> > > El 22 ago 2022, a las 22:45, feng wang <snailsoar at hotmail.com> escribi?:
> > >
> > > Hi Jose,
> > >
> > > I think I might have solved my problem. I have some uninitialized values in my part of code to compute the right hand side. so it produces a NAN when it tries to compute the matrix-vector product.
> > >
> > > Many thanks for your help!
> > >
> > > Best regards,
> > > Feng
> > > From: Jose E. Roman <jroman at dsic.upv.es>
> > > Sent: 22 August 2022 19:32
> > > To: feng wang <snailsoar at hotmail.com>
> > > Cc: petsc-users at mcs.anl.gov <petsc-users at mcs.anl.gov>
> > > Subject: Re: [petsc-users] Slepc Question, shell-matrix
> > >
> > > This is very strange. This error appears when the solver employs the B-inner product, but in your case you don't have a B matrix, so you should never see that error. Try running under valgrind to see if it gives more hints.
> > >
> > > Jose
> > >
> > >
> > > > El 22 ago 2022, a las 20:45, feng wang <snailsoar at hotmail.com> escribi?:
> > > >
> > > > Hello,
> > > >
> > > > I am new to Slepc and trying to work out the eigenvalues and eigenvectors of my Jacobian matrix. I am using a shell matrix to work out the matrix-vector product and I am using the default Krylov-schur method.
> > > >
> > > > My first attempt was not successful and I got the following errors:
> > > >
> > > > [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> > > > [0]PETSC ERROR: Missing or incorrect user input
> > > > [0]PETSC ERROR: The inner product is not well defined: indefinite matrix
> > > > [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
> > > > [0]PETSC ERROR: Petsc Release Version 3.17.4, unknown
> > > > [0]PETSC ERROR: cfdtest on a arch-debug named ming by feng Mon Aug 22 19:21:41 2022
> > > > [0]PETSC ERROR: Configure options --with-cc=mpicc --with-cxx=mpicxx --with-fc=0 PETSC_ARCH=arch-debug
> > > > [0]PETSC ERROR: #1 BV_SafeSqrt() at /home/feng/cfd/slepc-3.17.1/include/slepc/private/bvimpl.h:130
> > > > [0]PETSC ERROR: #2 BV_SquareRoot_Default() at /home/feng/cfd/slepc-3.17.1/include/slepc/private/bvimpl.h:365
> > > > [0]PETSC ERROR: #3 BVOrthogonalizeCGS1() at /home/feng/cfd/slepc-3.17.1/src/sys/classes/bv/interface/bvorthog.c:101
> > > > [0]PETSC ERROR: #4 BVOrthogonalizeGS() at /home/feng/cfd/slepc-3.17.1/src/sys/classes/bv/interface/bvorthog.c:177
> > > > [0]PETSC ERROR: #5 BVOrthonormalizeColumn() at /home/feng/cfd/slepc-3.17.1/src/sys/classes/bv/interface/bvorthog.c:402
> > > > [0]PETSC ERROR: #6 BVMatArnoldi() at /home/feng/cfd/slepc-3.17.1/src/sys/classes/bv/interface/bvkrylov.c:91
> > > > [0]PETSC ERROR: #7 EPSSolve_KrylovSchur_Default() at /home/feng/cfd/slepc-3.17.1/src/eps/impls/krylov/krylovschur/krylovschur.c:261
> > > > [0]PETSC ERROR: #8 EPSSolve() at /home/feng/cfd/slepc-3.17.1/src/eps/interface/epssolve.c:147
> > > > [0]PETSC ERROR: #9 slepc_eigen_comp() at domain/cfd/slepc_eigen_solve.cpp:77
> > > >
> > > > Could someone please shine some light on this? I have also attached my code. The code is part of my big library and I cannot attach the whole code, sorry about this. but I am happy to provide more information. The attached code has some arrangements for halo exchange, but for the moment, it assumes it is a serial run.
> > > >
> > > > Many thanks,
> > > > Feng
> > > > <slepc_eigen_solve.cpp>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220824/47313171/attachment.html>

From jroman at dsic.upv.es  Wed Aug 24 11:23:44 2022
From: jroman at dsic.upv.es (Jose E. Roman)
Date: Wed, 24 Aug 2022 18:23:44 +0200
Subject: [petsc-users] Slepc Question, shell-matrix
In-Reply-To: <ME3PR01MB762495A98C93E2D6EA7E30ABB9739@ME3PR01MB7624.ausprd01.prod.outlook.com>
References: <ME3PR01MB7624DF748CD2568A0BAAC5B6B9719@ME3PR01MB7624.ausprd01.prod.outlook.com>
	<C1E1C97D-7F02-4764-9732-9330A6076438@dsic.upv.es>
	<ME3PR01MB7624ADA914B555F17A490226B9719@ME3PR01MB7624.ausprd01.prod.outlook.com>
	<8B78A886-8CB5-49A9-AFED-E67C22696400@dsic.upv.es>
	<SYZPR01MB76275849E71CDD1083D79937B9709@SYZPR01MB7627.ausprd01.prod.outlook.com>
	<17F56A61-2DB1-416B-85CE-8E585051F1EB@dsic.upv.es>
	<ME3PR01MB762414A8D756C6C68BC95E92B9709@ME3PR01MB7624.ausprd01.prod.outlook.com>
	<3E25E7D6-C81A-4A75-A215-A5FCFB7F3CB6@dsic.upv.es>
	<ME3PR01MB762495A98C93E2D6EA7E30ABB9739@ME3PR01MB7624.ausprd01.prod.outlook.com>
Message-ID: <6BC033FF-0A55-4564-923D-97EFDD7C7FF7@dsic.upv.es>



> El 24 ago 2022, a las 17:03, feng wang <snailsoar at hotmail.com> escribi?:
> 
> Hi Jose,
> 
> Thanks for your reply.
> 
> I have tried -eps_view_mat0 binary:amatrix.bin to save my shell matrix. It seems not saving the shell matrix and just created an empty file.

Oops, I thought the shell matrix would be converted automatically. Try converting it yourself:

PetscCall(MatConvert(slepc_A_mf,MATDENSE,MAT_INITIAL_MATRIX,&Adense));

Then use Adense in EPSSetOperators(). Or save it to disk and send it to me so that I can give it a try.

> 
> Besides, I am trying to understand the output of slepc.  I can set  -eps_tol to a low value, but the final relative residual norm is still high , what does this tell me?

I don't know. It should not happen.
Jose

> 
> Best regards,
> Feng
> 
> From: Jose E. Roman <jroman at dsic.upv.es>
> Sent: 23 August 2022 11:06
> To: feng wang <snailsoar at hotmail.com>
> Cc: petsc-users <petsc-users at mcs.anl.gov>
> Subject: Re: [petsc-users] Slepc Question, shell-matrix
>  
> You can try the following. Save your shell matrix to disk with the option
> -eps_view_mat0 binary:amatrix.bin
> then repeat the computation with ex4.c loading this file.
> 
> Note that this should be done for a small problem size, because the conversion from a shell matrix implies a matrix-vector product per each column.
> 
> Jose
> 
> > El 23 ago 2022, a las 12:57, feng wang <snailsoar at hotmail.com> escribi?:
> > 
> > Hi Jose,
> > 
> > Thanks for your reply.
> > 
> > It represents a linear operator. In my shell matrix, I am computing the non-linear residuals twice with perturbed flow variables. The matrix-vector product is computed as:
> > 
> > A*v = (R(q+eps*v) - R(q-eps*v))/(2*eps)
> > 
> > R is the non-linear residual. q is my flow variable and it does not change. eps is the perturbation. A is my Jacobian matrix. Besides, for some background, I am computing a steady RANS flow with finite volume method and trying to do a global stability analysis by looking at the Jacobian matrix. 
> > 
> > Thanks,
> > Feng
> > 
> > From: Jose E. Roman <jroman at dsic.upv.es>
> > Sent: 23 August 2022 10:32
> > To: feng wang <snailsoar at hotmail.com>
> > Cc: petsc-users <petsc-users at mcs.anl.gov>
> > Subject: Re: [petsc-users] Slepc Question, shell-matrix
> >  
> > The relative residual norms that are printed at the end are too large. For NHEP problems, they should be below the tolerance. Don't know what is happening. Does your shell matrix represent a linear (constant) operator? Or does it change slightly depending on the input vector? 
> > 
> > > El 23 ago 2022, a las 12:14, feng wang <snailsoar at hotmail.com> escribi?:
> > > 
> > > Hi Jose,
> > > 
> > > I think the previous problem comes from my side. I have some uninitialized values in my part of code to compute the non-linear residuals. so, it produces a NAN when it tries to compute the matrix-vector product using finite difference.  This might make the slepc/pestc do unexpected things.
> > > 
> > > Now It seems I've got slepc running. eps_nev is set to 3 and I am trying to compute the ones with the largest amplitudes. Below is the slepc output. 
> > > 
> > >  14 EPS converged value (error) #0 -0.000164282 (5.36813206e-09)
> > >  16 EPS converged value (error) #1 -0.000160691+2.17113e-05i (3.37429620e-09)
> > >  16 EPS converged value (error) #2 -0.000160691-2.17113e-05i (3.37429620e-09)
> > >  Solution method: krylovschur
> > > 
> > >  Number of requested eigenvalues: 2
> > >  Linear eigensolve converged (3 eigenpairs) due to CONVERGED_TOL; iterations 16
> > >  ---------------------- --------------------
> > >             k             ||Ax-kx||/||kx||
> > >  ---------------------- --------------------
> > >        -0.000164              0.0613788
> > >   -0.000161+0.000022i         0.0773339
> > >   -0.000161-0.000022i         0.0774536
> > >  ---------------------- --------------------
> > > 
> > > The values in the brackets are the absolute error (I believe) and they seem very low. The relative error seems quite large.  Could you please comment on this? 
> > > 
> > > 
> > > Best regards,
> > > Feng
> > > 
> > > From: Jose E. Roman <jroman at dsic.upv.es>
> > > Sent: 23 August 2022 5:24
> > > To: feng wang <snailsoar at hotmail.com>
> > > Cc: petsc-users <petsc-users at mcs.anl.gov>
> > > Subject: Re: [petsc-users] Slepc Question, shell-matrix
> > >  
> > > Please always respond to the list, otherwise the thread appears as unresolved in the archives of the mailing list.
> > > 
> > > 
> > > > El 22 ago 2022, a las 22:45, feng wang <snailsoar at hotmail.com> escribi?:
> > > > 
> > > > Hi Jose,
> > > > 
> > > > I think I might have solved my problem. I have some uninitialized values in my part of code to compute the right hand side. so it produces a NAN when it tries to compute the matrix-vector product. 
> > > > 
> > > > Many thanks for your help!
> > > > 
> > > > Best regards,
> > > > Feng
> > > > From: Jose E. Roman <jroman at dsic.upv.es>
> > > > Sent: 22 August 2022 19:32
> > > > To: feng wang <snailsoar at hotmail.com>
> > > > Cc: petsc-users at mcs.anl.gov <petsc-users at mcs.anl.gov>
> > > > Subject: Re: [petsc-users] Slepc Question, shell-matrix
> > > >  
> > > > This is very strange. This error appears when the solver employs the B-inner product, but in your case you don't have a B matrix, so you should never see that error. Try running under valgrind to see if it gives more hints.
> > > > 
> > > > Jose
> > > > 
> > > > 
> > > > > El 22 ago 2022, a las 20:45, feng wang <snailsoar at hotmail.com> escribi?:
> > > > > 
> > > > > Hello,
> > > > > 
> > > > > I am new to Slepc and trying to work out the eigenvalues and eigenvectors of my Jacobian matrix. I am using a shell matrix to work out the matrix-vector product and I am using the default Krylov-schur method.  
> > > > > 
> > > > > My first attempt was not successful and I got the following errors:
> > > > > 
> > > > > [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> > > > > [0]PETSC ERROR: Missing or incorrect user input 
> > > > > [0]PETSC ERROR: The inner product is not well defined: indefinite matrix
> > > > > [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
> > > > > [0]PETSC ERROR: Petsc Release Version 3.17.4, unknown 
> > > > > [0]PETSC ERROR: cfdtest on a arch-debug named ming by feng Mon Aug 22 19:21:41 2022
> > > > > [0]PETSC ERROR: Configure options --with-cc=mpicc --with-cxx=mpicxx --with-fc=0 PETSC_ARCH=arch-debug
> > > > > [0]PETSC ERROR: #1 BV_SafeSqrt() at /home/feng/cfd/slepc-3.17.1/include/slepc/private/bvimpl.h:130
> > > > > [0]PETSC ERROR: #2 BV_SquareRoot_Default() at /home/feng/cfd/slepc-3.17.1/include/slepc/private/bvimpl.h:365
> > > > > [0]PETSC ERROR: #3 BVOrthogonalizeCGS1() at /home/feng/cfd/slepc-3.17.1/src/sys/classes/bv/interface/bvorthog.c:101
> > > > > [0]PETSC ERROR: #4 BVOrthogonalizeGS() at /home/feng/cfd/slepc-3.17.1/src/sys/classes/bv/interface/bvorthog.c:177
> > > > > [0]PETSC ERROR: #5 BVOrthonormalizeColumn() at /home/feng/cfd/slepc-3.17.1/src/sys/classes/bv/interface/bvorthog.c:402
> > > > > [0]PETSC ERROR: #6 BVMatArnoldi() at /home/feng/cfd/slepc-3.17.1/src/sys/classes/bv/interface/bvkrylov.c:91
> > > > > [0]PETSC ERROR: #7 EPSSolve_KrylovSchur_Default() at /home/feng/cfd/slepc-3.17.1/src/eps/impls/krylov/krylovschur/krylovschur.c:261
> > > > > [0]PETSC ERROR: #8 EPSSolve() at /home/feng/cfd/slepc-3.17.1/src/eps/interface/epssolve.c:147
> > > > > [0]PETSC ERROR: #9 slepc_eigen_comp() at domain/cfd/slepc_eigen_solve.cpp:77
> > > > > 
> > > > > Could someone please shine some light on this? I have also attached my code. The code is part of my big library and I cannot attach the whole code, sorry about this. but I am happy to provide more information. The attached code has some arrangements for halo exchange, but for the moment, it assumes it is a serial run.
> > > > > 
> > > > > Many thanks,
> > > > > Feng
> > > > > <slepc_eigen_solve.cpp>


From snailsoar at hotmail.com  Wed Aug 24 12:35:22 2022
From: snailsoar at hotmail.com (feng wang)
Date: Wed, 24 Aug 2022 17:35:22 +0000
Subject: [petsc-users] Slepc Question, shell-matrix
In-Reply-To: <6BC033FF-0A55-4564-923D-97EFDD7C7FF7@dsic.upv.es>
References: <ME3PR01MB7624DF748CD2568A0BAAC5B6B9719@ME3PR01MB7624.ausprd01.prod.outlook.com>
	<C1E1C97D-7F02-4764-9732-9330A6076438@dsic.upv.es>
	<ME3PR01MB7624ADA914B555F17A490226B9719@ME3PR01MB7624.ausprd01.prod.outlook.com>
	<8B78A886-8CB5-49A9-AFED-E67C22696400@dsic.upv.es>
	<SYZPR01MB76275849E71CDD1083D79937B9709@SYZPR01MB7627.ausprd01.prod.outlook.com>
	<17F56A61-2DB1-416B-85CE-8E585051F1EB@dsic.upv.es>
	<ME3PR01MB762414A8D756C6C68BC95E92B9709@ME3PR01MB7624.ausprd01.prod.outlook.com>
	<3E25E7D6-C81A-4A75-A215-A5FCFB7F3CB6@dsic.upv.es>
	<ME3PR01MB762495A98C93E2D6EA7E30ABB9739@ME3PR01MB7624.ausprd01.prod.outlook.com>
	<6BC033FF-0A55-4564-923D-97EFDD7C7FF7@dsic.upv.es>
Message-ID: <ME3PR01MB762467CDD5598EC8E2AC5A50B9739@ME3PR01MB7624.ausprd01.prod.outlook.com>

Hi Jose,

Thanks for your reply.

I have tried PetscCall(MatConvert(slepc_A_mf,MATDENSE,MAT_INITIAL_MATRIX,&Adense));  and If I do MatView(Adense,viewer), All I see was NAN. Have I missed anything here?

Thanks,
Feng

________________________________
From: Jose E. Roman <jroman at dsic.upv.es>
Sent: 24 August 2022 16:23
To: feng wang <snailsoar at hotmail.com>
Cc: petsc-users <petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] Slepc Question, shell-matrix



> El 24 ago 2022, a las 17:03, feng wang <snailsoar at hotmail.com> escribi?:
>
> Hi Jose,
>
> Thanks for your reply.
>
> I have tried -eps_view_mat0 binary:amatrix.bin to save my shell matrix. It seems not saving the shell matrix and just created an empty file.

Oops, I thought the shell matrix would be converted automatically. Try converting it yourself:

PetscCall(MatConvert(slepc_A_mf,MATDENSE,MAT_INITIAL_MATRIX,&Adense));

Then use Adense in EPSSetOperators(). Or save it to disk and send it to me so that I can give it a try.

>
> Besides, I am trying to understand the output of slepc.  I can set  -eps_tol to a low value, but the final relative residual norm is still high , what does this tell me?

I don't know. It should not happen.
Jose

>
> Best regards,
> Feng
>
> From: Jose E. Roman <jroman at dsic.upv.es>
> Sent: 23 August 2022 11:06
> To: feng wang <snailsoar at hotmail.com>
> Cc: petsc-users <petsc-users at mcs.anl.gov>
> Subject: Re: [petsc-users] Slepc Question, shell-matrix
>
> You can try the following. Save your shell matrix to disk with the option
> -eps_view_mat0 binary:amatrix.bin
> then repeat the computation with ex4.c loading this file.
>
> Note that this should be done for a small problem size, because the conversion from a shell matrix implies a matrix-vector product per each column.
>
> Jose
>
> > El 23 ago 2022, a las 12:57, feng wang <snailsoar at hotmail.com> escribi?:
> >
> > Hi Jose,
> >
> > Thanks for your reply.
> >
> > It represents a linear operator. In my shell matrix, I am computing the non-linear residuals twice with perturbed flow variables. The matrix-vector product is computed as:
> >
> > A*v = (R(q+eps*v) - R(q-eps*v))/(2*eps)
> >
> > R is the non-linear residual. q is my flow variable and it does not change. eps is the perturbation. A is my Jacobian matrix. Besides, for some background, I am computing a steady RANS flow with finite volume method and trying to do a global stability analysis by looking at the Jacobian matrix.
> >
> > Thanks,
> > Feng
> >
> > From: Jose E. Roman <jroman at dsic.upv.es>
> > Sent: 23 August 2022 10:32
> > To: feng wang <snailsoar at hotmail.com>
> > Cc: petsc-users <petsc-users at mcs.anl.gov>
> > Subject: Re: [petsc-users] Slepc Question, shell-matrix
> >
> > The relative residual norms that are printed at the end are too large. For NHEP problems, they should be below the tolerance. Don't know what is happening. Does your shell matrix represent a linear (constant) operator? Or does it change slightly depending on the input vector?
> >
> > > El 23 ago 2022, a las 12:14, feng wang <snailsoar at hotmail.com> escribi?:
> > >
> > > Hi Jose,
> > >
> > > I think the previous problem comes from my side. I have some uninitialized values in my part of code to compute the non-linear residuals. so, it produces a NAN when it tries to compute the matrix-vector product using finite difference.  This might make the slepc/pestc do unexpected things.
> > >
> > > Now It seems I've got slepc running. eps_nev is set to 3 and I am trying to compute the ones with the largest amplitudes. Below is the slepc output.
> > >
> > >  14 EPS converged value (error) #0 -0.000164282 (5.36813206e-09)
> > >  16 EPS converged value (error) #1 -0.000160691+2.17113e-05i (3.37429620e-09)
> > >  16 EPS converged value (error) #2 -0.000160691-2.17113e-05i (3.37429620e-09)
> > >  Solution method: krylovschur
> > >
> > >  Number of requested eigenvalues: 2
> > >  Linear eigensolve converged (3 eigenpairs) due to CONVERGED_TOL; iterations 16
> > >  ---------------------- --------------------
> > >             k             ||Ax-kx||/||kx||
> > >  ---------------------- --------------------
> > >        -0.000164              0.0613788
> > >   -0.000161+0.000022i         0.0773339
> > >   -0.000161-0.000022i         0.0774536
> > >  ---------------------- --------------------
> > >
> > > The values in the brackets are the absolute error (I believe) and they seem very low. The relative error seems quite large.  Could you please comment on this?
> > >
> > >
> > > Best regards,
> > > Feng
> > >
> > > From: Jose E. Roman <jroman at dsic.upv.es>
> > > Sent: 23 August 2022 5:24
> > > To: feng wang <snailsoar at hotmail.com>
> > > Cc: petsc-users <petsc-users at mcs.anl.gov>
> > > Subject: Re: [petsc-users] Slepc Question, shell-matrix
> > >
> > > Please always respond to the list, otherwise the thread appears as unresolved in the archives of the mailing list.
> > >
> > >
> > > > El 22 ago 2022, a las 22:45, feng wang <snailsoar at hotmail.com> escribi?:
> > > >
> > > > Hi Jose,
> > > >
> > > > I think I might have solved my problem. I have some uninitialized values in my part of code to compute the right hand side. so it produces a NAN when it tries to compute the matrix-vector product.
> > > >
> > > > Many thanks for your help!
> > > >
> > > > Best regards,
> > > > Feng
> > > > From: Jose E. Roman <jroman at dsic.upv.es>
> > > > Sent: 22 August 2022 19:32
> > > > To: feng wang <snailsoar at hotmail.com>
> > > > Cc: petsc-users at mcs.anl.gov <petsc-users at mcs.anl.gov>
> > > > Subject: Re: [petsc-users] Slepc Question, shell-matrix
> > > >
> > > > This is very strange. This error appears when the solver employs the B-inner product, but in your case you don't have a B matrix, so you should never see that error. Try running under valgrind to see if it gives more hints.
> > > >
> > > > Jose
> > > >
> > > >
> > > > > El 22 ago 2022, a las 20:45, feng wang <snailsoar at hotmail.com> escribi?:
> > > > >
> > > > > Hello,
> > > > >
> > > > > I am new to Slepc and trying to work out the eigenvalues and eigenvectors of my Jacobian matrix. I am using a shell matrix to work out the matrix-vector product and I am using the default Krylov-schur method.
> > > > >
> > > > > My first attempt was not successful and I got the following errors:
> > > > >
> > > > > [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> > > > > [0]PETSC ERROR: Missing or incorrect user input
> > > > > [0]PETSC ERROR: The inner product is not well defined: indefinite matrix
> > > > > [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
> > > > > [0]PETSC ERROR: Petsc Release Version 3.17.4, unknown
> > > > > [0]PETSC ERROR: cfdtest on a arch-debug named ming by feng Mon Aug 22 19:21:41 2022
> > > > > [0]PETSC ERROR: Configure options --with-cc=mpicc --with-cxx=mpicxx --with-fc=0 PETSC_ARCH=arch-debug
> > > > > [0]PETSC ERROR: #1 BV_SafeSqrt() at /home/feng/cfd/slepc-3.17.1/include/slepc/private/bvimpl.h:130
> > > > > [0]PETSC ERROR: #2 BV_SquareRoot_Default() at /home/feng/cfd/slepc-3.17.1/include/slepc/private/bvimpl.h:365
> > > > > [0]PETSC ERROR: #3 BVOrthogonalizeCGS1() at /home/feng/cfd/slepc-3.17.1/src/sys/classes/bv/interface/bvorthog.c:101
> > > > > [0]PETSC ERROR: #4 BVOrthogonalizeGS() at /home/feng/cfd/slepc-3.17.1/src/sys/classes/bv/interface/bvorthog.c:177
> > > > > [0]PETSC ERROR: #5 BVOrthonormalizeColumn() at /home/feng/cfd/slepc-3.17.1/src/sys/classes/bv/interface/bvorthog.c:402
> > > > > [0]PETSC ERROR: #6 BVMatArnoldi() at /home/feng/cfd/slepc-3.17.1/src/sys/classes/bv/interface/bvkrylov.c:91
> > > > > [0]PETSC ERROR: #7 EPSSolve_KrylovSchur_Default() at /home/feng/cfd/slepc-3.17.1/src/eps/impls/krylov/krylovschur/krylovschur.c:261
> > > > > [0]PETSC ERROR: #8 EPSSolve() at /home/feng/cfd/slepc-3.17.1/src/eps/interface/epssolve.c:147
> > > > > [0]PETSC ERROR: #9 slepc_eigen_comp() at domain/cfd/slepc_eigen_solve.cpp:77
> > > > >
> > > > > Could someone please shine some light on this? I have also attached my code. The code is part of my big library and I cannot attach the whole code, sorry about this. but I am happy to provide more information. The attached code has some arrangements for halo exchange, but for the moment, it assumes it is a serial run.
> > > > >
> > > > > Many thanks,
> > > > > Feng
> > > > > <slepc_eigen_solve.cpp>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220824/1081eb31/attachment.html>

From sagitter at fedoraproject.org  Wed Aug 24 14:32:48 2022
From: sagitter at fedoraproject.org (Antonio T. sagitter)
Date: Wed, 24 Aug 2022 21:32:48 +0200
Subject: [petsc-users] PETSc on fedora 36. Solved
In-Reply-To: <af11ac6f-a545-0927-7053-4e56e9b5e994@upc.edu>
References: <aa3cbfe2-2ccc-93cc-6004-5c2ae6429047@upc.edu>
	<358BFF34-D941-42AB-8648-871186EF64DB@petsc.dev>
	<65d7f5ad-fcd5-2cc9-0f74-8c9c8f5eaeb8@upc.edu>
	<5f57fbe-c49a-baad-faa-24e49ed6ed45@mcs.anl.gov>
	<af11ac6f-a545-0927-7053-4e56e9b5e994@upc.edu>
Message-ID: <860ac430-a8c8-388a-7205-0cfc07b1b0b1@fedoraproject.org>

New builds will be in the `updates-testing` repositories soon: 
https://bodhi.fedoraproject.org/updates/FEDORA-2022-b584048fe7

Or get them from 
https://koji.fedoraproject.org/koji/buildinfo?buildID=2051418

On 23/08/22 18:03, Rafel Amer Ramon wrote:
> Hi,
> 
> I have solved the problem installing only the petsc-openmpi packages:
> 
> ~]# rpm -qa | grep petsc
> petsc-openmpi-3.16.4-3.fc36.x86_64
> petsc-openmpi-devel-3.16.4-3.fc36.x86_64
> python3-petsc-openmpi-3.16.4-3.fc36.x86_64
> ~]#
> 
> ~$ cat e.c
> #include <petsc.h>
> 
> int main(int argc, char **argv) {
>    PetscErrorCode ierr;
>    PetscMPIInt    rank;
>    char processor_name[MPI_MAX_PROCESSOR_NAME];
>    int  namelen;
> 
>    ierr = PetscInitialize(&argc,&argv,NULL,
>        "Compute e in parallel with PETSc.\n\n"); if (ierr) return ierr;
> 	
>    ierr = MPI_Comm_rank(PETSC_COMM_WORLD,&rank); CHKERRQ(ierr);
>    MPI_Get_processor_name(processor_name,&namelen);
> 
>    ierr = PetscPrintf(PETSC_COMM_SELF,"My rank is %d on %s\n",rank,processor_name); CHKERRQ(ierr);
>     
>    return PetscFinalize();
> }
> 
> ~$ mpicc -o e e.c -I/usr/include/openmpi-x86_64/petsc -lpetsc
> 
> ~$ $ mpirun -np 16 --hostfile ~/hosts ./e
> My rank is 4 on mce-eseiaat.com
> My rank is 5 on mce-eseiaat.com
> My rank is 6 on mce-eseiaat.com
> My rank is 7 on mce-eseiaat.com
> My rank is 0 on mce-eseiaat.com
> My rank is 1 on mce-eseiaat.com
> My rank is 2 on mce-eseiaat.com
> My rank is 3 on mce-eseiaat.com
> My rank is 15 on worker2.mce-eseiaat.com
> My rank is 11 on worker1.mce-eseiaat.com
> My rank is 12 on worker2.mce-eseiaat.com
> My rank is 9 on worker1.mce-eseiaat.com
> My rank is 13 on worker2.mce-eseiaat.com
> My rank is 10 on worker1.mce-eseiaat.com
> My rank is 14 on worker2.mce-eseiaat.com
> My rank is 8 on worker1.mce-eseiaat.com
> ~$
> 
> Thank you!!!
> 
> Best regards,
> 
> Rafel Amer
> 
>   
> 
> 
> 
> 

-- 
---
Antonio Trande
Fedora Project
mailto: sagitter at fedoraproject.org
GPG key: 0xCC1CFEF30920C8AE
GPG key server: https://keyserver1.pgp.com/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_0xCC1CFEF30920C8AE.asc
Type: application/pgp-keys
Size: 1798 bytes
Desc: OpenPGP public key
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220824/b840122e/attachment-0001.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature
Type: application/pgp-signature
Size: 495 bytes
Desc: OpenPGP digital signature
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220824/b840122e/attachment-0001.sig>

From jroman at dsic.upv.es  Thu Aug 25 02:21:06 2022
From: jroman at dsic.upv.es (Jose E. Roman)
Date: Thu, 25 Aug 2022 09:21:06 +0200
Subject: [petsc-users] Slepc Question, shell-matrix
In-Reply-To: <ME3PR01MB762467CDD5598EC8E2AC5A50B9739@ME3PR01MB7624.ausprd01.prod.outlook.com>
References: <ME3PR01MB7624DF748CD2568A0BAAC5B6B9719@ME3PR01MB7624.ausprd01.prod.outlook.com>
	<C1E1C97D-7F02-4764-9732-9330A6076438@dsic.upv.es>
	<ME3PR01MB7624ADA914B555F17A490226B9719@ME3PR01MB7624.ausprd01.prod.outlook.com>
	<8B78A886-8CB5-49A9-AFED-E67C22696400@dsic.upv.es>
	<SYZPR01MB76275849E71CDD1083D79937B9709@SYZPR01MB7627.ausprd01.prod.outlook.com>
	<17F56A61-2DB1-416B-85CE-8E585051F1EB@dsic.upv.es>
	<ME3PR01MB762414A8D756C6C68BC95E92B9709@ME3PR01MB7624.ausprd01.prod.outlook.com>
	<3E25E7D6-C81A-4A75-A215-A5FCFB7F3CB6@dsic.upv.es>
	<ME3PR01MB762495A98C93E2D6EA7E30ABB9739@ME3PR01MB7624.ausprd01.prod.outlook.com>
	<6BC033FF-0A55-4564-923D-97EFDD7C7FF7@dsic.upv.es>
	<ME3PR01MB762467CDD5598EC8E2AC5A50B9739@ME3PR01MB7624.ausprd01.prod.outlook.com>
Message-ID: <8E59CD67-862C-463C-B3ED-1E530ECAE333@dsic.upv.es>

This works for simple examples such as ex9.c
It may be an indication that there is a problem with your shell matrix.

Jose


> El 24 ago 2022, a las 19:35, feng wang <snailsoar at hotmail.com> escribi?:
> 
> Hi Jose,
> 
> Thanks for your reply.
> 
> I have tried PetscCall(MatConvert(slepc_A_mf,MATDENSE,MAT_INITIAL_MATRIX,&Adense));  and If I do MatView(Adense,viewer), All I see was NAN. Have I missed anything here?
> 
> Thanks,
> Feng
> 
> From: Jose E. Roman <jroman at dsic.upv.es>
> Sent: 24 August 2022 16:23
> To: feng wang <snailsoar at hotmail.com>
> Cc: petsc-users <petsc-users at mcs.anl.gov>
> Subject: Re: [petsc-users] Slepc Question, shell-matrix
>  
> 
> 
> > El 24 ago 2022, a las 17:03, feng wang <snailsoar at hotmail.com> escribi?:
> > 
> > Hi Jose,
> > 
> > Thanks for your reply.
> > 
> > I have tried -eps_view_mat0 binary:amatrix.bin to save my shell matrix. It seems not saving the shell matrix and just created an empty file.
> 
> Oops, I thought the shell matrix would be converted automatically. Try converting it yourself:
> 
> PetscCall(MatConvert(slepc_A_mf,MATDENSE,MAT_INITIAL_MATRIX,&Adense));
> 
> Then use Adense in EPSSetOperators(). Or save it to disk and send it to me so that I can give it a try.
> 
> > 
> > Besides, I am trying to understand the output of slepc.  I can set  -eps_tol to a low value, but the final relative residual norm is still high , what does this tell me?
> 
> I don't know. It should not happen.
> Jose
> 
> > 
> > Best regards,
> > Feng
> > 
> > From: Jose E. Roman <jroman at dsic.upv.es>
> > Sent: 23 August 2022 11:06
> > To: feng wang <snailsoar at hotmail.com>
> > Cc: petsc-users <petsc-users at mcs.anl.gov>
> > Subject: Re: [petsc-users] Slepc Question, shell-matrix
> >  
> > You can try the following. Save your shell matrix to disk with the option
> > -eps_view_mat0 binary:amatrix.bin
> > then repeat the computation with ex4.c loading this file.
> > 
> > Note that this should be done for a small problem size, because the conversion from a shell matrix implies a matrix-vector product per each column.
> > 
> > Jose
> > 
> > > El 23 ago 2022, a las 12:57, feng wang <snailsoar at hotmail.com> escribi?:
> > > 
> > > Hi Jose,
> > > 
> > > Thanks for your reply.
> > > 
> > > It represents a linear operator. In my shell matrix, I am computing the non-linear residuals twice with perturbed flow variables. The matrix-vector product is computed as:
> > > 
> > > A*v = (R(q+eps*v) - R(q-eps*v))/(2*eps)
> > > 
> > > R is the non-linear residual. q is my flow variable and it does not change. eps is the perturbation. A is my Jacobian matrix. Besides, for some background, I am computing a steady RANS flow with finite volume method and trying to do a global stability analysis by looking at the Jacobian matrix. 
> > > 
> > > Thanks,
> > > Feng
> > > 
> > > From: Jose E. Roman <jroman at dsic.upv.es>
> > > Sent: 23 August 2022 10:32
> > > To: feng wang <snailsoar at hotmail.com>
> > > Cc: petsc-users <petsc-users at mcs.anl.gov>
> > > Subject: Re: [petsc-users] Slepc Question, shell-matrix
> > >  
> > > The relative residual norms that are printed at the end are too large. For NHEP problems, they should be below the tolerance. Don't know what is happening. Does your shell matrix represent a linear (constant) operator? Or does it change slightly depending on the input vector? 
> > > 
> > > > El 23 ago 2022, a las 12:14, feng wang <snailsoar at hotmail.com> escribi?:
> > > > 
> > > > Hi Jose,
> > > > 
> > > > I think the previous problem comes from my side. I have some uninitialized values in my part of code to compute the non-linear residuals. so, it produces a NAN when it tries to compute the matrix-vector product using finite difference.  This might make the slepc/pestc do unexpected things.
> > > > 
> > > > Now It seems I've got slepc running. eps_nev is set to 3 and I am trying to compute the ones with the largest amplitudes. Below is the slepc output. 
> > > > 
> > > >  14 EPS converged value (error) #0 -0.000164282 (5.36813206e-09)
> > > >  16 EPS converged value (error) #1 -0.000160691+2.17113e-05i (3.37429620e-09)
> > > >  16 EPS converged value (error) #2 -0.000160691-2.17113e-05i (3.37429620e-09)
> > > >  Solution method: krylovschur
> > > > 
> > > >  Number of requested eigenvalues: 2
> > > >  Linear eigensolve converged (3 eigenpairs) due to CONVERGED_TOL; iterations 16
> > > >  ---------------------- --------------------
> > > >             k             ||Ax-kx||/||kx||
> > > >  ---------------------- --------------------
> > > >        -0.000164              0.0613788
> > > >   -0.000161+0.000022i         0.0773339
> > > >   -0.000161-0.000022i         0.0774536
> > > >  ---------------------- --------------------
> > > > 
> > > > The values in the brackets are the absolute error (I believe) and they seem very low. The relative error seems quite large.  Could you please comment on this? 
> > > > 
> > > > 
> > > > Best regards,
> > > > Feng
> > > > 
> > > > From: Jose E. Roman <jroman at dsic.upv.es>
> > > > Sent: 23 August 2022 5:24
> > > > To: feng wang <snailsoar at hotmail.com>
> > > > Cc: petsc-users <petsc-users at mcs.anl.gov>
> > > > Subject: Re: [petsc-users] Slepc Question, shell-matrix
> > > >  
> > > > Please always respond to the list, otherwise the thread appears as unresolved in the archives of the mailing list.
> > > > 
> > > > 
> > > > > El 22 ago 2022, a las 22:45, feng wang <snailsoar at hotmail.com> escribi?:
> > > > > 
> > > > > Hi Jose,
> > > > > 
> > > > > I think I might have solved my problem. I have some uninitialized values in my part of code to compute the right hand side. so it produces a NAN when it tries to compute the matrix-vector product. 
> > > > > 
> > > > > Many thanks for your help!
> > > > > 
> > > > > Best regards,
> > > > > Feng
> > > > > From: Jose E. Roman <jroman at dsic.upv.es>
> > > > > Sent: 22 August 2022 19:32
> > > > > To: feng wang <snailsoar at hotmail.com>
> > > > > Cc: petsc-users at mcs.anl.gov <petsc-users at mcs.anl.gov>
> > > > > Subject: Re: [petsc-users] Slepc Question, shell-matrix
> > > > >  
> > > > > This is very strange. This error appears when the solver employs the B-inner product, but in your case you don't have a B matrix, so you should never see that error. Try running under valgrind to see if it gives more hints.
> > > > > 
> > > > > Jose
> > > > > 
> > > > > 
> > > > > > El 22 ago 2022, a las 20:45, feng wang <snailsoar at hotmail.com> escribi?:
> > > > > > 
> > > > > > Hello,
> > > > > > 
> > > > > > I am new to Slepc and trying to work out the eigenvalues and eigenvectors of my Jacobian matrix. I am using a shell matrix to work out the matrix-vector product and I am using the default Krylov-schur method.  
> > > > > > 
> > > > > > My first attempt was not successful and I got the following errors:
> > > > > > 
> > > > > > [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> > > > > > [0]PETSC ERROR: Missing or incorrect user input 
> > > > > > [0]PETSC ERROR: The inner product is not well defined: indefinite matrix
> > > > > > [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
> > > > > > [0]PETSC ERROR: Petsc Release Version 3.17.4, unknown 
> > > > > > [0]PETSC ERROR: cfdtest on a arch-debug named ming by feng Mon Aug 22 19:21:41 2022
> > > > > > [0]PETSC ERROR: Configure options --with-cc=mpicc --with-cxx=mpicxx --with-fc=0 PETSC_ARCH=arch-debug
> > > > > > [0]PETSC ERROR: #1 BV_SafeSqrt() at /home/feng/cfd/slepc-3.17.1/include/slepc/private/bvimpl.h:130
> > > > > > [0]PETSC ERROR: #2 BV_SquareRoot_Default() at /home/feng/cfd/slepc-3.17.1/include/slepc/private/bvimpl.h:365
> > > > > > [0]PETSC ERROR: #3 BVOrthogonalizeCGS1() at /home/feng/cfd/slepc-3.17.1/src/sys/classes/bv/interface/bvorthog.c:101
> > > > > > [0]PETSC ERROR: #4 BVOrthogonalizeGS() at /home/feng/cfd/slepc-3.17.1/src/sys/classes/bv/interface/bvorthog.c:177
> > > > > > [0]PETSC ERROR: #5 BVOrthonormalizeColumn() at /home/feng/cfd/slepc-3.17.1/src/sys/classes/bv/interface/bvorthog.c:402
> > > > > > [0]PETSC ERROR: #6 BVMatArnoldi() at /home/feng/cfd/slepc-3.17.1/src/sys/classes/bv/interface/bvkrylov.c:91
> > > > > > [0]PETSC ERROR: #7 EPSSolve_KrylovSchur_Default() at /home/feng/cfd/slepc-3.17.1/src/eps/impls/krylov/krylovschur/krylovschur.c:261
> > > > > > [0]PETSC ERROR: #8 EPSSolve() at /home/feng/cfd/slepc-3.17.1/src/eps/interface/epssolve.c:147
> > > > > > [0]PETSC ERROR: #9 slepc_eigen_comp() at domain/cfd/slepc_eigen_solve.cpp:77
> > > > > > 
> > > > > > Could someone please shine some light on this? I have also attached my code. The code is part of my big library and I cannot attach the whole code, sorry about this. but I am happy to provide more information. The attached code has some arrangements for halo exchange, but for the moment, it assumes it is a serial run.
> > > > > > 
> > > > > > Many thanks,
> > > > > > Feng
> > > > > > <slepc_eigen_solve.cpp>


From snailsoar at hotmail.com  Thu Aug 25 04:34:19 2022
From: snailsoar at hotmail.com (feng wang)
Date: Thu, 25 Aug 2022 09:34:19 +0000
Subject: [petsc-users] Slepc Question, shell-matrix
In-Reply-To: <8E59CD67-862C-463C-B3ED-1E530ECAE333@dsic.upv.es>
References: <ME3PR01MB7624DF748CD2568A0BAAC5B6B9719@ME3PR01MB7624.ausprd01.prod.outlook.com>
	<C1E1C97D-7F02-4764-9732-9330A6076438@dsic.upv.es>
	<ME3PR01MB7624ADA914B555F17A490226B9719@ME3PR01MB7624.ausprd01.prod.outlook.com>
	<8B78A886-8CB5-49A9-AFED-E67C22696400@dsic.upv.es>
	<SYZPR01MB76275849E71CDD1083D79937B9709@SYZPR01MB7627.ausprd01.prod.outlook.com>
	<17F56A61-2DB1-416B-85CE-8E585051F1EB@dsic.upv.es>
	<ME3PR01MB762414A8D756C6C68BC95E92B9709@ME3PR01MB7624.ausprd01.prod.outlook.com>
	<3E25E7D6-C81A-4A75-A215-A5FCFB7F3CB6@dsic.upv.es>
	<ME3PR01MB762495A98C93E2D6EA7E30ABB9739@ME3PR01MB7624.ausprd01.prod.outlook.com>
	<6BC033FF-0A55-4564-923D-97EFDD7C7FF7@dsic.upv.es>
	<ME3PR01MB762467CDD5598EC8E2AC5A50B9739@ME3PR01MB7624.ausprd01.prod.outlook.com>
	<8E59CD67-862C-463C-B3ED-1E530ECAE333@dsic.upv.es>
Message-ID: <ME3PR01MB7624F86687223C01A676BA28B9729@ME3PR01MB7624.ausprd01.prod.outlook.com>

Hi Jose,

Thanks for your reply. I have fixed one thing in my shell matrix. Now the conversion seems working.

If I use the converted dense matrix and give it to EPSSetOperators, it actually converges!

 50 EPS nconv=0 first unconverged value (error) 2.89521e-06+2.13473e-06i (7.73051721e-08)
 51 EPS nconv=0 first unconverged value (error) 2.89521e-06+2.13473e-06i (5.42681175e-08)
 52 EPS nconv=0 first unconverged value (error) 2.89521e-06+2.13473e-06i (2.48352708e-08)
 53 EPS nconv=0 first unconverged value (error) 2.89521e-06+2.13473e-06i (1.76912430e-08)
 54 EPS nconv=0 first unconverged value (error) 2.89521e-06+2.13473e-06i (2.07480734e-08)
 55 EPS nconv=0 first unconverged value (error) 2.89521e-06+2.13473e-06i (1.13588981e-08)
 56 EPS nconv=2 first unconverged value (error) 1.10143e-06+9.80005e-06i (8.32889697e-06)
 Solution method: krylovschur

 Number of requested eigenvalues: 1
 Linear eigensolve converged (2 eigenpairs) due to CONVERGED_TOL; iterations 56
 ---------------------- --------------------
            k             ||Ax-kx||/||kx||
 ---------------------- --------------------
   0.000003+0.000002i       7.90447e-09
   0.000003-0.000002i       7.90447e-09
 ---------------------- --------------------

If I use the shell matrix directly, the relative norm is still large. This really puzzles me......

Thanks,
Feng

________________________________
From: Jose E. Roman <jroman at dsic.upv.es>
Sent: 25 August 2022 7:21
To: feng wang <snailsoar at hotmail.com>
Cc: petsc-users <petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] Slepc Question, shell-matrix

This works for simple examples such as ex9.c
It may be an indication that there is a problem with your shell matrix.

Jose


> El 24 ago 2022, a las 19:35, feng wang <snailsoar at hotmail.com> escribi?:
>
> Hi Jose,
>
> Thanks for your reply.
>
> I have tried PetscCall(MatConvert(slepc_A_mf,MATDENSE,MAT_INITIAL_MATRIX,&Adense));  and If I do MatView(Adense,viewer), All I see was NAN. Have I missed anything here?
>
> Thanks,
> Feng
>
> From: Jose E. Roman <jroman at dsic.upv.es>
> Sent: 24 August 2022 16:23
> To: feng wang <snailsoar at hotmail.com>
> Cc: petsc-users <petsc-users at mcs.anl.gov>
> Subject: Re: [petsc-users] Slepc Question, shell-matrix
>
>
>
> > El 24 ago 2022, a las 17:03, feng wang <snailsoar at hotmail.com> escribi?:
> >
> > Hi Jose,
> >
> > Thanks for your reply.
> >
> > I have tried -eps_view_mat0 binary:amatrix.bin to save my shell matrix. It seems not saving the shell matrix and just created an empty file.
>
> Oops, I thought the shell matrix would be converted automatically. Try converting it yourself:
>
> PetscCall(MatConvert(slepc_A_mf,MATDENSE,MAT_INITIAL_MATRIX,&Adense));
>
> Then use Adense in EPSSetOperators(). Or save it to disk and send it to me so that I can give it a try.
>
> >
> > Besides, I am trying to understand the output of slepc.  I can set  -eps_tol to a low value, but the final relative residual norm is still high , what does this tell me?
>
> I don't know. It should not happen.
> Jose
>
> >
> > Best regards,
> > Feng
> >
> > From: Jose E. Roman <jroman at dsic.upv.es>
> > Sent: 23 August 2022 11:06
> > To: feng wang <snailsoar at hotmail.com>
> > Cc: petsc-users <petsc-users at mcs.anl.gov>
> > Subject: Re: [petsc-users] Slepc Question, shell-matrix
> >
> > You can try the following. Save your shell matrix to disk with the option
> > -eps_view_mat0 binary:amatrix.bin
> > then repeat the computation with ex4.c loading this file.
> >
> > Note that this should be done for a small problem size, because the conversion from a shell matrix implies a matrix-vector product per each column.
> >
> > Jose
> >
> > > El 23 ago 2022, a las 12:57, feng wang <snailsoar at hotmail.com> escribi?:
> > >
> > > Hi Jose,
> > >
> > > Thanks for your reply.
> > >
> > > It represents a linear operator. In my shell matrix, I am computing the non-linear residuals twice with perturbed flow variables. The matrix-vector product is computed as:
> > >
> > > A*v = (R(q+eps*v) - R(q-eps*v))/(2*eps)
> > >
> > > R is the non-linear residual. q is my flow variable and it does not change. eps is the perturbation. A is my Jacobian matrix. Besides, for some background, I am computing a steady RANS flow with finite volume method and trying to do a global stability analysis by looking at the Jacobian matrix.
> > >
> > > Thanks,
> > > Feng
> > >
> > > From: Jose E. Roman <jroman at dsic.upv.es>
> > > Sent: 23 August 2022 10:32
> > > To: feng wang <snailsoar at hotmail.com>
> > > Cc: petsc-users <petsc-users at mcs.anl.gov>
> > > Subject: Re: [petsc-users] Slepc Question, shell-matrix
> > >
> > > The relative residual norms that are printed at the end are too large. For NHEP problems, they should be below the tolerance. Don't know what is happening. Does your shell matrix represent a linear (constant) operator? Or does it change slightly depending on the input vector?
> > >
> > > > El 23 ago 2022, a las 12:14, feng wang <snailsoar at hotmail.com> escribi?:
> > > >
> > > > Hi Jose,
> > > >
> > > > I think the previous problem comes from my side. I have some uninitialized values in my part of code to compute the non-linear residuals. so, it produces a NAN when it tries to compute the matrix-vector product using finite difference.  This might make the slepc/pestc do unexpected things.
> > > >
> > > > Now It seems I've got slepc running. eps_nev is set to 3 and I am trying to compute the ones with the largest amplitudes. Below is the slepc output.
> > > >
> > > >  14 EPS converged value (error) #0 -0.000164282 (5.36813206e-09)
> > > >  16 EPS converged value (error) #1 -0.000160691+2.17113e-05i (3.37429620e-09)
> > > >  16 EPS converged value (error) #2 -0.000160691-2.17113e-05i (3.37429620e-09)
> > > >  Solution method: krylovschur
> > > >
> > > >  Number of requested eigenvalues: 2
> > > >  Linear eigensolve converged (3 eigenpairs) due to CONVERGED_TOL; iterations 16
> > > >  ---------------------- --------------------
> > > >             k             ||Ax-kx||/||kx||
> > > >  ---------------------- --------------------
> > > >        -0.000164              0.0613788
> > > >   -0.000161+0.000022i         0.0773339
> > > >   -0.000161-0.000022i         0.0774536
> > > >  ---------------------- --------------------
> > > >
> > > > The values in the brackets are the absolute error (I believe) and they seem very low. The relative error seems quite large.  Could you please comment on this?
> > > >
> > > >
> > > > Best regards,
> > > > Feng
> > > >
> > > > From: Jose E. Roman <jroman at dsic.upv.es>
> > > > Sent: 23 August 2022 5:24
> > > > To: feng wang <snailsoar at hotmail.com>
> > > > Cc: petsc-users <petsc-users at mcs.anl.gov>
> > > > Subject: Re: [petsc-users] Slepc Question, shell-matrix
> > > >
> > > > Please always respond to the list, otherwise the thread appears as unresolved in the archives of the mailing list.
> > > >
> > > >
> > > > > El 22 ago 2022, a las 22:45, feng wang <snailsoar at hotmail.com> escribi?:
> > > > >
> > > > > Hi Jose,
> > > > >
> > > > > I think I might have solved my problem. I have some uninitialized values in my part of code to compute the right hand side. so it produces a NAN when it tries to compute the matrix-vector product.
> > > > >
> > > > > Many thanks for your help!
> > > > >
> > > > > Best regards,
> > > > > Feng
> > > > > From: Jose E. Roman <jroman at dsic.upv.es>
> > > > > Sent: 22 August 2022 19:32
> > > > > To: feng wang <snailsoar at hotmail.com>
> > > > > Cc: petsc-users at mcs.anl.gov <petsc-users at mcs.anl.gov>
> > > > > Subject: Re: [petsc-users] Slepc Question, shell-matrix
> > > > >
> > > > > This is very strange. This error appears when the solver employs the B-inner product, but in your case you don't have a B matrix, so you should never see that error. Try running under valgrind to see if it gives more hints.
> > > > >
> > > > > Jose
> > > > >
> > > > >
> > > > > > El 22 ago 2022, a las 20:45, feng wang <snailsoar at hotmail.com> escribi?:
> > > > > >
> > > > > > Hello,
> > > > > >
> > > > > > I am new to Slepc and trying to work out the eigenvalues and eigenvectors of my Jacobian matrix. I am using a shell matrix to work out the matrix-vector product and I am using the default Krylov-schur method.
> > > > > >
> > > > > > My first attempt was not successful and I got the following errors:
> > > > > >
> > > > > > [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> > > > > > [0]PETSC ERROR: Missing or incorrect user input
> > > > > > [0]PETSC ERROR: The inner product is not well defined: indefinite matrix
> > > > > > [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
> > > > > > [0]PETSC ERROR: Petsc Release Version 3.17.4, unknown
> > > > > > [0]PETSC ERROR: cfdtest on a arch-debug named ming by feng Mon Aug 22 19:21:41 2022
> > > > > > [0]PETSC ERROR: Configure options --with-cc=mpicc --with-cxx=mpicxx --with-fc=0 PETSC_ARCH=arch-debug
> > > > > > [0]PETSC ERROR: #1 BV_SafeSqrt() at /home/feng/cfd/slepc-3.17.1/include/slepc/private/bvimpl.h:130
> > > > > > [0]PETSC ERROR: #2 BV_SquareRoot_Default() at /home/feng/cfd/slepc-3.17.1/include/slepc/private/bvimpl.h:365
> > > > > > [0]PETSC ERROR: #3 BVOrthogonalizeCGS1() at /home/feng/cfd/slepc-3.17.1/src/sys/classes/bv/interface/bvorthog.c:101
> > > > > > [0]PETSC ERROR: #4 BVOrthogonalizeGS() at /home/feng/cfd/slepc-3.17.1/src/sys/classes/bv/interface/bvorthog.c:177
> > > > > > [0]PETSC ERROR: #5 BVOrthonormalizeColumn() at /home/feng/cfd/slepc-3.17.1/src/sys/classes/bv/interface/bvorthog.c:402
> > > > > > [0]PETSC ERROR: #6 BVMatArnoldi() at /home/feng/cfd/slepc-3.17.1/src/sys/classes/bv/interface/bvkrylov.c:91
> > > > > > [0]PETSC ERROR: #7 EPSSolve_KrylovSchur_Default() at /home/feng/cfd/slepc-3.17.1/src/eps/impls/krylov/krylovschur/krylovschur.c:261
> > > > > > [0]PETSC ERROR: #8 EPSSolve() at /home/feng/cfd/slepc-3.17.1/src/eps/interface/epssolve.c:147
> > > > > > [0]PETSC ERROR: #9 slepc_eigen_comp() at domain/cfd/slepc_eigen_solve.cpp:77
> > > > > >
> > > > > > Could someone please shine some light on this? I have also attached my code. The code is part of my big library and I cannot attach the whole code, sorry about this. but I am happy to provide more information. The attached code has some arrangements for halo exchange, but for the moment, it assumes it is a serial run.
> > > > > >
> > > > > > Many thanks,
> > > > > > Feng
> > > > > > <slepc_eigen_solve.cpp>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220825/21f2b9b2/attachment-0001.html>

From jroman at dsic.upv.es  Thu Aug 25 04:45:24 2022
From: jroman at dsic.upv.es (Jose E. Roman)
Date: Thu, 25 Aug 2022 11:45:24 +0200
Subject: [petsc-users] Slepc Question, shell-matrix
In-Reply-To: <ME3PR01MB7624F86687223C01A676BA28B9729@ME3PR01MB7624.ausprd01.prod.outlook.com>
References: <ME3PR01MB7624DF748CD2568A0BAAC5B6B9719@ME3PR01MB7624.ausprd01.prod.outlook.com>
	<C1E1C97D-7F02-4764-9732-9330A6076438@dsic.upv.es>
	<ME3PR01MB7624ADA914B555F17A490226B9719@ME3PR01MB7624.ausprd01.prod.outlook.com>
	<8B78A886-8CB5-49A9-AFED-E67C22696400@dsic.upv.es>
	<SYZPR01MB76275849E71CDD1083D79937B9709@SYZPR01MB7627.ausprd01.prod.outlook.com>
	<17F56A61-2DB1-416B-85CE-8E585051F1EB@dsic.upv.es>
	<ME3PR01MB762414A8D756C6C68BC95E92B9709@ME3PR01MB7624.ausprd01.prod.outlook.com>
	<3E25E7D6-C81A-4A75-A215-A5FCFB7F3CB6@dsic.upv.es>
	<ME3PR01MB762495A98C93E2D6EA7E30ABB9739@ME3PR01MB7624.ausprd01.prod.outlook.com>
	<6BC033FF-0A55-4564-923D-97EFDD7C7FF7@dsic.upv.es>
	<ME3PR01MB762467CDD5598EC8E2AC5A50B9739@ME3PR01MB7624.ausprd01.prod.outlook.com>
	<8E59CD67-862C-463C-B3ED-1E530ECAE333@dsic.upv.es>
	<ME3PR01MB7624F86687223C01A676BA28B9729@ME3PR01MB7624.ausprd01.prod.outlook.com>
Message-ID: <EDA8B12C-5637-4F23-8745-CB44C8A2367E@dsic.upv.es>

Probably the explanation is the following. Your shell matrix runs an iterative procedure that computes the action of the matrix to a certain accuracy. This is equivalent to multiplying by the exact matrix plus a perturbation, A+E. But every time you apply the shell matrix, the perturbation E is slightly different, so the eigensolver does not really see a constant matrix, it changes at every iteration.

This also happens in nested iterations, such as inexact shift-and-invert, where e.g. GMRES is run within each iteration of the eigensolver. For this to work, the GMRES tolerance should be smaller than the tolerance of the eigensolver. Similarly, if your shell matrix runs an iterative procedure, you have to use a tolerance that is more stringent than the one used in the eigensolver.

Hope this helps.
Jose


> El 25 ago 2022, a las 11:34, feng wang <snailsoar at hotmail.com> escribi?:
> 
> Hi Jose,
> 
> Thanks for your reply. I have fixed one thing in my shell matrix. Now the conversion seems working.
> 
> If I use the converted dense matrix and give it to EPSSetOperators, it actually converges!
> 
>  50 EPS nconv=0 first unconverged value (error) 2.89521e-06+2.13473e-06i (7.73051721e-08)
>  51 EPS nconv=0 first unconverged value (error) 2.89521e-06+2.13473e-06i (5.42681175e-08)
>  52 EPS nconv=0 first unconverged value (error) 2.89521e-06+2.13473e-06i (2.48352708e-08)
>  53 EPS nconv=0 first unconverged value (error) 2.89521e-06+2.13473e-06i (1.76912430e-08)
>  54 EPS nconv=0 first unconverged value (error) 2.89521e-06+2.13473e-06i (2.07480734e-08)
>  55 EPS nconv=0 first unconverged value (error) 2.89521e-06+2.13473e-06i (1.13588981e-08)
>  56 EPS nconv=2 first unconverged value (error) 1.10143e-06+9.80005e-06i (8.32889697e-06)
>  Solution method: krylovschur
> 
>  Number of requested eigenvalues: 1
>  Linear eigensolve converged (2 eigenpairs) due to CONVERGED_TOL; iterations 56
>  ---------------------- --------------------
>             k             ||Ax-kx||/||kx||
>  ---------------------- --------------------
>    0.000003+0.000002i       7.90447e-09
>    0.000003-0.000002i       7.90447e-09
>  ---------------------- --------------------
> 
> If I use the shell matrix directly, the relative norm is still large. This really puzzles me...... 
> 
> Thanks,
> Feng
> 
> From: Jose E. Roman <jroman at dsic.upv.es>
> Sent: 25 August 2022 7:21
> To: feng wang <snailsoar at hotmail.com>
> Cc: petsc-users <petsc-users at mcs.anl.gov>
> Subject: Re: [petsc-users] Slepc Question, shell-matrix
>  
> This works for simple examples such as ex9.c
> It may be an indication that there is a problem with your shell matrix.
> 
> Jose
> 
> 
> > El 24 ago 2022, a las 19:35, feng wang <snailsoar at hotmail.com> escribi?:
> > 
> > Hi Jose,
> > 
> > Thanks for your reply.
> > 
> > I have tried PetscCall(MatConvert(slepc_A_mf,MATDENSE,MAT_INITIAL_MATRIX,&Adense));  and If I do MatView(Adense,viewer), All I see was NAN. Have I missed anything here?
> > 
> > Thanks,
> > Feng
> > 
> > From: Jose E. Roman <jroman at dsic.upv.es>
> > Sent: 24 August 2022 16:23
> > To: feng wang <snailsoar at hotmail.com>
> > Cc: petsc-users <petsc-users at mcs.anl.gov>
> > Subject: Re: [petsc-users] Slepc Question, shell-matrix
> >  
> > 
> > 
> > > El 24 ago 2022, a las 17:03, feng wang <snailsoar at hotmail.com> escribi?:
> > > 
> > > Hi Jose,
> > > 
> > > Thanks for your reply.
> > > 
> > > I have tried -eps_view_mat0 binary:amatrix.bin to save my shell matrix. It seems not saving the shell matrix and just created an empty file.
> > 
> > Oops, I thought the shell matrix would be converted automatically. Try converting it yourself:
> > 
> > PetscCall(MatConvert(slepc_A_mf,MATDENSE,MAT_INITIAL_MATRIX,&Adense));
> > 
> > Then use Adense in EPSSetOperators(). Or save it to disk and send it to me so that I can give it a try.
> > 
> > > 
> > > Besides, I am trying to understand the output of slepc.  I can set  -eps_tol to a low value, but the final relative residual norm is still high , what does this tell me?
> > 
> > I don't know. It should not happen.
> > Jose
> > 
> > > 
> > > Best regards,
> > > Feng
> > > 
> > > From: Jose E. Roman <jroman at dsic.upv.es>
> > > Sent: 23 August 2022 11:06
> > > To: feng wang <snailsoar at hotmail.com>
> > > Cc: petsc-users <petsc-users at mcs.anl.gov>
> > > Subject: Re: [petsc-users] Slepc Question, shell-matrix
> > >  
> > > You can try the following. Save your shell matrix to disk with the option
> > > -eps_view_mat0 binary:amatrix.bin
> > > then repeat the computation with ex4.c loading this file.
> > > 
> > > Note that this should be done for a small problem size, because the conversion from a shell matrix implies a matrix-vector product per each column.
> > > 
> > > Jose
> > > 
> > > > El 23 ago 2022, a las 12:57, feng wang <snailsoar at hotmail.com> escribi?:
> > > > 
> > > > Hi Jose,
> > > > 
> > > > Thanks for your reply.
> > > > 
> > > > It represents a linear operator. In my shell matrix, I am computing the non-linear residuals twice with perturbed flow variables. The matrix-vector product is computed as:
> > > > 
> > > > A*v = (R(q+eps*v) - R(q-eps*v))/(2*eps)
> > > > 
> > > > R is the non-linear residual. q is my flow variable and it does not change. eps is the perturbation. A is my Jacobian matrix. Besides, for some background, I am computing a steady RANS flow with finite volume method and trying to do a global stability analysis by looking at the Jacobian matrix. 
> > > > 
> > > > Thanks,
> > > > Feng
> > > > 
> > > > From: Jose E. Roman <jroman at dsic.upv.es>
> > > > Sent: 23 August 2022 10:32
> > > > To: feng wang <snailsoar at hotmail.com>
> > > > Cc: petsc-users <petsc-users at mcs.anl.gov>
> > > > Subject: Re: [petsc-users] Slepc Question, shell-matrix
> > > >  
> > > > The relative residual norms that are printed at the end are too large. For NHEP problems, they should be below the tolerance. Don't know what is happening. Does your shell matrix represent a linear (constant) operator? Or does it change slightly depending on the input vector? 
> > > > 
> > > > > El 23 ago 2022, a las 12:14, feng wang <snailsoar at hotmail.com> escribi?:
> > > > > 
> > > > > Hi Jose,
> > > > > 
> > > > > I think the previous problem comes from my side. I have some uninitialized values in my part of code to compute the non-linear residuals. so, it produces a NAN when it tries to compute the matrix-vector product using finite difference.  This might make the slepc/pestc do unexpected things.
> > > > > 
> > > > > Now It seems I've got slepc running. eps_nev is set to 3 and I am trying to compute the ones with the largest amplitudes. Below is the slepc output. 
> > > > > 
> > > > >  14 EPS converged value (error) #0 -0.000164282 (5.36813206e-09)
> > > > >  16 EPS converged value (error) #1 -0.000160691+2.17113e-05i (3.37429620e-09)
> > > > >  16 EPS converged value (error) #2 -0.000160691-2.17113e-05i (3.37429620e-09)
> > > > >  Solution method: krylovschur
> > > > > 
> > > > >  Number of requested eigenvalues: 2
> > > > >  Linear eigensolve converged (3 eigenpairs) due to CONVERGED_TOL; iterations 16
> > > > >  ---------------------- --------------------
> > > > >             k             ||Ax-kx||/||kx||
> > > > >  ---------------------- --------------------
> > > > >        -0.000164              0.0613788
> > > > >   -0.000161+0.000022i         0.0773339
> > > > >   -0.000161-0.000022i         0.0774536
> > > > >  ---------------------- --------------------
> > > > > 
> > > > > The values in the brackets are the absolute error (I believe) and they seem very low. The relative error seems quite large.  Could you please comment on this? 
> > > > > 
> > > > > 
> > > > > Best regards,
> > > > > Feng
> > > > > 
> > > > > From: Jose E. Roman <jroman at dsic.upv.es>
> > > > > Sent: 23 August 2022 5:24
> > > > > To: feng wang <snailsoar at hotmail.com>
> > > > > Cc: petsc-users <petsc-users at mcs.anl.gov>
> > > > > Subject: Re: [petsc-users] Slepc Question, shell-matrix
> > > > >  
> > > > > Please always respond to the list, otherwise the thread appears as unresolved in the archives of the mailing list.
> > > > > 
> > > > > 
> > > > > > El 22 ago 2022, a las 22:45, feng wang <snailsoar at hotmail.com> escribi?:
> > > > > > 
> > > > > > Hi Jose,
> > > > > > 
> > > > > > I think I might have solved my problem. I have some uninitialized values in my part of code to compute the right hand side. so it produces a NAN when it tries to compute the matrix-vector product. 
> > > > > > 
> > > > > > Many thanks for your help!
> > > > > > 
> > > > > > Best regards,
> > > > > > Feng
> > > > > > From: Jose E. Roman <jroman at dsic.upv.es>
> > > > > > Sent: 22 August 2022 19:32
> > > > > > To: feng wang <snailsoar at hotmail.com>
> > > > > > Cc: petsc-users at mcs.anl.gov <petsc-users at mcs.anl.gov>
> > > > > > Subject: Re: [petsc-users] Slepc Question, shell-matrix
> > > > > >  
> > > > > > This is very strange. This error appears when the solver employs the B-inner product, but in your case you don't have a B matrix, so you should never see that error. Try running under valgrind to see if it gives more hints.
> > > > > > 
> > > > > > Jose
> > > > > > 
> > > > > > 
> > > > > > > El 22 ago 2022, a las 20:45, feng wang <snailsoar at hotmail.com> escribi?:
> > > > > > > 
> > > > > > > Hello,
> > > > > > > 
> > > > > > > I am new to Slepc and trying to work out the eigenvalues and eigenvectors of my Jacobian matrix. I am using a shell matrix to work out the matrix-vector product and I am using the default Krylov-schur method.  
> > > > > > > 
> > > > > > > My first attempt was not successful and I got the following errors:
> > > > > > > 
> > > > > > > [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> > > > > > > [0]PETSC ERROR: Missing or incorrect user input 
> > > > > > > [0]PETSC ERROR: The inner product is not well defined: indefinite matrix
> > > > > > > [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
> > > > > > > [0]PETSC ERROR: Petsc Release Version 3.17.4, unknown 
> > > > > > > [0]PETSC ERROR: cfdtest on a arch-debug named ming by feng Mon Aug 22 19:21:41 2022
> > > > > > > [0]PETSC ERROR: Configure options --with-cc=mpicc --with-cxx=mpicxx --with-fc=0 PETSC_ARCH=arch-debug
> > > > > > > [0]PETSC ERROR: #1 BV_SafeSqrt() at /home/feng/cfd/slepc-3.17.1/include/slepc/private/bvimpl.h:130
> > > > > > > [0]PETSC ERROR: #2 BV_SquareRoot_Default() at /home/feng/cfd/slepc-3.17.1/include/slepc/private/bvimpl.h:365
> > > > > > > [0]PETSC ERROR: #3 BVOrthogonalizeCGS1() at /home/feng/cfd/slepc-3.17.1/src/sys/classes/bv/interface/bvorthog.c:101
> > > > > > > [0]PETSC ERROR: #4 BVOrthogonalizeGS() at /home/feng/cfd/slepc-3.17.1/src/sys/classes/bv/interface/bvorthog.c:177
> > > > > > > [0]PETSC ERROR: #5 BVOrthonormalizeColumn() at /home/feng/cfd/slepc-3.17.1/src/sys/classes/bv/interface/bvorthog.c:402
> > > > > > > [0]PETSC ERROR: #6 BVMatArnoldi() at /home/feng/cfd/slepc-3.17.1/src/sys/classes/bv/interface/bvkrylov.c:91
> > > > > > > [0]PETSC ERROR: #7 EPSSolve_KrylovSchur_Default() at /home/feng/cfd/slepc-3.17.1/src/eps/impls/krylov/krylovschur/krylovschur.c:261
> > > > > > > [0]PETSC ERROR: #8 EPSSolve() at /home/feng/cfd/slepc-3.17.1/src/eps/interface/epssolve.c:147
> > > > > > > [0]PETSC ERROR: #9 slepc_eigen_comp() at domain/cfd/slepc_eigen_solve.cpp:77
> > > > > > > 
> > > > > > > Could someone please shine some light on this? I have also attached my code. The code is part of my big library and I cannot attach the whole code, sorry about this. but I am happy to provide more information. The attached code has some arrangements for halo exchange, but for the moment, it assumes it is a serial run.
> > > > > > > 
> > > > > > > Many thanks,
> > > > > > > Feng
> > > > > > > <slepc_eigen_solve.cpp>


From snailsoar at hotmail.com  Thu Aug 25 05:31:03 2022
From: snailsoar at hotmail.com (feng wang)
Date: Thu, 25 Aug 2022 10:31:03 +0000
Subject: [petsc-users] Slepc Question, shell-matrix
In-Reply-To: <EDA8B12C-5637-4F23-8745-CB44C8A2367E@dsic.upv.es>
References: <ME3PR01MB7624DF748CD2568A0BAAC5B6B9719@ME3PR01MB7624.ausprd01.prod.outlook.com>
	<C1E1C97D-7F02-4764-9732-9330A6076438@dsic.upv.es>
	<ME3PR01MB7624ADA914B555F17A490226B9719@ME3PR01MB7624.ausprd01.prod.outlook.com>
	<8B78A886-8CB5-49A9-AFED-E67C22696400@dsic.upv.es>
	<SYZPR01MB76275849E71CDD1083D79937B9709@SYZPR01MB7627.ausprd01.prod.outlook.com>
	<17F56A61-2DB1-416B-85CE-8E585051F1EB@dsic.upv.es>
	<ME3PR01MB762414A8D756C6C68BC95E92B9709@ME3PR01MB7624.ausprd01.prod.outlook.com>
	<3E25E7D6-C81A-4A75-A215-A5FCFB7F3CB6@dsic.upv.es>
	<ME3PR01MB762495A98C93E2D6EA7E30ABB9739@ME3PR01MB7624.ausprd01.prod.outlook.com>
	<6BC033FF-0A55-4564-923D-97EFDD7C7FF7@dsic.upv.es>
	<ME3PR01MB762467CDD5598EC8E2AC5A50B9739@ME3PR01MB7624.ausprd01.prod.outlook.com>
	<8E59CD67-862C-463C-B3ED-1E530ECAE333@dsic.upv.es>
	<ME3PR01MB7624F86687223C01A676BA28B9729@ME3PR01MB7624.ausprd01.prod.outlook.com>
	<EDA8B12C-5637-4F23-8745-CB44C8A2367E@dsic.upv.es>
Message-ID: <ME3PR01MB76243360ABDAA3EEB57F9844B9729@ME3PR01MB7624.ausprd01.prod.outlook.com>

Hi Jose,

Thanks for your explanation. It's very helpful!

In my shell matrix, I am using finite difference to approximate the matrix-vector product. such as

A*v = (R(q+epsilon*v) - R(q-epsilon*v))/(2*epsilon).

q is my base flow variable and is held constant. epsilon is a variable and its values changes with "v". The way it changes follows the petsc user guide. From your explanation, this seems to be causing the problem, as epsilon actually keeps changing and to the eigensolver, this is not a constant matrix?

Maybe I am better off using the finite different approach to assemble a sparse matrix and then give it to slepc?

Thanks,
Feng
________________________________
From: Jose E. Roman <jroman at dsic.upv.es>
Sent: 25 August 2022 9:45
To: feng wang <snailsoar at hotmail.com>
Cc: petsc-users <petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] Slepc Question, shell-matrix

Probably the explanation is the following. Your shell matrix runs an iterative procedure that computes the action of the matrix to a certain accuracy. This is equivalent to multiplying by the exact matrix plus a perturbation, A+E. But every time you apply the shell matrix, the perturbation E is slightly different, so the eigensolver does not really see a constant matrix, it changes at every iteration.

This also happens in nested iterations, such as inexact shift-and-invert, where e.g. GMRES is run within each iteration of the eigensolver. For this to work, the GMRES tolerance should be smaller than the tolerance of the eigensolver. Similarly, if your shell matrix runs an iterative procedure, you have to use a tolerance that is more stringent than the one used in the eigensolver.

Hope this helps.
Jose


> El 25 ago 2022, a las 11:34, feng wang <snailsoar at hotmail.com> escribi?:
>
> Hi Jose,
>
> Thanks for your reply. I have fixed one thing in my shell matrix. Now the conversion seems working.
>
> If I use the converted dense matrix and give it to EPSSetOperators, it actually converges!
>
>  50 EPS nconv=0 first unconverged value (error) 2.89521e-06+2.13473e-06i (7.73051721e-08)
>  51 EPS nconv=0 first unconverged value (error) 2.89521e-06+2.13473e-06i (5.42681175e-08)
>  52 EPS nconv=0 first unconverged value (error) 2.89521e-06+2.13473e-06i (2.48352708e-08)
>  53 EPS nconv=0 first unconverged value (error) 2.89521e-06+2.13473e-06i (1.76912430e-08)
>  54 EPS nconv=0 first unconverged value (error) 2.89521e-06+2.13473e-06i (2.07480734e-08)
>  55 EPS nconv=0 first unconverged value (error) 2.89521e-06+2.13473e-06i (1.13588981e-08)
>  56 EPS nconv=2 first unconverged value (error) 1.10143e-06+9.80005e-06i (8.32889697e-06)
>  Solution method: krylovschur
>
>  Number of requested eigenvalues: 1
>  Linear eigensolve converged (2 eigenpairs) due to CONVERGED_TOL; iterations 56
>  ---------------------- --------------------
>             k             ||Ax-kx||/||kx||
>  ---------------------- --------------------
>    0.000003+0.000002i       7.90447e-09
>    0.000003-0.000002i       7.90447e-09
>  ---------------------- --------------------
>
> If I use the shell matrix directly, the relative norm is still large. This really puzzles me......
>
> Thanks,
> Feng
>
> From: Jose E. Roman <jroman at dsic.upv.es>
> Sent: 25 August 2022 7:21
> To: feng wang <snailsoar at hotmail.com>
> Cc: petsc-users <petsc-users at mcs.anl.gov>
> Subject: Re: [petsc-users] Slepc Question, shell-matrix
>
> This works for simple examples such as ex9.c
> It may be an indication that there is a problem with your shell matrix.
>
> Jose
>
>
> > El 24 ago 2022, a las 19:35, feng wang <snailsoar at hotmail.com> escribi?:
> >
> > Hi Jose,
> >
> > Thanks for your reply.
> >
> > I have tried PetscCall(MatConvert(slepc_A_mf,MATDENSE,MAT_INITIAL_MATRIX,&Adense));  and If I do MatView(Adense,viewer), All I see was NAN. Have I missed anything here?
> >
> > Thanks,
> > Feng
> >
> > From: Jose E. Roman <jroman at dsic.upv.es>
> > Sent: 24 August 2022 16:23
> > To: feng wang <snailsoar at hotmail.com>
> > Cc: petsc-users <petsc-users at mcs.anl.gov>
> > Subject: Re: [petsc-users] Slepc Question, shell-matrix
> >
> >
> >
> > > El 24 ago 2022, a las 17:03, feng wang <snailsoar at hotmail.com> escribi?:
> > >
> > > Hi Jose,
> > >
> > > Thanks for your reply.
> > >
> > > I have tried -eps_view_mat0 binary:amatrix.bin to save my shell matrix. It seems not saving the shell matrix and just created an empty file.
> >
> > Oops, I thought the shell matrix would be converted automatically. Try converting it yourself:
> >
> > PetscCall(MatConvert(slepc_A_mf,MATDENSE,MAT_INITIAL_MATRIX,&Adense));
> >
> > Then use Adense in EPSSetOperators(). Or save it to disk and send it to me so that I can give it a try.
> >
> > >
> > > Besides, I am trying to understand the output of slepc.  I can set  -eps_tol to a low value, but the final relative residual norm is still high , what does this tell me?
> >
> > I don't know. It should not happen.
> > Jose
> >
> > >
> > > Best regards,
> > > Feng
> > >
> > > From: Jose E. Roman <jroman at dsic.upv.es>
> > > Sent: 23 August 2022 11:06
> > > To: feng wang <snailsoar at hotmail.com>
> > > Cc: petsc-users <petsc-users at mcs.anl.gov>
> > > Subject: Re: [petsc-users] Slepc Question, shell-matrix
> > >
> > > You can try the following. Save your shell matrix to disk with the option
> > > -eps_view_mat0 binary:amatrix.bin
> > > then repeat the computation with ex4.c loading this file.
> > >
> > > Note that this should be done for a small problem size, because the conversion from a shell matrix implies a matrix-vector product per each column.
> > >
> > > Jose
> > >
> > > > El 23 ago 2022, a las 12:57, feng wang <snailsoar at hotmail.com> escribi?:
> > > >
> > > > Hi Jose,
> > > >
> > > > Thanks for your reply.
> > > >
> > > > It represents a linear operator. In my shell matrix, I am computing the non-linear residuals twice with perturbed flow variables. The matrix-vector product is computed as:
> > > >
> > > > A*v = (R(q+eps*v) - R(q-eps*v))/(2*eps)
> > > >
> > > > R is the non-linear residual. q is my flow variable and it does not change. eps is the perturbation. A is my Jacobian matrix. Besides, for some background, I am computing a steady RANS flow with finite volume method and trying to do a global stability analysis by looking at the Jacobian matrix.
> > > >
> > > > Thanks,
> > > > Feng
> > > >
> > > > From: Jose E. Roman <jroman at dsic.upv.es>
> > > > Sent: 23 August 2022 10:32
> > > > To: feng wang <snailsoar at hotmail.com>
> > > > Cc: petsc-users <petsc-users at mcs.anl.gov>
> > > > Subject: Re: [petsc-users] Slepc Question, shell-matrix
> > > >
> > > > The relative residual norms that are printed at the end are too large. For NHEP problems, they should be below the tolerance. Don't know what is happening. Does your shell matrix represent a linear (constant) operator? Or does it change slightly depending on the input vector?
> > > >
> > > > > El 23 ago 2022, a las 12:14, feng wang <snailsoar at hotmail.com> escribi?:
> > > > >
> > > > > Hi Jose,
> > > > >
> > > > > I think the previous problem comes from my side. I have some uninitialized values in my part of code to compute the non-linear residuals. so, it produces a NAN when it tries to compute the matrix-vector product using finite difference.  This might make the slepc/pestc do unexpected things.
> > > > >
> > > > > Now It seems I've got slepc running. eps_nev is set to 3 and I am trying to compute the ones with the largest amplitudes. Below is the slepc output.
> > > > >
> > > > >  14 EPS converged value (error) #0 -0.000164282 (5.36813206e-09)
> > > > >  16 EPS converged value (error) #1 -0.000160691+2.17113e-05i (3.37429620e-09)
> > > > >  16 EPS converged value (error) #2 -0.000160691-2.17113e-05i (3.37429620e-09)
> > > > >  Solution method: krylovschur
> > > > >
> > > > >  Number of requested eigenvalues: 2
> > > > >  Linear eigensolve converged (3 eigenpairs) due to CONVERGED_TOL; iterations 16
> > > > >  ---------------------- --------------------
> > > > >             k             ||Ax-kx||/||kx||
> > > > >  ---------------------- --------------------
> > > > >        -0.000164              0.0613788
> > > > >   -0.000161+0.000022i         0.0773339
> > > > >   -0.000161-0.000022i         0.0774536
> > > > >  ---------------------- --------------------
> > > > >
> > > > > The values in the brackets are the absolute error (I believe) and they seem very low. The relative error seems quite large.  Could you please comment on this?
> > > > >
> > > > >
> > > > > Best regards,
> > > > > Feng
> > > > >
> > > > > From: Jose E. Roman <jroman at dsic.upv.es>
> > > > > Sent: 23 August 2022 5:24
> > > > > To: feng wang <snailsoar at hotmail.com>
> > > > > Cc: petsc-users <petsc-users at mcs.anl.gov>
> > > > > Subject: Re: [petsc-users] Slepc Question, shell-matrix
> > > > >
> > > > > Please always respond to the list, otherwise the thread appears as unresolved in the archives of the mailing list.
> > > > >
> > > > >
> > > > > > El 22 ago 2022, a las 22:45, feng wang <snailsoar at hotmail.com> escribi?:
> > > > > >
> > > > > > Hi Jose,
> > > > > >
> > > > > > I think I might have solved my problem. I have some uninitialized values in my part of code to compute the right hand side. so it produces a NAN when it tries to compute the matrix-vector product.
> > > > > >
> > > > > > Many thanks for your help!
> > > > > >
> > > > > > Best regards,
> > > > > > Feng
> > > > > > From: Jose E. Roman <jroman at dsic.upv.es>
> > > > > > Sent: 22 August 2022 19:32
> > > > > > To: feng wang <snailsoar at hotmail.com>
> > > > > > Cc: petsc-users at mcs.anl.gov <petsc-users at mcs.anl.gov>
> > > > > > Subject: Re: [petsc-users] Slepc Question, shell-matrix
> > > > > >
> > > > > > This is very strange. This error appears when the solver employs the B-inner product, but in your case you don't have a B matrix, so you should never see that error. Try running under valgrind to see if it gives more hints.
> > > > > >
> > > > > > Jose
> > > > > >
> > > > > >
> > > > > > > El 22 ago 2022, a las 20:45, feng wang <snailsoar at hotmail.com> escribi?:
> > > > > > >
> > > > > > > Hello,
> > > > > > >
> > > > > > > I am new to Slepc and trying to work out the eigenvalues and eigenvectors of my Jacobian matrix. I am using a shell matrix to work out the matrix-vector product and I am using the default Krylov-schur method.
> > > > > > >
> > > > > > > My first attempt was not successful and I got the following errors:
> > > > > > >
> > > > > > > [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> > > > > > > [0]PETSC ERROR: Missing or incorrect user input
> > > > > > > [0]PETSC ERROR: The inner product is not well defined: indefinite matrix
> > > > > > > [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
> > > > > > > [0]PETSC ERROR: Petsc Release Version 3.17.4, unknown
> > > > > > > [0]PETSC ERROR: cfdtest on a arch-debug named ming by feng Mon Aug 22 19:21:41 2022
> > > > > > > [0]PETSC ERROR: Configure options --with-cc=mpicc --with-cxx=mpicxx --with-fc=0 PETSC_ARCH=arch-debug
> > > > > > > [0]PETSC ERROR: #1 BV_SafeSqrt() at /home/feng/cfd/slepc-3.17.1/include/slepc/private/bvimpl.h:130
> > > > > > > [0]PETSC ERROR: #2 BV_SquareRoot_Default() at /home/feng/cfd/slepc-3.17.1/include/slepc/private/bvimpl.h:365
> > > > > > > [0]PETSC ERROR: #3 BVOrthogonalizeCGS1() at /home/feng/cfd/slepc-3.17.1/src/sys/classes/bv/interface/bvorthog.c:101
> > > > > > > [0]PETSC ERROR: #4 BVOrthogonalizeGS() at /home/feng/cfd/slepc-3.17.1/src/sys/classes/bv/interface/bvorthog.c:177
> > > > > > > [0]PETSC ERROR: #5 BVOrthonormalizeColumn() at /home/feng/cfd/slepc-3.17.1/src/sys/classes/bv/interface/bvorthog.c:402
> > > > > > > [0]PETSC ERROR: #6 BVMatArnoldi() at /home/feng/cfd/slepc-3.17.1/src/sys/classes/bv/interface/bvkrylov.c:91
> > > > > > > [0]PETSC ERROR: #7 EPSSolve_KrylovSchur_Default() at /home/feng/cfd/slepc-3.17.1/src/eps/impls/krylov/krylovschur/krylovschur.c:261
> > > > > > > [0]PETSC ERROR: #8 EPSSolve() at /home/feng/cfd/slepc-3.17.1/src/eps/interface/epssolve.c:147
> > > > > > > [0]PETSC ERROR: #9 slepc_eigen_comp() at domain/cfd/slepc_eigen_solve.cpp:77
> > > > > > >
> > > > > > > Could someone please shine some light on this? I have also attached my code. The code is part of my big library and I cannot attach the whole code, sorry about this. but I am happy to provide more information. The attached code has some arrangements for halo exchange, but for the moment, it assumes it is a serial run.
> > > > > > >
> > > > > > > Many thanks,
> > > > > > > Feng
> > > > > > > <slepc_eigen_solve.cpp>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220825/25c400ba/attachment-0001.html>

From lbgpublic at gmail.com  Thu Aug 25 05:50:24 2022
From: lbgpublic at gmail.com (byron)
Date: Thu, 25 Aug 2022 11:50:24 +0100
Subject: [petsc-users] unable to build petsc with cuda / superlu-dist enabled
Message-ID: <CAJ+YQyrctNbkcdbUD-DmzdQRzUx8w12yy2dBbuntLm4Awj6McQ@mail.gmail.com>

Hi All

I am having trouble getting petsc to build with cuda / superlu-dist
enabled.  I have been trying to build it as part of another software tool,
moose, hence the choice of options being passed to configure

python3 ./configure --download-hypre=1 \
--with-shared-libraries=1 \
--download-hdf5=1 \
--download-hdf5-fortran-bindings=0 \
--with-debugging=no \
--download-fblaslapack=1 \
--download-metis=1 \
--download-ptscotch=1 \
--download-parmetis=1 \
--download-superlu_dist=1 \
--download-mumps=1 \
--download-strumpack=1 \
--download-scalapack=1 \
--download-slepc=1 \
--with-mpi=1 \
--with-openmp=1 \
--with-cxx-dialect=C++11 \
--with-fortran-bindings=0 \
--with-sowing=0 \
--with-64-bit-indices \
--with-cuda \
--prefix=/home/blah/blah

it fails with

=============================================================================================

                                Installing PTScotch; this may take several
minutes

 =============================================================================================


=============================================================================================

                                Trying to download git://
https://github.com/xiaoyeli/superlu_dist for SUPERLU_DIST


 =============================================================================================


=============================================================================================

                                Configuring SUPERLU_DIST with cmake; this
may take several minutes

 =============================================================================================


=============================================================================================

                                Compiling and installing SUPERLU_DIST; this
may take several minutes

 =============================================================================================


*******************************************************************************
         UNABLE to CONFIGURE with GIVEN OPTIONS    (see configure.log for
details):
-------------------------------------------------------------------------------
Error running make on  SUPERLU_DIST
*******************************************************************************

I've attached the configure.log.

I've tried a few different things but nothing that resulted in a
successfull build.  The most promising was I tried to build superlu_dist
separately with the configure options that petsc passes it and then kept
trying the build over and over droping out different configure options
until I got it to work.  In that case removing a "-std=c++14" entry
appeared to fix the problem but then when I made the same change in how
petsc builds superlu-dist the make failed on superlu-dist with a different
error.  I get the feeling I'm dealing with symptoms of the problem rather
than the cause.

Thanks
Byron
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220825/8418a2a1/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: configure.log
Type: application/octet-stream
Size: 2545023 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220825/8418a2a1/attachment-0001.obj>

From bsmith at petsc.dev  Thu Aug 25 07:44:56 2022
From: bsmith at petsc.dev (Barry Smith)
Date: Thu, 25 Aug 2022 08:44:56 -0400
Subject: [petsc-users] unable to build petsc with cuda / superlu-dist
 enabled
In-Reply-To: <CAJ+YQyrctNbkcdbUD-DmzdQRzUx8w12yy2dBbuntLm4Awj6McQ@mail.gmail.com>
References: <CAJ+YQyrctNbkcdbUD-DmzdQRzUx8w12yy2dBbuntLm4Awj6McQ@mail.gmail.com>
Message-ID: <C4B564E7-0DD0-4FAC-A9F3-185F0229653F@petsc.dev>


nvcc fatal   : redefinition of argument 'optimize'
make[2]: *** [SRC/CMakeFiles/superlu_dist.dir/build.make:499: SRC/CMakeFiles/superlu_dist.dir/superlu_gpu_utils.cu.o] Error 1
make[2]: *** Waiting for unfinished jobs....
nvcc fatal   : redefinition of argument 'optimize'
make[2]: *** [SRC/CMakeFiles/superlu_dist-static.dir/build.make:499: SRC/CMakeFiles/superlu_dist-static.dir/superlu_gpu_utils.cu.o] Error 1
make[2]: *** Waiting for unfinished jobs....

It seems that nvcc is unhappy with something in superlu_gpu_utils.cu  Could you check that file for any "redefinition of optimize"? Unfortunately the nvcc compiler isn't kind enough to print the line number of the redefinition or that information is lost somehow in the build process.

Barry

> On Aug 25, 2022, at 6:50 AM, byron <lbgpublic at gmail.com> wrote:
> 
> Hi All
> 
> I am having trouble getting petsc to build with cuda / superlu-dist enabled.  I have been trying to build it as part of another software tool, moose, hence the choice of options being passed to configure
> 
> python3 ./configure --download-hypre=1 \
> --with-shared-libraries=1 \
> --download-hdf5=1 \
> --download-hdf5-fortran-bindings=0 \
> --with-debugging=no \
> --download-fblaslapack=1 \
> --download-metis=1 \
> --download-ptscotch=1 \
> --download-parmetis=1 \
> --download-superlu_dist=1 \
> --download-mumps=1 \
> --download-strumpack=1 \
> --download-scalapack=1 \
> --download-slepc=1 \
> --with-mpi=1 \
> --with-openmp=1 \
> --with-cxx-dialect=C++11 \
> --with-fortran-bindings=0 \
> --with-sowing=0 \
> --with-64-bit-indices \
> --with-cuda \
> --prefix=/home/blah/blah
> 
> it fails with 
> 
> =============================================================================================                                                                                                             Installing PTScotch; this may take several minutes                                                                                                                                            =============================================================================================                                                                                                       =============================================================================================                                                                                                             Trying to download git://https://github.com/xiaoyeli/superlu_dist <https://github.com/xiaoyeli/superlu_dist> for SUPERLU_DIST                                                                                                            =============================================================================================                                                                                                       =============================================================================================                                                                                                             Configuring SUPERLU_DIST with cmake; this may take several minutes                                                                                                                            =============================================================================================                                                                                                       =============================================================================================                                                                                                             Compiling and installing SUPERLU_DIST; this may take several minutes                                                                                                                          =============================================================================================                                                                                                                             *******************************************************************************
>          UNABLE to CONFIGURE with GIVEN OPTIONS    (see configure.log for details):
> -------------------------------------------------------------------------------
> Error running make on  SUPERLU_DIST
> *******************************************************************************
> 
> I've attached the configure.log.
> 
> I've tried a few different things but nothing that resulted in a successfull build.  The most promising was I tried to build superlu_dist separately with the configure options that petsc passes it and then kept trying the build over and over droping out different configure options until I got it to work.  In that case removing a "-std=c++14" entry appeared to fix the problem but then when I made the same change in how petsc builds superlu-dist the make failed on superlu-dist with a different error.  I get the feeling I'm dealing with symptoms of the problem rather than the cause.  
> 
> Thanks
> Byron
>           
> 
> 
>  
> <configure.log>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220825/0e9f9980/attachment.html>

From lbgpublic at gmail.com  Thu Aug 25 08:38:22 2022
From: lbgpublic at gmail.com (byron)
Date: Thu, 25 Aug 2022 14:38:22 +0100
Subject: [petsc-users] unable to build petsc with cuda / superlu-dist
 enabled
In-Reply-To: <C4B564E7-0DD0-4FAC-A9F3-185F0229653F@petsc.dev>
References: <CAJ+YQyrctNbkcdbUD-DmzdQRzUx8w12yy2dBbuntLm4Awj6McQ@mail.gmail.com>
	<C4B564E7-0DD0-4FAC-A9F3-185F0229653F@petsc.dev>
Message-ID: <CAJ+YQyr9uhyR0LvCsuDwcGEhDXb1k5kSKNgP4Z+77jf97Pmfhg@mail.gmail.com>

Hi Barry

There doesn't appear to be any reference to 'optimize' in the only copy of
superlu_gpu_utils.cu I could find, I've attached a copy.

Thanks



On Thu, Aug 25, 2022 at 1:44 PM Barry Smith <bsmith at petsc.dev> wrote:

>
> nvcc fatal   : redefinition of argument 'optimize'
> make[2]: *** [SRC/CMakeFiles/superlu_dist.dir/build.make:499:
> SRC/CMakeFiles/superlu_dist.dir/superlu_gpu_utils.cu.o] Error 1
> make[2]: *** Waiting for unfinished jobs....
> nvcc fatal   : redefinition of argument 'optimize'
> make[2]: *** [SRC/CMakeFiles/superlu_dist-static.dir/build.make:499:
> SRC/CMakeFiles/superlu_dist-static.dir/superlu_gpu_utils.cu.o] Error 1
> make[2]: *** Waiting for unfinished jobs....
>
> It seems that nvcc is unhappy with something in superlu_gpu_utils.cu
>  Could you check that file for any "redefinition of optimize"?
> Unfortunately the nvcc compiler isn't kind enough to print the line number
> of the redefinition or that information is lost somehow in the build
> process.
>
> Barry
>
> On Aug 25, 2022, at 6:50 AM, byron <lbgpublic at gmail.com> wrote:
>
> Hi All
>
> I am having trouble getting petsc to build with cuda / superlu-dist
> enabled.  I have been trying to build it as part of another software tool,
> moose, hence the choice of options being passed to configure
>
> python3 ./configure --download-hypre=1 \
> --with-shared-libraries=1 \
> --download-hdf5=1 \
> --download-hdf5-fortran-bindings=0 \
> --with-debugging=no \
> --download-fblaslapack=1 \
> --download-metis=1 \
> --download-ptscotch=1 \
> --download-parmetis=1 \
> --download-superlu_dist=1 \
> --download-mumps=1 \
> --download-strumpack=1 \
> --download-scalapack=1 \
> --download-slepc=1 \
> --with-mpi=1 \
> --with-openmp=1 \
> --with-cxx-dialect=C++11 \
> --with-fortran-bindings=0 \
> --with-sowing=0 \
> --with-64-bit-indices \
> --with-cuda \
> --prefix=/home/blah/blah
>
> it fails with
>
> =============================================================================================
>
>                                 Installing PTScotch; this may take several
> minutes
>
>  =============================================================================================
>
>
> =============================================================================================
>
>                                 Trying to download git://
> https://github.com/xiaoyeli/superlu_dist for SUPERLU_DIST
>
>
>  =============================================================================================
>
>
> =============================================================================================
>
>                                 Configuring SUPERLU_DIST with cmake; this
> may take several minutes
>
>  =============================================================================================
>
>
> =============================================================================================
>
>                                 Compiling and installing SUPERLU_DIST; this
> may take several minutes
>
>  =============================================================================================
>
>
> *******************************************************************************
>          UNABLE to CONFIGURE with GIVEN OPTIONS    (see configure.log for
> details):
>
> -------------------------------------------------------------------------------
> Error running make on  SUPERLU_DIST
>
> *******************************************************************************
>
> I've attached the configure.log.
>
> I've tried a few different things but nothing that resulted in a
> successfull build.  The most promising was I tried to build superlu_dist
> separately with the configure options that petsc passes it and then kept
> trying the build over and over droping out different configure options
> until I got it to work.  In that case removing a "-std=c++14" entry
> appeared to fix the problem but then when I made the same change in how
> petsc builds superlu-dist the make failed on superlu-dist with a different
> error.  I get the feeling I'm dealing with symptoms of the problem rather
> than the cause.
>
> Thanks
> Byron
>
>
>
>
> <configure.log>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220825/5b80d419/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: superlu_gpu_utils.cu
Type: application/octet-stream
Size: 2166 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220825/5b80d419/attachment-0001.obj>

From bsmith at petsc.dev  Thu Aug 25 10:05:48 2022
From: bsmith at petsc.dev (Barry Smith)
Date: Thu, 25 Aug 2022 11:05:48 -0400
Subject: [petsc-users] unable to build petsc with cuda / superlu-dist
 enabled
In-Reply-To: <CAJ+YQyr9uhyR0LvCsuDwcGEhDXb1k5kSKNgP4Z+77jf97Pmfhg@mail.gmail.com>
References: <CAJ+YQyrctNbkcdbUD-DmzdQRzUx8w12yy2dBbuntLm4Awj6McQ@mail.gmail.com>
	<C4B564E7-0DD0-4FAC-A9F3-185F0229653F@petsc.dev>
	<CAJ+YQyr9uhyR0LvCsuDwcGEhDXb1k5kSKNgP4Z+77jf97Pmfhg@mail.gmail.com>
Message-ID: <2616A45C-68CF-4C94-A880-B7858C9B7882@petsc.dev>


  Oh yes, my mistake I misinterpreted the error message. It is actually the nvcc compiler complaining about multiple command line arguments related to optimization (which it is really picky about even if the two arguments are identical.)  I reformatted some of the configure output to show this

-ccbin mpicxx -std=c++14 -Xcompiler -fPIC -O3 -gencode arch=compute_75,code=sm_75  -Wno-deprecated-gpu-targets
-I/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi -I/usr/lib/x86_64-linux-gnu/openmpi/include
-DDEBUGlevel=0 -DPRNTlevel=0

the next fragment is 

-O3 -DNDEBUG

--generate-code=arch=compute_75,code=[compute_75,sm_75]

-Xcompiler=-fPIC -std=c++11 -x cu -rdc=true -c /home/blahblah/Projects/220824-petsc-example-for_forum/petsc/arch-linux-c-opt/externalpackages/git.superlu_dist/SRC/superlu_gpu_utils.cu -o

Hence the -O3 appears twice, once from the CMAKE argument PETSc passes to superlu_dist 

-DCMAKE_CUDA_FLAGS=" -ccbin mpicxx -std=c++14 -Xcompiler -fPIC -O3 -gencode arch=compute_75,code=sm_75  -Wno-deprecated-gpu-targets -I/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi -I/usr/lib/x86_64-linux-gnu/openmpi/include   -DDEBUGlevel=0 -DPRNTlevel=0"

and then presumably, the superlu_dist cmake process adds its own arguments which overlap.

If I remember correctly this is fixed in the main branch of PETSc, could you do an identical build with the main branch? If that works we will just need to figure out how to get MOOSE to work with main PETSc branch.

  Barry

Sadly each one makes different -std=c++11 vs  -std=c++14 choices as well.


--------------------

[ 15%] Building CUDA object SRC/CMakeFiles/superlu_dist.dir/superlu_gpu_utils.cu.o
cd /home/blahblah/Projects/220824-petsc-example-for_forum/petsc/arch-linux-c-opt/externalpackages/git.superlu_dist/petsc-build/SRC && /usr/bin/nvcc

-DSUPERLU_DIST_EXPORTS -Dsuperlu_dist_EXPORTS

-I/home/blahblah/Projects/220824-petsc-example-for_forum/petsc/arch-linux-c-opt/externalpackages/git.superlu_dist/petsc-build/SRC

-I/home/blahblah/Projects/220824-petsc-build/installation/include -I/usr/lib/x86_64-linux-gnu/openmpi/lib

-DUSE_VENDOR_BLAS  -ccbin mpicxx -std=c++14 -Xcompiler -fPIC -O3 -gencode arch=compute_75,code=sm_75  -Wno-deprecated-gpu-targets

-I/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi -I/usr/lib/x86_64-linux-gnu/openmpi/include

-DDEBUGlevel=0 -DPRNTlevel=0 -O3 -DNDEBUG

--generate-code=arch=compute_75,code=[compute_75,sm_75]

-Xcompiler=-fPIC -std=c++11 -x cu -rdc=true -c /home/blahblah/Projects/220824-petsc-example-for_forum/petsc/arch-linux-c-opt/externalpackages/git.superlu_dist/SRC/superlu_gpu_utils.cu -o

CMakeFiles/superlu_dist.dir/superlu_gpu_utils.cu



> On Aug 25, 2022, at 9:38 AM, byron <lbgpublic at gmail.com> wrote:
> 
> Hi Barry 
> 
> There doesn't appear to be any reference to 'optimize' in the only copy of  superlu_gpu_utils.cu <http://superlu_gpu_utils.cu/> I could find, I've attached a copy.
> 
> Thanks
> 
> 
> 
> On Thu, Aug 25, 2022 at 1:44 PM Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>> wrote:
> 
> nvcc fatal   : redefinition of argument 'optimize'
> make[2]: *** [SRC/CMakeFiles/superlu_dist.dir/build.make:499: SRC/CMakeFiles/superlu_dist.dir/superlu_gpu_utils.cu.o] Error 1
> make[2]: *** Waiting for unfinished jobs....
> nvcc fatal   : redefinition of argument 'optimize'
> make[2]: *** [SRC/CMakeFiles/superlu_dist-static.dir/build.make:499: SRC/CMakeFiles/superlu_dist-static.dir/superlu_gpu_utils.cu.o] Error 1
> make[2]: *** Waiting for unfinished jobs....
> 
> It seems that nvcc is unhappy with something in superlu_gpu_utils.cu <http://superlu_gpu_utils.cu/>  Could you check that file for any "redefinition of optimize"? Unfortunately the nvcc compiler isn't kind enough to print the line number of the redefinition or that information is lost somehow in the build process.
> 
> Barry
> 
>> On Aug 25, 2022, at 6:50 AM, byron <lbgpublic at gmail.com <mailto:lbgpublic at gmail.com>> wrote:
>> 
>> Hi All
>> 
>> I am having trouble getting petsc to build with cuda / superlu-dist enabled.  I have been trying to build it as part of another software tool, moose, hence the choice of options being passed to configure
>> 
>> python3 ./configure --download-hypre=1 \
>> --with-shared-libraries=1 \
>> --download-hdf5=1 \
>> --download-hdf5-fortran-bindings=0 \
>> --with-debugging=no \
>> --download-fblaslapack=1 \
>> --download-metis=1 \
>> --download-ptscotch=1 \
>> --download-parmetis=1 \
>> --download-superlu_dist=1 \
>> --download-mumps=1 \
>> --download-strumpack=1 \
>> --download-scalapack=1 \
>> --download-slepc=1 \
>> --with-mpi=1 \
>> --with-openmp=1 \
>> --with-cxx-dialect=C++11 \
>> --with-fortran-bindings=0 \
>> --with-sowing=0 \
>> --with-64-bit-indices \
>> --with-cuda \
>> --prefix=/home/blah/blah
>> 
>> it fails with 
>> 
>> =============================================================================================                                                                                                             Installing PTScotch; this may take several minutes                                                                                                                                            =============================================================================================                                                                                                       =============================================================================================                                                                                                             Trying to download git://https://github.com/xiaoyeli/superlu_dist <https://github.com/xiaoyeli/superlu_dist> for SUPERLU_DIST                                                                                                            =============================================================================================                                                                                                       =============================================================================================                                                                                                             Configuring SUPERLU_DIST with cmake; this may take several minutes                                                                                                                            =============================================================================================                                                                                                       =============================================================================================                                                                                                             Compiling and installing SUPERLU_DIST; this may take several minutes                                                                                                                          =============================================================================================                                                                                                                             *******************************************************************************
>>          UNABLE to CONFIGURE with GIVEN OPTIONS    (see configure.log for details):
>> -------------------------------------------------------------------------------
>> Error running make on  SUPERLU_DIST
>> *******************************************************************************
>> 
>> I've attached the configure.log.
>> 
>> I've tried a few different things but nothing that resulted in a successfull build.  The most promising was I tried to build superlu_dist separately with the configure options that petsc passes it and then kept trying the build over and over droping out different configure options until I got it to work.  In that case removing a "-std=c++14" entry appeared to fix the problem but then when I made the same change in how petsc builds superlu-dist the make failed on superlu-dist with a different error.  I get the feeling I'm dealing with symptoms of the problem rather than the cause.  
>> 
>> Thanks
>> Byron
>>           
>> 
>> 
>>  
>> <configure.log>
> 
> <superlu_gpu_utils.cu>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220825/b3b77143/attachment.html>

From jacob.fai at gmail.com  Thu Aug 25 10:15:16 2022
From: jacob.fai at gmail.com (Jacob Faibussowitsch)
Date: Thu, 25 Aug 2022 11:15:16 -0400
Subject: [petsc-users] unable to build petsc with cuda / superlu-dist
 enabled
In-Reply-To: <2616A45C-68CF-4C94-A880-B7858C9B7882@petsc.dev>
References: <CAJ+YQyrctNbkcdbUD-DmzdQRzUx8w12yy2dBbuntLm4Awj6McQ@mail.gmail.com>
	<C4B564E7-0DD0-4FAC-A9F3-185F0229653F@petsc.dev>
	<CAJ+YQyr9uhyR0LvCsuDwcGEhDXb1k5kSKNgP4Z+77jf97Pmfhg@mail.gmail.com>
	<2616A45C-68CF-4C94-A880-B7858C9B7882@petsc.dev>
Message-ID: <6BA4C5E3-29A4-4E68-A872-AE6A06185D64@gmail.com>

> Sadly each one makes different -std=c++11 vs  -std=c++14 choices as well.

Main should hopefully *also* warn about this, but unless you really really want -std=c++11 or -std=c++14, it is better to leave out this configure flag (--with-cxx-dialect). 

If --with-cxx-dialect is unspecified (or set to ?auto?) configure will pick the most appropriate flag for you, taking your package environment into account when choosing. If you specify the flag configure must assume the user knows something it doesn?t and will blindly take what is given.

Best regards,

Jacob Faibussowitsch
(Jacob Fai - booss - oh - vitch)

> On Aug 25, 2022, at 11:05, Barry Smith <bsmith at petsc.dev> wrote:
> 
> 
>   Oh yes, my mistake I misinterpreted the error message. It is actually the nvcc compiler complaining about multiple command line arguments related to optimization (which it is really picky about even if the two arguments are identical.)  I reformatted some of the configure output to show this
> 
> -ccbin mpicxx -std=c++14 -Xcompiler -fPIC -O3 -gencode arch=compute_75,code=sm_75  -Wno-deprecated-gpu-targets
> -I/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi -I/usr/lib/x86_64-linux-gnu/openmpi/include
> -DDEBUGlevel=0 -DPRNTlevel=0
> 
> the next fragment is 
> 
> -O3 -DNDEBUG
> 
> --generate-code=arch=compute_75,code=[compute_75,sm_75]
> 
> -Xcompiler=-fPIC -std=c++11 -x cu -rdc=true -c /home/blahblah/Projects/220824-petsc-example-for_forum/petsc/arch-linux-c-opt/externalpackages/git.superlu_dist/SRC/superlu_gpu_utils.cu -o
> 
> Hence the -O3 appears twice, once from the CMAKE argument PETSc passes to superlu_dist 
> 
> -DCMAKE_CUDA_FLAGS=" -ccbin mpicxx -std=c++14 -Xcompiler -fPIC -O3 -gencode arch=compute_75,code=sm_75  -Wno-deprecated-gpu-targets -I/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi -I/usr/lib/x86_64-linux-gnu/openmpi/include   -DDEBUGlevel=0 -DPRNTlevel=0"
> 
> and then presumably, the superlu_dist cmake process adds its own arguments which overlap.
> 
> If I remember correctly this is fixed in the main branch of PETSc, could you do an identical build with the main branch? If that works we will just need to figure out how to get MOOSE to work with main PETSc branch.
> 
>   Barry
> 
> Sadly each one makes different -std=c++11 vs  -std=c++14 choices as well.
> 
> 
> --------------------
> 
> [ 15%] Building CUDA object SRC/CMakeFiles/superlu_dist.dir/superlu_gpu_utils.cu.o
> cd /home/blahblah/Projects/220824-petsc-example-for_forum/petsc/arch-linux-c-opt/externalpackages/git.superlu_dist/petsc-build/SRC && /usr/bin/nvcc
> 
> -DSUPERLU_DIST_EXPORTS -Dsuperlu_dist_EXPORTS
> 
> -I/home/blahblah/Projects/220824-petsc-example-for_forum/petsc/arch-linux-c-opt/externalpackages/git.superlu_dist/petsc-build/SRC
> 
> -I/home/blahblah/Projects/220824-petsc-build/installation/include -I/usr/lib/x86_64-linux-gnu/openmpi/lib
> 
> -DUSE_VENDOR_BLAS  -ccbin mpicxx -std=c++14 -Xcompiler -fPIC -O3 -gencode arch=compute_75,code=sm_75  -Wno-deprecated-gpu-targets
> 
> -I/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi -I/usr/lib/x86_64-linux-gnu/openmpi/include
> 
> -DDEBUGlevel=0 -DPRNTlevel=0 -O3 -DNDEBUG
> 
> --generate-code=arch=compute_75,code=[compute_75,sm_75]
> 
> -Xcompiler=-fPIC -std=c++11 -x cu -rdc=true -c /home/blahblah/Projects/220824-petsc-example-for_forum/petsc/arch-linux-c-opt/externalpackages/git.superlu_dist/SRC/superlu_gpu_utils.cu -o
> 
> CMakeFiles/superlu_dist.dir/superlu_gpu_utils.cu
> 
> 
> 
>> On Aug 25, 2022, at 9:38 AM, byron <lbgpublic at gmail.com> wrote:
>> 
>> Hi Barry 
>> 
>> There doesn't appear to be any reference to 'optimize' in the only copy of  superlu_gpu_utils.cu I could find, I've attached a copy.
>> 
>> Thanks
>> 
>> 
>> 
>> On Thu, Aug 25, 2022 at 1:44 PM Barry Smith <bsmith at petsc.dev> wrote:
>> 
>> nvcc fatal   : redefinition of argument 'optimize'
>> make[2]: *** [SRC/CMakeFiles/superlu_dist.dir/build.make:499: SRC/CMakeFiles/superlu_dist.dir/superlu_gpu_utils.cu.o] Error 1
>> make[2]: *** Waiting for unfinished jobs....
>> nvcc fatal   : redefinition of argument 'optimize'
>> make[2]: *** [SRC/CMakeFiles/superlu_dist-static.dir/build.make:499: SRC/CMakeFiles/superlu_dist-static.dir/superlu_gpu_utils.cu.o] Error 1
>> make[2]: *** Waiting for unfinished jobs....
>> 
>> It seems that nvcc is unhappy with something in superlu_gpu_utils.cu  Could you check that file for any "redefinition of optimize"? Unfortunately the nvcc compiler isn't kind enough to print the line number of the redefinition or that information is lost somehow in the build process.
>> 
>> Barry
>> 
>>> On Aug 25, 2022, at 6:50 AM, byron <lbgpublic at gmail.com> wrote:
>>> 
>>> Hi All
>>> 
>>> I am having trouble getting petsc to build with cuda / superlu-dist enabled.  I have been trying to build it as part of another software tool, moose, hence the choice of options being passed to configure
>>> 
>>> python3 ./configure --download-hypre=1 \
>>> --with-shared-libraries=1 \
>>> --download-hdf5=1 \
>>> --download-hdf5-fortran-bindings=0 \
>>> --with-debugging=no \
>>> --download-fblaslapack=1 \
>>> --download-metis=1 \
>>> --download-ptscotch=1 \
>>> --download-parmetis=1 \
>>> --download-superlu_dist=1 \
>>> --download-mumps=1 \
>>> --download-strumpack=1 \
>>> --download-scalapack=1 \
>>> --download-slepc=1 \
>>> --with-mpi=1 \
>>> --with-openmp=1 \
>>> --with-cxx-dialect=C++11 \
>>> --with-fortran-bindings=0 \
>>> --with-sowing=0 \
>>> --with-64-bit-indices \
>>> --with-cuda \
>>> --prefix=/home/blah/blah
>>> 
>>> it fails with 
>>> 
>>> =============================================================================================                                                                                                             Installing PTScotch; this may take several minutes                                                                                                                                            =============================================================================================                                                                                                       =============================================================================================                                                                                                             Trying to download git://https://github.com/xiaoyeli/superlu_dist for SUPERLU_DIST                                                                                                            =============================================================================================                                                                                                       =============================================================================================                                                                                                             Configuring SUPERLU_DIST with cmake; this may take several minutes                                                                                                                            =============================================================================================                                                                                                       =============================================================================================                                                                                                             Compiling and installing SUPERLU_DIST; this may take several minutes                                                                                                                          =============================================================================================                                                                                                                             *******************************************************************************
>>>          UNABLE to CONFIGURE with GIVEN OPTIONS    (see configure.log for details):
>>> -------------------------------------------------------------------------------
>>> Error running make on  SUPERLU_DIST
>>> *******************************************************************************
>>> 
>>> I've attached the configure.log.
>>> 
>>> I've tried a few different things but nothing that resulted in a successfull build.  The most promising was I tried to build superlu_dist separately with the configure options that petsc passes it and then kept trying the build over and over droping out different configure options until I got it to work.  In that case removing a "-std=c++14" entry appeared to fix the problem but then when I made the same change in how petsc builds superlu-dist the make failed on superlu-dist with a different error.  I get the feeling I'm dealing with symptoms of the problem rather than the cause.  
>>> 
>>> Thanks
>>> Byron
>>>           
>>> 
>>> 
>>>  
>>> <configure.log>
>> 
>> <superlu_gpu_utils.cu>
> 


From snailsoar at hotmail.com  Thu Aug 25 11:07:44 2022
From: snailsoar at hotmail.com (feng wang)
Date: Thu, 25 Aug 2022 16:07:44 +0000
Subject: [petsc-users] Slepc Question, shell-matrix
In-Reply-To: <EDA8B12C-5637-4F23-8745-CB44C8A2367E@dsic.upv.es>
References: <ME3PR01MB7624DF748CD2568A0BAAC5B6B9719@ME3PR01MB7624.ausprd01.prod.outlook.com>
	<C1E1C97D-7F02-4764-9732-9330A6076438@dsic.upv.es>
	<ME3PR01MB7624ADA914B555F17A490226B9719@ME3PR01MB7624.ausprd01.prod.outlook.com>
	<8B78A886-8CB5-49A9-AFED-E67C22696400@dsic.upv.es>
	<SYZPR01MB76275849E71CDD1083D79937B9709@SYZPR01MB7627.ausprd01.prod.outlook.com>
	<17F56A61-2DB1-416B-85CE-8E585051F1EB@dsic.upv.es>
	<ME3PR01MB762414A8D756C6C68BC95E92B9709@ME3PR01MB7624.ausprd01.prod.outlook.com>
	<3E25E7D6-C81A-4A75-A215-A5FCFB7F3CB6@dsic.upv.es>
	<ME3PR01MB762495A98C93E2D6EA7E30ABB9739@ME3PR01MB7624.ausprd01.prod.outlook.com>
	<6BC033FF-0A55-4564-923D-97EFDD7C7FF7@dsic.upv.es>
	<ME3PR01MB762467CDD5598EC8E2AC5A50B9739@ME3PR01MB7624.ausprd01.prod.outlook.com>
	<8E59CD67-862C-463C-B3ED-1E530ECAE333@dsic.upv.es>
	<ME3PR01MB7624F86687223C01A676BA28B9729@ME3PR01MB7624.ausprd01.prod.outlook.com>
	<EDA8B12C-5637-4F23-8745-CB44C8A2367E@dsic.upv.es>
Message-ID: <ME3PR01MB7624B60950A0AD91E26DF144B9729@ME3PR01MB7624.ausprd01.prod.outlook.com>

Hi Jose,

Following my previous email, just some update,  the epsilon actually does not change in my shell matrix to compute the finite difference Jacobian. I don't have an iterative procedure in the shell matrix. It simply computes the non-linear residuals twice with perturbed flow variables. I have attached my code. Any suggestions are welcome.

Thanks,
Feng
________________________________
From: Jose E. Roman <jroman at dsic.upv.es>
Sent: 25 August 2022 9:45
To: feng wang <snailsoar at hotmail.com>
Cc: petsc-users <petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] Slepc Question, shell-matrix

Probably the explanation is the following. Your shell matrix runs an iterative procedure that computes the action of the matrix to a certain accuracy. This is equivalent to multiplying by the exact matrix plus a perturbation, A+E. But every time you apply the shell matrix, the perturbation E is slightly different, so the eigensolver does not really see a constant matrix, it changes at every iteration.

This also happens in nested iterations, such as inexact shift-and-invert, where e.g. GMRES is run within each iteration of the eigensolver. For this to work, the GMRES tolerance should be smaller than the tolerance of the eigensolver. Similarly, if your shell matrix runs an iterative procedure, you have to use a tolerance that is more stringent than the one used in the eigensolver.

Hope this helps.
Jose


> El 25 ago 2022, a las 11:34, feng wang <snailsoar at hotmail.com> escribi?:
>
> Hi Jose,
>
> Thanks for your reply. I have fixed one thing in my shell matrix. Now the conversion seems working.
>
> If I use the converted dense matrix and give it to EPSSetOperators, it actually converges!
>
>  50 EPS nconv=0 first unconverged value (error) 2.89521e-06+2.13473e-06i (7.73051721e-08)
>  51 EPS nconv=0 first unconverged value (error) 2.89521e-06+2.13473e-06i (5.42681175e-08)
>  52 EPS nconv=0 first unconverged value (error) 2.89521e-06+2.13473e-06i (2.48352708e-08)
>  53 EPS nconv=0 first unconverged value (error) 2.89521e-06+2.13473e-06i (1.76912430e-08)
>  54 EPS nconv=0 first unconverged value (error) 2.89521e-06+2.13473e-06i (2.07480734e-08)
>  55 EPS nconv=0 first unconverged value (error) 2.89521e-06+2.13473e-06i (1.13588981e-08)
>  56 EPS nconv=2 first unconverged value (error) 1.10143e-06+9.80005e-06i (8.32889697e-06)
>  Solution method: krylovschur
>
>  Number of requested eigenvalues: 1
>  Linear eigensolve converged (2 eigenpairs) due to CONVERGED_TOL; iterations 56
>  ---------------------- --------------------
>             k             ||Ax-kx||/||kx||
>  ---------------------- --------------------
>    0.000003+0.000002i       7.90447e-09
>    0.000003-0.000002i       7.90447e-09
>  ---------------------- --------------------
>
> If I use the shell matrix directly, the relative norm is still large. This really puzzles me......
>
> Thanks,
> Feng
>
> From: Jose E. Roman <jroman at dsic.upv.es>
> Sent: 25 August 2022 7:21
> To: feng wang <snailsoar at hotmail.com>
> Cc: petsc-users <petsc-users at mcs.anl.gov>
> Subject: Re: [petsc-users] Slepc Question, shell-matrix
>
> This works for simple examples such as ex9.c
> It may be an indication that there is a problem with your shell matrix.
>
> Jose
>
>
> > El 24 ago 2022, a las 19:35, feng wang <snailsoar at hotmail.com> escribi?:
> >
> > Hi Jose,
> >
> > Thanks for your reply.
> >
> > I have tried PetscCall(MatConvert(slepc_A_mf,MATDENSE,MAT_INITIAL_MATRIX,&Adense));  and If I do MatView(Adense,viewer), All I see was NAN. Have I missed anything here?
> >
> > Thanks,
> > Feng
> >
> > From: Jose E. Roman <jroman at dsic.upv.es>
> > Sent: 24 August 2022 16:23
> > To: feng wang <snailsoar at hotmail.com>
> > Cc: petsc-users <petsc-users at mcs.anl.gov>
> > Subject: Re: [petsc-users] Slepc Question, shell-matrix
> >
> >
> >
> > > El 24 ago 2022, a las 17:03, feng wang <snailsoar at hotmail.com> escribi?:
> > >
> > > Hi Jose,
> > >
> > > Thanks for your reply.
> > >
> > > I have tried -eps_view_mat0 binary:amatrix.bin to save my shell matrix. It seems not saving the shell matrix and just created an empty file.
> >
> > Oops, I thought the shell matrix would be converted automatically. Try converting it yourself:
> >
> > PetscCall(MatConvert(slepc_A_mf,MATDENSE,MAT_INITIAL_MATRIX,&Adense));
> >
> > Then use Adense in EPSSetOperators(). Or save it to disk and send it to me so that I can give it a try.
> >
> > >
> > > Besides, I am trying to understand the output of slepc.  I can set  -eps_tol to a low value, but the final relative residual norm is still high , what does this tell me?
> >
> > I don't know. It should not happen.
> > Jose
> >
> > >
> > > Best regards,
> > > Feng
> > >
> > > From: Jose E. Roman <jroman at dsic.upv.es>
> > > Sent: 23 August 2022 11:06
> > > To: feng wang <snailsoar at hotmail.com>
> > > Cc: petsc-users <petsc-users at mcs.anl.gov>
> > > Subject: Re: [petsc-users] Slepc Question, shell-matrix
> > >
> > > You can try the following. Save your shell matrix to disk with the option
> > > -eps_view_mat0 binary:amatrix.bin
> > > then repeat the computation with ex4.c loading this file.
> > >
> > > Note that this should be done for a small problem size, because the conversion from a shell matrix implies a matrix-vector product per each column.
> > >
> > > Jose
> > >
> > > > El 23 ago 2022, a las 12:57, feng wang <snailsoar at hotmail.com> escribi?:
> > > >
> > > > Hi Jose,
> > > >
> > > > Thanks for your reply.
> > > >
> > > > It represents a linear operator. In my shell matrix, I am computing the non-linear residuals twice with perturbed flow variables. The matrix-vector product is computed as:
> > > >
> > > > A*v = (R(q+eps*v) - R(q-eps*v))/(2*eps)
> > > >
> > > > R is the non-linear residual. q is my flow variable and it does not change. eps is the perturbation. A is my Jacobian matrix. Besides, for some background, I am computing a steady RANS flow with finite volume method and trying to do a global stability analysis by looking at the Jacobian matrix.
> > > >
> > > > Thanks,
> > > > Feng
> > > >
> > > > From: Jose E. Roman <jroman at dsic.upv.es>
> > > > Sent: 23 August 2022 10:32
> > > > To: feng wang <snailsoar at hotmail.com>
> > > > Cc: petsc-users <petsc-users at mcs.anl.gov>
> > > > Subject: Re: [petsc-users] Slepc Question, shell-matrix
> > > >
> > > > The relative residual norms that are printed at the end are too large. For NHEP problems, they should be below the tolerance. Don't know what is happening. Does your shell matrix represent a linear (constant) operator? Or does it change slightly depending on the input vector?
> > > >
> > > > > El 23 ago 2022, a las 12:14, feng wang <snailsoar at hotmail.com> escribi?:
> > > > >
> > > > > Hi Jose,
> > > > >
> > > > > I think the previous problem comes from my side. I have some uninitialized values in my part of code to compute the non-linear residuals. so, it produces a NAN when it tries to compute the matrix-vector product using finite difference.  This might make the slepc/pestc do unexpected things.
> > > > >
> > > > > Now It seems I've got slepc running. eps_nev is set to 3 and I am trying to compute the ones with the largest amplitudes. Below is the slepc output.
> > > > >
> > > > >  14 EPS converged value (error) #0 -0.000164282 (5.36813206e-09)
> > > > >  16 EPS converged value (error) #1 -0.000160691+2.17113e-05i (3.37429620e-09)
> > > > >  16 EPS converged value (error) #2 -0.000160691-2.17113e-05i (3.37429620e-09)
> > > > >  Solution method: krylovschur
> > > > >
> > > > >  Number of requested eigenvalues: 2
> > > > >  Linear eigensolve converged (3 eigenpairs) due to CONVERGED_TOL; iterations 16
> > > > >  ---------------------- --------------------
> > > > >             k             ||Ax-kx||/||kx||
> > > > >  ---------------------- --------------------
> > > > >        -0.000164              0.0613788
> > > > >   -0.000161+0.000022i         0.0773339
> > > > >   -0.000161-0.000022i         0.0774536
> > > > >  ---------------------- --------------------
> > > > >
> > > > > The values in the brackets are the absolute error (I believe) and they seem very low. The relative error seems quite large.  Could you please comment on this?
> > > > >
> > > > >
> > > > > Best regards,
> > > > > Feng
> > > > >
> > > > > From: Jose E. Roman <jroman at dsic.upv.es>
> > > > > Sent: 23 August 2022 5:24
> > > > > To: feng wang <snailsoar at hotmail.com>
> > > > > Cc: petsc-users <petsc-users at mcs.anl.gov>
> > > > > Subject: Re: [petsc-users] Slepc Question, shell-matrix
> > > > >
> > > > > Please always respond to the list, otherwise the thread appears as unresolved in the archives of the mailing list.
> > > > >
> > > > >
> > > > > > El 22 ago 2022, a las 22:45, feng wang <snailsoar at hotmail.com> escribi?:
> > > > > >
> > > > > > Hi Jose,
> > > > > >
> > > > > > I think I might have solved my problem. I have some uninitialized values in my part of code to compute the right hand side. so it produces a NAN when it tries to compute the matrix-vector product.
> > > > > >
> > > > > > Many thanks for your help!
> > > > > >
> > > > > > Best regards,
> > > > > > Feng
> > > > > > From: Jose E. Roman <jroman at dsic.upv.es>
> > > > > > Sent: 22 August 2022 19:32
> > > > > > To: feng wang <snailsoar at hotmail.com>
> > > > > > Cc: petsc-users at mcs.anl.gov <petsc-users at mcs.anl.gov>
> > > > > > Subject: Re: [petsc-users] Slepc Question, shell-matrix
> > > > > >
> > > > > > This is very strange. This error appears when the solver employs the B-inner product, but in your case you don't have a B matrix, so you should never see that error. Try running under valgrind to see if it gives more hints.
> > > > > >
> > > > > > Jose
> > > > > >
> > > > > >
> > > > > > > El 22 ago 2022, a las 20:45, feng wang <snailsoar at hotmail.com> escribi?:
> > > > > > >
> > > > > > > Hello,
> > > > > > >
> > > > > > > I am new to Slepc and trying to work out the eigenvalues and eigenvectors of my Jacobian matrix. I am using a shell matrix to work out the matrix-vector product and I am using the default Krylov-schur method.
> > > > > > >
> > > > > > > My first attempt was not successful and I got the following errors:
> > > > > > >
> > > > > > > [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> > > > > > > [0]PETSC ERROR: Missing or incorrect user input
> > > > > > > [0]PETSC ERROR: The inner product is not well defined: indefinite matrix
> > > > > > > [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
> > > > > > > [0]PETSC ERROR: Petsc Release Version 3.17.4, unknown
> > > > > > > [0]PETSC ERROR: cfdtest on a arch-debug named ming by feng Mon Aug 22 19:21:41 2022
> > > > > > > [0]PETSC ERROR: Configure options --with-cc=mpicc --with-cxx=mpicxx --with-fc=0 PETSC_ARCH=arch-debug
> > > > > > > [0]PETSC ERROR: #1 BV_SafeSqrt() at /home/feng/cfd/slepc-3.17.1/include/slepc/private/bvimpl.h:130
> > > > > > > [0]PETSC ERROR: #2 BV_SquareRoot_Default() at /home/feng/cfd/slepc-3.17.1/include/slepc/private/bvimpl.h:365
> > > > > > > [0]PETSC ERROR: #3 BVOrthogonalizeCGS1() at /home/feng/cfd/slepc-3.17.1/src/sys/classes/bv/interface/bvorthog.c:101
> > > > > > > [0]PETSC ERROR: #4 BVOrthogonalizeGS() at /home/feng/cfd/slepc-3.17.1/src/sys/classes/bv/interface/bvorthog.c:177
> > > > > > > [0]PETSC ERROR: #5 BVOrthonormalizeColumn() at /home/feng/cfd/slepc-3.17.1/src/sys/classes/bv/interface/bvorthog.c:402
> > > > > > > [0]PETSC ERROR: #6 BVMatArnoldi() at /home/feng/cfd/slepc-3.17.1/src/sys/classes/bv/interface/bvkrylov.c:91
> > > > > > > [0]PETSC ERROR: #7 EPSSolve_KrylovSchur_Default() at /home/feng/cfd/slepc-3.17.1/src/eps/impls/krylov/krylovschur/krylovschur.c:261
> > > > > > > [0]PETSC ERROR: #8 EPSSolve() at /home/feng/cfd/slepc-3.17.1/src/eps/interface/epssolve.c:147
> > > > > > > [0]PETSC ERROR: #9 slepc_eigen_comp() at domain/cfd/slepc_eigen_solve.cpp:77
> > > > > > >
> > > > > > > Could someone please shine some light on this? I have also attached my code. The code is part of my big library and I cannot attach the whole code, sorry about this. but I am happy to provide more information. The attached code has some arrangements for halo exchange, but for the moment, it assumes it is a serial run.
> > > > > > >
> > > > > > > Many thanks,
> > > > > > > Feng
> > > > > > > <slepc_eigen_solve.cpp>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220825/61529866/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: slepc_eigen_solve.cpp
Type: text/x-c++src
Size: 7181 bytes
Desc: slepc_eigen_solve.cpp
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220825/61529866/attachment-0001.bin>

From jroman at dsic.upv.es  Thu Aug 25 11:31:18 2022
From: jroman at dsic.upv.es (Jose E. Roman)
Date: Thu, 25 Aug 2022 18:31:18 +0200
Subject: [petsc-users] Slepc Question, shell-matrix
In-Reply-To: <ME3PR01MB7624B60950A0AD91E26DF144B9729@ME3PR01MB7624.ausprd01.prod.outlook.com>
References: <ME3PR01MB7624DF748CD2568A0BAAC5B6B9719@ME3PR01MB7624.ausprd01.prod.outlook.com>
	<C1E1C97D-7F02-4764-9732-9330A6076438@dsic.upv.es>
	<ME3PR01MB7624ADA914B555F17A490226B9719@ME3PR01MB7624.ausprd01.prod.outlook.com>
	<8B78A886-8CB5-49A9-AFED-E67C22696400@dsic.upv.es>
	<SYZPR01MB76275849E71CDD1083D79937B9709@SYZPR01MB7627.ausprd01.prod.outlook.com>
	<17F56A61-2DB1-416B-85CE-8E585051F1EB@dsic.upv.es>
	<ME3PR01MB762414A8D756C6C68BC95E92B9709@ME3PR01MB7624.ausprd01.prod.outlook.com>
	<3E25E7D6-C81A-4A75-A215-A5FCFB7F3CB6@dsic.upv.es>
	<ME3PR01MB762495A98C93E2D6EA7E30ABB9739@ME3PR01MB7624.ausprd01.prod.outlook.com>
	<6BC033FF-0A55-4564-923D-97EFDD7C7FF7@dsic.upv.es>
	<ME3PR01MB762467CDD5598EC8E2AC5A50B9739@ME3PR01MB7624.ausprd01.prod.outlook.com>
	<8E59CD67-862C-463C-B3ED-1E530ECAE333@dsic.upv.es>
	<ME3PR01MB7624F86687223C01A676BA28B9729@ME3PR01MB7624.ausprd01.prod.outlook.com>
	<EDA8B12C-5637-4F23-8745-CB44C8A2367E@dsic.upv.es>
	<ME3PR01MB7624B60950A0AD91E26DF144B9729@ME3PR01MB7624.ausprd01.prod.outlook.com>
Message-ID: <C6B97F7E-6D5F-4237-9F1F-54AEFB61F06E@dsic.upv.es>

Your formula
(R(q+epsilon*v) - R(q-epsilon*v))/(2*epsilon).
is not linear, it cannot be expressed as a constant matrix times a vector, A*v.

If you want to solve a nonlinear eigenvector problem
A(x)*x=lambda*x
then have a look at ex34.c https://slepc.upv.es/documentation/current/src/eps/tutorials/ex34.c.html

Jose


> El 25 ago 2022, a las 18:07, feng wang <snailsoar at hotmail.com> escribi?:
> 
> Hi Jose,
> 
> Following my previous email, just some update,  the epsilon actually does not change in my shell matrix to compute the finite difference Jacobian. I don't have an iterative procedure in the shell matrix. It simply computes the non-linear residuals twice with perturbed flow variables. I have attached my code. Any suggestions are welcome.
> 
> Thanks,
> Feng
> From: Jose E. Roman <jroman at dsic.upv.es>
> Sent: 25 August 2022 9:45
> To: feng wang <snailsoar at hotmail.com>
> Cc: petsc-users <petsc-users at mcs.anl.gov>
> Subject: Re: [petsc-users] Slepc Question, shell-matrix
>  
> Probably the explanation is the following. Your shell matrix runs an iterative procedure that computes the action of the matrix to a certain accuracy. This is equivalent to multiplying by the exact matrix plus a perturbation, A+E. But every time you apply the shell matrix, the perturbation E is slightly different, so the eigensolver does not really see a constant matrix, it changes at every iteration.
> 
> This also happens in nested iterations, such as inexact shift-and-invert, where e.g. GMRES is run within each iteration of the eigensolver. For this to work, the GMRES tolerance should be smaller than the tolerance of the eigensolver. Similarly, if your shell matrix runs an iterative procedure, you have to use a tolerance that is more stringent than the one used in the eigensolver.
> 
> Hope this helps.
> Jose
> 
> 
> > El 25 ago 2022, a las 11:34, feng wang <snailsoar at hotmail.com> escribi?:
> > 
> > Hi Jose,
> > 
> > Thanks for your reply. I have fixed one thing in my shell matrix. Now the conversion seems working.
> > 
> > If I use the converted dense matrix and give it to EPSSetOperators, it actually converges!
> > 
> >  50 EPS nconv=0 first unconverged value (error) 2.89521e-06+2.13473e-06i (7.73051721e-08)
> >  51 EPS nconv=0 first unconverged value (error) 2.89521e-06+2.13473e-06i (5.42681175e-08)
> >  52 EPS nconv=0 first unconverged value (error) 2.89521e-06+2.13473e-06i (2.48352708e-08)
> >  53 EPS nconv=0 first unconverged value (error) 2.89521e-06+2.13473e-06i (1.76912430e-08)
> >  54 EPS nconv=0 first unconverged value (error) 2.89521e-06+2.13473e-06i (2.07480734e-08)
> >  55 EPS nconv=0 first unconverged value (error) 2.89521e-06+2.13473e-06i (1.13588981e-08)
> >  56 EPS nconv=2 first unconverged value (error) 1.10143e-06+9.80005e-06i (8.32889697e-06)
> >  Solution method: krylovschur
> > 
> >  Number of requested eigenvalues: 1
> >  Linear eigensolve converged (2 eigenpairs) due to CONVERGED_TOL; iterations 56
> >  ---------------------- --------------------
> >             k             ||Ax-kx||/||kx||
> >  ---------------------- --------------------
> >    0.000003+0.000002i       7.90447e-09
> >    0.000003-0.000002i       7.90447e-09
> >  ---------------------- --------------------
> > 
> > If I use the shell matrix directly, the relative norm is still large. This really puzzles me...... 
> > 
> > Thanks,
> > Feng
> > 
> > From: Jose E. Roman <jroman at dsic.upv.es>
> > Sent: 25 August 2022 7:21
> > To: feng wang <snailsoar at hotmail.com>
> > Cc: petsc-users <petsc-users at mcs.anl.gov>
> > Subject: Re: [petsc-users] Slepc Question, shell-matrix
> >  
> > This works for simple examples such as ex9.c
> > It may be an indication that there is a problem with your shell matrix.
> > 
> > Jose
> > 
> > 
> > > El 24 ago 2022, a las 19:35, feng wang <snailsoar at hotmail.com> escribi?:
> > > 
> > > Hi Jose,
> > > 
> > > Thanks for your reply.
> > > 
> > > I have tried PetscCall(MatConvert(slepc_A_mf,MATDENSE,MAT_INITIAL_MATRIX,&Adense));  and If I do MatView(Adense,viewer), All I see was NAN. Have I missed anything here?
> > > 
> > > Thanks,
> > > Feng
> > > 
> > > From: Jose E. Roman <jroman at dsic.upv.es>
> > > Sent: 24 August 2022 16:23
> > > To: feng wang <snailsoar at hotmail.com>
> > > Cc: petsc-users <petsc-users at mcs.anl.gov>
> > > Subject: Re: [petsc-users] Slepc Question, shell-matrix
> > >  
> > > 
> > > 
> > > > El 24 ago 2022, a las 17:03, feng wang <snailsoar at hotmail.com> escribi?:
> > > > 
> > > > Hi Jose,
> > > > 
> > > > Thanks for your reply.
> > > > 
> > > > I have tried -eps_view_mat0 binary:amatrix.bin to save my shell matrix. It seems not saving the shell matrix and just created an empty file.
> > > 
> > > Oops, I thought the shell matrix would be converted automatically. Try converting it yourself:
> > > 
> > > PetscCall(MatConvert(slepc_A_mf,MATDENSE,MAT_INITIAL_MATRIX,&Adense));
> > > 
> > > Then use Adense in EPSSetOperators(). Or save it to disk and send it to me so that I can give it a try.
> > > 
> > > > 
> > > > Besides, I am trying to understand the output of slepc.  I can set  -eps_tol to a low value, but the final relative residual norm is still high , what does this tell me?
> > > 
> > > I don't know. It should not happen.
> > > Jose
> > > 
> > > > 
> > > > Best regards,
> > > > Feng
> > > > 
> > > > From: Jose E. Roman <jroman at dsic.upv.es>
> > > > Sent: 23 August 2022 11:06
> > > > To: feng wang <snailsoar at hotmail.com>
> > > > Cc: petsc-users <petsc-users at mcs.anl.gov>
> > > > Subject: Re: [petsc-users] Slepc Question, shell-matrix
> > > >  
> > > > You can try the following. Save your shell matrix to disk with the option
> > > > -eps_view_mat0 binary:amatrix.bin
> > > > then repeat the computation with ex4.c loading this file.
> > > > 
> > > > Note that this should be done for a small problem size, because the conversion from a shell matrix implies a matrix-vector product per each column.
> > > > 
> > > > Jose
> > > > 
> > > > > El 23 ago 2022, a las 12:57, feng wang <snailsoar at hotmail.com> escribi?:
> > > > > 
> > > > > Hi Jose,
> > > > > 
> > > > > Thanks for your reply.
> > > > > 
> > > > > It represents a linear operator. In my shell matrix, I am computing the non-linear residuals twice with perturbed flow variables. The matrix-vector product is computed as:
> > > > > 
> > > > > A*v = (R(q+eps*v) - R(q-eps*v))/(2*eps)
> > > > > 
> > > > > R is the non-linear residual. q is my flow variable and it does not change. eps is the perturbation. A is my Jacobian matrix. Besides, for some background, I am computing a steady RANS flow with finite volume method and trying to do a global stability analysis by looking at the Jacobian matrix. 
> > > > > 
> > > > > Thanks,
> > > > > Feng
> > > > > 
> > > > > From: Jose E. Roman <jroman at dsic.upv.es>
> > > > > Sent: 23 August 2022 10:32
> > > > > To: feng wang <snailsoar at hotmail.com>
> > > > > Cc: petsc-users <petsc-users at mcs.anl.gov>
> > > > > Subject: Re: [petsc-users] Slepc Question, shell-matrix
> > > > >  
> > > > > The relative residual norms that are printed at the end are too large. For NHEP problems, they should be below the tolerance. Don't know what is happening. Does your shell matrix represent a linear (constant) operator? Or does it change slightly depending on the input vector? 
> > > > > 
> > > > > > El 23 ago 2022, a las 12:14, feng wang <snailsoar at hotmail.com> escribi?:
> > > > > > 
> > > > > > Hi Jose,
> > > > > > 
> > > > > > I think the previous problem comes from my side. I have some uninitialized values in my part of code to compute the non-linear residuals. so, it produces a NAN when it tries to compute the matrix-vector product using finite difference.  This might make the slepc/pestc do unexpected things.
> > > > > > 
> > > > > > Now It seems I've got slepc running. eps_nev is set to 3 and I am trying to compute the ones with the largest amplitudes. Below is the slepc output. 
> > > > > > 
> > > > > >  14 EPS converged value (error) #0 -0.000164282 (5.36813206e-09)
> > > > > >  16 EPS converged value (error) #1 -0.000160691+2.17113e-05i (3.37429620e-09)
> > > > > >  16 EPS converged value (error) #2 -0.000160691-2.17113e-05i (3.37429620e-09)
> > > > > >  Solution method: krylovschur
> > > > > > 
> > > > > >  Number of requested eigenvalues: 2
> > > > > >  Linear eigensolve converged (3 eigenpairs) due to CONVERGED_TOL; iterations 16
> > > > > >  ---------------------- --------------------
> > > > > >             k             ||Ax-kx||/||kx||
> > > > > >  ---------------------- --------------------
> > > > > >        -0.000164              0.0613788
> > > > > >   -0.000161+0.000022i         0.0773339
> > > > > >   -0.000161-0.000022i         0.0774536
> > > > > >  ---------------------- --------------------
> > > > > > 
> > > > > > The values in the brackets are the absolute error (I believe) and they seem very low. The relative error seems quite large.  Could you please comment on this? 
> > > > > > 
> > > > > > 
> > > > > > Best regards,
> > > > > > Feng
> > > > > > 
> > > > > > From: Jose E. Roman <jroman at dsic.upv.es>
> > > > > > Sent: 23 August 2022 5:24
> > > > > > To: feng wang <snailsoar at hotmail.com>
> > > > > > Cc: petsc-users <petsc-users at mcs.anl.gov>
> > > > > > Subject: Re: [petsc-users] Slepc Question, shell-matrix
> > > > > >  
> > > > > > Please always respond to the list, otherwise the thread appears as unresolved in the archives of the mailing list.
> > > > > > 
> > > > > > 
> > > > > > > El 22 ago 2022, a las 22:45, feng wang <snailsoar at hotmail.com> escribi?:
> > > > > > > 
> > > > > > > Hi Jose,
> > > > > > > 
> > > > > > > I think I might have solved my problem. I have some uninitialized values in my part of code to compute the right hand side. so it produces a NAN when it tries to compute the matrix-vector product. 
> > > > > > > 
> > > > > > > Many thanks for your help!
> > > > > > > 
> > > > > > > Best regards,
> > > > > > > Feng
> > > > > > > From: Jose E. Roman <jroman at dsic.upv.es>
> > > > > > > Sent: 22 August 2022 19:32
> > > > > > > To: feng wang <snailsoar at hotmail.com>
> > > > > > > Cc: petsc-users at mcs.anl.gov <petsc-users at mcs.anl.gov>
> > > > > > > Subject: Re: [petsc-users] Slepc Question, shell-matrix
> > > > > > >  
> > > > > > > This is very strange. This error appears when the solver employs the B-inner product, but in your case you don't have a B matrix, so you should never see that error. Try running under valgrind to see if it gives more hints.
> > > > > > > 
> > > > > > > Jose
> > > > > > > 
> > > > > > > 
> > > > > > > > El 22 ago 2022, a las 20:45, feng wang <snailsoar at hotmail.com> escribi?:
> > > > > > > > 
> > > > > > > > Hello,
> > > > > > > > 
> > > > > > > > I am new to Slepc and trying to work out the eigenvalues and eigenvectors of my Jacobian matrix. I am using a shell matrix to work out the matrix-vector product and I am using the default Krylov-schur method.  
> > > > > > > > 
> > > > > > > > My first attempt was not successful and I got the following errors:
> > > > > > > > 
> > > > > > > > [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> > > > > > > > [0]PETSC ERROR: Missing or incorrect user input 
> > > > > > > > [0]PETSC ERROR: The inner product is not well defined: indefinite matrix
> > > > > > > > [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
> > > > > > > > [0]PETSC ERROR: Petsc Release Version 3.17.4, unknown 
> > > > > > > > [0]PETSC ERROR: cfdtest on a arch-debug named ming by feng Mon Aug 22 19:21:41 2022
> > > > > > > > [0]PETSC ERROR: Configure options --with-cc=mpicc --with-cxx=mpicxx --with-fc=0 PETSC_ARCH=arch-debug
> > > > > > > > [0]PETSC ERROR: #1 BV_SafeSqrt() at /home/feng/cfd/slepc-3.17.1/include/slepc/private/bvimpl.h:130
> > > > > > > > [0]PETSC ERROR: #2 BV_SquareRoot_Default() at /home/feng/cfd/slepc-3.17.1/include/slepc/private/bvimpl.h:365
> > > > > > > > [0]PETSC ERROR: #3 BVOrthogonalizeCGS1() at /home/feng/cfd/slepc-3.17.1/src/sys/classes/bv/interface/bvorthog.c:101
> > > > > > > > [0]PETSC ERROR: #4 BVOrthogonalizeGS() at /home/feng/cfd/slepc-3.17.1/src/sys/classes/bv/interface/bvorthog.c:177
> > > > > > > > [0]PETSC ERROR: #5 BVOrthonormalizeColumn() at /home/feng/cfd/slepc-3.17.1/src/sys/classes/bv/interface/bvorthog.c:402
> > > > > > > > [0]PETSC ERROR: #6 BVMatArnoldi() at /home/feng/cfd/slepc-3.17.1/src/sys/classes/bv/interface/bvkrylov.c:91
> > > > > > > > [0]PETSC ERROR: #7 EPSSolve_KrylovSchur_Default() at /home/feng/cfd/slepc-3.17.1/src/eps/impls/krylov/krylovschur/krylovschur.c:261
> > > > > > > > [0]PETSC ERROR: #8 EPSSolve() at /home/feng/cfd/slepc-3.17.1/src/eps/interface/epssolve.c:147
> > > > > > > > [0]PETSC ERROR: #9 slepc_eigen_comp() at domain/cfd/slepc_eigen_solve.cpp:77
> > > > > > > > 
> > > > > > > > Could someone please shine some light on this? I have also attached my code. The code is part of my big library and I cannot attach the whole code, sorry about this. but I am happy to provide more information. The attached code has some arrangements for halo exchange, but for the moment, it assumes it is a serial run.
> > > > > > > > 
> > > > > > > > Many thanks,
> > > > > > > > Feng
> > > > > > > > <slepc_eigen_solve.cpp>
> 
> <slepc_eigen_solve.cpp>


From mi.mike1021 at gmail.com  Thu Aug 25 11:52:34 2022
From: mi.mike1021 at gmail.com (Mike Michell)
Date: Thu, 25 Aug 2022 11:52:34 -0500
Subject: [petsc-users] DMPlex in Fortran
In-Reply-To: <CAMYG4GkFHyqzduzA+HU+B2DOf35QPiZd+2eQnUuFUC6qaa462g@mail.gmail.com>
References: <CAEc7osYQoP2TuS9FU8+ZEoMOb+22TtwkHs8LOf6BShSRTZ8sKg@mail.gmail.com>
	<CAMYG4GkFHyqzduzA+HU+B2DOf35QPiZd+2eQnUuFUC6qaa462g@mail.gmail.com>
Message-ID: <CAEc7osayv+YTMSRiMw5CAbF9Utt2n+7X1wbK-RoKDhV1Jqp+NA@mail.gmail.com>

Hi, I am trying to find the function you commented,
DMPlexCreateInterpolator(), from DMPlex manual page, but I do not think
this function is visible. Any comment about this?
Thanks,


> On Fri, Jul 8, 2022 at 11:26 PM Mike Michell <mi.mike1021 at gmail.com>
> wrote:
>
>> I am using DMPlex for a code with written in Fortran in 2D and 3D. There
>> were two questions.
>>
>> - As a follow up of the previous inquiry:
>> https://www.mail-archive.com/petsc-users at mcs.anl.gov/msg43856.html
>> Is the local-to-local halo exchange available in Fortran now or still
>> pending? Currently local-to-global and global-to-local communication are
>> used since local-to-local has not been allowed for Fortran.
>>
>
> Sorry, it is still on my TODO list. I am trying to get stuff cleared out.
>
>
>> - One code solves discretized equations at each vertex, and another code
>> I want to couple physics is solving equations at each cell centroid.
>> Basically, the value at cell centroid needs to be mapped to vertex (or vice
>> versa) through interpolation/extrapolation for coupling of two codes. Does
>> petsc function provide this kind of mapping between cell centroid and
>> vertex? The grids for the two codes can be overlapped. I was trying to find
>> some FEM infrastructures in petsc, but so far havent found that kind of
>> functionality. Can I get any comments on that?
>>
>
> Yes, you can create both P0 and P1 discretizations (PetscFECreateLagrange)
> in two different DMs using DMClone(), and then create an interpolation
> operator (DMPlexCreateInterpolator) which maps between them. Let me know if
> something is not clear there.
>
>   Thanks,
>
>     Matt
>
>
>> Thanks,
>> Mike
>>
>>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220825/7ff46c7a/attachment.html>

From knepley at gmail.com  Thu Aug 25 12:03:16 2022
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 25 Aug 2022 13:03:16 -0400
Subject: [petsc-users] DMPlex in Fortran
In-Reply-To: <CAEc7osayv+YTMSRiMw5CAbF9Utt2n+7X1wbK-RoKDhV1Jqp+NA@mail.gmail.com>
References: <CAEc7osYQoP2TuS9FU8+ZEoMOb+22TtwkHs8LOf6BShSRTZ8sKg@mail.gmail.com>
	<CAMYG4GkFHyqzduzA+HU+B2DOf35QPiZd+2eQnUuFUC6qaa462g@mail.gmail.com>
	<CAEc7osayv+YTMSRiMw5CAbF9Utt2n+7X1wbK-RoKDhV1Jqp+NA@mail.gmail.com>
Message-ID: <CAMYG4GneehDK9dXyE4YPv95NUYD9x4wmt_0V_Fh50vP9-iZzZA@mail.gmail.com>

On Thu, Aug 25, 2022 at 12:52 PM Mike Michell <mi.mike1021 at gmail.com> wrote:

> Hi, I am trying to find the function you commented,
> DMPlexCreateInterpolator(), from DMPlex manual page, but I do not think
> this function is visible. Any comment about this?
>

Sorry, it is DMCreateInterpolation(), which in turn calls
DMPlexComputeInterpolatorNested/General.

  Thanks,

      Matt


> Thanks,
>
>
>> On Fri, Jul 8, 2022 at 11:26 PM Mike Michell <mi.mike1021 at gmail.com>
>> wrote:
>>
>>> I am using DMPlex for a code with written in Fortran in 2D and 3D. There
>>> were two questions.
>>>
>>> - As a follow up of the previous inquiry:
>>> https://www.mail-archive.com/petsc-users at mcs.anl.gov/msg43856.html
>>> Is the local-to-local halo exchange available in Fortran now or still
>>> pending? Currently local-to-global and global-to-local communication are
>>> used since local-to-local has not been allowed for Fortran.
>>>
>>
>> Sorry, it is still on my TODO list. I am trying to get stuff cleared out.
>>
>>
>>> - One code solves discretized equations at each vertex, and another code
>>> I want to couple physics is solving equations at each cell centroid.
>>> Basically, the value at cell centroid needs to be mapped to vertex (or vice
>>> versa) through interpolation/extrapolation for coupling of two codes. Does
>>> petsc function provide this kind of mapping between cell centroid and
>>> vertex? The grids for the two codes can be overlapped. I was trying to find
>>> some FEM infrastructures in petsc, but so far havent found that kind of
>>> functionality. Can I get any comments on that?
>>>
>>
>> Yes, you can create both P0 and P1 discretizations
>> (PetscFECreateLagrange) in two different DMs using DMClone(), and then
>> create an interpolation operator (DMPlexCreateInterpolator) which maps
>> between them. Let me know if something is not clear there.
>>
>>   Thanks,
>>
>>     Matt
>>
>>
>>> Thanks,
>>> Mike
>>>
>>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/~knepley/>
>>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220825/ee872f6a/attachment.html>

From knepley at gmail.com  Thu Aug 25 12:08:22 2022
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 25 Aug 2022 13:08:22 -0400
Subject: [petsc-users] [petsc-maint] metis version of petsc -
 segmentation fault
In-Reply-To: <CAFdn=ppApyB1yJ9MDkJho20gcOOVv1fGoagnadOAU+xMah06Hg@mail.gmail.com>
References: <CAFdn=ppApyB1yJ9MDkJho20gcOOVv1fGoagnadOAU+xMah06Hg@mail.gmail.com>
Message-ID: <CAMYG4G=3LAiea8bTB+Q0J=PJdr2FtG+2noRgi7QNtZ_PoAzfkg@mail.gmail.com>

On Thu, Aug 25, 2022 at 12:38 PM Alexandre Halbach <
alexandre.halbach at gmail.com> wrote:

> Hello,
>
> I am using the metis package provided by the latest petsc version, using
> --download-metis (Linux, 32 bits). This question is arguably more a metis
> issue but since metis is used in petsc and mumps this seems relevant to ask
> here too.
>
> I systematically get a segmentation fault on this metis version (didn't
> try others) when I partition even simple and not too large graphs ( O(5
> millions)). I tried a lot of cases as well as both the 32 and 64 bits
> version of petsc and also increasing ulimit (on Linux).
> Here is the shortest possible representative example with that problem
> (c++). I double checked it and I hope there is no mistake in this example.
> The below example constructs the containers for a 1D line graph (n
> vertices connected by edges one by one as in a chain. Both end vertices
> only have a single neighbor while all other vertices have two neighbors).
> Either this is something I am doing wrong or there is a bug in at least
> the metis version proposed in petsc.
>
> Here is the example producing a segfault. If you don't get a segfault try
> a 10 millions or more vertices graph.
>

You are exceeding the stack space. I will rewrite to malloc() and take a
look.

  Thanks,

     Matt


> Thank you for your help and your great work!
>
> Alex
>
>
> #include "metis.h"
>
> int main(void)
> {
>     // Number of vertices:
>     int n = 1000000;
>     // Addresses should for example be [0 1 3 5 7 8] for 5 vertices
>     idx_t addresses[n+1];
>     addresses[0] = 0; addresses[1] = 1;
>     for (int i = 2; i < n; i++)
>         addresses[i] = addresses[i-1] + 2;
>     addresses[n] = addresses[n-1]+1;
>
>     // Connected vertices should for example be [1 0 2 1 3 2 4 3] for 5
> vertices
>     idx_t connectivities[2*(n-1)];
>     connectivities[0] = 1;
>     for (int i = 1; i < n-1; i++)
>     {
>         connectivities[2*i-1] = i-1;
>         connectivities[2*i+0] = i+1;
>     }
>     connectivities[2*(n-1)-1] = connectivities[2*(n-1)-2]-1;
>
>     idx_t nvtxs = n;
>     idx_t nEdges = n-1;
>
>     idx_t metisOptions[METIS_NOPTIONS];
>     METIS_SetDefaultOptions(metisOptions);
>     metisOptions[METIS_OPTION_NUMBERING] = 0; // c numbering (start at 0)
>
>     idx_t ncon = 1;
>     idx_t nParts = 2;
>     idx_t objval;
>     idx_t part[nvtxs];
>
>     int ret = METIS_PartGraphRecursive ( &nvtxs, &ncon, addresses,
> connectivities, NULL, NULL, NULL, &nParts, NULL, NULL, metisOptions,
> &objval, part );
> }
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220825/dc127ba3/attachment.html>

From knepley at gmail.com  Thu Aug 25 12:31:03 2022
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 25 Aug 2022 13:31:03 -0400
Subject: [petsc-users] [petsc-maint] metis version of petsc -
 segmentation fault
In-Reply-To: <CAMYG4G=3LAiea8bTB+Q0J=PJdr2FtG+2noRgi7QNtZ_PoAzfkg@mail.gmail.com>
References: <CAFdn=ppApyB1yJ9MDkJho20gcOOVv1fGoagnadOAU+xMah06Hg@mail.gmail.com>
	<CAMYG4G=3LAiea8bTB+Q0J=PJdr2FtG+2noRgi7QNtZ_PoAzfkg@mail.gmail.com>
Message-ID: <CAMYG4GnzuMBWxCztS9x4sHrfELGYRh2nHJXfifYU_LVoAD85Xg@mail.gmail.com>

On Thu, Aug 25, 2022 at 1:08 PM Matthew Knepley <knepley at gmail.com> wrote:

> On Thu, Aug 25, 2022 at 12:38 PM Alexandre Halbach <
> alexandre.halbach at gmail.com> wrote:
>
>> Hello,
>>
>> I am using the metis package provided by the latest petsc version, using
>> --download-metis (Linux, 32 bits). This question is arguably more a metis
>> issue but since metis is used in petsc and mumps this seems relevant to ask
>> here too.
>>
>> I systematically get a segmentation fault on this metis version (didn't
>> try others) when I partition even simple and not too large graphs ( O(5
>> millions)). I tried a lot of cases as well as both the 32 and 64 bits
>> version of petsc and also increasing ulimit (on Linux).
>> Here is the shortest possible representative example with that problem
>> (c++). I double checked it and I hope there is no mistake in this example.
>> The below example constructs the containers for a 1D line graph (n
>> vertices connected by edges one by one as in a chain. Both end vertices
>> only have a single neighbor while all other vertices have two neighbors).
>> Either this is something I am doing wrong or there is a bug in at least
>> the metis version proposed in petsc.
>>
>> Here is the example producing a segfault. If you don't get a segfault try
>> a 10 millions or more vertices graph.
>>
>
> You are exceeding the stack space. I will rewrite to malloc() and take a
> look.
>

I do not SEGV with the attached code.

  Thanks,

     Matt


>   Thanks,
>
>      Matt
>
>
>> Thank you for your help and your great work!
>>
>> Alex
>>
>>
>> #include "metis.h"
>>
>> int main(void)
>> {
>>     // Number of vertices:
>>     int n = 1000000;
>>     // Addresses should for example be [0 1 3 5 7 8] for 5 vertices
>>     idx_t addresses[n+1];
>>     addresses[0] = 0; addresses[1] = 1;
>>     for (int i = 2; i < n; i++)
>>         addresses[i] = addresses[i-1] + 2;
>>     addresses[n] = addresses[n-1]+1;
>>
>>     // Connected vertices should for example be [1 0 2 1 3 2 4 3] for 5
>> vertices
>>     idx_t connectivities[2*(n-1)];
>>     connectivities[0] = 1;
>>     for (int i = 1; i < n-1; i++)
>>     {
>>         connectivities[2*i-1] = i-1;
>>         connectivities[2*i+0] = i+1;
>>     }
>>     connectivities[2*(n-1)-1] = connectivities[2*(n-1)-2]-1;
>>
>>     idx_t nvtxs = n;
>>     idx_t nEdges = n-1;
>>
>>     idx_t metisOptions[METIS_NOPTIONS];
>>     METIS_SetDefaultOptions(metisOptions);
>>     metisOptions[METIS_OPTION_NUMBERING] = 0; // c numbering (start at 0)
>>
>>     idx_t ncon = 1;
>>     idx_t nParts = 2;
>>     idx_t objval;
>>     idx_t part[nvtxs];
>>
>>     int ret = METIS_PartGraphRecursive ( &nvtxs, &ncon, addresses,
>> connectivities, NULL, NULL, NULL, &nParts, NULL, NULL, metisOptions,
>> &objval, part );
>> }
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220825/743ed8d4/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: tester.c
Type: application/octet-stream
Size: 1479 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220825/743ed8d4/attachment-0001.obj>

From mi.mike1021 at gmail.com  Thu Aug 25 15:25:08 2022
From: mi.mike1021 at gmail.com (Mike Michell)
Date: Thu, 25 Aug 2022 15:25:08 -0500
Subject: [petsc-users] DMPlex in Fortran
In-Reply-To: <CAMYG4GneehDK9dXyE4YPv95NUYD9x4wmt_0V_Fh50vP9-iZzZA@mail.gmail.com>
References: <CAEc7osYQoP2TuS9FU8+ZEoMOb+22TtwkHs8LOf6BShSRTZ8sKg@mail.gmail.com>
	<CAMYG4GkFHyqzduzA+HU+B2DOf35QPiZd+2eQnUuFUC6qaa462g@mail.gmail.com>
	<CAEc7osayv+YTMSRiMw5CAbF9Utt2n+7X1wbK-RoKDhV1Jqp+NA@mail.gmail.com>
	<CAMYG4GneehDK9dXyE4YPv95NUYD9x4wmt_0V_Fh50vP9-iZzZA@mail.gmail.com>
Message-ID: <CAEc7osZxBZHbRKNwqzjA=p3WcE=K9shf6i2p4twb-qh7fqReXA@mail.gmail.com>

Thank you for the quick note.
To use PetscFECreateLagrange(), it is unclear about the meaning of "Nc",
which is explained as 'number of components' on the manual page.
I looked at an example with Nc=1 in
src/dm/impls/swarm/tutorials/ex1f90.F90, but could you please detail the
meaning of Nc? If only one value (component) is defined on each of p0 and
p1 DMs, is Nc=1?
Thanks,


> On Thu, Aug 25, 2022 at 12:52 PM Mike Michell <mi.mike1021 at gmail.com>
> wrote:
>
>> Hi, I am trying to find the function you commented,
>> DMPlexCreateInterpolator(), from DMPlex manual page, but I do not think
>> this function is visible. Any comment about this?
>>
>
> Sorry, it is DMCreateInterpolation(), which in turn calls
> DMPlexComputeInterpolatorNested/General.
>
>   Thanks,
>
>       Matt
>
>
>> Thanks,
>>
>>
>>> On Fri, Jul 8, 2022 at 11:26 PM Mike Michell <mi.mike1021 at gmail.com>
>>> wrote:
>>>
>>>> I am using DMPlex for a code with written in Fortran in 2D and 3D.
>>>> There were two questions.
>>>>
>>>> - As a follow up of the previous inquiry:
>>>> https://www.mail-archive.com/petsc-users at mcs.anl.gov/msg43856.html
>>>> Is the local-to-local halo exchange available in Fortran now or still
>>>> pending? Currently local-to-global and global-to-local communication are
>>>> used since local-to-local has not been allowed for Fortran.
>>>>
>>>
>>> Sorry, it is still on my TODO list. I am trying to get stuff cleared out.
>>>
>>>
>>>> - One code solves discretized equations at each vertex, and another
>>>> code I want to couple physics is solving equations at each cell centroid.
>>>> Basically, the value at cell centroid needs to be mapped to vertex (or vice
>>>> versa) through interpolation/extrapolation for coupling of two codes. Does
>>>> petsc function provide this kind of mapping between cell centroid and
>>>> vertex? The grids for the two codes can be overlapped. I was trying to find
>>>> some FEM infrastructures in petsc, but so far havent found that kind of
>>>> functionality. Can I get any comments on that?
>>>>
>>>
>>> Yes, you can create both P0 and P1 discretizations
>>> (PetscFECreateLagrange) in two different DMs using DMClone(), and then
>>> create an interpolation operator (DMPlexCreateInterpolator) which maps
>>> between them. Let me know if something is not clear there.
>>>
>>>   Thanks,
>>>
>>>     Matt
>>>
>>>
>>>> Thanks,
>>>> Mike
>>>>
>>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>> https://www.cse.buffalo.edu/~knepley/
>>> <http://www.cse.buffalo.edu/~knepley/>
>>>
>>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220825/849390f9/attachment.html>

From mi.mike1021 at gmail.com  Thu Aug 25 18:11:55 2022
From: mi.mike1021 at gmail.com (Mike Michell)
Date: Thu, 25 Aug 2022 18:11:55 -0500
Subject: [petsc-users] DMPlex Interpolation
Message-ID: <CAEc7osYgsXxW9DN_6aHof3wdyqHw3Wv6nLp+svWsZF5c0zz93w@mail.gmail.com>

Hi, this is a duplication of
https://lists.mcs.anl.gov/pipermail/petsc-users/2022-August/046746.html
for in-depth question.

I wrote a short code as attached to test interpolation between two DMPlex
objects. Goal is to map solution field defined on cell-centroid(DMPlex-1)
into vertex(DMPlex-2) field or vice versa.

Basically, DMCreateInterpolation() fails in the example code and it was not
allowed to get useful error messages. The DMPlex is created by loading a 2D
square box in Gmsh. Can I get some comments on that?

Thanks,
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220825/c28561a9/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: DMInterpolation.tar
Type: application/x-tar
Size: 92160 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220825/c28561a9/attachment-0001.tar>

From alexandre.halbach at gmail.com  Thu Aug 25 11:37:25 2022
From: alexandre.halbach at gmail.com (Alexandre Halbach)
Date: Thu, 25 Aug 2022 19:37:25 +0300
Subject: [petsc-users] metis version of petsc - segmentation fault
Message-ID: <CAFdn=ppApyB1yJ9MDkJho20gcOOVv1fGoagnadOAU+xMah06Hg@mail.gmail.com>

Hello,

I am using the metis package provided by the latest petsc version, using
--download-metis (Linux, 32 bits). This question is arguably more a metis
issue but since metis is used in petsc and mumps this seems relevant to ask
here too.

I systematically get a segmentation fault on this metis version (didn't try
others) when I partition even simple and not too large graphs ( O(5
millions)). I tried a lot of cases as well as both the 32 and 64 bits
version of petsc and also increasing ulimit (on Linux).
Here is the shortest possible representative example with that problem
(c++). I double checked it and I hope there is no mistake in this example.
The below example constructs the containers for a 1D line graph (n vertices
connected by edges one by one as in a chain. Both end vertices only have a
single neighbor while all other vertices have two neighbors).
Either this is something I am doing wrong or there is a bug in at least the
metis version proposed in petsc.

Here is the example producing a segfault. If you don't get a segfault try a
10 millions or more vertices graph.

Thank you for your help and your great work!

Alex


#include "metis.h"

int main(void)
{
    // Number of vertices:
    int n = 1000000;
    // Addresses should for example be [0 1 3 5 7 8] for 5 vertices
    idx_t addresses[n+1];
    addresses[0] = 0; addresses[1] = 1;
    for (int i = 2; i < n; i++)
        addresses[i] = addresses[i-1] + 2;
    addresses[n] = addresses[n-1]+1;

    // Connected vertices should for example be [1 0 2 1 3 2 4 3] for 5
vertices
    idx_t connectivities[2*(n-1)];
    connectivities[0] = 1;
    for (int i = 1; i < n-1; i++)
    {
        connectivities[2*i-1] = i-1;
        connectivities[2*i+0] = i+1;
    }
    connectivities[2*(n-1)-1] = connectivities[2*(n-1)-2]-1;

    idx_t nvtxs = n;
    idx_t nEdges = n-1;

    idx_t metisOptions[METIS_NOPTIONS];
    METIS_SetDefaultOptions(metisOptions);
    metisOptions[METIS_OPTION_NUMBERING] = 0; // c numbering (start at 0)

    idx_t ncon = 1;
    idx_t nParts = 2;
    idx_t objval;
    idx_t part[nvtxs];

    int ret = METIS_PartGraphRecursive ( &nvtxs, &ncon, addresses,
connectivities, NULL, NULL, NULL, &nParts, NULL, NULL, metisOptions,
&objval, part );
}
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220825/95ce898f/attachment.html>

From lbgpublic at gmail.com  Fri Aug 26 08:47:06 2022
From: lbgpublic at gmail.com (byron)
Date: Fri, 26 Aug 2022 14:47:06 +0100
Subject: [petsc-users] unable to build petsc with cuda / superlu-dist
 enabled
In-Reply-To: <6BA4C5E3-29A4-4E68-A872-AE6A06185D64@gmail.com>
References: <CAJ+YQyrctNbkcdbUD-DmzdQRzUx8w12yy2dBbuntLm4Awj6McQ@mail.gmail.com>
	<C4B564E7-0DD0-4FAC-A9F3-185F0229653F@petsc.dev>
	<CAJ+YQyr9uhyR0LvCsuDwcGEhDXb1k5kSKNgP4Z+77jf97Pmfhg@mail.gmail.com>
	<2616A45C-68CF-4C94-A880-B7858C9B7882@petsc.dev>
	<6BA4C5E3-29A4-4E68-A872-AE6A06185D64@gmail.com>
Message-ID: <CAJ+YQyo2CCKeKiBjqgqWPqPzYZLyWDmfeS=uAYD819bK8-c8+A@mail.gmail.com>

Hi Barry

I dont think I see the same thing.

When I do a build using the release branch petsc passes the following
arguements to superlu_dist (taken from
/home/blahblah/Projects/220826-vanilla-petsc-release-branch/petsc/arch-linux-c-opt/externalpackages/git.superlu_dist/superlu_dist.petscconf)
which has only one -O3 entry

-DCMAKE_INSTALL_PREFIX=/home/blahblah/Projects/220826-vanilla-petsc-release-branch/installation
-DCMAKE_INSTALL_NAME_DIR:STRING="/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib"
-DCMAKE_INSTALL_LIBDIR:STRING="lib" -DCMAKE_VERBOSE_MAKEFILE=1
-DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER="mpicc"
-DMPI_C_COMPILER="/usr/bin/mpicc" -DCMAKE_AR=/usr/bin/ar
-DCMAKE_RANLIB=/usr/bin/ranlib -DCMAKE_C_FLAGS:STRING="-fPIC
-Wno-lto-type-mismatch -fstack-protector -g -O -fopenmp  -DDEBUGlevel=0
-DPRNTlevel=0" -DCMAKE_C_FLAGS_DEBUG:STRING="-fPIC -Wno-lto-type-mismatch
-fstack-protector -g -O -fopenmp  -DDEBUGlevel=0 -DPRNTlevel=0"
-DCMAKE_C_FLAGS_RELEASE:STRING="-fPIC -Wno-lto-type-mismatch
-fstack-protector -g -O -fopenmp  -DDEBUGlevel=0 -DPRNTlevel=0"
-DCMAKE_CXX_COMPILER="mpicxx" -DMPI_CXX_COMPILER="/usr/bin/mpicxx"
-DCMAKE_CXX_FLAGS:STRING="-Wno-lto-type-mismatch -fstack-protector -g -O
-fopenmp -std=c++11 -fPIC -fopenmp  -DDEBUGlevel=0 -DPRNTlevel=0"
-DCMAKE_CXX_FLAGS_DEBUG:STRING="-Wno-lto-type-mismatch -fstack-protector -g
-O -fopenmp -std=c++11 -fPIC -fopenmp  -DDEBUGlevel=0 -DPRNTlevel=0"
-DCMAKE_CXX_FLAGS_RELEASE:STRING="-Wno-lto-type-mismatch -fstack-protector
-g -O -fopenmp -std=c++11 -fPIC -fopenmp  -DDEBUGlevel=0 -DPRNTlevel=0"
-DCMAKE_Fortran_COMPILER="mpif90" -DMPI_Fortran_COMPILER="/usr/bin/mpif90"
-DCMAKE_Fortran_FLAGS:STRING="-fPIC -ffree-line-length-0
-Wno-lto-type-mismatch -g -O -fopenmp"
-DCMAKE_Fortran_FLAGS_DEBUG:STRING="-fPIC -ffree-line-length-0
-Wno-lto-type-mismatch -g -O -fopenmp"
-DCMAKE_Fortran_FLAGS_RELEASE:STRING="-fPIC -ffree-line-length-0
-Wno-lto-type-mismatch -g -O -fopenmp" -DCMAKE_EXE_LINKER_FLAGS:STRING="
-fopenmp -fopenmp" -DBUILD_SHARED_LIBS:BOOL=ON -DTPL_ENABLE_CUDALIB=TRUE
-DTPL_CUDA_LIBRARIES="-lcudart -lcufft -lcublas -lcusparse -lcusolver
-lcurand -lcuda" -DCUDA_ARCH_FLAGS="-arch=sm_75"
-DCMAKE_CUDA_COMPILER="nvcc" -DCMAKE_CUDA_FLAGS=" -ccbin mpicxx -std=c++14
-Xcompiler -fPIC *-O3 *-gencode arch=compute_75,code=sm_75
 -Wno-deprecated-gpu-targets
-I/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi
-I/usr/lib/x86_64-linux-gnu/openmpi/include   -DDEBUGlevel=0 -DPRNTlevel=0"
-DUSE_XSDK_DEFAULTS=YES
-DTPL_BLAS_LIBRARIES="-Wl,-rpath,/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib
-L/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib
-lflapack
-Wl,-rpath,/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib
-L/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib
-lfblas -lcudart -lcufft -lcublas -lcusparse -lcusolver -lcurand -lcuda -lm
-lstdc++ -ldl -Wl,-rpath,/usr/lib/x86_64-linux-gnu/openmpi/lib
-L/usr/lib/x86_64-linux-gnu/openmpi/lib -lmpi_usempif08
-lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi -lgfortran -lm
-Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/9
-L/usr/lib/gcc/x86_64-linux-gnu/9 -lgfortran -lm -lgcc_s -lquadmath
-lpthread"
-DTPL_LAPACK_LIBRARIES="-Wl,-rpath,/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib
-L/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib
-lflapack
-Wl,-rpath,/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib
-L/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib
-lfblas -lcudart -lcufft -lcublas -lcusparse -lcusolver -lcurand -lcuda -lm
-lstdc++ -ldl -Wl,-rpath,/usr/lib/x86_64-linux-gnu/openmpi/lib
-L/usr/lib/x86_64-linux-gnu/openmpi/lib -lmpi_usempif08
-lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi -lgfortran -lm
-Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/9
-L/usr/lib/gcc/x86_64-linux-gnu/9 -lgfortran -lm -lgcc_s -lquadmath
-lpthread"
-DTPL_PARMETIS_INCLUDE_DIRS="/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/include;/usr/lib/x86_64-linux-gnu/openmpi/include;/usr/lib/x86_64-linux-gnu/openmpi/lib"
-DTPL_PARMETIS_LIBRARIES="-Wl,-rpath,/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib
-L/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib
-lparmetis
-Wl,-rpath,/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib
-L/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib
-lmetis -lm -lcudart -lcufft -lcublas -lcusparse -lcusolver -lcurand
-lcuda" -DXSDK_INDEX_SIZE=64 -DXSDK_ENABLE_Fortran=ON -Denable_tests=0
-Denable_examples=0 -DMPI_C_COMPILE_FLAGS:STRING=""
-DMPI_C_INCLUDE_PATH:STRING="" -DMPI_C_HEADER_DIR:STRING=""
-DMPI_C_LIBRARIES:STRING=""

Or am I looking at the wrong thing?

I also tried doing the build downloading petsc from the main branch but
that died at the same point and looked to give the same "redefinition of
argument 'optimize'" errors

Thanks
Byron



On Thu, Aug 25, 2022 at 4:15 PM Jacob Faibussowitsch <jacob.fai at gmail.com>
wrote:

> > Sadly each one makes different -std=c++11 vs  -std=c++14 choices as well.
>
> Main should hopefully *also* warn about this, but unless you really really
> want -std=c++11 or -std=c++14, it is better to leave out this configure
> flag (--with-cxx-dialect).
>
> If --with-cxx-dialect is unspecified (or set to ?auto?) configure will
> pick the most appropriate flag for you, taking your package environment
> into account when choosing. If you specify the flag configure must assume
> the user knows something it doesn?t and will blindly take what is given.
>
> Best regards,
>
> Jacob Faibussowitsch
> (Jacob Fai - booss - oh - vitch)
>
> > On Aug 25, 2022, at 11:05, Barry Smith <bsmith at petsc.dev> wrote:
> >
> >
> >   Oh yes, my mistake I misinterpreted the error message. It is actually
> the nvcc compiler complaining about multiple command line arguments related
> to optimization (which it is really picky about even if the two arguments
> are identical.)  I reformatted some of the configure output to show this
> >
> > -ccbin mpicxx -std=c++14 -Xcompiler -fPIC -O3 -gencode
> arch=compute_75,code=sm_75  -Wno-deprecated-gpu-targets
> > -I/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi
> -I/usr/lib/x86_64-linux-gnu/openmpi/include
> > -DDEBUGlevel=0 -DPRNTlevel=0
> >
> > the next fragment is
> >
> > -O3 -DNDEBUG
> >
> > --generate-code=arch=compute_75,code=[compute_75,sm_75]
> >
> > -Xcompiler=-fPIC -std=c++11 -x cu -rdc=true -c
> /home/blahblah/Projects/220824-petsc-example-for_forum/petsc/arch-linux-c-opt/externalpackages/git.superlu_dist/SRC/
> superlu_gpu_utils.cu -o
> >
> > Hence the -O3 appears twice, once from the CMAKE argument PETSc passes
> to superlu_dist
> >
> > -DCMAKE_CUDA_FLAGS=" -ccbin mpicxx -std=c++14 -Xcompiler -fPIC -O3
> -gencode arch=compute_75,code=sm_75  -Wno-deprecated-gpu-targets
> -I/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi
> -I/usr/lib/x86_64-linux-gnu/openmpi/include   -DDEBUGlevel=0 -DPRNTlevel=0"
> >
> > and then presumably, the superlu_dist cmake process adds its own
> arguments which overlap.
> >
> > If I remember correctly this is fixed in the main branch of PETSc, could
> you do an identical build with the main branch? If that works we will just
> need to figure out how to get MOOSE to work with main PETSc branch.
> >
> >   Barry
> >
> > Sadly each one makes different -std=c++11 vs  -std=c++14 choices as well.
> >
> >
> > --------------------
> >
> > [ 15%] Building CUDA object
> SRC/CMakeFiles/superlu_dist.dir/superlu_gpu_utils.cu.o
> > cd
> /home/blahblah/Projects/220824-petsc-example-for_forum/petsc/arch-linux-c-opt/externalpackages/git.superlu_dist/petsc-build/SRC
> && /usr/bin/nvcc
> >
> > -DSUPERLU_DIST_EXPORTS -Dsuperlu_dist_EXPORTS
> >
> >
> -I/home/blahblah/Projects/220824-petsc-example-for_forum/petsc/arch-linux-c-opt/externalpackages/git.superlu_dist/petsc-build/SRC
> >
> > -I/home/blahblah/Projects/220824-petsc-build/installation/include
> -I/usr/lib/x86_64-linux-gnu/openmpi/lib
> >
> > -DUSE_VENDOR_BLAS  -ccbin mpicxx -std=c++14 -Xcompiler -fPIC -O3
> -gencode arch=compute_75,code=sm_75  -Wno-deprecated-gpu-targets
> >
> > -I/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi
> -I/usr/lib/x86_64-linux-gnu/openmpi/include
> >
> > -DDEBUGlevel=0 -DPRNTlevel=0 -O3 -DNDEBUG
> >
> > --generate-code=arch=compute_75,code=[compute_75,sm_75]
> >
> > -Xcompiler=-fPIC -std=c++11 -x cu -rdc=true -c
> /home/blahblah/Projects/220824-petsc-example-for_forum/petsc/arch-linux-c-opt/externalpackages/git.superlu_dist/SRC/
> superlu_gpu_utils.cu -o
> >
> > CMakeFiles/superlu_dist.dir/superlu_gpu_utils.cu
> >
> >
> >
> >> On Aug 25, 2022, at 9:38 AM, byron <lbgpublic at gmail.com> wrote:
> >>
> >> Hi Barry
> >>
> >> There doesn't appear to be any reference to 'optimize' in the only copy
> of  superlu_gpu_utils.cu I could find, I've attached a copy.
> >>
> >> Thanks
> >>
> >>
> >>
> >> On Thu, Aug 25, 2022 at 1:44 PM Barry Smith <bsmith at petsc.dev> wrote:
> >>
> >> nvcc fatal   : redefinition of argument 'optimize'
> >> make[2]: *** [SRC/CMakeFiles/superlu_dist.dir/build.make:499:
> SRC/CMakeFiles/superlu_dist.dir/superlu_gpu_utils.cu.o] Error 1
> >> make[2]: *** Waiting for unfinished jobs....
> >> nvcc fatal   : redefinition of argument 'optimize'
> >> make[2]: *** [SRC/CMakeFiles/superlu_dist-static.dir/build.make:499:
> SRC/CMakeFiles/superlu_dist-static.dir/superlu_gpu_utils.cu.o] Error 1
> >> make[2]: *** Waiting for unfinished jobs....
> >>
> >> It seems that nvcc is unhappy with something in superlu_gpu_utils.cu
> Could you check that file for any "redefinition of optimize"? Unfortunately
> the nvcc compiler isn't kind enough to print the line number of the
> redefinition or that information is lost somehow in the build process.
> >>
> >> Barry
> >>
> >>> On Aug 25, 2022, at 6:50 AM, byron <lbgpublic at gmail.com> wrote:
> >>>
> >>> Hi All
> >>>
> >>> I am having trouble getting petsc to build with cuda / superlu-dist
> enabled.  I have been trying to build it as part of another software tool,
> moose, hence the choice of options being passed to configure
> >>>
> >>> python3 ./configure --download-hypre=1 \
> >>> --with-shared-libraries=1 \
> >>> --download-hdf5=1 \
> >>> --download-hdf5-fortran-bindings=0 \
> >>> --with-debugging=no \
> >>> --download-fblaslapack=1 \
> >>> --download-metis=1 \
> >>> --download-ptscotch=1 \
> >>> --download-parmetis=1 \
> >>> --download-superlu_dist=1 \
> >>> --download-mumps=1 \
> >>> --download-strumpack=1 \
> >>> --download-scalapack=1 \
> >>> --download-slepc=1 \
> >>> --with-mpi=1 \
> >>> --with-openmp=1 \
> >>> --with-cxx-dialect=C++11 \
> >>> --with-fortran-bindings=0 \
> >>> --with-sowing=0 \
> >>> --with-64-bit-indices \
> >>> --with-cuda \
> >>> --prefix=/home/blah/blah
> >>>
> >>> it fails with
> >>>
> >>>
> =============================================================================================
>
>                                Installing PTScotch; this may take several
> minutes
>
> =============================================================================================
>
>
>  =============================================================================================
>
>                                Trying to download git://
> https://github.com/xiaoyeli/superlu_dist for SUPERLU_DIST
>
>
> =============================================================================================
>
>
>  =============================================================================================
>
>                                Configuring SUPERLU_DIST with cmake; this
> may take several minutes
>
> =============================================================================================
>
>
>  =============================================================================================
>
>                                Compiling and installing SUPERLU_DIST; this
> may take several minutes
>
> =============================================================================================
>
>
>  *******************************************************************************
> >>>          UNABLE to CONFIGURE with GIVEN OPTIONS    (see configure.log
> for details):
> >>>
> -------------------------------------------------------------------------------
> >>> Error running make on  SUPERLU_DIST
> >>>
> *******************************************************************************
> >>>
> >>> I've attached the configure.log.
> >>>
> >>> I've tried a few different things but nothing that resulted in a
> successfull build.  The most promising was I tried to build superlu_dist
> separately with the configure options that petsc passes it and then kept
> trying the build over and over droping out different configure options
> until I got it to work.  In that case removing a "-std=c++14" entry
> appeared to fix the problem but then when I made the same change in how
> petsc builds superlu-dist the make failed on superlu-dist with a different
> error.  I get the feeling I'm dealing with symptoms of the problem rather
> than the cause.
> >>>
> >>> Thanks
> >>> Byron
> >>>
> >>>
> >>>
> >>>
> >>> <configure.log>
> >>
> >> <superlu_gpu_utils.cu>
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220826/dfa0f5e8/attachment-0001.html>

From snailsoar at hotmail.com  Fri Aug 26 08:51:07 2022
From: snailsoar at hotmail.com (feng wang)
Date: Fri, 26 Aug 2022 13:51:07 +0000
Subject: [petsc-users] Slepc Question, shell-matrix
In-Reply-To: <C6B97F7E-6D5F-4237-9F1F-54AEFB61F06E@dsic.upv.es>
References: <ME3PR01MB7624DF748CD2568A0BAAC5B6B9719@ME3PR01MB7624.ausprd01.prod.outlook.com>
	<C1E1C97D-7F02-4764-9732-9330A6076438@dsic.upv.es>
	<ME3PR01MB7624ADA914B555F17A490226B9719@ME3PR01MB7624.ausprd01.prod.outlook.com>
	<8B78A886-8CB5-49A9-AFED-E67C22696400@dsic.upv.es>
	<SYZPR01MB76275849E71CDD1083D79937B9709@SYZPR01MB7627.ausprd01.prod.outlook.com>
	<17F56A61-2DB1-416B-85CE-8E585051F1EB@dsic.upv.es>
	<ME3PR01MB762414A8D756C6C68BC95E92B9709@ME3PR01MB7624.ausprd01.prod.outlook.com>
	<3E25E7D6-C81A-4A75-A215-A5FCFB7F3CB6@dsic.upv.es>
	<ME3PR01MB762495A98C93E2D6EA7E30ABB9739@ME3PR01MB7624.ausprd01.prod.outlook.com>
	<6BC033FF-0A55-4564-923D-97EFDD7C7FF7@dsic.upv.es>
	<ME3PR01MB762467CDD5598EC8E2AC5A50B9739@ME3PR01MB7624.ausprd01.prod.outlook.com>
	<8E59CD67-862C-463C-B3ED-1E530ECAE333@dsic.upv.es>
	<ME3PR01MB7624F86687223C01A676BA28B9729@ME3PR01MB7624.ausprd01.prod.outlook.com>
	<EDA8B12C-5637-4F23-8745-CB44C8A2367E@dsic.upv.es>
	<ME3PR01MB7624B60950A0AD91E26DF144B9729@ME3PR01MB7624.ausprd01.prod.outlook.com>
	<C6B97F7E-6D5F-4237-9F1F-54AEFB61F06E@dsic.upv.es>
Message-ID: <ME3PR01MB76243335EB48CD80A2BD0A7EB9759@ME3PR01MB7624.ausprd01.prod.outlook.com>

Hi Jose,

Thanks for your reply. Then using finite difference in the shell matrix to compute the matrix-vector product is not a good idea here. I will try to find an alternative way. maybe assemble the matrix directly.

Thanks,
Feng
________________________________
From: Jose E. Roman <jroman at dsic.upv.es>
Sent: 25 August 2022 16:31
To: feng wang <snailsoar at hotmail.com>
Cc: petsc-users <petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] Slepc Question, shell-matrix

Your formula
(R(q+epsilon*v) - R(q-epsilon*v))/(2*epsilon).
is not linear, it cannot be expressed as a constant matrix times a vector, A*v.

If you want to solve a nonlinear eigenvector problem
A(x)*x=lambda*x
then have a look at ex34.c https://slepc.upv.es/documentation/current/src/eps/tutorials/ex34.c.html

Jose


> El 25 ago 2022, a las 18:07, feng wang <snailsoar at hotmail.com> escribi?:
>
> Hi Jose,
>
> Following my previous email, just some update,  the epsilon actually does not change in my shell matrix to compute the finite difference Jacobian. I don't have an iterative procedure in the shell matrix. It simply computes the non-linear residuals twice with perturbed flow variables. I have attached my code. Any suggestions are welcome.
>
> Thanks,
> Feng
> From: Jose E. Roman <jroman at dsic.upv.es>
> Sent: 25 August 2022 9:45
> To: feng wang <snailsoar at hotmail.com>
> Cc: petsc-users <petsc-users at mcs.anl.gov>
> Subject: Re: [petsc-users] Slepc Question, shell-matrix
>
> Probably the explanation is the following. Your shell matrix runs an iterative procedure that computes the action of the matrix to a certain accuracy. This is equivalent to multiplying by the exact matrix plus a perturbation, A+E. But every time you apply the shell matrix, the perturbation E is slightly different, so the eigensolver does not really see a constant matrix, it changes at every iteration.
>
> This also happens in nested iterations, such as inexact shift-and-invert, where e.g. GMRES is run within each iteration of the eigensolver. For this to work, the GMRES tolerance should be smaller than the tolerance of the eigensolver. Similarly, if your shell matrix runs an iterative procedure, you have to use a tolerance that is more stringent than the one used in the eigensolver.
>
> Hope this helps.
> Jose
>
>
> > El 25 ago 2022, a las 11:34, feng wang <snailsoar at hotmail.com> escribi?:
> >
> > Hi Jose,
> >
> > Thanks for your reply. I have fixed one thing in my shell matrix. Now the conversion seems working.
> >
> > If I use the converted dense matrix and give it to EPSSetOperators, it actually converges!
> >
> >  50 EPS nconv=0 first unconverged value (error) 2.89521e-06+2.13473e-06i (7.73051721e-08)
> >  51 EPS nconv=0 first unconverged value (error) 2.89521e-06+2.13473e-06i (5.42681175e-08)
> >  52 EPS nconv=0 first unconverged value (error) 2.89521e-06+2.13473e-06i (2.48352708e-08)
> >  53 EPS nconv=0 first unconverged value (error) 2.89521e-06+2.13473e-06i (1.76912430e-08)
> >  54 EPS nconv=0 first unconverged value (error) 2.89521e-06+2.13473e-06i (2.07480734e-08)
> >  55 EPS nconv=0 first unconverged value (error) 2.89521e-06+2.13473e-06i (1.13588981e-08)
> >  56 EPS nconv=2 first unconverged value (error) 1.10143e-06+9.80005e-06i (8.32889697e-06)
> >  Solution method: krylovschur
> >
> >  Number of requested eigenvalues: 1
> >  Linear eigensolve converged (2 eigenpairs) due to CONVERGED_TOL; iterations 56
> >  ---------------------- --------------------
> >             k             ||Ax-kx||/||kx||
> >  ---------------------- --------------------
> >    0.000003+0.000002i       7.90447e-09
> >    0.000003-0.000002i       7.90447e-09
> >  ---------------------- --------------------
> >
> > If I use the shell matrix directly, the relative norm is still large. This really puzzles me......
> >
> > Thanks,
> > Feng
> >
> > From: Jose E. Roman <jroman at dsic.upv.es>
> > Sent: 25 August 2022 7:21
> > To: feng wang <snailsoar at hotmail.com>
> > Cc: petsc-users <petsc-users at mcs.anl.gov>
> > Subject: Re: [petsc-users] Slepc Question, shell-matrix
> >
> > This works for simple examples such as ex9.c
> > It may be an indication that there is a problem with your shell matrix.
> >
> > Jose
> >
> >
> > > El 24 ago 2022, a las 19:35, feng wang <snailsoar at hotmail.com> escribi?:
> > >
> > > Hi Jose,
> > >
> > > Thanks for your reply.
> > >
> > > I have tried PetscCall(MatConvert(slepc_A_mf,MATDENSE,MAT_INITIAL_MATRIX,&Adense));  and If I do MatView(Adense,viewer), All I see was NAN. Have I missed anything here?
> > >
> > > Thanks,
> > > Feng
> > >
> > > From: Jose E. Roman <jroman at dsic.upv.es>
> > > Sent: 24 August 2022 16:23
> > > To: feng wang <snailsoar at hotmail.com>
> > > Cc: petsc-users <petsc-users at mcs.anl.gov>
> > > Subject: Re: [petsc-users] Slepc Question, shell-matrix
> > >
> > >
> > >
> > > > El 24 ago 2022, a las 17:03, feng wang <snailsoar at hotmail.com> escribi?:
> > > >
> > > > Hi Jose,
> > > >
> > > > Thanks for your reply.
> > > >
> > > > I have tried -eps_view_mat0 binary:amatrix.bin to save my shell matrix. It seems not saving the shell matrix and just created an empty file.
> > >
> > > Oops, I thought the shell matrix would be converted automatically. Try converting it yourself:
> > >
> > > PetscCall(MatConvert(slepc_A_mf,MATDENSE,MAT_INITIAL_MATRIX,&Adense));
> > >
> > > Then use Adense in EPSSetOperators(). Or save it to disk and send it to me so that I can give it a try.
> > >
> > > >
> > > > Besides, I am trying to understand the output of slepc.  I can set  -eps_tol to a low value, but the final relative residual norm is still high , what does this tell me?
> > >
> > > I don't know. It should not happen.
> > > Jose
> > >
> > > >
> > > > Best regards,
> > > > Feng
> > > >
> > > > From: Jose E. Roman <jroman at dsic.upv.es>
> > > > Sent: 23 August 2022 11:06
> > > > To: feng wang <snailsoar at hotmail.com>
> > > > Cc: petsc-users <petsc-users at mcs.anl.gov>
> > > > Subject: Re: [petsc-users] Slepc Question, shell-matrix
> > > >
> > > > You can try the following. Save your shell matrix to disk with the option
> > > > -eps_view_mat0 binary:amatrix.bin
> > > > then repeat the computation with ex4.c loading this file.
> > > >
> > > > Note that this should be done for a small problem size, because the conversion from a shell matrix implies a matrix-vector product per each column.
> > > >
> > > > Jose
> > > >
> > > > > El 23 ago 2022, a las 12:57, feng wang <snailsoar at hotmail.com> escribi?:
> > > > >
> > > > > Hi Jose,
> > > > >
> > > > > Thanks for your reply.
> > > > >
> > > > > It represents a linear operator. In my shell matrix, I am computing the non-linear residuals twice with perturbed flow variables. The matrix-vector product is computed as:
> > > > >
> > > > > A*v = (R(q+eps*v) - R(q-eps*v))/(2*eps)
> > > > >
> > > > > R is the non-linear residual. q is my flow variable and it does not change. eps is the perturbation. A is my Jacobian matrix. Besides, for some background, I am computing a steady RANS flow with finite volume method and trying to do a global stability analysis by looking at the Jacobian matrix.
> > > > >
> > > > > Thanks,
> > > > > Feng
> > > > >
> > > > > From: Jose E. Roman <jroman at dsic.upv.es>
> > > > > Sent: 23 August 2022 10:32
> > > > > To: feng wang <snailsoar at hotmail.com>
> > > > > Cc: petsc-users <petsc-users at mcs.anl.gov>
> > > > > Subject: Re: [petsc-users] Slepc Question, shell-matrix
> > > > >
> > > > > The relative residual norms that are printed at the end are too large. For NHEP problems, they should be below the tolerance. Don't know what is happening. Does your shell matrix represent a linear (constant) operator? Or does it change slightly depending on the input vector?
> > > > >
> > > > > > El 23 ago 2022, a las 12:14, feng wang <snailsoar at hotmail.com> escribi?:
> > > > > >
> > > > > > Hi Jose,
> > > > > >
> > > > > > I think the previous problem comes from my side. I have some uninitialized values in my part of code to compute the non-linear residuals. so, it produces a NAN when it tries to compute the matrix-vector product using finite difference.  This might make the slepc/pestc do unexpected things.
> > > > > >
> > > > > > Now It seems I've got slepc running. eps_nev is set to 3 and I am trying to compute the ones with the largest amplitudes. Below is the slepc output.
> > > > > >
> > > > > >  14 EPS converged value (error) #0 -0.000164282 (5.36813206e-09)
> > > > > >  16 EPS converged value (error) #1 -0.000160691+2.17113e-05i (3.37429620e-09)
> > > > > >  16 EPS converged value (error) #2 -0.000160691-2.17113e-05i (3.37429620e-09)
> > > > > >  Solution method: krylovschur
> > > > > >
> > > > > >  Number of requested eigenvalues: 2
> > > > > >  Linear eigensolve converged (3 eigenpairs) due to CONVERGED_TOL; iterations 16
> > > > > >  ---------------------- --------------------
> > > > > >             k             ||Ax-kx||/||kx||
> > > > > >  ---------------------- --------------------
> > > > > >        -0.000164              0.0613788
> > > > > >   -0.000161+0.000022i         0.0773339
> > > > > >   -0.000161-0.000022i         0.0774536
> > > > > >  ---------------------- --------------------
> > > > > >
> > > > > > The values in the brackets are the absolute error (I believe) and they seem very low. The relative error seems quite large.  Could you please comment on this?
> > > > > >
> > > > > >
> > > > > > Best regards,
> > > > > > Feng
> > > > > >
> > > > > > From: Jose E. Roman <jroman at dsic.upv.es>
> > > > > > Sent: 23 August 2022 5:24
> > > > > > To: feng wang <snailsoar at hotmail.com>
> > > > > > Cc: petsc-users <petsc-users at mcs.anl.gov>
> > > > > > Subject: Re: [petsc-users] Slepc Question, shell-matrix
> > > > > >
> > > > > > Please always respond to the list, otherwise the thread appears as unresolved in the archives of the mailing list.
> > > > > >
> > > > > >
> > > > > > > El 22 ago 2022, a las 22:45, feng wang <snailsoar at hotmail.com> escribi?:
> > > > > > >
> > > > > > > Hi Jose,
> > > > > > >
> > > > > > > I think I might have solved my problem. I have some uninitialized values in my part of code to compute the right hand side. so it produces a NAN when it tries to compute the matrix-vector product.
> > > > > > >
> > > > > > > Many thanks for your help!
> > > > > > >
> > > > > > > Best regards,
> > > > > > > Feng
> > > > > > > From: Jose E. Roman <jroman at dsic.upv.es>
> > > > > > > Sent: 22 August 2022 19:32
> > > > > > > To: feng wang <snailsoar at hotmail.com>
> > > > > > > Cc: petsc-users at mcs.anl.gov <petsc-users at mcs.anl.gov>
> > > > > > > Subject: Re: [petsc-users] Slepc Question, shell-matrix
> > > > > > >
> > > > > > > This is very strange. This error appears when the solver employs the B-inner product, but in your case you don't have a B matrix, so you should never see that error. Try running under valgrind to see if it gives more hints.
> > > > > > >
> > > > > > > Jose
> > > > > > >
> > > > > > >
> > > > > > > > El 22 ago 2022, a las 20:45, feng wang <snailsoar at hotmail.com> escribi?:
> > > > > > > >
> > > > > > > > Hello,
> > > > > > > >
> > > > > > > > I am new to Slepc and trying to work out the eigenvalues and eigenvectors of my Jacobian matrix. I am using a shell matrix to work out the matrix-vector product and I am using the default Krylov-schur method.
> > > > > > > >
> > > > > > > > My first attempt was not successful and I got the following errors:
> > > > > > > >
> > > > > > > > [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> > > > > > > > [0]PETSC ERROR: Missing or incorrect user input
> > > > > > > > [0]PETSC ERROR: The inner product is not well defined: indefinite matrix
> > > > > > > > [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
> > > > > > > > [0]PETSC ERROR: Petsc Release Version 3.17.4, unknown
> > > > > > > > [0]PETSC ERROR: cfdtest on a arch-debug named ming by feng Mon Aug 22 19:21:41 2022
> > > > > > > > [0]PETSC ERROR: Configure options --with-cc=mpicc --with-cxx=mpicxx --with-fc=0 PETSC_ARCH=arch-debug
> > > > > > > > [0]PETSC ERROR: #1 BV_SafeSqrt() at /home/feng/cfd/slepc-3.17.1/include/slepc/private/bvimpl.h:130
> > > > > > > > [0]PETSC ERROR: #2 BV_SquareRoot_Default() at /home/feng/cfd/slepc-3.17.1/include/slepc/private/bvimpl.h:365
> > > > > > > > [0]PETSC ERROR: #3 BVOrthogonalizeCGS1() at /home/feng/cfd/slepc-3.17.1/src/sys/classes/bv/interface/bvorthog.c:101
> > > > > > > > [0]PETSC ERROR: #4 BVOrthogonalizeGS() at /home/feng/cfd/slepc-3.17.1/src/sys/classes/bv/interface/bvorthog.c:177
> > > > > > > > [0]PETSC ERROR: #5 BVOrthonormalizeColumn() at /home/feng/cfd/slepc-3.17.1/src/sys/classes/bv/interface/bvorthog.c:402
> > > > > > > > [0]PETSC ERROR: #6 BVMatArnoldi() at /home/feng/cfd/slepc-3.17.1/src/sys/classes/bv/interface/bvkrylov.c:91
> > > > > > > > [0]PETSC ERROR: #7 EPSSolve_KrylovSchur_Default() at /home/feng/cfd/slepc-3.17.1/src/eps/impls/krylov/krylovschur/krylovschur.c:261
> > > > > > > > [0]PETSC ERROR: #8 EPSSolve() at /home/feng/cfd/slepc-3.17.1/src/eps/interface/epssolve.c:147
> > > > > > > > [0]PETSC ERROR: #9 slepc_eigen_comp() at domain/cfd/slepc_eigen_solve.cpp:77
> > > > > > > >
> > > > > > > > Could someone please shine some light on this? I have also attached my code. The code is part of my big library and I cannot attach the whole code, sorry about this. but I am happy to provide more information. The attached code has some arrangements for halo exchange, but for the moment, it assumes it is a serial run.
> > > > > > > >
> > > > > > > > Many thanks,
> > > > > > > > Feng
> > > > > > > > <slepc_eigen_solve.cpp>
>
> <slepc_eigen_solve.cpp>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220826/d17ff44d/attachment-0001.html>

From knepley at gmail.com  Fri Aug 26 09:30:44 2022
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 26 Aug 2022 10:30:44 -0400
Subject: [petsc-users] DMPlex in Fortran
In-Reply-To: <CAEc7osZxBZHbRKNwqzjA=p3WcE=K9shf6i2p4twb-qh7fqReXA@mail.gmail.com>
References: <CAEc7osYQoP2TuS9FU8+ZEoMOb+22TtwkHs8LOf6BShSRTZ8sKg@mail.gmail.com>
	<CAMYG4GkFHyqzduzA+HU+B2DOf35QPiZd+2eQnUuFUC6qaa462g@mail.gmail.com>
	<CAEc7osayv+YTMSRiMw5CAbF9Utt2n+7X1wbK-RoKDhV1Jqp+NA@mail.gmail.com>
	<CAMYG4GneehDK9dXyE4YPv95NUYD9x4wmt_0V_Fh50vP9-iZzZA@mail.gmail.com>
	<CAEc7osZxBZHbRKNwqzjA=p3WcE=K9shf6i2p4twb-qh7fqReXA@mail.gmail.com>
Message-ID: <CAMYG4Gmr-Sr2DQpEeH3NX-NJ9FD+tjxuW=eKA89bcaf1oiyN6w@mail.gmail.com>

On Thu, Aug 25, 2022 at 4:25 PM Mike Michell <mi.mike1021 at gmail.com> wrote:

> Thank you for the quick note.
> To use PetscFECreateLagrange(), it is unclear about the meaning of "Nc",
> which is explained as 'number of components' on the manual page.
> I looked at an example with Nc=1 in
> src/dm/impls/swarm/tutorials/ex1f90.F90, but could you please detail the
> meaning of Nc? If only one value (component) is defined on each of p0 and
> p1 DMs, is Nc=1?
>

Sure. Nc is the number of tensor components, although we usually think of
it as a rank-1 tensor (vector). For most elements, such as Lagrange
elements, we understand this as the Nc-fold tensor product of the scalar
element. For example, suppose you ask for P_2 with Nc = 3. That would mean
a vector field in 3D where each component is quadratic. On a tetrahedral
mesh, each vertex and edge would have 3 dogs, one for the x, y, and z
component of the field.

   Thanks,

     Matt


> Thanks,
>
>
>> On Thu, Aug 25, 2022 at 12:52 PM Mike Michell <mi.mike1021 at gmail.com>
>> wrote:
>>
>>> Hi, I am trying to find the function you commented,
>>> DMPlexCreateInterpolator(), from DMPlex manual page, but I do not think
>>> this function is visible. Any comment about this?
>>>
>>
>> Sorry, it is DMCreateInterpolation(), which in turn calls
>> DMPlexComputeInterpolatorNested/General.
>>
>>   Thanks,
>>
>>       Matt
>>
>>
>>> Thanks,
>>>
>>>
>>>> On Fri, Jul 8, 2022 at 11:26 PM Mike Michell <mi.mike1021 at gmail.com>
>>>> wrote:
>>>>
>>>>> I am using DMPlex for a code with written in Fortran in 2D and 3D.
>>>>> There were two questions.
>>>>>
>>>>> - As a follow up of the previous inquiry:
>>>>> https://www.mail-archive.com/petsc-users at mcs.anl.gov/msg43856.html
>>>>> Is the local-to-local halo exchange available in Fortran now or still
>>>>> pending? Currently local-to-global and global-to-local communication are
>>>>> used since local-to-local has not been allowed for Fortran.
>>>>>
>>>>
>>>> Sorry, it is still on my TODO list. I am trying to get stuff cleared
>>>> out.
>>>>
>>>>
>>>>> - One code solves discretized equations at each vertex, and another
>>>>> code I want to couple physics is solving equations at each cell centroid.
>>>>> Basically, the value at cell centroid needs to be mapped to vertex (or vice
>>>>> versa) through interpolation/extrapolation for coupling of two codes. Does
>>>>> petsc function provide this kind of mapping between cell centroid and
>>>>> vertex? The grids for the two codes can be overlapped. I was trying to find
>>>>> some FEM infrastructures in petsc, but so far havent found that kind of
>>>>> functionality. Can I get any comments on that?
>>>>>
>>>>
>>>> Yes, you can create both P0 and P1 discretizations
>>>> (PetscFECreateLagrange) in two different DMs using DMClone(), and then
>>>> create an interpolation operator (DMPlexCreateInterpolator) which maps
>>>> between them. Let me know if something is not clear there.
>>>>
>>>>   Thanks,
>>>>
>>>>     Matt
>>>>
>>>>
>>>>> Thanks,
>>>>> Mike
>>>>>
>>>>>
>>>>
>>>> --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>> https://www.cse.buffalo.edu/~knepley/
>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>
>>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/~knepley/>
>>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220826/095ca51c/attachment.html>

From knepley at gmail.com  Fri Aug 26 09:47:54 2022
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 26 Aug 2022 10:47:54 -0400
Subject: [petsc-users] DMPlex Interpolation
In-Reply-To: <CAEc7osYgsXxW9DN_6aHof3wdyqHw3Wv6nLp+svWsZF5c0zz93w@mail.gmail.com>
References: <CAEc7osYgsXxW9DN_6aHof3wdyqHw3Wv6nLp+svWsZF5c0zz93w@mail.gmail.com>
Message-ID: <CAMYG4GmiXaNTG5sgjoH=-CdY21OkRaKZh5y5OdPcSiEMj-KfXg@mail.gmail.com>

On Thu, Aug 25, 2022 at 7:12 PM Mike Michell <mi.mike1021 at gmail.com> wrote:

> Hi, this is a duplication of
> https://lists.mcs.anl.gov/pipermail/petsc-users/2022-August/046746.html
> for in-depth question.
>
> I wrote a short code as attached to test interpolation between two DMPlex
> objects. Goal is to map solution field defined on cell-centroid(DMPlex-1)
> into vertex(DMPlex-2) field or vice versa.
>
> Basically, DMCreateInterpolation() fails in the example code and it was
> not allowed to get useful error messages. The DMPlex is created by loading
> a 2D square box in Gmsh. Can I get some comments on that?
>

Sure, I am looking at it.

  Thanks,

     Matt


> Thanks,
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220826/00a617f4/attachment.html>

From lbgpublic at gmail.com  Fri Aug 26 09:49:06 2022
From: lbgpublic at gmail.com (byron)
Date: Fri, 26 Aug 2022 15:49:06 +0100
Subject: [petsc-users] unable to build petsc with cuda / superlu-dist
 enabled
In-Reply-To: <CAJ+YQyo2CCKeKiBjqgqWPqPzYZLyWDmfeS=uAYD819bK8-c8+A@mail.gmail.com>
References: <CAJ+YQyrctNbkcdbUD-DmzdQRzUx8w12yy2dBbuntLm4Awj6McQ@mail.gmail.com>
	<C4B564E7-0DD0-4FAC-A9F3-185F0229653F@petsc.dev>
	<CAJ+YQyr9uhyR0LvCsuDwcGEhDXb1k5kSKNgP4Z+77jf97Pmfhg@mail.gmail.com>
	<2616A45C-68CF-4C94-A880-B7858C9B7882@petsc.dev>
	<6BA4C5E3-29A4-4E68-A872-AE6A06185D64@gmail.com>
	<CAJ+YQyo2CCKeKiBjqgqWPqPzYZLyWDmfeS=uAYD819bK8-c8+A@mail.gmail.com>
Message-ID: <CAJ+YQyqeoBqESuJh_cJJzwdpNC5wxaaK-8QTWC==m7zK-015OA@mail.gmail.com>

Hi Jacob

I tried with "--with-cxx-dialect=auto" which did look to clean up all the
references to c++14 / c++11 from the options used to configure superlu-dist
but it still didnt work and looked to generate the same "nvcc fatal" errors.

Thanks
Byron

On Fri, Aug 26, 2022 at 2:47 PM byron <lbgpublic at gmail.com> wrote:

> Hi Barry
>
> I dont think I see the same thing.
>
> When I do a build using the release branch petsc passes the following
> arguements to superlu_dist (taken from
> /home/blahblah/Projects/220826-vanilla-petsc-release-branch/petsc/arch-linux-c-opt/externalpackages/git.superlu_dist/superlu_dist.petscconf)
> which has only one -O3 entry
>
> -DCMAKE_INSTALL_PREFIX=/home/blahblah/Projects/220826-vanilla-petsc-release-branch/installation
> -DCMAKE_INSTALL_NAME_DIR:STRING="/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib"
> -DCMAKE_INSTALL_LIBDIR:STRING="lib" -DCMAKE_VERBOSE_MAKEFILE=1
> -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER="mpicc"
> -DMPI_C_COMPILER="/usr/bin/mpicc" -DCMAKE_AR=/usr/bin/ar
> -DCMAKE_RANLIB=/usr/bin/ranlib -DCMAKE_C_FLAGS:STRING="-fPIC
> -Wno-lto-type-mismatch -fstack-protector -g -O -fopenmp  -DDEBUGlevel=0
> -DPRNTlevel=0" -DCMAKE_C_FLAGS_DEBUG:STRING="-fPIC -Wno-lto-type-mismatch
> -fstack-protector -g -O -fopenmp  -DDEBUGlevel=0 -DPRNTlevel=0"
> -DCMAKE_C_FLAGS_RELEASE:STRING="-fPIC -Wno-lto-type-mismatch
> -fstack-protector -g -O -fopenmp  -DDEBUGlevel=0 -DPRNTlevel=0"
> -DCMAKE_CXX_COMPILER="mpicxx" -DMPI_CXX_COMPILER="/usr/bin/mpicxx"
> -DCMAKE_CXX_FLAGS:STRING="-Wno-lto-type-mismatch -fstack-protector -g -O
> -fopenmp -std=c++11 -fPIC -fopenmp  -DDEBUGlevel=0 -DPRNTlevel=0"
> -DCMAKE_CXX_FLAGS_DEBUG:STRING="-Wno-lto-type-mismatch -fstack-protector -g
> -O -fopenmp -std=c++11 -fPIC -fopenmp  -DDEBUGlevel=0 -DPRNTlevel=0"
> -DCMAKE_CXX_FLAGS_RELEASE:STRING="-Wno-lto-type-mismatch -fstack-protector
> -g -O -fopenmp -std=c++11 -fPIC -fopenmp  -DDEBUGlevel=0 -DPRNTlevel=0"
> -DCMAKE_Fortran_COMPILER="mpif90" -DMPI_Fortran_COMPILER="/usr/bin/mpif90"
> -DCMAKE_Fortran_FLAGS:STRING="-fPIC -ffree-line-length-0
> -Wno-lto-type-mismatch -g -O -fopenmp"
> -DCMAKE_Fortran_FLAGS_DEBUG:STRING="-fPIC -ffree-line-length-0
> -Wno-lto-type-mismatch -g -O -fopenmp"
> -DCMAKE_Fortran_FLAGS_RELEASE:STRING="-fPIC -ffree-line-length-0
> -Wno-lto-type-mismatch -g -O -fopenmp" -DCMAKE_EXE_LINKER_FLAGS:STRING="
> -fopenmp -fopenmp" -DBUILD_SHARED_LIBS:BOOL=ON -DTPL_ENABLE_CUDALIB=TRUE
> -DTPL_CUDA_LIBRARIES="-lcudart -lcufft -lcublas -lcusparse -lcusolver
> -lcurand -lcuda" -DCUDA_ARCH_FLAGS="-arch=sm_75"
> -DCMAKE_CUDA_COMPILER="nvcc" -DCMAKE_CUDA_FLAGS=" -ccbin mpicxx -std=c++14
> -Xcompiler -fPIC *-O3 *-gencode arch=compute_75,code=sm_75
>  -Wno-deprecated-gpu-targets
> -I/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi
> -I/usr/lib/x86_64-linux-gnu/openmpi/include   -DDEBUGlevel=0 -DPRNTlevel=0"
> -DUSE_XSDK_DEFAULTS=YES
> -DTPL_BLAS_LIBRARIES="-Wl,-rpath,/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib
> -L/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib
> -lflapack
> -Wl,-rpath,/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib
> -L/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib
> -lfblas -lcudart -lcufft -lcublas -lcusparse -lcusolver -lcurand -lcuda -lm
> -lstdc++ -ldl -Wl,-rpath,/usr/lib/x86_64-linux-gnu/openmpi/lib
> -L/usr/lib/x86_64-linux-gnu/openmpi/lib -lmpi_usempif08
> -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi -lgfortran -lm
> -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/9
> -L/usr/lib/gcc/x86_64-linux-gnu/9 -lgfortran -lm -lgcc_s -lquadmath
> -lpthread"
> -DTPL_LAPACK_LIBRARIES="-Wl,-rpath,/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib
> -L/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib
> -lflapack
> -Wl,-rpath,/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib
> -L/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib
> -lfblas -lcudart -lcufft -lcublas -lcusparse -lcusolver -lcurand -lcuda -lm
> -lstdc++ -ldl -Wl,-rpath,/usr/lib/x86_64-linux-gnu/openmpi/lib
> -L/usr/lib/x86_64-linux-gnu/openmpi/lib -lmpi_usempif08
> -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi -lgfortran -lm
> -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/9
> -L/usr/lib/gcc/x86_64-linux-gnu/9 -lgfortran -lm -lgcc_s -lquadmath
> -lpthread"
> -DTPL_PARMETIS_INCLUDE_DIRS="/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/include;/usr/lib/x86_64-linux-gnu/openmpi/include;/usr/lib/x86_64-linux-gnu/openmpi/lib"
> -DTPL_PARMETIS_LIBRARIES="-Wl,-rpath,/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib
> -L/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib
> -lparmetis
> -Wl,-rpath,/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib
> -L/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib
> -lmetis -lm -lcudart -lcufft -lcublas -lcusparse -lcusolver -lcurand
> -lcuda" -DXSDK_INDEX_SIZE=64 -DXSDK_ENABLE_Fortran=ON -Denable_tests=0
> -Denable_examples=0 -DMPI_C_COMPILE_FLAGS:STRING=""
> -DMPI_C_INCLUDE_PATH:STRING="" -DMPI_C_HEADER_DIR:STRING=""
> -DMPI_C_LIBRARIES:STRING=""
>
> Or am I looking at the wrong thing?
>
> I also tried doing the build downloading petsc from the main branch but
> that died at the same point and looked to give the same "redefinition of
> argument 'optimize'" errors
>
> Thanks
> Byron
>
>
>
> On Thu, Aug 25, 2022 at 4:15 PM Jacob Faibussowitsch <jacob.fai at gmail.com>
> wrote:
>
>> > Sadly each one makes different -std=c++11 vs  -std=c++14 choices as
>> well.
>>
>> Main should hopefully *also* warn about this, but unless you really
>> really want -std=c++11 or -std=c++14, it is better to leave out this
>> configure flag (--with-cxx-dialect).
>>
>> If --with-cxx-dialect is unspecified (or set to ?auto?) configure will
>> pick the most appropriate flag for you, taking your package environment
>> into account when choosing. If you specify the flag configure must assume
>> the user knows something it doesn?t and will blindly take what is given.
>>
>> Best regards,
>>
>> Jacob Faibussowitsch
>> (Jacob Fai - booss - oh - vitch)
>>
>> > On Aug 25, 2022, at 11:05, Barry Smith <bsmith at petsc.dev> wrote:
>> >
>> >
>> >   Oh yes, my mistake I misinterpreted the error message. It is actually
>> the nvcc compiler complaining about multiple command line arguments related
>> to optimization (which it is really picky about even if the two arguments
>> are identical.)  I reformatted some of the configure output to show this
>> >
>> > -ccbin mpicxx -std=c++14 -Xcompiler -fPIC -O3 -gencode
>> arch=compute_75,code=sm_75  -Wno-deprecated-gpu-targets
>> > -I/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi
>> -I/usr/lib/x86_64-linux-gnu/openmpi/include
>> > -DDEBUGlevel=0 -DPRNTlevel=0
>> >
>> > the next fragment is
>> >
>> > -O3 -DNDEBUG
>> >
>> > --generate-code=arch=compute_75,code=[compute_75,sm_75]
>> >
>> > -Xcompiler=-fPIC -std=c++11 -x cu -rdc=true -c
>> /home/blahblah/Projects/220824-petsc-example-for_forum/petsc/arch-linux-c-opt/externalpackages/git.superlu_dist/SRC/
>> superlu_gpu_utils.cu -o
>> >
>> > Hence the -O3 appears twice, once from the CMAKE argument PETSc passes
>> to superlu_dist
>> >
>> > -DCMAKE_CUDA_FLAGS=" -ccbin mpicxx -std=c++14 -Xcompiler -fPIC -O3
>> -gencode arch=compute_75,code=sm_75  -Wno-deprecated-gpu-targets
>> -I/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi
>> -I/usr/lib/x86_64-linux-gnu/openmpi/include   -DDEBUGlevel=0 -DPRNTlevel=0"
>> >
>> > and then presumably, the superlu_dist cmake process adds its own
>> arguments which overlap.
>> >
>> > If I remember correctly this is fixed in the main branch of PETSc,
>> could you do an identical build with the main branch? If that works we will
>> just need to figure out how to get MOOSE to work with main PETSc branch.
>> >
>> >   Barry
>> >
>> > Sadly each one makes different -std=c++11 vs  -std=c++14 choices as
>> well.
>> >
>> >
>> > --------------------
>> >
>> > [ 15%] Building CUDA object
>> SRC/CMakeFiles/superlu_dist.dir/superlu_gpu_utils.cu.o
>> > cd
>> /home/blahblah/Projects/220824-petsc-example-for_forum/petsc/arch-linux-c-opt/externalpackages/git.superlu_dist/petsc-build/SRC
>> && /usr/bin/nvcc
>> >
>> > -DSUPERLU_DIST_EXPORTS -Dsuperlu_dist_EXPORTS
>> >
>> >
>> -I/home/blahblah/Projects/220824-petsc-example-for_forum/petsc/arch-linux-c-opt/externalpackages/git.superlu_dist/petsc-build/SRC
>> >
>> > -I/home/blahblah/Projects/220824-petsc-build/installation/include
>> -I/usr/lib/x86_64-linux-gnu/openmpi/lib
>> >
>> > -DUSE_VENDOR_BLAS  -ccbin mpicxx -std=c++14 -Xcompiler -fPIC -O3
>> -gencode arch=compute_75,code=sm_75  -Wno-deprecated-gpu-targets
>> >
>> > -I/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi
>> -I/usr/lib/x86_64-linux-gnu/openmpi/include
>> >
>> > -DDEBUGlevel=0 -DPRNTlevel=0 -O3 -DNDEBUG
>> >
>> > --generate-code=arch=compute_75,code=[compute_75,sm_75]
>> >
>> > -Xcompiler=-fPIC -std=c++11 -x cu -rdc=true -c
>> /home/blahblah/Projects/220824-petsc-example-for_forum/petsc/arch-linux-c-opt/externalpackages/git.superlu_dist/SRC/
>> superlu_gpu_utils.cu -o
>> >
>> > CMakeFiles/superlu_dist.dir/superlu_gpu_utils.cu
>> >
>> >
>> >
>> >> On Aug 25, 2022, at 9:38 AM, byron <lbgpublic at gmail.com> wrote:
>> >>
>> >> Hi Barry
>> >>
>> >> There doesn't appear to be any reference to 'optimize' in the only
>> copy of  superlu_gpu_utils.cu I could find, I've attached a copy.
>> >>
>> >> Thanks
>> >>
>> >>
>> >>
>> >> On Thu, Aug 25, 2022 at 1:44 PM Barry Smith <bsmith at petsc.dev> wrote:
>> >>
>> >> nvcc fatal   : redefinition of argument 'optimize'
>> >> make[2]: *** [SRC/CMakeFiles/superlu_dist.dir/build.make:499:
>> SRC/CMakeFiles/superlu_dist.dir/superlu_gpu_utils.cu.o] Error 1
>> >> make[2]: *** Waiting for unfinished jobs....
>> >> nvcc fatal   : redefinition of argument 'optimize'
>> >> make[2]: *** [SRC/CMakeFiles/superlu_dist-static.dir/build.make:499:
>> SRC/CMakeFiles/superlu_dist-static.dir/superlu_gpu_utils.cu.o] Error 1
>> >> make[2]: *** Waiting for unfinished jobs....
>> >>
>> >> It seems that nvcc is unhappy with something in superlu_gpu_utils.cu
>> Could you check that file for any "redefinition of optimize"? Unfortunately
>> the nvcc compiler isn't kind enough to print the line number of the
>> redefinition or that information is lost somehow in the build process.
>> >>
>> >> Barry
>> >>
>> >>> On Aug 25, 2022, at 6:50 AM, byron <lbgpublic at gmail.com> wrote:
>> >>>
>> >>> Hi All
>> >>>
>> >>> I am having trouble getting petsc to build with cuda / superlu-dist
>> enabled.  I have been trying to build it as part of another software tool,
>> moose, hence the choice of options being passed to configure
>> >>>
>> >>> python3 ./configure --download-hypre=1 \
>> >>> --with-shared-libraries=1 \
>> >>> --download-hdf5=1 \
>> >>> --download-hdf5-fortran-bindings=0 \
>> >>> --with-debugging=no \
>> >>> --download-fblaslapack=1 \
>> >>> --download-metis=1 \
>> >>> --download-ptscotch=1 \
>> >>> --download-parmetis=1 \
>> >>> --download-superlu_dist=1 \
>> >>> --download-mumps=1 \
>> >>> --download-strumpack=1 \
>> >>> --download-scalapack=1 \
>> >>> --download-slepc=1 \
>> >>> --with-mpi=1 \
>> >>> --with-openmp=1 \
>> >>> --with-cxx-dialect=C++11 \
>> >>> --with-fortran-bindings=0 \
>> >>> --with-sowing=0 \
>> >>> --with-64-bit-indices \
>> >>> --with-cuda \
>> >>> --prefix=/home/blah/blah
>> >>>
>> >>> it fails with
>> >>>
>> >>>
>> =============================================================================================
>>
>>                                Installing PTScotch; this may take several
>> minutes
>>
>> =============================================================================================
>>
>>
>>  =============================================================================================
>>
>>                                Trying to download git://
>> https://github.com/xiaoyeli/superlu_dist for SUPERLU_DIST
>>
>>
>> =============================================================================================
>>
>>
>>  =============================================================================================
>>
>>                                Configuring SUPERLU_DIST with cmake; this
>> may take several minutes
>>
>> =============================================================================================
>>
>>
>>  =============================================================================================
>>
>>                                Compiling and installing SUPERLU_DIST; this
>> may take several minutes
>>
>> =============================================================================================
>>
>>
>>  *******************************************************************************
>> >>>          UNABLE to CONFIGURE with GIVEN OPTIONS    (see configure.log
>> for details):
>> >>>
>> -------------------------------------------------------------------------------
>> >>> Error running make on  SUPERLU_DIST
>> >>>
>> *******************************************************************************
>> >>>
>> >>> I've attached the configure.log.
>> >>>
>> >>> I've tried a few different things but nothing that resulted in a
>> successfull build.  The most promising was I tried to build superlu_dist
>> separately with the configure options that petsc passes it and then kept
>> trying the build over and over droping out different configure options
>> until I got it to work.  In that case removing a "-std=c++14" entry
>> appeared to fix the problem but then when I made the same change in how
>> petsc builds superlu-dist the make failed on superlu-dist with a different
>> error.  I get the feeling I'm dealing with symptoms of the problem rather
>> than the cause.
>> >>>
>> >>> Thanks
>> >>> Byron
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> <configure.log>
>> >>
>> >> <superlu_gpu_utils.cu>
>> >
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220826/17116c1f/attachment-0001.html>

From bsmith at petsc.dev  Fri Aug 26 10:06:15 2022
From: bsmith at petsc.dev (Barry Smith)
Date: Fri, 26 Aug 2022 11:06:15 -0400
Subject: [petsc-users] unable to build petsc with cuda / superlu-dist
 enabled
In-Reply-To: <CAJ+YQyo2CCKeKiBjqgqWPqPzYZLyWDmfeS=uAYD819bK8-c8+A@mail.gmail.com>
References: <CAJ+YQyrctNbkcdbUD-DmzdQRzUx8w12yy2dBbuntLm4Awj6McQ@mail.gmail.com>
	<C4B564E7-0DD0-4FAC-A9F3-185F0229653F@petsc.dev>
	<CAJ+YQyr9uhyR0LvCsuDwcGEhDXb1k5kSKNgP4Z+77jf97Pmfhg@mail.gmail.com>
	<2616A45C-68CF-4C94-A880-B7858C9B7882@petsc.dev>
	<6BA4C5E3-29A4-4E68-A872-AE6A06185D64@gmail.com>
	<CAJ+YQyo2CCKeKiBjqgqWPqPzYZLyWDmfeS=uAYD819bK8-c8+A@mail.gmail.com>
Message-ID: <A50D7893-39DF-46AA-B1C9-BFBF508266EA@petsc.dev>


  Yes, PETSc sends down a single -O3 to SuperLU_DIST CMake but internally SuperLU_DIST CMake is adding its own set of nvcc compiler flags to use and these that get added to the one's PETSc passes in (they do not replace them). A little further down in the file it displays a compile line with nvcc that has two sets of -O3 (as well as two sets of other things as well).

  I think we changed the main branch PETSc to not pass down the optimization flags to the CMake CUDA to prevent the problem from happening. In other words, we let the package decide its optimization flags instead of trying to tell it what to use. 

  Barry

The root of the problem is that compiler APIs are a bunch of strings passed on the command line to the compiler without any structure.


> On Aug 26, 2022, at 9:47 AM, byron <lbgpublic at gmail.com> wrote:
> 
> Hi Barry 
> 
> I dont think I see the same thing.
> 
> When I do a build using the release branch petsc passes the following arguements to superlu_dist (taken from /home/blahblah/Projects/220826-vanilla-petsc-release-branch/petsc/arch-linux-c-opt/externalpackages/git.superlu_dist/superlu_dist.petscconf)  which has only one -O3 entry
> 
> -DCMAKE_INSTALL_PREFIX=/home/blahblah/Projects/220826-vanilla-petsc-release-branch/installation -DCMAKE_INSTALL_NAME_DIR:STRING="/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib" -DCMAKE_INSTALL_LIBDIR:STRING="lib" -DCMAKE_VERBOSE_MAKEFILE=1 -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER="mpicc" -DMPI_C_COMPILER="/usr/bin/mpicc" -DCMAKE_AR=/usr/bin/ar -DCMAKE_RANLIB=/usr/bin/ranlib -DCMAKE_C_FLAGS:STRING="-fPIC -Wno-lto-type-mismatch -fstack-protector -g -O -fopenmp  -DDEBUGlevel=0 -DPRNTlevel=0" -DCMAKE_C_FLAGS_DEBUG:STRING="-fPIC -Wno-lto-type-mismatch -fstack-protector -g -O -fopenmp  -DDEBUGlevel=0 -DPRNTlevel=0" -DCMAKE_C_FLAGS_RELEASE:STRING="-fPIC -Wno-lto-type-mismatch -fstack-protector -g -O -fopenmp  -DDEBUGlevel=0 -DPRNTlevel=0" -DCMAKE_CXX_COMPILER="mpicxx" -DMPI_CXX_COMPILER="/usr/bin/mpicxx" -DCMAKE_CXX_FLAGS:STRING="-Wno-lto-type-mismatch -fstack-protector -g -O -fopenmp -std=c++11 -fPIC -fopenmp  -DDEBUGlevel=0 -DPRNTlevel=0" -DCMAKE_CXX_FLAGS_DEBUG:STRING="-Wno-lto-type-mismatch -fstack-protector -g -O -fopenmp -std=c++11 -fPIC -fopenmp  -DDEBUGlevel=0 -DPRNTlevel=0" -DCMAKE_CXX_FLAGS_RELEASE:STRING="-Wno-lto-type-mismatch -fstack-protector -g -O -fopenmp -std=c++11 -fPIC -fopenmp  -DDEBUGlevel=0 -DPRNTlevel=0" -DCMAKE_Fortran_COMPILER="mpif90" -DMPI_Fortran_COMPILER="/usr/bin/mpif90" -DCMAKE_Fortran_FLAGS:STRING="-fPIC -ffree-line-length-0 -Wno-lto-type-mismatch -g -O -fopenmp" -DCMAKE_Fortran_FLAGS_DEBUG:STRING="-fPIC -ffree-line-length-0 -Wno-lto-type-mismatch -g -O -fopenmp" -DCMAKE_Fortran_FLAGS_RELEASE:STRING="-fPIC -ffree-line-length-0 -Wno-lto-type-mismatch -g -O -fopenmp" -DCMAKE_EXE_LINKER_FLAGS:STRING=" -fopenmp -fopenmp" -DBUILD_SHARED_LIBS:BOOL=ON -DTPL_ENABLE_CUDALIB=TRUE -DTPL_CUDA_LIBRARIES="-lcudart -lcufft -lcublas -lcusparse -lcusolver -lcurand -lcuda" -DCUDA_ARCH_FLAGS="-arch=sm_75" -DCMAKE_CUDA_COMPILER="nvcc" -DCMAKE_CUDA_FLAGS=" -ccbin mpicxx -std=c++14 -Xcompiler -fPIC -O3 -gencode arch=compute_75,code=sm_75  -Wno-deprecated-gpu-targets -I/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi -I/usr/lib/x86_64-linux-gnu/openmpi/include   -DDEBUGlevel=0 -DPRNTlevel=0" -DUSE_XSDK_DEFAULTS=YES -DTPL_BLAS_LIBRARIES="-Wl,-rpath,/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib -L/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib -lflapack -Wl,-rpath,/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib -L/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib -lfblas -lcudart -lcufft -lcublas -lcusparse -lcusolver -lcurand -lcuda -lm -lstdc++ -ldl -Wl,-rpath,/usr/lib/x86_64-linux-gnu/openmpi/lib -L/usr/lib/x86_64-linux-gnu/openmpi/lib -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi -lgfortran -lm -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/9 -L/usr/lib/gcc/x86_64-linux-gnu/9 -lgfortran -lm -lgcc_s -lquadmath -lpthread" -DTPL_LAPACK_LIBRARIES="-Wl,-rpath,/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib -L/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib -lflapack -Wl,-rpath,/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib -L/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib -lfblas -lcudart -lcufft -lcublas -lcusparse -lcusolver -lcurand -lcuda -lm -lstdc++ -ldl -Wl,-rpath,/usr/lib/x86_64-linux-gnu/openmpi/lib -L/usr/lib/x86_64-linux-gnu/openmpi/lib -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi -lgfortran -lm -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/9 -L/usr/lib/gcc/x86_64-linux-gnu/9 -lgfortran -lm -lgcc_s -lquadmath -lpthread" -DTPL_PARMETIS_INCLUDE_DIRS="/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/include;/usr/lib/x86_64-linux-gnu/openmpi/include;/usr/lib/x86_64-linux-gnu/openmpi/lib" -DTPL_PARMETIS_LIBRARIES="-Wl,-rpath,/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib -L/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib -lparmetis -Wl,-rpath,/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib -L/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib -lmetis -lm -lcudart -lcufft -lcublas -lcusparse -lcusolver -lcurand -lcuda" -DXSDK_INDEX_SIZE=64 -DXSDK_ENABLE_Fortran=ON -Denable_tests=0 -Denable_examples=0 -DMPI_C_COMPILE_FLAGS:STRING="" -DMPI_C_INCLUDE_PATH:STRING="" -DMPI_C_HEADER_DIR:STRING="" -DMPI_C_LIBRARIES:STRING=""
> 
> Or am I looking at the wrong thing?
> 
> I also tried doing the build downloading petsc from the main branch but that died at the same point and looked to give the same "redefinition of argument 'optimize'" errors
> 
> Thanks
> Byron
> 
> 
> 
> On Thu, Aug 25, 2022 at 4:15 PM Jacob Faibussowitsch <jacob.fai at gmail.com <mailto:jacob.fai at gmail.com>> wrote:
> > Sadly each one makes different -std=c++11 vs  -std=c++14 choices as well.
> 
> Main should hopefully *also* warn about this, but unless you really really want -std=c++11 or -std=c++14, it is better to leave out this configure flag (--with-cxx-dialect). 
> 
> If --with-cxx-dialect is unspecified (or set to ?auto?) configure will pick the most appropriate flag for you, taking your package environment into account when choosing. If you specify the flag configure must assume the user knows something it doesn?t and will blindly take what is given.
> 
> Best regards,
> 
> Jacob Faibussowitsch
> (Jacob Fai - booss - oh - vitch)
> 
> > On Aug 25, 2022, at 11:05, Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>> wrote:
> > 
> > 
> >   Oh yes, my mistake I misinterpreted the error message. It is actually the nvcc compiler complaining about multiple command line arguments related to optimization (which it is really picky about even if the two arguments are identical.)  I reformatted some of the configure output to show this
> > 
> > -ccbin mpicxx -std=c++14 -Xcompiler -fPIC -O3 -gencode arch=compute_75,code=sm_75  -Wno-deprecated-gpu-targets
> > -I/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi -I/usr/lib/x86_64-linux-gnu/openmpi/include
> > -DDEBUGlevel=0 -DPRNTlevel=0
> > 
> > the next fragment is 
> > 
> > -O3 -DNDEBUG
> > 
> > --generate-code=arch=compute_75,code=[compute_75,sm_75]
> > 
> > -Xcompiler=-fPIC -std=c++11 -x cu -rdc=true -c /home/blahblah/Projects/220824-petsc-example-for_forum/petsc/arch-linux-c-opt/externalpackages/git.superlu_dist/SRC/superlu_gpu_utils.cu <http://superlu_gpu_utils.cu/> -o
> > 
> > Hence the -O3 appears twice, once from the CMAKE argument PETSc passes to superlu_dist 
> > 
> > -DCMAKE_CUDA_FLAGS=" -ccbin mpicxx -std=c++14 -Xcompiler -fPIC -O3 -gencode arch=compute_75,code=sm_75  -Wno-deprecated-gpu-targets -I/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi -I/usr/lib/x86_64-linux-gnu/openmpi/include   -DDEBUGlevel=0 -DPRNTlevel=0"
> > 
> > and then presumably, the superlu_dist cmake process adds its own arguments which overlap.
> > 
> > If I remember correctly this is fixed in the main branch of PETSc, could you do an identical build with the main branch? If that works we will just need to figure out how to get MOOSE to work with main PETSc branch.
> > 
> >   Barry
> > 
> > Sadly each one makes different -std=c++11 vs  -std=c++14 choices as well.
> > 
> > 
> > --------------------
> > 
> > [ 15%] Building CUDA object SRC/CMakeFiles/superlu_dist.dir/superlu_gpu_utils.cu.o
> > cd /home/blahblah/Projects/220824-petsc-example-for_forum/petsc/arch-linux-c-opt/externalpackages/git.superlu_dist/petsc-build/SRC && /usr/bin/nvcc
> > 
> > -DSUPERLU_DIST_EXPORTS -Dsuperlu_dist_EXPORTS
> > 
> > -I/home/blahblah/Projects/220824-petsc-example-for_forum/petsc/arch-linux-c-opt/externalpackages/git.superlu_dist/petsc-build/SRC
> > 
> > -I/home/blahblah/Projects/220824-petsc-build/installation/include -I/usr/lib/x86_64-linux-gnu/openmpi/lib
> > 
> > -DUSE_VENDOR_BLAS  -ccbin mpicxx -std=c++14 -Xcompiler -fPIC -O3 -gencode arch=compute_75,code=sm_75  -Wno-deprecated-gpu-targets
> > 
> > -I/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi -I/usr/lib/x86_64-linux-gnu/openmpi/include
> > 
> > -DDEBUGlevel=0 -DPRNTlevel=0 -O3 -DNDEBUG
> > 
> > --generate-code=arch=compute_75,code=[compute_75,sm_75]
> > 
> > -Xcompiler=-fPIC -std=c++11 -x cu -rdc=true -c /home/blahblah/Projects/220824-petsc-example-for_forum/petsc/arch-linux-c-opt/externalpackages/git.superlu_dist/SRC/superlu_gpu_utils.cu <http://superlu_gpu_utils.cu/> -o
> > 
> > CMakeFiles/superlu_dist.dir/superlu_gpu_utils.cu <http://superlu_gpu_utils.cu/>
> > 
> > 
> > 
> >> On Aug 25, 2022, at 9:38 AM, byron <lbgpublic at gmail.com <mailto:lbgpublic at gmail.com>> wrote:
> >> 
> >> Hi Barry 
> >> 
> >> There doesn't appear to be any reference to 'optimize' in the only copy of  superlu_gpu_utils.cu <http://superlu_gpu_utils.cu/> I could find, I've attached a copy.
> >> 
> >> Thanks
> >> 
> >> 
> >> 
> >> On Thu, Aug 25, 2022 at 1:44 PM Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>> wrote:
> >> 
> >> nvcc fatal   : redefinition of argument 'optimize'
> >> make[2]: *** [SRC/CMakeFiles/superlu_dist.dir/build.make:499: SRC/CMakeFiles/superlu_dist.dir/superlu_gpu_utils.cu.o] Error 1
> >> make[2]: *** Waiting for unfinished jobs....
> >> nvcc fatal   : redefinition of argument 'optimize'
> >> make[2]: *** [SRC/CMakeFiles/superlu_dist-static.dir/build.make:499: SRC/CMakeFiles/superlu_dist-static.dir/superlu_gpu_utils.cu.o] Error 1
> >> make[2]: *** Waiting for unfinished jobs....
> >> 
> >> It seems that nvcc is unhappy with something in superlu_gpu_utils.cu <http://superlu_gpu_utils.cu/>  Could you check that file for any "redefinition of optimize"? Unfortunately the nvcc compiler isn't kind enough to print the line number of the redefinition or that information is lost somehow in the build process.
> >> 
> >> Barry
> >> 
> >>> On Aug 25, 2022, at 6:50 AM, byron <lbgpublic at gmail.com <mailto:lbgpublic at gmail.com>> wrote:
> >>> 
> >>> Hi All
> >>> 
> >>> I am having trouble getting petsc to build with cuda / superlu-dist enabled.  I have been trying to build it as part of another software tool, moose, hence the choice of options being passed to configure
> >>> 
> >>> python3 ./configure --download-hypre=1 \
> >>> --with-shared-libraries=1 \
> >>> --download-hdf5=1 \
> >>> --download-hdf5-fortran-bindings=0 \
> >>> --with-debugging=no \
> >>> --download-fblaslapack=1 \
> >>> --download-metis=1 \
> >>> --download-ptscotch=1 \
> >>> --download-parmetis=1 \
> >>> --download-superlu_dist=1 \
> >>> --download-mumps=1 \
> >>> --download-strumpack=1 \
> >>> --download-scalapack=1 \
> >>> --download-slepc=1 \
> >>> --with-mpi=1 \
> >>> --with-openmp=1 \
> >>> --with-cxx-dialect=C++11 \
> >>> --with-fortran-bindings=0 \
> >>> --with-sowing=0 \
> >>> --with-64-bit-indices \
> >>> --with-cuda \
> >>> --prefix=/home/blah/blah
> >>> 
> >>> it fails with 
> >>> 
> >>> =============================================================================================                                                                                                             Installing PTScotch; this may take several minutes                                                                                                                                            =============================================================================================                                                                                                       =============================================================================================                                                                                                             Trying to download git://https://github.com/xiaoyeli/superlu_dist <https://github.com/xiaoyeli/superlu_dist> for SUPERLU_DIST                                                                                                            =============================================================================================                                                                                                       =============================================================================================                                                                                                             Configuring SUPERLU_DIST with cmake; this may take several minutes                                                                                                                            =============================================================================================                                                                                                       =============================================================================================                                                                                                             Compiling and installing SUPERLU_DIST; this may take several minutes                                                                                                                          =============================================================================================                                                                                                                             *******************************************************************************
> >>>          UNABLE to CONFIGURE with GIVEN OPTIONS    (see configure.log for details):
> >>> -------------------------------------------------------------------------------
> >>> Error running make on  SUPERLU_DIST
> >>> *******************************************************************************
> >>> 
> >>> I've attached the configure.log.
> >>> 
> >>> I've tried a few different things but nothing that resulted in a successfull build.  The most promising was I tried to build superlu_dist separately with the configure options that petsc passes it and then kept trying the build over and over droping out different configure options until I got it to work.  In that case removing a "-std=c++14" entry appeared to fix the problem but then when I made the same change in how petsc builds superlu-dist the make failed on superlu-dist with a different error.  I get the feeling I'm dealing with symptoms of the problem rather than the cause.  
> >>> 
> >>> Thanks
> >>> Byron
> >>>           
> >>> 
> >>> 
> >>>  
> >>> <configure.log>
> >> 
> >> <superlu_gpu_utils.cu <http://superlu_gpu_utils.cu/>>
> > 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220826/99f860c2/attachment-0001.html>

From lbgpublic at gmail.com  Fri Aug 26 11:13:35 2022
From: lbgpublic at gmail.com (byron)
Date: Fri, 26 Aug 2022 17:13:35 +0100
Subject: [petsc-users] unable to build petsc with cuda / superlu-dist
 enabled
In-Reply-To: <A50D7893-39DF-46AA-B1C9-BFBF508266EA@petsc.dev>
References: <CAJ+YQyrctNbkcdbUD-DmzdQRzUx8w12yy2dBbuntLm4Awj6McQ@mail.gmail.com>
	<C4B564E7-0DD0-4FAC-A9F3-185F0229653F@petsc.dev>
	<CAJ+YQyr9uhyR0LvCsuDwcGEhDXb1k5kSKNgP4Z+77jf97Pmfhg@mail.gmail.com>
	<2616A45C-68CF-4C94-A880-B7858C9B7882@petsc.dev>
	<6BA4C5E3-29A4-4E68-A872-AE6A06185D64@gmail.com>
	<CAJ+YQyo2CCKeKiBjqgqWPqPzYZLyWDmfeS=uAYD819bK8-c8+A@mail.gmail.com>
	<A50D7893-39DF-46AA-B1C9-BFBF508266EA@petsc.dev>
Message-ID: <CAJ+YQyq8rwwZ9fWG_V=Vu5Vdrd_N5UGL79ZbDW0pu7YwsUikkw@mail.gmail.com>

It looks like I've (we've) cracked it, at least the petsc stage of what I'm
trying to do.  To get the configure to work I used the following

python3 ./configure --download-hypre=1 --with-shared-libraries=1
--download-hdf5=1 --download-hdf5-fortran-bindings=0 --with-debugging=no
--download-fblaslapack=1 --download-metis=1 --download-ptscotch=1
--download-parmetis=1 --download-superlu_dist=1 --download-mumps=1
--download-strumpack=1 --download-scalapack=1 --download-slepc=1
--with-mpi=1 --with-openmp=1 --with-cxx-dialect=C++11
-with-fortran-bindings=0 --with-sowing=0 --with-64-bit-indices --with-cuda
-CUDAOPTFLAGS="" --with-cuda-dialect=cxx11
--prefix=/home/lgoodman/Projects/petscExamples/main-build-CUDAOPTFLAGS-zero-cxx-dialect-unspc-cuda-dial-11/installation

Where -CUDAOPTFLAGS=""  was the solutions to getting rid of the duplicate
-O3

then I was getting errors about 'redefinition of argument 'std'"  so i added

--with-cxx-dialect=C++11
and
--with-cuda-dialect=cxx11

and that got rid of those errors.  I image there are cleaner / more elegant
ways of doing this but I'll settle for this for now.

Thanks a lot for the help guys, much appreciated

Best regards
Byron

On Fri, Aug 26, 2022 at 4:06 PM Barry Smith <bsmith at petsc.dev> wrote:

>
>   Yes, PETSc sends down a single -O3 to SuperLU_DIST CMake but internally
> SuperLU_DIST CMake is adding its own set of nvcc compiler flags to use and
> these that get added to the one's PETSc passes in (they do not replace
> them). A little further down in the file it displays a compile line with
> nvcc that has two sets of -O3 (as well as two sets of other things as well).
>
>   I think we changed the main branch PETSc to not pass down the
> optimization flags to the CMake CUDA to prevent the problem from happening.
> In other words, we let the package decide its optimization flags instead of
> trying to tell it what to use.
>
>   Barry
>
> The root of the problem is that compiler APIs are a bunch of strings
> passed on the command line to the compiler without any structure.
>
>
> On Aug 26, 2022, at 9:47 AM, byron <lbgpublic at gmail.com> wrote:
>
> Hi Barry
>
> I dont think I see the same thing.
>
> When I do a build using the release branch petsc passes the following
> arguements to superlu_dist (taken from
> /home/blahblah/Projects/220826-vanilla-petsc-release-branch/petsc/arch-linux-c-opt/externalpackages/git.superlu_dist/superlu_dist.petscconf)
> which has only one -O3 entry
>
> -DCMAKE_INSTALL_PREFIX=/home/blahblah/Projects/220826-vanilla-petsc-release-branch/installation
> -DCMAKE_INSTALL_NAME_DIR:STRING="/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib"
> -DCMAKE_INSTALL_LIBDIR:STRING="lib" -DCMAKE_VERBOSE_MAKEFILE=1
> -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER="mpicc"
> -DMPI_C_COMPILER="/usr/bin/mpicc" -DCMAKE_AR=/usr/bin/ar
> -DCMAKE_RANLIB=/usr/bin/ranlib -DCMAKE_C_FLAGS:STRING="-fPIC
> -Wno-lto-type-mismatch -fstack-protector -g -O -fopenmp  -DDEBUGlevel=0
> -DPRNTlevel=0" -DCMAKE_C_FLAGS_DEBUG:STRING="-fPIC -Wno-lto-type-mismatch
> -fstack-protector -g -O -fopenmp  -DDEBUGlevel=0 -DPRNTlevel=0"
> -DCMAKE_C_FLAGS_RELEASE:STRING="-fPIC -Wno-lto-type-mismatch
> -fstack-protector -g -O -fopenmp  -DDEBUGlevel=0 -DPRNTlevel=0"
> -DCMAKE_CXX_COMPILER="mpicxx" -DMPI_CXX_COMPILER="/usr/bin/mpicxx"
> -DCMAKE_CXX_FLAGS:STRING="-Wno-lto-type-mismatch -fstack-protector -g -O
> -fopenmp -std=c++11 -fPIC -fopenmp  -DDEBUGlevel=0 -DPRNTlevel=0"
> -DCMAKE_CXX_FLAGS_DEBUG:STRING="-Wno-lto-type-mismatch -fstack-protector -g
> -O -fopenmp -std=c++11 -fPIC -fopenmp  -DDEBUGlevel=0 -DPRNTlevel=0"
> -DCMAKE_CXX_FLAGS_RELEASE:STRING="-Wno-lto-type-mismatch -fstack-protector
> -g -O -fopenmp -std=c++11 -fPIC -fopenmp  -DDEBUGlevel=0 -DPRNTlevel=0"
> -DCMAKE_Fortran_COMPILER="mpif90" -DMPI_Fortran_COMPILER="/usr/bin/mpif90"
> -DCMAKE_Fortran_FLAGS:STRING="-fPIC -ffree-line-length-0
> -Wno-lto-type-mismatch -g -O -fopenmp"
> -DCMAKE_Fortran_FLAGS_DEBUG:STRING="-fPIC -ffree-line-length-0
> -Wno-lto-type-mismatch -g -O -fopenmp"
> -DCMAKE_Fortran_FLAGS_RELEASE:STRING="-fPIC -ffree-line-length-0
> -Wno-lto-type-mismatch -g -O -fopenmp" -DCMAKE_EXE_LINKER_FLAGS:STRING="
> -fopenmp -fopenmp" -DBUILD_SHARED_LIBS:BOOL=ON -DTPL_ENABLE_CUDALIB=TRUE
> -DTPL_CUDA_LIBRARIES="-lcudart -lcufft -lcublas -lcusparse -lcusolver
> -lcurand -lcuda" -DCUDA_ARCH_FLAGS="-arch=sm_75"
> -DCMAKE_CUDA_COMPILER="nvcc" -DCMAKE_CUDA_FLAGS=" -ccbin mpicxx -std=c++14
> -Xcompiler -fPIC *-O3 *-gencode arch=compute_75,code=sm_75
>  -Wno-deprecated-gpu-targets
> -I/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi
> -I/usr/lib/x86_64-linux-gnu/openmpi/include   -DDEBUGlevel=0 -DPRNTlevel=0"
> -DUSE_XSDK_DEFAULTS=YES
> -DTPL_BLAS_LIBRARIES="-Wl,-rpath,/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib
> -L/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib
> -lflapack
> -Wl,-rpath,/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib
> -L/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib
> -lfblas -lcudart -lcufft -lcublas -lcusparse -lcusolver -lcurand -lcuda -lm
> -lstdc++ -ldl -Wl,-rpath,/usr/lib/x86_64-linux-gnu/openmpi/lib
> -L/usr/lib/x86_64-linux-gnu/openmpi/lib -lmpi_usempif08
> -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi -lgfortran -lm
> -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/9
> -L/usr/lib/gcc/x86_64-linux-gnu/9 -lgfortran -lm -lgcc_s -lquadmath
> -lpthread"
> -DTPL_LAPACK_LIBRARIES="-Wl,-rpath,/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib
> -L/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib
> -lflapack
> -Wl,-rpath,/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib
> -L/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib
> -lfblas -lcudart -lcufft -lcublas -lcusparse -lcusolver -lcurand -lcuda -lm
> -lstdc++ -ldl -Wl,-rpath,/usr/lib/x86_64-linux-gnu/openmpi/lib
> -L/usr/lib/x86_64-linux-gnu/openmpi/lib -lmpi_usempif08
> -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi -lgfortran -lm
> -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/9
> -L/usr/lib/gcc/x86_64-linux-gnu/9 -lgfortran -lm -lgcc_s -lquadmath
> -lpthread"
> -DTPL_PARMETIS_INCLUDE_DIRS="/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/include;/usr/lib/x86_64-linux-gnu/openmpi/include;/usr/lib/x86_64-linux-gnu/openmpi/lib"
> -DTPL_PARMETIS_LIBRARIES="-Wl,-rpath,/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib
> -L/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib
> -lparmetis
> -Wl,-rpath,/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib
> -L/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib
> -lmetis -lm -lcudart -lcufft -lcublas -lcusparse -lcusolver -lcurand
> -lcuda" -DXSDK_INDEX_SIZE=64 -DXSDK_ENABLE_Fortran=ON -Denable_tests=0
> -Denable_examples=0 -DMPI_C_COMPILE_FLAGS:STRING=""
> -DMPI_C_INCLUDE_PATH:STRING="" -DMPI_C_HEADER_DIR:STRING=""
> -DMPI_C_LIBRARIES:STRING=""
>
> Or am I looking at the wrong thing?
>
> I also tried doing the build downloading petsc from the main branch but
> that died at the same point and looked to give the same "redefinition of
> argument 'optimize'" errors
>
> Thanks
> Byron
>
>
>
> On Thu, Aug 25, 2022 at 4:15 PM Jacob Faibussowitsch <jacob.fai at gmail.com>
> wrote:
>
>> > Sadly each one makes different -std=c++11 vs  -std=c++14 choices as
>> well.
>>
>> Main should hopefully *also* warn about this, but unless you really
>> really want -std=c++11 or -std=c++14, it is better to leave out this
>> configure flag (--with-cxx-dialect).
>>
>> If --with-cxx-dialect is unspecified (or set to ?auto?) configure will
>> pick the most appropriate flag for you, taking your package environment
>> into account when choosing. If you specify the flag configure must assume
>> the user knows something it doesn?t and will blindly take what is given.
>>
>> Best regards,
>>
>> Jacob Faibussowitsch
>> (Jacob Fai - booss - oh - vitch)
>>
>> > On Aug 25, 2022, at 11:05, Barry Smith <bsmith at petsc.dev> wrote:
>> >
>> >
>> >   Oh yes, my mistake I misinterpreted the error message. It is actually
>> the nvcc compiler complaining about multiple command line arguments related
>> to optimization (which it is really picky about even if the two arguments
>> are identical.)  I reformatted some of the configure output to show this
>> >
>> > -ccbin mpicxx -std=c++14 -Xcompiler -fPIC -O3 -gencode
>> arch=compute_75,code=sm_75  -Wno-deprecated-gpu-targets
>> > -I/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi
>> -I/usr/lib/x86_64-linux-gnu/openmpi/include
>> > -DDEBUGlevel=0 -DPRNTlevel=0
>> >
>> > the next fragment is
>> >
>> > -O3 -DNDEBUG
>> >
>> > --generate-code=arch=compute_75,code=[compute_75,sm_75]
>> >
>> > -Xcompiler=-fPIC -std=c++11 -x cu -rdc=true -c
>> /home/blahblah/Projects/220824-petsc-example-for_forum/petsc/arch-linux-c-opt/externalpackages/git.superlu_dist/SRC/
>> superlu_gpu_utils.cu -o
>> >
>> > Hence the -O3 appears twice, once from the CMAKE argument PETSc passes
>> to superlu_dist
>> >
>> > -DCMAKE_CUDA_FLAGS=" -ccbin mpicxx -std=c++14 -Xcompiler -fPIC -O3
>> -gencode arch=compute_75,code=sm_75  -Wno-deprecated-gpu-targets
>> -I/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi
>> -I/usr/lib/x86_64-linux-gnu/openmpi/include   -DDEBUGlevel=0 -DPRNTlevel=0"
>> >
>> > and then presumably, the superlu_dist cmake process adds its own
>> arguments which overlap.
>> >
>> > If I remember correctly this is fixed in the main branch of PETSc,
>> could you do an identical build with the main branch? If that works we will
>> just need to figure out how to get MOOSE to work with main PETSc branch.
>> >
>> >   Barry
>> >
>> > Sadly each one makes different -std=c++11 vs  -std=c++14 choices as
>> well.
>> >
>> >
>> > --------------------
>> >
>> > [ 15%] Building CUDA object
>> SRC/CMakeFiles/superlu_dist.dir/superlu_gpu_utils.cu.o
>> > cd
>> /home/blahblah/Projects/220824-petsc-example-for_forum/petsc/arch-linux-c-opt/externalpackages/git.superlu_dist/petsc-build/SRC
>> && /usr/bin/nvcc
>> >
>> > -DSUPERLU_DIST_EXPORTS -Dsuperlu_dist_EXPORTS
>> >
>> >
>> -I/home/blahblah/Projects/220824-petsc-example-for_forum/petsc/arch-linux-c-opt/externalpackages/git.superlu_dist/petsc-build/SRC
>> >
>> > -I/home/blahblah/Projects/220824-petsc-build/installation/include
>> -I/usr/lib/x86_64-linux-gnu/openmpi/lib
>> >
>> > -DUSE_VENDOR_BLAS  -ccbin mpicxx -std=c++14 -Xcompiler -fPIC -O3
>> -gencode arch=compute_75,code=sm_75  -Wno-deprecated-gpu-targets
>> >
>> > -I/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi
>> -I/usr/lib/x86_64-linux-gnu/openmpi/include
>> >
>> > -DDEBUGlevel=0 -DPRNTlevel=0 -O3 -DNDEBUG
>> >
>> > --generate-code=arch=compute_75,code=[compute_75,sm_75]
>> >
>> > -Xcompiler=-fPIC -std=c++11 -x cu -rdc=true -c
>> /home/blahblah/Projects/220824-petsc-example-for_forum/petsc/arch-linux-c-opt/externalpackages/git.superlu_dist/SRC/
>> superlu_gpu_utils.cu -o
>> >
>> > CMakeFiles/superlu_dist.dir/superlu_gpu_utils.cu
>> >
>> >
>> >
>> >> On Aug 25, 2022, at 9:38 AM, byron <lbgpublic at gmail.com> wrote:
>> >>
>> >> Hi Barry
>> >>
>> >> There doesn't appear to be any reference to 'optimize' in the only
>> copy of  superlu_gpu_utils.cu I could find, I've attached a copy.
>> >>
>> >> Thanks
>> >>
>> >>
>> >>
>> >> On Thu, Aug 25, 2022 at 1:44 PM Barry Smith <bsmith at petsc.dev> wrote:
>> >>
>> >> nvcc fatal   : redefinition of argument 'optimize'
>> >> make[2]: *** [SRC/CMakeFiles/superlu_dist.dir/build.make:499:
>> SRC/CMakeFiles/superlu_dist.dir/superlu_gpu_utils.cu.o] Error 1
>> >> make[2]: *** Waiting for unfinished jobs....
>> >> nvcc fatal   : redefinition of argument 'optimize'
>> >> make[2]: *** [SRC/CMakeFiles/superlu_dist-static.dir/build.make:499:
>> SRC/CMakeFiles/superlu_dist-static.dir/superlu_gpu_utils.cu.o] Error 1
>> >> make[2]: *** Waiting for unfinished jobs....
>> >>
>> >> It seems that nvcc is unhappy with something in superlu_gpu_utils.cu
>> Could you check that file for any "redefinition of optimize"? Unfortunately
>> the nvcc compiler isn't kind enough to print the line number of the
>> redefinition or that information is lost somehow in the build process.
>> >>
>> >> Barry
>> >>
>> >>> On Aug 25, 2022, at 6:50 AM, byron <lbgpublic at gmail.com> wrote:
>> >>>
>> >>> Hi All
>> >>>
>> >>> I am having trouble getting petsc to build with cuda / superlu-dist
>> enabled.  I have been trying to build it as part of another software tool,
>> moose, hence the choice of options being passed to configure
>> >>>
>> >>> python3 ./configure --download-hypre=1 \
>> >>> --with-shared-libraries=1 \
>> >>> --download-hdf5=1 \
>> >>> --download-hdf5-fortran-bindings=0 \
>> >>> --with-debugging=no \
>> >>> --download-fblaslapack=1 \
>> >>> --download-metis=1 \
>> >>> --download-ptscotch=1 \
>> >>> --download-parmetis=1 \
>> >>> --download-superlu_dist=1 \
>> >>> --download-mumps=1 \
>> >>> --download-strumpack=1 \
>> >>> --download-scalapack=1 \
>> >>> --download-slepc=1 \
>> >>> --with-mpi=1 \
>> >>> --with-openmp=1 \
>> >>> --with-cxx-dialect=C++11 \
>> >>> --with-fortran-bindings=0 \
>> >>> --with-sowing=0 \
>> >>> --with-64-bit-indices \
>> >>> --with-cuda \
>> >>> --prefix=/home/blah/blah
>> >>>
>> >>> it fails with
>> >>>
>> >>>
>> =============================================================================================
>>
>>                                Installing PTScotch; this may take several
>> minutes
>>
>> =============================================================================================
>>
>>
>>  =============================================================================================
>>
>>                                Trying to download git://
>> https://github.com/xiaoyeli/superlu_dist for SUPERLU_DIST
>>
>>
>> =============================================================================================
>>
>>
>>  =============================================================================================
>>
>>                                Configuring SUPERLU_DIST with cmake; this
>> may take several minutes
>>
>> =============================================================================================
>>
>>
>>  =============================================================================================
>>
>>                                Compiling and installing SUPERLU_DIST; this
>> may take several minutes
>>
>> =============================================================================================
>>
>>
>>  *******************************************************************************
>> >>>          UNABLE to CONFIGURE with GIVEN OPTIONS    (see configure.log
>> for details):
>> >>>
>> -------------------------------------------------------------------------------
>> >>> Error running make on  SUPERLU_DIST
>> >>>
>> *******************************************************************************
>> >>>
>> >>> I've attached the configure.log.
>> >>>
>> >>> I've tried a few different things but nothing that resulted in a
>> successfull build.  The most promising was I tried to build superlu_dist
>> separately with the configure options that petsc passes it and then kept
>> trying the build over and over droping out different configure options
>> until I got it to work.  In that case removing a "-std=c++14" entry
>> appeared to fix the problem but then when I made the same change in how
>> petsc builds superlu-dist the make failed on superlu-dist with a different
>> error.  I get the feeling I'm dealing with symptoms of the problem rather
>> than the cause.
>> >>>
>> >>> Thanks
>> >>> Byron
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> <configure.log>
>> >>
>> >> <superlu_gpu_utils.cu>
>> >
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220826/6730cf0f/attachment-0001.html>

From bsmith at petsc.dev  Fri Aug 26 12:10:27 2022
From: bsmith at petsc.dev (Barry Smith)
Date: Fri, 26 Aug 2022 13:10:27 -0400
Subject: [petsc-users] unable to build petsc with cuda / superlu-dist
 enabled
In-Reply-To: <CAJ+YQyq8rwwZ9fWG_V=Vu5Vdrd_N5UGL79ZbDW0pu7YwsUikkw@mail.gmail.com>
References: <CAJ+YQyrctNbkcdbUD-DmzdQRzUx8w12yy2dBbuntLm4Awj6McQ@mail.gmail.com>
	<C4B564E7-0DD0-4FAC-A9F3-185F0229653F@petsc.dev>
	<CAJ+YQyr9uhyR0LvCsuDwcGEhDXb1k5kSKNgP4Z+77jf97Pmfhg@mail.gmail.com>
	<2616A45C-68CF-4C94-A880-B7858C9B7882@petsc.dev>
	<6BA4C5E3-29A4-4E68-A872-AE6A06185D64@gmail.com>
	<CAJ+YQyo2CCKeKiBjqgqWPqPzYZLyWDmfeS=uAYD819bK8-c8+A@mail.gmail.com>
	<A50D7893-39DF-46AA-B1C9-BFBF508266EA@petsc.dev>
	<CAJ+YQyq8rwwZ9fWG_V=Vu5Vdrd_N5UGL79ZbDW0pu7YwsUikkw@mail.gmail.com>
Message-ID: <561684F8-2789-42DD-A59D-19BC4237E9FB@petsc.dev>


  Glad it worked. The plan is the next release will not require these hacks.

> On Aug 26, 2022, at 12:13 PM, byron <lbgpublic at gmail.com> wrote:
> 
> It looks like I've (we've) cracked it, at least the petsc stage of what I'm trying to do.  To get the configure to work I used the following 
> 
> python3 ./configure --download-hypre=1 --with-shared-libraries=1 --download-hdf5=1 --download-hdf5-fortran-bindings=0 --with-debugging=no --download-fblaslapack=1 --download-metis=1 --download-ptscotch=1 --download-parmetis=1 --download-superlu_dist=1 --download-mumps=1 --download-strumpack=1 --download-scalapack=1 --download-slepc=1 --with-mpi=1 --with-openmp=1 --with-cxx-dialect=C++11 -with-fortran-bindings=0 --with-sowing=0 --with-64-bit-indices --with-cuda -CUDAOPTFLAGS="" --with-cuda-dialect=cxx11 --prefix=/home/lgoodman/Projects/petscExamples/main-build-CUDAOPTFLAGS-zero-cxx-dialect-unspc-cuda-dial-11/installation
> 
> Where -CUDAOPTFLAGS=""  was the solutions to getting rid of the duplicate -O3
> 
> then I was getting errors about 'redefinition of argument 'std'"  so i added
> 
> --with-cxx-dialect=C++11
> and 
> --with-cuda-dialect=cxx11 
> 
> and that got rid of those errors.  I image there are cleaner / more elegant ways of doing this but I'll settle for this for now.
> 
> Thanks a lot for the help guys, much appreciated
> 
> Best regards
> Byron
> 
> On Fri, Aug 26, 2022 at 4:06 PM Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>> wrote:
> 
>   Yes, PETSc sends down a single -O3 to SuperLU_DIST CMake but internally SuperLU_DIST CMake is adding its own set of nvcc compiler flags to use and these that get added to the one's PETSc passes in (they do not replace them). A little further down in the file it displays a compile line with nvcc that has two sets of -O3 (as well as two sets of other things as well).
> 
>   I think we changed the main branch PETSc to not pass down the optimization flags to the CMake CUDA to prevent the problem from happening. In other words, we let the package decide its optimization flags instead of trying to tell it what to use. 
> 
>   Barry
> 
> The root of the problem is that compiler APIs are a bunch of strings passed on the command line to the compiler without any structure.
> 
> 
>> On Aug 26, 2022, at 9:47 AM, byron <lbgpublic at gmail.com <mailto:lbgpublic at gmail.com>> wrote:
>> 
>> Hi Barry 
>> 
>> I dont think I see the same thing.
>> 
>> When I do a build using the release branch petsc passes the following arguements to superlu_dist (taken from /home/blahblah/Projects/220826-vanilla-petsc-release-branch/petsc/arch-linux-c-opt/externalpackages/git.superlu_dist/superlu_dist.petscconf)  which has only one -O3 entry
>> 
>> -DCMAKE_INSTALL_PREFIX=/home/blahblah/Projects/220826-vanilla-petsc-release-branch/installation -DCMAKE_INSTALL_NAME_DIR:STRING="/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib" -DCMAKE_INSTALL_LIBDIR:STRING="lib" -DCMAKE_VERBOSE_MAKEFILE=1 -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER="mpicc" -DMPI_C_COMPILER="/usr/bin/mpicc" -DCMAKE_AR=/usr/bin/ar -DCMAKE_RANLIB=/usr/bin/ranlib -DCMAKE_C_FLAGS:STRING="-fPIC -Wno-lto-type-mismatch -fstack-protector -g -O -fopenmp  -DDEBUGlevel=0 -DPRNTlevel=0" -DCMAKE_C_FLAGS_DEBUG:STRING="-fPIC -Wno-lto-type-mismatch -fstack-protector -g -O -fopenmp  -DDEBUGlevel=0 -DPRNTlevel=0" -DCMAKE_C_FLAGS_RELEASE:STRING="-fPIC -Wno-lto-type-mismatch -fstack-protector -g -O -fopenmp  -DDEBUGlevel=0 -DPRNTlevel=0" -DCMAKE_CXX_COMPILER="mpicxx" -DMPI_CXX_COMPILER="/usr/bin/mpicxx" -DCMAKE_CXX_FLAGS:STRING="-Wno-lto-type-mismatch -fstack-protector -g -O -fopenmp -std=c++11 -fPIC -fopenmp  -DDEBUGlevel=0 -DPRNTlevel=0" -DCMAKE_CXX_FLAGS_DEBUG:STRING="-Wno-lto-type-mismatch -fstack-protector -g -O -fopenmp -std=c++11 -fPIC -fopenmp  -DDEBUGlevel=0 -DPRNTlevel=0" -DCMAKE_CXX_FLAGS_RELEASE:STRING="-Wno-lto-type-mismatch -fstack-protector -g -O -fopenmp -std=c++11 -fPIC -fopenmp  -DDEBUGlevel=0 -DPRNTlevel=0" -DCMAKE_Fortran_COMPILER="mpif90" -DMPI_Fortran_COMPILER="/usr/bin/mpif90" -DCMAKE_Fortran_FLAGS:STRING="-fPIC -ffree-line-length-0 -Wno-lto-type-mismatch -g -O -fopenmp" -DCMAKE_Fortran_FLAGS_DEBUG:STRING="-fPIC -ffree-line-length-0 -Wno-lto-type-mismatch -g -O -fopenmp" -DCMAKE_Fortran_FLAGS_RELEASE:STRING="-fPIC -ffree-line-length-0 -Wno-lto-type-mismatch -g -O -fopenmp" -DCMAKE_EXE_LINKER_FLAGS:STRING=" -fopenmp -fopenmp" -DBUILD_SHARED_LIBS:BOOL=ON -DTPL_ENABLE_CUDALIB=TRUE -DTPL_CUDA_LIBRARIES="-lcudart -lcufft -lcublas -lcusparse -lcusolver -lcurand -lcuda" -DCUDA_ARCH_FLAGS="-arch=sm_75" -DCMAKE_CUDA_COMPILER="nvcc" -DCMAKE_CUDA_FLAGS=" -ccbin mpicxx -std=c++14 -Xcompiler -fPIC -O3 -gencode arch=compute_75,code=sm_75  -Wno-deprecated-gpu-targets -I/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi -I/usr/lib/x86_64-linux-gnu/openmpi/include   -DDEBUGlevel=0 -DPRNTlevel=0" -DUSE_XSDK_DEFAULTS=YES -DTPL_BLAS_LIBRARIES="-Wl,-rpath,/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib -L/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib -lflapack -Wl,-rpath,/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib -L/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib -lfblas -lcudart -lcufft -lcublas -lcusparse -lcusolver -lcurand -lcuda -lm -lstdc++ -ldl -Wl,-rpath,/usr/lib/x86_64-linux-gnu/openmpi/lib -L/usr/lib/x86_64-linux-gnu/openmpi/lib -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi -lgfortran -lm -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/9 -L/usr/lib/gcc/x86_64-linux-gnu/9 -lgfortran -lm -lgcc_s -lquadmath -lpthread" -DTPL_LAPACK_LIBRARIES="-Wl,-rpath,/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib -L/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib -lflapack -Wl,-rpath,/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib -L/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib -lfblas -lcudart -lcufft -lcublas -lcusparse -lcusolver -lcurand -lcuda -lm -lstdc++ -ldl -Wl,-rpath,/usr/lib/x86_64-linux-gnu/openmpi/lib -L/usr/lib/x86_64-linux-gnu/openmpi/lib -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi -lgfortran -lm -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/9 -L/usr/lib/gcc/x86_64-linux-gnu/9 -lgfortran -lm -lgcc_s -lquadmath -lpthread" -DTPL_PARMETIS_INCLUDE_DIRS="/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/include;/usr/lib/x86_64-linux-gnu/openmpi/include;/usr/lib/x86_64-linux-gnu/openmpi/lib" -DTPL_PARMETIS_LIBRARIES="-Wl,-rpath,/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib -L/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib -lparmetis -Wl,-rpath,/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib -L/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib -lmetis -lm -lcudart -lcufft -lcublas -lcusparse -lcusolver -lcurand -lcuda" -DXSDK_INDEX_SIZE=64 -DXSDK_ENABLE_Fortran=ON -Denable_tests=0 -Denable_examples=0 -DMPI_C_COMPILE_FLAGS:STRING="" -DMPI_C_INCLUDE_PATH:STRING="" -DMPI_C_HEADER_DIR:STRING="" -DMPI_C_LIBRARIES:STRING=""
>> 
>> Or am I looking at the wrong thing?
>> 
>> I also tried doing the build downloading petsc from the main branch but that died at the same point and looked to give the same "redefinition of argument 'optimize'" errors
>> 
>> Thanks
>> Byron
>> 
>> 
>> 
>> On Thu, Aug 25, 2022 at 4:15 PM Jacob Faibussowitsch <jacob.fai at gmail.com <mailto:jacob.fai at gmail.com>> wrote:
>> > Sadly each one makes different -std=c++11 vs  -std=c++14 choices as well.
>> 
>> Main should hopefully *also* warn about this, but unless you really really want -std=c++11 or -std=c++14, it is better to leave out this configure flag (--with-cxx-dialect). 
>> 
>> If --with-cxx-dialect is unspecified (or set to ?auto?) configure will pick the most appropriate flag for you, taking your package environment into account when choosing. If you specify the flag configure must assume the user knows something it doesn?t and will blindly take what is given.
>> 
>> Best regards,
>> 
>> Jacob Faibussowitsch
>> (Jacob Fai - booss - oh - vitch)
>> 
>> > On Aug 25, 2022, at 11:05, Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>> wrote:
>> > 
>> > 
>> >   Oh yes, my mistake I misinterpreted the error message. It is actually the nvcc compiler complaining about multiple command line arguments related to optimization (which it is really picky about even if the two arguments are identical.)  I reformatted some of the configure output to show this
>> > 
>> > -ccbin mpicxx -std=c++14 -Xcompiler -fPIC -O3 -gencode arch=compute_75,code=sm_75  -Wno-deprecated-gpu-targets
>> > -I/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi -I/usr/lib/x86_64-linux-gnu/openmpi/include
>> > -DDEBUGlevel=0 -DPRNTlevel=0
>> > 
>> > the next fragment is 
>> > 
>> > -O3 -DNDEBUG
>> > 
>> > --generate-code=arch=compute_75,code=[compute_75,sm_75]
>> > 
>> > -Xcompiler=-fPIC -std=c++11 -x cu -rdc=true -c /home/blahblah/Projects/220824-petsc-example-for_forum/petsc/arch-linux-c-opt/externalpackages/git.superlu_dist/SRC/superlu_gpu_utils.cu <http://superlu_gpu_utils.cu/> -o
>> > 
>> > Hence the -O3 appears twice, once from the CMAKE argument PETSc passes to superlu_dist 
>> > 
>> > -DCMAKE_CUDA_FLAGS=" -ccbin mpicxx -std=c++14 -Xcompiler -fPIC -O3 -gencode arch=compute_75,code=sm_75  -Wno-deprecated-gpu-targets -I/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi -I/usr/lib/x86_64-linux-gnu/openmpi/include   -DDEBUGlevel=0 -DPRNTlevel=0"
>> > 
>> > and then presumably, the superlu_dist cmake process adds its own arguments which overlap.
>> > 
>> > If I remember correctly this is fixed in the main branch of PETSc, could you do an identical build with the main branch? If that works we will just need to figure out how to get MOOSE to work with main PETSc branch.
>> > 
>> >   Barry
>> > 
>> > Sadly each one makes different -std=c++11 vs  -std=c++14 choices as well.
>> > 
>> > 
>> > --------------------
>> > 
>> > [ 15%] Building CUDA object SRC/CMakeFiles/superlu_dist.dir/superlu_gpu_utils.cu.o
>> > cd /home/blahblah/Projects/220824-petsc-example-for_forum/petsc/arch-linux-c-opt/externalpackages/git.superlu_dist/petsc-build/SRC && /usr/bin/nvcc
>> > 
>> > -DSUPERLU_DIST_EXPORTS -Dsuperlu_dist_EXPORTS
>> > 
>> > -I/home/blahblah/Projects/220824-petsc-example-for_forum/petsc/arch-linux-c-opt/externalpackages/git.superlu_dist/petsc-build/SRC
>> > 
>> > -I/home/blahblah/Projects/220824-petsc-build/installation/include -I/usr/lib/x86_64-linux-gnu/openmpi/lib
>> > 
>> > -DUSE_VENDOR_BLAS  -ccbin mpicxx -std=c++14 -Xcompiler -fPIC -O3 -gencode arch=compute_75,code=sm_75  -Wno-deprecated-gpu-targets
>> > 
>> > -I/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi -I/usr/lib/x86_64-linux-gnu/openmpi/include
>> > 
>> > -DDEBUGlevel=0 -DPRNTlevel=0 -O3 -DNDEBUG
>> > 
>> > --generate-code=arch=compute_75,code=[compute_75,sm_75]
>> > 
>> > -Xcompiler=-fPIC -std=c++11 -x cu -rdc=true -c /home/blahblah/Projects/220824-petsc-example-for_forum/petsc/arch-linux-c-opt/externalpackages/git.superlu_dist/SRC/superlu_gpu_utils.cu <http://superlu_gpu_utils.cu/> -o
>> > 
>> > CMakeFiles/superlu_dist.dir/superlu_gpu_utils.cu <http://superlu_gpu_utils.cu/>
>> > 
>> > 
>> > 
>> >> On Aug 25, 2022, at 9:38 AM, byron <lbgpublic at gmail.com <mailto:lbgpublic at gmail.com>> wrote:
>> >> 
>> >> Hi Barry 
>> >> 
>> >> There doesn't appear to be any reference to 'optimize' in the only copy of  superlu_gpu_utils.cu <http://superlu_gpu_utils.cu/> I could find, I've attached a copy.
>> >> 
>> >> Thanks
>> >> 
>> >> 
>> >> 
>> >> On Thu, Aug 25, 2022 at 1:44 PM Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>> wrote:
>> >> 
>> >> nvcc fatal   : redefinition of argument 'optimize'
>> >> make[2]: *** [SRC/CMakeFiles/superlu_dist.dir/build.make:499: SRC/CMakeFiles/superlu_dist.dir/superlu_gpu_utils.cu.o] Error 1
>> >> make[2]: *** Waiting for unfinished jobs....
>> >> nvcc fatal   : redefinition of argument 'optimize'
>> >> make[2]: *** [SRC/CMakeFiles/superlu_dist-static.dir/build.make:499: SRC/CMakeFiles/superlu_dist-static.dir/superlu_gpu_utils.cu.o] Error 1
>> >> make[2]: *** Waiting for unfinished jobs....
>> >> 
>> >> It seems that nvcc is unhappy with something in superlu_gpu_utils.cu <http://superlu_gpu_utils.cu/>  Could you check that file for any "redefinition of optimize"? Unfortunately the nvcc compiler isn't kind enough to print the line number of the redefinition or that information is lost somehow in the build process.
>> >> 
>> >> Barry
>> >> 
>> >>> On Aug 25, 2022, at 6:50 AM, byron <lbgpublic at gmail.com <mailto:lbgpublic at gmail.com>> wrote:
>> >>> 
>> >>> Hi All
>> >>> 
>> >>> I am having trouble getting petsc to build with cuda / superlu-dist enabled.  I have been trying to build it as part of another software tool, moose, hence the choice of options being passed to configure
>> >>> 
>> >>> python3 ./configure --download-hypre=1 \
>> >>> --with-shared-libraries=1 \
>> >>> --download-hdf5=1 \
>> >>> --download-hdf5-fortran-bindings=0 \
>> >>> --with-debugging=no \
>> >>> --download-fblaslapack=1 \
>> >>> --download-metis=1 \
>> >>> --download-ptscotch=1 \
>> >>> --download-parmetis=1 \
>> >>> --download-superlu_dist=1 \
>> >>> --download-mumps=1 \
>> >>> --download-strumpack=1 \
>> >>> --download-scalapack=1 \
>> >>> --download-slepc=1 \
>> >>> --with-mpi=1 \
>> >>> --with-openmp=1 \
>> >>> --with-cxx-dialect=C++11 \
>> >>> --with-fortran-bindings=0 \
>> >>> --with-sowing=0 \
>> >>> --with-64-bit-indices \
>> >>> --with-cuda \
>> >>> --prefix=/home/blah/blah
>> >>> 
>> >>> it fails with 
>> >>> 
>> >>> =============================================================================================                                                                                                             Installing PTScotch; this may take several minutes                                                                                                                                            =============================================================================================                                                                                                       =============================================================================================                                                                                                             Trying to download git://https://github.com/xiaoyeli/superlu_dist <https://github.com/xiaoyeli/superlu_dist> for SUPERLU_DIST                                                                                                            =============================================================================================                                                                                                       =============================================================================================                                                                                                             Configuring SUPERLU_DIST with cmake; this may take several minutes                                                                                                                            =============================================================================================                                                                                                       =============================================================================================                                                                                                             Compiling and installing SUPERLU_DIST; this may take several minutes                                                                                                                          =============================================================================================                                                                                                                             *******************************************************************************
>> >>>          UNABLE to CONFIGURE with GIVEN OPTIONS    (see configure.log for details):
>> >>> -------------------------------------------------------------------------------
>> >>> Error running make on  SUPERLU_DIST
>> >>> *******************************************************************************
>> >>> 
>> >>> I've attached the configure.log.
>> >>> 
>> >>> I've tried a few different things but nothing that resulted in a successfull build.  The most promising was I tried to build superlu_dist separately with the configure options that petsc passes it and then kept trying the build over and over droping out different configure options until I got it to work.  In that case removing a "-std=c++14" entry appeared to fix the problem but then when I made the same change in how petsc builds superlu-dist the make failed on superlu-dist with a different error.  I get the feeling I'm dealing with symptoms of the problem rather than the cause.  
>> >>> 
>> >>> Thanks
>> >>> Byron
>> >>>           
>> >>> 
>> >>> 
>> >>>  
>> >>> <configure.log>
>> >> 
>> >> <superlu_gpu_utils.cu <http://superlu_gpu_utils.cu/>>
>> > 
>> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220826/acd05594/attachment-0001.html>

From wangxq2020 at mail.ustc.edu.cn  Sat Aug 27 06:16:40 2022
From: wangxq2020 at mail.ustc.edu.cn (wangxq2020 at mail.ustc.edu.cn)
Date: Sat, 27 Aug 2022 19:16:40 +0800 (GMT+08:00)
Subject: [petsc-users] ./test1: error while loading shared libraries:
 libopenblas.so.0: cannot open shared object file: No such file or
 directory.
Message-ID: <cee35bd.483ca.182df04a6c0.Coremail.wangxq2020@mail.ustc.edu.cn>

Hi


Submitting a program fails by srun using slurm. This is my command 
srun -n 8 --partition=test --qos=testqos ./test1 -options_file options/test1.opts 
and returns 
./test1: error while loading shared libraries: libopenblas.so.0: cannot open shared object file: No such file or directory.


But when I using /gpfs/home/ses/wangxq2020/petsc_projects/petsc-3.17.2/arch-linux-c-debug/bin/mpiexec -n 1 ./test1,the program can be running. When I installed the related library PETSC of this program, it failed to use the mpich on the supercomputer, so I downloaded mpich myself. I think it was srun that executed the mpich in /opt or other directories. srun was not using the mpich I downloaded, causing the error. How can I make srun run this /gpfs/home/ses/wangxq2020/petsc_projects/petsc-3.17.2/arch-linux-c-debug/bin/mpiexec?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220827/34d93f16/attachment.html>

From knepley at gmail.com  Sat Aug 27 08:29:02 2022
From: knepley at gmail.com (Matthew Knepley)
Date: Sat, 27 Aug 2022 09:29:02 -0400
Subject: [petsc-users] ./test1: error while loading shared libraries:
 libopenblas.so.0: cannot open shared object file: No such file or
 directory.
In-Reply-To: <cee35bd.483ca.182df04a6c0.Coremail.wangxq2020@mail.ustc.edu.cn>
References: <cee35bd.483ca.182df04a6c0.Coremail.wangxq2020@mail.ustc.edu.cn>
Message-ID: <CAMYG4GmX602r9CoavBBL-Q2=xT2i+8=+JReYbLwkai-miRMhnQ@mail.gmail.com>

On Sat, Aug 27, 2022 at 7:17 AM wangxq2020--- via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Hi
>
> Submitting a program fails by srun using slurm. This is my command
> srun -n 8 --partition=test --qos=testqos ./test1 -options_file
> options/test1.opts
> and returns
> ./test1: error while loading shared libraries: libopenblas.so.0: cannot
> open shared object file: No such file or directory.
>
> But when I
> using /gpfs/home/ses/wangxq2020/petsc_projects/petsc-3.17.2/arch-linux-c-debug/bin/mpiexec
> -n 1 ./test1,the program can be running. When I installed the related
> library PETSC of this program, it failed to use the mpich on the
> supercomputer, so I downloaded mpich myself. I think it was srun that
> executed the mpich in /opt or other directories. srun was not using the
> mpich I downloaded, causing the error. How can I make srun run this
> /gpfs/home/ses/wangxq2020/petsc_projects/petsc-3.17.2/arch-linux-c-debug/bin/mpiexec?
>

There may also be a problem with MPI, but this problem is with BLAS. You
built PETSc with OpenBLAS, however it appears that OpenBLAS is only
installed on the node you built PETSc on, not on the compute nodes used by
srun. You can verify this by asking the sysadmin.

  Thanks,

     Matt

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220827/9f16ddc8/attachment.html>

From mi.mike1021 at gmail.com  Sun Aug 28 07:51:26 2022
From: mi.mike1021 at gmail.com (Mike Michell)
Date: Sun, 28 Aug 2022 07:51:26 -0500
Subject: [petsc-users] DMPlex Interpolation
In-Reply-To: <CAMYG4GmiXaNTG5sgjoH=-CdY21OkRaKZh5y5OdPcSiEMj-KfXg@mail.gmail.com>
References: <CAEc7osYgsXxW9DN_6aHof3wdyqHw3Wv6nLp+svWsZF5c0zz93w@mail.gmail.com>
	<CAMYG4GmiXaNTG5sgjoH=-CdY21OkRaKZh5y5OdPcSiEMj-KfXg@mail.gmail.com>
Message-ID: <CAEc7osb8HG7ZQQy3=6Jcq-J8jMB1vWXHo-7Y5=fc7A3QR9BQGA@mail.gmail.com>

Hi, thank you for the reply.

I was able to manage mapping from cell-center to vertex. Basically, in
Fortran, it seems DMCreateInterpolation() requires the optional scaling
vector as a mandatory argument, which is strange.

My dmplex has zero overlap layer over the procs, and there is no ghost cell
inside physical boundary. In this case, it seems mapping between cell
center to vertex returns zero (in case the global cell vector initialized
to zero) value at the nodes, which are located at physical boundary. To
manage this problem, is it mandatory to have ghost cells. Is this correct
understanding?

Also, my mapping accuracy itself seems to be improved. For simple test,
cell-center's x-coordinate value of each cell mapped to node and printed to
the vertex field as .vtu format, and the mapped x-vertex-coordinate is
quite different with the actual nodal coordinate values what PETSc
intrinsically provides through DMGetCoordinatesLocal(). I believe I am
doing something wrong, probably arguments in PetscFECreateLagrange() can
improve the mapping accuracy in finite-element space?

Thanks,


> On Thu, Aug 25, 2022 at 7:12 PM Mike Michell <mi.mike1021 at gmail.com>
> wrote:
>
>> Hi, this is a duplication of
>> https://lists.mcs.anl.gov/pipermail/petsc-users/2022-August/046746.html
>> for in-depth question.
>>
>> I wrote a short code as attached to test interpolation between two DMPlex
>> objects. Goal is to map solution field defined on cell-centroid(DMPlex-1)
>> into vertex(DMPlex-2) field or vice versa.
>>
>> Basically, DMCreateInterpolation() fails in the example code and it was
>> not allowed to get useful error messages. The DMPlex is created by loading
>> a 2D square box in Gmsh. Can I get some comments on that?
>>
>
> Sure, I am looking at it.
>
>   Thanks,
>
>      Matt
>
>
>> Thanks,
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220828/1c0a32f1/attachment.html>

From knepley at gmail.com  Sun Aug 28 08:05:47 2022
From: knepley at gmail.com (Matthew Knepley)
Date: Sun, 28 Aug 2022 09:05:47 -0400
Subject: [petsc-users] DMPlex Interpolation
In-Reply-To: <CAEc7osb8HG7ZQQy3=6Jcq-J8jMB1vWXHo-7Y5=fc7A3QR9BQGA@mail.gmail.com>
References: <CAEc7osYgsXxW9DN_6aHof3wdyqHw3Wv6nLp+svWsZF5c0zz93w@mail.gmail.com>
	<CAMYG4GmiXaNTG5sgjoH=-CdY21OkRaKZh5y5OdPcSiEMj-KfXg@mail.gmail.com>
	<CAEc7osb8HG7ZQQy3=6Jcq-J8jMB1vWXHo-7Y5=fc7A3QR9BQGA@mail.gmail.com>
Message-ID: <CAMYG4GnfNRGa9yOeCBzfviSfsJ+FxAy73_aRo07VqyCoQETJ+g@mail.gmail.com>

On Sun, Aug 28, 2022 at 8:51 AM Mike Michell <mi.mike1021 at gmail.com> wrote:

> Hi, thank you for the reply.
>
> I was able to manage mapping from cell-center to vertex. Basically, in
> Fortran, it seems DMCreateInterpolation() requires the optional scaling
> vector as a mandatory argument, which is strange.
>

It needs a custom Fortran wrapper to check for the NULL input from Fortran.


> My dmplex has zero overlap layer over the procs, and there is no ghost
> cell inside physical boundary. In this case, it seems mapping between cell
> center to vertex returns zero (in case the global cell vector initialized
> to zero) value at the nodes, which are located at physical boundary. To
> manage this problem, is it mandatory to have ghost cells. Is this correct
> understanding?
>

I cannot quite understand. Are you saying that you have inhomogeneous
Dirichlet conditions on the boundary, and a 0 guess in the interior, and
you get all zeros from interpolation? Yes, we have no way of getting
boundary values into the operator. You could conceivably make an operator
that mapped between local vectors, but it is hard to see why you would want
this. In the solver, we usually  just care about the unknowns we are
solving for.

Did I understand your question, or is it about something else?


> Also, my mapping accuracy itself seems to be improved. For simple test,
> cell-center's x-coordinate value of each cell mapped to node and printed to
> the vertex field as .vtu format, and the mapped x-vertex-coordinate is
> quite different with the actual nodal coordinate values what PETSc
> intrinsically provides through DMGetCoordinatesLocal(). I believe I am
> doing something wrong, probably arguments in PetscFECreateLagrange() can
> improve the mapping accuracy in finite-element space?
>

Yes, a cellwise field is much less accurate than a linear field. Pushing
the values up to a linear field does not make them more accurate.

  Thanks,

     Matt


> Thanks,
>
>
>> On Thu, Aug 25, 2022 at 7:12 PM Mike Michell <mi.mike1021 at gmail.com>
>> wrote:
>>
>>> Hi, this is a duplication of
>>> https://lists.mcs.anl.gov/pipermail/petsc-users/2022-August/046746.html
>>> for in-depth question.
>>>
>>> I wrote a short code as attached to test interpolation between two
>>> DMPlex objects. Goal is to map solution field defined on
>>> cell-centroid(DMPlex-1) into vertex(DMPlex-2) field or vice versa.
>>>
>>> Basically, DMCreateInterpolation() fails in the example code and it was
>>> not allowed to get useful error messages. The DMPlex is created by loading
>>> a 2D square box in Gmsh. Can I get some comments on that?
>>>
>>
>> Sure, I am looking at it.
>>
>>   Thanks,
>>
>>      Matt
>>
>>
>>> Thanks,
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/~knepley/>
>>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220828/c0bb5467/attachment.html>

From mi.mike1021 at gmail.com  Sun Aug 28 11:11:32 2022
From: mi.mike1021 at gmail.com (Mike Michell)
Date: Sun, 28 Aug 2022 11:11:32 -0500
Subject: [petsc-users] DMPlex Interpolation
In-Reply-To: <CAMYG4GnfNRGa9yOeCBzfviSfsJ+FxAy73_aRo07VqyCoQETJ+g@mail.gmail.com>
References: <CAEc7osYgsXxW9DN_6aHof3wdyqHw3Wv6nLp+svWsZF5c0zz93w@mail.gmail.com>
	<CAMYG4GmiXaNTG5sgjoH=-CdY21OkRaKZh5y5OdPcSiEMj-KfXg@mail.gmail.com>
	<CAEc7osb8HG7ZQQy3=6Jcq-J8jMB1vWXHo-7Y5=fc7A3QR9BQGA@mail.gmail.com>
	<CAMYG4GnfNRGa9yOeCBzfviSfsJ+FxAy73_aRo07VqyCoQETJ+g@mail.gmail.com>
Message-ID: <CAEc7osad2ee=T6QH5=VHFRjJDsyJ4mO3SYnTF6_BpqkJN0Ppfw@mail.gmail.com>

Thank you for the reply.

*I cannot quite understand. Are you saying that you have inhomogeneous
Dirichlet conditions on the boundary, and a 0 guess in the interior, and
you get all zeros from interpolation? Yes, we have no way of getting
boundary values into the operator. You could conceivably make an operator
that mapped between local vectors, but it is hard to see why you would want
this. In the solver, we usually  just care about the unknowns we are
solving for. Did I understand your question, or is it about something else?*
--> It is not imposing a physical boundary condition (Dirichlet condition
for example), but my solver needs to import an external solution vector
field resolved to each cell-centroid, then I need to map it to a vertex
field in which my solver is resolved for solving governing equations. Then,
my code solves its equations. After my solver is done solving the
equations, the vertex field should be re-map into cell-field and then push
back the cell field to the external code. With this context, I have zero
values returned at the vertices on the physical boundaries if I try mapping
from cell-center to vertex field without having ghost cell layers inside
physical boundaries. For interior vertices, I get reasonable mapped values,
although the mapping accuracy seems to be able to be improved as I
mentioned in the earlier email.
If dmplex for cell-center field would be defined to have ghost cell layer,
and fill it (it meant import ghost cell values as well from the external
code), and map it to vertex-field, can it be possible to fill vertex points
on the physical boundaries? I was trying to test this strategy, but it
seems DMPlexComputeCellGeometryFVM() gives NaN values if dmplex has ghost
cell by calling DMPlexConstructGhostCells(). In detail,
DMPlexComputeCellGeometryFVM() gives zero area for each cell, and returns
NaN values for cell centroids and normals for the ghost cells.

Thanks,


> On Sun, Aug 28, 2022 at 8:51 AM Mike Michell <mi.mike1021 at gmail.com>
> wrote:
>
>> Hi, thank you for the reply.
>>
>> I was able to manage mapping from cell-center to vertex. Basically, in
>> Fortran, it seems DMCreateInterpolation() requires the optional scaling
>> vector as a mandatory argument, which is strange.
>>
>
> It needs a custom Fortran wrapper to check for the NULL input from Fortran.
>
>
>> My dmplex has zero overlap layer over the procs, and there is no ghost
>> cell inside physical boundary. In this case, it seems mapping between cell
>> center to vertex returns zero (in case the global cell vector initialized
>> to zero) value at the nodes, which are located at physical boundary. To
>> manage this problem, is it mandatory to have ghost cells. Is this correct
>> understanding?
>>
>
> I cannot quite understand. Are you saying that you have inhomogeneous
> Dirichlet conditions on the boundary, and a 0 guess in the interior, and
> you get all zeros from interpolation? Yes, we have no way of getting
> boundary values into the operator. You could conceivably make an operator
> that mapped between local vectors, but it is hard to see why you would want
> this. In the solver, we usually  just care about the unknowns we are
> solving for.
>
> Did I understand your question, or is it about something else?
>
>
>> Also, my mapping accuracy itself seems to be improved. For simple test,
>> cell-center's x-coordinate value of each cell mapped to node and printed to
>> the vertex field as .vtu format, and the mapped x-vertex-coordinate is
>> quite different with the actual nodal coordinate values what PETSc
>> intrinsically provides through DMGetCoordinatesLocal(). I believe I am
>> doing something wrong, probably arguments in PetscFECreateLagrange() can
>> improve the mapping accuracy in finite-element space?
>>
>
> Yes, a cellwise field is much less accurate than a linear field. Pushing
> the values up to a linear field does not make them more accurate.
>
>   Thanks,
>
>      Matt
>
>
>> Thanks,
>>
>>
>>> On Thu, Aug 25, 2022 at 7:12 PM Mike Michell <mi.mike1021 at gmail.com>
>>> wrote:
>>>
>>>> Hi, this is a duplication of
>>>> https://lists.mcs.anl.gov/pipermail/petsc-users/2022-August/046746.html
>>>>
>>>> for in-depth question.
>>>>
>>>> I wrote a short code as attached to test interpolation between two
>>>> DMPlex objects. Goal is to map solution field defined on
>>>> cell-centroid(DMPlex-1) into vertex(DMPlex-2) field or vice versa.
>>>>
>>>> Basically, DMCreateInterpolation() fails in the example code and it was
>>>> not allowed to get useful error messages. The DMPlex is created by loading
>>>> a 2D square box in Gmsh. Can I get some comments on that?
>>>>
>>>
>>> Sure, I am looking at it.
>>>
>>>   Thanks,
>>>
>>>      Matt
>>>
>>>
>>>> Thanks,
>>>>
>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>> https://www.cse.buffalo.edu/~knepley/
>>> <http://www.cse.buffalo.edu/~knepley/>
>>>
>>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220828/162700ca/attachment-0001.html>

From knepley at gmail.com  Sun Aug 28 11:21:44 2022
From: knepley at gmail.com (Matthew Knepley)
Date: Sun, 28 Aug 2022 12:21:44 -0400
Subject: [petsc-users] DMPlex Interpolation
In-Reply-To: <CAEc7osad2ee=T6QH5=VHFRjJDsyJ4mO3SYnTF6_BpqkJN0Ppfw@mail.gmail.com>
References: <CAEc7osYgsXxW9DN_6aHof3wdyqHw3Wv6nLp+svWsZF5c0zz93w@mail.gmail.com>
	<CAMYG4GmiXaNTG5sgjoH=-CdY21OkRaKZh5y5OdPcSiEMj-KfXg@mail.gmail.com>
	<CAEc7osb8HG7ZQQy3=6Jcq-J8jMB1vWXHo-7Y5=fc7A3QR9BQGA@mail.gmail.com>
	<CAMYG4GnfNRGa9yOeCBzfviSfsJ+FxAy73_aRo07VqyCoQETJ+g@mail.gmail.com>
	<CAEc7osad2ee=T6QH5=VHFRjJDsyJ4mO3SYnTF6_BpqkJN0Ppfw@mail.gmail.com>
Message-ID: <CAMYG4Gkzjf=OXBQsErWCumMscGykzTZbLFsuzDbhCtL155o_vA@mail.gmail.com>

On Sun, Aug 28, 2022 at 12:11 PM Mike Michell <mi.mike1021 at gmail.com> wrote:

> Thank you for the reply.
>
> *I cannot quite understand. Are you saying that you have inhomogeneous
> Dirichlet conditions on the boundary, and a 0 guess in the interior, and
> you get all zeros from interpolation? Yes, we have no way of getting
> boundary values into the operator. You could conceivably make an operator
> that mapped between local vectors, but it is hard to see why you would want
> this. In the solver, we usually  just care about the unknowns we are
> solving for. Did I understand your question, or is it about something else?*
> --> It is not imposing a physical boundary condition (Dirichlet condition
> for example), but my solver needs to import an external solution vector
> field resolved to each cell-centroid, then I need to map it to a vertex
> field in which my solver is resolved for solving governing equations. Then,
> my code solves its equations. After my solver is done solving the
> equations, the vertex field should be re-map into cell-field and then push
> back the cell field to the external code. With this context, I have zero
> values returned at the vertices on the physical boundaries
>

Does your section mark these unknowns as constrained?


> if I try mapping from cell-center to vertex field without having ghost
> cell layers inside physical boundaries. For interior vertices, I get
> reasonable mapped values, although the mapping accuracy seems to be able to
> be improved as I mentioned in the earlier email.
>

As I replied, I think this is not true.


> If dmplex for cell-center field would be defined to have ghost cell layer,
> and fill it (it meant import ghost cell values as well from the external
> code), and map it to vertex-field, can it be possible to fill vertex points
> on the physical boundaries?
>

You should not get 0 unless you constrained these variables in the section.
If not, you should see it use the constant values in the cell.

  Thanks,

     Matt


> I was trying to test this strategy, but it
> seems DMPlexComputeCellGeometryFVM() gives NaN values if dmplex has ghost
> cell by calling DMPlexConstructGhostCells(). In detail,
> DMPlexComputeCellGeometryFVM() gives zero area for each cell, and returns
> NaN values for cell centroids and normals for the ghost cells.
>
> Thanks,
>
>
>> On Sun, Aug 28, 2022 at 8:51 AM Mike Michell <mi.mike1021 at gmail.com>
>> wrote:
>>
>>> Hi, thank you for the reply.
>>>
>>> I was able to manage mapping from cell-center to vertex. Basically, in
>>> Fortran, it seems DMCreateInterpolation() requires the optional scaling
>>> vector as a mandatory argument, which is strange.
>>>
>>
>> It needs a custom Fortran wrapper to check for the NULL input from
>> Fortran.
>>
>>
>>> My dmplex has zero overlap layer over the procs, and there is no ghost
>>> cell inside physical boundary. In this case, it seems mapping between cell
>>> center to vertex returns zero (in case the global cell vector initialized
>>> to zero) value at the nodes, which are located at physical boundary. To
>>> manage this problem, is it mandatory to have ghost cells. Is this correct
>>> understanding?
>>>
>>
>> I cannot quite understand. Are you saying that you have inhomogeneous
>> Dirichlet conditions on the boundary, and a 0 guess in the interior, and
>> you get all zeros from interpolation? Yes, we have no way of getting
>> boundary values into the operator. You could conceivably make an operator
>> that mapped between local vectors, but it is hard to see why you would want
>> this. In the solver, we usually  just care about the unknowns we are
>> solving for.
>>
>> Did I understand your question, or is it about something else?
>>
>>
>>> Also, my mapping accuracy itself seems to be improved. For simple test,
>>> cell-center's x-coordinate value of each cell mapped to node and printed to
>>> the vertex field as .vtu format, and the mapped x-vertex-coordinate is
>>> quite different with the actual nodal coordinate values what PETSc
>>> intrinsically provides through DMGetCoordinatesLocal(). I believe I am
>>> doing something wrong, probably arguments in PetscFECreateLagrange() can
>>> improve the mapping accuracy in finite-element space?
>>>
>>
>> Yes, a cellwise field is much less accurate than a linear field. Pushing
>> the values up to a linear field does not make them more accurate.
>>
>>   Thanks,
>>
>>      Matt
>>
>>
>>> Thanks,
>>>
>>>
>>>> On Thu, Aug 25, 2022 at 7:12 PM Mike Michell <mi.mike1021 at gmail.com>
>>>> wrote:
>>>>
>>>>> Hi, this is a duplication of
>>>>> https://lists.mcs.anl.gov/pipermail/petsc-users/2022-August/046746.html
>>>>>
>>>>> for in-depth question.
>>>>>
>>>>> I wrote a short code as attached to test interpolation between two
>>>>> DMPlex objects. Goal is to map solution field defined on
>>>>> cell-centroid(DMPlex-1) into vertex(DMPlex-2) field or vice versa.
>>>>>
>>>>> Basically, DMCreateInterpolation() fails in the example code and it
>>>>> was not allowed to get useful error messages. The DMPlex is created by
>>>>> loading a 2D square box in Gmsh. Can I get some comments on that?
>>>>>
>>>>
>>>> Sure, I am looking at it.
>>>>
>>>>   Thanks,
>>>>
>>>>      Matt
>>>>
>>>>
>>>>> Thanks,
>>>>>
>>>>
>>>>
>>>> --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>> https://www.cse.buffalo.edu/~knepley/
>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>
>>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/~knepley/>
>>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220828/9d64263c/attachment.html>

From narnoldm at umich.edu  Sun Aug 28 11:43:41 2022
From: narnoldm at umich.edu (Nicholas Arnold-Medabalimi)
Date: Sun, 28 Aug 2022 12:43:41 -0400
Subject: [petsc-users] Passive DM Variables
Message-ID: <CAB1dcV41VmLJFdctiN0T2iZUdXDQSXT4cUsbQ=X6H=ydjUUokQ@mail.gmail.com>

Hi

I'm working on a finite volume implementation using DMPlex. I was wondering
what the best-considered way to add cell variables to the DM that we don't
want to be part of the ksp/snes solve. As far as I can tell, if I add an
extra section with those, they will still be part of the F and U vectors in
the solver callbacks (to be expected). But my impression is that that will
just inflate the system size even if they have an F=0? Or is the solver
able to notice variables that are not dependent for the solve.

Alternatively, I have been just making a separate DM with those but that
feels a bit clunky especially if any mesh refinement occurs.

Any advice would be much appreciated.

Thanks
Nicholas

-- 
Nicholas Arnold-Medabalimi

Ph.D. Candidate
Computational Aeroscience Lab
University of Michigan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220828/b18f3a86/attachment.html>

From andreas.hauffe at tu-dresden.de  Sun Aug 28 13:02:26 2022
From: andreas.hauffe at tu-dresden.de (Andreas Hauffe)
Date: Sun, 28 Aug 2022 20:02:26 +0200
Subject: [petsc-users] Possible Bug in PETSC/MUMPS?
Message-ID: <be6a4940-2d4d-f722-cbf4-86cca9e2b601@tu-dresden.de>

Hi list,

I tried to compile PETSC with MUMPS on a Windows platform with intel 
compilers. I think there is a bug in "mumps_io.h" since there is a 
additional underscore in the upper case function names in line 36 and 
line 41. With my configuration I have to change line 36 from 
"F_SYMBOL(dumprhsbinary_c,DUMP_RHSBINARY_C)" to 
"F_SYMBOL(dumprhsbinary_c,DUMPRHSBINARY_C)" and line 41 
"F_SYMBOL(dumpmatbinary_c,DUMP_MATBINARY_C)" to 
"F_SYMBOL(dumpmatbinary_c,DUMPMATBINARY_C)" to compile without errors. 
Is this a bug or am I doing something wrong?

Regards,

-- 
Andreas Hauffe**
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220828/c438d8e7/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5779 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220828/c438d8e7/attachment.p7s>

From andreas.hauffe at tu-dresden.de  Sun Aug 28 13:09:29 2022
From: andreas.hauffe at tu-dresden.de (Andreas Hauffe)
Date: Sun, 28 Aug 2022 20:09:29 +0200
Subject: [petsc-users] Hint for compiling PETSC on Windows with non-english
 language settings
Message-ID: <d4530189-d600-4538-9572-0762fcb1af28@tu-dresden.de>

Hi,

perhaps it is possible to add a short hint in the documentation 
(https://petsc.org/main/install/windows/) for Windows OS with a 
non-english default language. It is important, that the compiler is 
installed with english language pack, so that all compiler output is in 
english. Otherwise the filter during the PETSC configuration will not 
work correctly.

Regards,

-- 
Andreas Hauffe
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220828/ee0a9543/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5779 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220828/ee0a9543/attachment.p7s>

From bsmith at petsc.dev  Sun Aug 28 13:14:58 2022
From: bsmith at petsc.dev (Barry Smith)
Date: Sun, 28 Aug 2022 14:14:58 -0400
Subject: [petsc-users] Hint for compiling PETSC on Windows with
 non-english language settings
In-Reply-To: <d4530189-d600-4538-9572-0762fcb1af28@tu-dresden.de>
References: <d4530189-d600-4538-9572-0762fcb1af28@tu-dresden.de>
Message-ID: <1C8031ED-E998-49A7-A7C1-872FD0619781@petsc.dev>


  Is there a way to automatically check the package language? If so, we will do that check right at the beginning of configure to limit the user's need to read the PETSc documentation before getting to work :-)

> On Aug 28, 2022, at 2:09 PM, Andreas Hauffe <andreas.hauffe at tu-dresden.de> wrote:
> 
> Hi,
> 
> perhaps it is possible to add a short hint in the documentation (https://petsc.org/main/install/windows/ <https://petsc.org/main/install/windows/>) for Windows OS with a non-english default language. It is important, that the compiler is installed with english language pack, so that all compiler output is in english. Otherwise the filter during the PETSC configuration will not work correctly.
> 
> Regards,
> 
> -- 
> Andreas Hauffe

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220828/48ba096a/attachment-0001.html>

From mi.mike1021 at gmail.com  Sun Aug 28 16:35:55 2022
From: mi.mike1021 at gmail.com (Mike Michell)
Date: Sun, 28 Aug 2022 16:35:55 -0500
Subject: [petsc-users] DMPlex Interpolation
In-Reply-To: <CAMYG4Gkzjf=OXBQsErWCumMscGykzTZbLFsuzDbhCtL155o_vA@mail.gmail.com>
References: <CAEc7osYgsXxW9DN_6aHof3wdyqHw3Wv6nLp+svWsZF5c0zz93w@mail.gmail.com>
	<CAMYG4GmiXaNTG5sgjoH=-CdY21OkRaKZh5y5OdPcSiEMj-KfXg@mail.gmail.com>
	<CAEc7osb8HG7ZQQy3=6Jcq-J8jMB1vWXHo-7Y5=fc7A3QR9BQGA@mail.gmail.com>
	<CAMYG4GnfNRGa9yOeCBzfviSfsJ+FxAy73_aRo07VqyCoQETJ+g@mail.gmail.com>
	<CAEc7osad2ee=T6QH5=VHFRjJDsyJ4mO3SYnTF6_BpqkJN0Ppfw@mail.gmail.com>
	<CAMYG4Gkzjf=OXBQsErWCumMscGykzTZbLFsuzDbhCtL155o_vA@mail.gmail.com>
Message-ID: <CAEc7osbxHuf3W66poK_7GpKOTDMExZvN-7KWXpfqe9Oo3CUTvw@mail.gmail.com>

Thank you for the reply.

I think it can be more helpful for me if the attached sample code
(DMInterpolation_Mod.tar) could be checked by you. If you run the sample
code, basically, you would get "VertexField_Map_from_Cell.vtu" file which
displays a solution vector field defined on each vertex, which is mapped
from another solution field defined on each cell-centroid. The vertex
solution field in the vtu file should match with "x-coordinate" value in
the same file. Basically, the attached code tries to reproduce vertex's
x-coordinate value using cell-centroid's x-coordinate value by means of
finite-element space mapping between Q0 and Q1.
>From the vtu file, it is obvious that at the physical boundaries (left,
right, top, and bottom) x-coordinate value is not properly mapped from
cell-centroid, whereas interior vertex's solution seems okay. What I do not
understand is why the boundary vertices cannot have properly mapped
coordinate values? I believe I am doing something wrong and I was
suspicious about the ghost cell, which I do not have. But your comment
seems to say that the problem is not related to the ghost cell. What am I
doing wrong?

Does your section mark these unknowns as constrained?
--> Yes, as shown in the code, I defined one section for each dmplex, one
for vertex and one for cell, respectively, with 1-dof for each (See code
line from 91 to 124).

You should not get 0 unless you constrained these variables in the section.
If not, you should see it use the constant values in the cell.
--> Yes, you are correct. Now I do not have zero value at the physical
boundaries, however, it is still not proper value and seems underscaled,
compared to the proper answer (i.e., compared with the properly mapped
interior values).

Thanks,


> On Sun, Aug 28, 2022 at 12:11 PM Mike Michell <mi.mike1021 at gmail.com>
> wrote:
>
>> Thank you for the reply.
>>
>> *I cannot quite understand. Are you saying that you have inhomogeneous
>> Dirichlet conditions on the boundary, and a 0 guess in the interior, and
>> you get all zeros from interpolation? Yes, we have no way of getting
>> boundary values into the operator. You could conceivably make an operator
>> that mapped between local vectors, but it is hard to see why you would want
>> this. In the solver, we usually  just care about the unknowns we are
>> solving for. Did I understand your question, or is it about something else?*
>> --> It is not imposing a physical boundary condition (Dirichlet condition
>> for example), but my solver needs to import an external solution vector
>> field resolved to each cell-centroid, then I need to map it to a vertex
>> field in which my solver is resolved for solving governing equations. Then,
>> my code solves its equations. After my solver is done solving the
>> equations, the vertex field should be re-map into cell-field and then push
>> back the cell field to the external code. With this context, I have zero
>> values returned at the vertices on the physical boundaries
>>
>
> Does your section mark these unknowns as constrained?
>
>
>> if I try mapping from cell-center to vertex field without having ghost
>> cell layers inside physical boundaries. For interior vertices, I get
>> reasonable mapped values, although the mapping accuracy seems to be able to
>> be improved as I mentioned in the earlier email.
>>
>
> As I replied, I think this is not true.
>
>
>> If dmplex for cell-center field would be defined to have ghost cell
>> layer, and fill it (it meant import ghost cell values as well from the
>> external code), and map it to vertex-field, can it be possible to fill
>> vertex points on the physical boundaries?
>>
>
> You should not get 0 unless you constrained these variables in the
> section. If not, you should see it use the constant values in the cell.
>
>   Thanks,
>
>      Matt
>
>
>> I was trying to test this strategy, but it
>> seems DMPlexComputeCellGeometryFVM() gives NaN values if dmplex has ghost
>> cell by calling DMPlexConstructGhostCells(). In detail,
>> DMPlexComputeCellGeometryFVM() gives zero area for each cell, and returns
>> NaN values for cell centroids and normals for the ghost cells.
>>
>> Thanks,
>>
>>
>>> On Sun, Aug 28, 2022 at 8:51 AM Mike Michell <mi.mike1021 at gmail.com>
>>> wrote:
>>>
>>>> Hi, thank you for the reply.
>>>>
>>>> I was able to manage mapping from cell-center to vertex. Basically, in
>>>> Fortran, it seems DMCreateInterpolation() requires the optional scaling
>>>> vector as a mandatory argument, which is strange.
>>>>
>>>
>>> It needs a custom Fortran wrapper to check for the NULL input from
>>> Fortran.
>>>
>>>
>>>> My dmplex has zero overlap layer over the procs, and there is no ghost
>>>> cell inside physical boundary. In this case, it seems mapping between cell
>>>> center to vertex returns zero (in case the global cell vector initialized
>>>> to zero) value at the nodes, which are located at physical boundary. To
>>>> manage this problem, is it mandatory to have ghost cells. Is this correct
>>>> understanding?
>>>>
>>>
>>> I cannot quite understand. Are you saying that you have inhomogeneous
>>> Dirichlet conditions on the boundary, and a 0 guess in the interior, and
>>> you get all zeros from interpolation? Yes, we have no way of getting
>>> boundary values into the operator. You could conceivably make an operator
>>> that mapped between local vectors, but it is hard to see why you would want
>>> this. In the solver, we usually  just care about the unknowns we are
>>> solving for.
>>>
>>> Did I understand your question, or is it about something else?
>>>
>>>
>>>> Also, my mapping accuracy itself seems to be improved. For simple test,
>>>> cell-center's x-coordinate value of each cell mapped to node and printed to
>>>> the vertex field as .vtu format, and the mapped x-vertex-coordinate is
>>>> quite different with the actual nodal coordinate values what PETSc
>>>> intrinsically provides through DMGetCoordinatesLocal(). I believe I am
>>>> doing something wrong, probably arguments in PetscFECreateLagrange() can
>>>> improve the mapping accuracy in finite-element space?
>>>>
>>>
>>> Yes, a cellwise field is much less accurate than a linear field. Pushing
>>> the values up to a linear field does not make them more accurate.
>>>
>>>   Thanks,
>>>
>>>      Matt
>>>
>>>
>>>> Thanks,
>>>>
>>>>
>>>>> On Thu, Aug 25, 2022 at 7:12 PM Mike Michell <mi.mike1021 at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi, this is a duplication of
>>>>>>
>>>>>> https://lists.mcs.anl.gov/pipermail/petsc-users/2022-August/046746.html
>>>>>>
>>>>>> for in-depth question.
>>>>>>
>>>>>> I wrote a short code as attached to test interpolation between two
>>>>>> DMPlex objects. Goal is to map solution field defined on
>>>>>> cell-centroid(DMPlex-1) into vertex(DMPlex-2) field or vice versa.
>>>>>>
>>>>>> Basically, DMCreateInterpolation() fails in the example code and it
>>>>>> was not allowed to get useful error messages. The DMPlex is created by
>>>>>> loading a 2D square box in Gmsh. Can I get some comments on that?
>>>>>>
>>>>>
>>>>> Sure, I am looking at it.
>>>>>
>>>>>   Thanks,
>>>>>
>>>>>      Matt
>>>>>
>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> What most experimenters take for granted before they begin their
>>>>> experiments is infinitely more interesting than any results to which their
>>>>> experiments lead.
>>>>> -- Norbert Wiener
>>>>>
>>>>> https://www.cse.buffalo.edu/~knepley/
>>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>>
>>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>> https://www.cse.buffalo.edu/~knepley/
>>> <http://www.cse.buffalo.edu/~knepley/>
>>>
>>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220828/4358ffba/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: DMInterpolation_Mod.tar
Type: application/x-tar
Size: 143360 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220828/4358ffba/attachment-0001.tar>

From knepley at gmail.com  Sun Aug 28 18:17:37 2022
From: knepley at gmail.com (Matthew Knepley)
Date: Sun, 28 Aug 2022 19:17:37 -0400
Subject: [petsc-users] DMPlex Interpolation
In-Reply-To: <CAEc7osbxHuf3W66poK_7GpKOTDMExZvN-7KWXpfqe9Oo3CUTvw@mail.gmail.com>
References: <CAEc7osYgsXxW9DN_6aHof3wdyqHw3Wv6nLp+svWsZF5c0zz93w@mail.gmail.com>
	<CAMYG4GmiXaNTG5sgjoH=-CdY21OkRaKZh5y5OdPcSiEMj-KfXg@mail.gmail.com>
	<CAEc7osb8HG7ZQQy3=6Jcq-J8jMB1vWXHo-7Y5=fc7A3QR9BQGA@mail.gmail.com>
	<CAMYG4GnfNRGa9yOeCBzfviSfsJ+FxAy73_aRo07VqyCoQETJ+g@mail.gmail.com>
	<CAEc7osad2ee=T6QH5=VHFRjJDsyJ4mO3SYnTF6_BpqkJN0Ppfw@mail.gmail.com>
	<CAMYG4Gkzjf=OXBQsErWCumMscGykzTZbLFsuzDbhCtL155o_vA@mail.gmail.com>
	<CAEc7osbxHuf3W66poK_7GpKOTDMExZvN-7KWXpfqe9Oo3CUTvw@mail.gmail.com>
Message-ID: <CAMYG4G=L9sWMVWyUuT8fiNxE1VEw3vg4NxrihTEBsvqV3O7nHw@mail.gmail.com>

On Sun, Aug 28, 2022 at 5:36 PM Mike Michell <mi.mike1021 at gmail.com> wrote:

> Thank you for the reply.
>
> I think it can be more helpful for me if the attached sample code
> (DMInterpolation_Mod.tar) could be checked by you.
>

Okay, you are right. I will run it tomorrow.

  Thanks,

     Matt


> If you run the sample code, basically, you would get
> "VertexField_Map_from_Cell.vtu" file which displays a solution vector field
> defined on each vertex, which is mapped from another solution field defined
> on each cell-centroid. The vertex solution field in the vtu file should
> match with "x-coordinate" value in the same file. Basically, the attached
> code tries to reproduce vertex's x-coordinate value using
> cell-centroid's x-coordinate value by means of finite-element space mapping
> between Q0 and Q1.
> From the vtu file, it is obvious that at the physical boundaries (left,
> right, top, and bottom) x-coordinate value is not properly mapped from
> cell-centroid, whereas interior vertex's solution seems okay. What I do not
> understand is why the boundary vertices cannot have properly mapped
> coordinate values? I believe I am doing something wrong and I was
> suspicious about the ghost cell, which I do not have. But your comment
> seems to say that the problem is not related to the ghost cell. What am I
> doing wrong?
>
> Does your section mark these unknowns as constrained?
> --> Yes, as shown in the code, I defined one section for each dmplex, one
> for vertex and one for cell, respectively, with 1-dof for each (See code
> line from 91 to 124).
>
> You should not get 0 unless you constrained these variables in the
> section. If not, you should see it use the constant values in the cell.
> --> Yes, you are correct. Now I do not have zero value at the physical
> boundaries, however, it is still not proper value and seems underscaled,
> compared to the proper answer (i.e., compared with the properly mapped
> interior values).
>
> Thanks,
>
>
>> On Sun, Aug 28, 2022 at 12:11 PM Mike Michell <mi.mike1021 at gmail.com>
>> wrote:
>>
>>> Thank you for the reply.
>>>
>>> *I cannot quite understand. Are you saying that you have inhomogeneous
>>> Dirichlet conditions on the boundary, and a 0 guess in the interior, and
>>> you get all zeros from interpolation? Yes, we have no way of getting
>>> boundary values into the operator. You could conceivably make an operator
>>> that mapped between local vectors, but it is hard to see why you would want
>>> this. In the solver, we usually  just care about the unknowns we are
>>> solving for. Did I understand your question, or is it about something else?*
>>> --> It is not imposing a physical boundary condition (Dirichlet
>>> condition for example), but my solver needs to import an external solution
>>> vector field resolved to each cell-centroid, then I need to map it to a
>>> vertex field in which my solver is resolved for solving governing
>>> equations. Then, my code solves its equations. After my solver is done
>>> solving the equations, the vertex field should be re-map into cell-field
>>> and then push back the cell field to the external code. With this context,
>>> I have zero values returned at the vertices on the physical boundaries
>>>
>>
>> Does your section mark these unknowns as constrained?
>>
>>
>>> if I try mapping from cell-center to vertex field without having ghost
>>> cell layers inside physical boundaries. For interior vertices, I get
>>> reasonable mapped values, although the mapping accuracy seems to be able to
>>> be improved as I mentioned in the earlier email.
>>>
>>
>> As I replied, I think this is not true.
>>
>>
>>> If dmplex for cell-center field would be defined to have ghost cell
>>> layer, and fill it (it meant import ghost cell values as well from the
>>> external code), and map it to vertex-field, can it be possible to fill
>>> vertex points on the physical boundaries?
>>>
>>
>> You should not get 0 unless you constrained these variables in the
>> section. If not, you should see it use the constant values in the cell.
>>
>>   Thanks,
>>
>>      Matt
>>
>>
>>> I was trying to test this strategy, but it
>>> seems DMPlexComputeCellGeometryFVM() gives NaN values if dmplex has ghost
>>> cell by calling DMPlexConstructGhostCells(). In detail,
>>> DMPlexComputeCellGeometryFVM() gives zero area for each cell, and returns
>>> NaN values for cell centroids and normals for the ghost cells.
>>>
>>> Thanks,
>>>
>>>
>>>> On Sun, Aug 28, 2022 at 8:51 AM Mike Michell <mi.mike1021 at gmail.com>
>>>> wrote:
>>>>
>>>>> Hi, thank you for the reply.
>>>>>
>>>>> I was able to manage mapping from cell-center to vertex. Basically, in
>>>>> Fortran, it seems DMCreateInterpolation() requires the optional scaling
>>>>> vector as a mandatory argument, which is strange.
>>>>>
>>>>
>>>> It needs a custom Fortran wrapper to check for the NULL input from
>>>> Fortran.
>>>>
>>>>
>>>>> My dmplex has zero overlap layer over the procs, and there is no ghost
>>>>> cell inside physical boundary. In this case, it seems mapping between cell
>>>>> center to vertex returns zero (in case the global cell vector initialized
>>>>> to zero) value at the nodes, which are located at physical boundary. To
>>>>> manage this problem, is it mandatory to have ghost cells. Is this correct
>>>>> understanding?
>>>>>
>>>>
>>>> I cannot quite understand. Are you saying that you have inhomogeneous
>>>> Dirichlet conditions on the boundary, and a 0 guess in the interior, and
>>>> you get all zeros from interpolation? Yes, we have no way of getting
>>>> boundary values into the operator. You could conceivably make an operator
>>>> that mapped between local vectors, but it is hard to see why you would want
>>>> this. In the solver, we usually  just care about the unknowns we are
>>>> solving for.
>>>>
>>>> Did I understand your question, or is it about something else?
>>>>
>>>>
>>>>> Also, my mapping accuracy itself seems to be improved. For simple
>>>>> test, cell-center's x-coordinate value of each cell mapped to node and
>>>>> printed to the vertex field as .vtu format, and the mapped
>>>>> x-vertex-coordinate is quite different with the actual nodal coordinate
>>>>> values what PETSc intrinsically provides through DMGetCoordinatesLocal(). I
>>>>> believe I am doing something wrong, probably arguments
>>>>> in PetscFECreateLagrange() can improve the mapping accuracy in
>>>>> finite-element space?
>>>>>
>>>>
>>>> Yes, a cellwise field is much less accurate than a linear field.
>>>> Pushing the values up to a linear field does not make them more accurate.
>>>>
>>>>   Thanks,
>>>>
>>>>      Matt
>>>>
>>>>
>>>>> Thanks,
>>>>>
>>>>>
>>>>>> On Thu, Aug 25, 2022 at 7:12 PM Mike Michell <mi.mike1021 at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi, this is a duplication of
>>>>>>>
>>>>>>> https://lists.mcs.anl.gov/pipermail/petsc-users/2022-August/046746.html
>>>>>>>
>>>>>>> for in-depth question.
>>>>>>>
>>>>>>> I wrote a short code as attached to test interpolation between two
>>>>>>> DMPlex objects. Goal is to map solution field defined on
>>>>>>> cell-centroid(DMPlex-1) into vertex(DMPlex-2) field or vice versa.
>>>>>>>
>>>>>>> Basically, DMCreateInterpolation() fails in the example code and it
>>>>>>> was not allowed to get useful error messages. The DMPlex is created by
>>>>>>> loading a 2D square box in Gmsh. Can I get some comments on that?
>>>>>>>
>>>>>>
>>>>>> Sure, I am looking at it.
>>>>>>
>>>>>>   Thanks,
>>>>>>
>>>>>>      Matt
>>>>>>
>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> What most experimenters take for granted before they begin their
>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>> experiments lead.
>>>>>> -- Norbert Wiener
>>>>>>
>>>>>> https://www.cse.buffalo.edu/~knepley/
>>>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>>>
>>>>>
>>>>
>>>> --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>> https://www.cse.buffalo.edu/~knepley/
>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>
>>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/~knepley/>
>>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220828/25ea6792/attachment.html>

From mfadams at lbl.gov  Mon Aug 29 06:34:35 2022
From: mfadams at lbl.gov (Mark Adams)
Date: Mon, 29 Aug 2022 07:34:35 -0400
Subject: [petsc-users] Passive DM Variables
In-Reply-To: <CAB1dcV41VmLJFdctiN0T2iZUdXDQSXT4cUsbQ=X6H=ydjUUokQ@mail.gmail.com>
References: <CAB1dcV41VmLJFdctiN0T2iZUdXDQSXT4cUsbQ=X6H=ydjUUokQ@mail.gmail.com>
Message-ID: <CADOhEh5W+f-O4hSR9vmDuO_9VRsx8SS_rauDYbNbrVvkKmLmkg@mail.gmail.com>

You want "auxiliary" variables.

There may be better examples but, try looking at
https://petsc.org/release/src/snes/tutorials/ex12.c.html
And search for "aux".

Mark

On Sun, Aug 28, 2022 at 12:44 PM Nicholas Arnold-Medabalimi <
narnoldm at umich.edu> wrote:

> Hi
>
> I'm working on a finite volume implementation using DMPlex. I was
> wondering what the best-considered way to add cell variables to the DM that
> we don't want to be part of the ksp/snes solve. As far as I can tell, if I
> add an extra section with those, they will still be part of the F and U
> vectors in the solver callbacks (to be expected). But my impression is that
> that will just inflate the system size even if they have an F=0? Or is the
> solver able to notice variables that are not dependent for the solve.
>
> Alternatively, I have been just making a separate DM with those but that
> feels a bit clunky especially if any mesh refinement occurs.
>
> Any advice would be much appreciated.
>
> Thanks
> Nicholas
>
> --
> Nicholas Arnold-Medabalimi
>
> Ph.D. Candidate
> Computational Aeroscience Lab
> University of Michigan
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220829/f8b0ead5/attachment-0001.html>

From knepley at gmail.com  Mon Aug 29 07:01:12 2022
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 29 Aug 2022 08:01:12 -0400
Subject: [petsc-users] Passive DM Variables
In-Reply-To: <CAB1dcV41VmLJFdctiN0T2iZUdXDQSXT4cUsbQ=X6H=ydjUUokQ@mail.gmail.com>
References: <CAB1dcV41VmLJFdctiN0T2iZUdXDQSXT4cUsbQ=X6H=ydjUUokQ@mail.gmail.com>
Message-ID: <CAMYG4GmcneNvhkEoQ1SunWBgk5JM2aAHLdWY=ceKKMoibGV8zQ@mail.gmail.com>

On Sun, Aug 28, 2022 at 12:44 PM Nicholas Arnold-Medabalimi <
narnoldm at umich.edu> wrote:

> Hi
>
> I'm working on a finite volume implementation using DMPlex. I was
> wondering what the best-considered way to add cell variables to the DM that
> we don't want to be part of the ksp/snes solve. As far as I can tell, if I
> add an extra section with those, they will still be part of the F and U
> vectors in the solver callbacks (to be expected). But my impression is that
> that will just inflate the system size even if they have an F=0? Or is the
> solver able to notice variables that are not dependent for the solve.
>
> Alternatively, I have been just making a separate DM with those but that
> feels a bit clunky especially if any mesh refinement occurs.
>

Mark is correct. We have been using a two DM strategy. You can attach this
auxiliary data using

  https://petsc.org/main/docs/manualpages/DM/DMSetAuxiliaryVec/

The idea is that you define a layout for your extra information using a new
DM, then create a Vec for it, and attach it to the DM. This
can  be pulled out by any callback function, and the DM can be pulled out
of the vector. In the Plex pointwise functions for residual
and Jacobian, auxiliary data is automatically interpolated to the
quadrature points as well.

Is this what you were looking for?

  Thanks,

     Matt


> Any advice would be much appreciated.
>
> Thanks
> Nicholas
>
> --
> Nicholas Arnold-Medabalimi
>
> Ph.D. Candidate
> Computational Aeroscience Lab
> University of Michigan
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220829/ed0b4512/attachment.html>

From knepley at gmail.com  Mon Aug 29 08:55:02 2022
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 29 Aug 2022 09:55:02 -0400
Subject: [petsc-users] DMPlex Interpolation
In-Reply-To: <CAMYG4G=L9sWMVWyUuT8fiNxE1VEw3vg4NxrihTEBsvqV3O7nHw@mail.gmail.com>
References: <CAEc7osYgsXxW9DN_6aHof3wdyqHw3Wv6nLp+svWsZF5c0zz93w@mail.gmail.com>
	<CAMYG4GmiXaNTG5sgjoH=-CdY21OkRaKZh5y5OdPcSiEMj-KfXg@mail.gmail.com>
	<CAEc7osb8HG7ZQQy3=6Jcq-J8jMB1vWXHo-7Y5=fc7A3QR9BQGA@mail.gmail.com>
	<CAMYG4GnfNRGa9yOeCBzfviSfsJ+FxAy73_aRo07VqyCoQETJ+g@mail.gmail.com>
	<CAEc7osad2ee=T6QH5=VHFRjJDsyJ4mO3SYnTF6_BpqkJN0Ppfw@mail.gmail.com>
	<CAMYG4Gkzjf=OXBQsErWCumMscGykzTZbLFsuzDbhCtL155o_vA@mail.gmail.com>
	<CAEc7osbxHuf3W66poK_7GpKOTDMExZvN-7KWXpfqe9Oo3CUTvw@mail.gmail.com>
	<CAMYG4G=L9sWMVWyUuT8fiNxE1VEw3vg4NxrihTEBsvqV3O7nHw@mail.gmail.com>
Message-ID: <CAMYG4GksbOAY1-3L2v97A3B0mD-bKiB4dCOks7EOsL1UvaAa=w@mail.gmail.com>

On Sun, Aug 28, 2022 at 7:17 PM Matthew Knepley <knepley at gmail.com> wrote:

> On Sun, Aug 28, 2022 at 5:36 PM Mike Michell <mi.mike1021 at gmail.com>
> wrote:
>
>> Thank you for the reply.
>>
>> I think it can be more helpful for me if the attached sample code
>> (DMInterpolation_Mod.tar) could be checked by you.
>>
>
> Okay, you are right. I will run it tomorrow.
>

The problem was somewhat conceptual. When you are restricting state,
meaning a function you want to use directly, rather than
residuals, meaning some average or integral, you want to preserve scale.
This is what the 'vscale' vector is for. I added the
VecPointwiseMult to your code (attached) to do this. I also changed to a
more general mehs creation. To get your run, use

  ./App.x -dm_plex_filename square.msh

There is inaccuracy, but it is order O(h) just like the interpolation error
from constants. You can see this by refining the mesh

  ./App.x -dm_plex_simplex 0 -dm_plex_box_faces 25,25

I am attaching the picture. The last value is 0.98 instead of 1, but the
error is decreasing like h.

  Thanks,

     Matt


>   Thanks,
>
>      Matt
>
>
>> If you run the sample code, basically, you would get
>> "VertexField_Map_from_Cell.vtu" file which displays a solution vector field
>> defined on each vertex, which is mapped from another solution field defined
>> on each cell-centroid. The vertex solution field in the vtu file should
>> match with "x-coordinate" value in the same file. Basically, the attached
>> code tries to reproduce vertex's x-coordinate value using
>> cell-centroid's x-coordinate value by means of finite-element space mapping
>> between Q0 and Q1.
>> From the vtu file, it is obvious that at the physical boundaries (left,
>> right, top, and bottom) x-coordinate value is not properly mapped from
>> cell-centroid, whereas interior vertex's solution seems okay. What I do not
>> understand is why the boundary vertices cannot have properly mapped
>> coordinate values? I believe I am doing something wrong and I was
>> suspicious about the ghost cell, which I do not have. But your comment
>> seems to say that the problem is not related to the ghost cell. What am I
>> doing wrong?
>>
>> Does your section mark these unknowns as constrained?
>> --> Yes, as shown in the code, I defined one section for each dmplex, one
>> for vertex and one for cell, respectively, with 1-dof for each (See code
>> line from 91 to 124).
>>
>> You should not get 0 unless you constrained these variables in the
>> section. If not, you should see it use the constant values in the cell.
>> --> Yes, you are correct. Now I do not have zero value at the physical
>> boundaries, however, it is still not proper value and seems underscaled,
>> compared to the proper answer (i.e., compared with the properly mapped
>> interior values).
>>
>> Thanks,
>>
>>
>>> On Sun, Aug 28, 2022 at 12:11 PM Mike Michell <mi.mike1021 at gmail.com>
>>> wrote:
>>>
>>>> Thank you for the reply.
>>>>
>>>> *I cannot quite understand. Are you saying that you have inhomogeneous
>>>> Dirichlet conditions on the boundary, and a 0 guess in the interior, and
>>>> you get all zeros from interpolation? Yes, we have no way of getting
>>>> boundary values into the operator. You could conceivably make an operator
>>>> that mapped between local vectors, but it is hard to see why you would want
>>>> this. In the solver, we usually  just care about the unknowns we are
>>>> solving for. Did I understand your question, or is it about something else?*
>>>> --> It is not imposing a physical boundary condition (Dirichlet
>>>> condition for example), but my solver needs to import an external solution
>>>> vector field resolved to each cell-centroid, then I need to map it to a
>>>> vertex field in which my solver is resolved for solving governing
>>>> equations. Then, my code solves its equations. After my solver is done
>>>> solving the equations, the vertex field should be re-map into cell-field
>>>> and then push back the cell field to the external code. With this context,
>>>> I have zero values returned at the vertices on the physical boundaries
>>>>
>>>
>>> Does your section mark these unknowns as constrained?
>>>
>>>
>>>> if I try mapping from cell-center to vertex field without having ghost
>>>> cell layers inside physical boundaries. For interior vertices, I get
>>>> reasonable mapped values, although the mapping accuracy seems to be able to
>>>> be improved as I mentioned in the earlier email.
>>>>
>>>
>>> As I replied, I think this is not true.
>>>
>>>
>>>> If dmplex for cell-center field would be defined to have ghost cell
>>>> layer, and fill it (it meant import ghost cell values as well from the
>>>> external code), and map it to vertex-field, can it be possible to fill
>>>> vertex points on the physical boundaries?
>>>>
>>>
>>> You should not get 0 unless you constrained these variables in the
>>> section. If not, you should see it use the constant values in the cell.
>>>
>>>   Thanks,
>>>
>>>      Matt
>>>
>>>
>>>> I was trying to test this strategy, but it
>>>> seems DMPlexComputeCellGeometryFVM() gives NaN values if dmplex has ghost
>>>> cell by calling DMPlexConstructGhostCells(). In detail,
>>>> DMPlexComputeCellGeometryFVM() gives zero area for each cell, and returns
>>>> NaN values for cell centroids and normals for the ghost cells.
>>>>
>>>> Thanks,
>>>>
>>>>
>>>>> On Sun, Aug 28, 2022 at 8:51 AM Mike Michell <mi.mike1021 at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi, thank you for the reply.
>>>>>>
>>>>>> I was able to manage mapping from cell-center to vertex. Basically,
>>>>>> in Fortran, it seems DMCreateInterpolation() requires the optional scaling
>>>>>> vector as a mandatory argument, which is strange.
>>>>>>
>>>>>
>>>>> It needs a custom Fortran wrapper to check for the NULL input from
>>>>> Fortran.
>>>>>
>>>>>
>>>>>> My dmplex has zero overlap layer over the procs, and there is no
>>>>>> ghost cell inside physical boundary. In this case, it seems mapping between
>>>>>> cell center to vertex returns zero (in case the global cell vector
>>>>>> initialized to zero) value at the nodes, which are located at physical
>>>>>> boundary. To manage this problem, is it mandatory to have ghost cells. Is
>>>>>> this correct understanding?
>>>>>>
>>>>>
>>>>> I cannot quite understand. Are you saying that you have inhomogeneous
>>>>> Dirichlet conditions on the boundary, and a 0 guess in the interior, and
>>>>> you get all zeros from interpolation? Yes, we have no way of getting
>>>>> boundary values into the operator. You could conceivably make an operator
>>>>> that mapped between local vectors, but it is hard to see why you would want
>>>>> this. In the solver, we usually  just care about the unknowns we are
>>>>> solving for.
>>>>>
>>>>> Did I understand your question, or is it about something else?
>>>>>
>>>>>
>>>>>> Also, my mapping accuracy itself seems to be improved. For simple
>>>>>> test, cell-center's x-coordinate value of each cell mapped to node and
>>>>>> printed to the vertex field as .vtu format, and the mapped
>>>>>> x-vertex-coordinate is quite different with the actual nodal coordinate
>>>>>> values what PETSc intrinsically provides through DMGetCoordinatesLocal(). I
>>>>>> believe I am doing something wrong, probably arguments
>>>>>> in PetscFECreateLagrange() can improve the mapping accuracy in
>>>>>> finite-element space?
>>>>>>
>>>>>
>>>>> Yes, a cellwise field is much less accurate than a linear field.
>>>>> Pushing the values up to a linear field does not make them more accurate.
>>>>>
>>>>>   Thanks,
>>>>>
>>>>>      Matt
>>>>>
>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>>
>>>>>>> On Thu, Aug 25, 2022 at 7:12 PM Mike Michell <mi.mike1021 at gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi, this is a duplication of
>>>>>>>>
>>>>>>>> https://lists.mcs.anl.gov/pipermail/petsc-users/2022-August/046746.html
>>>>>>>>
>>>>>>>> for in-depth question.
>>>>>>>>
>>>>>>>> I wrote a short code as attached to test interpolation between two
>>>>>>>> DMPlex objects. Goal is to map solution field defined on
>>>>>>>> cell-centroid(DMPlex-1) into vertex(DMPlex-2) field or vice versa.
>>>>>>>>
>>>>>>>> Basically, DMCreateInterpolation() fails in the example code and it
>>>>>>>> was not allowed to get useful error messages. The DMPlex is created by
>>>>>>>> loading a 2D square box in Gmsh. Can I get some comments on that?
>>>>>>>>
>>>>>>>
>>>>>>> Sure, I am looking at it.
>>>>>>>
>>>>>>>   Thanks,
>>>>>>>
>>>>>>>      Matt
>>>>>>>
>>>>>>>
>>>>>>>> Thanks,
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> What most experimenters take for granted before they begin their
>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>> experiments lead.
>>>>>>> -- Norbert Wiener
>>>>>>>
>>>>>>> https://www.cse.buffalo.edu/~knepley/
>>>>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> What most experimenters take for granted before they begin their
>>>>> experiments is infinitely more interesting than any results to which their
>>>>> experiments lead.
>>>>> -- Norbert Wiener
>>>>>
>>>>> https://www.cse.buffalo.edu/~knepley/
>>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>>
>>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>> https://www.cse.buffalo.edu/~knepley/
>>> <http://www.cse.buffalo.edu/~knepley/>
>>>
>>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220829/48914cf0/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: test.F90
Type: application/octet-stream
Size: 10351 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220829/48914cf0/attachment-0001.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: x.png
Type: image/png
Size: 194379 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220829/48914cf0/attachment-0001.png>

From wence at gmx.li  Mon Aug 29 09:22:28 2022
From: wence at gmx.li (Lawrence Mitchell)
Date: Mon, 29 Aug 2022 15:22:28 +0100
Subject: [petsc-users] DMPlex Interpolation
In-Reply-To: <CAMYG4GksbOAY1-3L2v97A3B0mD-bKiB4dCOks7EOsL1UvaAa=w@mail.gmail.com>
References: <CAEc7osYgsXxW9DN_6aHof3wdyqHw3Wv6nLp+svWsZF5c0zz93w@mail.gmail.com>
	<CAMYG4GmiXaNTG5sgjoH=-CdY21OkRaKZh5y5OdPcSiEMj-KfXg@mail.gmail.com>
	<CAEc7osb8HG7ZQQy3=6Jcq-J8jMB1vWXHo-7Y5=fc7A3QR9BQGA@mail.gmail.com>
	<CAMYG4GnfNRGa9yOeCBzfviSfsJ+FxAy73_aRo07VqyCoQETJ+g@mail.gmail.com>
	<CAEc7osad2ee=T6QH5=VHFRjJDsyJ4mO3SYnTF6_BpqkJN0Ppfw@mail.gmail.com>
	<CAMYG4Gkzjf=OXBQsErWCumMscGykzTZbLFsuzDbhCtL155o_vA@mail.gmail.com>
	<CAEc7osbxHuf3W66poK_7GpKOTDMExZvN-7KWXpfqe9Oo3CUTvw@mail.gmail.com>
	<CAMYG4G=L9sWMVWyUuT8fiNxE1VEw3vg4NxrihTEBsvqV3O7nHw@mail.gmail.com>
	<CAMYG4GksbOAY1-3L2v97A3B0mD-bKiB4dCOks7EOsL1UvaAa=w@mail.gmail.com>
Message-ID: <CA+wRr2=jsm1hbdVWyL=TAH478DhkXrR-wY_Ue4y1NUBDXEAULg@mail.gmail.com>

On Mon, 29 Aug 2022 at 14:55, Matthew Knepley <knepley at gmail.com> wrote:
>
> On Sun, Aug 28, 2022 at 7:17 PM Matthew Knepley <knepley at gmail.com> wrote:
>>
>> On Sun, Aug 28, 2022 at 5:36 PM Mike Michell <mi.mike1021 at gmail.com> wrote:
>>>
>>> Thank you for the reply.
>>>
>>> I think it can be more helpful for me if the attached sample code (DMInterpolation_Mod.tar) could be checked by you.
>>
>>
>> Okay, you are right. I will run it tomorrow.
>
>
> The problem was somewhat conceptual. When you are restricting state, meaning a function you want to use directly, rather than
> residuals, meaning some average or integral, you want to preserve scale. This is what the 'vscale' vector is for. I added the
> VecPointwiseMult to your code (attached) to do this. I also changed to a more general mehs creation. To get your run, use

The right answer is to use DMCreateInjection
(https://petsc.org/release/docs/manualpages/DM/DMCreateInjection.html)
to move fine grid state to coarse grid state (rather than
DMCreateInterpolation and then VecPointwiseMult, which doesn't work
for most discretisations).

Matt: the interface to DMCreateInjection is now correct (it returns a
Mat, rather than a VecScatter) so DMPlexComputeInjectorFEM
(https://petsc.org/release/docs/manualpages/DMPLEX/DMPlexComputeInjectorFEM.html)
should be updated to take advantage.

Lawrence

From narnoldm at umich.edu  Mon Aug 29 10:24:51 2022
From: narnoldm at umich.edu (Nicholas Arnold-Medabalimi)
Date: Mon, 29 Aug 2022 11:24:51 -0400
Subject: [petsc-users] Passive DM Variables
In-Reply-To: <CAMYG4GmcneNvhkEoQ1SunWBgk5JM2aAHLdWY=ceKKMoibGV8zQ@mail.gmail.com>
References: <CAB1dcV41VmLJFdctiN0T2iZUdXDQSXT4cUsbQ=X6H=ydjUUokQ@mail.gmail.com>
	<CAMYG4GmcneNvhkEoQ1SunWBgk5JM2aAHLdWY=ceKKMoibGV8zQ@mail.gmail.com>
Message-ID: <CAB1dcV63yng5LZWdS1njg4H7SM5W8RmnzHK3xWVUpWNs8uKorg@mail.gmail.com>

Yes, this is exactly what I was looking for. Thank you all.

On Mon, Aug 29, 2022 at 8:02 AM Matthew Knepley <knepley at gmail.com> wrote:

> On Sun, Aug 28, 2022 at 12:44 PM Nicholas Arnold-Medabalimi <
> narnoldm at umich.edu> wrote:
>
>> Hi
>>
>> I'm working on a finite volume implementation using DMPlex. I was
>> wondering what the best-considered way to add cell variables to the DM that
>> we don't want to be part of the ksp/snes solve. As far as I can tell, if I
>> add an extra section with those, they will still be part of the F and U
>> vectors in the solver callbacks (to be expected). But my impression is that
>> that will just inflate the system size even if they have an F=0? Or is the
>> solver able to notice variables that are not dependent for the solve.
>>
>> Alternatively, I have been just making a separate DM with those but that
>> feels a bit clunky especially if any mesh refinement occurs.
>>
>
> Mark is correct. We have been using a two DM strategy. You can attach this
> auxiliary data using
>
>   https://petsc.org/main/docs/manualpages/DM/DMSetAuxiliaryVec/
>
> The idea is that you define a layout for your extra information using a
> new DM, then create a Vec for it, and attach it to the DM. This
> can  be pulled out by any callback function, and the DM can be pulled out
> of the vector. In the Plex pointwise functions for residual
> and Jacobian, auxiliary data is automatically interpolated to the
> quadrature points as well.
>
> Is this what you were looking for?
>
>   Thanks,
>
>      Matt
>
>
>> Any advice would be much appreciated.
>>
>> Thanks
>> Nicholas
>>
>> --
>> Nicholas Arnold-Medabalimi
>>
>> Ph.D. Candidate
>> Computational Aeroscience Lab
>> University of Michigan
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>


-- 
Nicholas Arnold-Medabalimi

Ph.D. Candidate
Computational Aeroscience Lab
University of Michigan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220829/0ad06dbd/attachment.html>

From knepley at gmail.com  Mon Aug 29 12:58:07 2022
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 29 Aug 2022 13:58:07 -0400
Subject: [petsc-users] DMPlex Interpolation
In-Reply-To: <CA+wRr2=jsm1hbdVWyL=TAH478DhkXrR-wY_Ue4y1NUBDXEAULg@mail.gmail.com>
References: <CAEc7osYgsXxW9DN_6aHof3wdyqHw3Wv6nLp+svWsZF5c0zz93w@mail.gmail.com>
	<CAMYG4GmiXaNTG5sgjoH=-CdY21OkRaKZh5y5OdPcSiEMj-KfXg@mail.gmail.com>
	<CAEc7osb8HG7ZQQy3=6Jcq-J8jMB1vWXHo-7Y5=fc7A3QR9BQGA@mail.gmail.com>
	<CAMYG4GnfNRGa9yOeCBzfviSfsJ+FxAy73_aRo07VqyCoQETJ+g@mail.gmail.com>
	<CAEc7osad2ee=T6QH5=VHFRjJDsyJ4mO3SYnTF6_BpqkJN0Ppfw@mail.gmail.com>
	<CAMYG4Gkzjf=OXBQsErWCumMscGykzTZbLFsuzDbhCtL155o_vA@mail.gmail.com>
	<CAEc7osbxHuf3W66poK_7GpKOTDMExZvN-7KWXpfqe9Oo3CUTvw@mail.gmail.com>
	<CAMYG4G=L9sWMVWyUuT8fiNxE1VEw3vg4NxrihTEBsvqV3O7nHw@mail.gmail.com>
	<CAMYG4GksbOAY1-3L2v97A3B0mD-bKiB4dCOks7EOsL1UvaAa=w@mail.gmail.com>
	<CA+wRr2=jsm1hbdVWyL=TAH478DhkXrR-wY_Ue4y1NUBDXEAULg@mail.gmail.com>
Message-ID: <CAMYG4GmQ5Ke8u9c3dqqWmqT3apkWh2fTLd_-TSU0U8ZjcaoJ8w@mail.gmail.com>

On Mon, Aug 29, 2022 at 10:22 AM Lawrence Mitchell <wence at gmx.li> wrote:

> On Mon, 29 Aug 2022 at 14:55, Matthew Knepley <knepley at gmail.com> wrote:
> >
> > On Sun, Aug 28, 2022 at 7:17 PM Matthew Knepley <knepley at gmail.com>
> wrote:
> >>
> >> On Sun, Aug 28, 2022 at 5:36 PM Mike Michell <mi.mike1021 at gmail.com>
> wrote:
> >>>
> >>> Thank you for the reply.
> >>>
> >>> I think it can be more helpful for me if the attached sample code
> (DMInterpolation_Mod.tar) could be checked by you.
> >>
> >>
> >> Okay, you are right. I will run it tomorrow.
> >
> >
> > The problem was somewhat conceptual. When you are restricting state,
> meaning a function you want to use directly, rather than
> > residuals, meaning some average or integral, you want to preserve scale.
> This is what the 'vscale' vector is for. I added the
> > VecPointwiseMult to your code (attached) to do this. I also changed to a
> more general mehs creation. To get your run, use
>
> The right answer is to use DMCreateInjection
> (https://petsc.org/release/docs/manualpages/DM/DMCreateInjection.html)
> to move fine grid state to coarse grid state (rather than
> DMCreateInterpolation and then VecPointwiseMult, which doesn't work
> for most discretisations).
>
> Matt: the interface to DMCreateInjection is now correct (it returns a
> Mat, rather than a VecScatter) so DMPlexComputeInjectorFEM
> (
> https://petsc.org/release/docs/manualpages/DMPLEX/DMPlexComputeInjectorFEM.html
> )
> should be updated to take advantage.
>

I am not sure Injection is going to "do the right thing" when you move
between spaces. At least my memory of the implementation
was that it just did local interpolation, which is not right when injecting
P0 into P1.

  Thanks,

      Matt


> Lawrence
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220829/367e524d/attachment.html>

From wence at gmx.li  Mon Aug 29 14:12:55 2022
From: wence at gmx.li (Lawrence Mitchell)
Date: Mon, 29 Aug 2022 20:12:55 +0100
Subject: [petsc-users] DMPlex Interpolation
In-Reply-To: <CAMYG4GmQ5Ke8u9c3dqqWmqT3apkWh2fTLd_-TSU0U8ZjcaoJ8w@mail.gmail.com>
References: <CAEc7osYgsXxW9DN_6aHof3wdyqHw3Wv6nLp+svWsZF5c0zz93w@mail.gmail.com>
	<CAMYG4GmiXaNTG5sgjoH=-CdY21OkRaKZh5y5OdPcSiEMj-KfXg@mail.gmail.com>
	<CAEc7osb8HG7ZQQy3=6Jcq-J8jMB1vWXHo-7Y5=fc7A3QR9BQGA@mail.gmail.com>
	<CAMYG4GnfNRGa9yOeCBzfviSfsJ+FxAy73_aRo07VqyCoQETJ+g@mail.gmail.com>
	<CAEc7osad2ee=T6QH5=VHFRjJDsyJ4mO3SYnTF6_BpqkJN0Ppfw@mail.gmail.com>
	<CAMYG4Gkzjf=OXBQsErWCumMscGykzTZbLFsuzDbhCtL155o_vA@mail.gmail.com>
	<CAEc7osbxHuf3W66poK_7GpKOTDMExZvN-7KWXpfqe9Oo3CUTvw@mail.gmail.com>
	<CAMYG4G=L9sWMVWyUuT8fiNxE1VEw3vg4NxrihTEBsvqV3O7nHw@mail.gmail.com>
	<CAMYG4GksbOAY1-3L2v97A3B0mD-bKiB4dCOks7EOsL1UvaAa=w@mail.gmail.com>
	<CA+wRr2=jsm1hbdVWyL=TAH478DhkXrR-wY_Ue4y1NUBDXEAULg@mail.gmail.com>
	<CAMYG4GmQ5Ke8u9c3dqqWmqT3apkWh2fTLd_-TSU0U8ZjcaoJ8w@mail.gmail.com>
Message-ID: <CA+wRr2m+M3Xrm8CrKSu-BT6o2T2jTk2ZA8+j9-vG2Ca1i8K0sQ@mail.gmail.com>

>
> I am not sure Injection is going to "do the right thing" when you move between spaces. At least my memory of the implementation
> was that it just did local interpolation, which is not right when injecting P0 into P1.

Ah, that could be the case. For discontinuous things I think what you
want is the L2 projection of the fine function into the coarse space
(which on nested meshes you can compute locally), this is what I
implemented in Firedrake IIRC.

I agree that point interpolating won't work, and I guess in that case
some kind of scaling of a dual thing is approximating the L2
projection.

Lawrence

From mi.mike1021 at gmail.com  Mon Aug 29 14:16:34 2022
From: mi.mike1021 at gmail.com (Mike Michell)
Date: Mon, 29 Aug 2022 14:16:34 -0500
Subject: [petsc-users] DMPlex Interpolation
In-Reply-To: <CA+wRr2m+M3Xrm8CrKSu-BT6o2T2jTk2ZA8+j9-vG2Ca1i8K0sQ@mail.gmail.com>
References: <CAEc7osYgsXxW9DN_6aHof3wdyqHw3Wv6nLp+svWsZF5c0zz93w@mail.gmail.com>
	<CAMYG4GmiXaNTG5sgjoH=-CdY21OkRaKZh5y5OdPcSiEMj-KfXg@mail.gmail.com>
	<CAEc7osb8HG7ZQQy3=6Jcq-J8jMB1vWXHo-7Y5=fc7A3QR9BQGA@mail.gmail.com>
	<CAMYG4GnfNRGa9yOeCBzfviSfsJ+FxAy73_aRo07VqyCoQETJ+g@mail.gmail.com>
	<CAEc7osad2ee=T6QH5=VHFRjJDsyJ4mO3SYnTF6_BpqkJN0Ppfw@mail.gmail.com>
	<CAMYG4Gkzjf=OXBQsErWCumMscGykzTZbLFsuzDbhCtL155o_vA@mail.gmail.com>
	<CAEc7osbxHuf3W66poK_7GpKOTDMExZvN-7KWXpfqe9Oo3CUTvw@mail.gmail.com>
	<CAMYG4G=L9sWMVWyUuT8fiNxE1VEw3vg4NxrihTEBsvqV3O7nHw@mail.gmail.com>
	<CAMYG4GksbOAY1-3L2v97A3B0mD-bKiB4dCOks7EOsL1UvaAa=w@mail.gmail.com>
	<CA+wRr2=jsm1hbdVWyL=TAH478DhkXrR-wY_Ue4y1NUBDXEAULg@mail.gmail.com>
	<CAMYG4GmQ5Ke8u9c3dqqWmqT3apkWh2fTLd_-TSU0U8ZjcaoJ8w@mail.gmail.com>
	<CA+wRr2m+M3Xrm8CrKSu-BT6o2T2jTk2ZA8+j9-vG2Ca1i8K0sQ@mail.gmail.com>
Message-ID: <CAEc7osYY0fhhVW8Sd6K2Xh5yfYYYPvn8XeqGPM9KH8YG7HDLAg@mail.gmail.com>

Hi,
@Matt-This is exactly what I wanted to do. Thank you for the help. Mapping
for both directions (Q0->Q1 and Q1->Q0) works well.
@Lawrence-Thank you for the comment. So far, Matt's comment fits my
purpose.

Thanks,


> >
> > I am not sure Injection is going to "do the right thing" when you move
> between spaces. At least my memory of the implementation
> > was that it just did local interpolation, which is not right when
> injecting P0 into P1.
>
> Ah, that could be the case. For discontinuous things I think what you
> want is the L2 projection of the fine function into the coarse space
> (which on nested meshes you can compute locally), this is what I
> implemented in Firedrake IIRC.
>
> I agree that point interpolating won't work, and I guess in that case
> some kind of scaling of a dual thing is approximating the L2
> projection.
>
> Lawrence
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220829/c1bda675/attachment.html>

From yyang85 at alumni.stanford.edu  Tue Aug 30 01:03:47 2022
From: yyang85 at alumni.stanford.edu (Yuyun Yang)
Date: Tue, 30 Aug 2022 14:03:47 +0800
Subject: [petsc-users] Question on PetscViewerFileSetMode
Message-ID: <CAF_VkU7SJ07Oy8TOTxhh4bHQf4Yh_kyHYFfdV5+3zJEzbaOM7g@mail.gmail.com>

Hello team,

I recently updated by PETSc version and this error emerged:
"Cannot change mode to APPEND after setup"

I used to  use PetscViewerFileSetMode to set the mode as WRITE when I first
initiate the file, and for following time steps I set it to APPEND to open
the existing file and append data. Has the functionality changed?

Thanks!

Best regards,
Yuyun
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220830/07ad9219/attachment-0001.html>

From bsmith at petsc.dev  Tue Aug 30 09:21:58 2022
From: bsmith at petsc.dev (Barry Smith)
Date: Tue, 30 Aug 2022 10:21:58 -0400
Subject: [petsc-users] Question on PetscViewerFileSetMode
In-Reply-To: <CAF_VkU7SJ07Oy8TOTxhh4bHQf4Yh_kyHYFfdV5+3zJEzbaOM7g@mail.gmail.com>
References: <CAF_VkU7SJ07Oy8TOTxhh4bHQf4Yh_kyHYFfdV5+3zJEzbaOM7g@mail.gmail.com>
Message-ID: <143C41AE-6C96-4DB1-BFA6-8CC41102323C@petsc.dev>


   You should only use PetscViewerFileSetMode to APPEND  when you first create the viewer. You would only set it to APPEND if you destroy and recreate the viewer for each time-step.

   The append tells the viewer that when it opens the file it should move its pointer to the end of the file (so that the next writes append instead of replacing the old material). But when the file is already open and written to the pointer always points to the end of the file so there is no reason to append.

   I guess in the past we had less error checking so that your use did not trigger an error condition; but also in the past setting to append later did not break anything so it worked fine for you. Just remove the unneeded extra call to PetscViewerFileSetMode

  Barry


> On Aug 30, 2022, at 2:03 AM, Yuyun Yang <yyang85 at alumni.stanford.edu> wrote:
> 
> Hello team,
> 
> I recently updated by PETSc version and this error emerged:
> "Cannot change mode to APPEND after setup"
> 
> I used to  use PetscViewerFileSetMode to set the mode as WRITE when I first initiate the file, and for following time steps I set it to APPEND to open the existing file and append data. Has the functionality changed?
> 
> Thanks!
> 
> Best regards,
> Yuyun


From narnoldm at umich.edu  Tue Aug 30 17:37:13 2022
From: narnoldm at umich.edu (Nicholas Arnold-Medabalimi)
Date: Tue, 30 Aug 2022 18:37:13 -0400
Subject: [petsc-users] DMLabel Synchronize
Message-ID: <CAB1dcV6pLBRK8-ShycJe-PpSy_YtZeZEDDkzam-i-gCiu_rocw@mail.gmail.com>

Hi

Thanks for all the help. I'm looking to tag several cells based on metrics
occurring at run time. In addition to the metric-defined cells I also want
to label the surrounding cells. This is no problem as I can traverse the
DMPlex chart to get the neighboring cells. The issue only arises if an
identified cell is on a partition boundary. this leads to the neighbor
cell in the overlap halo being tagged. However, the root cell on the
adjacent partition is not tagged. Is there a built-in way to synchronize
the labels from the leafs to the roots?

Right now, I iterate through the root and leaf cells of each processor, but
it is somewhat wasteful as I don't see a way for the root cell partition to
know if a corresponding leaf cell label value without communicating it
first.

Thanks
Nicholas

-- 
Nicholas Arnold-Medabalimi

Ph.D. Candidate
Computational Aeroscience Lab
University of Michigan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220830/a0be23b1/attachment.html>

From knepley at gmail.com  Wed Aug 31 06:44:54 2022
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 31 Aug 2022 07:44:54 -0400
Subject: [petsc-users] DMLabel Synchronize
In-Reply-To: <CAB1dcV6pLBRK8-ShycJe-PpSy_YtZeZEDDkzam-i-gCiu_rocw@mail.gmail.com>
References: <CAB1dcV6pLBRK8-ShycJe-PpSy_YtZeZEDDkzam-i-gCiu_rocw@mail.gmail.com>
Message-ID: <CAMYG4GnoJ-X0=n6TRSrJ7FN0e1E2T5qvjKz8FydoYk2KHdytow@mail.gmail.com>

On Tue, Aug 30, 2022 at 6:37 PM Nicholas Arnold-Medabalimi <
narnoldm at umich.edu> wrote:

> Hi
>
> Thanks for all the help. I'm looking to tag several cells based on metrics
> occurring at run time. In addition to the metric-defined cells I also want
> to label the surrounding cells. This is no problem as I can traverse the
> DMPlex chart to get the neighboring cells. The issue only arises if an
> identified cell is on a partition boundary. this leads to the neighbor
> cell in the overlap halo being tagged. However, the root cell on the
> adjacent partition is not tagged. Is there a built-in way to synchronize
> the labels from the leafs to the roots?
>
> Right now, I iterate through the root and leaf cells of each processor,
> but it is somewhat wasteful as I don't see a way for the root cell
> partition to know if a corresponding leaf cell label value without
> communicating it first.
>

What I think you want is this:

  https://petsc.org/main/docs/manualpages/DMLabel/DMLabelPropagateBegin/

I use this to do the same kind of labeling I think. It is setup to allow
multiple rounds,
so that you can use the communicated labeling to continue on the new rank.

Will this work?

  Thanks,

     Matt


> Thanks
> Nicholas
>
> --
> Nicholas Arnold-Medabalimi
>
> Ph.D. Candidate
> Computational Aeroscience Lab
> University of Michigan
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220831/503fa8af/attachment.html>

From wangxq2020 at mail.ustc.edu.cn  Wed Aug 31 08:39:38 2022
From: wangxq2020 at mail.ustc.edu.cn (wangxq2020 at mail.ustc.edu.cn)
Date: Wed, 31 Aug 2022 21:39:38 +0800 (GMT+08:00)
Subject: [petsc-users] Problem about optimized version
Message-ID: <4852f88a.f69.182f420f9eb.Coremail.wangxq2020@mail.ustc.edu.cn>

Hi
I run the program on the debuged version of PETSC successfuly. Then I want using the optimized version. And configure by ./configure --with-debugging=no --with-packages-download-dir=~/packages_dir --with-mpich-dir="/public3/soft/mpich/mpich-3.4.2". Then I installed the Hypre package by ./configure --download-hypre=/public3/home/scg6368/packages_dir/hypre-2.24.0.tar.gz --with-mpich-dir="/public3/soft/mpich/mpich-3.4.2"
. The result as below


=============================================================================================
                         Configuring PETSc to compile on your system                         
=============================================================================================
=============================================================================================                                            ***** WARNING: You have a version of GNU make older than 4.0. It will work,                                                        but may not support all the parallel testing options. You can install the                                                          latest GNU make with your package manager, such as brew or macports, or use                                                        the --download-make option to get the latest GNU make *****                                                                  =============================================================================================                                      =============================================================================================                                            Running configure on HYPRE; this may take several minutes                                                                    =============================================================================================                                      =============================================================================================                                            Running make on HYPRE; this may take several minutes                                                                         =============================================================================================                                      =============================================================================================                                            Running make install on HYPRE; this may take several minutes                                                                 =============================================================================================                                      Compilers:                                                                                                                         
  C Compiler:         /public3/soft/mpich/mpich-3.4.2/bin/mpicc  -fPIC -Wall -Wwrite-strings -Wno-unknown-pragmas -Wno-lto-type-mismatch -fstack-protector -fvisibility=hidden -g3 -O0  
    Version: gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-44)
  C++ Compiler:         /public3/soft/mpich/mpich-3.4.2/bin/mpicxx  -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -Wno-lto-type-mismatch -fstack-protector -fvisibility=hidden -g -O0  -std=gnu++11 -fPIC 
    Version: g++ (GCC) 4.8.5 20150623 (Red Hat 4.8.5-44)
  Fortran Compiler:         /public3/soft/mpich/mpich-3.4.2/bin/mpif90  -fPIC -Wall -ffree-line-length-0 -Wno-lto-type-mismatch -Wno-unused-dummy-argument -g -O0  
    Version: GNU Fortran (GCC) 4.8.5 20150623 (Red Hat 4.8.5-44)
Linkers:
  Shared linker:   /public3/soft/mpich/mpich-3.4.2/bin/mpicc  -shared  -fPIC -Wall -Wwrite-strings -Wno-unknown-pragmas -Wno-lto-type-mismatch -fstack-protector -fvisibility=hidden -g3 -O0
  Dynamic linker:   /public3/soft/mpich/mpich-3.4.2/bin/mpicc  -shared  -fPIC -Wall -Wwrite-strings -Wno-unknown-pragmas -Wno-lto-type-mismatch -fstack-protector -fvisibility=hidden -g3 -O0
  Libraries linked against:   -lquadmath -lstdc++ -ldl 
BlasLapack:
  Library:  -llapack -lblas
  Unknown if this uses OpenMP (try export OMP_NUM_THREADS=<1-4> yourprogram -log_view) 
  uses 4 byte integers
MPI:
  Version:  3
  Includes: -I/public3/soft/mpich/mpich-3.4.2/include
  mpiexec: /public3/soft/mpich/mpich-3.4.2/bin/mpiexec
  Implementation: mpich3
  MPICH_NUMVERSION: 30402300
X:
  Library:  -lX11
pthread:
  Library:  -lpthread
cmake:
  Version:  2.8.12
  /usr/bin/cmake
hypre:
  Version:  2.24.0
  Includes: -I/public3/home/scg6368/petsc-3.17.2-opt/arch-linux-c-debug/include
  Library:  -Wl,-rpath,/public3/home/scg6368/petsc-3.17.2-opt/arch-linux-c-debug/lib -L/public3/home/scg6368/petsc-3.17.2-opt/arch-linux-c-debug/lib -lHYPRE
regex:
  Language used to compile PETSc: C
PETSc:
  PETSC_ARCH: arch-linux-c-debug
  PETSC_DIR: /public3/home/scg6368/petsc-3.17.2-opt
  Prefix: <inplace installation>
  Scalar type: real
  Precision: double
  Support for __float128
  Integer size: 4 bytes
  Single library: yes
  Shared libraries: yes
  Memory alignment from malloc(): 16 bytes
  Using GNU make: /usr/bin/gmake
xxx=========================================================================xxx
 Configure stage complete. Now build PETSc libraries with:
   make PETSC_DIR=/public3/home/scg6368/petsc-3.17.2-opt PETSC_ARCH=arch-linux-c-debug all
xxx=========================================================================xxx
[scg6368 at ln1:~/petsc-3.17.2-opt]$ make PETSC_DIR=/public3/home/scg6368/petsc-3.17.2-opt PETSC_ARCH=arch-linux-c-debug all  


But after I compiled the C file, run the program on the login node by mpiexec -n 2 ./test37. It appears that the program is processed by two cores serially, not in parallel. The result is 
 INITIALIZATION TO COMPUTE INITIAL PRESSURE TO CREATE FRACTURE WITH INITIAL VOLUME AT INJECTION RATE = 0.000000e+00 ......
      snesU converged in 0 iterations 2.
      snesV converged in 0 iterations 2.
      V min / max:     1.000000e+00 1.000000e+00
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
[0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[0]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind
[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple MacOS to find memory corruption errors
[0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
[0]PETSC ERROR: to get more information on the crash.
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Signal received
[0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.17.2, Jun 02, 2022 
[0]PETSC ERROR: ./test37 on a arch-linux-c-opt named ln1.para.bscc by scg6368 Wed Aug 31 21:27:26 2022
[0]PETSC ERROR: Configure options --with-debugging=no
[0]PETSC ERROR: #1 User provided function() at unknown file:0
[0]PETSC ERROR: Run with -malloc_debug to check if memory corruption is causing the crash.
Abort(59) on node 0 (rank 0 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
      snesU converged in 0 iterations 2.
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
      snesV converged in 0 iterations 2.
      V min / max:     1.000000e+00 1.000000e+00
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
[0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[0]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind
[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple MacOS to find memory corruption errors
[0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
[0]PETSC ERROR: to get more information on the crash.
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Signal received
[0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.17.2, Jun 02, 2022 


Then I run the program on the compute nodes by sbatch scipt. It appears that there is no hypre. The content of the script is as follows?the job error file is attached. I don't know where the problem is and would like your help. Thank you very much?


#!/bin/sh
#An example for MPI  job.
#SBATCH -J PFM_frac53
#SBATCH -o job_37-1-%j.log
#SBATCH -e job_37-1-%j.err
#SBATCH -N 1
#SBATCH -n 20
#SBATCH --partition=amd_512
echo Time is `date`
echo Directory is $PWD
echo This job runs on the following nodes:
echo$SLURM_JOB_NODELIST
echo This job has allocated $SLURM_JOB_CPUS_PER_NODE cpu cores.
#module load intelmpi/2018.update4
#module load intelmpi/2019.update5
#module load intelmpi/2020
#module load hpcx/2.9.0/hpcx-intel-2019.update5
module load mpi/openmpi/4.1.1-gcc7.3.0
module load mpich/3.4.2
#module load openmpi/4.0.2/gcc/4.8.5
#module load openmpi/3.0.5/gcc/9.2.0
#module load gcc/9.2.0
#MPIRUN=mpirun #Intel mpi and Open MPI
MPIRUN=mpiexec #MPICH
#MPIOPT="-env I_MPI_FABRICS shm:ofi" #Intel MPI 2018 ofa, 2019 and 2020 ofi
#MPIOPT="--mca btl self,sm,openib --mca btl_openib_cpc_include rdmacm --mca btl_openib_if_include mlx5_0:1" #Open MPI
MPIOPT="-iface ib0"#MPICH3
#MPIOPT=
#$MPIRUN $MPIOPT ./mpiring
$MPIRUN ./test37 -options_file options/test37.opts -p runtest37-1
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220831/0b7402ee/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: job_37-1-1233983.err
Type: application/octet-stream
Size: 8781 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220831/0b7402ee/attachment-0001.obj>

From bourdin at mcmaster.ca  Wed Aug 31 08:53:03 2022
From: bourdin at mcmaster.ca (Blaise Bourdin)
Date: Wed, 31 Aug 2022 13:53:03 +0000
Subject: [petsc-users] Problem about optimized version
In-Reply-To: <4852f88a.f69.182f420f9eb.Coremail.wangxq2020@mail.ustc.edu.cn>
References: <4852f88a.f69.182f420f9eb.Coremail.wangxq2020@mail.ustc.edu.cn>
Message-ID: <4390515B-85E1-4C1E-9180-4DC6196E6593@mcmaster.ca>

An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220831/f6cb4407/attachment-0001.html>

From knepley at gmail.com  Wed Aug 31 08:55:05 2022
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 31 Aug 2022 09:55:05 -0400
Subject: [petsc-users] Problem about optimized version
In-Reply-To: <4852f88a.f69.182f420f9eb.Coremail.wangxq2020@mail.ustc.edu.cn>
References: <4852f88a.f69.182f420f9eb.Coremail.wangxq2020@mail.ustc.edu.cn>
Message-ID: <CAMYG4Gn-G7NRWCJJsDj2K6J8rvR1wcEPP7-G8kOvDDL1kxaNzw@mail.gmail.com>

On Wed, Aug 31, 2022 at 9:40 AM wangxq2020--- via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Hi
> I run the program on the debuged version of PETSC successfuly. Then I want
> using the optimized version. And configure by ./configure
> --with-debugging=no --with-packages-download-dir=~/packages_dir
> --with-mpich-dir="/public3/soft/mpich/mpich-3.4.2". Then I installed the
> Hypre package by ./configure
> --download-hypre=/public3/home/scg6368/packages_dir/hypre-2.24.0.tar.gz
> --with-mpich-dir="/public3/soft/mpich/mpich-3.4.2"
> . The result as below
>
>
> =============================================================================================
>                          Configuring PETSc to compile on your system
>
>
> =============================================================================================
> =============================================================================================
>                                           ***** WARNING: You have a version
> of GNU make older than 4.0. It will work,
>                       but may not support all the parallel testing options.
> You can install the
>   latest GNU make with your package manager, such as brew or macports, or
> use                                                        the
> --download-make option to get the latest GNU make *****
>
> =============================================================================================
>
> =============================================================================================
>                                           Running configure on HYPRE; this
> may take several minutes
>
> =============================================================================================
>
> =============================================================================================
>                                           Running make on HYPRE; this may
> take several minutes
>
>  =============================================================================================
>
> =============================================================================================
>                                           Running make install on HYPRE;
> this may take several minutes
>
>  =============================================================================================
>                                     Compilers:
>
>
>   C Compiler:         /public3/soft/mpich/mpich-3.4.2/bin/mpicc  -fPIC
> -Wall -Wwrite-strings -Wno-unknown-pragmas -Wno-lto-type-mismatch
> -fstack-protector -fvisibility=hidden -g3 -O0
>     Version: gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-44)
>   C++ Compiler:         /public3/soft/mpich/mpich-3.4.2/bin/mpicxx  -Wall
> -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas
> -Wno-lto-type-mismatch -fstack-protector -fvisibility=hidden -g -O0
> -std=gnu++11 -fPIC
>     Version: g++ (GCC) 4.8.5 20150623 (Red Hat 4.8.5-44)
>   Fortran Compiler:         /public3/soft/mpich/mpich-3.4.2/bin/mpif90
> -fPIC -Wall -ffree-line-length-0 -Wno-lto-type-mismatch
> -Wno-unused-dummy-argument -g -O0
>     Version: GNU Fortran (GCC) 4.8.5 20150623 (Red Hat 4.8.5-44)
> Linkers:
>   Shared linker:   /public3/soft/mpich/mpich-3.4.2/bin/mpicc  -shared
> -fPIC -Wall -Wwrite-strings -Wno-unknown-pragmas -Wno-lto-type-mismatch
> -fstack-protector -fvisibility=hidden -g3 -O0
>   Dynamic linker:   /public3/soft/mpich/mpich-3.4.2/bin/mpicc  -shared
> -fPIC -Wall -Wwrite-strings -Wno-unknown-pragmas -Wno-lto-type-mismatch
> -fstack-protector -fvisibility=hidden -g3 -O0
>   Libraries linked against:   -lquadmath -lstdc++ -ldl
> BlasLapack:
>   Library:  -llapack -lblas
>   Unknown if this uses OpenMP (try export OMP_NUM_THREADS=<1-4>
> yourprogram -log_view)
>   uses 4 byte integers
> MPI:
>   Version:  3
>   Includes: -I/public3/soft/mpich/mpich-3.4.2/include
>   mpiexec: /public3/soft/mpich/mpich-3.4.2/bin/mpiexec
>   Implementation: mpich3
>   MPICH_NUMVERSION: 30402300
> X:
>   Library:  -lX11
> pthread:
>   Library:  -lpthread
> cmake:
>   Version:  2.8.12
>   /usr/bin/cmake
> hypre:
>   Version:  2.24.0
>   Includes:
> -I/public3/home/scg6368/petsc-3.17.2-opt/arch-linux-c-debug/include
>   Library:
> -Wl,-rpath,/public3/home/scg6368/petsc-3.17.2-opt/arch-linux-c-debug/lib
> -L/public3/home/scg6368/petsc-3.17.2-opt/arch-linux-c-debug/lib -lHYPRE
> regex:
>   Language used to compile PETSc: C
> PETSc:
>   PETSC_ARCH: arch-linux-c-debug
>   PETSC_DIR: /public3/home/scg6368/petsc-3.17.2-opt
>   Prefix: <inplace installation>
>   Scalar type: real
>   Precision: double
>   Support for __float128
>   Integer size: 4 bytes
>   Single library: yes
>   Shared libraries: yes
>   Memory alignment from malloc(): 16 bytes
>   Using GNU make: /usr/bin/gmake
>
> xxx=========================================================================xxx
>  Configure stage complete. Now build PETSc libraries with:
>    make PETSC_DIR=/public3/home/scg6368/petsc-3.17.2-opt
> PETSC_ARCH=arch-linux-c-debug all
>
> xxx=========================================================================xxx
> [scg6368 at ln1:~/petsc-3.17.2-opt]$ make
> PETSC_DIR=/public3/home/scg6368/petsc-3.17.2-opt
> PETSC_ARCH=arch-linux-c-debug all
>
> But after I compiled the C file, run the program on the login node by
> mpiexec -n 2 ./test37. It appears that the program is processed by two
> cores serially, not in parallel. The result is
>

This usually means that you have run with an mpiexec from a different MPI
installation than the one you compiled PETSc with.

  Thanks,

    Matt


>  INITIALIZATION TO COMPUTE INITIAL PRESSURE TO CREATE FRACTURE WITH
> INITIAL VOLUME AT INJECTION RATE = 0.000000e+00 ......
>       snesU converged in 0 iterations 2.
>       snesV converged in 0 iterations 2.
>       V min / max:     1.000000e+00 1.000000e+00
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
> probably memory access out of range
> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [0]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind
> [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple MacOS
> to find memory corruption errors
> [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and
> run
> [0]PETSC ERROR: to get more information on the crash.
> [0]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [0]PETSC ERROR: Signal received
> [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.17.2, Jun 02, 2022
> [0]PETSC ERROR: ./test37 on a arch-linux-c-opt named ln1.para.bscc by
> scg6368 Wed Aug 31 21:27:26 2022
> [0]PETSC ERROR: Configure options --with-debugging=no
> [0]PETSC ERROR: #1 User provided function() at unknown file:0
> [0]PETSC ERROR: Run with -malloc_debug to check if memory corruption is
> causing the crash.
> Abort(59) on node 0 (rank 0 in comm 0): application called
> MPI_Abort(MPI_COMM_WORLD, 59) - process 0
>       snesU converged in 0 iterations 2.
> --------------------------------------------------------------------------
> Primary job  terminated normally, but 1 process returned
> a non-zero exit code. Per user-direction, the job has been aborted.
> --------------------------------------------------------------------------
>       snesV converged in 0 iterations 2.
>       V min / max:     1.000000e+00 1.000000e+00
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
> probably memory access out of range
> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [0]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind
> [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple MacOS
> to find memory corruption errors
> [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and
> run
> [0]PETSC ERROR: to get more information on the crash.
> [0]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [0]PETSC ERROR: Signal received
> [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.17.2, Jun 02, 2022
>
> Then I run the program on the compute nodes by sbatch scipt. It appears
> that there is no hypre. The content of the script is as follows?the job
> error file is attached. I don't know where the problem is and would like
> your help. Thank you very much?
>
> #!/bin/sh
> #An example for MPI  job.
> #SBATCH -J PFM_frac53
> #SBATCH -o job_37-1-%j.log
> #SBATCH -e job_37-1-%j.err
> #SBATCH -N 1
> #SBATCH -n 20
> #SBATCH --partition=amd_512
> echo Time is `date`
> echo Directory is $PWD
> echo This job runs on the following nodes:
> echo $SLURM_JOB_NODELIST
> echo This job has allocated $SLURM_JOB_CPUS_PER_NODE cpu cores.
> #module load intelmpi/2018.update4
> #module load intelmpi/2019.update5
> #module load intelmpi/2020
> #module load hpcx/2.9.0/hpcx-intel-2019.update5
> module load mpi/openmpi/4.1.1-gcc7.3.0
> module load mpich/3.4.2
> #module load openmpi/4.0.2/gcc/4.8.5
> #module load openmpi/3.0.5/gcc/9.2.0
> #module load gcc/9.2.0
> #MPIRUN=mpirun #Intel mpi and Open MPI
> MPIRUN=mpiexec #MPICH
> #MPIOPT="-env I_MPI_FABRICS shm:ofi" #Intel MPI 2018 ofa, 2019 and 2020 ofi
> #MPIOPT="--mca btl self,sm,openib --mca btl_openib_cpc_include rdmacm
> --mca btl_openib_if_include mlx5_0:1" #Open MPI
> MPIOPT="-iface ib0" #MPICH3
> #MPIOPT=
> #$MPIRUN $MPIOPT ./mpiring
> $MPIRUN ./test37 -options_file options/test37.opts -p runtest37-1
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220831/87d64a6e/attachment-0001.html>

From bsmith at petsc.dev  Wed Aug 31 09:00:40 2022
From: bsmith at petsc.dev (Barry Smith)
Date: Wed, 31 Aug 2022 10:00:40 -0400
Subject: [petsc-users] Problem about optimized version
In-Reply-To: <4852f88a.f69.182f420f9eb.Coremail.wangxq2020@mail.ustc.edu.cn>
References: <4852f88a.f69.182f420f9eb.Coremail.wangxq2020@mail.ustc.edu.cn>
Message-ID: <DE2D9FE9-5C31-465A-9467-2EC9967147BB@petsc.dev>


  Please send configure.log and make.log to petsc-maint at mcs.anl.gov <mailto:petsc-maint at mcs.anl.gov> (too large for petsc-users)

  Did you use the exact same configure options for the optimized and debugger versions of PETSc except for the option --with-debugging=no ?

  The "is processed by two cores serially, not in parallel", usually comes from using a different mpiexec than the one associated with the MPI library that you linked against.

Verify that you use the mpiexec in /public3/soft/mpich/mpich-3.4.2/bin also 
verify that the MPI libraries linked codes can actually be run on the front-end; it may be that such executables can only be run on compute nodes in your system.
cd src/snes/tutorials/; make ex19 and try running that code with mpiexec

Barry



> On Aug 31, 2022, at 9:39 AM, wangxq2020--- via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
> Hi
> 	I run the program on the debuged version of PETSC successfuly. Then I want using the optimized version. And configure by ./configure --with-debugging=no --with-packages-download-dir=~/packages_dir --with-mpich-dir="/public3/soft/mpich/mpich-3.4.2". Then I installed the Hypre package by ./configure --download-hypre=/public3/home/scg6368/packages_dir/hypre-2.24.0.tar.gz --with-mpich-dir="/public3/soft/mpich/mpich-3.4.2"
> . The result as below
> 
> =============================================================================================
>                          Configuring PETSc to compile on your system                         
> =============================================================================================
> =============================================================================================                                            ***** WARNING: You have a version of GNU make older than 4.0. It will work,                                                        but may not support all the parallel testing options. You can install the                                                          latest GNU make with your package manager, such as brew or macports, or use                                                        the --download-make option to get the latest GNU make *****                                                                  =============================================================================================                                      =============================================================================================                                            Running configure on HYPRE; this may take several minutes                                                                    =============================================================================================                                      =============================================================================================                                            Running make on HYPRE; this may take several minutes                                                                         =============================================================================================                                      =============================================================================================                                            Running make install on HYPRE; this may take several minutes                                                                 =============================================================================================                                      Compilers:                                                                                                                         
>   C Compiler:         /public3/soft/mpich/mpich-3.4.2/bin/mpicc  -fPIC -Wall -Wwrite-strings -Wno-unknown-pragmas -Wno-lto-type-mismatch -fstack-protector -fvisibility=hidden -g3 -O0  
>     Version: gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-44)
>   C++ Compiler:         /public3/soft/mpich/mpich-3.4.2/bin/mpicxx  -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -Wno-lto-type-mismatch -fstack-protector -fvisibility=hidden -g -O0  -std=gnu++11 -fPIC 
>     Version: g++ (GCC) 4.8.5 20150623 (Red Hat 4.8.5-44)
>   Fortran Compiler:         /public3/soft/mpich/mpich-3.4.2/bin/mpif90  -fPIC -Wall -ffree-line-length-0 -Wno-lto-type-mismatch -Wno-unused-dummy-argument -g -O0  
>     Version: GNU Fortran (GCC) 4.8.5 20150623 (Red Hat 4.8.5-44)
> Linkers:
>   Shared linker:   /public3/soft/mpich/mpich-3.4.2/bin/mpicc  -shared  -fPIC -Wall -Wwrite-strings -Wno-unknown-pragmas -Wno-lto-type-mismatch -fstack-protector -fvisibility=hidden -g3 -O0
>   Dynamic linker:   /public3/soft/mpich/mpich-3.4.2/bin/mpicc  -shared  -fPIC -Wall -Wwrite-strings -Wno-unknown-pragmas -Wno-lto-type-mismatch -fstack-protector -fvisibility=hidden -g3 -O0
>   Libraries linked against:   -lquadmath -lstdc++ -ldl 
> BlasLapack:
>   Library:  -llapack -lblas
>   Unknown if this uses OpenMP (try export OMP_NUM_THREADS=<1-4> yourprogram -log_view) 
>   uses 4 byte integers
> MPI:
>   Version:  3
>   Includes: -I/public3/soft/mpich/mpich-3.4.2/include
>   mpiexec: /public3/soft/mpich/mpich-3.4.2/bin/mpiexec
>   Implementation: mpich3
>   MPICH_NUMVERSION: 30402300
> X:
>   Library:  -lX11
> pthread:
>   Library:  -lpthread
> cmake:
>   Version:  2.8.12
>   /usr/bin/cmake
> hypre:
>   Version:  2.24.0
>   Includes: -I/public3/home/scg6368/petsc-3.17.2-opt/arch-linux-c-debug/include
>   Library:  -Wl,-rpath,/public3/home/scg6368/petsc-3.17.2-opt/arch-linux-c-debug/lib -L/public3/home/scg6368/petsc-3.17.2-opt/arch-linux-c-debug/lib -lHYPRE
> regex:
>   Language used to compile PETSc: C
> PETSc:
>   PETSC_ARCH: arch-linux-c-debug
>   PETSC_DIR: /public3/home/scg6368/petsc-3.17.2-opt
>   Prefix: <inplace installation>
>   Scalar type: real
>   Precision: double
>   Support for __float128
>   Integer size: 4 bytes
>   Single library: yes
>   Shared libraries: yes
>   Memory alignment from malloc(): 16 bytes
>   Using GNU make: /usr/bin/gmake
> xxx=========================================================================xxx
>  Configure stage complete. Now build PETSc libraries with:
>    make PETSC_DIR=/public3/home/scg6368/petsc-3.17.2-opt PETSC_ARCH=arch-linux-c-debug all
> xxx=========================================================================xxx
> [scg6368 at ln1:~/petsc-3.17.2-opt]$ make PETSC_DIR=/public3/home/scg6368/petsc-3.17.2-opt PETSC_ARCH=arch-linux-c-debug all  
> 
> But after I compiled the C file, run the program on the login node by mpiexec -n 2 ./test37. It appears that the program is processed by two cores serially, not in parallel. The result is 
>  INITIALIZATION TO COMPUTE INITIAL PRESSURE TO CREATE FRACTURE WITH INITIAL VOLUME AT INJECTION RATE = 0.000000e+00 ......
>       snesU converged in 0 iterations 2.
>       snesV converged in 0 iterations 2.
>       V min / max:     1.000000e+00 1.000000e+00
> [0]PETSC ERROR: ------------------------------------------------------------------------
> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [0]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind
> [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple MacOS to find memory corruption errors
> [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
> [0]PETSC ERROR: to get more information on the crash.
> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [0]PETSC ERROR: Signal received
> [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.17.2, Jun 02, 2022 
> [0]PETSC ERROR: ./test37 on a arch-linux-c-opt named ln1.para.bscc by scg6368 Wed Aug 31 21:27:26 2022
> [0]PETSC ERROR: Configure options --with-debugging=no
> [0]PETSC ERROR: #1 User provided function() at unknown file:0
> [0]PETSC ERROR: Run with -malloc_debug to check if memory corruption is causing the crash.
> Abort(59) on node 0 (rank 0 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
>       snesU converged in 0 iterations 2.
> --------------------------------------------------------------------------
> Primary job  terminated normally, but 1 process returned
> a non-zero exit code. Per user-direction, the job has been aborted.
> --------------------------------------------------------------------------
>       snesV converged in 0 iterations 2.
>       V min / max:     1.000000e+00 1.000000e+00
> [0]PETSC ERROR: ------------------------------------------------------------------------
> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [0]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind
> [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple MacOS to find memory corruption errors
> [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
> [0]PETSC ERROR: to get more information on the crash.
> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [0]PETSC ERROR: Signal received
> [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.17.2, Jun 02, 2022 
> 
> Then I run the program on the compute nodes by sbatch scipt. It appears that there is no hypre. The content of the script is as follows?the job error file is attached. I don't know where the problem is and would like your help. Thank you very much?
> 
> #!/bin/sh
> #An example for MPI  job.
> #SBATCH -J PFM_frac53
> #SBATCH -o job_37-1-%j.log
> #SBATCH -e job_37-1-%j.err
> #SBATCH -N 1
> #SBATCH -n 20
> #SBATCH --partition=amd_512
> echo Time is `date`
> echo Directory is $PWD
> echo This job runs on the following nodes:
> echo $SLURM_JOB_NODELIST
> echo This job has allocated $SLURM_JOB_CPUS_PER_NODE cpu cores.
> #module load intelmpi/2018.update4
> #module load intelmpi/2019.update5
> #module load intelmpi/2020
> #module load hpcx/2.9.0/hpcx-intel-2019.update5
> module load mpi/openmpi/4.1.1-gcc7.3.0
> module load mpich/3.4.2
> #module load openmpi/4.0.2/gcc/4.8.5
> #module load openmpi/3.0.5/gcc/9.2.0
> #module load gcc/9.2.0
> #MPIRUN=mpirun #Intel mpi and Open MPI
> MPIRUN=mpiexec #MPICH
> #MPIOPT="-env I_MPI_FABRICS shm:ofi" #Intel MPI 2018 ofa, 2019 and 2020 ofi
> #MPIOPT="--mca btl self,sm,openib --mca btl_openib_cpc_include rdmacm --mca btl_openib_if_include mlx5_0:1" #Open MPI
> MPIOPT="-iface ib0" #MPICH3
> #MPIOPT=
> #$MPIRUN $MPIOPT ./mpiring
> $MPIRUN ./test37 -options_file options/test37.opts -p runtest37-1
> 
> <job_37-1-1233983.err>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220831/7ce848fb/attachment-0001.html>

From narnoldm at umich.edu  Wed Aug 31 20:29:45 2022
From: narnoldm at umich.edu (Nicholas Arnold-Medabalimi)
Date: Wed, 31 Aug 2022 21:29:45 -0400
Subject: [petsc-users] DMLabel Synchronize
In-Reply-To: <CAMYG4GnoJ-X0=n6TRSrJ7FN0e1E2T5qvjKz8FydoYk2KHdytow@mail.gmail.com>
References: <CAB1dcV6pLBRK8-ShycJe-PpSy_YtZeZEDDkzam-i-gCiu_rocw@mail.gmail.com>
	<CAMYG4GnoJ-X0=n6TRSrJ7FN0e1E2T5qvjKz8FydoYk2KHdytow@mail.gmail.com>
Message-ID: <CAB1dcV5OhLqW9soBcH=kExKgDSYyyj=ZgQ6xiE+3=TxpYE_5Ng@mail.gmail.com>

Hi

Thanks, this looks like what I need, but I don't seem to have it in my
release (3.17.4 from
https://ftp.mcs.anl.gov/pub/petsc/release-snapshots/petsc-3.17.4.tar.gz). I
see it in the current repository, though. Is the 3.17.4 tarball just
missing it, or will it be added in the next release?

Thanks
Nicholas

On Wed, Aug 31, 2022 at 7:45 AM Matthew Knepley <knepley at gmail.com> wrote:

> On Tue, Aug 30, 2022 at 6:37 PM Nicholas Arnold-Medabalimi <
> narnoldm at umich.edu> wrote:
>
>> Hi
>>
>> Thanks for all the help. I'm looking to tag several cells based on
>> metrics occurring at run time. In addition to the metric-defined cells I
>> also want to label the surrounding cells. This is no problem as I can
>> traverse the DMPlex chart to get the neighboring cells. The issue only
>> arises if an identified cell is on a partition boundary. this leads to the
>> neighbor cell in the overlap halo being tagged. However, the root cell on
>> the adjacent partition is not tagged. Is there a built-in way to
>> synchronize the labels from the leafs to the roots?
>>
>> Right now, I iterate through the root and leaf cells of each processor,
>> but it is somewhat wasteful as I don't see a way for the root cell
>> partition to know if a corresponding leaf cell label value without
>> communicating it first.
>>
>
> What I think you want is this:
>
>   https://petsc.org/main/docs/manualpages/DMLabel/DMLabelPropagateBegin/
>
> I use this to do the same kind of labeling I think. It is setup to allow
> multiple rounds,
> so that you can use the communicated labeling to continue on the new rank.
>
> Will this work?
>
>   Thanks,
>
>      Matt
>
>
>> Thanks
>> Nicholas
>>
>> --
>> Nicholas Arnold-Medabalimi
>>
>> Ph.D. Candidate
>> Computational Aeroscience Lab
>> University of Michigan
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>


-- 
Nicholas Arnold-Medabalimi

Ph.D. Candidate
Computational Aeroscience Lab
University of Michigan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220831/eb712007/attachment.html>