[petsc-users] 2D Partitioning matrix-shell and KSP

[Sreeram R Venkat]

I have a custom implementation of a matrix-vector product that inherently
relies on a 2D processor partitioning of the matrix. That is, if the matrix
looks like:

A B C D
E F G H
I  J K L

in block form, we use 12 processors, each having one block. The input
vector is partitioned across each row, and the output vector is partitioned
across each column.

Each processor has 3 communicators: the WORLD_COMM, a ROW_COMM, and a
COL_COMM. The ROW/COL communicators are used to do reductions over
rows/columns of processors.

With this setup, I am a bit confused about how to set up the matrix shell.
The "MatCreateShell" function only accepts one communicator. If I give the
WORLD_COMM, the local/global sizes won't match since PETSc will try to
multiply local_size * total_processors instead of local_size *
processors_per_row (or col). I have gotten around this temporarily by
giving ROW_COMM here instead. What I think happens is a different MatShell
is created on each row, but when computing the matvec, they all work
together.

However, if I try to use KSP (CG) with this setup (giving ROW_COMM as the
communicator), the process hangs. I believe this is due to the partitioning
of the input/output vectors. The matvec itself is fine, but the inner
products and other steps of CG fail. In fact, if I restrict to the case
where I only have one row of processors, I am able to successfully use KSP.

Is there a way to use KSP with this 2D partitioning setup when there are
multiple rows of processors? I'd also prefer to work with one global
MatShell object instead of this one object per row thing that I'm doing
right now.

Thanks for your help,
Sreeram
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20230919/9863008f/attachment.html>