[petsc-users] 2D Partitioning matrix-shell and KSP

[Barry Smith]

 (  a )       (  A  B  C  D ) (   w )
 (  b )   =  (  E  F  G H  ) (  x )
 (  c )        ( I    J   K L  )  ( y  )
                                       ( z  )

I have no idea what "The input vector is partitioned across each row, and the output vector is partitioned across each column" means.

Anyways the shell matrix needs to live on MPI_COMM_WORLD, as do both the (a,b,c) and (w,x,y,z) vector. 

Now how many MPI ranks do you want to do the compution on? 12?  
Do you want one matrix A .. Z on each rank?

Do you want the (a,b,c) vector spread over all ranks? What about the (w,x,y,z) vector?

  Barry



> On Sep 19, 2023, at 4:42 PM, Sreeram R Venkat <srvenkat at utexas.edu> wrote:
> 
> I have a custom implementation of a matrix-vector product that inherently relies on a 2D processor partitioning of the matrix. That is, if the matrix looks like:
> 
> A B C D
> E F G H
> I  J K L
> in block form, we use 12 processors, each having one block. The input vector is partitioned across each row, and the output vector is partitioned across each column.
> 
> Each processor has 3 communicators: the WORLD_COMM, a ROW_COMM, and a COL_COMM. The ROW/COL communicators are used to do reductions over rows/columns of processors.
> 
> With this setup, I am a bit confused about how to set up the matrix shell. The "MatCreateShell" function only accepts one communicator. If I give the WORLD_COMM, the local/global sizes won't match since PETSc will try to multiply local_size * total_processors instead of local_size * processors_per_row (or col). I have gotten around this temporarily by giving ROW_COMM here instead. What I think happens is a different MatShell is created on each row, but when computing the matvec, they all work together. 
> 
> However, if I try to use KSP (CG) with this setup (giving ROW_COMM as the communicator), the process hangs. I believe this is due to the partitioning of the input/output vectors. The matvec itself is fine, but the inner products and other steps of CG fail. In fact, if I restrict to the case where I only have one row of processors, I am able to successfully use KSP. 
> 
> Is there a way to use KSP with this 2D partitioning setup when there are multiple rows of processors? I'd also prefer to work with one global MatShell object instead of this one object per row thing that I'm doing right now.
> 
> Thanks for your help,
> Sreeram

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20230919/135740bf/attachment-0001.html>