<html><head><meta http-equiv="content-type" content="text/html; charset=us-ascii"></head><body style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;"><div><br></div> ( a ) ( A B C D ) ( w )<div> ( b ) = ( E F G H ) ( x )<br><div> ( c ) ( I J K L ) ( y )</div><div> ( z )</div><div><br></div><div>I have no idea what "<span style="color: rgb(34, 34, 34);">The input vector is partitioned across each row, and the output vector is partitioned across each column" means.</span></div><div><span style="color: rgb(34, 34, 34);"><br></span></div><div><span style="color: rgb(34, 34, 34);">Anyways the shell matrix needs to live on MPI_COMM_WORLD, as do both the (a,b,c) and (w,x,y,z) vector. </span></div><div><span style="color: rgb(34, 34, 34);"><br></span></div><div><span style="color: rgb(34, 34, 34);">Now how many MPI ranks do you want to do the compution on? 12? </span></div><div><span style="color: rgb(34, 34, 34);">Do you want one matrix A .. Z on each rank?</span></div><div><span style="color: rgb(34, 34, 34);"><br></span></div><div><span style="color: rgb(34, 34, 34);">Do you want the (a,b,c) vector spread over all ranks? What about the </span><span style="caret-color: rgb(34, 34, 34); color: rgb(34, 34, 34);">(w,x,y,z) vector?</span></div><div><span style="caret-color: rgb(34, 34, 34); color: rgb(34, 34, 34);"><br></span></div><div><span style="caret-color: rgb(34, 34, 34); color: rgb(34, 34, 34);"> Barry</span></div><div><span style="caret-color: rgb(34, 34, 34); color: rgb(34, 34, 34);"><br></span></div><div><br></div><div><br><blockquote type="cite"><div>On Sep 19, 2023, at 4:42 PM, Sreeram R Venkat <srvenkat@utexas.edu> wrote:</div><br class="Apple-interchange-newline"><div><div dir="ltr"><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div style="color:rgb(34,34,34)">I have a custom implementation of a matrix-vector product that inherently relies on a 2D processor partitioning of the matrix. That is, if the matrix looks like:</div><div style="color:rgb(34,34,34)"><br></div><div style="color:rgb(34,34,34)">A B C D</div><div style="color:rgb(34,34,34)">E F G H</div><div style="color:rgb(34,34,34)">I J K L</div></div></div></div></div></blockquote><blockquote type="cite"><div><div dir="ltr"><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div style="color:rgb(34,34,34)">in block form, we use 12 processors, each having one block. The input vector is partitioned across each row, and the output vector is partitioned across each column.</div><div style="color:rgb(34,34,34)"><br></div><div style="color:rgb(34,34,34)">Each processor has 3 communicators: the WORLD_COMM, a ROW_COMM, and a COL_COMM. The ROW/COL communicators are used to do reductions over rows/columns of processors.</div><div style="color:rgb(34,34,34)"><br></div><div style="color:rgb(34,34,34)">With this setup, I am a bit confused about how to set up the matrix shell. The "MatCreateShell" function only accepts one communicator. If I give the WORLD_COMM, the local/global sizes won't match since PETSc will try to multiply local_size * total_processors instead of local_size * processors_per_row (or col). I have gotten around this temporarily by giving ROW_COMM here instead. What I think happens is a different MatShell is created on each row, but when computing the matvec, they all work together. </div><div style="color:rgb(34,34,34)"><br></div><div style="color:rgb(34,34,34)">However, if I try to use KSP (CG) with this setup (giving ROW_COMM as the communicator), the process hangs. I believe this is due to the partitioning of the input/output vectors. The matvec itself is fine, but the inner products and other steps of CG fail. In fact, if I restrict to the case where I only have one row of processors, I am able to successfully use KSP. </div><div style="color:rgb(34,34,34)"><br></div><div style="color:rgb(34,34,34)">Is there a way to use KSP with this 2D partitioning setup when there are multiple rows of processors? I'd also prefer to work with one global MatShell object instead of this one object per row thing that I'm doing right now.</div><div style="color:rgb(34,34,34)"><br></div><div style="color:rgb(34,34,34)">Thanks for your help,</div><div style="color:rgb(34,34,34)">Sreeram</div></div></div></div>
</div></blockquote></div><br></div></body></html>