On Mon, Mar 7, 2011 at 11:32 AM, Barry Smith <span dir="ltr"><<a href="mailto:bsmith@mcs.anl.gov">bsmith@mcs.anl.gov</a>></span> wrote:<br><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div><div></div><div class="h5"><br>
On Mar 6, 2011, at 11:03 PM, Jack Poulson wrote:<br>
<br>
> Sorry to hit the list so quickly again, but the PCShellSetApply documentation doesn't say anything about how the 'xin' and 'xout' vectors will be distributed, or if it's up to the user to determine it.<br>
><br>
> Given that a distributed Mat implies a right and left vector distribution, I would assume that the 'xin' should be distributed like the 'left' vector distribution for a left preconditioner.<br>
><br>
> Is it up to the application routine to determine the 'xout' vector distribution? If so, is there a performance penalty for making it something other than the 'right' vector distribution implied by the matrix registered with the KSP? If not, what distribution does it need to be?<br>
<br>
</div></div> The PCShell doesn't determine the vector layouts at all. That is determined by the KSP that is using the PC. So basically your PC should handle any distribution; if your PCShell can only handle certain distributions then you need to make sure that the Vecs and Mats you provide to KSP or SNES (or TS) match what the PCShell needs.<br>
<br>
Barry<br>
<br></blockquote><div><br></div><div>It turns out that I had simply misunderstood the way PETSc was distributing matrices under the hood. I was assuming that it was best to only distribute the rows of the matrix using something along the lines of MatSetSizes( A, N/nproc, N, N, N ), but Jed informed me that the local size arguments for MatSetSizes only constrain the left and right vector distributions implied by the matrix distribution (through rmap and cmap, respectively). Also, I now know that KSP expects the right and left vector distributions to be the same for square matrices, so I should instead create my matrix along the lines of MatSetSizes( A, N/nproc, N/nproc, N, N ), and then both the in and out vectors for the PCShell application routine should be distributed in the same manner.</div>
<div><br></div><div>Thank you both for clearing up my misconceptions.</div><div><br></div><div>Jack</div></div><br>