<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On Sep 15, 2020, at 12:32 AM, Pierre Jolivet <<a href="mailto:pierre.jolivet@enseeiht.fr" class="">pierre.jolivet@enseeiht.fr</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><meta http-equiv="Content-Type" content="text/html; charset=utf-8" class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><br class=""><div class=""><br class=""><blockquote type="cite" class=""><div class="">On 15 Sep 2020, at 2:21 AM, Abhyankar, Shrirang G <<a href="mailto:shrirang.abhyankar@pnnl.gov" class="">shrirang.abhyankar@pnnl.gov</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class="WordSection1" style="page: WordSection1; caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt; font-family: "Times New Roman", serif;" class="">Hello Pierre,<o:p class=""></o:p></span></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt; font-family: "Times New Roman", serif;" class=""> PDIPM works in parallel so you can have distributed Hessian, Jacobians, constraints, variables, gradients in any layout you want. If you are using a DM then you can have it generate the Hessian.<span class="Apple-converted-space"> </span></span></div></div></div></blockquote><div class=""><br class=""></div><div class="">Could you please show an example where this is the case?</div><div class="">pdipm->x, which I’m assuming is a working vector, is both used as input for Hessian and Jacobian functions, e.g., <a href="https://gitlab.com/petsc/petsc/-/blob/master/src/tao/constrained/impls/ipm/pdipm.c#L369" class="">https://gitlab.com/petsc/petsc/-/blob/master/src/tao/constrained/impls/ipm/pdipm.c#L369</a> (Hessian) + <a href="https://gitlab.com/petsc/petsc/-/blob/master/src/tao/constrained/impls/ipm/pdipm.c#L473" class="">https://gitlab.com/petsc/petsc/-/blob/master/src/tao/constrained/impls/ipm/pdipm.c#L473</a> (Jacobian)</div><div class="">I thus doubt that it is possible to have different layouts?</div><div class="">In practice, I end up with the following error when I try this (2 processes, distributed Hessian with centralized Jacobian):</div><div class=""><div class="">[1]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------</div><div class="">[1]PETSC ERROR: Nonconforming object sizes</div><div class="">[1]PETSC ERROR: Vector wrong size 14172 for scatter 0 (scatter reverse and vector to != ctx from size)</div><div class="">[1]PETSC ERROR: #1 VecScatterBegin() line 96 in /Users/jolivet/Documents/repositories/petsc/src/vec/vscat/interface/vscatfce.c</div><div class="">[1]PETSC ERROR: #2 MatMultTransposeAdd_MPIAIJ() line 1223 in /Users/jolivet/Documents/repositories/petsc/src/mat/impls/aij/mpi/mpiaij.c</div><div class="">[1]PETSC ERROR: #3 MatMultTransposeAdd() line 2648 in /Users/jolivet/Documents/repositories/petsc/src/mat/interface/matrix.c</div><div class="">[0]PETSC ERROR: Nonconforming object sizes</div><div class="">[0]PETSC ERROR: Vector wrong size 13790 for scatter 27962 (scatter reverse and vector to != ctx from size)</div><div class="">[1]PETSC ERROR: #4 TaoSNESFunction_PDIPM() line 510 in /Users/jolivet/Documents/repositories/petsc/src/tao/constrained/impls/ipm/pdipm.c</div><div class="">[0]PETSC ERROR: #5 TaoSolve_PDIPM() line 712 in /Users/jolivet/Documents/repositories/petsc/src/tao/constrained/impls/ipm/pdipm.c</div><div class="">[1]PETSC ERROR: #6 TaoSolve() line 222 in /Users/jolivet/Documents/repositories/petsc/src/tao/interface/taosolver.c</div><div class="">[0]PETSC ERROR: #1 VecScatterBegin() line 96 in /Users/jolivet/Documents/repositories/petsc/src/vec/vscat/interface/vscatfce.c</div><div class="">[0]PETSC ERROR: #2 MatMultTransposeAdd_MPIAIJ() line 1223 in /Users/jolivet/Documents/repositories/petsc/src/mat/impls/aij/mpi/mpiaij.c</div><div class="">[0]PETSC ERROR: #3 MatMultTransposeAdd() line 2648 in /Users/jolivet/Documents/repositories/petsc/src/mat/interface/matrix.c</div><div class="">[0]PETSC ERROR: #4 TaoSNESFunction_PDIPM() line 510 in /Users/jolivet/Documents/repositories/petsc/src/tao/constrained/impls/ipm/pdipm.c</div><div class="">[0]PETSC ERROR: #5 TaoSolve_PDIPM() line 712 in /Users/jolivet/Documents/repositories/petsc/src/tao/constrained/impls/ipm/pdipm.c</div><div class="">[0]PETSC ERROR: #6 TaoSolve() line 222 in /Users/jolivet/Documents/repositories/petsc/src/tao/interface/taosolver.c</div></div><div class=""><br class=""></div><div class="">I think this can be reproduced by ex1.c by just distributing the Hessian instead of having it centralized on rank 0.</div><br class=""><blockquote type="cite" class=""><div class=""><div class="WordSection1" style="page: WordSection1; caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt; font-family: "Times New Roman", serif;" class="">Ideally, you want to have the layout below to minimize movement of matrix/vector elements across ranks.<o:p class=""></o:p></span></div><ul type="disc" style="margin-bottom: 0in; margin-top: 0in;" class=""><li class="MsoListParagraph" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 12pt; font-family: "Times New Roman", serif;" class="">The layout of vectors x, bounds on x, and gradient is same.<o:p class=""></o:p></span></li><li class="MsoListParagraph" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 12pt; font-family: "Times New Roman", serif;" class="">The row layout of the equality/inequality Jacobian is same as the equality/inequality constraint vector layout.<o:p class=""></o:p></span></li><li class="MsoListParagraph" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 12pt; font-family: "Times New Roman", serif;" class="">The column layout of the equality/inequality Jacobian is same as that for x.</span></li></ul></div></div></blockquote><div class=""><br class=""></div><div class="">Very naive question, but given that I have a single constraint, how do I split a 1 x N matrix column-wise? I thought it was not possible.</div></div></div></div></blockquote><div><br class=""></div> For the standard PETSc matrix implementations dense, AIJ, BAIJ, SBAIJ, that partition matrices by row it is not possible to have different column entires stored on different ranks. </div><div><br class=""></div><div> The two PetscLayouts associated with a matrix need to match the layouts of the vector input and output for a matrix vector product. So for a matrix with a single row the output vector needs to have a layout of 1 entry on the first rank and 0 entries on the rest. The input vector can have many layouts, one layout would be for all the entries to be on the first rank, but one could also distribute the vector entries across the ranks. But for the standard PETSc matrix implementations all the matrix entries would be on the first rank and all input vector entries would be communicated to the first process before doing the local matrix-vector product. </div><div><br class=""></div><div> So to store the single row matrix entries across the ranks one would need a custom storage format and communication pattern. For each let's assume the input vector to the matrix-vector product has a conventional layout of [0:rend0-1,rend0-1:rend1-1,...] and let's store the matrix entries for this layout by rank so that the local multiplies can be done without any communication, hence rank 0 will have matrix columns 0,rend0-1, rank 1 will have rend0-1:rend1-1 etc. </div><div><br class=""></div><div> Let Mat_MPIColumn (for lack of a better name) be our custom data structure.</div><div><br class=""></div><div> typedef struct {</div><div> Mat seq;</div><div> Vec work;</div><div> } Mat_MPIColumn;</div><div><br class=""></div><div> It would have a sequential matrix (AIJ or dense, or others) with 1 row and rend0-1:rend1-1 columns, call it seq. The local matrix-vector product could be calculated by MatMult(col->seq,inputvec,col->work); then a MPI_Reduce() could be used to accumulate the results from the col->work into the output vector on rank 0. </div><div><br class=""></div><div> Note this design would seem to work for not just a row matrix on process 0 but for having multiply rows on process 0 if needed. </div><div><br class=""></div><div> The custom matrix would also need a custom MatSetValues() which seems straightforward, by checking the column of the input entry it would determine what rank the entry belonged to and on the correct rank the entry in the local seq matrix would be determined by subtracting rend(rank -1)-1 from the global column.</div><div><br class=""></div><div> So in conclusion what is required is a relatively straight-forward new matrix class that is intended for "short-fat" matrices that are stored across the ranks but whose resulting matrix-vector product is on rank 0. I haven't thought about things like MatMultTranspose() hopefully they are not too convoluted.</div><div><br class=""></div><div> Questions, comments?</div><div><br class=""></div><div> Barry</div><div><br class=""></div><div><br class=""></div><div><br class=""></div><div><blockquote type="cite" class=""><div class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class=""><div class=""><br class=""></div><div class="">Thanks,</div><div class="">Pierre</div><br class=""><blockquote type="cite" class=""><div class=""><div class="WordSection1" style="page: WordSection1; caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><ul type="disc" style="margin-bottom: 0in; margin-top: 0in;" class=""><li class="MsoListParagraph" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 12pt; font-family: "Times New Roman", serif;" class="">The row and column layout for the Hessian is same as x.<o:p class=""></o:p></span></li></ul><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt; font-family: "Times New Roman", serif;" class=""><o:p class=""> </o:p></span></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt; font-family: "Times New Roman", serif;" class="">The tutorial example ex1 is extremely small (only 2 variables) so its implementation is very simplistic. I think, in parallel, it ships off constraints etc. to rank 0. It’s not an ideal example w.r.t demonstrating a parallel implementation. We aim to add more examples as we develop PDIPM. If you have an example to contribute then we would most welcome it and provide help on adding it.<o:p class=""></o:p></span></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt; font-family: "Times New Roman", serif;" class=""><o:p class=""> </o:p></span></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt; font-family: "Times New Roman", serif;" class="">Thanks,<o:p class=""></o:p></span></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt; font-family: "Times New Roman", serif;" class="">Shri<o:p class=""></o:p></span></div></div><div style="border-style: solid none none; border-top-width: 1pt; border-top-color: rgb(181, 196, 223); padding: 3pt 0in 0in;" class=""><div style="margin: 0in 0in 0in 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><b class=""><span style="font-size: 12pt;" class="">From:<span class="Apple-converted-space"> </span></span></b><span style="font-size: 12pt;" class="">petsc-dev <<a href="mailto:petsc-dev-bounces@mcs.anl.gov" class="">petsc-dev-bounces@mcs.anl.gov</a>> on behalf of Pierre Jolivet <<a href="mailto:pierre.jolivet@enseeiht.fr" class="">pierre.jolivet@enseeiht.fr</a>><br class=""><b class="">Date:<span class="Apple-converted-space"> </span></b>Monday, September 14, 2020 at 1:52 PM<br class=""><b class="">To:<span class="Apple-converted-space"> </span></b>PETSc Development <<a href="mailto:petsc-dev@mcs.anl.gov" class="">petsc-dev@mcs.anl.gov</a>><br class=""><b class="">Subject:<span class="Apple-converted-space"> </span></b>[petsc-dev] PDIPDM questions<o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0in 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div></div><div class=""><div style="margin: 0in 0in 0in 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Hello,<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in 0in 0in 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">In my quest to help users migrate from Ipopt to Tao, I’ve a new question.<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in 0in 0in 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">When looking at src/tao/constrained/tutorials/ex1.c, it seems that almost everything is centralized on rank 0 (local sizes are 0 but on rank 0).<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in 0in 0in 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">I’d like to have my Hessian distributed more naturally, as in (almost?) all other SNES/TS examples, but still keep the Jacobian of my equality constraint, which is of dimension 1 x N (N >> 1), centralized on rank 0.<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in 0in 0in 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Is this possible?<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in 0in 0in 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">If not, is it possible to supply the transpose of the Jacobian, of dimension N x 1, which could then be distributed row-wise like the Hessian?<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in 0in 0in 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Or maybe use some trick to distribute a MatAIJ/MatDense of dimension 1 x N column-wise? Use a MatNest with as many blocks as processes?<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in 0in 0in 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div></div><div class=""><div style="margin: 0in 0in 0in 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">So, just to sum up, how can I have a distributed Hessian with a Jacobian with a single row?<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in 0in 0in 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div></div><div class=""><div style="margin: 0in 0in 0in 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Thanks in advance for your help,<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in 0in 0in 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Pierre</div></div></div></div></blockquote></div><br class=""></div></div></blockquote></div><br class=""></body></html>