<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
Pierre,</div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div style="line-break:after-white-space">
<div>
<div><br>
</div>
<blockquote type="cite">
<div>
<div style="font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; line-break:after-white-space">
<div>1) Is there any hope to get PDIPDM to use a MatNest?</div>
<div><br>
</div>
<div class="x_ContentPasted1"><span style="color: rgb(48, 58, 242);">KKT matrix is indefinite and ill-conditioned, which must be solved using a direct matrix factorization method.</span></div>
</div>
</div>
</blockquote>
<div>But you are using PCBJACOBI in the paper you attached?</div>
<div>In any case, there are many such systems, e.g., a discretization of Stokes equations, that can be solved with something else than a direct factorization.</div>
<div><br>
</div>
<div><span style="color: rgb(79, 67, 209);">The KKT in this paper is very special, each block represents a time step with loose and weak couplings. We tested larger and slightly varying physical parameters, PCZBJACOBI fails convergence while CHOLESKY encounters
out of memory. </span></div>
<blockquote type="cite">
<div>
<div style="font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; line-break:after-white-space">
<div class="x_ContentPasted1"><span style="color: rgb(48, 58, 242);">For the current implementation, we use MUMPS Cholesky as default. To use MatNest, what direct solver to use, SCHUR_FACTOR? I do not know how to get it work. </span></div>
</div>
</div>
</blockquote>
<div>On the one hand, MatNest can efficiently convert to AIJ or SBAIJ if you want to stick to PCLU or PCCHOLESKY.</div>
<div>On the other hand, it allows to easily switch to PCFIELDSPLIT which can be used to solve saddle-point problems.</div>
<div><br>
</div>
<div><span style="font-family: "Segoe UI Web (West European)", "Segoe UI", -apple-system, "system-ui", Roboto, "Helvetica Neue", sans-serif; font-size: 12pt; display: inline !important; color: rgb(79, 67, 209); background-color: rgb(255, 255, 255);" class="ContentPasted0 ContentPasted2">MatNest
would be a natural way for these type of applications. PETSc has MatConvert_Nest_AIJ(), which could </span><span style="font-size: 12pt; color: rgb(79, 67, 209);">enable users to assemble their matrices in MatNest, and apply mumps/superlu direct solvers. I'm
not concerned about the overhead of matrix conversion, because matrix factorization dominates computation in general. </span></div>
<blockquote type="cite">
<div>
<div style="font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; line-break:after-white-space">
<div>2) Is this fixed <a href="https://lists.mcs.anl.gov/pipermail/petsc-dev/2020-September/026398.html" data-auth="NotApplicable">https://lists.mcs.anl.gov/pipermail/petsc-dev/2020-September/026398.html</a> ?</div>
<div>I cannot get users to transition away from Ipopt because of these two missing features.</div>
<div><br>
</div>
<div><span style="color: rgb(48, 58, 242);">The existing pdipm is the result of a MS student intern project. None of us involved are experts on the optimization solvers. We made a straightforward parallelization of Ipopt. It indeed needs further work, e.g.,
more features, better matrix storage, convergence criteria... To our knowledge, parallel pdipm is not available other than our pdipm.</span></div>
</div>
</div>
</blockquote>
<div>Ipopt can use MUMPS and PARDISO internally, so it’s in some sense parallel (using shared memory).</div>
<div>Also, this is not a very potent selling point.<br>
</div>
<div>My users that are satisfied with Ipopt as a "non-parallel" black box don’t want to have to touch part of their code just to stick it in a parallel black box which is limited to the same kind of linear solver and which has severe limitations with respect
to Hessian/Jacobian/constraint distributions.</div>
<div><br>
</div>
<div><span style="color: rgb(79, 67, 209);">We saw exiting 'parallel' pdipm papers, which conduct sequential computation and send KKT matrices to multiple processors, then they show scaling results on matrix computation only. Again, </span><span style="color: rgb(79, 67, 209);">I
wish someone can help to improve tao/pdipm. This is a useful solver.</span></div>
<blockquote type="cite">
<div>
<div style="font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; line-break:after-white-space">
<div><span style="color: rgb(48, 58, 242);">We should improve our pdipm. </span></div>
<div><span style="color: rgb(48, 58, 242);">Hong</span></div>
<div><br>
</div>
<div>
<div>
<blockquote type="cite">
<div>On 20 Apr 2023, at 5:47 PM, Zhang, Hong via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" data-auth="NotApplicable">petsc-users@mcs.anl.gov</a>> wrote:</div>
<br class="x_x_Apple-interchange-newline">
<div>
<div class="x_x_elementToProof x_x_ContentPasted0 x_x_ContentPasted1" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; background-color: rgb(255, 255, 255);">
<span class="x_x_ContentPasted2" style="font-family: Calibri, sans-serif; font-size: 14.6667px; display: inline !important; background-color: rgb(255, 255, 255);">Karthik,</span><br>
</div>
<div class="x_x_elementToProof x_x_ContentPasted0 x_x_ContentPasted1" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; background-color: rgb(255, 255, 255);">
We built a KKT matrix in TaoSNESJacobian_PDIPM() (see petsc/src/tao/constrained/impls/ipm/pdipm.c) which assembles several small matrices into a large KKT matrix in mpiaij format. You could take the same approach to insert P and P^T into your K.</div>
<div class="x_x_elementToProof x_x_ContentPasted0 x_x_ContentPasted1" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; background-color: rgb(255, 255, 255);">
FYI, I attached our paper.</div>
<div class="x_x_elementToProof x_x_ContentPasted0 x_x_ContentPasted1" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; background-color: rgb(255, 255, 255);">
Hong</div>
<div id="x_x_appendonsend" style="font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
</div>
<hr tabindex="-1" style="font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; display:inline-block; width:826.125px">
<span class="x_Apple-converted-space"> </span><span style="font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; float:none; display:inline!important"></span>
<div id="x_x_divRplyFwdMsg" dir="ltr" style="font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
<font face="Calibri, sans-serif" style="font-size:11pt"><b>From:</b><span class="x_x_Apple-converted-space"> </span>petsc-users <<a href="mailto:petsc-users-bounces@mcs.anl.gov" data-auth="NotApplicable">petsc-users-bounces@mcs.anl.gov</a>> on behalf of Matthew
Knepley <<a href="mailto:knepley@gmail.com" data-auth="NotApplicable">knepley@gmail.com</a>><br>
<b>Sent:</b><span class="x_x_Apple-converted-space"> </span>Thursday, April 20, 2023 5:37 AM<br>
<b>To:</b><span class="x_x_Apple-converted-space"> </span>Karthikeyan Chockalingam - STFC UKRI <<a href="mailto:karthikeyan.chockalingam@stfc.ac.uk" data-auth="NotApplicable">karthikeyan.chockalingam@stfc.ac.uk</a>><br>
<b>Cc:</b><span class="x_x_Apple-converted-space"> </span><a href="mailto:petsc-users@mcs.anl.gov" data-auth="NotApplicable">petsc-users@mcs.anl.gov</a><span class="x_Apple-converted-space"> </span><<a href="mailto:petsc-users@mcs.anl.gov" data-auth="NotApplicable">petsc-users@mcs.anl.gov</a>><br>
<b>Subject:</b><span class="x_x_Apple-converted-space"> </span>Re: [petsc-users] question about MatSetLocalToGlobalMapping</font>
<div> </div>
</div>
<div style="font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
<div dir="ltr">
<div dir="ltr">On Thu, Apr 20, 2023 at 6:13 AM Karthikeyan Chockalingam - STFC UKRI via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" data-auth="NotApplicable">petsc-users@mcs.anl.gov</a>> wrote:<br>
</div>
<div class="x_x_x_gmail_quote">
<blockquote class="x_x_x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left-width:1px; border-left-style:solid; border-left-color:rgb(204,204,204); padding-left:1ex">
<div class="x_x_x_msg4059408548263094035">
<div lang="EN-GB">
<div class="x_x_x_m_-6455518359002065503WordSection1">
<div style="margin-top:0px; margin-bottom:0px">Hello,</div>
<div style="margin-top:0px; margin-bottom:0px"><u></u> <u></u></div>
<div style="margin-top:0px; margin-bottom:0px">I created a new thread, thought would it be more appropriate (and is a continuation of my previous post). I want to construct the below K matrix (which is composed of submatrices)</div>
<div style="margin-top:0px; margin-bottom:0px"><u></u> <u></u></div>
<div style="margin-top:0px; margin-bottom:0px"><span>K = [A P^T</span><span style="font-size: 10pt; color: rgb(33, 33, 33);"><u></u><u></u></span></div>
<div style="margin-top:0px; margin-bottom:0px"><span> P 0]<u></u><u></u></span></div>
<div style="margin-top:0px; margin-bottom:0px"><span><u></u> <u></u></span></div>
<div style="margin-top:0px; margin-bottom:0px"><span>Where K is of type MatMPIAIJ. I first constructed the top left [A] using MatSetValues().<u></u><u></u></span></div>
<div style="margin-top:0px; margin-bottom:0px"><span><u></u> <u></u></span></div>
<div style="margin-top:0px; margin-bottom:0px"><span>Now, I would like to construct the bottom left [p] and top right [p^T]<span class="x_x_Apple-converted-space"> </span></span>using MatSetValuesLocal().</div>
<div style="margin-top:0px; margin-bottom:0px"><u></u> <u></u></div>
<div style="margin-top:0px; margin-bottom:0px">To use MatSetValuesLocal(), I first have to create a local-to-global mapping using ISLocalToGlobalMappingCreate. I have created two mapping row_mapping and column_mapping.</div>
</div>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>I do not understand why they are not the same map. Maybe I was unclear before. It looks like you have two fields, say phi and lambda, where lambda is a Lagrange multiplier imposing some constraint. Then you get a saddle point like this. You can imagine
matrices</div>
<div><br>
</div>
<div> (phi, phi) --> A</div>
<div> (phi, lambda) --> P^T</div>
<div> (lambda, phi) --> P</div>
<div><br>
</div>
<div>So you make a L2G map for the phi field and the lambda field. Oh, you are calling them row and col map, but they are my phi and lambda</div>
<div>maps. I do not like the row and col names since in P they reverse.</div>
<div> </div>
<blockquote class="x_x_x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left-width:1px; border-left-style:solid; border-left-color:rgb(204,204,204); padding-left:1ex">
<div class="x_x_x_msg4059408548263094035">
<div lang="EN-GB">
<div class="x_x_x_m_-6455518359002065503WordSection1">
<div style="margin-top:0px; margin-bottom:0px"><span>Q1) At what point should I declare<span class="x_x_Apple-converted-space"> </span></span>MatSetLocalToGlobalMapping – is it just before I use MatSetValuesLocal()?</div>
</div>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>Okay, it is good you are asking this because my thinking was somewhat confused. I think the precise steps are:</div>
<div><br>
</div>
<div> 1) Create the large saddle point matrix K</div>
<div><br>
</div>
<div> 1a) We must call <a href="https://petsc.org/main/manualpages/Mat/MatSetLocalToGlobalMapping/" data-auth="NotApplicable">https://petsc.org/main/manualpages/Mat/MatSetLocalToGlobalMapping/</a><span class="x_x_Apple-converted-space"> </span>on it. In
the simplest case, this just maps</div>
<div> the local rows numbers [0, Nrows) to the global rows numbers [rowStart, rowStart + Nrows).</div>
<div><br>
</div>
<div> 2) To form each piece:</div>
<div><br>
</div>
<div> 2a) Extract that block using <a href="https://petsc.org/main/manualpages/Mat/MatGetLocalSubMatrix/" data-auth="NotApplicable">https://petsc.org/main/manualpages/Mat/MatGetLocalSubMatrix/</a></div>
<div><br>
</div>
<div> This gives back a Mat object that you subsequently restore using <a href="https://petsc.org/main/manualpages/Mat/MatRestoreLocalSubMatrix/" data-auth="NotApplicable">https://petsc.org/main/manualpages/Mat/MatRestoreLocalSubMatrix/</a></div>
<div><br>
</div>
<div> 2b) Insert values using<span class="x_x_Apple-converted-space"> </span><a href="https://petsc.org/main/manualpages/Mat/MatSetValuesLocal/" data-auth="NotApplicable">https://petsc.org/main/manualpages/Mat/MatSetValuesLocal/</a></div>
<div><br>
</div>
<div> The local indices used for insertion here are indices relative to the block itself, and the L2G map for this matrix</div>
<div> has been rewritten to insert into that block in the larger matrix. Thus this looks like just calling MatSetValuesLocal()</div>
<div> on the smaller matrix block, but inserts correctly into the larger matrix.</div>
<div><br>
</div>
<div>Therefore, the code you write code in 2) could work equally well making the large matrix from 1), or independent smaller matrix blocks.</div>
<div><br>
</div>
<div>Does this make sense?</div>
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote class="x_x_x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left-width:1px; border-left-style:solid; border-left-color:rgb(204,204,204); padding-left:1ex">
<div class="x_x_x_msg4059408548263094035">
<div lang="EN-GB">
<div class="x_x_x_m_-6455518359002065503WordSection1">
<div style="margin-top:0px; margin-bottom:0px">I will use MatSetLocalToGlobalMapping(K, row_mapping, column_mapping) to build<span class="x_x_Apple-converted-space"> </span><span>the bottom left [P].<u></u><u></u></span></div>
<div style="margin-top:0px; margin-bottom:0px"><span><u></u> <u></u></span></div>
<div style="margin-top:0px; margin-bottom:0px"><span><u></u> <u></u></span></div>
<div style="margin-top:0px; margin-bottom:0px"><span>Q2) Can now I reset the mapping as<span class="x_x_Apple-converted-space"> </span></span>MatSetLocalToGlobalMapping(K, column_mapping, row_mapping) to build<span class="x_x_Apple-converted-space"> </span><span>the
top right [P^T]?<span class="x_x_Apple-converted-space"> </span><u></u><u></u></span></div>
<div style="margin-top:0px; margin-bottom:0px"><span><u></u> <u></u></span></div>
<div style="margin-top:0px; margin-bottom:0px"><u></u> <u></u></div>
<div style="margin-top:0px; margin-bottom:0px">Many thanks!<span><u></u><u></u></span></div>
<div style="margin-top:0px; margin-bottom:0px"><span><u></u> <u></u></span></div>
<div style="margin-top:0px; margin-bottom:0px"><span>Kind regards,<u></u><u></u></span></div>
<div style="margin-top:0px; margin-bottom:0px"><span>Karthik.<u></u><u></u></span></div>
<div style="margin-top:0px; margin-bottom:0px"><span><u></u> <u></u></span></div>
<div style="margin-top:0px; margin-bottom:0px"><span><u></u> <u></u></span></div>
<div style="margin-top:0px; margin-bottom:0px"><span><u></u> <u></u></span></div>
<div style="margin-top:0px; margin-bottom:0px"><span><u></u> <u></u></span></div>
<div style="margin-top:0px; margin-bottom:0px"><span><u></u> <u></u></span></div>
<div style="margin-top:0px; margin-bottom:0px"><u></u> <u></u></div>
<div style="margin-top:0px; margin-bottom:0px"><u></u> <u></u></div>
<div style="margin-top:0px; margin-bottom:0px"><span style="font-size: 10pt; color: rgb(33, 33, 33);"><u></u> <u></u></span></div>
<div style="margin-top:0px; margin-bottom:0px"><u></u> <u></u></div>
</div>
</div>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
<span class="x_x_x_gmail_signature_prefix">--<span class="x_x_Apple-converted-space"> </span></span><br>
<div dir="ltr" class="x_x_x_gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener</div>
<div><br>
</div>
<div><a href="http://www.cse.buffalo.edu/~knepley/" data-auth="NotApplicable">https://www.cse.buffalo.edu/~knepley/</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<span id="x_x_cid:AA825F93-5460-43A9-BD3E-EC5B64921BC3"><IET Generation Trans Dist - 2022 - Sundermann - Parallel primal‐dual interior point method for the solution of dynamic.pdf></span></div>
</blockquote>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</body>
</html>