<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<span style="font-size:12pt">Barry,</span>
<div style="font-size:12pt"><br>
</div>
<div style="font-size:12pt">Yes, you are right, Anear is the preconditioner that needs to be passed to KSPSetOperators() as the second matrix. </div>
<div style="font-size:12pt"><br>
</div>
<div style="font-size:12pt">I think I have figured out how to provide a preconditioner for matrix-free method. I was confused by matrix A created by MatCreateMFFD(), which is type MFFD. I thought I have to use it as the first matrix passed to KSPSetOperators().
Since it was not assembled (and can't) so that Petsc gave error message "matrix unassembled". In fact, I should ignore A, but use other two matrixes AA and B, which are type MPIAIJ, assemble them, pass them to KSPSetOperators() as first and second matrix.
Since function passed into MatMFFDSetFunction() does not take a matrix as an argument, matrix AA needs to be declared in user context so that it can be used in that function to perform matrix-vector product (partially) using MatMult() as you suggested. </div>
<div style="font-size:12pt"><br>
</div>
<div style="font-size:12pt">I have used a small test program to solve an 8x8 linear equation system and the code produces the same answers with/without the preconditioner. The iteration for this small problem, of course no improvement with the preconditioner
but the residual norms with/without the preconditioner are slightly different. I suppose the preconditioner takes effect.<br>
</div>
<div style="font-size:12pt"><br>
</div>
<div style="font-size:12pt">Now I need to think about if the applied preconditioner does the right job on the other part of matrix-vector product (non near part) for the problem I am working on.</div>
<div style="font-size:12pt"><br>
</div>
<div style="font-size:12pt">Thank you very very much and greatly appreciate your help!</div>
<div style="font-size:12pt"><br>
</div>
<span style="font-size:12pt">Jiannan</span><br>
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Barry Smith <bsmith@petsc.dev><br>
<b>Sent:</b> Monday, August 22, 2022 12:42 PM<br>
<b>To:</b> Tu, Jiannan <Jiannan_Tu@uml.edu><br>
<b>Cc:</b> petsc-users@mcs.anl.gov <petsc-users@mcs.anl.gov><br>
<b>Subject:</b> Re: [petsc-users] Using matrix-free with KSP</font>
<div> </div>
</div>
<div class="" style="word-wrap:break-word; line-break:after-white-space">
<p></p>
<div style="background-color:#FFCACA; border-style:hidden; border-color:#FFCACA; border-width:1pt; padding:2pt; font-size:12pt; line-height:16pt; font-family:Calibri; color:Black; text-align:left">
<span style="font-weight:bold">CAUTION:</span> This email was sent from outside the UMass Lowell network.</div>
<br>
<p></p>
<div><br class="">
<div><br class="">
<blockquote type="cite" class="">
<div class="">On Aug 22, 2022, at 12:25 PM, Tu, Jiannan <<a href="mailto:Jiannan_Tu@uml.edu" class="">Jiannan_Tu@uml.edu</a>> wrote:</div>
<br class="x_Apple-interchange-newline">
<div class="">
<div class="x_elementToProof" style="font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt">
The matrix-vector product involves part of matrix elements explicitly as shown below so that I need to insert values into this part of the matrix.</div>
<div class="x_elementToProof" style="font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt">
<span id="x_cid:07fd6bc2-973b-4485-8416-e22ba5302a62"><image.png></span></div>
<div class="x_elementToProof" style="font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt">
Initially I wrote my own code for calculating and storing matrix A^near_mi without using any Petsc matrix and then performed matrix-vector product using above equation. The code runs but converges very slow. Apparently a preconditioned is needed.<span class="x_Apple-converted-space"> </span><br class="">
</div>
<div class="x_elementToProof" style="font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt">
<br class="">
</div>
<div class="x_elementToProof" style="font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt">
I am not clear how Petsc can provide a preconditioner for my case.<br class="">
</div>
</div>
</blockquote>
<div><br class="">
</div>
I am now confused. I thought you were forming Anear as a PETSc matrix and passing that to KSPSetOperators as the second matrix and so the preconditioner was being built with Anear?</div>
<div><br class="">
</div>
<div> For your shell matrix vector product you should call MatMult(Anear,x,y) inside your custom routine and then add the rest of the matrix-vector product using your fast multipole products. What am I missing?</div>
<div><br class="">
</div>
<div> Barry</div>
<div><br class="">
<blockquote type="cite" class="">
<div class="">
<div id="x_appendonsend" class="" style="font-family:Helvetica; font-size:18px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
</div>
<hr tabindex="-1" class="" style="font-family:Helvetica; font-size:18px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; display:inline-block; width:977.046875px">
<span class="" style="font-family:Helvetica; font-size:18px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; float:none; display:inline!important"></span>
<div id="x_divRplyFwdMsg" dir="ltr" class="" style="font-family:Helvetica; font-size:18px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
<font face="Calibri, sans-serif" class="" style="font-size:11pt"><b class="">From:</b><span class="x_Apple-converted-space"> </span>Barry Smith <<a href="mailto:bsmith@petsc.dev" class="">bsmith@petsc.dev</a>><br class="">
<b class="">Sent:</b><span class="x_Apple-converted-space"> </span>Monday, August 22, 2022 8:42 AM<br class="">
<b class="">To:</b><span class="x_Apple-converted-space"> </span>Tu, Jiannan <<a href="mailto:Jiannan_Tu@uml.edu" class="">Jiannan_Tu@uml.edu</a>><br class="">
<b class="">Cc:</b><span class="x_Apple-converted-space"> </span><a href="mailto:petsc-users@mcs.anl.gov" class="">petsc-users@mcs.anl.gov</a><span class="x_Apple-converted-space"> </span><<a href="mailto:petsc-users@mcs.anl.gov" class="">petsc-users@mcs.anl.gov</a>><br class="">
<b class="">Subject:</b><span class="x_Apple-converted-space"> </span>Re: [petsc-users] Using matrix-free with KSP</font>
<div class=""> </div>
</div>
<div class="" style="font-family:Helvetica; font-size:18px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; word-wrap:break-word; line-break:after-white-space">
<p class="" style="margin-top:0px; margin-bottom:0px"></p>
<div class="" style="background-color:rgb(255,202,202); border:1pt hidden rgb(255,202,202); padding:2pt; font-size:12pt; line-height:16pt; font-family:Calibri; text-align:left">
<span class="" style="font-weight:bold">CAUTION:</span><span class="x_Apple-converted-space"> </span>This email was sent from outside the UMass Lowell network.</div>
<br class="">
<p class="" style="margin-top:0px; margin-bottom:0px"></p>
<div class=""><br class="">
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">On Aug 21, 2022, at 11:18 PM, Tu, Jiannan <<a href="mailto:Jiannan_Tu@uml.edu" class="">Jiannan_Tu@uml.edu</a>> wrote:</div>
<br class="x_x_Apple-interchange-newline">
<div class="">
<div class="x_x_WordSection1" style="font-family:Helvetica; font-size:18px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt">Barry,</span></div>
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"> </span></div>
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt">You are right. With PC type none, the same error appears.</span></div>
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"> </span></div>
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt">Now I use MatShell and I literally have three matrixes. A is a shell matrix, B is a preconditioner, and C is a matrix just for preallocation purpose.</span></div>
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"> </span></div>
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt">First C is used to determine non-zero structure. Then use MatPreallocatorPreallocation(C, PETSC_FALSE, B) to preallocate for B. After that do insertion and assembly for B. These steps work fine.</span></div>
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"> </span></div>
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt">Now the question is how to insert and assemble shell matrix A. MatPreallocatorPreallocation doesn’t work with shell matrix as indicated by an error message. MatSetValues or Mat assembly doesn’t support shell matrix either.<span class="x_x_Apple-converted-space"> </span></span></div>
</div>
</div>
</blockquote>
<div class=""><br class="">
</div>
<div class=""> A shell matrix is completely under your control. You decide on what data you want to store in it, and you provide the matrix-vector product routine. <font face="Calibri, sans-serif" class="">MatPreallocatorPreallocation and MatSetValues don't
work for shell matrix unless you provide code for them to work, which normally you would not do.</font></div>
<div class=""><font face="Calibri, sans-serif" class=""><br class="">
</font></div>
<div class=""><font face="Calibri, sans-serif" class=""> I don't understand why you want to use MatSetValues for your shell matrix. If you can use MatSetValues() and just build an explicit matrix, then why would you use a shell matrix, just use MATAIJ matrix? </font></div>
<div class=""><font face="Calibri, sans-serif" class=""><br class="">
</font></div>
<div class=""><font face="Calibri, sans-serif" class=""> Or does part of the operation inside your MatShell require a "regular" PETSc matrix? This would be fine, but it is your responsibility to design your Mat shell context to contain the "regular matrix"
and your responsibility to fill up that regular matrix inside your shell context, calling whatever code you choose to write. So for example your shell context would be a C struct and one of the fields would be a regular PETSc matrix, also part of your custom
matrix-vector product routine would call a regular MatMult() on the regular PETSc matrix stored inside the C struct.</font></div>
<div class=""><font face="Calibri, sans-serif" class=""><br class="">
</font></div>
<div class=""><font face="Calibri, sans-serif" class=""> Barry</font></div>
<div class=""><font face="Calibri, sans-serif" class=""><br class="">
</font></div>
<br class="">
<blockquote type="cite" class="">
<div class="">
<div class="x_x_WordSection1" style="font-family:Helvetica; font-size:18px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"></span></div>
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"> </span></div>
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt">Thank you,</span></div>
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt">Jiannan</span></div>
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"> </span></div>
<div class="">
<div class="" style="border-style:solid none none; border-top-width:1pt; border-top-color:rgb(225,225,225); padding:3pt 0in 0in">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<b class="">From:</b><span class="x_x_Apple-converted-space"> </span>Barry Smith <<a href="mailto:bsmith@petsc.dev" class="" style="color:purple; text-decoration:underline">bsmith@petsc.dev</a>><span class="x_x_Apple-converted-space"> </span><br class="">
<b class="">Sent:</b><span class="x_x_Apple-converted-space"> </span>Friday, August 19, 2022 4:02 PM<br class="">
<b class="">To:</b><span class="x_x_Apple-converted-space"> </span>Tu, Jiannan <<a href="mailto:Jiannan_Tu@uml.edu" class="" style="color:purple; text-decoration:underline">Jiannan_Tu@uml.edu</a>><br class="">
<b class="">Cc:</b><span class="x_x_Apple-converted-space"> </span><a href="mailto:petsc-users@mcs.anl.gov" class="" style="color:purple; text-decoration:underline">petsc-users@mcs.anl.gov</a><br class="">
<b class="">Subject:</b><span class="x_x_Apple-converted-space"> </span>Re: [petsc-users] Using matrix-free with KSP</div>
</div>
</div>
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
</div>
<div class="" style="border:1pt hidden rgb(255,202,202); padding:2pt">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif; line-height:16pt; background-color:rgb(255,202,202)">
<b class=""><span class="" style="font-size:12pt">CAUTION:</span></b><span class="" style="font-size:12pt"><span class="x_x_Apple-converted-space"> </span>This email was sent from outside the UMass Lowell network.</span></div>
</div>
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
</div>
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
</div>
</div>
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
I don't know why changing the PCType could matter, can you please try with your new code but use the -pc_type none option to turn off the use of the preconditioner to see if the error still appears (I think it should). </div>
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
</div>
</div>
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
Also if you have a custom function that can do the matrix-vector product (not using any differencing) then you should use MatShell() and provide the matrix-vector product operation with your code.</div>
</div>
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
</div>
</div>
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
Barry</div>
</div>
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
</div>
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<br class="">
<br class="">
</div>
<blockquote class="" style="margin-top:5pt; margin-bottom:5pt">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
On Aug 19, 2022, at 11:47 AM, Tu, Jiannan <<a href="mailto:Jiannan_Tu@uml.edu" class="" style="color:purple; text-decoration:underline">Jiannan_Tu@uml.edu</a>> wrote:</div>
</div>
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
</div>
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt">Barry,</span></div>
</div>
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"> </span></div>
</div>
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt">I have my own matrix-vector product but not use MatShell. I followed your previous instruction to use</span></div>
</div>
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"> </span></div>
</div>
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"> MatCreateMFFD(MPI_COMM_WORLD, PETSC_DECIDE, PETSC_DECIDE, N, N, &A);</span></div>
</div>
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"> MatMFFDSetFunction(A, formfunction, ¶ms);</span></div>
</div>
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"> MatSetFromOptions(A);</span></div>
</div>
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"> MatMFFDSetBase(A, X, NULL);</span></div>
</div>
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"> </span></div>
</div>
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt">The customized matrix-vector product is implemented in formfunction() provided to MatMFFDSetFunction(). In this way, the matrix A is not used. I think that is why Petsc complains the matrix is not assembled. However, when
PC type is none, the code runs without error although KSP solver doesn’t converge.</span></div>
</div>
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"> </span></div>
</div>
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt">The question then becomes how to link A to matrix-free matrix-vector multiplication. And where to put MatShell?</span></div>
</div>
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"> </span></div>
</div>
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt">Thank you,</span></div>
</div>
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt">Jiannan</span></div>
</div>
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"> </span></div>
</div>
<div class="">
<div class="" style="border-style:solid none none; border-top-width:1pt; border-top-color:rgb(225,225,225); padding:3pt 0in 0in">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<b class="">From:</b><span class="x_x_apple-converted-space"> </span>Barry Smith <<a href="mailto:bsmith@petsc.dev" class="" style="color:purple; text-decoration:underline">bsmith@petsc.dev</a>><span class="x_x_apple-converted-space"> </span><br class="">
<b class="">Sent:</b><span class="x_x_apple-converted-space"> </span>Friday, August 19, 2022 10:27 AM<br class="">
<b class="">To:</b><span class="x_x_apple-converted-space"> </span>Tu, Jiannan <<a href="mailto:Jiannan_Tu@uml.edu" class="" style="color:purple; text-decoration:underline">Jiannan_Tu@uml.edu</a>><br class="">
<b class="">Cc:</b><span class="x_x_apple-converted-space"> </span><a href="mailto:petsc-users@mcs.anl.gov" class="" style="color:purple; text-decoration:underline">petsc-users@mcs.anl.gov</a><br class="">
<b class="">Subject:</b><span class="x_x_apple-converted-space"> </span>Re: [petsc-users] Using matrix-free with KSP</div>
</div>
</div>
</div>
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
</div>
</div>
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif; line-height:16pt; background-color:rgb(255,202,202)">
<b class=""><span class="" style="font-size:12pt">CAUTION:</span></b><span class="x_x_apple-converted-space"><span class="" style="font-size:12pt"> </span></span><span class="" style="font-size:12pt">This email was sent from outside the UMass Lowell network.</span></div>
</div>
</div>
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
</div>
</div>
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
</div>
</div>
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<br class="">
<br class="">
<br class="">
</div>
</div>
<blockquote class="" style="margin-top:5pt; margin-bottom:5pt">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
On Aug 18, 2022, at 11:41 PM, Tu, Jiannan <<a href="mailto:Jiannan_Tu@uml.edu" class="" style="color:purple; text-decoration:underline">Jiannan_Tu@uml.edu</a>> wrote:</div>
</div>
</div>
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt">The program calls KSPSolve just once and the error occurs after the first iteration.</span></div>
</div>
</div>
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"> </span></div>
</div>
</div>
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt">I realized the matrix-vector product in my case is different from that approximating matrix-vector multiply by finite difference of function vector. It consists of two parts. One is multiplication of part of the matrix
with part of vector, and another part results from so called fast multipole method. I’m not clear how to relate such calculation of matrix-vector product to the finite differencing of function vector.</span></div>
</div>
</div>
</div>
</blockquote>
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
</div>
</div>
</div>
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
Are you using finite differencing at all in a matrix-vector product, or do you have your own custom matrix-vector product code (with MatShell)? </div>
</div>
</div>
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
</div>
</div>
</div>
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
Anyways the easiest thing to do is </div>
</div>
</div>
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
</div>
</div>
</div>
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
use the run time option -on_error_attach_debugger noxterm and run the code, </div>
</div>
</div>
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
</div>
</div>
</div>
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
when it stops in the debugger you can type bt (for backtrace)</div>
</div>
</div>
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
</div>
</div>
</div>
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
then do up a few times to move up the stack until you get to the PETSc function call that generates the error</div>
</div>
</div>
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
</div>
</div>
</div>
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
you can then do "call MatView(thematrixvarible,0) to have PETSc display what the matrix is and</div>
</div>
</div>
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
</div>
</div>
</div>
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
you can also do print *thematrixvariable to see the exact struct with all its fields</div>
</div>
</div>
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
</div>
</div>
</div>
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
If this doesn't tell you why the troublesome matrix is not assembled then cut and paste ALL of the output and email it</div>
</div>
</div>
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
</div>
</div>
</div>
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
Barry</div>
</div>
</div>
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<br class="">
<br class="">
<br class="">
</div>
</div>
<blockquote class="" style="margin-top:5pt; margin-bottom:5pt">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"> </span></div>
</div>
</div>
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"> </span></div>
</div>
</div>
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"> </span></div>
</div>
</div>
<div class="">
<div class="" style="border-style:solid none none; border-top-width:1pt; border-top-color:rgb(225,225,225); padding:3pt 0in 0in">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<b class="">From:</b><span class="x_x_apple-converted-space"> </span>Barry Smith <<a href="mailto:bsmith@petsc.dev" class="" style="color:purple; text-decoration:underline">bsmith@petsc.dev</a>><span class="x_x_apple-converted-space"> </span><br class="">
<b class="">Sent:</b><span class="x_x_apple-converted-space"> </span>Thursday, August 18, 2022 7:44 PM<br class="">
<b class="">To:</b><span class="x_x_apple-converted-space"> </span>Tu, Jiannan <<a href="mailto:Jiannan_Tu@uml.edu" class="" style="color:purple; text-decoration:underline">Jiannan_Tu@uml.edu</a>><br class="">
<b class="">Cc:</b><span class="x_x_apple-converted-space"> </span><a href="mailto:petsc-users@mcs.anl.gov" class="" style="color:purple; text-decoration:underline">petsc-users@mcs.anl.gov</a><br class="">
<b class="">Subject:</b><span class="x_x_apple-converted-space"> </span>Re: [petsc-users] Using matrix-free with KSP</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif; line-height:16pt; background-color:rgb(255,202,202)">
<b class=""><span class="" style="font-size:12pt">CAUTION:</span></b><span class="x_x_apple-converted-space"><span class="" style="font-size:12pt"> </span></span><span class="" style="font-size:12pt">This email was sent from outside the UMass Lowell network.</span></div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
Is it in the first iteration of the first KSPSolve that you get this message? Or in the second call to KSPSolve? </div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
The error comes from a matrix vector product with your A matrix, what is that and where did it come from. The error message is not from your B matrix, that is not used in the MatMult()</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<br class="">
<br class="">
<br class="">
<br class="">
</div>
</div>
</div>
<blockquote class="" style="margin-top:5pt; margin-bottom:5pt">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
On Aug 18, 2022, at 3:05 PM, Tu, Jiannan <<a href="mailto:Jiannan_Tu@uml.edu" class="" style="color:purple; text-decoration:underline">Jiannan_Tu@uml.edu</a>> wrote:</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt">Barry,<span class="x_x_apple-converted-space"> </span></span></div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"> </span></div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt">Thanks. The method works. Assembly of actual matrix now completes quickly. The convergency is much better for the first iteration. The residual normal is in the order of 10^-10 instead of 10^-2. However, there is an error
message after the first iteration: “Object is in wrong state. Not for unassembled matrix” even though both preallocator and actual matrix have been assembled (see the code snippet below). Could you please give me some hints what’s going wrong? I use KSP type
bcgs and PC type mat</span></div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"> </span></div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt">Thank you,</span></div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt">Jiannan</span></div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"> </span></div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt">-----------------------------------------------------</span></div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt">Solve linear equations ...</span></div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"> 0 KSP Residual norm 9.841179147519e-10</span></div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt">[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------</span></div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt">[0]PETSC ERROR: Object is in wrong state</span></div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt">[0]PETSC ERROR: Not for unassembled matrix</span></div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt">[0]PETSC ERROR: See<span class="x_x_apple-converted-space"> </span></span><a href="https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Frelease%2Ffaq%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7Cc8226b4a8af040e8ca0208da845d5e8b%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637967833829887546%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=PFsyWAtTw7CvGpV9csURQXWxsSFCbZ3xGUPkE3cARgo%3D&reserved=0" originalsrc="https://petsc.org/release/faq/" shash="z9nIZw0GQNkMRlE96A8klfJZiEQ8hnEHUkJ3vg5QUUcX8Oc5y1MCLwxONJH5MJdOWLYMlhLi9N/2Vvq942BMaoUCkepe3Vk5XItY4I8gkkgzls/ctiM/HltglKXyhcIleTne3i2mNVM9FkxeKIH06ICA9e8aBQLdxTB4VmhQ2b8=" originalsrc="https://petsc.org/release/faq/" shash="NK5JcBNe8TpkAOj3jIMr17o4nm6AIQZbSK7NTXB0uZWu1CVq6TzZSEn4h7Q0XiPzCCn7p3JmTcGzxLuWkOKcQbmrTzKxiC0y0MsLQei7kCi1m58XtWg2UYoglfMgntqWryDRWjVRDilRUmGQphDymyBpFhfYrc2eWAxAsgI78JE=" class="" style="color:purple; text-decoration:underline"><span class="" style="font-size:12pt">https://petsc.org/release/faq/</span></a><span class="x_x_apple-converted-space"><span class="" style="font-size:12pt"> </span></span><span class="" style="font-size:12pt">for
trouble shooting.</span></div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt">[0]PETSC ERROR: Petsc Release Version 3.16.6, Mar 30, 2022</span></div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt">[0]PETSC ERROR: ./iesolver on a named REXNET-WS4 by jiannantu Thu Aug 18 14:48:43 2022</span></div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt">[0]PETSC ERROR: Configure options --download-f2cblaslapack=yes --with-mpi-dir=/usr/local --with-fc=0 --prefix=/home/jiannantu/petsc --with-scalar-type=complex --with-64-bit-indices=1</span></div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt">[0]PETSC ERROR: #1 MatMult() at /home/jiannantu/petsc-3.16.6/src/mat/interface/matrix.c:2425</span></div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt">[0]PETSC ERROR: #2 PCApplyBAorAB() at /home/jiannantu/petsc-3.16.6/src/ksp/pc/interface/precon.c:730</span></div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt">[0]PETSC ERROR: #3 KSP_PCApplyBAorAB() at /home/jiannantu/petsc-3.16.6/include/petsc/private/kspimpl.h:421</span></div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt">[0]PETSC ERROR: Linear equations not solved. Diverged reason: 0</span></div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt">#4 KSPSolve_BCGS() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/impls/bcgs/bcgs.c:87</span></div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt">[0]PETSC ERROR: #5 KSPSolve_Private() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itfunc.c:914</span></div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt">[0]PETSC ERROR: #6 KSPSolve() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itfunc.c:1086</span></div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"> </span></div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt">-----------------------------------------------</span></div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"> MatCreate(MPI_COMM_WORLD, &B);</span></div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"> MatSetType(B, MATMPIAIJ);</span></div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"> MatSetSizes(B, PETSC_DECIDE, PETSC_DECIDE, N, N);</span></div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"> </span></div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"> Mat preall;</span></div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"> MatCreate(MPI_COMM_WORLD, &preall);</span></div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"> MatSetType(preall, MATPREALLOCATOR);</span></div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"> MatSetSizes(preall, PETSC_DECIDE, PETSC_DECIDE, N, N);</span></div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"> </span></div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"> MatMPIAIJSetPreallocation(preall, 0, d_nnz, 0, o_nnz);</span></div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"> MatSetUp(preall);</span></div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif; text-indent:10.5pt">
<span class="" style="font-size:12pt">MatSetOption(preall, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE);</span></div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif; text-indent:10.5pt">
<span class="" style="font-size:12pt"> </span></div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif; text-indent:10.5pt">
<span class="" style="font-size:12pt">//set values row by row</span></div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif; text-indent:10.5pt">
<span class="" style="font-size:12pt">MatSetValues(preall, 1, &m, 1, nCols, matCols, INSERT_VALUES);</span></div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif; text-indent:10.5pt">
<span class="" style="font-size:12pt"> </span></div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif; text-indent:10.5pt">
<span class="" style="font-size:12pt">MatAssemblyBegin(preall, MAT_FINAL_ASSEMBLY);</span></div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif; text-indent:10.5pt">
<span class="" style="font-size:12pt">MatAssemblyEnd(preall, MAT_FINAL_ASSEMBLY);</span></div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif; text-indent:10.5pt">
<span class="" style="font-size:12pt"> </span></div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif; text-indent:10.5pt">
<span class="" style="font-size:12pt">//set values for actual matrix row by row</span></div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif; text-indent:10.5pt">
<span class="" style="font-size:12pt">MatSetValues(B, 1, &m, 1, nCols, matCols, INSERT_VALUES);</span></div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif; text-indent:10.5pt">
<span class="" style="font-size:12pt"> </span></div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif; text-indent:10.5pt">
<span class="" style="font-size:12pt">MatAssemblyBegin(B, MAT_FINAL_ASSEMBLY);</span></div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif; text-indent:10.5pt">
<span class="" style="font-size:12pt">MatAssemblyEnd(B, MAT_FINAL_ASSEMBLY);</span></div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif; text-indent:10.5pt">
<span class="" style="font-size:12pt"> </span></div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"> KSPCreate(MPI_COMM_WORLD, &ksp);</span></div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"> KSPSetOperators(ksp, A, B);</span></div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"> KSPSetFromOptions(ksp);</span></div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"> </span></div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"> PC pc;</span></div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"> KSPGetPC(ksp, &pc);</span></div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"> PCSetFromOptions(pc);</span></div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"> </span></div>
</div>
</div>
</div>
<div class="">
<div class="" style="border-style:solid none none; border-top-width:1pt; border-top-color:rgb(225,225,225); padding:3pt 0in 0in">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<b class="">From:</b><span class="x_x_apple-converted-space"> </span>Barry Smith <<a href="mailto:bsmith@petsc.dev" class="" style="color:purple; text-decoration:underline">bsmith@petsc.dev</a>><span class="x_x_apple-converted-space"> </span><br class="">
<b class="">Sent:</b><span class="x_x_apple-converted-space"> </span>Thursday, August 18, 2022 12:59 PM<br class="">
<b class="">To:</b><span class="x_x_apple-converted-space"> </span>Tu, Jiannan <<a href="mailto:Jiannan_Tu@uml.edu" class="" style="color:purple; text-decoration:underline">Jiannan_Tu@uml.edu</a>><br class="">
<b class="">Cc:</b><span class="x_x_apple-converted-space"> </span><a href="mailto:petsc-users@mcs.anl.gov" class="" style="color:purple; text-decoration:underline">petsc-users@mcs.anl.gov</a><br class="">
<b class="">Subject:</b><span class="x_x_apple-converted-space"> </span>Re: [petsc-users] Using matrix-free with KSP</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif; line-height:16pt; background-color:rgb(255,202,202)">
<b class=""><span class="" style="font-size:12pt">CAUTION:</span></b><span class="x_x_apple-converted-space"><span class="" style="font-size:12pt"> </span></span><span class="" style="font-size:12pt">This email was sent from outside the UMass Lowell network.</span></div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
Yes, once you have preallocated the real matrix you can destroy the preallocation matrix whose only job is to gather the preallocation information</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<br class="">
<br class="">
<br class="">
<br class="">
<br class="">
</div>
</div>
</div>
</div>
<blockquote class="" style="margin-top:5pt; margin-bottom:5pt">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
On Aug 18, 2022, at 12:52 PM, Tu, Jiannan <<a href="mailto:Jiannan_Tu@uml.edu" class="" style="color:purple; text-decoration:underline">Jiannan_Tu@uml.edu</a>> wrote:</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt">Thanks, Barry.</span></div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"> </span></div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt">So I need actually two matrixes, one for preallocator and one for actual matrix that can be passed to KSPSetOperators(). -mat_type preallocator option is used to speed up doing insertion into preallocator, then use MatPreallocatorPreallocate()
to preallocate actual matrix and do actual insertion of values into it., right?</span></div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"> </span></div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt">The code runs in parallel. Each process owns number of rows that equals to number of unknowns (that is, xm in 1D DM) it owns.</span></div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"> </span></div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt">Jiannan</span></div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"> </span></div>
</div>
</div>
</div>
</div>
<div class="">
<div class="" style="border-style:solid none none; border-top-width:1pt; border-top-color:rgb(225,225,225); padding:3pt 0in 0in">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<b class="">From:</b><span class="x_x_apple-converted-space"> </span>Barry Smith <<a href="mailto:bsmith@petsc.dev" class="" style="color:purple; text-decoration:underline">bsmith@petsc.dev</a>><span class="x_x_apple-converted-space"> </span><br class="">
<b class="">Sent:</b><span class="x_x_apple-converted-space"> </span>Thursday, August 18, 2022 11:37 AM<br class="">
<b class="">To:</b><span class="x_x_apple-converted-space"> </span>Tu, Jiannan <<a href="mailto:Jiannan_Tu@uml.edu" class="" style="color:purple; text-decoration:underline">Jiannan_Tu@uml.edu</a>><br class="">
<b class="">Cc:</b><span class="x_x_apple-converted-space"> </span><a href="mailto:petsc-users@mcs.anl.gov" class="" style="color:purple; text-decoration:underline">petsc-users@mcs.anl.gov</a><br class="">
<b class="">Subject:</b><span class="x_x_apple-converted-space"> </span>Re: [petsc-users] Using matrix-free with KSP</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif; line-height:16pt; background-color:rgb(255,202,202)">
<b class=""><span class="" style="font-size:12pt">CAUTION:</span></b><span class="x_x_apple-converted-space"><span class="" style="font-size:12pt"> </span></span><span class="" style="font-size:12pt">This email was sent from outside the UMass Lowell network.</span></div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
The preallocator MatType matrix cannot be passed to the KSPSetOperators(), you need to create an actual matrix, for example MATAIJ and use the preallocator to set its preallocation and then fill up the MATAIJ matrix with the usual MatSetValues and pass that
matrix to the KSPSetOperators.</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<br class="">
<br class="">
<br class="">
<br class="">
<br class="">
<br class="">
</div>
</div>
</div>
</div>
</div>
<blockquote class="" style="margin-top:5pt; margin-bottom:5pt">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
On Aug 18, 2022, at 11:30 AM, Tu, Jiannan <<a href="mailto:Jiannan_Tu@uml.edu" class="" style="color:purple; text-decoration:underline">Jiannan_Tu@uml.edu</a>> wrote:</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt">Hi Barry,</span></div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"> </span></div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt">The MATPREALLOCATOR solved problem of slow matrix assembly. But the solver failed because of “Matrix type preallocator does not have a multiply defined”.</span></div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"> </span></div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt">I guess this is because the matrix-free is used. I am wondering how a preconditioner is applied to the matrix-vector multiply in Petsc.</span></div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"> </span></div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt">Thank you,</span></div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt">Jiannan</span></div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"> </span></div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt">[0]PETSC ERROR: No support for this operation for this object type</span></div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt">[0]PETSC ERROR: Matrix type preallocator does not have a multiply defined</span></div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt">[0]PETSC ERROR: See<span class="x_x_apple-converted-space"> </span></span><a href="https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Frelease%2Ffaq%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7Cc8226b4a8af040e8ca0208da845d5e8b%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637967833829887546%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=PFsyWAtTw7CvGpV9csURQXWxsSFCbZ3xGUPkE3cARgo%3D&reserved=0" originalsrc="https://petsc.org/release/faq/" shash="z9nIZw0GQNkMRlE96A8klfJZiEQ8hnEHUkJ3vg5QUUcX8Oc5y1MCLwxONJH5MJdOWLYMlhLi9N/2Vvq942BMaoUCkepe3Vk5XItY4I8gkkgzls/ctiM/HltglKXyhcIleTne3i2mNVM9FkxeKIH06ICA9e8aBQLdxTB4VmhQ2b8=" originalsrc="https://petsc.org/release/faq/" shash="NK5JcBNe8TpkAOj3jIMr17o4nm6AIQZbSK7NTXB0uZWu1CVq6TzZSEn4h7Q0XiPzCCn7p3JmTcGzxLuWkOKcQbmrTzKxiC0y0MsLQei7kCi1m58XtWg2UYoglfMgntqWryDRWjVRDilRUmGQphDymyBpFhfYrc2eWAxAsgI78JE=" class="" style="color:purple; text-decoration:underline"><span class="" style="font-size:12pt">https://petsc.org/release/faq/</span></a><span class="x_x_apple-converted-space"><span class="" style="font-size:12pt"> </span></span><span class="" style="font-size:12pt">for
trouble shooting.</span></div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt">[0]PETSC ERROR: Petsc Release Version 3.16.6, Mar 30, 2022</span></div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt">[0]PETSC ERROR: ./iesolver on a named REXNET-WS4 by jiannantu Thu Aug 18 11:21:55 2022</span></div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt">[0]PETSC ERROR: Configure options --download-f2cblaslapack=yes --with-mpi-dir=/usr/local --with-fc=0 --prefix=/home/jiannantu/petsc --with-scalar-type=complex --with-64-bit-indices=1</span></div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt">[0]PETSC ERROR: #1 MatMult() at /home/jiannantu/petsc-3.16.6/src/mat/interface/matrix.c:2437</span></div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt">[0]PETSC ERROR: #2 PCApply_Mat() at /home/jiannantu/petsc-3.16.6/src/ksp/pc/impls/mat/pcmat.c:9</span></div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt">[0]PETSC ERROR: #3 PCApply() at /home/jiannantu/petsc-3.16.6/src/ksp/pc/interface/precon.c:445</span></div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt">[0]PETSC ERROR: #4 KSP_PCApply() at /home/jiannantu/petsc-3.16.6/include/petsc/private/kspimpl.h:382</span></div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt">[0]PETSC ERROR: #5 KSPInitialResidual() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itres.c:65</span></div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt">[0]PETSC ERROR: #6 KSPSolve_BCGS() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/impls/bcgs/bcgs.c:43</span></div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt">[0]PETSC ERROR: #7 KSPSolve_Private() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itfunc.c:914</span></div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt">[0]PETSC ERROR: #8 KSPSolve() at /home/jiannantu/petsc-3.16.6/src/ksp/ksp/interface/itfunc.c:1086</span></div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"> </span></div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="" style="border-style:solid none none; border-top-width:1pt; border-top-color:rgb(225,225,225); padding:3pt 0in 0in">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<b class="">From:</b><span class="x_x_apple-converted-space"> </span>Barry Smith <<a href="mailto:bsmith@petsc.dev" class="" style="color:purple; text-decoration:underline">bsmith@petsc.dev</a>><span class="x_x_apple-converted-space"> </span><br class="">
<b class="">Sent:</b><span class="x_x_apple-converted-space"> </span>Thursday, August 18, 2022 8:35 AM<br class="">
<b class="">To:</b><span class="x_x_apple-converted-space"> </span>Tu, Jiannan <<a href="mailto:Jiannan_Tu@uml.edu" class="" style="color:purple; text-decoration:underline">Jiannan_Tu@uml.edu</a>><br class="">
<b class="">Cc:</b><span class="x_x_apple-converted-space"> </span><a href="mailto:petsc-users@mcs.anl.gov" class="" style="color:purple; text-decoration:underline">petsc-users@mcs.anl.gov</a><br class="">
<b class="">Subject:</b><span class="x_x_apple-converted-space"> </span>Re: [petsc-users] Using matrix-free with KSP</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif; line-height:16pt; background-color:rgb(255,202,202)">
<b class=""><span class="" style="font-size:12pt">CAUTION:</span></b><span class="x_x_apple-converted-space"><span class="" style="font-size:12pt"> </span></span><span class="" style="font-size:12pt">This email was sent from outside the UMass Lowell network.</span></div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
Slow assembly is usually due to under preallocation. You can change to using </div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<blockquote class="" style="margin-top:5pt; margin-bottom:5pt">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt">MatSetOption(B, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_TRUE);</span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
to detect if you are under preallocating. See<span class="x_x_apple-converted-space"> </span><a href="https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanual%2Fmat%2F%23sec-matsparse&data=05%7C01%7CJiannan_Tu%40uml.edu%7Cc8226b4a8af040e8ca0208da845d5e8b%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637967833829887546%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=xdOGjLTnr9umc%2FOmbWXPu3BFQd7kjxeYJF2RS%2F1vGMg%3D&reserved=0" originalsrc="https://petsc.org/main/docs/manual/mat/#sec-matsparse" shash="gmLUms+cc/qGMZScjLQRkTEjqKlVF2wvPcNbkY6+ssr8UhXeLwirqanPF8z6NklFEZIu8SouIf8BDSqG+8+z7yj7skzvKMMkXe1CaT6Eq6WbNN0B+C1TudChg4jGTjI+BbHrH9EAmY2l/ePPNKekTw7AhFwUa+EAE7T2/bkExNA=" originalsrc="https://petsc.org/main/docs/manual/mat/#sec-matsparse" shash="EhcDIEvVgclMVgeh9dBj5Prbhf+szUl397OWv1SnzFC+B8diJ0JSS5qTOPks1HIpyYUBjyYSiPwJkKAeInVZa1jzWUMFgaeEACUL9SeUGXXAbPb6kOY66wglAatANp0rk3bqOxNEYbaohAYXW5s0Cd6WIFT6pU8UOD2R0tkhnNY=" class="" style="color:purple; text-decoration:underline">https://petsc.org/main/docs/manual/mat/#sec-matsparse</a>.
That section clearly needs some sprucing up. If it is difficult to determine good preallocation values you can use <a href="https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMATPREALLOCATOR%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7Cc8226b4a8af040e8ca0208da845d5e8b%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637967833829887546%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=D%2BPftcvN3AjHrPPNNiZP%2B0GiF2DJobBtqO3IIBmRDf8%3D&reserved=0" originalsrc="https://petsc.org/main/docs/manualpages/Mat/MATPREALLOCATOR/" shash="OkNj75HFqtjK3q62tpejkYnfGHDCgmuK7eB1TN8uFj/GuoC0134vKJtDx9003VybMDPmdigQfPPxv3Nm0EiINz4H4+Fzo2a4Ij3PSqEsYkkleQNbww36IhDHDjFgkFOW+oHPR3+DTdLPMvF/JzxFaEe2dWoiHOzco1S0hq724eo=" originalsrc="https://petsc.org/main/docs/manualpages/Mat/MATPREALLOCATOR/" shash="TTyhTzJ6QAurO5rtp4+hViePnG+ny9uz9d23NH+hztEiu+nMkpwX8XIgkVeV34eGD+yHyxwqHXT7t2aG+49k4E5KEsamNxCQAOQJP7KlmMeBa4qk2srUFa89lA6IwhtYoqKx5Is7cCiykHxSGpGBsbReBWzWi8qJ/Y7SmJplIFw=" class="" style="color:purple; text-decoration:underline">https://petsc.org/main/docs/manualpages/Mat/MATPREALLOCATOR/</a> to
compute the needed preallocation efficiently.</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
Are you running in parallel? If so how are you determining which rows of entries to compute on each MPI rank? You will want most of the rows to be computed on the rank where the values are stored, you can determine this with <a href="https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMatGetOwnershipRange.html&data=05%7C01%7CJiannan_Tu%40uml.edu%7Cc8226b4a8af040e8ca0208da845d5e8b%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637967833829887546%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=dmqAfAFRE%2FTvZ4b2yb4btNZe48pZlL0DbLpOLLRqOd8%3D&reserved=0" originalsrc="https://petsc.org/main/docs/manualpages/Mat/MatGetOwnershipRange.html" shash="rIs+KzLeFS6swdydFeJKy8ZowaJugqRtUMwB9/w9EOYbp8svkCtlF5qo/FTC9Tdc6DX2pItC2/0YYce4n25tSHUvTLBkWc9N8TMSZnCxrwIu3/HuDyZQq1j9tr8ayrXNgSYbfo9yrxZgFH9KvvITrQkJphMA3lLUdgYba02956E=" originalsrc="https://petsc.org/main/docs/manualpages/Mat/MatGetOwnershipRange.html" shash="ZOUZ1re4BOuAHtczVnVsv2E+529csm5p8JVdsLK1Bu+YCIjBNIISfQJI9yy+HBRCgspHRwAE3tQra9y5mvAJftZESFbmjAJhrbmUCLfJjoWFf1t49iOimbSW+PXSLRe+4bQHK464Zq3YTmeKFzUA7PDjG4pKgrRuIsZh6FAYLmM=" class="" style="color:purple; text-decoration:underline">https://petsc.org/main/docs/manualpages/Mat/MatGetOwnershipRange.html</a></div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
Barry</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-family:"Segoe UI",sans-serif; color:rgb(51,51,51)"><br class="">
<br class="">
<br class="">
<br class="">
<br class="">
<br class="">
<br class="">
</span></div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<br class="">
<br class="">
<br class="">
<br class="">
<br class="">
<br class="">
<br class="">
</div>
</div>
</div>
</div>
</div>
</div>
<blockquote class="" style="margin-top:5pt; margin-bottom:5pt">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
On Aug 18, 2022, at 12:50 AM, Tu, Jiannan <<a href="mailto:Jiannan_Tu@uml.edu" class="" style="color:purple; text-decoration:underline">Jiannan_Tu@uml.edu</a>> wrote:</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt">I implemented the preconditioner. The solver converges quickly when the problem size is small. But when the size increases, say to 100k unknowns, the code hangs at assembly of the preconditioner matrix. The function call
sequence is like this</span></div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"> </span></div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"> MatCreate(MPI_COMM_WORLD, &B);</span></div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"> MatSetType(B, MATMPIAIJ);</span></div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"> MatSetSizes(B, PETSC_DECIDE, PETSC_DECIDE, N, N);</span></div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"> MatSetFromOptions(B);</span></div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"> </span></div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"> // The number of diagonal and off diagonal non zeros only can be determined run-time.</span></div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"> MatMPIAIJSetPreallocation(B, 0, d_nnz, 0, o_nnz);</span></div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"> MatSetOption(B, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE);</span></div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"> </span></div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"> //here insert values row by row.</span></div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif; text-indent:10.5pt">
<span class="" style="font-size:12pt">MatSetValues(B, 1, &m, nn, nCols, matCols, INSERT_VALUES);</span></div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif; text-indent:10.5pt">
<span class="" style="font-size:12pt"> </span></div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif; text-indent:10.5pt">
<span class="" style="font-size:12pt">MatAssemblyBegin(B, MAT_FINAL_ASSEMBLY);</span></div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif; text-indent:10.5pt">
<span class="" style="font-size:12pt">MatAssemblyEnd(B, MAT_FINAL_ASSEMBLY);</span></div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif; text-indent:10.5pt">
<span class="" style="font-size:12pt"> </span></div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif; text-indent:10.5pt">
<span class="" style="font-size:12pt">Could you please tell me what I have done wrong?</span></div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif; text-indent:10.5pt">
<span class="" style="font-size:12pt"> </span></div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif; text-indent:10.5pt">
<span class="" style="font-size:12pt">Thank you,</span></div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif; text-indent:10.5pt">
<span class="" style="font-size:12pt">Jiannan</span></div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"> <span class="x_x_apple-converted-space"> </span></span></div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="" style="border-style:solid none none; border-top-width:1pt; border-top-color:rgb(225,225,225); padding:3pt 0in 0in">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<b class="">From:</b><span class="x_x_apple-converted-space"> </span>Barry Smith <<a href="mailto:bsmith@petsc.dev" class="" style="color:purple; text-decoration:underline">bsmith@petsc.dev</a>><span class="x_x_apple-converted-space"> </span><br class="">
<b class="">Sent:</b><span class="x_x_apple-converted-space"> </span>Tuesday, August 16, 2022 2:00 PM<br class="">
<b class="">To:</b><span class="x_x_apple-converted-space"> </span>Tu, Jiannan <<a href="mailto:Jiannan_Tu@uml.edu" class="" style="color:purple; text-decoration:underline">Jiannan_Tu@uml.edu</a>><br class="">
<b class="">Cc:</b><span class="x_x_apple-converted-space"> </span><a href="mailto:petsc-users@mcs.anl.gov" class="" style="color:purple; text-decoration:underline">petsc-users@mcs.anl.gov</a><br class="">
<b class="">Subject:</b><span class="x_x_apple-converted-space"> </span>Re: [petsc-users] Using matrix-free with KSP</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif; line-height:16pt; background-color:rgb(255,202,202)">
<b class=""><span class="" style="font-size:12pt">CAUTION:</span></b><span class="x_x_apple-converted-space"><span class="" style="font-size:12pt"> </span></span><span class="" style="font-size:12pt">This email was sent from outside the UMass Lowell network.</span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
Create another matrix B that contains "<span class="" style="font-size:12pt">Selected parts of the matrix can be pre-calculated and stored to serve as the preconditioner." When you call KSPSetOperators() pass in B as the second matrix argument.</span></div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"> Now if literally application of B is the preconditioner you want to use, then simply use PCMAT as the preconditioner, and it will apply B each time it needs to apply the preconditioner. With this, you do not even need
a PCShell. </span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"> If your preconditioner uses B to construct your own specific data structure needed to apply your preconditioner, then you can use PCShellSetSetUp() and have your routine call PCGetOperators() to pull out the B matrix
and use it to construct your preconditioner. </span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"> If you want PETSc to construct a standard preconditioner from B, then you don't need the PCShell; you simply pass in the B, as above, and use -pc_type type to set the preconditioner you want to use. </span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"> Barry</span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<br class="">
<br class="">
<br class="">
<br class="">
<br class="">
<br class="">
<br class="">
<br class="">
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<blockquote class="" style="margin-top:5pt; margin-bottom:5pt">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
On Aug 16, 2022, at 12:31 PM, Tu, Jiannan <<a href="mailto:Jiannan_Tu@uml.edu" class="" style="color:purple; text-decoration:underline">Jiannan_Tu@uml.edu</a>> wrote:</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt">Barry,</span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"> </span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt">Thank you very much for your instructions. I am sorry I may not ask clearer questions.</span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"> </span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt">I don't use SNES. What I am trying to solve is a very large linear equation system with dense and ill-conditioned matrix. Selected parts of the matrix can be pre-calculated and stored to serve as the preconditioner. My
question is how I can apply this matrix as the preconditioner.<span class="x_x_apple-converted-space"> </span></span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"> </span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt">Here is the part of code from my linear equations solver. MatrixFreePreconditioner is supposed to be the routine to provide the preconditioner. But I don't understand how to use it. Should I use SNESSetJacobian() even if
this is a linear problem?<span class="x_x_apple-converted-space"> </span></span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"> </span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"> MatCreateMFFD(MPI_COMM_WORLD, PETSC_DECIDE, PETSC_DECIDE, N, N, &A);</span></div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"> MatMFFDSetFunction(A, formfunction, ¶ms);</span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"> MatSetFromOptions(A);</span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"> MatMFFDSetBase(A, X, NULL);</span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"> </span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"> KSPCreate(MPI_COMM_WORLD, &ksp);</span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"> KSPSetOperators(ksp, A, A);</span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"> KSPSetFromOptions(ksp);</span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"> </span></div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"> PC pc;</span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"> KSPGetPC(ksp,&pc);</span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"> PCSetType(pc,PCSHELL);</span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"> PCShellSetApply(pc,MatrixFreePreconditioner);</span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"> </span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt">Thank you,</span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt">Jiannan</span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"> </span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="x_x_MsoNormal" align="center" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif; text-align:center">
<hr size="2" width="1025" align="center" class="" style="width:768.8pt">
</div>
<div id="x_x_divRplyFwdMsg" class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<b class="">From:</b><span class="x_x_apple-converted-space"> </span>Barry Smith <<a href="mailto:bsmith@petsc.dev" class="" style="color:purple; text-decoration:underline">bsmith@petsc.dev</a>><br class="">
<b class="">Sent:</b><span class="x_x_apple-converted-space"> </span>Tuesday, August 16, 2022 10:10 AM<br class="">
<b class="">To:</b><span class="x_x_apple-converted-space"> </span>Tu, Jiannan <<a href="mailto:Jiannan_Tu@uml.edu" class="" style="color:purple; text-decoration:underline">Jiannan_Tu@uml.edu</a>><br class="">
<b class="">Cc:</b><span class="x_x_apple-converted-space"> </span><a href="mailto:petsc-users@mcs.anl.gov" class="" style="color:purple; text-decoration:underline">petsc-users@mcs.anl.gov</a><span class="x_x_apple-converted-space"> </span><<a href="mailto:petsc-users@mcs.anl.gov" class="" style="color:purple; text-decoration:underline">petsc-users@mcs.anl.gov</a>><br class="">
<b class="">Subject:</b><span class="x_x_apple-converted-space"> </span>Re: [petsc-users] Using matrix-free with KSP</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:13.5pt; font-family:Helvetica,sans-serif"> </span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif; line-height:16pt; background-color:rgb(255,202,202)">
<b class=""><span class="" style="font-size:12pt">CAUTION:</span></b><span class="x_x_apple-converted-space"><span class="" style="font-size:12pt"> </span></span><span class="" style="font-size:12pt">This email was sent from outside the UMass Lowell network.</span></div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:13.5pt; font-family:Helvetica,sans-serif"> </span></div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:13.5pt; font-family:Helvetica,sans-serif"> </span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:13.5pt; font-family:Helvetica,sans-serif"> I don't fully understand your question so I will answer questions that I do know the answer to :-)</span></div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:13.5pt; font-family:Helvetica,sans-serif"> </span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:13.5pt; font-family:Helvetica,sans-serif"> The function you provide to PCShellSetApply() applies the preconditioner to the input vector and puts the result in the output vector. This input vector changes at every application
of the preconditioner so should not be used in constructing the preconditioner.</span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:13.5pt; font-family:Helvetica,sans-serif"> </span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:13.5pt; font-family:Helvetica,sans-serif"> If you are using the standard PETSc matrix-free matrix-vector product via finite difference, -snes_mf_operator or via MatCreateSNESMF() or you are using a MatShell, the matrix-vector
product routines input changes for each multiply and should not be used in constructing the preconditioner.</span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:13.5pt; font-family:Helvetica,sans-serif"> </span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:13.5pt; font-family:Helvetica,sans-serif"> The function you provide with SNESSetJacobian() is where you can compute anything you need to help build your preconditioner. The input vector to that function is the location at which
the Jacobian is needed and is what you should use to build your approximate/part of Jacobian. You can store your approximate/part of Jacobian that you need to compute in the pmat and then in a PCShellSetUp() function that you provide you will compute what
you will need latter to apply the preconditioner in your PCShellSetApply() function. Note that the values you put in the pmat do not need to be the full Jacobian; they can be anything you want since this pmat is not used to apply the matrix vector product.</span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:13.5pt; font-family:Helvetica,sans-serif"> </span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:13.5pt; font-family:Helvetica,sans-serif"> If this is unclear, feel free to ask additional questions.</span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:13.5pt; font-family:Helvetica,sans-serif"> </span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:13.5pt; font-family:Helvetica,sans-serif"> Barry</span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:13.5pt; font-family:Helvetica,sans-serif"> </span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:13.5pt; font-family:Helvetica,sans-serif"><br class="">
<br class="">
<br class="">
<br class="">
<br class="">
<br class="">
<br class="">
<br class="">
</span></div>
</div>
</div>
</div>
</div>
</div>
</div>
<blockquote class="" style="margin-top:5pt; margin-bottom:5pt">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:13.5pt; font-family:Helvetica,sans-serif">On Aug 16, 2022, at 9:40 AM, Tu, Jiannan <</span><a href="mailto:Jiannan_Tu@uml.edu" class="" style="color:purple; text-decoration:underline"><span class="" style="font-size:13.5pt; font-family:Helvetica,sans-serif">Jiannan_Tu@uml.edu</span></a><span class="" style="font-size:13.5pt; font-family:Helvetica,sans-serif">>
wrote:</span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:13.5pt; font-family:Helvetica,sans-serif"> </span></div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt">I am trying to apply a user-defined preconditioner to speed up the convergency of matrix-free KSP solver. The user's manual provides an example of how to call the user-defined preconditioner routine. The routine has PC,
input Vec and output Vec as arguments. I want to use part of the Jacobian as preconditioner matrix, the elements of which can be calculated using input Vec. My question is what the role of output Vec from the routine is in matrix-vector product? Or I just
have the preconditioner matrix elements calculated in the routine and then Petsc handles applying the preconditioner to matrix-vector product? I used PCSetType(pc, PCSHELL) and PCShellSetApply.</span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"> </span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt">Thank you and appreciate your help!</span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"> </span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt">Jiannan<span class="x_x_xapple-converted-space"> </span></span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="x_x_MsoNormal" align="center" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif; text-align:center">
<span class="" style="font-size:13.5pt; font-family:Helvetica,sans-serif">
<hr size="2" width="1025" align="center" class="" style="width:768.8pt">
</span></div>
<div id="x_x_x_divRplyFwdMsg" class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<b class="">From:</b><span class="x_x_xapple-converted-space"> </span>Tu, Jiannan <<a href="mailto:Jiannan_Tu@uml.edu" class="" style="color:purple; text-decoration:underline">Jiannan_Tu@uml.edu</a>><br class="">
<b class="">Sent:</b><span class="x_x_xapple-converted-space"> </span>Wednesday, July 6, 2022 2:34 PM<br class="">
<b class="">To:</b><span class="x_x_xapple-converted-space"> </span>Barry Smith <<a href="mailto:bsmith@petsc.dev" class="" style="color:purple; text-decoration:underline">bsmith@petsc.dev</a>><br class="">
<b class="">Cc:</b><span class="x_x_xapple-converted-space"> </span>Jed Brown <<a href="mailto:jed@jedbrown.org" class="" style="color:purple; text-decoration:underline">jed@jedbrown.org</a>>;<span class="x_x_xapple-converted-space"> </span><a href="mailto:petsc-users@mcs.anl.gov" class="" style="color:purple; text-decoration:underline">petsc-users@mcs.anl.gov</a><span class="x_x_xapple-converted-space"> </span><<a href="mailto:petsc-users@mcs.anl.gov" class="" style="color:purple; text-decoration:underline">petsc-users@mcs.anl.gov</a>><br class="">
<b class="">Subject:</b><span class="x_x_xapple-converted-space"> </span>Re: [petsc-users] Using matrix-free with KSP</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:13.5pt; font-family:Helvetica,sans-serif"> </span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt">Barry,</span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"> </span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt">VecScatterCreateToAll solves the problem! The code now gives the correct answer with multiple processes.<span class="x_x_xapple-converted-space"> </span></span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"> </span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt">Thank you all so much!</span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"> </span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt">Jiannan</span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="x_x_MsoNormal" align="center" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif; text-align:center">
<span class="" style="font-size:13.5pt; font-family:Helvetica,sans-serif">
<hr size="2" width="1025" align="center" class="" style="width:768.8pt">
</span></div>
<div id="x_x_x_x_divRplyFwdMsg" class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<b class="">From:</b><span class="x_x_xapple-converted-space"> </span>Barry Smith <<a href="mailto:bsmith@petsc.dev" class="" style="color:purple; text-decoration:underline">bsmith@petsc.dev</a>><br class="">
<b class="">Sent:</b><span class="x_x_xapple-converted-space"> </span>Wednesday, July 6, 2022 2:08 PM<br class="">
<b class="">To:</b><span class="x_x_xapple-converted-space"> </span>Tu, Jiannan <<a href="mailto:Jiannan_Tu@uml.edu" class="" style="color:purple; text-decoration:underline">Jiannan_Tu@uml.edu</a>><br class="">
<b class="">Cc:</b><span class="x_x_xapple-converted-space"> </span>Jed Brown <<a href="mailto:jed@jedbrown.org" class="" style="color:purple; text-decoration:underline">jed@jedbrown.org</a>>;<span class="x_x_xapple-converted-space"> </span><a href="mailto:petsc-users@mcs.anl.gov" class="" style="color:purple; text-decoration:underline">petsc-users@mcs.anl.gov</a><span class="x_x_xapple-converted-space"> </span><<a href="mailto:petsc-users@mcs.anl.gov" class="" style="color:purple; text-decoration:underline">petsc-users@mcs.anl.gov</a>><br class="">
<b class="">Subject:</b><span class="x_x_xapple-converted-space"> </span>Re: [petsc-users] Using matrix-free with KSP</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:13.5pt; font-family:Helvetica,sans-serif"> </span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:13.5pt; font-family:Helvetica,sans-serif"> </span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:13.5pt; font-family:Helvetica,sans-serif"> So your operator is a "dense" operator in that its matrix representation would be essentially dense. In that case, you can use specialized routines to do this efficiently. You can use VecScatterCreateToAll()
and then VecScatterBegin/End to do the communication.</span></div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:13.5pt; font-family:Helvetica,sans-serif"> </span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:13.5pt; font-family:Helvetica,sans-serif"> Barry</span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:13.5pt; font-family:Helvetica,sans-serif"> </span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:13.5pt; font-family:Helvetica,sans-serif"> </span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:13.5pt; font-family:Helvetica,sans-serif"><br class="">
<br class="">
<br class="">
<br class="">
<br class="">
<br class="">
<br class="">
<br class="">
</span></div>
</div>
</div>
</div>
</div>
</div>
</div>
<blockquote class="" style="margin-top:5pt; margin-bottom:5pt">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:13.5pt; font-family:Helvetica,sans-serif">On Jul 6, 2022, at 1:39 PM, Tu, Jiannan <</span><a href="mailto:Jiannan_Tu@uml.edu" class="" style="color:purple; text-decoration:underline"><span class="" style="font-size:13.5pt; font-family:Helvetica,sans-serif">Jiannan_Tu@uml.edu</span></a><span class="" style="font-size:13.5pt; font-family:Helvetica,sans-serif">><span class="x_Apple-converted-space"> </span>
wrote:</span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:13.5pt; font-family:Helvetica,sans-serif"> </span></div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt">Jed, thank you very much for the reply.<span class="x_x_xxxapple-converted-space"> </span></span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"> </span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt">I'm not sure GlobaltoLocal will work. Say the solution vector is of length 10. Two processes then each process can see 5 elements of the vector. For A x = b (10 equations), each process performs matrix-vector product for
five equations. And for each equation i , sum_j A_ij * x_j = b_i requires the process has access to all 10 unknows x_i (i=0,1,...,9).<span class="x_x_xxxapple-converted-space"> </span></span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"> </span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt">Can VecScatter and PetscSF make the entire vector accessible to each process? I am reading the manual and trying to understand how VecScatter and PetscSF work.</span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt"> </span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:12pt">Jiannan</span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="x_x_MsoNormal" align="center" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif; text-align:center">
<span class="" style="font-size:13.5pt; font-family:Helvetica,sans-serif">
<hr size="2" width="1262" align="center" class="" style="width:946.7pt">
</span></div>
<div id="x_x_x_x_x_divRplyFwdMsg" class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<b class="">From:</b><span class="x_x_xxxapple-converted-space"> </span>Jed Brown <<a href="mailto:jed@jedbrown.org" class="" style="color:purple; text-decoration:underline">jed@jedbrown.org</a>><br class="">
<b class="">Sent:</b><span class="x_x_xxxapple-converted-space"> </span>Wednesday, July 6, 2022 10:09 AM<br class="">
<b class="">To:</b><span class="x_x_xxxapple-converted-space"> </span>Tu, Jiannan <<a href="mailto:Jiannan_Tu@uml.edu" class="" style="color:purple; text-decoration:underline">Jiannan_Tu@uml.edu</a>>; Barry Smith <<a href="mailto:bsmith@petsc.dev" class="" style="color:purple; text-decoration:underline">bsmith@petsc.dev</a>><br class="">
<b class="">Cc:</b><span class="x_x_xxxapple-converted-space"> </span><a href="mailto:petsc-users@mcs.anl.gov" class="" style="color:purple; text-decoration:underline">petsc-users@mcs.anl.gov</a><span class="x_x_xxxapple-converted-space"> </span><<a href="mailto:petsc-users@mcs.anl.gov" class="" style="color:purple; text-decoration:underline">petsc-users@mcs.anl.gov</a>><br class="">
<b class="">Subject:</b><span class="x_x_xxxapple-converted-space"> </span>Re: [petsc-users] Using matrix-free with KSP</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:13.5pt; font-family:Helvetica,sans-serif"> </span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-family:Helvetica,sans-serif">You'll usually have a GlobalToLocal operation for each rank to get the halo data it needs, then either a LocalToGlobal to collect the result (e.g., finite element methods) or the local compute will write
directly into the owned portion of the global vector. If you're doing the communication "raw", then you may find VecScatter or PetscSF useful to perform the necessary communication.<br class="">
<br class="">
"Tu, Jiannan" <</span><a href="mailto:Jiannan_Tu@uml.edu" class="" style="color:purple; text-decoration:underline"><span class="" style="font-family:Helvetica,sans-serif">Jiannan_Tu@uml.edu</span></a><span class="" style="font-family:Helvetica,sans-serif">>
writes:<br class="">
<br class="">
> Hi Barry,<br class="">
><br class="">
> Following your instructions I implemented the matrix-free method to solve a large linear equation system resulted from a surface integration equation. The KSP solver works fine for single process, but it is problematic with multiple processes. The problem
is that each process only can access its own part of solution vector so that each process only conducts part of matrix-vector multiplication. MPI can be used to assemble these partial matrix-vector multiplication together. Does Petsc provide any ways to implement
multi-process matrix-free method?<br class="">
><br class="">
> Thanks,<br class="">
> Jiannan<br class="">
> ________________________________<br class="">
> From: Barry Smith <</span><a href="mailto:bsmith@petsc.dev" class="" style="color:purple; text-decoration:underline"><span class="" style="font-family:Helvetica,sans-serif">bsmith@petsc.dev</span></a><span class="" style="font-family:Helvetica,sans-serif">><br class="">
> Sent: Tuesday, May 24, 2022 2:12 PM<br class="">
> To: Tu, Jiannan <</span><a href="mailto:Jiannan_Tu@uml.edu" class="" style="color:purple; text-decoration:underline"><span class="" style="font-family:Helvetica,sans-serif">Jiannan_Tu@uml.edu</span></a><span class="" style="font-family:Helvetica,sans-serif">><br class="">
> Cc:<span class="x_x_xxxapple-converted-space"> </span></span><a href="mailto:petsc-users@mcs.anl.gov" class="" style="color:purple; text-decoration:underline"><span class="" style="font-family:Helvetica,sans-serif">petsc-users@mcs.anl.gov</span></a><span class="x_x_xxxapple-converted-space"><span class="" style="font-family:Helvetica,sans-serif"> </span></span><span class="" style="font-family:Helvetica,sans-serif"><</span><a href="mailto:petsc-users@mcs.anl.gov" class="" style="color:purple; text-decoration:underline"><span class="" style="font-family:Helvetica,sans-serif">petsc-users@mcs.anl.gov</span></a><span class="" style="font-family:Helvetica,sans-serif">><br class="">
> Subject: Re: [petsc-users] Using matrix-free with KSP<br class="">
><br class="">
> This e-mail originated from outside the UMass Lowell network.<br class="">
> ________________________________<br class="">
><br class="">
> You can use MatCreateMFFD<span class="x_x_xxxapple-converted-space"> </span></span><a href="https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMatCreateMFFD%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7Cc8226b4a8af040e8ca0208da845d5e8b%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637967833829887546%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=pQ3eVZvOw0WEJdHYm%2FP3leoX3tBNKpgLDQo5Xm10akw%3D&reserved=0" originalsrc="https://petsc.org/main/docs/manualpages/Mat/MatCreateMFFD/" shash="bitJXsYw7R1snElYbYwMNp91h7DxUeh+BqAnq+Zct/44Mh0j4IN7QZTPy6lJBEkX7VtwavJZOF31gmgDbee6kKq6ZyTGxE6XlmsnBbWQ2vGH5zp5MIj3IYISQ3E3piUwjqsi47+bvl7K1Gn9TTnhW7LQILz1iBtY+jWk3P9/axg=" originalsrc="https://petsc.org/main/docs/manualpages/Mat/MatCreateMFFD/" shash="hQ0qceTENqR4QH1ERl/EmpJSwhb2NaL/uHAchowVcO2EPvCxQa6lKxijS7vFwd9rsQo3/C4Nms91dMcZh/KapFbXwxCpc7YgV3sgIXg2eOl7Nde9+42f2Si03l27qolVj8XvEPQ+urCDOBTsHchFCgdCmO68J7doJQ1JU7zmpt0=" class="" style="color:purple; text-decoration:underline"><span class="" style="font-family:Helvetica,sans-serif">https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMatCreateMFFD%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7Cab0859d8d154471590bc08da5f5918d6%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637927133525745809%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=A7wqegTfh94No5BpDiWLK3VxOuR44U2wlWHVm2k7l60%3D&reserved=0<https://urldefense.com/v3/__https://petsc.org/main/docs/manualpages/Mat/MatCreateMFFD/__;!!PVKG_VDCxu5g!stxq2NWGVjUihvHBQC9Rlk0aHVoowcy0PTQYMcQxQ4DqOEC05KSw6TmstSXKckiUgelHzy4ue-d10-zlXkw$></span></a><span class="x_x_xxxapple-converted-space"><span class="" style="font-family:Helvetica,sans-serif"> </span></span><span class="" style="font-family:Helvetica,sans-serif">MatMFFDSetFunction
MatSetFromOptions MatMFFDSetBase and provide the matrix to KSP. Note you will need to use -pc_type none or provide your own custom preconditioner with</span><a href="https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FPC%2FPCSHELL%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7Cc8226b4a8af040e8ca0208da845d5e8b%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637967833829887546%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=OwryOVq7EBof9oYaYJmN3klaLNwxtAxcZBgj44BF%2BJo%3D&reserved=0" originalsrc="https://petsc.org/main/docs/manualpages/PC/PCSHELL/" shash="TlJAaWNfHqt34EUijZ6wK42gDhfFNztxCAFNzyngMlZlb6bftxMU4vWsHarSa+a2pxNTzZQAodRjvVS8LQ64iA93HK2CHeU/0141HVPu7Qes3hIrzI/QSJ7ly5rX5Q6ZoEvqDnMl4ggVJEs5oa99+ybUXBu6d2i1/CTv4i4LLvo=" originalsrc="https://petsc.org/main/docs/manualpages/PC/PCSHELL/" shash="lQ9ELiY59spuoqN0NWftq9B1jYkk6T882YuwQr2dwd1fD/wGMP91cQJ+NxltQyxeZrhPeQfVYfoPpRFSGXCmLIoy97czH8U/TjIN6D5+drlrvA7VdYmzeHWeW0JQEU8Scz4UYsZJ2d2yfFnHBU8vpWUXDFSGzK9+PGfQbXJnfZ8=" class="" style="color:purple; text-decoration:underline"><span class="" style="font-family:Helvetica,sans-serif">https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FPC%2FPCSHELL%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7Cab0859d8d154471590bc08da5f5918d6%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637927133525745809%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Gb%2F9uFG3jp%2FbfWaSh2cSEQLItHS9CuLwarzN0KcNiJY%3D&reserved=0<https://urldefense.com/v3/__https://petsc.org/main/docs/manualpages/PC/PCSHELL/__;!!PVKG_VDCxu5g!stxq2NWGVjUihvHBQC9Rlk0aHVoowcy0PTQYMcQxQ4DqOEC05KSw6TmstSXKckiUgelHzy4ue-d1cA4fKGw$></span></a><span class="" style="font-family:Helvetica,sans-serif"><br class="">
><br class="">
><br class="">
><br class="">
> On May 24, 2022, at 1:21 PM, Tu, Jiannan <</span><a href="mailto:Jiannan_Tu@uml.edu" class="" style="color:purple; text-decoration:underline"><span class="" style="font-family:Helvetica,sans-serif">Jiannan_Tu@uml.edu</span></a><span class="" style="font-family:Helvetica,sans-serif"><</span><a href="mailto:Jiannan_Tu@uml.edu" class="" style="color:purple; text-decoration:underline"><span class="" style="font-family:Helvetica,sans-serif">mailto:Jiannan_Tu@uml.edu</span></a><span class="" style="font-family:Helvetica,sans-serif">>>
wrote:<br class="">
><br class="">
> I want to use a matrix-free matrix to solve a large linear equation system because the matrix is too large to be stored. Petsc user manual describes matrix-free method for SNES with examples. The matrix-free matrices section explains how to set up such a
matrix, but I can't find any example of matrix-free method with KSP. I am wondering how to apply the method to KSP iterative solver in parallel.<br class="">
><br class="">
> I greatly appreciate your help for me to understand this topic.<br class="">
><br class="">
> Jiannan Tu</span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</body>
</html>