<html><head><meta http-equiv="Content-Type" content="text/html; charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class=""><br class=""></div> Slow assembly is usually due to under preallocation. You can change to using <div class=""><br class=""></div><div class=""><blockquote type="cite" class=""><div class="WordSection1" style="page: WordSection1;"><div class="" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;"><span class="" style="font-size: 12pt;">MatSetOption(B, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_TRUE);</span></div></div></blockquote><div class=""><br class=""></div><div class=""> to detect if you are under preallocating. See <a href="https://petsc.org/main/docs/manual/mat/#sec-matsparse" class="">https://petsc.org/main/docs/manual/mat/#sec-matsparse</a>. That section clearly needs some sprucing up. If it is difficult to determine good preallocation values you can use <a href="https://petsc.org/main/docs/manualpages/Mat/MATPREALLOCATOR/" class="">https://petsc.org/main/docs/manualpages/Mat/MATPREALLOCATOR/</a> to compute the needed preallocation efficiently.</div><div class=""><br class=""></div><div class=""> Are you running in parallel? If so how are you determining which rows of entries to compute on each MPI rank? You will want most of the rows to be computed on the rank where the values are stored, you can determine this with <a href="https://petsc.org/main/docs/manualpages/Mat/MatGetOwnershipRange.html" class="">https://petsc.org/main/docs/manualpages/Mat/MatGetOwnershipRange.html</a></div><div class=""><br class=""></div><div class=""> Barry</div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><font color="#333333" face="-apple-system, BlinkMacSystemFont, Segoe UI, Helvetica Neue, Arial, sans-serif, Apple Color Emoji, Segoe UI Emoji, Segoe UI Symbol" class=""><span style="caret-color: rgb(51, 51, 51);" class=""><br class=""></span></font><div class=""><br class=""></div><div><br class=""><blockquote type="cite" class=""><div class="">On Aug 18, 2022, at 12:50 AM, Tu, Jiannan <<a href="mailto:Jiannan_Tu@uml.edu" class="">Jiannan_Tu@uml.edu</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><meta charset="UTF-8" class=""><div class="WordSection1" style="page: WordSection1; caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt;" class="">I implemented the preconditioner. The solver converges quickly when the problem size is small. But when the size increases, say to 100k unknowns, the code hangs at assembly of the preconditioner matrix. The function call sequence is like this<o:p class=""></o:p></span></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt;" class=""><o:p class=""> </o:p></span></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt;" class=""> MatCreate(MPI_COMM_WORLD, &B);<o:p class=""></o:p></span></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt;" class=""> MatSetType(B, MATMPIAIJ);<o:p class=""></o:p></span></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt;" class=""> MatSetSizes(B, PETSC_DECIDE, PETSC_DECIDE, N, N);<o:p class=""></o:p></span></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt;" class=""> MatSetFromOptions(B);<o:p class=""></o:p></span></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt;" class=""><o:p class=""> </o:p></span></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt;" class=""> // The number of diagonal and off diagonal non zeros only can be determined run-time.<o:p class=""></o:p></span></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt;" class=""> MatMPIAIJSetPreallocation(B, 0, d_nnz, 0, o_nnz);<o:p class=""></o:p></span></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt;" class=""> MatSetOption(B, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE);<o:p class=""></o:p></span></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt;" class=""><o:p class=""> </o:p></span></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt;" class=""> //here insert values row by row.<o:p class=""></o:p></span></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif; text-indent: 10.5pt;" class=""><span style="font-size: 12pt;" class="">MatSetValues(B, 1, &m, nn, nCols, matCols, INSERT_VALUES);<o:p class=""></o:p></span></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif; text-indent: 10.5pt;" class=""><span style="font-size: 12pt;" class=""><o:p class=""> </o:p></span></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif; text-indent: 10.5pt;" class=""><span style="font-size: 12pt;" class="">MatAssemblyBegin(B, MAT_FINAL_ASSEMBLY);<o:p class=""></o:p></span></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif; text-indent: 10.5pt;" class=""><span style="font-size: 12pt;" class="">MatAssemblyEnd(B, MAT_FINAL_ASSEMBLY);<o:p class=""></o:p></span></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif; text-indent: 10.5pt;" class=""><span style="font-size: 12pt;" class=""><o:p class=""> </o:p></span></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif; text-indent: 10.5pt;" class=""><span style="font-size: 12pt;" class="">Could you please tell me what I have done wrong?<o:p class=""></o:p></span></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif; text-indent: 10.5pt;" class=""><span style="font-size: 12pt;" class=""><o:p class=""> </o:p></span></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif; text-indent: 10.5pt;" class=""><span style="font-size: 12pt;" class="">Thank you,<o:p class=""></o:p></span></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif; text-indent: 10.5pt;" class=""><span style="font-size: 12pt;" class="">Jiannan<o:p class=""></o:p></span></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt;" class=""> <span class="Apple-converted-space"> </span><o:p class=""></o:p></span></div><div class=""><div style="border-style: solid none none; border-top-width: 1pt; border-top-color: rgb(225, 225, 225); padding: 3pt 0in 0in;" class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><b class="">From:</b><span class="Apple-converted-space"> </span>Barry Smith <<a href="mailto:bsmith@petsc.dev" style="color: blue; text-decoration: underline;" class="">bsmith@petsc.dev</a>><span class="Apple-converted-space"> </span><br class=""><b class="">Sent:</b><span class="Apple-converted-space"> </span>Tuesday, August 16, 2022 2:00 PM<br class=""><b class="">To:</b><span class="Apple-converted-space"> </span>Tu, Jiannan <<a href="mailto:Jiannan_Tu@uml.edu" style="color: blue; text-decoration: underline;" class="">Jiannan_Tu@uml.edu</a>><br class=""><b class="">Cc:</b><span class="Apple-converted-space"> </span><a href="mailto:petsc-users@mcs.anl.gov" style="color: blue; text-decoration: underline;" class="">petsc-users@mcs.anl.gov</a><br class=""><b class="">Subject:</b><span class="Apple-converted-space"> </span>Re: [petsc-users] Using matrix-free with KSP<o:p class=""></o:p></div></div></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div><div style="border: 1pt hidden rgb(255, 202, 202); padding: 2pt;" class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif; line-height: 16pt; background-color: rgb(255, 202, 202);" class=""><b class=""><span style="font-size: 12pt;" class="">CAUTION:</span></b><span style="font-size: 12pt;" class=""><span class="Apple-converted-space"> </span>This email was sent from outside the UMass Lowell network.<o:p class=""></o:p></span></div></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> Create another matrix B that contains "<span style="font-size: 12pt;" class="">Selected parts of the matrix can be pre-calculated and stored to serve as the preconditioner." When you call KSPSetOperators() pass in B as the second matrix argument.</span><o:p class=""></o:p></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt;" class=""> Now if literally application of B is the preconditioner you want to use, then simply use PCMAT as the preconditioner, and it will apply B each time it needs to apply the preconditioner. With this, you do not even need a PCShell. </span><o:p class=""></o:p></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt;" class=""> If your preconditioner uses B to construct your own specific data structure needed to apply your preconditioner, then you can use PCShellSetSetUp() and have your routine call PCGetOperators() to pull out the B matrix and use it to construct your preconditioner. </span><o:p class=""></o:p></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt;" class=""> If you want PETSc to construct a standard preconditioner from B, then you don't need the PCShell; you simply pass in the B, as above, and use -pc_type type to set the preconditioner you want to use. </span><o:p class=""></o:p></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt;" class=""> Barry</span><o:p class=""></o:p></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><br class=""><br class=""><o:p class=""></o:p></div><blockquote style="margin-top: 5pt; margin-bottom: 5pt;" class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">On Aug 16, 2022, at 12:31 PM, Tu, Jiannan <<a href="mailto:Jiannan_Tu@uml.edu" style="color: blue; text-decoration: underline;" class="">Jiannan_Tu@uml.edu</a>> wrote:<o:p class=""></o:p></div></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt;" class="">Barry,<o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt;" class=""><o:p class=""> </o:p></span></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt;" class="">Thank you very much for your instructions. I am sorry I may not ask clearer questions.<o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt;" class=""><o:p class=""> </o:p></span></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt;" class="">I don't use SNES. What I am trying to solve is a very large linear equation system with dense and ill-conditioned matrix. Selected parts of the matrix can be pre-calculated and stored to serve as the preconditioner. My question is how I can apply this matrix as the preconditioner.<span class="apple-converted-space"> </span><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt;" class=""><o:p class=""> </o:p></span></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt;" class="">Here is the part of code from my linear equations solver. MatrixFreePreconditioner is supposed to be the routine to provide the preconditioner. But I don't understand how to use it. Should I use SNESSetJacobian() even if this is a linear problem?<span class="apple-converted-space"> </span><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt;" class=""><o:p class=""> </o:p></span></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt;" class=""> MatCreateMFFD(MPI_COMM_WORLD, PETSC_DECIDE, PETSC_DECIDE, N, N, &A);<o:p class=""></o:p></span></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt;" class=""> MatMFFDSetFunction(A, formfunction, ¶ms);<o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt;" class=""> MatSetFromOptions(A);<o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt;" class=""> MatMFFDSetBase(A, X, NULL);<o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt;" class=""><o:p class=""> </o:p></span></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt;" class=""> KSPCreate(MPI_COMM_WORLD, &ksp);<o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt;" class=""> KSPSetOperators(ksp, A, A);<o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt;" class=""> KSPSetFromOptions(ksp);<o:p class=""></o:p></span></div></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt;" class=""><o:p class=""> </o:p></span></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt;" class=""> PC pc;<o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt;" class=""> KSPGetPC(ksp,&pc);<o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt;" class=""> PCSetType(pc,PCSHELL);<o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt;" class=""> PCShellSetApply(pc,MatrixFreePreconditioner);<o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt;" class=""><o:p class=""> </o:p></span></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt;" class="">Thank you,<o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt;" class="">Jiannan<o:p class=""></o:p></span></div></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt;" class=""><o:p class=""> </o:p></span></div></div><div class="MsoNormal" align="center" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif; text-align: center;"><hr size="2" width="1025" align="center" style="width: 768.8pt;" class=""></div><div id="divRplyFwdMsg" class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><b class="">From:</b><span class="apple-converted-space"> </span>Barry Smith <<a href="mailto:bsmith@petsc.dev" style="color: blue; text-decoration: underline;" class="">bsmith@petsc.dev</a>><br class=""><b class="">Sent:</b><span class="apple-converted-space"> </span>Tuesday, August 16, 2022 10:10 AM<br class=""><b class="">To:</b><span class="apple-converted-space"> </span>Tu, Jiannan <<a href="mailto:Jiannan_Tu@uml.edu" style="color: blue; text-decoration: underline;" class="">Jiannan_Tu@uml.edu</a>><br class=""><b class="">Cc:</b><span class="apple-converted-space"> </span><a href="mailto:petsc-users@mcs.anl.gov" style="color: blue; text-decoration: underline;" class="">petsc-users@mcs.anl.gov</a><span class="apple-converted-space"> </span><<a href="mailto:petsc-users@mcs.anl.gov" style="color: blue; text-decoration: underline;" class="">petsc-users@mcs.anl.gov</a>><br class=""><b class="">Subject:</b><span class="apple-converted-space"> </span>Re: [petsc-users] Using matrix-free with KSP<span style="font-size: 13.5pt; font-family: Helvetica, sans-serif;" class=""><o:p class=""></o:p></span></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 13.5pt; font-family: Helvetica, sans-serif;" class=""> <o:p class=""></o:p></span></div></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif; line-height: 16pt; background-color: rgb(255, 202, 202);" class=""><b class=""><span style="font-size: 12pt;" class="">CAUTION:</span></b><span class="apple-converted-space"><span style="font-size: 12pt;" class=""> </span></span><span style="font-size: 12pt;" class="">This email was sent from outside the UMass Lowell network.</span><span style="font-size: 12pt;" class=""><o:p class=""></o:p></span></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 13.5pt; font-family: Helvetica, sans-serif;" class=""><o:p class=""> </o:p></span></div><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 13.5pt; font-family: Helvetica, sans-serif;" class=""><o:p class=""> </o:p></span></div></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 13.5pt; font-family: Helvetica, sans-serif;" class=""> I don't fully understand your question so I will answer questions that I do know the answer to :-)<o:p class=""></o:p></span></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 13.5pt; font-family: Helvetica, sans-serif;" class=""><o:p class=""> </o:p></span></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 13.5pt; font-family: Helvetica, sans-serif;" class=""> The function you provide to PCShellSetApply() applies the preconditioner to the input vector and puts the result in the output vector. This input vector changes at every application of the preconditioner so should not be used in constructing the preconditioner.<o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 13.5pt; font-family: Helvetica, sans-serif;" class=""><o:p class=""> </o:p></span></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 13.5pt; font-family: Helvetica, sans-serif;" class=""> If you are using the standard PETSc matrix-free matrix-vector product via finite difference, -snes_mf_operator or via MatCreateSNESMF() or you are using a MatShell, the matrix-vector product routines input changes for each multiply and should not be used in constructing the preconditioner.<o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 13.5pt; font-family: Helvetica, sans-serif;" class=""><o:p class=""> </o:p></span></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 13.5pt; font-family: Helvetica, sans-serif;" class=""> The function you provide with SNESSetJacobian() is where you can compute anything you need to help build your preconditioner. The input vector to that function is the location at which the Jacobian is needed and is what you should use to build your approximate/part of Jacobian. You can store your approximate/part of Jacobian that you need to compute in the pmat and then in a PCShellSetUp() function that you provide you will compute what you will need latter to apply the preconditioner in your PCShellSetApply() function. Note that the values you put in the pmat do not need to be the full Jacobian; they can be anything you want since this pmat is not used to apply the matrix vector product.<o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 13.5pt; font-family: Helvetica, sans-serif;" class=""><o:p class=""> </o:p></span></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 13.5pt; font-family: Helvetica, sans-serif;" class=""> If this is unclear, feel free to ask additional questions.<o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 13.5pt; font-family: Helvetica, sans-serif;" class=""><o:p class=""> </o:p></span></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 13.5pt; font-family: Helvetica, sans-serif;" class=""> Barry<o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 13.5pt; font-family: Helvetica, sans-serif;" class=""><o:p class=""> </o:p></span></div></div><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 13.5pt; font-family: Helvetica, sans-serif;" class=""><br class=""><br class=""><o:p class=""></o:p></span></div><blockquote style="margin-top: 5pt; margin-bottom: 5pt;" class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 13.5pt; font-family: Helvetica, sans-serif;" class="">On Aug 16, 2022, at 9:40 AM, Tu, Jiannan <<a href="mailto:Jiannan_Tu@uml.edu" style="color: blue; text-decoration: underline;" class="">Jiannan_Tu@uml.edu</a>> wrote:<o:p class=""></o:p></span></div></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 13.5pt; font-family: Helvetica, sans-serif;" class=""><o:p class=""> </o:p></span></div><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt;" class="">I am trying to apply a user-defined preconditioner to speed up the convergency of matrix-free KSP solver. The user's manual provides an example of how to call the user-defined preconditioner routine. The routine has PC, input Vec and output Vec as arguments. I want to use part of the Jacobian as preconditioner matrix, the elements of which can be calculated using input Vec. My question is what the role of output Vec from the routine is in matrix-vector product? Or I just have the preconditioner matrix elements calculated in the routine and then Petsc handles applying the preconditioner to matrix-vector product? I used PCSetType(pc, PCSHELL) and PCShellSetApply.<o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt;" class=""><o:p class=""> </o:p></span></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt;" class="">Thank you and appreciate your help!<o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt;" class=""><o:p class=""> </o:p></span></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt;" class="">Jiannan<span class="xapple-converted-space"> </span><o:p class=""></o:p></span></div></div><div class="MsoNormal" align="center" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif; text-align: center;"><span style="font-size: 13.5pt; font-family: Helvetica, sans-serif;" class=""><hr size="2" width="1025" align="center" style="width: 768.8pt;" class=""></span></div><div id="x_divRplyFwdMsg" class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><b class="">From:</b><span class="xapple-converted-space"> </span>Tu, Jiannan <<a href="mailto:Jiannan_Tu@uml.edu" style="color: blue; text-decoration: underline;" class="">Jiannan_Tu@uml.edu</a>><br class=""><b class="">Sent:</b><span class="xapple-converted-space"> </span>Wednesday, July 6, 2022 2:34 PM<br class=""><b class="">To:</b><span class="xapple-converted-space"> </span>Barry Smith <<a href="mailto:bsmith@petsc.dev" style="color: blue; text-decoration: underline;" class="">bsmith@petsc.dev</a>><br class=""><b class="">Cc:</b><span class="xapple-converted-space"> </span>Jed Brown <<a href="mailto:jed@jedbrown.org" style="color: blue; text-decoration: underline;" class="">jed@jedbrown.org</a>>;<span class="xapple-converted-space"> </span><a href="mailto:petsc-users@mcs.anl.gov" style="color: blue; text-decoration: underline;" class="">petsc-users@mcs.anl.gov</a><span class="xapple-converted-space"> </span><<a href="mailto:petsc-users@mcs.anl.gov" style="color: blue; text-decoration: underline;" class="">petsc-users@mcs.anl.gov</a>><br class=""><b class="">Subject:</b><span class="xapple-converted-space"> </span>Re: [petsc-users] Using matrix-free with KSP<span style="font-size: 13.5pt; font-family: Helvetica, sans-serif;" class=""><o:p class=""></o:p></span></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 13.5pt; font-family: Helvetica, sans-serif;" class=""> <o:p class=""></o:p></span></div></div></div><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt;" class="">Barry,<o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt;" class=""><o:p class=""> </o:p></span></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt;" class="">VecScatterCreateToAll solves the problem! The code now gives the correct answer with multiple processes.<span class="xapple-converted-space"> </span><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt;" class=""><o:p class=""> </o:p></span></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt;" class="">Thank you all so much!<o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt;" class=""><o:p class=""> </o:p></span></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt;" class="">Jiannan<o:p class=""></o:p></span></div></div><div class="MsoNormal" align="center" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif; text-align: center;"><span style="font-size: 13.5pt; font-family: Helvetica, sans-serif;" class=""><hr size="2" width="1025" align="center" style="width: 768.8pt;" class=""></span></div><div id="x_x_divRplyFwdMsg" class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><b class="">From:</b><span class="xapple-converted-space"> </span>Barry Smith <<a href="mailto:bsmith@petsc.dev" style="color: blue; text-decoration: underline;" class="">bsmith@petsc.dev</a>><br class=""><b class="">Sent:</b><span class="xapple-converted-space"> </span>Wednesday, July 6, 2022 2:08 PM<br class=""><b class="">To:</b><span class="xapple-converted-space"> </span>Tu, Jiannan <<a href="mailto:Jiannan_Tu@uml.edu" style="color: blue; text-decoration: underline;" class="">Jiannan_Tu@uml.edu</a>><br class=""><b class="">Cc:</b><span class="xapple-converted-space"> </span>Jed Brown <<a href="mailto:jed@jedbrown.org" style="color: blue; text-decoration: underline;" class="">jed@jedbrown.org</a>>;<span class="xapple-converted-space"> </span><a href="mailto:petsc-users@mcs.anl.gov" style="color: blue; text-decoration: underline;" class="">petsc-users@mcs.anl.gov</a><span class="xapple-converted-space"> </span><<a href="mailto:petsc-users@mcs.anl.gov" style="color: blue; text-decoration: underline;" class="">petsc-users@mcs.anl.gov</a>><br class=""><b class="">Subject:</b><span class="xapple-converted-space"> </span>Re: [petsc-users] Using matrix-free with KSP<span style="font-size: 13.5pt; font-family: Helvetica, sans-serif;" class=""><o:p class=""></o:p></span></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 13.5pt; font-family: Helvetica, sans-serif;" class=""> <o:p class=""></o:p></span></div></div></div><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 13.5pt; font-family: Helvetica, sans-serif;" class=""><o:p class=""> </o:p></span></div></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 13.5pt; font-family: Helvetica, sans-serif;" class=""> So your operator is a "dense" operator in that its matrix representation would be essentially dense. In that case, you can use specialized routines to do this efficiently. You can use VecScatterCreateToAll() and then VecScatterBegin/End to do the communication.<o:p class=""></o:p></span></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 13.5pt; font-family: Helvetica, sans-serif;" class=""><o:p class=""> </o:p></span></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 13.5pt; font-family: Helvetica, sans-serif;" class=""> Barry<o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 13.5pt; font-family: Helvetica, sans-serif;" class=""><o:p class=""> </o:p></span></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 13.5pt; font-family: Helvetica, sans-serif;" class=""><o:p class=""> </o:p></span></div></div><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 13.5pt; font-family: Helvetica, sans-serif;" class=""><br class=""><br class=""><o:p class=""></o:p></span></div><blockquote style="margin-top: 5pt; margin-bottom: 5pt;" class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 13.5pt; font-family: Helvetica, sans-serif;" class="">On Jul 6, 2022, at 1:39 PM, Tu, Jiannan <<a href="mailto:Jiannan_Tu@uml.edu" style="color: blue; text-decoration: underline;" class="">Jiannan_Tu@uml.edu</a>> wrote:<o:p class=""></o:p></span></div></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 13.5pt; font-family: Helvetica, sans-serif;" class=""><o:p class=""> </o:p></span></div><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt;" class="">Jed, thank you very much for the reply.<span class="xxxapple-converted-space"> </span><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt;" class=""><o:p class=""> </o:p></span></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt;" class="">I'm not sure GlobaltoLocal will work. Say the solution vector is of length 10. Two processes then each process can see 5 elements of the vector. For A x = b (10 equations), each process performs matrix-vector product for five equations. And for each equation i , sum_j A_ij * x_j = b_i requires the process has access to all 10 unknows x_i (i=0,1,...,9).<span class="xxxapple-converted-space"> </span><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt;" class=""><o:p class=""> </o:p></span></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt;" class="">Can VecScatter and PetscSF make the entire vector accessible to each process? I am reading the manual and trying to understand how VecScatter and PetscSF work.<o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt;" class=""><o:p class=""> </o:p></span></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt;" class="">Jiannan<o:p class=""></o:p></span></div></div><div class="MsoNormal" align="center" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif; text-align: center;"><span style="font-size: 13.5pt; font-family: Helvetica, sans-serif;" class=""><hr size="2" width="1262" align="center" style="width: 946.7pt;" class=""></span></div><div id="x_x_x_divRplyFwdMsg" class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><b class="">From:</b><span class="xxxapple-converted-space"> </span>Jed Brown <<a href="mailto:jed@jedbrown.org" style="color: blue; text-decoration: underline;" class="">jed@jedbrown.org</a>><br class=""><b class="">Sent:</b><span class="xxxapple-converted-space"> </span>Wednesday, July 6, 2022 10:09 AM<br class=""><b class="">To:</b><span class="xxxapple-converted-space"> </span>Tu, Jiannan <<a href="mailto:Jiannan_Tu@uml.edu" style="color: blue; text-decoration: underline;" class="">Jiannan_Tu@uml.edu</a>>; Barry Smith <<a href="mailto:bsmith@petsc.dev" style="color: blue; text-decoration: underline;" class="">bsmith@petsc.dev</a>><br class=""><b class="">Cc:</b><span class="xxxapple-converted-space"> </span><a href="mailto:petsc-users@mcs.anl.gov" style="color: blue; text-decoration: underline;" class="">petsc-users@mcs.anl.gov</a><span class="xxxapple-converted-space"> </span><<a href="mailto:petsc-users@mcs.anl.gov" style="color: blue; text-decoration: underline;" class="">petsc-users@mcs.anl.gov</a>><br class=""><b class="">Subject:</b><span class="xxxapple-converted-space"> </span>Re: [petsc-users] Using matrix-free with KSP<span style="font-size: 13.5pt; font-family: Helvetica, sans-serif;" class=""><o:p class=""></o:p></span></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 13.5pt; font-family: Helvetica, sans-serif;" class=""> <o:p class=""></o:p></span></div></div></div><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-family: Helvetica, sans-serif;" class="">You'll usually have a GlobalToLocal operation for each rank to get the halo data it needs, then either a LocalToGlobal to collect the result (e.g., finite element methods) or the local compute will write directly into the owned portion of the global vector. If you're doing the communication "raw", then you may find VecScatter or PetscSF useful to perform the necessary communication.<br class=""><br class="">"Tu, Jiannan" <<a href="mailto:Jiannan_Tu@uml.edu" style="color: blue; text-decoration: underline;" class="">Jiannan_Tu@uml.edu</a>> writes:<br class=""><br class="">> Hi Barry,<br class="">><br class="">> Following your instructions I implemented the matrix-free method to solve a large linear equation system resulted from a surface integration equation. The KSP solver works fine for single process, but it is problematic with multiple processes. The problem is that each process only can access its own part of solution vector so that each process only conducts part of matrix-vector multiplication. MPI can be used to assemble these partial matrix-vector multiplication together. Does Petsc provide any ways to implement multi-process matrix-free method?<br class="">><br class="">> Thanks,<br class="">> Jiannan<br class="">> ________________________________<br class="">> From: Barry Smith <<a href="mailto:bsmith@petsc.dev" style="color: blue; text-decoration: underline;" class="">bsmith@petsc.dev</a>><br class="">> Sent: Tuesday, May 24, 2022 2:12 PM<br class="">> To: Tu, Jiannan <<a href="mailto:Jiannan_Tu@uml.edu" style="color: blue; text-decoration: underline;" class="">Jiannan_Tu@uml.edu</a>><br class="">> Cc:<span class="xxxapple-converted-space"> </span><a href="mailto:petsc-users@mcs.anl.gov" style="color: blue; text-decoration: underline;" class="">petsc-users@mcs.anl.gov</a><span class="xxxapple-converted-space"> </span><<a href="mailto:petsc-users@mcs.anl.gov" style="color: blue; text-decoration: underline;" class="">petsc-users@mcs.anl.gov</a>><br class="">> Subject: Re: [petsc-users] Using matrix-free with KSP<br class="">><br class="">> This e-mail originated from outside the UMass Lowell network.<br class="">> ________________________________<br class="">><br class="">> You can use MatCreateMFFD<span class="xxxapple-converted-space"> </span><a href="https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMatCreateMFFD%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7C454a1eb8de0745be453508da7fb114e4%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637962695800558712%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=%2FausyQFMwKFHd4FnNaKAOCGwskUc%2F8jIlSxtWMdIjC4%3D&reserved=0" style="color: blue; text-decoration: underline;" class="">https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMatCreateMFFD%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7Cab0859d8d154471590bc08da5f5918d6%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637927133525745809%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=A7wqegTfh94No5BpDiWLK3VxOuR44U2wlWHVm2k7l60%3D&reserved=0<https://urldefense.com/v3/__https://petsc.org/main/docs/manualpages/Mat/MatCreateMFFD/__;!!PVKG_VDCxu5g!stxq2NWGVjUihvHBQC9Rlk0aHVoowcy0PTQYMcQxQ4DqOEC05KSw6TmstSXKckiUgelHzy4ue-d10-zlXkw$></a><span class="xxxapple-converted-space"> </span>MatMFFDSetFunction MatSetFromOptions MatMFFDSetBase and provide the matrix to KSP. Note you will need to use -pc_type none or provide your own custom preconditioner with<a href="https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FPC%2FPCSHELL%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7C454a1eb8de0745be453508da7fb114e4%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637962695800558712%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=%2F5%2BLn2AdyPMZYfBS11Dd87S5anXtl3QPPNEst%2BjDGE4%3D&reserved=0" style="color: blue; text-decoration: underline;" class="">https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FPC%2FPCSHELL%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7Cab0859d8d154471590bc08da5f5918d6%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637927133525745809%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Gb%2F9uFG3jp%2FbfWaSh2cSEQLItHS9CuLwarzN0KcNiJY%3D&reserved=0<https://urldefense.com/v3/__https://petsc.org/main/docs/manualpages/PC/PCSHELL/__;!!PVKG_VDCxu5g!stxq2NWGVjUihvHBQC9Rlk0aHVoowcy0PTQYMcQxQ4DqOEC05KSw6TmstSXKckiUgelHzy4ue-d1cA4fKGw$></a><br class="">><br class="">><br class="">><br class="">> On May 24, 2022, at 1:21 PM, Tu, Jiannan <<a href="mailto:Jiannan_Tu@uml.edu" style="color: blue; text-decoration: underline;" class="">Jiannan_Tu@uml.edu</a><<a href="mailto:Jiannan_Tu@uml.edu" style="color: blue; text-decoration: underline;" class="">mailto:Jiannan_Tu@uml.edu</a>>> wrote:<br class="">><br class="">> I want to use a matrix-free matrix to solve a large linear equation system because the matrix is too large to be stored. Petsc user manual describes matrix-free method for SNES with examples. The matrix-free matrices section explains how to set up such a matrix, but I can't find any example of matrix-free method with KSP. I am wondering how to apply the method to KSP iterative solver in parallel.<br class="">><br class="">> I greatly appreciate your help for me to understand this topic.<br class="">><br class="">> Jiannan Tu</span></div></div></div></div></blockquote></div></div></div></div></div></blockquote></div></div></div></div></div></blockquote></div></div></div></div></div></blockquote></div><br class=""></div></div></body></html>