<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class=""><br class=""></div> 50,000 * 50,000 > 2^31 (roughly 2 billion) hence we cannot represent even the length of a vector using 32 bit integers; just to do "book keeping" of dimensions requires using 64 bit integers. Using 64 bit integers should not slow PETSc significantly, especially in matrix-free mode.<div class=""><br class=""></div><div class=""><br class=""><div class=""><font face="Calibri, sans-serif" class=""><span style="font-size: 14.666666984558105px;" class=""><br class=""></span></font></div><div class=""><font face="Calibri, sans-serif" class=""><span style="font-size: 14.666666984558105px;" class=""><br class=""></span></font><div><br class=""><blockquote type="cite" class=""><div class="">On May 30, 2022, at 7:33 PM, Tu, Jiannan <<a href="mailto:Jiannan_Tu@uml.edu" class="">Jiannan_Tu@uml.edu</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><meta charset="UTF-8" class=""><div class="WordSection1" style="page: WordSection1; caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">I tried the matrix-free method. Since the number of the unknown is large, say 50000, then the number of matrix elements is 50000^2, which is greater than the max number represented by the 32-bit int. PETSC gives an error message: “Mesh of 50,000 by 50,000 by 1 (dof) is too large for 32 bit indices.”<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">I know using the 64-bit int option when compiling PETSC can avoid this error. However, it seems that 64-bit int makes the PETSC relatively slow. I wonder why the matrix-free method still counts the total number of matrix elements even though they are not stored. Any other ways around it?<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Thank you,<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Jiannan<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div><div class=""><div style="border-style: solid none none; border-top-width: 1pt; border-top-color: rgb(225, 225, 225); padding: 3pt 0in 0in;" class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><b class="">From:</b><span class="Apple-converted-space"> </span>Barry Smith <<a href="mailto:bsmith@petsc.dev" style="color: blue; text-decoration: underline;" class="">bsmith@petsc.dev</a>><span class="Apple-converted-space"> </span><br class=""><b class="">Sent:</b><span class="Apple-converted-space"> </span>Tuesday, May 24, 2022 2:13 PM<br class=""><b class="">To:</b><span class="Apple-converted-space"> </span>Tu, Jiannan <<a href="mailto:Jiannan_Tu@uml.edu" style="color: blue; text-decoration: underline;" class="">Jiannan_Tu@uml.edu</a>><br class=""><b class="">Cc:</b><span class="Apple-converted-space"> </span><a href="mailto:petsc-users@mcs.anl.gov" style="color: blue; text-decoration: underline;" class="">petsc-users@mcs.anl.gov</a><br class=""><b class="">Subject:</b><span class="Apple-converted-space"> </span>Re: [petsc-users] Using matrix-free with KSP<o:p class=""></o:p></div></div></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><b class=""><span style="color: red;" class="">This e-mail originated from outside the UMass Lowell network.</span></b><o:p class=""></o:p></div><div class="MsoNormal" align="center" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif; text-align: center;"><hr size="2" width="100%" align="center" class=""></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> You can use MatCreateMFFD <a href="https://urldefense.com/v3/__https:/petsc.org/main/docs/manualpages/Mat/MatCreateMFFD/__;!!PVKG_VDCxu5g!stxq2NWGVjUihvHBQC9Rlk0aHVoowcy0PTQYMcQxQ4DqOEC05KSw6TmstSXKckiUgelHzy4ue-d10-zlXkw$" style="color: blue; text-decoration: underline;" class="">https://petsc.org/main/docs/manualpages/Mat/MatCreateMFFD/</a><span class="Apple-converted-space"> </span>MatMFFDSetFunction MatSetFromOptions MatMFFDSetBase and provide the matrix to KSP. Note you will need to use -pc_type none or provide your own custom preconditioner with <a href="https://urldefense.com/v3/__https:/petsc.org/main/docs/manualpages/PC/PCSHELL/__;!!PVKG_VDCxu5g!stxq2NWGVjUihvHBQC9Rlk0aHVoowcy0PTQYMcQxQ4DqOEC05KSw6TmstSXKckiUgelHzy4ue-d1cA4fKGw$" style="color: blue; text-decoration: underline;" class="">https://petsc.org/main/docs/manualpages/PC/PCSHELL/</a><o:p class=""></o:p></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><br class=""><br class=""><o:p class=""></o:p></div><blockquote style="margin-top: 5pt; margin-bottom: 5pt;" class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">On May 24, 2022, at 1:21 PM, Tu, Jiannan <<a href="mailto:Jiannan_Tu@uml.edu" style="color: blue; text-decoration: underline;" class="">Jiannan_Tu@uml.edu</a>> wrote:<o:p class=""></o:p></div></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt;" class="">I want to use a matrix-free matrix to solve a large linear equation system because the matrix is too large to be stored. Petsc user manual describes matrix-free method for SNES with examples. The matrix-free matrices section explains how to set up such a matrix, but I can't find any example of matrix-free method with KSP. <span style="background-color: white;" class="">I am wondering how to apply the method to KSP iterative solver in parallel.</span><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt;" class=""><o:p class=""> </o:p></span></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt; background-color: white;" class="">I greatly appreciate your help for me to understand this topic. </span><span style="font-size: 12pt;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt;" class=""><o:p class=""> </o:p></span></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt; background-color: white;" class="">Jiannan Tu</span></div></div></div></blockquote></div></div></div></div></blockquote></div><br class=""></div></div></body></html>