<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Barry,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
VecScatterCreateToAll solves the problem! The code now gives the correct answer with multiple processes.
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Thank you all so much!</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Jiannan<br>
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Barry Smith <bsmith@petsc.dev><br>
<b>Sent:</b> Wednesday, July 6, 2022 2:08 PM<br>
<b>To:</b> Tu, Jiannan <Jiannan_Tu@uml.edu><br>
<b>Cc:</b> Jed Brown <jed@jedbrown.org>; petsc-users@mcs.anl.gov <petsc-users@mcs.anl.gov><br>
<b>Subject:</b> Re: [petsc-users] Using matrix-free with KSP</font>
<div> </div>
</div>
<div class="" style="word-wrap:break-word; line-break:after-white-space">
<div class=""><br class="">
</div>
So your operator is a "dense" operator in that its matrix representation would be essentially dense. In that case, you can use specialized routines to do this efficiently. You can use VecScatterCreateToAll() and then VecScatterBegin/End to do the communication.
<div class=""><br class="">
</div>
<div class=""> Barry</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class="">
<div><br class="">
<blockquote type="cite" class="">
<div class="">On Jul 6, 2022, at 1:39 PM, Tu, Jiannan <<a href="mailto:Jiannan_Tu@uml.edu" class="">Jiannan_Tu@uml.edu</a>> wrote:</div>
<br class="x_Apple-interchange-newline">
<div class="">
<div class="x_elementToProof" style="font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt">
Jed, thank you very much for the reply.<span class="x_Apple-converted-space"> </span><br class="">
</div>
<div class="x_elementToProof" style="font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt">
<br class="">
</div>
<div class="x_elementToProof" style="font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt">
I'm not sure GlobaltoLocal will work. Say the solution vector is of length 10. Two processes then each process can see 5 elements of the vector. For A x = b (10 equations), each process performs matrix-vector product for five equations. And for each equation
i , sum_j A_ij * x_j = b_i requires the process has access to all 10 unknows x_i (i=0,1,...,9).<span class="x_Apple-converted-space"> </span><br class="">
</div>
<div class="x_elementToProof" style="font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt">
<br class="">
</div>
<div class="x_elementToProof" style="font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt">
Can VecScatter and PetscSF make the entire vector accessible to each process? I am reading the manual and trying to understand how VecScatter and PetscSF work.</div>
<div class="x_elementToProof" style="font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt">
<br class="">
</div>
<div class="x_elementToProof" style="font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt">
Jiannan<br class="">
</div>
<div id="x_appendonsend" class="" style="font-family:Helvetica; font-size:18px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
</div>
<hr tabindex="-1" class="" style="font-family:Helvetica; font-size:18px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; display:inline-block; width:1262.234375px">
<span class="" style="font-family:Helvetica; font-size:18px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; float:none; display:inline!important"></span>
<div id="x_divRplyFwdMsg" dir="ltr" class="" style="font-family:Helvetica; font-size:18px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
<font face="Calibri, sans-serif" class="" style="font-size:11pt"><b class="">From:</b><span class="x_Apple-converted-space"> </span>Jed Brown <<a href="mailto:jed@jedbrown.org" class="">jed@jedbrown.org</a>><br class="">
<b class="">Sent:</b><span class="x_Apple-converted-space"> </span>Wednesday, July 6, 2022 10:09 AM<br class="">
<b class="">To:</b><span class="x_Apple-converted-space"> </span>Tu, Jiannan <<a href="mailto:Jiannan_Tu@uml.edu" class="">Jiannan_Tu@uml.edu</a>>; Barry Smith <<a href="mailto:bsmith@petsc.dev" class="">bsmith@petsc.dev</a>><br class="">
<b class="">Cc:</b><span class="x_Apple-converted-space"> </span><a href="mailto:petsc-users@mcs.anl.gov" class="">petsc-users@mcs.anl.gov</a><span class="x_Apple-converted-space"> </span><<a href="mailto:petsc-users@mcs.anl.gov" class="">petsc-users@mcs.anl.gov</a>><br class="">
<b class="">Subject:</b><span class="x_Apple-converted-space"> </span>Re: [petsc-users] Using matrix-free with KSP</font>
<div class=""> </div>
</div>
<div class="x_BodyFragment" style="font-family:Helvetica; font-size:18px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
<font size="2" class=""><span class="" style="font-size:11pt">
<div class="x_PlainText">You'll usually have a GlobalToLocal operation for each rank to get the halo data it needs, then either a LocalToGlobal to collect the result (e.g., finite element methods) or the local compute will write directly into the owned portion
of the global vector. If you're doing the communication "raw", then you may find VecScatter or PetscSF useful to perform the necessary communication.<br class="">
<br class="">
"Tu, Jiannan" <<a href="mailto:Jiannan_Tu@uml.edu" class="">Jiannan_Tu@uml.edu</a>> writes:<br class="">
<br class="">
> Hi Barry,<br class="">
><br class="">
> Following your instructions I implemented the matrix-free method to solve a large linear equation system resulted from a surface integration equation. The KSP solver works fine for single process, but it is problematic with multiple processes. The problem
is that each process only can access its own part of solution vector so that each process only conducts part of matrix-vector multiplication. MPI can be used to assemble these partial matrix-vector multiplication together. Does Petsc provide any ways to implement
multi-process matrix-free method?<br class="">
><br class="">
> Thanks,<br class="">
> Jiannan<br class="">
> ________________________________<br class="">
> From: Barry Smith <<a href="mailto:bsmith@petsc.dev" class="">bsmith@petsc.dev</a>><br class="">
> Sent: Tuesday, May 24, 2022 2:12 PM<br class="">
> To: Tu, Jiannan <<a href="mailto:Jiannan_Tu@uml.edu" class="">Jiannan_Tu@uml.edu</a>><br class="">
> Cc:<span class="x_Apple-converted-space"> </span><a href="mailto:petsc-users@mcs.anl.gov" class="">petsc-users@mcs.anl.gov</a><span class="x_Apple-converted-space"> </span><<a href="mailto:petsc-users@mcs.anl.gov" class="">petsc-users@mcs.anl.gov</a>><br class="">
> Subject: Re: [petsc-users] Using matrix-free with KSP<br class="">
><br class="">
> This e-mail originated from outside the UMass Lowell network.<br class="">
> ________________________________<br class="">
><br class="">
> You can use MatCreateMFFD<span class="x_Apple-converted-space"> </span><a href="https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMatCreateMFFD%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7C60675432e0b5409f004908da5f7a8b29%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637927277198462552%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=7SEmNO9TybDxwnhwcmyW1vFVc2eo7ys8JGBv9%2F%2FR2eI%3D&reserved=0" originalsrc="https://petsc.org/main/docs/manualpages/Mat/MatCreateMFFD/" shash="aDXgeXur7ORlzTSlpJb/H46OYbtpSJXvrh+25Vg+WDxpEyd+Ew7acmEg8N2x7IHPrm2drkVfe5GMYVI07xRaG9cixL3rZw9iPxeetO+w3Orua6MSnf+G2wStt4kKsQZE3XQ4kZVOuMOfeq5evSahl1jRC5oLuMm5fQ2WbR+pRlg=" class="">https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMatCreateMFFD%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7Cab0859d8d154471590bc08da5f5918d6%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637927133525745809%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=A7wqegTfh94No5BpDiWLK3VxOuR44U2wlWHVm2k7l60%3D&reserved=0<https://urldefense.com/v3/__https://petsc.org/main/docs/manualpages/Mat/MatCreateMFFD/__;!!PVKG_VDCxu5g!stxq2NWGVjUihvHBQC9Rlk0aHVoowcy0PTQYMcQxQ4DqOEC05KSw6TmstSXKckiUgelHzy4ue-d10-zlXkw$></a><span class="x_Apple-converted-space"> </span>MatMFFDSetFunction
MatSetFromOptions MatMFFDSetBase and provide the matrix to KSP. Note you will need to use -pc_type none or provide your own custom preconditioner with<a href="https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FPC%2FPCSHELL%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7C60675432e0b5409f004908da5f7a8b29%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637927277198462552%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=FDGqXk2UbO%2FYef5n21SIAUYrte093iKgL2p%2FwoA3qec%3D&reserved=0" originalsrc="https://petsc.org/main/docs/manualpages/PC/PCSHELL/" shash="Z4pJesuexL2BFCAIu7og3DctOBvfaWiT6IFHCDNfMM5ws0N+ouYygV+RADfWpQDFMVHBjGFf9SoRo9Vx5fWc0H/Fe8F9qZyiYq4jiuFx2dEKe8DOpP8wa6SSi35ku98lNHCaI1sLrTgMmyxGxvCpJbjaKJmu9q9CqJq7FcIjCFA=" class="">https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FPC%2FPCSHELL%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7Cab0859d8d154471590bc08da5f5918d6%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637927133525745809%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Gb%2F9uFG3jp%2FbfWaSh2cSEQLItHS9CuLwarzN0KcNiJY%3D&reserved=0<https://urldefense.com/v3/__https://petsc.org/main/docs/manualpages/PC/PCSHELL/__;!!PVKG_VDCxu5g!stxq2NWGVjUihvHBQC9Rlk0aHVoowcy0PTQYMcQxQ4DqOEC05KSw6TmstSXKckiUgelHzy4ue-d1cA4fKGw$></a><br class="">
><br class="">
><br class="">
><br class="">
> On May 24, 2022, at 1:21 PM, Tu, Jiannan <<a href="mailto:Jiannan_Tu@uml.edu" class="">Jiannan_Tu@uml.edu</a><<a href="mailto:Jiannan_Tu@uml.edu" class="">mailto:Jiannan_Tu@uml.edu</a>>> wrote:<br class="">
><br class="">
> I want to use a matrix-free matrix to solve a large linear equation system because the matrix is too large to be stored. Petsc user manual describes matrix-free method for SNES with examples. The matrix-free matrices section explains how to set up such a
matrix, but I can't find any example of matrix-free method with KSP. I am wondering how to apply the method to KSP iterative solver in parallel.<br class="">
><br class="">
> I greatly appreciate your help for me to understand this topic.<br class="">
><br class="">
> Jiannan Tu</div>
</span></font></div>
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</body>
</html>