<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
What type of filter are you implementing?<br>
<div>Convolution filters are expensive to parallelize since you need an overlap of the size of the support of the filter, but it may still not be worst than doing it sequentially (typically the filter size is only one or 2 element diameters). Or you may be
able to apply the filter in Fourier space.</div>
<div>PDE-filters are typically elliptic and can be parallelized.<br>
<div><br>
</div>
<div>Blaise</div>
<div><br>
<blockquote type="cite">
<div>On Jan 17, 2023, at 4:38 PM, Venugopal,<span class="Apple-tab-span" style="white-space:pre">
</span>Vysakh (venugovh) via petsc-users <petsc-users@mcs.anl.gov> wrote:</div>
<br class="Apple-interchange-newline">
<div>
<div class="WordSection1" style="page: WordSection1; caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-family: Arial, sans-serif;">Thank you! I am doing a structural optimization filter that inherently cannot be parallelized.<o:p></o:p></span></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-family: Arial, sans-serif;"><o:p> </o:p></span></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-family: Arial, sans-serif;">Vysakh<o:p></o:p></span></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-family: Arial, sans-serif;"><o:p> </o:p></span></div>
<div>
<div style="border-style: solid none none; border-top-width: 1pt; border-top-color: rgb(225, 225, 225); padding: 3pt 0in 0in;">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;"><b>From:</b><span class="Apple-converted-space"> </span>Barry Smith <<a href="mailto:bsmith@petsc.dev" style="color: rgb(5, 99, 193); text-decoration: underline;">bsmith@petsc.dev</a>><span class="Apple-converted-space"> </span><br>
<b>Sent:</b><span class="Apple-converted-space"> </span>Tuesday, January 17, 2023 3:28 PM<br>
<b>To:</b><span class="Apple-converted-space"> </span>Venugopal, Vysakh (venugovh) <<a href="mailto:venugovh@mail.uc.edu" style="color: rgb(5, 99, 193); text-decoration: underline;">venugovh@mail.uc.edu</a>><br>
<b>Cc:</b><span class="Apple-converted-space"> </span><a href="mailto:petsc-users@mcs.anl.gov" style="color: rgb(5, 99, 193); text-decoration: underline;">petsc-users@mcs.anl.gov</a><br>
<b>Subject:</b><span class="Apple-converted-space"> </span>Re: [petsc-users] about repeat of expensive functions using VecScatterCreateToAll<o:p></o:p></div>
</div>
</div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;"><o:p> </o:p></div>
<p><strong><span style="font-family: "Open Sans", sans-serif; color: white; background-color: rgb(224, 1, 34);">External Email: Use Caution</span></strong><span style="color: white;"><o:p></o:p></span></p>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="color: white;"><br>
<br>
</span><o:p></o:p></div>
<div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;"><o:p> </o:p></div>
<div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;"><br>
<br>
<o:p></o:p></div>
<blockquote style="margin-top: 5pt; margin-bottom: 5pt;">
<div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;">On Jan 17, 2023, at 3:12 PM, Venugopal, Vysakh (venugovh) via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" style="color: rgb(5, 99, 193); text-decoration: underline;">petsc-users@mcs.anl.gov</a>>
wrote:<o:p></o:p></div>
</div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;"><o:p> </o:p></div>
<div>
<div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-family: Arial, sans-serif;">Hi,</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-family: Arial, sans-serif;"> </span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-family: Arial, sans-serif;">I am doing the following thing.</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-family: Arial, sans-serif;"> </span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-family: Arial, sans-serif;">Step 1. Create DM object and get global vector ‘V’ using DMGetGlobalVector.</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-family: Arial, sans-serif;">Step 2. Doing some parallel operations on V.</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-family: Arial, sans-serif;">Step 3. I am using VecScatterCreateToAll on V to create a sequential vector ‘V_SEQ’ using VecScatterBegin/End with SCATTER_FORWARD.</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-family: Arial, sans-serif;">Step 4. I am performing an expensive operation on V_SEQ and outputting the updated V_SEQ.</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-family: Arial, sans-serif;">Step 5. I am using VecScatterBegin/End with SCATTER_REVERSE (global and sequential flipped) to get V that is updated with new values from
V_SEQ.</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-family: Arial, sans-serif;">Step 6. I continue using this new V on the rest of the parallelized program.</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-family: Arial, sans-serif;"> </span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-family: Arial, sans-serif;">Question: Suppose I have n MPI processes, is the expensive operation in Step 4 repeated n times? If yes, is there a workaround such that
the operation in Step 4 is performed only once? I would like to follow the same structure as steps 1 to 6 with step 4 only performed once.</span><o:p></o:p></div>
</div>
</div>
</blockquote>
<div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;"><o:p> </o:p></div>
</div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;"> Each MPI rank is doing the same operations on its copy of the sequential vector. Since they are running in parallel it probably does not matter much that each is doing the same computation.
Step 5 does not require any MPI since only part of the sequential vector (which everyone has) is needed in the parallel vector.<o:p></o:p></div>
</div>
<div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;"><o:p> </o:p></div>
</div>
<div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;"> You could use VecScatterCreateToZero() but then step 3 would require less communication but step 5 would require communication to get parts of the solution from rank 0 to the other
ranks. The time for step 4 would be roughly the same.<o:p></o:p></div>
</div>
<div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;"><o:p> </o:p></div>
</div>
<div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;"> You will likely only see a worthwhile improvement in performance if you can parallelize the computation in 4. What are you doing that is computational intense and requires all the
data on a rank?<o:p></o:p></div>
</div>
<div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;"><o:p> </o:p></div>
</div>
<div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;">Barry<o:p></o:p></div>
</div>
<div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;"><br>
<br>
<o:p></o:p></div>
<blockquote style="margin-top: 5pt; margin-bottom: 5pt;">
<div>
<div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-family: Arial, sans-serif;"> </span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-family: Arial, sans-serif;">Thanks,</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-family: Arial, sans-serif;"> </span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-family: Arial, sans-serif;">Vysakh Venugopal</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif; background-color: white;">
<span style="font-family: Arial, sans-serif; border: 1pt none windowtext; padding: 0in;">---</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif; background-color: white;">
<span style="font-family: Arial, sans-serif; border: 1pt none windowtext; padding: 0in;">Vysakh Venugopal</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif; background-color: white;">
<span style="font-family: Arial, sans-serif; border: 1pt none windowtext; padding: 0in;">Ph.D. Candidate</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif; background-color: white;">
<span style="font-family: Arial, sans-serif; border: 1pt none windowtext; padding: 0in;">Department of Mechanical Engineering</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif; background-color: white;">
<span style="font-family: Arial, sans-serif; border: 1pt none windowtext; padding: 0in;">University of Cincinnati, Cincinnati, OH 45221-0072</span></div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
<div>
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
<div>— <br>
Canada Research Chair in Mathematical and Computational Aspects of Solid Mechanics (Tier 1)</div>
<div>Professor, Department of Mathematics & Statistics<br>
Hamilton Hall room 409A, McMaster University<br>
1280 Main Street West, Hamilton, Ontario L8S 4K1, Canada <br>
https://www.math.mcmaster.ca/bourdin | +1 (905) 525 9140 ext. 27243</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
</div>
</body>
</html>