<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Helvetica;
panose-1:2 11 5 4 2 2 2 2 2 4;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"Open Sans";
panose-1:2 11 6 6 3 5 4 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.apple-converted-space
{mso-style-name:apple-converted-space;}
span.EmailStyle23
{mso-style-type:personal-reply;
font-family:"Arial",sans-serif;
color:windowtext;
font-weight:normal;
font-style:normal;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif">Sure, I will try this. I will update this thread once I get it working using the suggested method. Thank you!<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif">Vysakh<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> Blaise Bourdin <bourdin@mcmaster.ca> <br>
<b>Sent:</b> Tuesday, January 17, 2023 5:13 PM<br>
<b>To:</b> Venugopal, Vysakh (venugovh) <venugovh@mail.uc.edu><br>
<b>Cc:</b> Barry Smith <bsmith@petsc.dev>; petsc-users@mcs.anl.gov<br>
<b>Subject:</b> Re: [petsc-users] about repeat of expensive functions using VecScatterCreateToAll<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p><strong><span style="font-family:"Open Sans",sans-serif;color:white;background:#E00122">External Email: Use Caution</span></strong><span style="color:white"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:white"><br>
<br>
</span><o:p></o:p></p>
<div>
<p class="MsoNormal">Got it. Can you partition your mesh with only one processor in the z-direction? (Trivial if using DMDA)
<o:p></o:p></p>
<div>
<p class="MsoNormal">Blaise<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal"><br>
<br>
<o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal">On Jan 17, 2023, at 4:49 PM, Venugopal, Vysakh (venugovh) <<a href="mailto:venugovh@mail.uc.edu">venugovh@mail.uc.edu</a>> wrote:<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif">This is the support structure minimization filter. So I need to go layer-by-layer from the bottommost slice of the array and update it as I move up. Every slice needs the updated values below
that slice.</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif">Vysakh</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif"> </span><o:p></o:p></p>
</div>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<div>
<p class="MsoNormal"><b>From:</b><span class="apple-converted-space"> </span>Blaise Bourdin <<a href="mailto:bourdin@mcmaster.ca">bourdin@mcmaster.ca</a>><span class="apple-converted-space"> </span><br>
<b>Sent:</b><span class="apple-converted-space"> </span>Tuesday, January 17, 2023 4:47 PM<br>
<b>To:</b><span class="apple-converted-space"> </span>Venugopal, Vysakh (venugovh) <<a href="mailto:venugovh@mail.uc.edu">venugovh@mail.uc.edu</a>><br>
<b>Cc:</b><span class="apple-converted-space"> </span>Barry Smith <<a href="mailto:bsmith@petsc.dev">bsmith@petsc.dev</a>>;<span class="apple-converted-space"> </span><a href="mailto:petsc-users@mcs.anl.gov">petsc-users@mcs.anl.gov</a><br>
<b>Subject:</b><span class="apple-converted-space"> </span>Re: [petsc-users] about repeat of expensive functions using VecScatterCreateToAll<o:p></o:p></p>
</div>
</div>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<p><strong><span style="font-size:9.0pt;font-family:"Open Sans",sans-serif;color:white;background:#E00122">External Email: Use Caution</span></strong><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif"><o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span style="color:white"><br>
<br>
<br>
</span><o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal">What type of filter are you implementing?<o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal">Convolution filters are expensive to parallelize since you need an overlap of the size of the support of the filter, but it may still not be worst than doing it sequentially (typically the filter size is only one or 2 element diameters).
Or you may be able to apply the filter in Fourier space.<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">PDE-filters are typically elliptic and can be parallelized.<o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">Blaise<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><br>
<br>
<br>
<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal">On Jan 17, 2023, at 4:38 PM, Venugopal,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Vysakh (venugovh) via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov">petsc-users@mcs.anl.gov</a>> wrote:<o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif">Thank you! I am doing a structural optimization filter that inherently cannot be parallelized.</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif"> </span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif">Vysakh</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif"> </span><o:p></o:p></p>
</div>
</div>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<div>
<div>
<p class="MsoNormal"><b>From:</b><span class="apple-converted-space"> </span>Barry Smith <<a href="mailto:bsmith@petsc.dev"><span style="color:#0563C1">bsmith@petsc.dev</span></a>><span class="apple-converted-space"> </span><br>
<b>Sent:</b><span class="apple-converted-space"> </span>Tuesday, January 17, 2023 3:28 PM<br>
<b>To:</b><span class="apple-converted-space"> </span>Venugopal, Vysakh (venugovh) <<a href="mailto:venugovh@mail.uc.edu"><span style="color:#0563C1">venugovh@mail.uc.edu</span></a>><br>
<b>Cc:</b><span class="apple-converted-space"> </span><a href="mailto:petsc-users@mcs.anl.gov"><span style="color:#0563C1">petsc-users@mcs.anl.gov</span></a><br>
<b>Subject:</b><span class="apple-converted-space"> </span>Re: [petsc-users] about repeat of expensive functions using VecScatterCreateToAll<o:p></o:p></p>
</div>
</div>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<p><strong><span style="font-size:9.0pt;font-family:"Open Sans",sans-serif;color:white;background:#E00122">External Email: Use Caution</span></strong><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif"><o:p></o:p></span></p>
<div>
<div>
<p class="MsoNormal"><span style="color:white"><br>
<br>
<br>
<br>
</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><br>
<br>
<br>
<br>
<o:p></o:p></p>
</div>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<div>
<div>
<p class="MsoNormal">On Jan 17, 2023, at 3:12 PM, Venugopal, Vysakh (venugovh) via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov"><span style="color:#0563C1">petsc-users@mcs.anl.gov</span></a>> wrote:<o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif">Hi,</span><o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif"> </span><o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif">I am doing the following thing.</span><o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif"> </span><o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif">Step 1. Create DM object and get global vector ‘V’ using DMGetGlobalVector.</span><o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif">Step 2. Doing some parallel operations on V.</span><o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif">Step 3. I am using VecScatterCreateToAll on V to create a sequential vector ‘V_SEQ’ using VecScatterBegin/End with SCATTER_FORWARD.</span><o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif">Step 4. I am performing an expensive operation on V_SEQ and outputting the updated V_SEQ.</span><o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif">Step 5. I am using VecScatterBegin/End with SCATTER_REVERSE (global and sequential flipped) to get V that is updated with new values from V_SEQ.</span><o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif">Step 6. I continue using this new V on the rest of the parallelized program.</span><o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif"> </span><o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif">Question: Suppose I have n MPI processes, is the expensive operation in Step 4 repeated n times? If yes, is there a workaround such that the operation in Step 4 is performed only once? I would
like to follow the same structure as steps 1 to 6 with step 4 only performed once.</span><o:p></o:p></p>
</div>
</div>
</div>
</div>
</blockquote>
<div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> Each MPI rank is doing the same operations on its copy of the sequential vector. Since they are running in parallel it probably does not matter much that each is doing the same computation. Step 5 does not require any MPI since only part
of the sequential vector (which everyone has) is needed in the parallel vector.<o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"> You could use VecScatterCreateToZero() but then step 3 would require less communication but step 5 would require communication to get parts of the solution from rank 0 to the other ranks. The time for step 4 would be roughly the same.<o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"> You will likely only see a worthwhile improvement in performance if you can parallelize the computation in 4. What are you doing that is computational intense and requires all the data on a rank?<o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal">Barry<o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><br>
<br>
<br>
<br>
<o:p></o:p></p>
</div>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif"> </span><o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif">Thanks,</span><o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif"> </span><o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif">Vysakh Venugopal</span><o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-family:"Arial",sans-serif;color:black;border:none windowtext 1.0pt;padding:0in">---</span><o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-family:"Arial",sans-serif;color:black;border:none windowtext 1.0pt;padding:0in">Vysakh Venugopal</span><o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-family:"Arial",sans-serif;color:black;border:none windowtext 1.0pt;padding:0in">Ph.D. Candidate</span><o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-family:"Arial",sans-serif;color:black;border:none windowtext 1.0pt;padding:0in">Department of Mechanical Engineering</span><o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-family:"Arial",sans-serif;color:black;border:none windowtext 1.0pt;padding:0in">University of Cincinnati, Cincinnati, OH 45221-0072</span><o:p></o:p></p>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal">— <br>
Canada Research Chair in Mathematical and Computational Aspects of Solid Mechanics (Tier 1)<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">Professor, Department of Mathematics & Statistics<br>
Hamilton Hall room 409A, McMaster University<br>
1280 Main Street West, Hamilton, Ontario L8S 4K1, Canada <br>
<a href="https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.math.mcmaster.ca%2Fbourdin&data=05%7C01%7Cvenugovh%40mail.uc.edu%7C3a1025fb4ab04e99932508daf8d80e07%7Cf5222e6c5fc648eb8f0373db18203b63%7C1%7C0%7C638095904080757941%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=1Gm1hWVHEAiKmMvLH5sMb8pdJpjykUtWTAI3f4XgwmU%3D&reserved=0">https://www.math.mcmaster.ca/bourdin</a><span class="apple-converted-space"> </span>|
+1 (905) 525 9140 ext. 27243<o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="color:black">— <br>
Canada Research Chair in Mathematical and Computational Aspects of Solid Mechanics (Tier 1)<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">Professor, Department of Mathematics & Statistics<br>
Hamilton Hall room 409A, McMaster University<br>
1280 Main Street West, Hamilton, Ontario L8S 4K1, Canada <br>
<a href="https://www.math.mcmaster.ca/bourdin">https://www.math.mcmaster.ca/bourdin</a> | +1 (905) 525 9140 ext. 27243<o:p></o:p></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
</body>
</html>