<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;"><br><div><br><blockquote type="cite"><div>On Jan 17, 2023, at 3:12 PM, Venugopal, Vysakh (venugovh) via petsc-users <petsc-users@mcs.anl.gov> wrote:</div><br class="Apple-interchange-newline"><div><meta charset="UTF-8"><div class="WordSection1" style="page: WordSection1; caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-family: Arial, sans-serif;">Hi,<o:p></o:p></span></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-family: Arial, sans-serif;"><o:p> </o:p></span></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-family: Arial, sans-serif;">I am doing the following thing.<o:p></o:p></span></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-family: Arial, sans-serif;"><o:p> </o:p></span></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-family: Arial, sans-serif;">Step 1. Create DM object and get global vector ‘V’ using DMGetGlobalVector.<o:p></o:p></span></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-family: Arial, sans-serif;">Step 2. Doing some parallel operations on V.<o:p></o:p></span></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-family: Arial, sans-serif;">Step 3. I am using VecScatterCreateToAll on V to create a sequential vector ‘V_SEQ’ using VecScatterBegin/End with SCATTER_FORWARD.<o:p></o:p></span></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-family: Arial, sans-serif;">Step 4. I am performing an expensive operation on V_SEQ and outputting the updated V_SEQ.<o:p></o:p></span></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-family: Arial, sans-serif;">Step 5. I am using VecScatterBegin/End with SCATTER_REVERSE (global and sequential flipped) to get V that is updated with new values from V_SEQ.<o:p></o:p></span></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-family: Arial, sans-serif;">Step 6. I continue using this new V on the rest of the parallelized program.<o:p></o:p></span></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-family: Arial, sans-serif;"><o:p> </o:p></span></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-family: Arial, sans-serif;">Question: Suppose I have n MPI processes, is the expensive operation in Step 4 repeated n times? If yes, is there a workaround such that the operation in Step 4 is performed only once? I would like to follow the same structure as steps 1 to 6 with step 4 only performed once.</span></div></div></div></blockquote><div><br></div> Each MPI rank is doing the same operations on its copy of the sequential vector. Since they are running in parallel it probably does not matter much that each is doing the same computation. Step 5 does not require any MPI since only part of the sequential vector (which everyone has) is needed in the parallel vector.</div><div><br></div><div> You could use VecScatterCreateToZero() but then step 3 would require less communication but step 5 would require communication to get parts of the solution from rank 0 to the other ranks. The time for step 4 would be roughly the same.</div><div><br></div><div> You will likely only see a worthwhile improvement in performance if you can parallelize the computation in 4. What are you doing that is computational intense and requires all the data on a rank?</div><div><br></div><div>Barry</div><div><br><blockquote type="cite"><div><div class="WordSection1" style="page: WordSection1; caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-family: Arial, sans-serif;"><o:p></o:p></span></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-family: Arial, sans-serif;"><o:p> </o:p></span></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-family: Arial, sans-serif;">Thanks,<o:p></o:p></span></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-family: Arial, sans-serif;"><o:p> </o:p></span></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-family: Arial, sans-serif;">Vysakh Venugopal<o:p></o:p></span></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif; background-color: white;"><span style="font-family: Arial, sans-serif; border: 1pt none windowtext; padding: 0in;">---<o:p></o:p></span></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif; background-color: white;"><span style="font-family: Arial, sans-serif; border: 1pt none windowtext; padding: 0in;">Vysakh Venugopal</span><span style="font-family: Arial, sans-serif;"><o:p></o:p></span></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif; background-color: white;"><span style="font-family: Arial, sans-serif; border: 1pt none windowtext; padding: 0in;">Ph.D. Candidate</span><span style="font-family: Arial, sans-serif;"><o:p></o:p></span></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif; background-color: white;"><span style="font-family: Arial, sans-serif; border: 1pt none windowtext; padding: 0in;">Department of Mechanical Engineering</span><span style="font-family: Arial, sans-serif;"><o:p></o:p></span></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif; background-color: white;"><span style="font-family: Arial, sans-serif; border: 1pt none windowtext; padding: 0in;">University of Cincinnati, Cincinnati, OH 45221-0072</span></div></div></div></blockquote></div><br></body></html>