<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">Isn’t xxvec(0) basically a pointer to n1 after the call to VecCreateSeqWithArray? If so, the VecScatter should update the values in n1 and setting a_n1=n1 in the end makes sense.<div class=""><br class=""></div><div class="">Robert<br class=""><div class=""><br class=""></div><div class=""><br class=""><div><blockquote type="cite" class=""><div class="">On Aug 25, 2016, at 3:16 PM, Matthew Knepley <<a href="mailto:knepley@gmail.com" class="">knepley@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><div class="gmail_extra"><div class="gmail_quote">On Thu, Aug 25, 2016 at 1:59 PM, Mark Adams<span class="Apple-converted-space"> </span><span dir="ltr" class=""><<a href="mailto:mfadams@lbl.gov" target="_blank" class="">mfadams@lbl.gov</a>></span><span class="Apple-converted-space"> </span>wrote:<br class=""><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;"><div dir="ltr" class=""><div class="">We have the subroutine below that scatters three vectors. We have used this code on many machines and it works fine but on one machine data does not get scattered correctly. The first scatter looks OK, but it looks like the other two are missing data.</div><div class=""><br class=""></div><div class="">Am I using this correctly? Should we use VecGetArray in here instead of just using the pointer used for construction? Is there a race condition here that I'm missing?</div><div class=""><br class=""></div><div class="">Thanks,</div><div class="">Mark</div><div class=""><br class=""></div><div class="">subroutine scatter_to_xgc(a_ts,a_XX,a_n1,<wbr class="">a_apar,a_phi,ierr)</div><div class=""> <span class="Apple-converted-space"> </span>use petscts</div><div class=""> <span class="Apple-converted-space"> </span>use sml_module,only:sml_mype</div><div class=""> <span class="Apple-converted-space"> </span>use xgc_ts_module</div><div class=""> <span class="Apple-converted-space"> </span>implicit none</div><div class=""> <span class="Apple-converted-space"> </span>type(xgc_ts),intent(in)::a_ts</div><div class=""> <span class="Apple-converted-space"> </span>Vec,intent(in)::a_XX</div><div class=""> <span class="Apple-converted-space"> </span>real (kind=8),dimension(a_ts%nnode)<wbr class="">::a_n1,a_apar,a_phi</div><div class=""> <span class="Apple-converted-space"> </span>PetscErrorCode,intent(out)::<wbr class="">ierr</div><div class=""> <span class="Apple-converted-space"> </span>! locals</div><div class=""> <span class="Apple-converted-space"> </span>PetscInt,parameter::ione=1</div><div class=""> <span class="Apple-converted-space"> </span>PetscScalar,dimension(a_ts%<wbr class="">nnode)::n1,apar,phi</div><div class=""> <span class="Apple-converted-space"> </span>Vec::xxvec(0:2)</div><div class=""><br class=""></div><div class=""> <span class="Apple-converted-space"> </span>! scatter solution back - n1</div><div class=""> <span class="Apple-converted-space"> </span>n1 = a_n1</div></div></blockquote><div class=""><br class=""></div><div class="">Let me be clearer. Here n1 is equal to a_n1, and used to back xxvec(0). It does not change</div><div class="">in the course of the function, so why would you have</div><div class=""><br class=""></div><div class=""> a_n1 = n</div><div class=""><br class=""></div><div class="">later on? What am I missing?</div><div class=""><br class=""></div><div class=""> Matt</div><div class=""> </div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;"><div dir="ltr" class=""><div class=""> <span class="Apple-converted-space"> </span>call VecCreateSeqWithArray(PETSC_<wbr class="">COMM_SELF,ione,a_ts%nnode,n1,<wbr class="">xxvec(0),ierr);CHKERRQ(ierr)</div><div class=""> <span class="Apple-converted-space"> </span>call VecScatterBegin(a_ts%from_<wbr class="">petsc(0),a_XX,xxvec(0),INSERT_<wbr class="">VALUES,SCATTER_FORWARD,ierr)</div><div class=""><br class=""></div><div class=""> <span class="Apple-converted-space"> </span>! scatter solution back - apar</div><div class=""> <span class="Apple-converted-space"> </span>apar = a_apar</div><div class=""> <span class="Apple-converted-space"> </span>call VecCreateSeqWithArray(PETSC_<wbr class="">COMM_SELF,ione,a_ts%nnode,<wbr class="">apar,xxvec(1),ierr);CHKERRQ(<wbr class="">ierr)</div><div class=""> <span class="Apple-converted-space"> </span>call VecScatterBegin(a_ts%from_<wbr class="">petsc(1),a_XX,xxvec(1),INSERT_<wbr class="">VALUES,SCATTER_FORWARD,ierr)</div><div class=""><br class=""></div><div class=""> <span class="Apple-converted-space"> </span>! scatter solution back - phi</div><div class=""> <span class="Apple-converted-space"> </span>phi = a_phi</div><div class=""> <span class="Apple-converted-space"> </span>call VecCreateSeqWithArray(PETSC_<wbr class="">COMM_SELF,ione,a_ts%nnode,phi,<wbr class="">xxvec(2),ierr);CHKERRQ(ierr)</div><div class=""> <span class="Apple-converted-space"> </span>call VecScatterBegin(a_ts%from_<wbr class="">petsc(2),a_XX,xxvec(2),INSERT_<wbr class="">VALUES,SCATTER_FORWARD,ierr)</div><div class=""><br class=""></div><div class=""> <span class="Apple-converted-space"> </span>! end</div><div class=""> <span class="Apple-converted-space"> </span>call VecScatterEnd( a_ts%from_petsc(0),a_XX,<wbr class="">xxvec(0),INSERT_VALUES,<wbr class="">SCATTER_FORWARD,ierr)</div><div class=""> <span class="Apple-converted-space"> </span>a_n1 = n1</div><div class=""> <span class="Apple-converted-space"> </span>call VecDestroy(xxvec(0),ierr)</div><div class=""><br class=""></div><div class=""> <span class="Apple-converted-space"> </span>call VecScatterEnd( a_ts%from_petsc(1),a_XX,<wbr class="">xxvec(1),INSERT_VALUES,<wbr class="">SCATTER_FORWARD,ierr)</div><div class=""> <span class="Apple-converted-space"> </span>a_apar = apar</div><div class=""> <span class="Apple-converted-space"> </span>call VecDestroy(xxvec(1),ierr)</div><div class=""><br class=""></div><div class=""> <span class="Apple-converted-space"> </span>call VecScatterEnd( a_ts%from_petsc(2),a_XX,<wbr class="">xxvec(2),INSERT_VALUES,<wbr class="">SCATTER_FORWARD,ierr)</div><div class=""> <span class="Apple-converted-space"> </span>a_phi = phi</div><div class=""> <span class="Apple-converted-space"> </span>call VecDestroy(xxvec(2),ierr)</div><div class=""><br class=""></div><div class=""> <span class="Apple-converted-space"> </span>return</div><div class="">end subroutine scatter_to_xgc</div></div></blockquote></div><br class=""><br clear="all" class=""><div class=""><br class=""></div>--<span class="Apple-converted-space"> </span><br class=""><div class="gmail_signature" data-smartmail="gmail_signature">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br class="">-- Norbert Wiener</div></div></div></div></blockquote></div><br class=""></div></div></body></html>