<div dir="ltr">Matt is getting confused with Fortran, I just talked with him and he gets it now,</div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Aug 25, 2016 at 4:58 PM, Robert Hager <span dir="ltr"><<a href="mailto:rhager@pppl.gov" target="_blank">rhager@pppl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word">Now I am confused.  n1, which is a local variable in this subroutine, is not declared as a pointer or as allocatable here, so there is memory connected to n1. And in my understanding whatever values we get out of XX with the VecScatter are written to this memory. After the scatter has finished, we want those values to go into a_n1, which is an in/output of this subroutine. That’s what the assignment a_n1=n1 does.<span class="HOEnZb"><font color="#888888"><div><br></div></font></span><div><span class="HOEnZb"><font color="#888888">Robert</font></span><div><div class="h5"><br><div><br></div><div><br><div><blockquote type="cite"><div>On Aug 25, 2016, at 3:56 PM, Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:</div><br><div><div dir="ltr" style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><div class="gmail_extra"><div class="gmail_quote">On Thu, Aug 25, 2016 at 2:38 PM, Robert Hager<span> </span><span dir="ltr"><<a href="mailto:rhager@pppl.gov" target="_blank">rhager@pppl.gov</a>></span><span> </span>wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div style="word-wrap:break-word">Isn’t xxvec(0) basically a pointer to n1 after the call to VecCreateSeqWithArray? If so, the VecScatter should update the values in n1 and setting a_n1=n1 in the end makes sense.</div></blockquote><div><br></div><div>xxvec(0) holds a pointer (n1) to the data. That pointer does not change when the scatter writes in to the Vec, so</div><div>at the end the pointer n1 is still the same, namely a_n1.</div><div><br></div><div>   Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div style="word-wrap:break-word"><div><span><font color="#888888">Robert</font></span><div><div><br><div><br></div><div><br><div><blockquote type="cite"><div>On Aug 25, 2016, at 3:16 PM, Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:</div><br><div><div dir="ltr" style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><div class="gmail_extra"><div class="gmail_quote">On Thu, Aug 25, 2016 at 1:59 PM, Mark Adams<span> </span><span dir="ltr"><<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>></span><span> </span>wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div>We have the subroutine below that scatters three vectors.  We have used this code on many machines and it works fine but on one machine data does not get scattered correctly. The first scatter looks OK, but it looks like the other two are missing data.</div><div><br></div><div>Am I using this correctly?  Should we use VecGetArray in here instead of just using the pointer used for construction? Is there a race condition here that I'm missing?</div><div><br></div><div>Thanks,</div><div>Mark</div><div><br></div><div>subroutine scatter_to_xgc(a_ts,a_XX,a_n1,<wbr>a_apar,a_phi,ierr)</div><div> <span> </span>use petscts</div><div> <span> </span>use sml_module,only:sml_mype</div><div> <span> </span>use xgc_ts_module</div><div> <span> </span>implicit none</div><div> <span> </span>type(xgc_ts),intent(in)::a_t<wbr>s</div><div> <span> </span>Vec,intent(in)::a_XX</div><div> <span> </span>real (kind=8),dimension(a_ts%nnode)<wbr>::a_n1,a_apar,a_phi</div><div> <span> </span>PetscErrorCode,intent(out)::<wbr>ierr</div><div> <span> </span>! locals</div><div> <span> </span>PetscInt,parameter::ione=1</div><div> <span> </span>PetscScalar,dimension(a_ts%n<wbr>node)::n1,apar,phi</div><div> <span> </span>Vec::xxvec(0:2)</div><div><br></div><div> <span> </span>! scatter solution back - n1</div><div> <span> </span>n1 = a_n1</div></div></blockquote><div><br></div><div>Let me be clearer. Here n1 is equal to a_n1, and used to back xxvec(0). It does not change</div><div>in the course of the function, so why would you have</div><div><br></div><div> <span> </span>a_n1 = n</div><div><br></div><div>later on? What am I missing?</div><div><br></div><div>   Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div> <span> </span>call VecCreateSeqWithArray(PETSC_CO<wbr>MM_SELF,ione,a_ts%nnode,n1,xxv<wbr>ec(0),ierr);CHKERRQ(ierr)</div><div> <span> </span>call VecScatterBegin(a_ts%from_pets<wbr>c(0),a_XX,xxvec(0),INSERT_VALU<wbr>ES,SCATTER_FORWARD,ierr)</div><div><br></div><div> <span> </span>! scatter solution back - apar</div><div> <span> </span>apar = a_apar</div><div> <span> </span>call VecCreateSeqWithArray(PETSC_CO<wbr>MM_SELF,ione,a_ts%nnode,apar,x<wbr>xvec(1),ierr);CHKERRQ(ierr)</div><div> <span> </span>call VecScatterBegin(a_ts%from_pets<wbr>c(1),a_XX,xxvec(1),INSERT_VALU<wbr>ES,SCATTER_FORWARD,ierr)</div><div><br></div><div> <span> </span>! scatter solution back - phi</div><div> <span> </span>phi = a_phi</div><div> <span> </span>call VecCreateSeqWithArray(PETSC_CO<wbr>MM_SELF,ione,a_ts%nnode,phi,xx<wbr>vec(2),ierr);CHKERRQ(ierr)</div><div> <span> </span>call VecScatterBegin(a_ts%from_pets<wbr>c(2),a_XX,xxvec(2),INSERT_VALU<wbr>ES,SCATTER_FORWARD,ierr)</div><div><br></div><div> <span> </span>! end</div><div> <span> </span>call VecScatterEnd(  a_ts%from_petsc(0),a_XX,xxvec<wbr>(0),INSERT_VALUES,SCATTER_FORW<wbr>ARD,ierr)</div><div> <span> </span>a_n1 = n1</div><div> <span> </span>call VecDestroy(xxvec(0),ierr)</div><div><br></div><div> <span> </span>call VecScatterEnd(  a_ts%from_petsc(1),a_XX,xxvec<wbr>(1),INSERT_VALUES,SCATTER_FORW<wbr>ARD,ierr)</div><div> <span> </span>a_apar = apar</div><div> <span> </span>call VecDestroy(xxvec(1),ierr)</div><div><br></div><div> <span> </span>call VecScatterEnd(  a_ts%from_petsc(2),a_XX,xxvec<wbr>(2),INSERT_VALUES,SCATTER_FORW<wbr>ARD,ierr)</div><div> <span> </span>a_phi = phi</div><div> <span> </span>call VecDestroy(xxvec(2),ierr)</div><div><br></div><div> <span> </span>return</div><div>end subroutine scatter_to_xgc</div></div></blockquote></div><br><br clear="all"><div><br></div>--<span> </span><br><div data-smartmail="gmail_signature">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div></div></div></div></blockquote></div><br></div></div></div></div></div></blockquote></div><br><br clear="all"><div><br></div>--<span> </span><br><div data-smartmail="gmail_signature">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div></div></div></div></blockquote></div><br></div></div></div></div></div></blockquote></div><br></div>