[petsc-dev] VecScatter problem

Thu Aug 25 16:36:09 CDT 2016

“Steve: are you using the debug version of PETSc that I built?”

I am.

“Steve: perhaps you could try setting apar = -1.d0, or something and look at your plots to verify that it is not getting updated. Perhaps it is getting updated with 0.0.  And now that I think about it, this assignment should put some reasonable value in "apar" so if was not updated it should not be zero.  Is that correct or could a_par be zero coming into this routine?”

So, Robert and I tested this previously by creating dummy vectors filled with ‘1’, scattering them to petsc, scattering them out of petsc, and then dumping the output. We were still seeing just part of the domain, so I don’t think it’s a problem with the array coming in.

I’m reasonably confident that the original vector makes it to petsc just fine when we ‘scatter_from_xgc’, but the whole thing doesn’t make it back when we scatter out of petsc with ‘scatter_from_xgc.’

The troubling bit about this, which caused me to suggest that it was a race condition, was that if I modify the code a little bit I can get the correct result from the scatter. For example, we weren’t always checking the returned error flag. If I add in additional error checks, the output shows the entire field. If I do a VecGetArray … VecRestoreArray, I get the whole domain. If I move all the VecScatterEnds and put in an MPI_Barrier before doing the final assignment, I get the whole domain.

It is only with the original code that I don’t get the full domain. My hope was that we were making a mistake in syntax and copying the array before it had been flushed from PETSc’s internal buffers to the backing array.

Since that’s not the case, it means it’s something more insidious. Which means, as Barry just suggested, I need to fire up valgrind or some other memory checker.

Thanks,
--steve

On 8/25/16, 5:18 PM, "Mark Adams" <mfadams at lbl.gov<mailto:mfadams at lbl.gov>> wrote:

So, this code works on lots of machines and the first scatter does work.

And it sounds like there is nothing wrong with this syntax, there is no known race condition here.

Steve: are you using the debug version of PETSc that I built?

Note, this assignment:

  ! scatter solution back - apar
  apar = a_apar !!!! THIS ONE
  call VecCreateSeqWithArray(PETSC_COMM_SELF,ione,a_ts%nnode,apar,xxvec(1),ierr);CHKERRQ(ierr)
  call VecScatterBegin(a_ts%from_petsc(1),a_XX,xxvec(1),INSERT_VALUES,SCATTER_FORWARD,ierr)

Is just so that any boundary conditions in nodes that PETSc does not touch, get passed through this routine.

Steve: perhaps you could try setting apar = -1.d0, or something and look at your plots to verify that it is not getting updated. Perhaps it is getting updated with 0.0.  And now that I think about it, this assignment should put some reasonable value in "apar" so if was not updated it should not be zero.  Is that correct or could a_par be zero coming into this routine?

If need be I could add print statements in PETSc to dig into this.

On Thu, Aug 25, 2016 at 5:04 PM, Mark Adams <mfadams at lbl.gov<mailto:mfadams at lbl.gov>> wrote:
Matt is getting confused with Fortran, I just talked with him and he gets it now,

On Thu, Aug 25, 2016 at 4:58 PM, Robert Hager <rhager at pppl.gov<mailto:rhager at pppl.gov>> wrote:
Now I am confused.  n1, which is a local variable in this subroutine, is not declared as a pointer or as allocatable here, so there is memory connected to n1. And in my understanding whatever values we get out of XX with the VecScatter are written to this memory. After the scatter has finished, we want those values to go into a_n1, which is an in/output of this subroutine. That’s what the assignment a_n1=n1 does.

Robert

On Aug 25, 2016, at 3:56 PM, Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>> wrote:

On Thu, Aug 25, 2016 at 2:38 PM, Robert Hager <rhager at pppl.gov<mailto:rhager at pppl.gov>> wrote:
Isn’t xxvec(0) basically a pointer to n1 after the call to VecCreateSeqWithArray? If so, the VecScatter should update the values in n1 and setting a_n1=n1 in the end makes sense.

xxvec(0) holds a pointer (n1) to the data. That pointer does not change when the scatter writes in to the Vec, so
at the end the pointer n1 is still the same, namely a_n1.

   Matt

Robert

On Aug 25, 2016, at 3:16 PM, Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>> wrote:

On Thu, Aug 25, 2016 at 1:59 PM, Mark Adams <mfadams at lbl.gov<mailto:mfadams at lbl.gov>> wrote:
We have the subroutine below that scatters three vectors.  We have used this code on many machines and it works fine but on one machine data does not get scattered correctly. The first scatter looks OK, but it looks like the other two are missing data.

Am I using this correctly?  Should we use VecGetArray in here instead of just using the pointer used for construction? Is there a race condition here that I'm missing?

Thanks,
Mark

subroutine scatter_to_xgc(a_ts,a_XX,a_n1,a_apar,a_phi,ierr)
  use petscts
  use sml_module,only:sml_mype
  use xgc_ts_module
  implicit none
  type(xgc_ts),intent(in)::a_ts
  Vec,intent(in)::a_XX
  real (kind=8),dimension(a_ts%nnode)::a_n1,a_apar,a_phi
  PetscErrorCode,intent(out)::ierr
  ! locals
  PetscInt,parameter::ione=1
  PetscScalar,dimension(a_ts%nnode)::n1,apar,phi
  Vec::xxvec(0:2)

  ! scatter solution back - n1
  n1 = a_n1

Let me be clearer. Here n1 is equal to a_n1, and used to back xxvec(0). It does not change
in the course of the function, so why would you have

  a_n1 = n

later on? What am I missing?

   Matt

  call VecCreateSeqWithArray(PETSC_COMM_SELF,ione,a_ts%nnode,n1,xxvec(0),ierr);CHKERRQ(ierr)
  call VecScatterBegin(a_ts%from_petsc(0),a_XX,xxvec(0),INSERT_VALUES,SCATTER_FORWARD,ierr)

  ! scatter solution back - apar
  apar = a_apar
  call VecCreateSeqWithArray(PETSC_COMM_SELF,ione,a_ts%nnode,apar,xxvec(1),ierr);CHKERRQ(ierr)
  call VecScatterBegin(a_ts%from_petsc(1),a_XX,xxvec(1),INSERT_VALUES,SCATTER_FORWARD,ierr)

  ! scatter solution back - phi
  phi = a_phi
  call VecCreateSeqWithArray(PETSC_COMM_SELF,ione,a_ts%nnode,phi,xxvec(2),ierr);CHKERRQ(ierr)
  call VecScatterBegin(a_ts%from_petsc(2),a_XX,xxvec(2),INSERT_VALUES,SCATTER_FORWARD,ierr)

  ! end
  call VecScatterEnd(  a_ts%from_petsc(0),a_XX,xxvec(0),INSERT_VALUES,SCATTER_FORWARD,ierr)
  a_n1 = n1
  call VecDestroy(xxvec(0),ierr)

  call VecScatterEnd(  a_ts%from_petsc(1),a_XX,xxvec(1),INSERT_VALUES,SCATTER_FORWARD,ierr)
  a_apar = apar
  call VecDestroy(xxvec(1),ierr)

  call VecScatterEnd(  a_ts%from_petsc(2),a_XX,xxvec(2),INSERT_VALUES,SCATTER_FORWARD,ierr)
  a_phi = phi
  call VecDestroy(xxvec(2),ierr)

  return
end subroutine scatter_to_xgc

--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20160825/ea3863ed/attachment.html>