[petsc-users] DMGlobalToLocal acting weird

[Smith, Barry F.]

> On Jul 2, 2018, at 6:48 PM, Manuel Valera <mvalera-w at sdsu.edu> wrote:
> 
> 
> 
> On Mon, Jul 2, 2018 at 4:23 PM, Smith, Barry F. <bsmith at mcs.anl.gov> wrote:
> 
> 
> > On Jul 2, 2018, at 6:13 PM, Manuel Valera <mvalera-w at sdsu.edu> wrote:
> > 
> > Ok so i fixed the '99 business' by making sure the whole DMDA array was filled, there was row at the end that wasn't and hence that problem,
> > 
> > The local vector problem persists, though,
> 
>   Exactly what error? When you print the entire local vector it doesn't match the global vector? Or something else. 
> 
> So, still debugging with extremas, if i do:
>     
>     call DMGlobalToLocalBegin(daScalars,gTemperature,INSERT_VALUES,LocTemperature,ierr)
>     call DMGlobalToLocalEnd(daScalars,gTemperature,INSERT_VALUES,LocTemperature,ierr)

   What about LocTempature? Is that correct or not?

   using min() helps you determine there is a problem but doesn't help much in determine exactly where the problem is. Print all of the entries in LocTempature and in DimTempature to see which ones are bad, or are they all bad?


>     call DMCreateLocalVector(daScalars,DimTemperature,ierr)
>     call VecCopy(LocTemperature,DimTemperature,ierr)
> 
>   I get a wrong minimum, zero instead of the actual value, 
> 
> If instead i do:
> 
>     call DMCreateGlobalVector(daScalars,DimTemperature,ierr)
>     call VecCopy(gTemperature,DimTemperature,ierr)
> 
> I get the right extreme values,
> 
> 
> 
>   DMGlobalToLocalBegin/End() are used in dozens of tests in PETSc; very unlikely there is a bug that is causing you a problem especially on one process where it just copies the values from the global vector to the local vector.
> 
> Yes, this is baffling for me also, but the parallelization of my code is based on this paradigm of local vectors so i need to be sure is working properly, and it wasn't until these changes,
> 
> Thanks,
> 
> 
> 
> 
>  
> 
> 
> > i solved it for now using a global vector instead of a local one, is this because of the warning on globaltolocalbegin/end? 
> > 
> > Thanks,
> > 
> > 
> > 
> > On Mon, Jul 2, 2018 at 3:42 PM, Manuel Valera <mvalera-w at sdsu.edu> wrote:
> > 
> > 
> > On Mon, Jul 2, 2018 at 3:04 PM, Smith, Barry F. <bsmith at mcs.anl.gov> wrote:
> > 
> >    First make sure that getCenterInfo(daScalars,  xstart, ystart, zstart, xend, yend, zend) returns what it should. 
> > 
> > It does, i am also working in one processor only for now.
> > 
> >  
> >    remove the 99 business and print out the arrtemp() and TO() values. Are they correct?
> > 
> > They are not, i'm using maximum and minimum as probes to print into, but the minimum of T0 is 1 and is 0 for arrtemp, 
> >  
> > 
> >    What is DimTemperature? Shouldn't it be LocTempature?
> > 
> > Yes, sorry about that,
> >  
> > 
> >    Print out the values in tdim().
> > 
> > Yes, the same as with arrtemp/T0
> > 
> > Sorry i didn't mention i was debugging by printing the extremas of the arrays,
> > 
> > From your answer you seem to think the problem is in my code's side and it may very well be, but from the DMGlobalToLocalBegin there is a warning saying INSERT_VALUES doesn't work with DMDAs, is this still an issue or have been fixed? i understand it looks like these are two different problems in my code,
> > 
> > Thanks,
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> >  
> > 
> >    Barry
> > 
> > 
> > 
> > 
> > > On Jul 2, 2018, at 2:58 PM, Manuel Valera <mvalera-w at sdsu.edu> wrote:
> > > 
> > > Hi guys,
> > > 
> > > I've noticed a bug in my code that seems to happen right after a call to DMGlobalToLocalBegin/End and i can't seem to find a reason, it goes like this:
> > > 
> > > I create the DMDA (daScalars) with the following: 
> > > 
> > >  bx = DM_BOUNDARY_GHOSTED
> > >  by = DM_BOUNDARY_PERIODIC
> > >  bz = DM_BOUNDARY_GHOSTED
> > >  dof = 1
> > >  stw = 3
> > > call DMDACreate3d(PETSC_COMM_WORLD,bx,by,bz,DMDA_STENCIL_BOX,gridx,gridy,gridz,PETSC_DECIDE,PETSC_DECIDE,PETSC_DECIDE, &                                dof,stw,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,daScalars,ierr)
> > > call DMSetFromOptions(daScalars,ierr)
> > > call DMSetUp(daScalars,ierr)
> > > 
> > > Then i read from a file the temperature and salinity fields of my problems, those two are giving the trouble, i do it with this template:
> > > 
> > > call DMCreateGlobalVector(daScalars,gTemperature,ierr)
> > > call DMDAVecGetArrayF90(daScalars,gTemperature,arrtemp,ierr)
> > >    arrtemp = 99.0d0      !no idea why this fixes the problem -mv 62518
> > > call getCenterInfo(daScalars,  xstart, ystart, zstart, xend, yend, zend)
> > >             do k=zstart,zend-1
> > >                 do j=ystart,yend-1
> > >                     do i=xstart,xend-1
> > >                         arrtemp(i,j,k) = T0(i+1,j+1,k+1)
> > >                     enddo
> > >                 enddo
> > >             enddo
> > > call DMDAVecRestoreArrayF90(daScalars,gTemperature,arrtemp,ierr)
> > > call DMCreateLocalVector(daScalars,LocTemperature,ierr)
> > > call DMGlobalToLocalBegin(daScalars,gTemperature,INSERT_VALUES,LocTemperature,ierr)
> > > call DMGlobalToLocalEnd(daScalars,gTemperature,INSERT_VALUES,LocTemperature,ierr)
> > > 
> > > 
> > > Now here is the first weirdness, if i assign a number to my array it does behave as it should for the global quantity, but it must be a greater number than the maximum of the field for this to happen, so i assign 99.0 arbitrarily, otherwise it doesn't read the array correctly, but it does as of now,
> > > 
> > > Next, the last two lines communicate the array to the local counterparts, but when i read LocTemperature with:
> > > 
> > >     call DMDAVecGetArrayReadF90(daScalars,DimTemperature,tdim,ierr)
> > >     print*,'Copied MinT0: ', minval(tdim)
> > >     call DMDAVecRestoreArrayReadF90(daScalars,DimTemperature,tdim,ierr)
> > > 
> > > That array minimum doesn't coincide with the global minimum, it should be 1.000 but it is just 0.000, the same happens with the Salinity following an identical approach,
> > > 
> > > Any ideas on where to start looking? i have last month's PETSc build,
> > > 
> > > I'm running in 1 processor for now,
> > > 
> > > Thanks,
> > > 
> > > Manuel
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > 
> > 
> >