[petsc-users] MatSetValues dropping non-local entries

[Randall Mackie]

Well, I only need this particular matrix to multiply a vector (ordering based on the DMDA grid), so I don’t need to do any ghost communication (like residual calculations). I just need to be able to set a few non-local entries. Is there no way to do that without increasing the stencil width of the DMDA?

Randy


> On Aug 24, 2016, at 2:39 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
> 
>> On Aug 24, 2016, at 4:27 PM, Randall Mackie <rlmackie862 at gmail.com> wrote:
>> 
>> I’ve run into a situation where MatSetValues seems to be dropping non-local entries. Most of the entries that are set are local, but a few are possibly non-local, and are only maximum a few grid points off the local part of the grid.
>> 
>> Specifically, I get the local to global mapping, and the indices like so:
>> 
>> call DMGetLocalToGlobalMapping(da,ltogm,ierr)
>> call ISLocalToGlobalMappingGetIndices(ltogm,ltog,idltog,ierr)
>> 
>> then set the row using ltog(idltog + row +1) etc
>> 
>> When run on 1 process, everything worked fine, but for > 1 I was not getting the right result (I know what the right answer should be for a simple case).
>> 
>> I found that when I increased the stencil width on the DA (in the call to DACreate3d) that was used in the DMGetLocalToGlobalMapping to be large enough that the non-local points would be in the ghost region, everything was fine even for > 1 process.
>> 
>> 
>> So, in conclusion, it seems like if I use a local to global mapping from DMGetLocalToGlobalMapping, then MatSetValues will drop any non-local entries that are not included in the ghost region.
>> 
>> 
>> Is this the correct behavior and if so, is there another way to set these non-local values so they don’t get dropped?
> 
>  Yes, this is the expected behavior. Note also that DMDA only allocates space in the matrix for these locations and if it did stick in your "extra" locations it would be very very slow because it would have to reallocate the matrix data structures. 
> 
>  Why would you want to put in matrix entries that are not represented in the ghosting? The whole point of the ghosting is to indicate what values need to be communicated so you putting additional values in that do not fit the ghosting does not match the paradigm.
> 
>  Barry
> 
>> 
>> 
>> Thanks, 
>> 
>> Randy
>