[petsc-users] MatSetValues dropping non-local entries

[Randall Mackie]

Thanks for the help. I now understand why the DMDA local to global mapping doesn’t work.

I was able to do what I wanted by computing the row/col numbers using natural ordering and then converting to PETSc ordering using AOApplicationToPetsc and the AO from DMDAGetAO, and that works fine.

Randy

> On Aug 24, 2016, at 4:46 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
> 
>   How important is this matrix for performance? Is it just a side operation you do rarely compared to linear solves and other stuff or is it used many times in a tight loop?
> 
>    If you do it rarely you can use a DMDAGlobalToNaturalBegin/End then do the matrix product (and build the matrix) using the normal natural ordering on a grid and then call DMDANaturalToGlobalBegin/End after. Not efficient but maybe good enough.
> 
>    Barry
> 
>> On Aug 24, 2016, at 5:23 PM, Matthew Knepley <knepley at gmail.com> wrote:
>> 
>> On Wed, Aug 24, 2016 at 5:01 PM, Randall Mackie <rlmackie862 at gmail.com> wrote:
>> I already create my own matrix with the appropriate size and layout. The problem seems to be the local to global mapping from DMGetLocalToGlobalMapping, which I suspect does not allow for these non-local entries outside the stencil width.
>> 
>> How is one suppose to determine the local to global mapping without a call to this?
>> 
>> @Matthew: I had tried MatSetValuesStencil with the same result, and in fact the web page says this: The columns and rows in the stencil passed in MUST be contained within the ghost region of the given process as set with DMDACreateXXX() or MatSetStencil(). 
>> 
>> You are right. In the deep past, we determined this directly using the dimensions. Now we use the map.
>> 
>> You will have to calculate the global indices you want by hand in the PETSc ordering, which means
>> knowing what process you want owns your (i,j,k). This is of course tedious, which is why we prefer to
>> use the maps, but its not possible to store every global index in the local map.
>> 
>>   Matt
>> 
>> Randy
>> 
>>> On Aug 24, 2016, at 2:52 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>> 
>>> 
>>>> On Aug 24, 2016, at 4:45 PM, Randall Mackie <rlmackie862 at gmail.com> wrote:
>>>> 
>>>> Well, I only need this particular matrix to multiply a vector (ordering based on the DMDA grid), so I don’t need to do any ghost communication (like residual calculations). I just need to be able to set a few non-local entries. Is there no way to do that without increasing the stencil width of the DMDA?
>>> 
>>>  Create your own matrix of the appropriate size and layout to match the DMDA vector and then put your values in it with MatSetValues(); don't use the matrix from DMCreateMatrix()
>>> 
>>> Barry
>>> 
>>>> 
>>>> Randy
>>>> 
>>>> 
>>>>> On Aug 24, 2016, at 2:39 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>> 
>>>>> 
>>>>>> On Aug 24, 2016, at 4:27 PM, Randall Mackie <rlmackie862 at gmail.com> wrote:
>>>>>> 
>>>>>> I’ve run into a situation where MatSetValues seems to be dropping non-local entries. Most of the entries that are set are local, but a few are possibly non-local, and are only maximum a few grid points off the local part of the grid.
>>>>>> 
>>>>>> Specifically, I get the local to global mapping, and the indices like so:
>>>>>> 
>>>>>> call DMGetLocalToGlobalMapping(da,ltogm,ierr)
>>>>>> call ISLocalToGlobalMappingGetIndices(ltogm,ltog,idltog,ierr)
>>>>>> 
>>>>>> then set the row using ltog(idltog + row +1) etc
>>>>>> 
>>>>>> When run on 1 process, everything worked fine, but for > 1 I was not getting the right result (I know what the right answer should be for a simple case).
>>>>>> 
>>>>>> I found that when I increased the stencil width on the DA (in the call to DACreate3d) that was used in the DMGetLocalToGlobalMapping to be large enough that the non-local points would be in the ghost region, everything was fine even for > 1 process.
>>>>>> 
>>>>>> 
>>>>>> So, in conclusion, it seems like if I use a local to global mapping from DMGetLocalToGlobalMapping, then MatSetValues will drop any non-local entries that are not included in the ghost region.
>>>>>> 
>>>>>> 
>>>>>> Is this the correct behavior and if so, is there another way to set these non-local values so they don’t get dropped?
>>>>> 
>>>>> Yes, this is the expected behavior. Note also that DMDA only allocates space in the matrix for these locations and if it did stick in your "extra" locations it would be very very slow because it would have to reallocate the matrix data structures. 
>>>>> 
>>>>> Why would you want to put in matrix entries that are not represented in the ghosting? The whole point of the ghosting is to indicate what values need to be communicated so you putting additional values in that do not fit the ghosting does not match the paradigm.
>>>>> 
>>>>> Barry
>>>>> 
>>>>>> 
>>>>>> 
>>>>>> Thanks, 
>>>>>> 
>>>>>> Randy
>>>>> 
>>>> 
>>> 
>> 
>> 
>> 
>> 
>> -- 
>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>> -- Norbert Wiener
>