[petsc-users] DMNetworkGetEdgeRange() in parallel

Miguel Angel Salazar de Troya salazardetroya at gmail.com
Tue Feb 24 18:42:08 CST 2015 

I implemented the code as agreed, but I don't get the results I expected.
When I create the vector with DMCreateGlobalVector(), I obtain a vector
with a layout similar to the original DMNetwork, instead of the cloned
network with the new PetscSection. The code is as follows:

DMClone(dm, &dmEdge);

PetscSectionCreate(PETSC_COMM_WORLD, &s);
PetscSectionSetNumFields(s, 1);
PetscSectionSetFieldComponents(s, 0, 1);

// Now to set the chart, I pick the edge range

DMNetworkGetEdgeRange(dmEdge, & eStart, & eEnd)

PetscSectionSetChart(s, eStart, eEnd);

for(PetscInt e = eStart; c < eEnd; ++e) {
     PetscSectionSetDof(s, e, 1);
     PetscSectionSetFieldDof(s, e, 0, 1);
}
PetscSectionSetUp(s);

DMSetDefaultSection(dmEdge s);
DMCreateGlobalVector(dmEdge, &globalVec);

When I get into DMCreateGlobalVector(dmEdge, &globalVec) in the debugger,
in the function DMCreateSubDM_Section_Private() I call PetscSectionView()
on the section obtained by DMGetDefaultGlobalSection(dm, &sectionGlobal),
and I obtain a PetscSection nothing like the one I see when I call
PetscSectionView()
on the PetscSection I created above. Does this have anything to do? I tried
to compare this strange PetscSection with the one from the original
DMNetwork, I call DMGetDefaultGlobalSection(dm, &sectionGlobal) before the
first line of the snippet above and I get this error message.

0]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------
[0]PETSC ERROR: Object is in wrong state
[0]PETSC ERROR: DM must have a default PetscSection in order to create a
global PetscSection

Thanks in advance
Miguel


On Mon, Feb 23, 2015 at 3:24 PM, Matthew Knepley <knepley at gmail.com> wrote:

> On Mon, Feb 23, 2015 at 2:15 PM, Miguel Angel Salazar de Troya <
> salazardetroya at gmail.com> wrote:
>
>> Thanks a lot, the partition should be done before setting up the section,
>> right?
>>
>
> The partition will be automatic. All you have to do is make the local
> section. The DM is already partitioned,
> and the Section will inherit that.
>
>   Matt
>
>
>> Miguel
>>
>> On Mon, Feb 23, 2015 at 2:05 PM, Matthew Knepley <knepley at gmail.com>
>> wrote:
>>
>>> On Mon, Feb 23, 2015 at 1:40 PM, Miguel Angel Salazar de Troya <
>>> salazardetroya at gmail.com> wrote:
>>>
>>>> Wouldn't including the edge variables in the global vector make the
>>>> code slower? I'm using the global vector in a TS, using one of the explicit
>>>> RK schemes. The edge variables would not be updated in the RHSFunction
>>>> evaluation. I only change the edge variables in the TSUpdate. If the global
>>>> vector had the edge variables, it would be a much larger vector, and all
>>>> the vector operations performed by the TS would be slower. Although the
>>>> vector F returned by the RHSFunction would be zero in the edge variable
>>>> components. I guess that being the vector sparse that would not be a
>>>> problem.
>>>>
>>>> I think I'm more interested in the PetscSection approach because it
>>>> might require less modifications in my code. However, I don't know how I
>>>> could do this. Maybe something like this?
>>>>
>>>> PetscSectionCreate(PETSC_COMM_WORLD, &s);
>>>> PetscSectionSetNumFields(s, 1);
>>>> PetscSectionSetFieldComponents(s, 0, 1);
>>>>
>>>> // Now to set the chart, I pick the edge range
>>>>
>>>> DMNetworkGetEdgeRange(dm, & eStart, & eEnd
>>>>
>>>> PetscSectionSetChart(s, eStart, eEnd);
>>>>
>>>> for(PetscInt e = eStart; c < eEnd; ++e) {
>>>>      PetscSectionSetDof(s, e, 1);
>>>>      PetscSectionSetFieldDof(s, e, 1, 1);
>>>>
>>>
>>> It should be PetscSectionSetFieldDof(s, e, 0, 1);
>>>
>>>
>>>> }
>>>> PetscSectionSetUp(s);
>>>>
>>>> Now in the manual I see this:
>>>>
>>>
>>> First you want to do:
>>>
>>>   DMClone(dm, &dmEdge);
>>>
>>> and then use dmEdge below.
>>>
>>>
>>>> DMSetDefaultSection(dm, s);
>>>> DMGetLocalVector(dm, &localVec);
>>>> DMGetGlobalVector(dm, &globalVec);
>>>>
>>>> Setting up the default section in the DM would interfere with the
>>>> section already set up with the variables in the vertices?
>>>>
>>>
>>> Yep, thats why you would use a clone.
>>>
>>>   Thanks,
>>>
>>>      Matt
>>>
>>>
>>>> Thanks a lot for your responses.
>>>>
>>>>
>>>>
>>>> On Mon, Feb 23, 2015 at 11:37 AM, Matthew Knepley <knepley at gmail.com>
>>>> wrote:
>>>>
>>>>> On Mon, Feb 23, 2015 at 9:27 AM, Miguel Angel Salazar de Troya <
>>>>> salazardetroya at gmail.com> wrote:
>>>>>
>>>>>> I'm iterating through local edges given in DMNetworkGetEdgeRange().
>>>>>> For each edge, I extract or modify its corresponding value in a global
>>>>>> petsc vector. Therefore that vector must have as many components as edges
>>>>>> there are in the network. To extract the value in the vector, I use
>>>>>> VecGetArray() and a variable counter that is incremented in each iteration.
>>>>>> The array that I obtain in VecGetArray() has to be the same size
>>>>>> than the edge range. That variable counter starts as 0, so if the array
>>>>>> that I obtained in VecGetArray() is x_array, x_array[0] must be the
>>>>>> component in the global vector that corresponds with the start edge given
>>>>>> in DMNetworkGetEdgeRange()
>>>>>>
>>>>>> I need that global petsc vector because I will use it in other
>>>>>> operations, it's not just data. Sorry for the confusion. Thanks in advance.
>>>>>>
>>>>>
>>>>> This sounds like an assembly operation. The usual paradigm is to
>>>>> compute in the local space, and then communicate to get to the global
>>>>> space. So you would make a PetscSection that had 1 (or some) unknowns on
>>>>> each cell (edge) and then you can use DMCreateGlobal/LocalVector() and
>>>>> DMLocalToGlobal() to do this.
>>>>>
>>>>> Does that make sense?
>>>>>
>>>>>   Thanks,
>>>>>
>>>>>      Matt
>>>>>
>>>>>
>>>>>> Miguel
>>>>>>
>>>>>>
>>>>>> On Mon, Feb 23, 2015 at 9:09 AM, Matthew Knepley <knepley at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> On Mon, Feb 23, 2015 at 8:42 AM, Miguel Angel Salazar de Troya <
>>>>>>> salazardetroya at gmail.com> wrote:
>>>>>>>
>>>>>>>> Thanks, that will help me. Now what I would like to have is the
>>>>>>>> following: if I have two processors and ten edges, the partitioning results
>>>>>>>> in the first processor having the edges 0-4 and the second processor, the
>>>>>>>> edges 5-9. I also have a global vector with as many components as edges,
>>>>>>>> 10. How can I partition it so the first processor also has the 0-4
>>>>>>>> components and the second, the 5-9 components of the vector?
>>>>>>>>
>>>>>>> I think it would help to know what you want to accomplish. This is
>>>>>>> how you are proposing to do it.'
>>>>>>>
>>>>>>> If you just want to put data on edges, DMNetwork has a facility for
>>>>>>> that already.
>>>>>>>
>>>>>>>   Thanks,
>>>>>>>
>>>>>>>      Matt
>>>>>>>
>>>>>>>
>>>>>>>> Miguel
>>>>>>>> On Feb 23, 2015 8:08 AM, "Abhyankar, Shrirang G." <
>>>>>>>> abhyshr at mcs.anl.gov> wrote:
>>>>>>>>
>>>>>>>>>  Miguel,
>>>>>>>>>    One possible way is to store the global numbering of any
>>>>>>>>> edge/vertex in the "component" attached to it. Once the mesh gets
>>>>>>>>> partitioned, the components are also distributed so you can easily retrieve
>>>>>>>>> the global number of any edge/vertex by accessing its component. This is
>>>>>>>>> what is done in the DMNetwork example pf.c although the global numbering is
>>>>>>>>> not used for anything.
>>>>>>>>>
>>>>>>>>>  Shri
>>>>>>>>>  From: Matthew Knepley <knepley at gmail.com>
>>>>>>>>> Date: Mon, 23 Feb 2015 07:54:34 -0600
>>>>>>>>> To: Miguel Angel Salazar de Troya <salazardetroya at gmail.com>
>>>>>>>>> Cc: "petsc-users at mcs.anl.gov" <petsc-users at mcs.anl.gov>
>>>>>>>>> Subject: Re: [petsc-users] DMNetworkGetEdgeRange() in parallel
>>>>>>>>>
>>>>>>>>>   On Sun, Feb 22, 2015 at 3:59 PM, Miguel Angel Salazar de Troya <
>>>>>>>>> salazardetroya at gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Thanks. Once I obtain that Index Set with the routine DMPlexCreateCellNumbering()
>>>>>>>>>> (I assume that the edges in DMNetwork correspond to cells in DMPlex) can I
>>>>>>>>>> use it to partition a vector with as many components as edges I have in my
>>>>>>>>>> network?
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>  I do not completely understand the question.
>>>>>>>>>
>>>>>>>>>  If you want a partition of the edges, you can use
>>>>>>>>> DMPlexCreatePartition() and its friend DMPlexDistribute(). What
>>>>>>>>> are you trying to do?
>>>>>>>>>
>>>>>>>>>     Matt
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>  Thanks
>>>>>>>>>> Miguel
>>>>>>>>>>
>>>>>>>>>> On Sun, Feb 22, 2015 at 12:15 PM, Matthew Knepley <
>>>>>>>>>> knepley at gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>>  On Sun, Feb 22, 2015 at 11:01 AM, Miguel Angel Salazar de
>>>>>>>>>>> Troya <salazardetroya at gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi
>>>>>>>>>>>>
>>>>>>>>>>>>  I noticed that the routine DMNetworkGetEdgeRange() returns
>>>>>>>>>>>> the local indices for the edge range. Is there any way to obtain the global
>>>>>>>>>>>> indices? So if my network has 10 edges, the processor 1 has the 0-4 edges
>>>>>>>>>>>> and the processor 2, the 5-9 edges, how can I obtain this information?
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>  One of the points of DMPlex is we do not require a global
>>>>>>>>>>> numbering. Everything is numbered
>>>>>>>>>>> locally, and the PetscSF maps local numbers to local numbers in
>>>>>>>>>>> order to determine ownership.
>>>>>>>>>>>
>>>>>>>>>>>  If you want to create a global numbering for some reason, you
>>>>>>>>>>> can using DMPlexCreatePointNumbering().
>>>>>>>>>>> There are also cell and vertex versions that we use for output,
>>>>>>>>>>> so you could do it just for edges as well.
>>>>>>>>>>>
>>>>>>>>>>>    Thanks,
>>>>>>>>>>>
>>>>>>>>>>>       Matt
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>  Thanks
>>>>>>>>>>>>  Miguel
>>>>>>>>>>>>
>>>>>>>>>>>>  --
>>>>>>>>>>>>  *Miguel Angel Salazar de Troya*
>>>>>>>>>>>> Graduate Research Assistant
>>>>>>>>>>>> Department of Mechanical Science and Engineering
>>>>>>>>>>>> University of Illinois at Urbana-Champaign
>>>>>>>>>>>> (217) 550-2360
>>>>>>>>>>>> salaza11 at illinois.edu
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>  --
>>>>>>>>>>> What most experimenters take for granted before they begin their
>>>>>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>>>>>> experiments lead.
>>>>>>>>>>> -- Norbert Wiener
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>  --
>>>>>>>>>>  *Miguel Angel Salazar de Troya*
>>>>>>>>>> Graduate Research Assistant
>>>>>>>>>> Department of Mechanical Science and Engineering
>>>>>>>>>> University of Illinois at Urbana-Champaign
>>>>>>>>>> (217) 550-2360
>>>>>>>>>> salaza11 at illinois.edu
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>  --
>>>>>>>>> What most experimenters take for granted before they begin their
>>>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>>>> experiments lead.
>>>>>>>>> -- Norbert Wiener
>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> What most experimenters take for granted before they begin their
>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>> experiments lead.
>>>>>>> -- Norbert Wiener
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> *Miguel Angel Salazar de Troya*
>>>>>> Graduate Research Assistant
>>>>>> Department of Mechanical Science and Engineering
>>>>>> University of Illinois at Urbana-Champaign
>>>>>> (217) 550-2360
>>>>>> salaza11 at illinois.edu
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> What most experimenters take for granted before they begin their
>>>>> experiments is infinitely more interesting than any results to which their
>>>>> experiments lead.
>>>>> -- Norbert Wiener
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> *Miguel Angel Salazar de Troya*
>>>> Graduate Research Assistant
>>>> Department of Mechanical Science and Engineering
>>>> University of Illinois at Urbana-Champaign
>>>> (217) 550-2360
>>>> salaza11 at illinois.edu
>>>>
>>>>
>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>
>>
>>
>> --
>> *Miguel Angel Salazar de Troya*
>> Graduate Research Assistant
>> Department of Mechanical Science and Engineering
>> University of Illinois at Urbana-Champaign
>> (217) 550-2360
>> salaza11 at illinois.edu
>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>



-- 
*Miguel Angel Salazar de Troya*
Graduate Research Assistant
Department of Mechanical Science and Engineering
University of Illinois at Urbana-Champaign
(217) 550-2360
salaza11 at illinois.edu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150224/7a5c36b3/attachment-0001.html>

More information about the petsc-users mailing list