[petsc-users] DMNetworkGetEdgeRange() in parallel
Matthew Knepley
knepley at gmail.com
Wed Feb 25 10:12:43 CST 2015
On Wed, Feb 25, 2015 at 9:59 AM, Abhyankar, Shrirang G. <abhyshr at mcs.anl.gov
> wrote:
> Miguel,
> I'm a bit tied up today. I'll try to debug this issue tomorrow and get
> back to you.
>
The problem is the way that DMNetwork is using the default section. When
DMCreateGlobalVec() is called,
it uses the default section for its included network->plex, but
DMSetDefaultSection() sets the section associated
with network. This is inconsistent.
I am sending the code which I hacked to show the correct result by using
the private header.
I think that
1) The underlying Plex should be exposed by DMNetworkGetPlex()
2) The default section business should be made consistent
Thanks,
Matt
> Thanks,
> Shri
>
> From: Miguel Angel Salazar de Troya <salazardetroya at gmail.com>
> Date: Wed, 25 Feb 2015 08:48:10 -0600
> To: Matthew Knepley <knepley at gmail.com>
> Cc: Shri <abhyshr at mcs.anl.gov>, "petsc-users at mcs.anl.gov" <
> petsc-users at mcs.anl.gov>
>
> Subject: Re: [petsc-users] DMNetworkGetEdgeRange() in parallel
>
> I modified the DMNetwork example to include the new DM with the
> modified section. It has the same problems. Please find attached the code
> to this email.
>
> Thanks
>
> On Tue, Feb 24, 2015 at 6:49 PM, Matthew Knepley <knepley at gmail.com>
> wrote:
>
>> On Tue, Feb 24, 2015 at 6:42 PM, Miguel Angel Salazar de Troya <
>> salazardetroya at gmail.com> wrote:
>>
>>> I implemented the code as agreed, but I don't get the results I
>>> expected. When I create the vector with DMCreateGlobalVector(), I obtain a
>>> vector with a layout similar to the original DMNetwork, instead of the
>>> cloned network with the new PetscSection. The code is as follows:
>>>
>>> DMClone(dm, &dmEdge);
>>>
>>> PetscSectionCreate(PETSC_COMM_WORLD, &s);
>>> PetscSectionSetNumFields(s, 1);
>>> PetscSectionSetFieldComponents(s, 0, 1);
>>>
>>> // Now to set the chart, I pick the edge range
>>>
>>> DMNetworkGetEdgeRange(dmEdge, & eStart, & eEnd)
>>>
>>> PetscSectionSetChart(s, eStart, eEnd);
>>>
>>> for(PetscInt e = eStart; c < eEnd; ++e) {
>>> PetscSectionSetDof(s, e, 1);
>>> PetscSectionSetFieldDof(s, e, 0, 1);
>>> }
>>> PetscSectionSetUp(s);
>>>
>>> DMSetDefaultSection(dmEdge s);
>>> DMCreateGlobalVector(dmEdge, &globalVec);
>>>
>>> When I get into DMCreateGlobalVector(dmEdge, &globalVec) in the
>>> debugger, in the function DMCreateSubDM_Section_Private() I call
>>> PetscSectionView() on the section
>>>
>>
>> I have no idea why you would be in DMCreateSubDM().
>>
>> Just view globalVec. If the code is as above, it will give you a vector
>> with that layout. If not
>> it should be trivial to make a small code and send it. I do this
>> everywhere is PETSc, so the
>> basic mechanism certainly works.
>>
>> Thanks,
>>
>> Matt
>>
>>
>>> obtained by DMGetDefaultGlobalSection(dm, §ionGlobal), and I
>>> obtain a PetscSection nothing like the one I see when I call PetscSectionView()
>>> on the PetscSection I created above. Does this have anything to do? I
>>> tried to compare this strange PetscSection with the one from the original
>>> DMNetwork, I call DMGetDefaultGlobalSection(dm, §ionGlobal) before
>>> the first line of the snippet above and I get this error message.
>>>
>>> 0]PETSC ERROR: --------------------- Error Message
>>> --------------------------------------------------------------
>>> [0]PETSC ERROR: Object is in wrong state
>>> [0]PETSC ERROR: DM must have a default PetscSection in order to create a
>>> global PetscSection
>>>
>>> Thanks in advance
>>> Miguel
>>>
>>>
>>> On Mon, Feb 23, 2015 at 3:24 PM, Matthew Knepley <knepley at gmail.com>
>>> wrote:
>>>
>>>> On Mon, Feb 23, 2015 at 2:15 PM, Miguel Angel Salazar de Troya <
>>>> salazardetroya at gmail.com> wrote:
>>>>
>>>>> Thanks a lot, the partition should be done before setting up the
>>>>> section, right?
>>>>>
>>>>
>>>> The partition will be automatic. All you have to do is make the local
>>>> section. The DM is already partitioned,
>>>> and the Section will inherit that.
>>>>
>>>> Matt
>>>>
>>>>
>>>>> Miguel
>>>>>
>>>>> On Mon, Feb 23, 2015 at 2:05 PM, Matthew Knepley <knepley at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> On Mon, Feb 23, 2015 at 1:40 PM, Miguel Angel Salazar de Troya <
>>>>>> salazardetroya at gmail.com> wrote:
>>>>>>
>>>>>>> Wouldn't including the edge variables in the global vector make the
>>>>>>> code slower? I'm using the global vector in a TS, using one of the explicit
>>>>>>> RK schemes. The edge variables would not be updated in the RHSFunction
>>>>>>> evaluation. I only change the edge variables in the TSUpdate. If the global
>>>>>>> vector had the edge variables, it would be a much larger vector, and all
>>>>>>> the vector operations performed by the TS would be slower. Although the
>>>>>>> vector F returned by the RHSFunction would be zero in the edge variable
>>>>>>> components. I guess that being the vector sparse that would not be a
>>>>>>> problem.
>>>>>>>
>>>>>>> I think I'm more interested in the PetscSection approach because
>>>>>>> it might require less modifications in my code. However, I don't know how I
>>>>>>> could do this. Maybe something like this?
>>>>>>>
>>>>>>> PetscSectionCreate(PETSC_COMM_WORLD, &s);
>>>>>>> PetscSectionSetNumFields(s, 1);
>>>>>>> PetscSectionSetFieldComponents(s, 0, 1);
>>>>>>>
>>>>>>> // Now to set the chart, I pick the edge range
>>>>>>>
>>>>>>> DMNetworkGetEdgeRange(dm, & eStart, & eEnd
>>>>>>>
>>>>>>> PetscSectionSetChart(s, eStart, eEnd);
>>>>>>>
>>>>>>> for(PetscInt e = eStart; c < eEnd; ++e) {
>>>>>>> PetscSectionSetDof(s, e, 1);
>>>>>>> PetscSectionSetFieldDof(s, e, 1, 1);
>>>>>>>
>>>>>>
>>>>>> It should be PetscSectionSetFieldDof(s, e, 0, 1);
>>>>>>
>>>>>>
>>>>>>> }
>>>>>>> PetscSectionSetUp(s);
>>>>>>>
>>>>>>> Now in the manual I see this:
>>>>>>>
>>>>>>
>>>>>> First you want to do:
>>>>>>
>>>>>> DMClone(dm, &dmEdge);
>>>>>>
>>>>>> and then use dmEdge below.
>>>>>>
>>>>>>
>>>>>>> DMSetDefaultSection(dm, s);
>>>>>>> DMGetLocalVector(dm, &localVec);
>>>>>>> DMGetGlobalVector(dm, &globalVec);
>>>>>>>
>>>>>>> Setting up the default section in the DM would interfere with the
>>>>>>> section already set up with the variables in the vertices?
>>>>>>>
>>>>>>
>>>>>> Yep, thats why you would use a clone.
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Matt
>>>>>>
>>>>>>
>>>>>>> Thanks a lot for your responses.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Feb 23, 2015 at 11:37 AM, Matthew Knepley <knepley at gmail.com
>>>>>>> > wrote:
>>>>>>>
>>>>>>>> On Mon, Feb 23, 2015 at 9:27 AM, Miguel Angel Salazar de Troya <
>>>>>>>> salazardetroya at gmail.com> wrote:
>>>>>>>>
>>>>>>>>> I'm iterating through local edges given in DMNetworkGetEdgeRange().
>>>>>>>>> For each edge, I extract or modify its corresponding value in a global
>>>>>>>>> petsc vector. Therefore that vector must have as many components as edges
>>>>>>>>> there are in the network. To extract the value in the vector, I use
>>>>>>>>> VecGetArray() and a variable counter that is incremented in each iteration.
>>>>>>>>> The array that I obtain in VecGetArray() has to be the same size
>>>>>>>>> than the edge range. That variable counter starts as 0, so if the array
>>>>>>>>> that I obtained in VecGetArray() is x_array, x_array[0] must be
>>>>>>>>> the component in the global vector that corresponds with the start edge
>>>>>>>>> given in DMNetworkGetEdgeRange()
>>>>>>>>>
>>>>>>>>> I need that global petsc vector because I will use it in other
>>>>>>>>> operations, it's not just data. Sorry for the confusion. Thanks in advance.
>>>>>>>>>
>>>>>>>>
>>>>>>>> This sounds like an assembly operation. The usual paradigm is to
>>>>>>>> compute in the local space, and then communicate to get to the global
>>>>>>>> space. So you would make a PetscSection that had 1 (or some) unknowns on
>>>>>>>> each cell (edge) and then you can use DMCreateGlobal/LocalVector() and
>>>>>>>> DMLocalToGlobal() to do this.
>>>>>>>>
>>>>>>>> Does that make sense?
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>>
>>>>>>>> Matt
>>>>>>>>
>>>>>>>>
>>>>>>>>> Miguel
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mon, Feb 23, 2015 at 9:09 AM, Matthew Knepley <
>>>>>>>>> knepley at gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> On Mon, Feb 23, 2015 at 8:42 AM, Miguel Angel Salazar de Troya <
>>>>>>>>>> salazardetroya at gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Thanks, that will help me. Now what I would like to have is the
>>>>>>>>>>> following: if I have two processors and ten edges, the partitioning results
>>>>>>>>>>> in the first processor having the edges 0-4 and the second processor, the
>>>>>>>>>>> edges 5-9. I also have a global vector with as many components as edges,
>>>>>>>>>>> 10. How can I partition it so the first processor also has the 0-4
>>>>>>>>>>> components and the second, the 5-9 components of the vector?
>>>>>>>>>>>
>>>>>>>>>> I think it would help to know what you want to accomplish. This
>>>>>>>>>> is how you are proposing to do it.'
>>>>>>>>>>
>>>>>>>>>> If you just want to put data on edges, DMNetwork has a facility
>>>>>>>>>> for that already.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>>
>>>>>>>>>> Matt
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> Miguel
>>>>>>>>>>> On Feb 23, 2015 8:08 AM, "Abhyankar, Shrirang G." <
>>>>>>>>>>> abhyshr at mcs.anl.gov> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Miguel,
>>>>>>>>>>>> One possible way is to store the global numbering of any
>>>>>>>>>>>> edge/vertex in the "component" attached to it. Once the mesh gets
>>>>>>>>>>>> partitioned, the components are also distributed so you can easily retrieve
>>>>>>>>>>>> the global number of any edge/vertex by accessing its component. This is
>>>>>>>>>>>> what is done in the DMNetwork example pf.c although the global numbering is
>>>>>>>>>>>> not used for anything.
>>>>>>>>>>>>
>>>>>>>>>>>> Shri
>>>>>>>>>>>> From: Matthew Knepley <knepley at gmail.com>
>>>>>>>>>>>> Date: Mon, 23 Feb 2015 07:54:34 -0600
>>>>>>>>>>>> To: Miguel Angel Salazar de Troya <salazardetroya at gmail.com>
>>>>>>>>>>>> Cc: "petsc-users at mcs.anl.gov" <petsc-users at mcs.anl.gov>
>>>>>>>>>>>> Subject: Re: [petsc-users] DMNetworkGetEdgeRange() in parallel
>>>>>>>>>>>>
>>>>>>>>>>>> On Sun, Feb 22, 2015 at 3:59 PM, Miguel Angel Salazar de
>>>>>>>>>>>> Troya <salazardetroya at gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks. Once I obtain that Index Set with the routine DMPlexCreateCellNumbering()
>>>>>>>>>>>>> (I assume that the edges in DMNetwork correspond to cells in DMPlex) can I
>>>>>>>>>>>>> use it to partition a vector with as many components as edges I have in my
>>>>>>>>>>>>> network?
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> I do not completely understand the question.
>>>>>>>>>>>>
>>>>>>>>>>>> If you want a partition of the edges, you can use
>>>>>>>>>>>> DMPlexCreatePartition() and its friend DMPlexDistribute(). What
>>>>>>>>>>>> are you trying to do?
>>>>>>>>>>>>
>>>>>>>>>>>> Matt
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>> Miguel
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Sun, Feb 22, 2015 at 12:15 PM, Matthew Knepley <
>>>>>>>>>>>>> knepley at gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Sun, Feb 22, 2015 at 11:01 AM, Miguel Angel Salazar de
>>>>>>>>>>>>>> Troya <salazardetroya at gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I noticed that the routine DMNetworkGetEdgeRange() returns
>>>>>>>>>>>>>>> the local indices for the edge range. Is there any way to obtain the global
>>>>>>>>>>>>>>> indices? So if my network has 10 edges, the processor 1 has the 0-4 edges
>>>>>>>>>>>>>>> and the processor 2, the 5-9 edges, how can I obtain this information?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> One of the points of DMPlex is we do not require a global
>>>>>>>>>>>>>> numbering. Everything is numbered
>>>>>>>>>>>>>> locally, and the PetscSF maps local numbers to local numbers
>>>>>>>>>>>>>> in order to determine ownership.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> If you want to create a global numbering for some reason,
>>>>>>>>>>>>>> you can using DMPlexCreatePointNumbering().
>>>>>>>>>>>>>> There are also cell and vertex versions that we use for
>>>>>>>>>>>>>> output, so you could do it just for edges as well.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Matt
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>> Miguel
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> *Miguel Angel Salazar de Troya*
>>>>>>>>>>>>>>> Graduate Research Assistant
>>>>>>>>>>>>>>> Department of Mechanical Science and Engineering
>>>>>>>>>>>>>>> University of Illinois at Urbana-Champaign
>>>>>>>>>>>>>>> (217) 550-2360
>>>>>>>>>>>>>>> salaza11 at illinois.edu
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> What most experimenters take for granted before they begin
>>>>>>>>>>>>>> their experiments is infinitely more interesting than any results to which
>>>>>>>>>>>>>> their experiments lead.
>>>>>>>>>>>>>> -- Norbert Wiener
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> *Miguel Angel Salazar de Troya*
>>>>>>>>>>>>> Graduate Research Assistant
>>>>>>>>>>>>> Department of Mechanical Science and Engineering
>>>>>>>>>>>>> University of Illinois at Urbana-Champaign
>>>>>>>>>>>>> (217) 550-2360
>>>>>>>>>>>>> salaza11 at illinois.edu
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> What most experimenters take for granted before they begin
>>>>>>>>>>>> their experiments is infinitely more interesting than any results to which
>>>>>>>>>>>> their experiments lead.
>>>>>>>>>>>> -- Norbert Wiener
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> What most experimenters take for granted before they begin their
>>>>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>>>>> experiments lead.
>>>>>>>>>> -- Norbert Wiener
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> *Miguel Angel Salazar de Troya*
>>>>>>>>> Graduate Research Assistant
>>>>>>>>> Department of Mechanical Science and Engineering
>>>>>>>>> University of Illinois at Urbana-Champaign
>>>>>>>>> (217) 550-2360
>>>>>>>>> salaza11 at illinois.edu
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> What most experimenters take for granted before they begin their
>>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>>> experiments lead.
>>>>>>>> -- Norbert Wiener
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> *Miguel Angel Salazar de Troya*
>>>>>>> Graduate Research Assistant
>>>>>>> Department of Mechanical Science and Engineering
>>>>>>> University of Illinois at Urbana-Champaign
>>>>>>> (217) 550-2360
>>>>>>> salaza11 at illinois.edu
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> What most experimenters take for granted before they begin their
>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>> experiments lead.
>>>>>> -- Norbert Wiener
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> *Miguel Angel Salazar de Troya*
>>>>> Graduate Research Assistant
>>>>> Department of Mechanical Science and Engineering
>>>>> University of Illinois at Urbana-Champaign
>>>>> (217) 550-2360
>>>>> salaza11 at illinois.edu
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>
>>>
>>>
>>> --
>>> *Miguel Angel Salazar de Troya*
>>> Graduate Research Assistant
>>> Department of Mechanical Science and Engineering
>>> University of Illinois at Urbana-Champaign
>>> (217) 550-2360
>>> salaza11 at illinois.edu
>>>
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>
>
>
> --
> *Miguel Angel Salazar de Troya*
> Graduate Research Assistant
> Department of Mechanical Science and Engineering
> University of Illinois at Urbana-Champaign
> (217) 550-2360
> salaza11 at illinois.edu
>
>
--
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150225/30276cba/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: pf.c
Type: text/x-csrc
Size: 23826 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150225/30276cba/attachment-0001.c>
More information about the petsc-users
mailing list