[petsc-users] DMNetworkGetEdgeRange() in parallel
Abhyankar, Shrirang G.
abhyshr at mcs.anl.gov
Wed Feb 25 10:56:58 CST 2015
Matt,
Thanks for debugging. I'll try to fix this tomorrow.
Shri
From: Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>>
Date: Wed, 25 Feb 2015 10:12:43 -0600
To: Shri <abhyshr at mcs.anl.gov<mailto:abhyshr at mcs.anl.gov>>
Cc: Miguel Angel Salazar de Troya <salazardetroya at gmail.com<mailto:salazardetroya at gmail.com>>, "petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>" <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] DMNetworkGetEdgeRange() in parallel
On Wed, Feb 25, 2015 at 9:59 AM, Abhyankar, Shrirang G. <abhyshr at mcs.anl.gov<mailto:abhyshr at mcs.anl.gov>> wrote:
Miguel,
I'm a bit tied up today. I'll try to debug this issue tomorrow and get back to you.
The problem is the way that DMNetwork is using the default section. When DMCreateGlobalVec() is called,
it uses the default section for its included network->plex, but DMSetDefaultSection() sets the section associated
with network. This is inconsistent.
I am sending the code which I hacked to show the correct result by using the private header.
I think that
1) The underlying Plex should be exposed by DMNetworkGetPlex()
2) The default section business should be made consistent
Thanks,
Matt
Thanks,
Shri
From: Miguel Angel Salazar de Troya <salazardetroya at gmail.com<mailto:salazardetroya at gmail.com>>
Date: Wed, 25 Feb 2015 08:48:10 -0600
To: Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>>
Cc: Shri <abhyshr at mcs.anl.gov<mailto:abhyshr at mcs.anl.gov>>, "petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>" <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] DMNetworkGetEdgeRange() in parallel
I modified the DMNetwork example to include the new DM with the modified section. It has the same problems. Please find attached the code to this email.
Thanks
On Tue, Feb 24, 2015 at 6:49 PM, Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>> wrote:
On Tue, Feb 24, 2015 at 6:42 PM, Miguel Angel Salazar de Troya <salazardetroya at gmail.com<mailto:salazardetroya at gmail.com>> wrote:
I implemented the code as agreed, but I don't get the results I expected. When I create the vector with DMCreateGlobalVector(), I obtain a vector with a layout similar to the original DMNetwork, instead of the cloned network with the new PetscSection. The code is as follows:
DMClone(dm, &dmEdge);
PetscSectionCreate(PETSC_COMM_WORLD, &s);
PetscSectionSetNumFields(s, 1);
PetscSectionSetFieldComponents(s, 0, 1);
// Now to set the chart, I pick the edge range
DMNetworkGetEdgeRange(dmEdge, & eStart, & eEnd)
PetscSectionSetChart(s, eStart, eEnd);
for(PetscInt e = eStart; c < eEnd; ++e) {
PetscSectionSetDof(s, e, 1);
PetscSectionSetFieldDof(s, e, 0, 1);
}
PetscSectionSetUp(s);
DMSetDefaultSection(dmEdge s);
DMCreateGlobalVector(dmEdge, &globalVec);
When I get into DMCreateGlobalVector(dmEdge, &globalVec) in the debugger, in the function DMCreateSubDM_Section_Private() I call PetscSectionView() on the section
I have no idea why you would be in DMCreateSubDM().
Just view globalVec. If the code is as above, it will give you a vector with that layout. If not
it should be trivial to make a small code and send it. I do this everywhere is PETSc, so the
basic mechanism certainly works.
Thanks,
Matt
obtained by DMGetDefaultGlobalSection(dm, §ionGlobal), and I obtain a PetscSection nothing like the one I see when I call PetscSectionView() on the PetscSection I created above. Does this have anything to do? I tried to compare this strange PetscSection with the one from the original DMNetwork, I call DMGetDefaultGlobalSection(dm, §ionGlobal) before the first line of the snippet above and I get this error message.
0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Object is in wrong state
[0]PETSC ERROR: DM must have a default PetscSection in order to create a global PetscSection
Thanks in advance
Miguel
On Mon, Feb 23, 2015 at 3:24 PM, Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>> wrote:
On Mon, Feb 23, 2015 at 2:15 PM, Miguel Angel Salazar de Troya <salazardetroya at gmail.com<mailto:salazardetroya at gmail.com>> wrote:
Thanks a lot, the partition should be done before setting up the section, right?
The partition will be automatic. All you have to do is make the local section. The DM is already partitioned,
and the Section will inherit that.
Matt
Miguel
On Mon, Feb 23, 2015 at 2:05 PM, Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>> wrote:
On Mon, Feb 23, 2015 at 1:40 PM, Miguel Angel Salazar de Troya <salazardetroya at gmail.com<mailto:salazardetroya at gmail.com>> wrote:
Wouldn't including the edge variables in the global vector make the code slower? I'm using the global vector in a TS, using one of the explicit RK schemes. The edge variables would not be updated in the RHSFunction evaluation. I only change the edge variables in the TSUpdate. If the global vector had the edge variables, it would be a much larger vector, and all the vector operations performed by the TS would be slower. Although the vector F returned by the RHSFunction would be zero in the edge variable components. I guess that being the vector sparse that would not be a problem.
I think I'm more interested in the PetscSection approach because it might require less modifications in my code. However, I don't know how I could do this. Maybe something like this?
PetscSectionCreate(PETSC_COMM_WORLD, &s);
PetscSectionSetNumFields(s, 1);
PetscSectionSetFieldComponents(s, 0, 1);
// Now to set the chart, I pick the edge range
DMNetworkGetEdgeRange(dm, & eStart, & eEnd
PetscSectionSetChart(s, eStart, eEnd);
for(PetscInt e = eStart; c < eEnd; ++e) {
PetscSectionSetDof(s, e, 1);
PetscSectionSetFieldDof(s, e, 1, 1);
It should be PetscSectionSetFieldDof(s, e, 0, 1);
}
PetscSectionSetUp(s);
Now in the manual I see this:
First you want to do:
DMClone(dm, &dmEdge);
and then use dmEdge below.
DMSetDefaultSection(dm, s);
DMGetLocalVector(dm, &localVec);
DMGetGlobalVector(dm, &globalVec);
Setting up the default section in the DM would interfere with the section already set up with the variables in the vertices?
Yep, thats why you would use a clone.
Thanks,
Matt
Thanks a lot for your responses.
On Mon, Feb 23, 2015 at 11:37 AM, Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>> wrote:
On Mon, Feb 23, 2015 at 9:27 AM, Miguel Angel Salazar de Troya <salazardetroya at gmail.com<mailto:salazardetroya at gmail.com>> wrote:
I'm iterating through local edges given in DMNetworkGetEdgeRange(). For each edge, I extract or modify its corresponding value in a global petsc vector. Therefore that vector must have as many components as edges there are in the network. To extract the value in the vector, I use VecGetArray() and a variable counter that is incremented in each iteration. The array that I obtain in VecGetArray() has to be the same size than the edge range. That variable counter starts as 0, so if the array that I obtained in VecGetArray() is x_array, x_array[0] must be the component in the global vector that corresponds with the start edge given in DMNetworkGetEdgeRange()
I need that global petsc vector because I will use it in other operations, it's not just data. Sorry for the confusion. Thanks in advance.
This sounds like an assembly operation. The usual paradigm is to compute in the local space, and then communicate to get to the global space. So you would make a PetscSection that had 1 (or some) unknowns on each cell (edge) and then you can use DMCreateGlobal/LocalVector() and DMLocalToGlobal() to do this.
Does that make sense?
Thanks,
Matt
Miguel
On Mon, Feb 23, 2015 at 9:09 AM, Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>> wrote:
On Mon, Feb 23, 2015 at 8:42 AM, Miguel Angel Salazar de Troya <salazardetroya at gmail.com<mailto:salazardetroya at gmail.com>> wrote:
Thanks, that will help me. Now what I would like to have is the following: if I have two processors and ten edges, the partitioning results in the first processor having the edges 0-4 and the second processor, the edges 5-9. I also have a global vector with as many components as edges, 10. How can I partition it so the first processor also has the 0-4 components and the second, the 5-9 components of the vector?
I think it would help to know what you want to accomplish. This is how you are proposing to do it.'
If you just want to put data on edges, DMNetwork has a facility for that already.
Thanks,
Matt
Miguel
On Feb 23, 2015 8:08 AM, "Abhyankar, Shrirang G." <abhyshr at mcs.anl.gov<mailto:abhyshr at mcs.anl.gov>> wrote:
Miguel,
One possible way is to store the global numbering of any edge/vertex in the "component" attached to it. Once the mesh gets partitioned, the components are also distributed so you can easily retrieve the global number of any edge/vertex by accessing its component. This is what is done in the DMNetwork example pf.c although the global numbering is not used for anything.
Shri
From: Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>>
Date: Mon, 23 Feb 2015 07:54:34 -0600
To: Miguel Angel Salazar de Troya <salazardetroya at gmail.com<mailto:salazardetroya at gmail.com>>
Cc: "petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>" <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] DMNetworkGetEdgeRange() in parallel
On Sun, Feb 22, 2015 at 3:59 PM, Miguel Angel Salazar de Troya <salazardetroya at gmail.com<mailto:salazardetroya at gmail.com>> wrote:
Thanks. Once I obtain that Index Set with the routine DMPlexCreateCellNumbering() (I assume that the edges in DMNetwork correspond to cells in DMPlex) can I use it to partition a vector with as many components as edges I have in my network?
I do not completely understand the question.
If you want a partition of the edges, you can use DMPlexCreatePartition() and its friend DMPlexDistribute(). What
are you trying to do?
Matt
Thanks
Miguel
On Sun, Feb 22, 2015 at 12:15 PM, Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>> wrote:
On Sun, Feb 22, 2015 at 11:01 AM, Miguel Angel Salazar de Troya <salazardetroya at gmail.com<mailto:salazardetroya at gmail.com>> wrote:
Hi
I noticed that the routine DMNetworkGetEdgeRange() returns the local indices for the edge range. Is there any way to obtain the global indices? So if my network has 10 edges, the processor 1 has the 0-4 edges and the processor 2, the 5-9 edges, how can I obtain this information?
One of the points of DMPlex is we do not require a global numbering. Everything is numbered
locally, and the PetscSF maps local numbers to local numbers in order to determine ownership.
If you want to create a global numbering for some reason, you can using DMPlexCreatePointNumbering().
There are also cell and vertex versions that we use for output, so you could do it just for edges as well.
Thanks,
Matt
Thanks
Miguel
--
Miguel Angel Salazar de Troya
Graduate Research Assistant
Department of Mechanical Science and Engineering
University of Illinois at Urbana-Champaign
(217) 550-2360<tel:%28217%29%20550-2360>
salaza11 at illinois.edu<mailto:salaza11 at illinois.edu>
--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener
--
Miguel Angel Salazar de Troya
Graduate Research Assistant
Department of Mechanical Science and Engineering
University of Illinois at Urbana-Champaign
(217) 550-2360<tel:%28217%29%20550-2360>
salaza11 at illinois.edu<mailto:salaza11 at illinois.edu>
--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener
--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener
--
Miguel Angel Salazar de Troya
Graduate Research Assistant
Department of Mechanical Science and Engineering
University of Illinois at Urbana-Champaign
(217) 550-2360<tel:%28217%29%20550-2360>
salaza11 at illinois.edu<mailto:salaza11 at illinois.edu>
--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener
--
Miguel Angel Salazar de Troya
Graduate Research Assistant
Department of Mechanical Science and Engineering
University of Illinois at Urbana-Champaign
(217) 550-2360<tel:%28217%29%20550-2360>
salaza11 at illinois.edu<mailto:salaza11 at illinois.edu>
--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener
--
Miguel Angel Salazar de Troya
Graduate Research Assistant
Department of Mechanical Science and Engineering
University of Illinois at Urbana-Champaign
(217) 550-2360<tel:%28217%29%20550-2360>
salaza11 at illinois.edu<mailto:salaza11 at illinois.edu>
--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener
--
Miguel Angel Salazar de Troya
Graduate Research Assistant
Department of Mechanical Science and Engineering
University of Illinois at Urbana-Champaign
(217) 550-2360<tel:%28217%29%20550-2360>
salaza11 at illinois.edu<mailto:salaza11 at illinois.edu>
--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener
--
Miguel Angel Salazar de Troya
Graduate Research Assistant
Department of Mechanical Science and Engineering
University of Illinois at Urbana-Champaign
(217) 550-2360<tel:%28217%29%20550-2360>
salaza11 at illinois.edu<mailto:salaza11 at illinois.edu>
--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150225/c65c0c80/attachment-0001.html>
More information about the petsc-users
mailing list