[MOAB-dev] Performance issues for finite-volume computations
Olivier Jamond
olivier.jamond at cea.fr
Fri Jun 21 05:43:37 CDT 2019
Thanks you all for your answers. I will try to cache the computed
connectivities as tags, this seems to be a very good idea!
Vijay, I may be wrong, but I feel that the sample code in my initial
message shows that the cell<->face adjacency queries after the "offline"
stage (those for which "create_if_missing" is set to false) do not
retrieve the adjacencies directly, but pass through a "vertex
intermediate step" (even the second time a call them). I paste the
sample code with some added comments...
Navamita, if the "tag trick" proves to be efficient in our simulation, I
think that this will be ok, at least in a first time. But maybe in a
quite near future, memory consumption will become more targeted, and
then maybe we will need AHF. At this time, if you are ok, I will come
back to you!
Thanks again!
Best,
Olivier
#include <iostream>
#include <sstream>
#include <vector>
#include "moab/Core.hpp"
//
// 3 ----- 5 ----- 5
// | | |
// | | |
// | | |
// 0 ----- 1 ----- 4
//
int main()
{
std::vector<moab::EntityHandle> adj;
// > init moab
moab::Core mb;
mb.set_dimension(2);
// <
// > create the vertices (foo coordinates)
double coords[3]={0.,0.,0.};
std::vector<moab::EntityHandle> nodes;
nodes.resize(6);
mb.create_vertex(coords, nodes[0]);
mb.create_vertex(coords, nodes[1]);
mb.create_vertex(coords, nodes[2]);
mb.create_vertex(coords, nodes[3]);
mb.create_vertex(coords, nodes[4]);
mb.create_vertex(coords, nodes[5]);
// <
// > create the 2 quadrangles
moab::EntityHandle q0;
std::vector<moab::EntityHandle> conn0;
conn0.resize(4);
conn0[0]=nodes[0];
conn0[1]=nodes[1];
conn0[2]=nodes[2];
conn0[3]=nodes[3];
mb.create_element(moab::MBQUAD, conn0.data(), 4, q0);
moab::EntityHandle q1;
std::vector<moab::EntityHandle> conn1;
conn1.resize(4);
conn1[0]=nodes[1];
conn1[1]=nodes[4];
conn1[2]=nodes[5];
conn1[3]=nodes[2];
mb.create_element(moab::MBQUAD, conn1.data(), 4, q1);
// <
// > compute and store cell<->face adjacencies
adj.clear();
mb.get_adjacencies (&q0, 1, 1, true, adj);
mb.add_adjacencies(q0, adj.data(), adj.size(), true);
adj.clear();
mb.get_adjacencies (&q1, 1, 1, true, adj);
mb.add_adjacencies(q1, adj.data(), adj.size(), true);
// <
// > example of queries
// > get the faces of the first cell
adj.clear();
mb.get_adjacencies (&q1, 1, 1, false, adj); // this adjacency query
seems not to be retrieved retrieved directly
adj.clear();
mb.get_adjacencies (&q1, 1, 1, false, adj); // this one neither ...
std::cout << "## face of the first cell -------------------#\n";
mb.list_entities(adj.data(), adj.size());
// <
// > get the face shared by the two cells
adj.clear();
moab::EntityHandle cells[2]={q0,q1};
mb.get_adjacencies(cells, 2, 1, false, adj); // this one neither ...
std::cout << "## face shared by the two cells -------------#\n";
mb.list_entities(adj.data(), adj.size());
// <
// > get the two cells from the middle face
moab::EntityHandle face=adj[0];
adj.clear();
mb.get_adjacencies(&face, 1, 2, false, adj); // this one neither ...
std::cout << "## cells adjacent to the middle face --------#\n";
mb.list_entities(adj.data(), adj.size());
// <
// <
return 0;
}
On 20/06/2019 04:55, Vijay S. Mahadevan wrote:
>> The basic sample code pasted in my original mail illustrates that.
> If you know the adjacency relation between face and element entities
> a-priori, you can set these with add_adjacencies and/or maintain this
> externally in a data-structure like Lorenzo and Navamita have
> suggested. If you call get_adjacencies during offline stage
> ("create_if_missing" set to true) and then make a call again with
> "create_if_missing" set to false, this should yield excellent speeds
> as well since the explicit adjacency information for the edge/face
> entity should be available in-memory. If you are seeing bad
> performance for such a case, then we do have a problem and I will
> certainly look into it.
>
>> Yes, but as far as I understand, there is the "vertex intermediate
> state" that is costly. Is that right? Is there a way to "bypass" this
> intermediate step and retrieve directly the stored adjacency?
>
> After the offline stage call, the explicitly stored adjacency for the
> entity should be retrieved directly on the next call for the same
> entity. This is what I mentioned above. If you can create a test that
> shows a different behavior (I have not run your example yet though),
> we will push a performance fix.
>
>> This is the code from AEntityFactory.cpp between lines 176 to 188
> I will take a look at this.
>
>> Another "lock" at this time for us to use AHF is that it forbids
> modified meshes. In a near future, we will add/delete cells frequently
> for Adapative Mesh Refinement and parallel load balancing
>
> A side note: remember to keep an eye on the local range of entities.
> Frequent creation/deletion without sufficient entity reordering can
> create large segmentation in the Range datastructure in MOAB, and may
> sometimes result in performance degradation. We had a proposal to
> address this but it got pushed down lower on the priority list.
>
> Vijay
>
> On Wed, Jun 19, 2019 at 1:18 PM Olivier Jamond <olivier.jamond at cea.fr> wrote:
>> Dear Vijay,
>>
>> Many thanks for your answer. For better clarity, I also answer inline!
>>
>>>> So we consider replacing our legacy data/structure by moab. We already did this replacement in a dedicated branch of our software, and it works (it succeed to pass our test base). But the cpu costs of adjacencies queries increased hugely... For example, for a finite-volume fluid computation (the cells "often" query their adjacent faces, and the faces "often" query their adjacent cells), with our legacy structure, the adjacency queries cost about nothing, and with moab cost about 50% of the total computation!
>>> Can you provide us a little more detail on the type of meshes you use
>>> in the applications ? I assume 3-D, with mixed hex/tets ? Also, do you
>>> create most meshes in memory or are the initial meshes loaded from a
>>> file.
>> In our applications, we use the most general meshes: 3D meshes with 1, 2
>> and 3 manifold cells. At this time, in a mesh we can have at the same
>> time hexs/wedges/tets/quads/tri/segs/isolated-points (and even pyramids,
>> but we try to avoid). In the future, we may consider polygons/hedra.
>>
>> We create the whole mesh in memory (we do not use the moab mesh readers).
>>
>>>> Maybe we just misuse moab? In the moab documentation, it is written that "Non-vertex entity to entity adjacencies are never stored, unless explicitly requested by the application.". Because memory consumption is not our primary concern, we tried that, assuming that the request for storing adjacencies is done with the "add_adjacencies" function (in the end of this message a very simple sample code representative of what we do with moab is provided). But the cost of the adjacency queries is still very high...
>>> MOAB takes a lazy approach to computing the cell adjacencies. What
>>> that means is that if the adjacency information for face to cell
>>> relation is unavailable on first query, it computes it and caches it
>>> in memory so that the next call to get the adjacency is much faster.
>>> Motivation is to get a balance between memory requirement and cost for
>>> adjacency query. Typically, if you create a mesh in memory, the user
>>> also sets the adjacencies as well so that it can be queried later. So
>>> the workflow would be to call add_adjacencies BEFORE get_adjacencies
>>> here. However, if you loaded a mesh from file, you can query the mesh
>>> with get_adjacencies in addition to "create_if_missing" set to true,
>>> in case the mesh file itself does not store explicit adjacency
>>> information.
>> Our code has the following workflow:
>> * in a "offline" step (when a cell is added to the mesh), it calls
>> "get_adjacencies" (with "create_if_missing" set to true) to compute the
>> adjacency, and just after "add_adjacencies" to store this computed
>> adjacency. It does not matters much if this "offline step" is quite costly.
>> * during the computations steps, it calls "get_adjacencies" (with
>> "create_if_missing" set to false). This adjacency queries during the
>> computation steps must be fast.
>> The basic sample code pasted in my original mail illustrates that.
>>
>> The problem seems to be that the stored adjacencies during the "offline
>> step", for example a cell/face adjacency, does not seems to be retrieved
>> directly in the "get_adjacencies" ("create_if_missing" set to false)
>> queries of the computation steps: as you mention in the following, it
>> goes through a "vertex intermediate state" that appears to be costly.
>>
>>>> - for each edge represented by its two vertices, loop over some existing edges represented by their EntityHandle to find which one matches the current edge (a large portion of the computation time is spent in "moab::AEntityFactory::entities_equivalent(unsigned long, unsigned long const*, int, moab::EntityType)" )
>>> MOAB adjacency datastructure is optimized for vertex to element type
>>> queries since these are explicitly stored and available by default. And this is
>>> why the implementation to get face->element adjacency information goes
>>> through that intermediate step to decipher the relation. The AHF datastructure
>>> does not have this indirection and may provide much faster queries depending
>>> on the use-case.
>>>> Does moab actually store the adjacencies with "add_adjacencies"? For example, by following in a debugger what moab does on the sample program provided when querying the edges adjacent to a quadrangular cell, it seems that whereas the adjacencies being stored, moab:
>>> Yes. If the user explicitly calls add_adjacencies, this information is
>>> stored internally so that it can be queried later.
>> Yes, but as far as I understand, there is the "vertex intermediate
>> state" that is costly. Is that right? Is there a way to "bypass" this
>> intermediate step and retrieve directly the stored adjacency?
>>
>>>> We see in the code that there is a "create_adjacency_option" that is not yet supported. Could this option help once implemented?
>>> Can you point me to this piece of the implementation ?
>> This is the code from AEntityFactory.cpp between lines 176 to 188
>>
>> //! get the element defined by the vertices in vertex_list, of the
>> //! type target_type, passing back in target_entity; if create_if_missing
>> //! is true and no entity is found, one is created; if
>> create_adjacency_option
>> //! is >= 0, adjacencies from entities of that dimension to target_entity
>> //! are created (only create_adjacency_option=0 is supported right now,
>> //! so that never creates other ancillary entities)
>> ErrorCode AEntityFactory::get_element(const EntityHandle *vertex_list,
>> const int vertex_list_size,
>> const EntityType target_type,
>> EntityHandle &target_entity,
>> const bool create_if_missing,
>> const EntityHandle source_entity,
>> const int
>> /*create_adjacency_option*/)
>>
>>>> We also checked the AHF feature. At this time, this feature cannot apply in our applications because it is not compatible with modified and mixed-type meshes. Is it planned for AHF to become compatible with modified and mixed-type meshes?
>>> I am also cc'ing Navamita Ray who implemented AHF [1] in MOAB.
>>> Navamita, can you talk more about the AHF support with the adjacency
>>> cost comparison with native MOAB implementation ? Also, if you can
>>> provide some thoughts about extending AHF to support mixed meshes and
>>> polytopes, that would be good as well.
>> Another "lock" at this time for us to use AHF is that it forbids
>> modified meshes. In a near future, we will add/delete cells frequently
>> for Adapative Mesh Refinement and parallel load balancing
>>
>> Thank again,
>> Olivier
>>
>>> Vijay
>>>
>>> [1] https://imr.sandia.gov/papers/imr22/IMR22_25_Dyedov.pdf
>>>
>>>
>>> On Wed, Jun 19, 2019 at 11:08 AM JAMOND Olivier via moab-dev
>>> <moab-dev at mcs.anl.gov> wrote:
>>>> Hello,
>>>>
>>>>
>>>>
>>>> I am working at the french CEA "French Alternative Energies and Atomic Energy Commission" on the development of a new finite-elements/finite-volume software for mechanical applications.
>>>>
>>>>
>>>>
>>>> Until now, we had an internal data base for storing and querying adjencencies/connectivities between mesh entities. In this legacy data structure, every adjacencies/connectivities are computed and stored at the beginning of the computation, and then queries during the actual computation steps just "read" in this data structure. Our problem is that this legacy data structure is way too "static": for example, it is very difficult to add/remove cells (for AMR, or load balancing for example) during the computation steps.
>>>>
>>>>
>>>>
>>>> So we consider replacing our legacy data/structure by moab. We already did this replacement in a dedicated branch of our software, and it works (it succeed to pass our test base). But the cpu costs of adjacencies queries increased hugely... For example, for a finite-volume fluid computation (the cells "often" query their adjacent faces, and the faces "often" query their adjacent cells), with our legacy structure, the adjacency queries cost about nothing, and with moab cost about 50% of the total computation!
>>>>
>>>>
>>>>
>>>> Maybe we just misuse moab? In the moab documentation, it is written that "Non-vertex entity to entity adjacencies are never stored, unless explicitly requested by the application.". Because memory consumption is not our primary concern, we tried that, assuming that the request for storing adjacencies is done with the "add_adjacencies" function (in the end of this message a very simple sample code representative of what we do with moab is provided). But the cost of the adjacency queries is still very high...
>>>>
>>>>
>>>>
>>>> Does moab actually store the adjacencies with "add_adjacencies"? For example, by following in a debugger what moab does on the sample program provided when querying the edges adjacent to a quadrangular cell, it seems that whereas the adjacencies being stored, moab:
>>>>
>>>> - get the nodes of each edges of the quadrangle
>>>>
>>>> - for each edge represented by its two vertices, loop over some existing edges represented by their EntityHandle to find which one matches the current edge (a large portion of the computation time is spent in "moab::AEntityFactory::entities_equivalent(unsigned long, unsigned long const*, int, moab::EntityType)" )
>>>>
>>>>
>>>>
>>>> We see in the code that there is a "create_adjacency_option" that is not yet supported. Could this option help once implemented? We also checked the AHF feature. At this time, this feature cannot apply in our applications because it is not compatible with modified and mixed-type meshes. Is it planned for AHF to become compatible with modified and mixed-type meshes?
>>>>
>>>>
>>>>
>>>> Is there something that we miss, or a more efficient way to use moab?
>>>>
>>>>
>>>>
>>>> Many thanks,
>>>>
>>>> _____________________________
>>>>
>>>> Olivier JAMOND
>>>>
>>>> Laboratoire d'études de dynamique
>>>>
>>>> CEA Saclay, DEN/DANS/DM2S/SEMT/DYN
>>>>
>>>> F-91191 GIF SUR YVETTE, FRANCE
>>>>
>>>> Tél. : 01.69.08.44.90
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> * ------------- Sample code --------------- *
>>>>
>>>>
>>>>
>>>> #include <iostream>
>>>> #include <sstream>
>>>> #include <vector>
>>>>
>>>> #include "moab/Core.hpp"
>>>>
>>>> //
>>>> // 3 ----- 5 ----- 5
>>>> // | | |
>>>> // | | |
>>>> // | | |
>>>> // 0 ----- 1 ----- 4
>>>> //
>>>>
>>>> int main()
>>>> {
>>>>
>>>> std::vector<moab::EntityHandle> adj;
>>>>
>>>> // > init moab
>>>> moab::Core mb;
>>>> mb.set_dimension(2);
>>>> // <
>>>>
>>>> // > create the vertices (foo coordinates)
>>>> double coords[3]={0.,0.,0.};
>>>>
>>>> std::vector<moab::EntityHandle> nodes;
>>>> nodes.resize(6);
>>>> mb.create_vertex(coords, nodes[0]);
>>>> mb.create_vertex(coords, nodes[1]);
>>>> mb.create_vertex(coords, nodes[2]);
>>>> mb.create_vertex(coords, nodes[3]);
>>>> mb.create_vertex(coords, nodes[4]);
>>>> mb.create_vertex(coords, nodes[5]);
>>>> // <
>>>>
>>>> // > create the 2 quadrangles
>>>> moab::EntityHandle q0;
>>>> std::vector<moab::EntityHandle> conn0;
>>>> conn0.resize(4);
>>>> conn0[0]=nodes[0];
>>>> conn0[1]=nodes[1];
>>>> conn0[2]=nodes[2];
>>>> conn0[3]=nodes[3];
>>>> mb.create_element(moab::MBQUAD, conn0.data(), 4, q0);
>>>>
>>>> moab::EntityHandle q1;
>>>> std::vector<moab::EntityHandle> conn1;
>>>> conn1.resize(4);
>>>> conn1[0]=nodes[1];
>>>> conn1[1]=nodes[4];
>>>> conn1[2]=nodes[5];
>>>> conn1[3]=nodes[2];
>>>> mb.create_element(moab::MBQUAD, conn1.data(), 4, q1);
>>>> // <
>>>>
>>>> // > compute and store cell<->face adjacencies
>>>> adj.clear();
>>>> mb.get_adjacencies (&q0, 1, 1, true, adj);
>>>> mb.add_adjacencies(q0, adj.data(), adj.size(), true);
>>>> adj.clear();
>>>> mb.get_adjacencies (&q1, 1, 1, true, adj);
>>>> mb.add_adjacencies(q1, adj.data(), adj.size(), true);
>>>> // <
>>>>
>>>> // > example of queries
>>>> // > get the faces of the first cell
>>>> adj.clear();
>>>> mb.get_adjacencies (&q1, 1, 1, false, adj);
>>>> std::cout << "## face of the first cell -------------------#\n";
>>>> mb.list_entities(adj.data(), adj.size());
>>>> // <
>>>>
>>>> // > get the face shared by the two cells
>>>> adj.clear();
>>>> moab::EntityHandle cells[2]={q0,q1};
>>>> mb.get_adjacencies(cells, 2, 1, false, adj);
>>>> std::cout << "## face shared by the two cells -------------#\n";
>>>> mb.list_entities(adj.data(), adj.size());
>>>> // <
>>>>
>>>> // > get the two cells from the middle face
>>>> moab::EntityHandle face=adj[0];
>>>> adj.clear();
>>>> mb.get_adjacencies(&face, 1, 2, false, adj);
>>>> std::cout << "## cells adjacent to the middle face --------#\n";
>>>> mb.list_entities(adj.data(), adj.size());
>>>> // <
>>>> // <
>>>>
>>>> return 0;
>>>> }
>>>>
>>>>
>>>>
>>>>
More information about the moab-dev
mailing list