[MOAB-dev] Performance issues for finite-volume computations
Navamita Ray
navamitaray at gmail.com
Wed Jun 19 14:14:33 CDT 2019
Hi Olivier,
It has been a while that I looked at MOAB code. As far as I remember, with
the native data structure one explicitly computes only the vertex—>entity
maps. So, when the add_adjacencies routine is called for new entities, it
will create and store only the vertex—>entity map. What this means is that
any other query would ultimately go through this look up as you correctly
mentioned.
An immediate work around would be to follow Lorenzo’s advice, and
explicitly store the computed adjacency as tags/fields which should have
constant lookup time.
AHF stores a different set of maps (facet-to-facet, vertex-to-facet) and
uses them to compute any other adjacency query. We did observe some speedup
in few of the adjacency queries, particularly the face-to-cell queries
compared to the native data structure.
We did extend AHF to support adaptive meshes, however, its only existing
implementation is in a MATLAB code base called NumGeom (developed at Stony
Brook University by Prof. Xiangmin Jiao’s group) as we didn’t get to a
point where we could add this extended feature to MOAB. Depending on your
time constraints, we can discuss further about adding this feature to MOAB.
Best..
Navamita
On Wed, Jun 19, 2019 at 11:53 AM lorenzo alessio botti <
lorenzoalessiobotti at gmail.com> wrote:
> You can also store adjacencies as tags using the specific type for moab
> entities. That's what I usually do to save computation type at the expense
> of storing integers.
> Bests
> Lorenxo
>
> On Wed, Jun 19, 2019, 17:32 Vijay S. Mahadevan via moab-dev <
> moab-dev at mcs.anl.gov> wrote:
>
>> Dear Jamond,
>>
>> Good to hear that you are considering MOAB as a mesh database for your
>> application. Please see my answers inline. Do let me know if something
>> is unclear.
>>
>> > So we consider replacing our legacy data/structure by moab. We already
>> did this replacement in a dedicated branch of our software, and it works
>> (it succeed to pass our test base). But the cpu costs of adjacencies
>> queries increased hugely... For example, for a finite-volume fluid
>> computation (the cells "often" query their adjacent faces, and the faces
>> "often" query their adjacent cells), with our legacy structure, the
>> adjacency queries cost about nothing, and with moab cost about 50% of the
>> total computation!
>>
>> Can you provide us a little more detail on the type of meshes you use
>> in the applications ? I assume 3-D, with mixed hex/tets ? Also, do you
>> create most meshes in memory or are the initial meshes loaded from a
>> file.
>>
>> > Maybe we just misuse moab? In the moab documentation, it is written
>> that "Non-vertex entity to entity adjacencies are never stored, unless
>> explicitly requested by the application.". Because memory consumption is
>> not our primary concern, we tried that, assuming that the request for
>> storing adjacencies is done with the "add_adjacencies" function (in the end
>> of this message a very simple sample code representative of what we do with
>> moab is provided). But the cost of the adjacency queries is still very
>> high...
>>
>> MOAB takes a lazy approach to computing the cell adjacencies. What
>> that means is that if the adjacency information for face to cell
>> relation is unavailable on first query, it computes it and caches it
>> in memory so that the next call to get the adjacency is much faster.
>> Motivation is to get a balance between memory requirement and cost for
>> adjacency query. Typically, if you create a mesh in memory, the user
>> also sets the adjacencies as well so that it can be queried later. So
>> the workflow would be to call add_adjacencies BEFORE get_adjacencies
>> here. However, if you loaded a mesh from file, you can query the mesh
>> with get_adjacencies in addition to "create_if_missing" set to true,
>> in case the mesh file itself does not store explicit adjacency
>> information.
>>
>> > - for each edge represented by its two vertices, loop over some
>> existing edges represented by their EntityHandle to find which one matches
>> the current edge (a large portion of the computation time is spent in
>> "moab::AEntityFactory::entities_equivalent(unsigned long, unsigned long
>> const*, int, moab::EntityType)" )
>>
>> MOAB adjacency datastructure is optimized for vertex to element type
>> queries since these are explicitly stored and available by default. And
>> this is
>> why the implementation to get face->element adjacency information goes
>> through that intermediate step to decipher the relation. The AHF
>> datastructure
>> does not have this indirection and may provide much faster queries
>> depending
>> on the use-case.
>>
>> > Does moab actually store the adjacencies with "add_adjacencies"? For
>> example, by following in a debugger what moab does on the sample program
>> provided when querying the edges adjacent to a quadrangular cell, it seems
>> that whereas the adjacencies being stored, moab:
>>
>> Yes. If the user explicitly calls add_adjacencies, this information is
>> stored internally so that it can be queried later.
>>
>> > We see in the code that there is a "create_adjacency_option" that is
>> not yet supported. Could this option help once implemented?
>>
>> Can you point me to this piece of the implementation ?
>>
>> > We also checked the AHF feature. At this time, this feature cannot
>> apply in our applications because it is not compatible with modified and
>> mixed-type meshes. Is it planned for AHF to become compatible with modified
>> and mixed-type meshes?
>>
>> I am also cc'ing Navamita Ray who implemented AHF [1] in MOAB.
>> Navamita, can you talk more about the AHF support with the adjacency
>> cost comparison with native MOAB implementation ? Also, if you can
>> provide some thoughts about extending AHF to support mixed meshes and
>> polytopes, that would be good as well.
>>
>> Vijay
>>
>> [1] https://imr.sandia.gov/papers/imr22/IMR22_25_Dyedov.pdf
>>
>>
>> On Wed, Jun 19, 2019 at 11:08 AM JAMOND Olivier via moab-dev
>> <moab-dev at mcs.anl.gov> wrote:
>> >
>> > Hello,
>> >
>> >
>> >
>> > I am working at the french CEA "French Alternative Energies and Atomic
>> Energy Commission" on the development of a new
>> finite-elements/finite-volume software for mechanical applications.
>> >
>> >
>> >
>> > Until now, we had an internal data base for storing and querying
>> adjencencies/connectivities between mesh entities. In this legacy data
>> structure, every adjacencies/connectivities are computed and stored at the
>> beginning of the computation, and then queries during the actual
>> computation steps just "read" in this data structure. Our problem is that
>> this legacy data structure is way too "static": for example, it is very
>> difficult to add/remove cells (for AMR, or load balancing for example)
>> during the computation steps.
>> >
>> >
>> >
>> > So we consider replacing our legacy data/structure by moab. We already
>> did this replacement in a dedicated branch of our software, and it works
>> (it succeed to pass our test base). But the cpu costs of adjacencies
>> queries increased hugely... For example, for a finite-volume fluid
>> computation (the cells "often" query their adjacent faces, and the faces
>> "often" query their adjacent cells), with our legacy structure, the
>> adjacency queries cost about nothing, and with moab cost about 50% of the
>> total computation!
>> >
>> >
>> >
>> > Maybe we just misuse moab? In the moab documentation, it is written
>> that "Non-vertex entity to entity adjacencies are never stored, unless
>> explicitly requested by the application.". Because memory consumption is
>> not our primary concern, we tried that, assuming that the request for
>> storing adjacencies is done with the "add_adjacencies" function (in the end
>> of this message a very simple sample code representative of what we do with
>> moab is provided). But the cost of the adjacency queries is still very
>> high...
>> >
>> >
>> >
>> > Does moab actually store the adjacencies with "add_adjacencies"? For
>> example, by following in a debugger what moab does on the sample program
>> provided when querying the edges adjacent to a quadrangular cell, it seems
>> that whereas the adjacencies being stored, moab:
>> >
>> > - get the nodes of each edges of the quadrangle
>> >
>> > - for each edge represented by its two vertices, loop over some
>> existing edges represented by their EntityHandle to find which one matches
>> the current edge (a large portion of the computation time is spent in
>> "moab::AEntityFactory::entities_equivalent(unsigned long, unsigned long
>> const*, int, moab::EntityType)" )
>> >
>> >
>> >
>> > We see in the code that there is a "create_adjacency_option" that is
>> not yet supported. Could this option help once implemented? We also checked
>> the AHF feature. At this time, this feature cannot apply in our
>> applications because it is not compatible with modified and mixed-type
>> meshes. Is it planned for AHF to become compatible with modified and
>> mixed-type meshes?
>> >
>> >
>> >
>> > Is there something that we miss, or a more efficient way to use moab?
>> >
>> >
>> >
>> > Many thanks,
>> >
>> > _____________________________
>> >
>> > Olivier JAMOND
>> >
>> > Laboratoire d'études de dynamique
>> >
>> > CEA Saclay, DEN/DANS/DM2S/SEMT/DYN
>> >
>> > F-91191 GIF SUR YVETTE, FRANCE
>> >
>> > Tél. : 01.69.08.44.90
>> >
>> >
>> >
>> >
>> >
>> > * ------------- Sample code --------------- *
>> >
>> >
>> >
>> > #include <iostream>
>> > #include <sstream>
>> > #include <vector>
>> >
>> > #include "moab/Core.hpp"
>> >
>> > //
>> > // 3 ----- 5 ----- 5
>> > // | | |
>> > // | | |
>> > // | | |
>> > // 0 ----- 1 ----- 4
>> > //
>> >
>> > int main()
>> > {
>> >
>> > std::vector<moab::EntityHandle> adj;
>> >
>> > // > init moab
>> > moab::Core mb;
>> > mb.set_dimension(2);
>> > // <
>> >
>> > // > create the vertices (foo coordinates)
>> > double coords[3]={0.,0.,0.};
>> >
>> > std::vector<moab::EntityHandle> nodes;
>> > nodes.resize(6);
>> > mb.create_vertex(coords, nodes[0]);
>> > mb.create_vertex(coords, nodes[1]);
>> > mb.create_vertex(coords, nodes[2]);
>> > mb.create_vertex(coords, nodes[3]);
>> > mb.create_vertex(coords, nodes[4]);
>> > mb.create_vertex(coords, nodes[5]);
>> > // <
>> >
>> > // > create the 2 quadrangles
>> > moab::EntityHandle q0;
>> > std::vector<moab::EntityHandle> conn0;
>> > conn0.resize(4);
>> > conn0[0]=nodes[0];
>> > conn0[1]=nodes[1];
>> > conn0[2]=nodes[2];
>> > conn0[3]=nodes[3];
>> > mb.create_element(moab::MBQUAD, conn0.data(), 4, q0);
>> >
>> > moab::EntityHandle q1;
>> > std::vector<moab::EntityHandle> conn1;
>> > conn1.resize(4);
>> > conn1[0]=nodes[1];
>> > conn1[1]=nodes[4];
>> > conn1[2]=nodes[5];
>> > conn1[3]=nodes[2];
>> > mb.create_element(moab::MBQUAD, conn1.data(), 4, q1);
>> > // <
>> >
>> > // > compute and store cell<->face adjacencies
>> > adj.clear();
>> > mb.get_adjacencies (&q0, 1, 1, true, adj);
>> > mb.add_adjacencies(q0, adj.data(), adj.size(), true);
>> > adj.clear();
>> > mb.get_adjacencies (&q1, 1, 1, true, adj);
>> > mb.add_adjacencies(q1, adj.data(), adj.size(), true);
>> > // <
>> >
>> > // > example of queries
>> > // > get the faces of the first cell
>> > adj.clear();
>> > mb.get_adjacencies (&q1, 1, 1, false, adj);
>> > std::cout << "## face of the first cell -------------------#\n";
>> > mb.list_entities(adj.data(), adj.size());
>> > // <
>> >
>> > // > get the face shared by the two cells
>> > adj.clear();
>> > moab::EntityHandle cells[2]={q0,q1};
>> > mb.get_adjacencies(cells, 2, 1, false, adj);
>> > std::cout << "## face shared by the two cells -------------#\n";
>> > mb.list_entities(adj.data(), adj.size());
>> > // <
>> >
>> > // > get the two cells from the middle face
>> > moab::EntityHandle face=adj[0];
>> > adj.clear();
>> > mb.get_adjacencies(&face, 1, 2, false, adj);
>> > std::cout << "## cells adjacent to the middle face --------#\n";
>> > mb.list_entities(adj.data(), adj.size());
>> > // <
>> > // <
>> >
>> > return 0;
>> > }
>> >
>> >
>> >
>> >
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/moab-dev/attachments/20190619/fdbc0c11/attachment.html>
More information about the moab-dev
mailing list