[MOAB-dev] Performance issues for finite-volume computations

[JAMOND Olivier]

Hello,

I am working at the french CEA "French Alternative Energies and Atomic Energy Commission" on the development of a new finite-elements/finite-volume software for mechanical applications.

Until now, we had an internal data base for storing and querying adjencencies/connectivities between mesh entities. In this legacy data structure, every adjacencies/connectivities are computed and stored at the beginning of the computation, and then queries during the actual computation steps just "read" in this data structure. Our problem is that this legacy data structure is way too "static": for example, it is very difficult to add/remove cells (for AMR, or load balancing for example) during the computation steps.

So we consider replacing our legacy data/structure by moab. We already did this replacement in a dedicated branch of our software, and it works (it succeed to pass our test base). But the cpu costs of adjacencies queries increased hugely... For example, for a finite-volume fluid computation (the cells "often" query their adjacent faces, and the faces "often" query their adjacent cells), with our legacy structure, the adjacency queries cost about nothing, and with moab cost about 50% of the total computation!

Maybe we just misuse moab? In the moab documentation, it is written that "Non-vertex entity to entity adjacencies are never stored, unless explicitly requested by the application.". Because memory consumption is not our primary concern, we tried that, assuming that the request for storing adjacencies is done with the "add_adjacencies" function (in the end of this message a very simple sample code representative of what we do with moab is provided). But the cost of the adjacency queries is still very high...

Does moab actually store the adjacencies with "add_adjacencies"? For example, by following in a debugger what moab does on the sample program provided when querying the edges adjacent to a quadrangular cell, it seems that whereas the adjacencies being stored, moab:
- get the nodes of each edges of the quadrangle
- for each edge represented by its two vertices, loop over some existing edges represented by their EntityHandle to find which one matches the current edge (a large portion of the computation time is spent in "moab::AEntityFactory::entities_equivalent(unsigned long, unsigned long const*, int, moab::EntityType)" )

We see in the code that there is a "create_adjacency_option" that is not yet supported. Could this option help once implemented? We also checked the AHF feature. At this time, this feature cannot apply in our applications because it is not compatible with modified and mixed-type meshes. Is it planned for AHF to become compatible with modified and mixed-type meshes?

Is there something that we miss, or a more efficient way to use moab?

Many thanks,
_____________________________
Olivier JAMOND
Laboratoire d'études de dynamique
CEA Saclay, DEN/DANS/DM2S/SEMT/DYN
F-91191 GIF SUR YVETTE, FRANCE
Tél. : 01.69.08.44.90


* ------------- Sample code --------------- *

#include <iostream>
#include <sstream>
#include <vector>

#include "moab/Core.hpp"

//
// 3 ----- 5 ----- 5
// |       |       |
// |       |       |
// |       |       |
// 0 ----- 1 ----- 4
//

int main()
{

  std::vector<moab::EntityHandle> adj;

  // > init moab
  moab::Core mb;
  mb.set_dimension(2);
  // <

  // > create the vertices (foo coordinates)
  double coords[3]={0.,0.,0.};

  std::vector<moab::EntityHandle> nodes;
  nodes.resize(6);
  mb.create_vertex(coords, nodes[0]);
  mb.create_vertex(coords, nodes[1]);
  mb.create_vertex(coords, nodes[2]);
  mb.create_vertex(coords, nodes[3]);
  mb.create_vertex(coords, nodes[4]);
  mb.create_vertex(coords, nodes[5]);
  // <

  // > create the 2 quadrangles
  moab::EntityHandle q0;
  std::vector<moab::EntityHandle> conn0;
  conn0.resize(4);
  conn0[0]=nodes[0];
  conn0[1]=nodes[1];
  conn0[2]=nodes[2];
  conn0[3]=nodes[3];
  mb.create_element(moab::MBQUAD, conn0.data(), 4, q0);

  moab::EntityHandle q1;
  std::vector<moab::EntityHandle> conn1;
  conn1.resize(4);
  conn1[0]=nodes[1];
  conn1[1]=nodes[4];
  conn1[2]=nodes[5];
  conn1[3]=nodes[2];
  mb.create_element(moab::MBQUAD, conn1.data(), 4, q1);
  // <

  // > compute and store cell<->face adjacencies
  adj.clear();
  mb.get_adjacencies (&q0, 1, 1, true, adj);
  mb.add_adjacencies(q0, adj.data(), adj.size(), true);
  adj.clear();
  mb.get_adjacencies (&q1, 1, 1, true, adj);
  mb.add_adjacencies(q1, adj.data(), adj.size(), true);
  // <

  // > example of queries
  // > get the faces of the first cell
  adj.clear();
  mb.get_adjacencies (&q1, 1, 1, false, adj);
  std::cout << "## face of the first cell -------------------#\n";
  mb.list_entities(adj.data(), adj.size());
  // <

  // > get the face shared by the two cells
  adj.clear();
  moab::EntityHandle cells[2]={q0,q1};
  mb.get_adjacencies(cells, 2, 1, false, adj);
  std::cout << "## face shared by the two cells -------------#\n";
  mb.list_entities(adj.data(), adj.size());
  // <

  // > get the two cells from the middle face
  moab::EntityHandle face=adj[0];
  adj.clear();
  mb.get_adjacencies(&face, 1, 2, false, adj);
  std::cout << "## cells adjacent to the middle face --------#\n";
  mb.list_entities(adj.data(), adj.size());
  // <
  // <

  return 0;
}


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/moab-dev/attachments/20190619/de7f5a72/attachment.html>