[MOAB-dev] Question: Ghosting

[Markus Mayr]

Dear List,

First of all, I really enjoy using MOAB for my parallel program. However,
I have got a problem in understanding how ghost entities work and how I
get them to do what I want.

For my computations I need to know the coordinates of each vertex and
of all of its neighbour vertices, i.e. all vertices that are connected
by one edge. Because I am using a tetrahedral mesh, I thought that it
would be a good idea to create a ghost layer that shares entities of
dimension 3 across a bridge dimension of 0 using 1 as the layer dimension,
i.e. what I get by using
  PARALLEL_GHOSTS=3.0.1
when loading the mesh. Then I used the PSTATUS_OWNED status bit to
check whether a certain process owns a certain entity. The odd thing
is that for example process 0 has no non-PSTATUS_OWNED entities. So
this is clearly not what I want. (The other processes have ghost
entities, by the way.)

Before I provide more code to give you more context, I will raise
all my questions:
  1.) Is this the expected result?
  2.) Is there a way to achieve what I need, i.e. to detect whether
      a vertex is owned by a given process and to get all of that
      vertex' neighbours? (I accomplished the first part already,
      I think.)

So ... more information: I am using MOAB-4.0.0, I created the mesh
using NETGEN, exported it to the GMSH format and then converted the
resulting file using mbconvert. Finally I partitioned it using
mbpart -n 8. The following short example produces the results which
I did not expect. I compiled it with

  mpicxx -Wall -DMPI_INCLUDED -lMOAB test.cpp

and ran it with

  mpirun -n 2 ./a.out
  mpirun -n 8 ./a.out

on my machine.

// SOURCE CODE STARTS HERE
#include <mpi.h>
#include <moab/ParallelComm.hpp>
#include <moab/Core.hpp>
#include <MBParallelConventions.h>

int main(int argc, char* argv[])
{
  MPI_Init(&argc, &argv);

  moab::EntityHandle local_entities;
  moab::Core* moab = new moab::Core();
  moab::ParallelComm* comm =
    new moab::ParallelComm(moab, MPI_COMM_WORLD);

  moab->create_meshset(static_cast<unsigned int>(0), local_entities, 0);
  moab->load_file("cube_8.h5m", &local_entities,
              "PARALLEL=READ_PART;"
              "PARALLEL_GHOSTS=3.0.1;"
              "PARTITION=PARALLEL_PARTITION;"
              "PARALLEL_RESOLVE_SHARED_ENTS");

  // Create an entity set for the local entities with the fitting type:
  moab::Range local_vertices_range;
  moab->get_entities_by_type(local_entities, moab::MBVERTEX,
              local_vertices_range);
  int number_of_local_vertices = local_vertices_range.size();

  // Filter out owned vertices:
  moab::Range owned_vertices_range;
  comm->filter_pstatus(local_vertices_range, PSTATUS_NOT_OWNED,
              PSTATUS_NOT, -1, &owned_vertices_range);
  int number_of_owned_vertices = owned_vertices_range.size();

  std::cerr << "Owning " << number_of_owned_vertices
            << " of " << number_of_local_vertices << " vertices."
            << std::endl;

  MPI_Finalize();
}
// SOURCE CODE ENDS HERE

Thank you for your precious help!

-- 
Best regards,
Markus Mayr