[MOAB-dev] ghost/halo elements

Vijay S. Mahadevan vijay.m at gmail.com
Mon Oct 29 11:54:07 CDT 2018 

The default set of tags do get transferred. These would be the
MATERIAL_SET, DIRICHLET/NEUMANN sets etc.

Iulian, did we add a separate method to transfer these to the ghosted
entities ? I vaguely remember something like that.

Vijay
On Mon, Oct 29, 2018 at 12:42 PM Ryan O'Kuinghttons
<ryan.okuinghttons at noaa.gov> wrote:
>
> Hi Vijay,
>
> I ran into another issue with the call to exchange_ghost_cells, but this
> is probably just a misunderstanding on my part. I find that the tag
> information that is part of the mesh before the the call does not
> transfer to the newly created faces. Is there some way to ask moab to
> transfer tags when creating ghost entities? (FWIW, I could not find this
> in the example you sent me) Thanks,
>
> Ryan
>
> On 10/26/18 07:36, Ryan O'Kuinghttons wrote:
> > Hi Vijay,
> >
> > I resolved the issue with not getting any shared, owned entities back
> > by using the other version of resolve_shared_ents, as demonstrated in
> > the example you indicated. I created a Range containing all elements
> > (faces for moab), and used that as input.
> >
> >   Range range_ent;
> > merr=mb->get_entities_by_dimension(0,this->sdim,range_ent);
> >   MBMESH_CHECK_ERR(merr, localrc);
> >
> >   merr = pcomm->resolve_shared_ents(0, range_ent, this->sdim, 1);
> >   MBMESH_CHECK_ERR(merr, localrc);
> >
> > The first version of the interface (which does not use a Range input)
> > does not work for me:
> >
> >   merr = pcomm->resolve_shared_ents(0, this->sdim, 1);
> >
> > Otherwise, everything else seems to be resolved with this issue,
> > thanks again for all of your help!!
> >
> > Ryan
> >
> > On 10/25/18 15:13, Vijay S. Mahadevan wrote:
> >> Ryan, I don't see anything obviously wrong in the code below. Does
> >> your shared_ents not have any entities after parallel resolve ? If you
> >> can replicate your issue in a small standalone test, we should be able
> >> to quickly find out what's going wrong.
> >>
> >> Look at the test_reduce_tag_explicit_dest routine in
> >> test/parallel/parallel_unit_tests.cpp:1617. It has the workflow where
> >> you create a mesh in memory, resolve shared entities, get shared
> >> entities and even set/reduce tag data.
> >>
> >> Vijay
> >> On Thu, Oct 25, 2018 at 4:25 PM Ryan O'Kuinghttons
> >> <ryan.okuinghttons at noaa.gov> wrote:
> >>> OK, that allows the resolve_shared_ents routine to complete without
> >>> error, but I still don't get any shared, owned entities back from the
> >>> get_shared_entities/filter_pstatus calls:
> >>>
> >>>     Range shared_ents;
> >>>     // Get entities shared with all other processors
> >>>     merr = pcomm->get_shared_entities(-1, shared_ents);
> >>>     MBMESH_CHECK_ERR(merr, localrc);
> >>>
> >>>     // Filter shared entities with not not_owned, which means owned
> >>>     Range owned_entities;
> >>>     merr = pcomm->filter_pstatus(shared_ents, PSTATUS_NOT_OWNED,
> >>> PSTATUS_NOT, -1, &owned_entities);
> >>>     MBMESH_CHECK_ERR(merr, localrc);
> >>>
> >>>     unsigned int nums[4] = {0}; // to store the owned entities per
> >>> dimension
> >>>     for (int i = 0; i < 4; i++)
> >>>       nums[i] = (int)owned_entities.num_of_dimension(i);
> >>>
> >>>     vector<int> rbuf(nprocs*4, 0);
> >>>     MPI_Gather(nums, 4, MPI_INT, &rbuf[0], 4, MPI_INT, 0, mpi_comm);
> >>>     // Print the stats gathered:
> >>>     if (0 == rank) {
> >>>       for (int i = 0; i < nprocs; i++)
> >>>         cout << " Shared, owned entities on proc " << i << ": " <<
> >>> rbuf[4*i] << " verts, " <<
> >>>             rbuf[4*i + 1] << " edges, " << rbuf[4*i + 2] << " faces,
> >>> " <<
> >>> rbuf[4*i + 3] << " elements" << endl;
> >>>     }
> >>>
> >>> OUTPUT:
> >>>
> >>>    Shared, owned entities on proc 0: 0 verts, 0 edges, 0 faces, 0
> >>> elements
> >>>    Shared, owned entities on proc 1: 0 verts, 0 edges, 0 faces, 0
> >>> elements
> >>>    Shared, owned entities on proc 2: 0 verts, 0 edges, 0 faces, 0
> >>> elements
> >>>    Shared, owned entities on proc 3: 0 verts, 0 edges, 0 faces, 0
> >>> elements
> >>>
> >>>
> >>> Note: I also tried changing the -1 to 2 in the
> >>> get_shared_entities/filter_pstatus calls to only retrieve elements.
> >>>
> >>>
> >>> On 10/25/18 13:57, Vijay S. Mahadevan wrote:
> >>>> No problem. Can you try 1 for the last argument to look for edges as
> >>>> the shared_dim.
> >>>>
> >>>> Vijay
> >>>> On Thu, Oct 25, 2018 at 3:41 PM Ryan O'Kuinghttons
> >>>> <ryan.okuinghttons at noaa.gov> wrote:
> >>>>> OK, thanks again for that explanation, I never would have been
> >>>>> able to figure out that this_set was the root set. I apologize for
> >>>>> all the dumb questions, hopefully they are all getting me closer
> >>>>> to self sufficiency..
> >>>>>
> >>>>> I'm getting an error that doesn't make sense to me. I specifying
> >>>>> the resolve_dim and shared_dim in the call:
> >>>>>
> >>>>>     // Get the ParallelComm instance
> >>>>>     ParallelComm* pcomm = new ParallelComm(mb, mpi_comm);
> >>>>>     int nprocs = pcomm->proc_config().proc_size();
> >>>>>     int rank = pcomm->proc_config().proc_rank();
> >>>>>
> >>>>>     // get the root set handle
> >>>>>     EntityHandle root_set = mb->get_root_set();
> >>>>>
> >>>>>     merr = pcomm->resolve_shared_ents(root_set, 2, -1);
> >>>>>
> >>>>>
> >>>>> but I'm getting this message:
> >>>>>
> >>>>> [0]MOAB ERROR: Unable to guess shared_dim or resolve_dim!
> >>>>>
> >>>>>
> >>>>> what am i missing?
> >>>>>
> >>>>>
> >>>>> On 10/25/18 12:51, Vijay S. Mahadevan wrote:
> >>>>>
> >>>>> It is your root set or fileset if you added entities to it. You
> >>>>> can get the root set handle from Core object.
> >>>>>
> >>>>> Vijay
> >>>>>
> >>>>> On Thu, Oct 25, 2018, 14:38 Ryan O'Kuinghttons
> >>>>> <ryan.okuinghttons at noaa.gov> wrote:
> >>>>>> Thanks again, Vijay. However, I still don't understand what this_set
> >>>>>> should be, is it an output maybe?
> >>>>>>
> >>>>>> Ryan
> >>>>>>
> >>>>>> On 10/25/18 12:27, Vijay S. Mahadevan wrote:
> >>>>>>> You can try either the first or the second variant instead [1] with
> >>>>>>> following arguments.
> >>>>>>>
> >>>>>>> ErrorCode moab::ParallelComm::resolve_shared_ents(EntityHandle
> >>>>>>> this_set,
> >>>>>>> int resolve_dim = 2,
> >>>>>>> int shared_dim = -1,
> >>>>>>> const Tag * id_tag = 0
> >>>>>>> )
> >>>>>>>
> >>>>>>> That should resolve 2-dim entities with shared edges across
> >>>>>>> partitions. You can leave id_tag pointer as zero since the
> >>>>>>> default is
> >>>>>>> GLOBAL_ID.
> >>>>>>>
> >>>>>>>> This brings up a more general question I've had about the moab
> >>>>>>> documentation for a while. In the doc for this routine, it only
> >>>>>>> lists 2
> >>>>>>> parameters, proc_ents and shared_dim, even though in the function
> >>>>>>> signature above it clearly shows more. I've had trouble
> >>>>>>> understanding
> >>>>>>> which parameters are relevant in the past, or what they do
> >>>>>>> because I'm
> >>>>>>> not quite sure how to read the documentation.
> >>>>>>>
> >>>>>>> This is an oversight. We will go through and rectify some of the
> >>>>>>> inconsistencies in the documentation. We are preparing for an
> >>>>>>> upcoming
> >>>>>>> release and I'll make sure that routines in Core/ParallelComm have
> >>>>>>> updated documentation that match the interfaces. Meanwhile, if you
> >>>>>>> have questions, feel free to shoot an email to the list.
> >>>>>>>
> >>>>>>> Hope that helps.
> >>>>>>>
> >>>>>>> Vijay
> >>>>>>>
> >>>>>>> [1]
> >>>>>>> http://ftp.mcs.anl.gov/pub/fathom/moab-docs-develop/classmoab_1_1ParallelComm.html#a29a3b834b3fc3b4ddb3a5d8a78a37c8a
> >>>>>>> On Thu, Oct 25, 2018 at 1:58 PM Ryan O'Kuinghttons
> >>>>>>> <ryan.okuinghttons at noaa.gov> wrote:
> >>>>>>>> Hi Vijay,
> >>>>>>>>
> >>>>>>>> Thanks again for that explanation.
> >>>>>>>>
> >>>>>>>> ESMF does use unique global ids for the vertices and set them
> >>>>>>>> to the
> >>>>>>>> GLOBAL_ID in the moab mesh. So I think we are good there.
> >>>>>>>>
> >>>>>>>> I can't quite figure out how to use resolve_shared_entities
> >>>>>>>> though.
> >>>>>>>> There are three versions of the call in the documentation, and
> >>>>>>>> I assume
> >>>>>>>> that I should use the first and pass in a Range containing all
> >>>>>>>> entities
> >>>>>>>> for proc_ents:
> >>>>>>>>
> >>>>>>>> http://ftp.mcs.anl.gov/pub/fathom/moab-docs-develop/classmoab_1_1ParallelComm.html#a59e35d9906f2e33fe010138a144a5cb6
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> However, I'm not sure what the this_set EntityHandle should be.
> >>>>>>>>
> >>>>>>>> This brings up a more general question I've had about the moab
> >>>>>>>> documentation for a while. In the doc for this routine, it only
> >>>>>>>> lists 2
> >>>>>>>> parameters, proc_ents and shared_dim, even though in the function
> >>>>>>>> signature above it clearly shows more. I've had trouble
> >>>>>>>> understanding
> >>>>>>>> which parameters are relevant in the past, or what they do
> >>>>>>>> because I'm
> >>>>>>>> not quite sure how to read the documentation.
> >>>>>>>>
> >>>>>>>> Any example or explanation you give me for
> >>>>>>>> resolve_shared_entities is
> >>>>>>>> much appreciated!
> >>>>>>>>
> >>>>>>>> Ryan
> >>>>>>>>
> >>>>>>>> On 10/25/18 09:40, Vijay S. Mahadevan wrote:
> >>>>>>>>> Hi Ryan,
> >>>>>>>>>
> >>>>>>>>> Glad the example helped.
> >>>>>>>>>
> >>>>>>>>>> The example shows the ghost exchange happening between shared
> >>>>>>>>>> owned
> >>>>>>>>> entities. In my experiments I've been unable to make the ghost
> >>>>>>>>> exchange
> >>>>>>>>> work, and I think that might be because my entities are not
> >>>>>>>>> shared.
> >>>>>>>>>
> >>>>>>>>> You need to call resolve_shared_entities on the entities prior to
> >>>>>>>>> doing ghost exchange. When we load the mesh from file, this
> >>>>>>>>> happens
> >>>>>>>>> automatically based on the read options but if you are forming
> >>>>>>>>> a mesh
> >>>>>>>>> in memory, you need to make sure that the shared vertices have
> >>>>>>>>> the
> >>>>>>>>> same GLOBAL_ID numbering consistently across processes. That
> >>>>>>>>> is shared
> >>>>>>>>> vertices are unique. Once that is set, shared entities
> >>>>>>>>> resolution will
> >>>>>>>>> work correctly out of the box and you will have shared
> >>>>>>>>> edges/entities
> >>>>>>>>> query working correctly. A call to get ghosted layers once
> >>>>>>>>> this is
> >>>>>>>>> done would be the way to go.
> >>>>>>>>>
> >>>>>>>>> I assume ESMF has a unique global numbering for the vertices ?
> >>>>>>>>> Use
> >>>>>>>>> that to set the GLOBAL_ID tag. Let us know if you are still
> >>>>>>>>> having
> >>>>>>>>> issues.
> >>>>>>>>>
> >>>>>>>>> Vijay
> >>>>>>>>> On Thu, Oct 25, 2018 at 11:17 AM Ryan O'Kuinghttons
> >>>>>>>>> <ryan.okuinghttons at noaa.gov> wrote:
> >>>>>>>>>> Hi Vijay,
> >>>>>>>>>>
> >>>>>>>>>> I've had some time to play with this now, and I have another
> >>>>>>>>>> question.
> >>>>>>>>>> Thank you for sending along the example code, it has been
> >>>>>>>>>> extremely
> >>>>>>>>>> helpful.
> >>>>>>>>>>
> >>>>>>>>>> The example shows the ghost exchange happening between shared
> >>>>>>>>>> owned
> >>>>>>>>>> entities. In my experiments I've been unable to make the
> >>>>>>>>>> ghost exchange
> >>>>>>>>>> work, and I think that might be because my entities are not
> >>>>>>>>>> shared. The
> >>>>>>>>>> situation I have is entities that are owned wholly on a single
> >>>>>>>>>> processor, which need to be communicated to other processors
> >>>>>>>>>> which
> >>>>>>>>>> require them as part of a halo region for mesh based
> >>>>>>>>>> computation. In
> >>>>>>>>>> this situation would I need to "share" my entities across the
> >>>>>>>>>> whole
> >>>>>>>>>> processor space, before requesting a ghost exchange? I'm not
> >>>>>>>>>> even really
> >>>>>>>>>> sure what shared entities mean, is there a good place to look
> >>>>>>>>>> in the
> >>>>>>>>>> documentation to learn more about the terminology? Thanks,
> >>>>>>>>>>
> >>>>>>>>>> Ryan
> >>>>>>>>>>
> >>>>>>>>>> On 10/8/18 12:17, Vijay S. Mahadevan wrote:
> >>>>>>>>>>> Ryan,
> >>>>>>>>>>>
> >>>>>>>>>>> You need to use the ParallelComm object in MOAB to call
> >>>>>>>>>>> exchange_ghost_cells [1] with appropriate levels of ghost
> >>>>>>>>>>> layers for
> >>>>>>>>>>> your mesh. This needs to be done after the mesh is loaded,
> >>>>>>>>>>> or you can
> >>>>>>>>>>> pass this information also as part of the options when
> >>>>>>>>>>> loading a file
> >>>>>>>>>>> and MOAB will internally load the file with ghosted layers.
> >>>>>>>>>>> Here's an
> >>>>>>>>>>> example [2].
> >>>>>>>>>>>
> >>>>>>>>>>> Vijay
> >>>>>>>>>>>
> >>>>>>>>>>> [1]
> >>>>>>>>>>> http://ftp.mcs.anl.gov/pub/fathom/moab-docs-develop/classmoab_1_1ParallelComm.html#a55dfa308f56fd368319bfb4244428878
> >>>>>>>>>>> [2]
> >>>>>>>>>>> http://ftp.mcs.anl.gov/pub/fathom/moab-docs-develop/HelloParMOAB_8cpp-example.html
> >>>>>>>>>>> On Mon, Oct 8, 2018 at 1:23 PM Ryan O'Kuinghttons
> >>>>>>>>>>> <ryan.okuinghttons at noaa.gov> wrote:
> >>>>>>>>>>>> Hi, I am wondering if there is a way to create ghost
> >>>>>>>>>>>> elements in MOAB.
> >>>>>>>>>>>> By this I mean a list of copies of elements surrounding a
> >>>>>>>>>>>> specific MOAB
> >>>>>>>>>>>> element, ghost elements may exist on a different processor
> >>>>>>>>>>>> than the
> >>>>>>>>>>>> source element. I see a ghost_elems class in the appData
> >>>>>>>>>>>> namespace, but
> >>>>>>>>>>>> there is not much documentation on how to use it.. Thank you,
> >>>>>>>>>>>>
> >>>>>>>>>>>> --
> >>>>>>>>>>>> Ryan O'Kuinghttons
> >>>>>>>>>>>> Cherokee Nation Management and Consulting
> >>>>>>>>>>>> NESII/NOAA/Earth System Research Laboratory
> >>>>>>>>>>>> ryan.okuinghttons at noaa.gov
> >>>>>>>>>>>> https://www.esrl.noaa.gov/gsd/nesii/
> >>>>>>>>>>>>

More information about the moab-dev mailing list