[MOAB-dev] ghost/halo elements

Vijay S. Mahadevan vijay.m at gmail.com
Thu Oct 25 16:13:11 CDT 2018 

Ryan, I don't see anything obviously wrong in the code below. Does
your shared_ents not have any entities after parallel resolve ? If you
can replicate your issue in a small standalone test, we should be able
to quickly find out what's going wrong.

Look at the test_reduce_tag_explicit_dest routine in
test/parallel/parallel_unit_tests.cpp:1617. It has the workflow where
you create a mesh in memory, resolve shared entities, get shared
entities and even set/reduce tag data.

Vijay
On Thu, Oct 25, 2018 at 4:25 PM Ryan O'Kuinghttons
<ryan.okuinghttons at noaa.gov> wrote:
>
> OK, that allows the resolve_shared_ents routine to complete without
> error, but I still don't get any shared, owned entities back from the
> get_shared_entities/filter_pstatus calls:
>
>    Range shared_ents;
>    // Get entities shared with all other processors
>    merr = pcomm->get_shared_entities(-1, shared_ents);
>    MBMESH_CHECK_ERR(merr, localrc);
>
>    // Filter shared entities with not not_owned, which means owned
>    Range owned_entities;
>    merr = pcomm->filter_pstatus(shared_ents, PSTATUS_NOT_OWNED,
> PSTATUS_NOT, -1, &owned_entities);
>    MBMESH_CHECK_ERR(merr, localrc);
>
>    unsigned int nums[4] = {0}; // to store the owned entities per dimension
>    for (int i = 0; i < 4; i++)
>      nums[i] = (int)owned_entities.num_of_dimension(i);
>
>    vector<int> rbuf(nprocs*4, 0);
>    MPI_Gather(nums, 4, MPI_INT, &rbuf[0], 4, MPI_INT, 0, mpi_comm);
>    // Print the stats gathered:
>    if (0 == rank) {
>      for (int i = 0; i < nprocs; i++)
>        cout << " Shared, owned entities on proc " << i << ": " <<
> rbuf[4*i] << " verts, " <<
>            rbuf[4*i + 1] << " edges, " << rbuf[4*i + 2] << " faces, " <<
> rbuf[4*i + 3] << " elements" << endl;
>    }
>
> OUTPUT:
>
>   Shared, owned entities on proc 0: 0 verts, 0 edges, 0 faces, 0 elements
>   Shared, owned entities on proc 1: 0 verts, 0 edges, 0 faces, 0 elements
>   Shared, owned entities on proc 2: 0 verts, 0 edges, 0 faces, 0 elements
>   Shared, owned entities on proc 3: 0 verts, 0 edges, 0 faces, 0 elements
>
>
> Note: I also tried changing the -1 to 2 in the
> get_shared_entities/filter_pstatus calls to only retrieve elements.
>
>
> On 10/25/18 13:57, Vijay S. Mahadevan wrote:
> > No problem. Can you try 1 for the last argument to look for edges as
> > the shared_dim.
> >
> > Vijay
> > On Thu, Oct 25, 2018 at 3:41 PM Ryan O'Kuinghttons
> > <ryan.okuinghttons at noaa.gov> wrote:
> >> OK, thanks again for that explanation, I never would have been able to figure out that this_set was the root set. I apologize for all the dumb questions, hopefully they are all getting me closer to self sufficiency..
> >>
> >> I'm getting an error that doesn't make sense to me. I specifying the resolve_dim and shared_dim in the call:
> >>
> >>    // Get the ParallelComm instance
> >>    ParallelComm* pcomm = new ParallelComm(mb, mpi_comm);
> >>    int nprocs = pcomm->proc_config().proc_size();
> >>    int rank = pcomm->proc_config().proc_rank();
> >>
> >>    // get the root set handle
> >>    EntityHandle root_set = mb->get_root_set();
> >>
> >>    merr = pcomm->resolve_shared_ents(root_set, 2, -1);
> >>
> >>
> >> but I'm getting this message:
> >>
> >> [0]MOAB ERROR: Unable to guess shared_dim or resolve_dim!
> >>
> >>
> >> what am i missing?
> >>
> >>
> >> On 10/25/18 12:51, Vijay S. Mahadevan wrote:
> >>
> >> It is your root set or fileset if you added entities to it. You can get the root set handle from Core object.
> >>
> >> Vijay
> >>
> >> On Thu, Oct 25, 2018, 14:38 Ryan O'Kuinghttons <ryan.okuinghttons at noaa.gov> wrote:
> >>> Thanks again, Vijay. However, I still don't understand what this_set
> >>> should be, is it an output maybe?
> >>>
> >>> Ryan
> >>>
> >>> On 10/25/18 12:27, Vijay S. Mahadevan wrote:
> >>>> You can try either the first or the second variant instead [1] with
> >>>> following arguments.
> >>>>
> >>>> ErrorCode moab::ParallelComm::resolve_shared_ents(EntityHandle this_set,
> >>>> int resolve_dim = 2,
> >>>> int shared_dim = -1,
> >>>> const Tag * id_tag = 0
> >>>> )
> >>>>
> >>>> That should resolve 2-dim entities with shared edges across
> >>>> partitions. You can leave id_tag pointer as zero since the default is
> >>>> GLOBAL_ID.
> >>>>
> >>>>> This brings up a more general question I've had about the moab
> >>>> documentation for a while. In the doc for this routine, it only lists 2
> >>>> parameters, proc_ents and shared_dim, even though in the function
> >>>> signature above it clearly shows more. I've had trouble understanding
> >>>> which parameters are relevant in the past, or what they do because I'm
> >>>> not quite sure how to read the documentation.
> >>>>
> >>>> This is an oversight. We will go through and rectify some of the
> >>>> inconsistencies in the documentation. We are preparing for an upcoming
> >>>> release and I'll make sure that routines in Core/ParallelComm have
> >>>> updated documentation that match the interfaces. Meanwhile, if you
> >>>> have questions, feel free to shoot an email to the list.
> >>>>
> >>>> Hope that helps.
> >>>>
> >>>> Vijay
> >>>>
> >>>> [1] http://ftp.mcs.anl.gov/pub/fathom/moab-docs-develop/classmoab_1_1ParallelComm.html#a29a3b834b3fc3b4ddb3a5d8a78a37c8a
> >>>> On Thu, Oct 25, 2018 at 1:58 PM Ryan O'Kuinghttons
> >>>> <ryan.okuinghttons at noaa.gov> wrote:
> >>>>> Hi Vijay,
> >>>>>
> >>>>> Thanks again for that explanation.
> >>>>>
> >>>>> ESMF does use unique global ids for the vertices and set them to the
> >>>>> GLOBAL_ID in the moab mesh. So I think we are good there.
> >>>>>
> >>>>> I can't quite figure out how to use resolve_shared_entities though.
> >>>>> There are three versions of the call in the documentation, and I assume
> >>>>> that I should use the first and pass in a Range containing all entities
> >>>>> for proc_ents:
> >>>>>
> >>>>> http://ftp.mcs.anl.gov/pub/fathom/moab-docs-develop/classmoab_1_1ParallelComm.html#a59e35d9906f2e33fe010138a144a5cb6
> >>>>>
> >>>>> However, I'm not sure what the this_set EntityHandle should be.
> >>>>>
> >>>>> This brings up a more general question I've had about the moab
> >>>>> documentation for a while. In the doc for this routine, it only lists 2
> >>>>> parameters, proc_ents and shared_dim, even though in the function
> >>>>> signature above it clearly shows more. I've had trouble understanding
> >>>>> which parameters are relevant in the past, or what they do because I'm
> >>>>> not quite sure how to read the documentation.
> >>>>>
> >>>>> Any example or explanation you give me for resolve_shared_entities is
> >>>>> much appreciated!
> >>>>>
> >>>>> Ryan
> >>>>>
> >>>>> On 10/25/18 09:40, Vijay S. Mahadevan wrote:
> >>>>>> Hi Ryan,
> >>>>>>
> >>>>>> Glad the example helped.
> >>>>>>
> >>>>>>> The example shows the ghost exchange happening between shared owned
> >>>>>> entities. In my experiments I've been unable to make the ghost exchange
> >>>>>> work, and I think that might be because my entities are not shared.
> >>>>>>
> >>>>>> You need to call resolve_shared_entities on the entities prior to
> >>>>>> doing ghost exchange. When we load the mesh from file, this happens
> >>>>>> automatically based on the read options but if you are forming a mesh
> >>>>>> in memory, you need to make sure that the shared vertices have the
> >>>>>> same GLOBAL_ID numbering consistently across processes. That is shared
> >>>>>> vertices are unique. Once that is set, shared entities resolution will
> >>>>>> work correctly out of the box and you will have shared edges/entities
> >>>>>> query working correctly. A call to get ghosted layers once this is
> >>>>>> done would be the way to go.
> >>>>>>
> >>>>>> I assume ESMF has a unique global numbering for the vertices ? Use
> >>>>>> that to set the GLOBAL_ID tag. Let us know if you are still having
> >>>>>> issues.
> >>>>>>
> >>>>>> Vijay
> >>>>>> On Thu, Oct 25, 2018 at 11:17 AM Ryan O'Kuinghttons
> >>>>>> <ryan.okuinghttons at noaa.gov> wrote:
> >>>>>>> Hi Vijay,
> >>>>>>>
> >>>>>>> I've had some time to play with this now, and I have another question.
> >>>>>>> Thank you for sending along the example code, it has been extremely
> >>>>>>> helpful.
> >>>>>>>
> >>>>>>> The example shows the ghost exchange happening between shared owned
> >>>>>>> entities. In my experiments I've been unable to make the ghost exchange
> >>>>>>> work, and I think that might be because my entities are not shared. The
> >>>>>>> situation I have is entities that are owned wholly on a single
> >>>>>>> processor, which need to be communicated to other processors which
> >>>>>>> require them as part of a halo region for mesh based computation. In
> >>>>>>> this situation would I need to "share" my entities across the whole
> >>>>>>> processor space, before requesting a ghost exchange? I'm not even really
> >>>>>>> sure what shared entities mean, is there a good place to look in the
> >>>>>>> documentation to learn more about the terminology? Thanks,
> >>>>>>>
> >>>>>>> Ryan
> >>>>>>>
> >>>>>>> On 10/8/18 12:17, Vijay S. Mahadevan wrote:
> >>>>>>>> Ryan,
> >>>>>>>>
> >>>>>>>> You need to use the ParallelComm object in MOAB to call
> >>>>>>>> exchange_ghost_cells [1] with appropriate levels of ghost layers for
> >>>>>>>> your mesh. This needs to be done after the mesh is loaded, or you can
> >>>>>>>> pass this information also as part of the options when loading a file
> >>>>>>>> and MOAB will internally load the file with ghosted layers. Here's an
> >>>>>>>> example [2].
> >>>>>>>>
> >>>>>>>> Vijay
> >>>>>>>>
> >>>>>>>> [1] http://ftp.mcs.anl.gov/pub/fathom/moab-docs-develop/classmoab_1_1ParallelComm.html#a55dfa308f56fd368319bfb4244428878
> >>>>>>>> [2] http://ftp.mcs.anl.gov/pub/fathom/moab-docs-develop/HelloParMOAB_8cpp-example.html
> >>>>>>>> On Mon, Oct 8, 2018 at 1:23 PM Ryan O'Kuinghttons
> >>>>>>>> <ryan.okuinghttons at noaa.gov> wrote:
> >>>>>>>>> Hi, I am wondering if there is a way to create ghost elements in MOAB.
> >>>>>>>>> By this I mean a list of copies of elements surrounding a specific MOAB
> >>>>>>>>> element, ghost elements may exist on a different processor than the
> >>>>>>>>> source element. I see a ghost_elems class in the appData namespace, but
> >>>>>>>>> there is not much documentation on how to use it.. Thank you,
> >>>>>>>>>
> >>>>>>>>> --
> >>>>>>>>> Ryan O'Kuinghttons
> >>>>>>>>> Cherokee Nation Management and Consulting
> >>>>>>>>> NESII/NOAA/Earth System Research Laboratory
> >>>>>>>>> ryan.okuinghttons at noaa.gov
> >>>>>>>>> https://www.esrl.noaa.gov/gsd/nesii/
> >>>>>>>>>

More information about the moab-dev mailing list