[MOAB-dev] ghost/halo elements
Vijay S. Mahadevan
vijay.m at gmail.com
Mon Oct 29 13:05:53 CDT 2018
Hi Ryan,
The method `augment_default_sets_with_ghosts` [1] in ParallelComm
enables this for some of the default tags. We haven't exposed this as
a very general mechanism as the user could define the tag data layout
in many possible ways and it would be difficult to cover all use cases
from a library standpoint. Remember that for custom defined Tags, you
can also use exchange_tag/reduce_tags in ParallelComm to get the data
that you need on shared entities. If exchange_tags does not suit your
needs, you can see if you can adapt augment_default_sets_with_ghosts
method for your use case. Let me know if that helps.
Vijay
[1] http://ftp.mcs.anl.gov/pub/fathom/moab-docs-develop/classmoab_1_1ParallelComm.html#a76dced0b6dbe2fb99340945c7ca3c0f2
On Mon, Oct 29, 2018 at 1:49 PM Ryan O'Kuinghttons
<ryan.okuinghttons at noaa.gov> wrote:
>
> Hi Iulian, Vijay,
>
> We have a set of custom tags that we use in ESMF, would it be possible
> to transfer these as well?
>
> Ryan
>
> On 10/29/18 10:54, Vijay S. Mahadevan wrote:
> > The default set of tags do get transferred. These would be the
> > MATERIAL_SET, DIRICHLET/NEUMANN sets etc.
> >
> > Iulian, did we add a separate method to transfer these to the ghosted
> > entities ? I vaguely remember something like that.
> >
> > Vijay
> > On Mon, Oct 29, 2018 at 12:42 PM Ryan O'Kuinghttons
> > <ryan.okuinghttons at noaa.gov> wrote:
> >> Hi Vijay,
> >>
> >> I ran into another issue with the call to exchange_ghost_cells, but this
> >> is probably just a misunderstanding on my part. I find that the tag
> >> information that is part of the mesh before the the call does not
> >> transfer to the newly created faces. Is there some way to ask moab to
> >> transfer tags when creating ghost entities? (FWIW, I could not find this
> >> in the example you sent me) Thanks,
> >>
> >> Ryan
> >>
> >> On 10/26/18 07:36, Ryan O'Kuinghttons wrote:
> >>> Hi Vijay,
> >>>
> >>> I resolved the issue with not getting any shared, owned entities back
> >>> by using the other version of resolve_shared_ents, as demonstrated in
> >>> the example you indicated. I created a Range containing all elements
> >>> (faces for moab), and used that as input.
> >>>
> >>> Range range_ent;
> >>> merr=mb->get_entities_by_dimension(0,this->sdim,range_ent);
> >>> MBMESH_CHECK_ERR(merr, localrc);
> >>>
> >>> merr = pcomm->resolve_shared_ents(0, range_ent, this->sdim, 1);
> >>> MBMESH_CHECK_ERR(merr, localrc);
> >>>
> >>> The first version of the interface (which does not use a Range input)
> >>> does not work for me:
> >>>
> >>> merr = pcomm->resolve_shared_ents(0, this->sdim, 1);
> >>>
> >>> Otherwise, everything else seems to be resolved with this issue,
> >>> thanks again for all of your help!!
> >>>
> >>> Ryan
> >>>
> >>> On 10/25/18 15:13, Vijay S. Mahadevan wrote:
> >>>> Ryan, I don't see anything obviously wrong in the code below. Does
> >>>> your shared_ents not have any entities after parallel resolve ? If you
> >>>> can replicate your issue in a small standalone test, we should be able
> >>>> to quickly find out what's going wrong.
> >>>>
> >>>> Look at the test_reduce_tag_explicit_dest routine in
> >>>> test/parallel/parallel_unit_tests.cpp:1617. It has the workflow where
> >>>> you create a mesh in memory, resolve shared entities, get shared
> >>>> entities and even set/reduce tag data.
> >>>>
> >>>> Vijay
> >>>> On Thu, Oct 25, 2018 at 4:25 PM Ryan O'Kuinghttons
> >>>> <ryan.okuinghttons at noaa.gov> wrote:
> >>>>> OK, that allows the resolve_shared_ents routine to complete without
> >>>>> error, but I still don't get any shared, owned entities back from the
> >>>>> get_shared_entities/filter_pstatus calls:
> >>>>>
> >>>>> Range shared_ents;
> >>>>> // Get entities shared with all other processors
> >>>>> merr = pcomm->get_shared_entities(-1, shared_ents);
> >>>>> MBMESH_CHECK_ERR(merr, localrc);
> >>>>>
> >>>>> // Filter shared entities with not not_owned, which means owned
> >>>>> Range owned_entities;
> >>>>> merr = pcomm->filter_pstatus(shared_ents, PSTATUS_NOT_OWNED,
> >>>>> PSTATUS_NOT, -1, &owned_entities);
> >>>>> MBMESH_CHECK_ERR(merr, localrc);
> >>>>>
> >>>>> unsigned int nums[4] = {0}; // to store the owned entities per
> >>>>> dimension
> >>>>> for (int i = 0; i < 4; i++)
> >>>>> nums[i] = (int)owned_entities.num_of_dimension(i);
> >>>>>
> >>>>> vector<int> rbuf(nprocs*4, 0);
> >>>>> MPI_Gather(nums, 4, MPI_INT, &rbuf[0], 4, MPI_INT, 0, mpi_comm);
> >>>>> // Print the stats gathered:
> >>>>> if (0 == rank) {
> >>>>> for (int i = 0; i < nprocs; i++)
> >>>>> cout << " Shared, owned entities on proc " << i << ": " <<
> >>>>> rbuf[4*i] << " verts, " <<
> >>>>> rbuf[4*i + 1] << " edges, " << rbuf[4*i + 2] << " faces,
> >>>>> " <<
> >>>>> rbuf[4*i + 3] << " elements" << endl;
> >>>>> }
> >>>>>
> >>>>> OUTPUT:
> >>>>>
> >>>>> Shared, owned entities on proc 0: 0 verts, 0 edges, 0 faces, 0
> >>>>> elements
> >>>>> Shared, owned entities on proc 1: 0 verts, 0 edges, 0 faces, 0
> >>>>> elements
> >>>>> Shared, owned entities on proc 2: 0 verts, 0 edges, 0 faces, 0
> >>>>> elements
> >>>>> Shared, owned entities on proc 3: 0 verts, 0 edges, 0 faces, 0
> >>>>> elements
> >>>>>
> >>>>>
> >>>>> Note: I also tried changing the -1 to 2 in the
> >>>>> get_shared_entities/filter_pstatus calls to only retrieve elements.
> >>>>>
> >>>>>
> >>>>> On 10/25/18 13:57, Vijay S. Mahadevan wrote:
> >>>>>> No problem. Can you try 1 for the last argument to look for edges as
> >>>>>> the shared_dim.
> >>>>>>
> >>>>>> Vijay
> >>>>>> On Thu, Oct 25, 2018 at 3:41 PM Ryan O'Kuinghttons
> >>>>>> <ryan.okuinghttons at noaa.gov> wrote:
> >>>>>>> OK, thanks again for that explanation, I never would have been
> >>>>>>> able to figure out that this_set was the root set. I apologize for
> >>>>>>> all the dumb questions, hopefully they are all getting me closer
> >>>>>>> to self sufficiency..
> >>>>>>>
> >>>>>>> I'm getting an error that doesn't make sense to me. I specifying
> >>>>>>> the resolve_dim and shared_dim in the call:
> >>>>>>>
> >>>>>>> // Get the ParallelComm instance
> >>>>>>> ParallelComm* pcomm = new ParallelComm(mb, mpi_comm);
> >>>>>>> int nprocs = pcomm->proc_config().proc_size();
> >>>>>>> int rank = pcomm->proc_config().proc_rank();
> >>>>>>>
> >>>>>>> // get the root set handle
> >>>>>>> EntityHandle root_set = mb->get_root_set();
> >>>>>>>
> >>>>>>> merr = pcomm->resolve_shared_ents(root_set, 2, -1);
> >>>>>>>
> >>>>>>>
> >>>>>>> but I'm getting this message:
> >>>>>>>
> >>>>>>> [0]MOAB ERROR: Unable to guess shared_dim or resolve_dim!
> >>>>>>>
> >>>>>>>
> >>>>>>> what am i missing?
> >>>>>>>
> >>>>>>>
> >>>>>>> On 10/25/18 12:51, Vijay S. Mahadevan wrote:
> >>>>>>>
> >>>>>>> It is your root set or fileset if you added entities to it. You
> >>>>>>> can get the root set handle from Core object.
> >>>>>>>
> >>>>>>> Vijay
> >>>>>>>
> >>>>>>> On Thu, Oct 25, 2018, 14:38 Ryan O'Kuinghttons
> >>>>>>> <ryan.okuinghttons at noaa.gov> wrote:
> >>>>>>>> Thanks again, Vijay. However, I still don't understand what this_set
> >>>>>>>> should be, is it an output maybe?
> >>>>>>>>
> >>>>>>>> Ryan
> >>>>>>>>
> >>>>>>>> On 10/25/18 12:27, Vijay S. Mahadevan wrote:
> >>>>>>>>> You can try either the first or the second variant instead [1] with
> >>>>>>>>> following arguments.
> >>>>>>>>>
> >>>>>>>>> ErrorCode moab::ParallelComm::resolve_shared_ents(EntityHandle
> >>>>>>>>> this_set,
> >>>>>>>>> int resolve_dim = 2,
> >>>>>>>>> int shared_dim = -1,
> >>>>>>>>> const Tag * id_tag = 0
> >>>>>>>>> )
> >>>>>>>>>
> >>>>>>>>> That should resolve 2-dim entities with shared edges across
> >>>>>>>>> partitions. You can leave id_tag pointer as zero since the
> >>>>>>>>> default is
> >>>>>>>>> GLOBAL_ID.
> >>>>>>>>>
> >>>>>>>>>> This brings up a more general question I've had about the moab
> >>>>>>>>> documentation for a while. In the doc for this routine, it only
> >>>>>>>>> lists 2
> >>>>>>>>> parameters, proc_ents and shared_dim, even though in the function
> >>>>>>>>> signature above it clearly shows more. I've had trouble
> >>>>>>>>> understanding
> >>>>>>>>> which parameters are relevant in the past, or what they do
> >>>>>>>>> because I'm
> >>>>>>>>> not quite sure how to read the documentation.
> >>>>>>>>>
> >>>>>>>>> This is an oversight. We will go through and rectify some of the
> >>>>>>>>> inconsistencies in the documentation. We are preparing for an
> >>>>>>>>> upcoming
> >>>>>>>>> release and I'll make sure that routines in Core/ParallelComm have
> >>>>>>>>> updated documentation that match the interfaces. Meanwhile, if you
> >>>>>>>>> have questions, feel free to shoot an email to the list.
> >>>>>>>>>
> >>>>>>>>> Hope that helps.
> >>>>>>>>>
> >>>>>>>>> Vijay
> >>>>>>>>>
> >>>>>>>>> [1]
> >>>>>>>>> http://ftp.mcs.anl.gov/pub/fathom/moab-docs-develop/classmoab_1_1ParallelComm.html#a29a3b834b3fc3b4ddb3a5d8a78a37c8a
> >>>>>>>>> On Thu, Oct 25, 2018 at 1:58 PM Ryan O'Kuinghttons
> >>>>>>>>> <ryan.okuinghttons at noaa.gov> wrote:
> >>>>>>>>>> Hi Vijay,
> >>>>>>>>>>
> >>>>>>>>>> Thanks again for that explanation.
> >>>>>>>>>>
> >>>>>>>>>> ESMF does use unique global ids for the vertices and set them
> >>>>>>>>>> to the
> >>>>>>>>>> GLOBAL_ID in the moab mesh. So I think we are good there.
> >>>>>>>>>>
> >>>>>>>>>> I can't quite figure out how to use resolve_shared_entities
> >>>>>>>>>> though.
> >>>>>>>>>> There are three versions of the call in the documentation, and
> >>>>>>>>>> I assume
> >>>>>>>>>> that I should use the first and pass in a Range containing all
> >>>>>>>>>> entities
> >>>>>>>>>> for proc_ents:
> >>>>>>>>>>
> >>>>>>>>>> http://ftp.mcs.anl.gov/pub/fathom/moab-docs-develop/classmoab_1_1ParallelComm.html#a59e35d9906f2e33fe010138a144a5cb6
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> However, I'm not sure what the this_set EntityHandle should be.
> >>>>>>>>>>
> >>>>>>>>>> This brings up a more general question I've had about the moab
> >>>>>>>>>> documentation for a while. In the doc for this routine, it only
> >>>>>>>>>> lists 2
> >>>>>>>>>> parameters, proc_ents and shared_dim, even though in the function
> >>>>>>>>>> signature above it clearly shows more. I've had trouble
> >>>>>>>>>> understanding
> >>>>>>>>>> which parameters are relevant in the past, or what they do
> >>>>>>>>>> because I'm
> >>>>>>>>>> not quite sure how to read the documentation.
> >>>>>>>>>>
> >>>>>>>>>> Any example or explanation you give me for
> >>>>>>>>>> resolve_shared_entities is
> >>>>>>>>>> much appreciated!
> >>>>>>>>>>
> >>>>>>>>>> Ryan
> >>>>>>>>>>
> >>>>>>>>>> On 10/25/18 09:40, Vijay S. Mahadevan wrote:
> >>>>>>>>>>> Hi Ryan,
> >>>>>>>>>>>
> >>>>>>>>>>> Glad the example helped.
> >>>>>>>>>>>
> >>>>>>>>>>>> The example shows the ghost exchange happening between shared
> >>>>>>>>>>>> owned
> >>>>>>>>>>> entities. In my experiments I've been unable to make the ghost
> >>>>>>>>>>> exchange
> >>>>>>>>>>> work, and I think that might be because my entities are not
> >>>>>>>>>>> shared.
> >>>>>>>>>>>
> >>>>>>>>>>> You need to call resolve_shared_entities on the entities prior to
> >>>>>>>>>>> doing ghost exchange. When we load the mesh from file, this
> >>>>>>>>>>> happens
> >>>>>>>>>>> automatically based on the read options but if you are forming
> >>>>>>>>>>> a mesh
> >>>>>>>>>>> in memory, you need to make sure that the shared vertices have
> >>>>>>>>>>> the
> >>>>>>>>>>> same GLOBAL_ID numbering consistently across processes. That
> >>>>>>>>>>> is shared
> >>>>>>>>>>> vertices are unique. Once that is set, shared entities
> >>>>>>>>>>> resolution will
> >>>>>>>>>>> work correctly out of the box and you will have shared
> >>>>>>>>>>> edges/entities
> >>>>>>>>>>> query working correctly. A call to get ghosted layers once
> >>>>>>>>>>> this is
> >>>>>>>>>>> done would be the way to go.
> >>>>>>>>>>>
> >>>>>>>>>>> I assume ESMF has a unique global numbering for the vertices ?
> >>>>>>>>>>> Use
> >>>>>>>>>>> that to set the GLOBAL_ID tag. Let us know if you are still
> >>>>>>>>>>> having
> >>>>>>>>>>> issues.
> >>>>>>>>>>>
> >>>>>>>>>>> Vijay
> >>>>>>>>>>> On Thu, Oct 25, 2018 at 11:17 AM Ryan O'Kuinghttons
> >>>>>>>>>>> <ryan.okuinghttons at noaa.gov> wrote:
> >>>>>>>>>>>> Hi Vijay,
> >>>>>>>>>>>>
> >>>>>>>>>>>> I've had some time to play with this now, and I have another
> >>>>>>>>>>>> question.
> >>>>>>>>>>>> Thank you for sending along the example code, it has been
> >>>>>>>>>>>> extremely
> >>>>>>>>>>>> helpful.
> >>>>>>>>>>>>
> >>>>>>>>>>>> The example shows the ghost exchange happening between shared
> >>>>>>>>>>>> owned
> >>>>>>>>>>>> entities. In my experiments I've been unable to make the
> >>>>>>>>>>>> ghost exchange
> >>>>>>>>>>>> work, and I think that might be because my entities are not
> >>>>>>>>>>>> shared. The
> >>>>>>>>>>>> situation I have is entities that are owned wholly on a single
> >>>>>>>>>>>> processor, which need to be communicated to other processors
> >>>>>>>>>>>> which
> >>>>>>>>>>>> require them as part of a halo region for mesh based
> >>>>>>>>>>>> computation. In
> >>>>>>>>>>>> this situation would I need to "share" my entities across the
> >>>>>>>>>>>> whole
> >>>>>>>>>>>> processor space, before requesting a ghost exchange? I'm not
> >>>>>>>>>>>> even really
> >>>>>>>>>>>> sure what shared entities mean, is there a good place to look
> >>>>>>>>>>>> in the
> >>>>>>>>>>>> documentation to learn more about the terminology? Thanks,
> >>>>>>>>>>>>
> >>>>>>>>>>>> Ryan
> >>>>>>>>>>>>
> >>>>>>>>>>>> On 10/8/18 12:17, Vijay S. Mahadevan wrote:
> >>>>>>>>>>>>> Ryan,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> You need to use the ParallelComm object in MOAB to call
> >>>>>>>>>>>>> exchange_ghost_cells [1] with appropriate levels of ghost
> >>>>>>>>>>>>> layers for
> >>>>>>>>>>>>> your mesh. This needs to be done after the mesh is loaded,
> >>>>>>>>>>>>> or you can
> >>>>>>>>>>>>> pass this information also as part of the options when
> >>>>>>>>>>>>> loading a file
> >>>>>>>>>>>>> and MOAB will internally load the file with ghosted layers.
> >>>>>>>>>>>>> Here's an
> >>>>>>>>>>>>> example [2].
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Vijay
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> [1]
> >>>>>>>>>>>>> http://ftp.mcs.anl.gov/pub/fathom/moab-docs-develop/classmoab_1_1ParallelComm.html#a55dfa308f56fd368319bfb4244428878
> >>>>>>>>>>>>> [2]
> >>>>>>>>>>>>> http://ftp.mcs.anl.gov/pub/fathom/moab-docs-develop/HelloParMOAB_8cpp-example.html
> >>>>>>>>>>>>> On Mon, Oct 8, 2018 at 1:23 PM Ryan O'Kuinghttons
> >>>>>>>>>>>>> <ryan.okuinghttons at noaa.gov> wrote:
> >>>>>>>>>>>>>> Hi, I am wondering if there is a way to create ghost
> >>>>>>>>>>>>>> elements in MOAB.
> >>>>>>>>>>>>>> By this I mean a list of copies of elements surrounding a
> >>>>>>>>>>>>>> specific MOAB
> >>>>>>>>>>>>>> element, ghost elements may exist on a different processor
> >>>>>>>>>>>>>> than the
> >>>>>>>>>>>>>> source element. I see a ghost_elems class in the appData
> >>>>>>>>>>>>>> namespace, but
> >>>>>>>>>>>>>> there is not much documentation on how to use it.. Thank you,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> --
> >>>>>>>>>>>>>> Ryan O'Kuinghttons
> >>>>>>>>>>>>>> Cherokee Nation Management and Consulting
> >>>>>>>>>>>>>> NESII/NOAA/Earth System Research Laboratory
> >>>>>>>>>>>>>> ryan.okuinghttons at noaa.gov
> >>>>>>>>>>>>>> https://www.esrl.noaa.gov/gsd/nesii/
> >>>>>>>>>>>>>>
More information about the moab-dev
mailing list