[MOAB-dev] ghost/halo elements

[Vijay S. Mahadevan]

No problem. Can you try 1 for the last argument to look for edges as
the shared_dim.

Vijay
On Thu, Oct 25, 2018 at 3:41 PM Ryan O'Kuinghttons
<ryan.okuinghttons at noaa.gov> wrote:
>
> OK, thanks again for that explanation, I never would have been able to figure out that this_set was the root set. I apologize for all the dumb questions, hopefully they are all getting me closer to self sufficiency..
>
> I'm getting an error that doesn't make sense to me. I specifying the resolve_dim and shared_dim in the call:
>
>   // Get the ParallelComm instance
>   ParallelComm* pcomm = new ParallelComm(mb, mpi_comm);
>   int nprocs = pcomm->proc_config().proc_size();
>   int rank = pcomm->proc_config().proc_rank();
>
>   // get the root set handle
>   EntityHandle root_set = mb->get_root_set();
>
>   merr = pcomm->resolve_shared_ents(root_set, 2, -1);
>
>
> but I'm getting this message:
>
> [0]MOAB ERROR: Unable to guess shared_dim or resolve_dim!
>
>
> what am i missing?
>
>
> On 10/25/18 12:51, Vijay S. Mahadevan wrote:
>
> It is your root set or fileset if you added entities to it. You can get the root set handle from Core object.
>
> Vijay
>
> On Thu, Oct 25, 2018, 14:38 Ryan O'Kuinghttons <ryan.okuinghttons at noaa.gov> wrote:
>>
>> Thanks again, Vijay. However, I still don't understand what this_set
>> should be, is it an output maybe?
>>
>> Ryan
>>
>> On 10/25/18 12:27, Vijay S. Mahadevan wrote:
>> > You can try either the first or the second variant instead [1] with
>> > following arguments.
>> >
>> > ErrorCode moab::ParallelComm::resolve_shared_ents(EntityHandle this_set,
>> > int resolve_dim = 2,
>> > int shared_dim = -1,
>> > const Tag * id_tag = 0
>> > )
>> >
>> > That should resolve 2-dim entities with shared edges across
>> > partitions. You can leave id_tag pointer as zero since the default is
>> > GLOBAL_ID.
>> >
>> >> This brings up a more general question I've had about the moab
>> > documentation for a while. In the doc for this routine, it only lists 2
>> > parameters, proc_ents and shared_dim, even though in the function
>> > signature above it clearly shows more. I've had trouble understanding
>> > which parameters are relevant in the past, or what they do because I'm
>> > not quite sure how to read the documentation.
>> >
>> > This is an oversight. We will go through and rectify some of the
>> > inconsistencies in the documentation. We are preparing for an upcoming
>> > release and I'll make sure that routines in Core/ParallelComm have
>> > updated documentation that match the interfaces. Meanwhile, if you
>> > have questions, feel free to shoot an email to the list.
>> >
>> > Hope that helps.
>> >
>> > Vijay
>> >
>> > [1] http://ftp.mcs.anl.gov/pub/fathom/moab-docs-develop/classmoab_1_1ParallelComm.html#a29a3b834b3fc3b4ddb3a5d8a78a37c8a
>> > On Thu, Oct 25, 2018 at 1:58 PM Ryan O'Kuinghttons
>> > <ryan.okuinghttons at noaa.gov> wrote:
>> >> Hi Vijay,
>> >>
>> >> Thanks again for that explanation.
>> >>
>> >> ESMF does use unique global ids for the vertices and set them to the
>> >> GLOBAL_ID in the moab mesh. So I think we are good there.
>> >>
>> >> I can't quite figure out how to use resolve_shared_entities though.
>> >> There are three versions of the call in the documentation, and I assume
>> >> that I should use the first and pass in a Range containing all entities
>> >> for proc_ents:
>> >>
>> >> http://ftp.mcs.anl.gov/pub/fathom/moab-docs-develop/classmoab_1_1ParallelComm.html#a59e35d9906f2e33fe010138a144a5cb6
>> >>
>> >> However, I'm not sure what the this_set EntityHandle should be.
>> >>
>> >> This brings up a more general question I've had about the moab
>> >> documentation for a while. In the doc for this routine, it only lists 2
>> >> parameters, proc_ents and shared_dim, even though in the function
>> >> signature above it clearly shows more. I've had trouble understanding
>> >> which parameters are relevant in the past, or what they do because I'm
>> >> not quite sure how to read the documentation.
>> >>
>> >> Any example or explanation you give me for resolve_shared_entities is
>> >> much appreciated!
>> >>
>> >> Ryan
>> >>
>> >> On 10/25/18 09:40, Vijay S. Mahadevan wrote:
>> >>> Hi Ryan,
>> >>>
>> >>> Glad the example helped.
>> >>>
>> >>>> The example shows the ghost exchange happening between shared owned
>> >>> entities. In my experiments I've been unable to make the ghost exchange
>> >>> work, and I think that might be because my entities are not shared.
>> >>>
>> >>> You need to call resolve_shared_entities on the entities prior to
>> >>> doing ghost exchange. When we load the mesh from file, this happens
>> >>> automatically based on the read options but if you are forming a mesh
>> >>> in memory, you need to make sure that the shared vertices have the
>> >>> same GLOBAL_ID numbering consistently across processes. That is shared
>> >>> vertices are unique. Once that is set, shared entities resolution will
>> >>> work correctly out of the box and you will have shared edges/entities
>> >>> query working correctly. A call to get ghosted layers once this is
>> >>> done would be the way to go.
>> >>>
>> >>> I assume ESMF has a unique global numbering for the vertices ? Use
>> >>> that to set the GLOBAL_ID tag. Let us know if you are still having
>> >>> issues.
>> >>>
>> >>> Vijay
>> >>> On Thu, Oct 25, 2018 at 11:17 AM Ryan O'Kuinghttons
>> >>> <ryan.okuinghttons at noaa.gov> wrote:
>> >>>> Hi Vijay,
>> >>>>
>> >>>> I've had some time to play with this now, and I have another question.
>> >>>> Thank you for sending along the example code, it has been extremely
>> >>>> helpful.
>> >>>>
>> >>>> The example shows the ghost exchange happening between shared owned
>> >>>> entities. In my experiments I've been unable to make the ghost exchange
>> >>>> work, and I think that might be because my entities are not shared. The
>> >>>> situation I have is entities that are owned wholly on a single
>> >>>> processor, which need to be communicated to other processors which
>> >>>> require them as part of a halo region for mesh based computation. In
>> >>>> this situation would I need to "share" my entities across the whole
>> >>>> processor space, before requesting a ghost exchange? I'm not even really
>> >>>> sure what shared entities mean, is there a good place to look in the
>> >>>> documentation to learn more about the terminology? Thanks,
>> >>>>
>> >>>> Ryan
>> >>>>
>> >>>> On 10/8/18 12:17, Vijay S. Mahadevan wrote:
>> >>>>> Ryan,
>> >>>>>
>> >>>>> You need to use the ParallelComm object in MOAB to call
>> >>>>> exchange_ghost_cells [1] with appropriate levels of ghost layers for
>> >>>>> your mesh. This needs to be done after the mesh is loaded, or you can
>> >>>>> pass this information also as part of the options when loading a file
>> >>>>> and MOAB will internally load the file with ghosted layers. Here's an
>> >>>>> example [2].
>> >>>>>
>> >>>>> Vijay
>> >>>>>
>> >>>>> [1] http://ftp.mcs.anl.gov/pub/fathom/moab-docs-develop/classmoab_1_1ParallelComm.html#a55dfa308f56fd368319bfb4244428878
>> >>>>> [2] http://ftp.mcs.anl.gov/pub/fathom/moab-docs-develop/HelloParMOAB_8cpp-example.html
>> >>>>> On Mon, Oct 8, 2018 at 1:23 PM Ryan O'Kuinghttons
>> >>>>> <ryan.okuinghttons at noaa.gov> wrote:
>> >>>>>> Hi, I am wondering if there is a way to create ghost elements in MOAB.
>> >>>>>> By this I mean a list of copies of elements surrounding a specific MOAB
>> >>>>>> element, ghost elements may exist on a different processor than the
>> >>>>>> source element. I see a ghost_elems class in the appData namespace, but
>> >>>>>> there is not much documentation on how to use it.. Thank you,
>> >>>>>>
>> >>>>>> --
>> >>>>>> Ryan O'Kuinghttons
>> >>>>>> Cherokee Nation Management and Consulting
>> >>>>>> NESII/NOAA/Earth System Research Laboratory
>> >>>>>> ryan.okuinghttons at noaa.gov
>> >>>>>> https://www.esrl.noaa.gov/gsd/nesii/
>> >>>>>>