[MOAB-dev] ghost/halo elements

[Ryan O'Kuinghttons]

Hi Vijay,

Thanks again for that explanation.

ESMF does use unique global ids for the vertices and set them to the 
GLOBAL_ID in the moab mesh. So I think we are good there.

I can't quite figure out how to use resolve_shared_entities though. 
There are three versions of the call in the documentation, and I assume 
that I should use the first and pass in a Range containing all entities 
for proc_ents:

http://ftp.mcs.anl.gov/pub/fathom/moab-docs-develop/classmoab_1_1ParallelComm.html#a59e35d9906f2e33fe010138a144a5cb6

However, I'm not sure what the this_set EntityHandle should be.

This brings up a more general question I've had about the moab 
documentation for a while. In the doc for this routine, it only lists 2 
parameters, proc_ents and shared_dim, even though in the function 
signature above it clearly shows more. I've had trouble understanding 
which parameters are relevant in the past, or what they do because I'm 
not quite sure how to read the documentation.

Any example or explanation you give me for resolve_shared_entities is 
much appreciated!

Ryan

On 10/25/18 09:40, Vijay S. Mahadevan wrote:
> Hi Ryan,
>
> Glad the example helped.
>
>> The example shows the ghost exchange happening between shared owned
> entities. In my experiments I've been unable to make the ghost exchange
> work, and I think that might be because my entities are not shared.
>
> You need to call resolve_shared_entities on the entities prior to
> doing ghost exchange. When we load the mesh from file, this happens
> automatically based on the read options but if you are forming a mesh
> in memory, you need to make sure that the shared vertices have the
> same GLOBAL_ID numbering consistently across processes. That is shared
> vertices are unique. Once that is set, shared entities resolution will
> work correctly out of the box and you will have shared edges/entities
> query working correctly. A call to get ghosted layers once this is
> done would be the way to go.
>
> I assume ESMF has a unique global numbering for the vertices ? Use
> that to set the GLOBAL_ID tag. Let us know if you are still having
> issues.
>
> Vijay
> On Thu, Oct 25, 2018 at 11:17 AM Ryan O'Kuinghttons
> <ryan.okuinghttons at noaa.gov> wrote:
>> Hi Vijay,
>>
>> I've had some time to play with this now, and I have another question.
>> Thank you for sending along the example code, it has been extremely
>> helpful.
>>
>> The example shows the ghost exchange happening between shared owned
>> entities. In my experiments I've been unable to make the ghost exchange
>> work, and I think that might be because my entities are not shared. The
>> situation I have is entities that are owned wholly on a single
>> processor, which need to be communicated to other processors which
>> require them as part of a halo region for mesh based computation. In
>> this situation would I need to "share" my entities across the whole
>> processor space, before requesting a ghost exchange? I'm not even really
>> sure what shared entities mean, is there a good place to look in the
>> documentation to learn more about the terminology? Thanks,
>>
>> Ryan
>>
>> On 10/8/18 12:17, Vijay S. Mahadevan wrote:
>>> Ryan,
>>>
>>> You need to use the ParallelComm object in MOAB to call
>>> exchange_ghost_cells [1] with appropriate levels of ghost layers for
>>> your mesh. This needs to be done after the mesh is loaded, or you can
>>> pass this information also as part of the options when loading a file
>>> and MOAB will internally load the file with ghosted layers. Here's an
>>> example [2].
>>>
>>> Vijay
>>>
>>> [1] http://ftp.mcs.anl.gov/pub/fathom/moab-docs-develop/classmoab_1_1ParallelComm.html#a55dfa308f56fd368319bfb4244428878
>>> [2] http://ftp.mcs.anl.gov/pub/fathom/moab-docs-develop/HelloParMOAB_8cpp-example.html
>>> On Mon, Oct 8, 2018 at 1:23 PM Ryan O'Kuinghttons
>>> <ryan.okuinghttons at noaa.gov> wrote:
>>>> Hi, I am wondering if there is a way to create ghost elements in MOAB.
>>>> By this I mean a list of copies of elements surrounding a specific MOAB
>>>> element, ghost elements may exist on a different processor than the
>>>> source element. I see a ghost_elems class in the appData namespace, but
>>>> there is not much documentation on how to use it.. Thank you,
>>>>
>>>> --
>>>> Ryan O'Kuinghttons
>>>> Cherokee Nation Management and Consulting
>>>> NESII/NOAA/Earth System Research Laboratory
>>>> ryan.okuinghttons at noaa.gov
>>>> https://www.esrl.noaa.gov/gsd/nesii/
>>>>