[MOAB-dev] ghost/halo elements

Thu Oct 25 13:37:58 CDT 2018

Thanks again, Vijay. However, I still don't understand what this_set 
should be, is it an output maybe?

Ryan

On 10/25/18 12:27, Vijay S. Mahadevan wrote:
> You can try either the first or the second variant instead [1] with
> following arguments.
>
> ErrorCode moab::ParallelComm::resolve_shared_ents(EntityHandle this_set,
> int resolve_dim = 2,
> int shared_dim = -1,
> const Tag * id_tag = 0
> )
>
> That should resolve 2-dim entities with shared edges across
> partitions. You can leave id_tag pointer as zero since the default is
> GLOBAL_ID.
>
>> This brings up a more general question I've had about the moab
> documentation for a while. In the doc for this routine, it only lists 2
> parameters, proc_ents and shared_dim, even though in the function
> signature above it clearly shows more. I've had trouble understanding
> which parameters are relevant in the past, or what they do because I'm
> not quite sure how to read the documentation.
>
> This is an oversight. We will go through and rectify some of the
> inconsistencies in the documentation. We are preparing for an upcoming
> release and I'll make sure that routines in Core/ParallelComm have
> updated documentation that match the interfaces. Meanwhile, if you
> have questions, feel free to shoot an email to the list.
>
> Hope that helps.
>
> Vijay
>
> [1] http://ftp.mcs.anl.gov/pub/fathom/moab-docs-develop/classmoab_1_1ParallelComm.html#a29a3b834b3fc3b4ddb3a5d8a78a37c8a
> On Thu, Oct 25, 2018 at 1:58 PM Ryan O'Kuinghttons
> <ryan.okuinghttons at noaa.gov> wrote:
>> Hi Vijay,
>>
>> Thanks again for that explanation.
>>
>> ESMF does use unique global ids for the vertices and set them to the
>> GLOBAL_ID in the moab mesh. So I think we are good there.
>>
>> I can't quite figure out how to use resolve_shared_entities though.
>> There are three versions of the call in the documentation, and I assume
>> that I should use the first and pass in a Range containing all entities
>> for proc_ents:
>>
>> http://ftp.mcs.anl.gov/pub/fathom/moab-docs-develop/classmoab_1_1ParallelComm.html#a59e35d9906f2e33fe010138a144a5cb6
>>
>> However, I'm not sure what the this_set EntityHandle should be.
>>
>> This brings up a more general question I've had about the moab
>> documentation for a while. In the doc for this routine, it only lists 2
>> parameters, proc_ents and shared_dim, even though in the function
>> signature above it clearly shows more. I've had trouble understanding
>> which parameters are relevant in the past, or what they do because I'm
>> not quite sure how to read the documentation.
>>
>> Any example or explanation you give me for resolve_shared_entities is
>> much appreciated!
>>
>> Ryan
>>
>> On 10/25/18 09:40, Vijay S. Mahadevan wrote:
>>> Hi Ryan,
>>>
>>> Glad the example helped.
>>>
>>>> The example shows the ghost exchange happening between shared owned
>>> entities. In my experiments I've been unable to make the ghost exchange
>>> work, and I think that might be because my entities are not shared.
>>>
>>> You need to call resolve_shared_entities on the entities prior to
>>> doing ghost exchange. When we load the mesh from file, this happens
>>> automatically based on the read options but if you are forming a mesh
>>> in memory, you need to make sure that the shared vertices have the
>>> same GLOBAL_ID numbering consistently across processes. That is shared
>>> vertices are unique. Once that is set, shared entities resolution will
>>> work correctly out of the box and you will have shared edges/entities
>>> query working correctly. A call to get ghosted layers once this is
>>> done would be the way to go.
>>>
>>> I assume ESMF has a unique global numbering for the vertices ? Use
>>> that to set the GLOBAL_ID tag. Let us know if you are still having
>>> issues.
>>>
>>> Vijay
>>> On Thu, Oct 25, 2018 at 11:17 AM Ryan O'Kuinghttons
>>> <ryan.okuinghttons at noaa.gov> wrote:
>>>> Hi Vijay,
>>>>
>>>> I've had some time to play with this now, and I have another question.
>>>> Thank you for sending along the example code, it has been extremely
>>>> helpful.
>>>>
>>>> The example shows the ghost exchange happening between shared owned
>>>> entities. In my experiments I've been unable to make the ghost exchange
>>>> work, and I think that might be because my entities are not shared. The
>>>> situation I have is entities that are owned wholly on a single
>>>> processor, which need to be communicated to other processors which
>>>> require them as part of a halo region for mesh based computation. In
>>>> this situation would I need to "share" my entities across the whole
>>>> processor space, before requesting a ghost exchange? I'm not even really
>>>> sure what shared entities mean, is there a good place to look in the
>>>> documentation to learn more about the terminology? Thanks,
>>>>
>>>> Ryan
>>>>
>>>> On 10/8/18 12:17, Vijay S. Mahadevan wrote:
>>>>> Ryan,
>>>>>
>>>>> You need to use the ParallelComm object in MOAB to call
>>>>> exchange_ghost_cells [1] with appropriate levels of ghost layers for
>>>>> your mesh. This needs to be done after the mesh is loaded, or you can
>>>>> pass this information also as part of the options when loading a file
>>>>> and MOAB will internally load the file with ghosted layers. Here's an
>>>>> example [2].
>>>>>
>>>>> Vijay
>>>>>
>>>>> [1] http://ftp.mcs.anl.gov/pub/fathom/moab-docs-develop/classmoab_1_1ParallelComm.html#a55dfa308f56fd368319bfb4244428878
>>>>> [2] http://ftp.mcs.anl.gov/pub/fathom/moab-docs-develop/HelloParMOAB_8cpp-example.html
>>>>> On Mon, Oct 8, 2018 at 1:23 PM Ryan O'Kuinghttons
>>>>> <ryan.okuinghttons at noaa.gov> wrote:
>>>>>> Hi, I am wondering if there is a way to create ghost elements in MOAB.
>>>>>> By this I mean a list of copies of elements surrounding a specific MOAB
>>>>>> element, ghost elements may exist on a different processor than the
>>>>>> source element. I see a ghost_elems class in the appData namespace, but
>>>>>> there is not much documentation on how to use it.. Thank you,
>>>>>>
>>>>>> --
>>>>>> Ryan O'Kuinghttons
>>>>>> Cherokee Nation Management and Consulting
>>>>>> NESII/NOAA/Earth System Research Laboratory
>>>>>> ryan.okuinghttons at noaa.gov
>>>>>> https://www.esrl.noaa.gov/gsd/nesii/
>>>>>>