[MOAB-dev] ghost/halo elements
Ryan O'Kuinghttons
ryan.okuinghttons at noaa.gov
Mon Oct 29 12:49:03 CDT 2018
Hi Iulian, Vijay,
We have a set of custom tags that we use in ESMF, would it be possible
to transfer these as well?
Ryan
On 10/29/18 10:54, Vijay S. Mahadevan wrote:
> The default set of tags do get transferred. These would be the
> MATERIAL_SET, DIRICHLET/NEUMANN sets etc.
>
> Iulian, did we add a separate method to transfer these to the ghosted
> entities ? I vaguely remember something like that.
>
> Vijay
> On Mon, Oct 29, 2018 at 12:42 PM Ryan O'Kuinghttons
> <ryan.okuinghttons at noaa.gov> wrote:
>> Hi Vijay,
>>
>> I ran into another issue with the call to exchange_ghost_cells, but this
>> is probably just a misunderstanding on my part. I find that the tag
>> information that is part of the mesh before the the call does not
>> transfer to the newly created faces. Is there some way to ask moab to
>> transfer tags when creating ghost entities? (FWIW, I could not find this
>> in the example you sent me) Thanks,
>>
>> Ryan
>>
>> On 10/26/18 07:36, Ryan O'Kuinghttons wrote:
>>> Hi Vijay,
>>>
>>> I resolved the issue with not getting any shared, owned entities back
>>> by using the other version of resolve_shared_ents, as demonstrated in
>>> the example you indicated. I created a Range containing all elements
>>> (faces for moab), and used that as input.
>>>
>>> Range range_ent;
>>> merr=mb->get_entities_by_dimension(0,this->sdim,range_ent);
>>> MBMESH_CHECK_ERR(merr, localrc);
>>>
>>> merr = pcomm->resolve_shared_ents(0, range_ent, this->sdim, 1);
>>> MBMESH_CHECK_ERR(merr, localrc);
>>>
>>> The first version of the interface (which does not use a Range input)
>>> does not work for me:
>>>
>>> merr = pcomm->resolve_shared_ents(0, this->sdim, 1);
>>>
>>> Otherwise, everything else seems to be resolved with this issue,
>>> thanks again for all of your help!!
>>>
>>> Ryan
>>>
>>> On 10/25/18 15:13, Vijay S. Mahadevan wrote:
>>>> Ryan, I don't see anything obviously wrong in the code below. Does
>>>> your shared_ents not have any entities after parallel resolve ? If you
>>>> can replicate your issue in a small standalone test, we should be able
>>>> to quickly find out what's going wrong.
>>>>
>>>> Look at the test_reduce_tag_explicit_dest routine in
>>>> test/parallel/parallel_unit_tests.cpp:1617. It has the workflow where
>>>> you create a mesh in memory, resolve shared entities, get shared
>>>> entities and even set/reduce tag data.
>>>>
>>>> Vijay
>>>> On Thu, Oct 25, 2018 at 4:25 PM Ryan O'Kuinghttons
>>>> <ryan.okuinghttons at noaa.gov> wrote:
>>>>> OK, that allows the resolve_shared_ents routine to complete without
>>>>> error, but I still don't get any shared, owned entities back from the
>>>>> get_shared_entities/filter_pstatus calls:
>>>>>
>>>>> Range shared_ents;
>>>>> // Get entities shared with all other processors
>>>>> merr = pcomm->get_shared_entities(-1, shared_ents);
>>>>> MBMESH_CHECK_ERR(merr, localrc);
>>>>>
>>>>> // Filter shared entities with not not_owned, which means owned
>>>>> Range owned_entities;
>>>>> merr = pcomm->filter_pstatus(shared_ents, PSTATUS_NOT_OWNED,
>>>>> PSTATUS_NOT, -1, &owned_entities);
>>>>> MBMESH_CHECK_ERR(merr, localrc);
>>>>>
>>>>> unsigned int nums[4] = {0}; // to store the owned entities per
>>>>> dimension
>>>>> for (int i = 0; i < 4; i++)
>>>>> nums[i] = (int)owned_entities.num_of_dimension(i);
>>>>>
>>>>> vector<int> rbuf(nprocs*4, 0);
>>>>> MPI_Gather(nums, 4, MPI_INT, &rbuf[0], 4, MPI_INT, 0, mpi_comm);
>>>>> // Print the stats gathered:
>>>>> if (0 == rank) {
>>>>> for (int i = 0; i < nprocs; i++)
>>>>> cout << " Shared, owned entities on proc " << i << ": " <<
>>>>> rbuf[4*i] << " verts, " <<
>>>>> rbuf[4*i + 1] << " edges, " << rbuf[4*i + 2] << " faces,
>>>>> " <<
>>>>> rbuf[4*i + 3] << " elements" << endl;
>>>>> }
>>>>>
>>>>> OUTPUT:
>>>>>
>>>>> Shared, owned entities on proc 0: 0 verts, 0 edges, 0 faces, 0
>>>>> elements
>>>>> Shared, owned entities on proc 1: 0 verts, 0 edges, 0 faces, 0
>>>>> elements
>>>>> Shared, owned entities on proc 2: 0 verts, 0 edges, 0 faces, 0
>>>>> elements
>>>>> Shared, owned entities on proc 3: 0 verts, 0 edges, 0 faces, 0
>>>>> elements
>>>>>
>>>>>
>>>>> Note: I also tried changing the -1 to 2 in the
>>>>> get_shared_entities/filter_pstatus calls to only retrieve elements.
>>>>>
>>>>>
>>>>> On 10/25/18 13:57, Vijay S. Mahadevan wrote:
>>>>>> No problem. Can you try 1 for the last argument to look for edges as
>>>>>> the shared_dim.
>>>>>>
>>>>>> Vijay
>>>>>> On Thu, Oct 25, 2018 at 3:41 PM Ryan O'Kuinghttons
>>>>>> <ryan.okuinghttons at noaa.gov> wrote:
>>>>>>> OK, thanks again for that explanation, I never would have been
>>>>>>> able to figure out that this_set was the root set. I apologize for
>>>>>>> all the dumb questions, hopefully they are all getting me closer
>>>>>>> to self sufficiency..
>>>>>>>
>>>>>>> I'm getting an error that doesn't make sense to me. I specifying
>>>>>>> the resolve_dim and shared_dim in the call:
>>>>>>>
>>>>>>> // Get the ParallelComm instance
>>>>>>> ParallelComm* pcomm = new ParallelComm(mb, mpi_comm);
>>>>>>> int nprocs = pcomm->proc_config().proc_size();
>>>>>>> int rank = pcomm->proc_config().proc_rank();
>>>>>>>
>>>>>>> // get the root set handle
>>>>>>> EntityHandle root_set = mb->get_root_set();
>>>>>>>
>>>>>>> merr = pcomm->resolve_shared_ents(root_set, 2, -1);
>>>>>>>
>>>>>>>
>>>>>>> but I'm getting this message:
>>>>>>>
>>>>>>> [0]MOAB ERROR: Unable to guess shared_dim or resolve_dim!
>>>>>>>
>>>>>>>
>>>>>>> what am i missing?
>>>>>>>
>>>>>>>
>>>>>>> On 10/25/18 12:51, Vijay S. Mahadevan wrote:
>>>>>>>
>>>>>>> It is your root set or fileset if you added entities to it. You
>>>>>>> can get the root set handle from Core object.
>>>>>>>
>>>>>>> Vijay
>>>>>>>
>>>>>>> On Thu, Oct 25, 2018, 14:38 Ryan O'Kuinghttons
>>>>>>> <ryan.okuinghttons at noaa.gov> wrote:
>>>>>>>> Thanks again, Vijay. However, I still don't understand what this_set
>>>>>>>> should be, is it an output maybe?
>>>>>>>>
>>>>>>>> Ryan
>>>>>>>>
>>>>>>>> On 10/25/18 12:27, Vijay S. Mahadevan wrote:
>>>>>>>>> You can try either the first or the second variant instead [1] with
>>>>>>>>> following arguments.
>>>>>>>>>
>>>>>>>>> ErrorCode moab::ParallelComm::resolve_shared_ents(EntityHandle
>>>>>>>>> this_set,
>>>>>>>>> int resolve_dim = 2,
>>>>>>>>> int shared_dim = -1,
>>>>>>>>> const Tag * id_tag = 0
>>>>>>>>> )
>>>>>>>>>
>>>>>>>>> That should resolve 2-dim entities with shared edges across
>>>>>>>>> partitions. You can leave id_tag pointer as zero since the
>>>>>>>>> default is
>>>>>>>>> GLOBAL_ID.
>>>>>>>>>
>>>>>>>>>> This brings up a more general question I've had about the moab
>>>>>>>>> documentation for a while. In the doc for this routine, it only
>>>>>>>>> lists 2
>>>>>>>>> parameters, proc_ents and shared_dim, even though in the function
>>>>>>>>> signature above it clearly shows more. I've had trouble
>>>>>>>>> understanding
>>>>>>>>> which parameters are relevant in the past, or what they do
>>>>>>>>> because I'm
>>>>>>>>> not quite sure how to read the documentation.
>>>>>>>>>
>>>>>>>>> This is an oversight. We will go through and rectify some of the
>>>>>>>>> inconsistencies in the documentation. We are preparing for an
>>>>>>>>> upcoming
>>>>>>>>> release and I'll make sure that routines in Core/ParallelComm have
>>>>>>>>> updated documentation that match the interfaces. Meanwhile, if you
>>>>>>>>> have questions, feel free to shoot an email to the list.
>>>>>>>>>
>>>>>>>>> Hope that helps.
>>>>>>>>>
>>>>>>>>> Vijay
>>>>>>>>>
>>>>>>>>> [1]
>>>>>>>>> http://ftp.mcs.anl.gov/pub/fathom/moab-docs-develop/classmoab_1_1ParallelComm.html#a29a3b834b3fc3b4ddb3a5d8a78a37c8a
>>>>>>>>> On Thu, Oct 25, 2018 at 1:58 PM Ryan O'Kuinghttons
>>>>>>>>> <ryan.okuinghttons at noaa.gov> wrote:
>>>>>>>>>> Hi Vijay,
>>>>>>>>>>
>>>>>>>>>> Thanks again for that explanation.
>>>>>>>>>>
>>>>>>>>>> ESMF does use unique global ids for the vertices and set them
>>>>>>>>>> to the
>>>>>>>>>> GLOBAL_ID in the moab mesh. So I think we are good there.
>>>>>>>>>>
>>>>>>>>>> I can't quite figure out how to use resolve_shared_entities
>>>>>>>>>> though.
>>>>>>>>>> There are three versions of the call in the documentation, and
>>>>>>>>>> I assume
>>>>>>>>>> that I should use the first and pass in a Range containing all
>>>>>>>>>> entities
>>>>>>>>>> for proc_ents:
>>>>>>>>>>
>>>>>>>>>> http://ftp.mcs.anl.gov/pub/fathom/moab-docs-develop/classmoab_1_1ParallelComm.html#a59e35d9906f2e33fe010138a144a5cb6
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> However, I'm not sure what the this_set EntityHandle should be.
>>>>>>>>>>
>>>>>>>>>> This brings up a more general question I've had about the moab
>>>>>>>>>> documentation for a while. In the doc for this routine, it only
>>>>>>>>>> lists 2
>>>>>>>>>> parameters, proc_ents and shared_dim, even though in the function
>>>>>>>>>> signature above it clearly shows more. I've had trouble
>>>>>>>>>> understanding
>>>>>>>>>> which parameters are relevant in the past, or what they do
>>>>>>>>>> because I'm
>>>>>>>>>> not quite sure how to read the documentation.
>>>>>>>>>>
>>>>>>>>>> Any example or explanation you give me for
>>>>>>>>>> resolve_shared_entities is
>>>>>>>>>> much appreciated!
>>>>>>>>>>
>>>>>>>>>> Ryan
>>>>>>>>>>
>>>>>>>>>> On 10/25/18 09:40, Vijay S. Mahadevan wrote:
>>>>>>>>>>> Hi Ryan,
>>>>>>>>>>>
>>>>>>>>>>> Glad the example helped.
>>>>>>>>>>>
>>>>>>>>>>>> The example shows the ghost exchange happening between shared
>>>>>>>>>>>> owned
>>>>>>>>>>> entities. In my experiments I've been unable to make the ghost
>>>>>>>>>>> exchange
>>>>>>>>>>> work, and I think that might be because my entities are not
>>>>>>>>>>> shared.
>>>>>>>>>>>
>>>>>>>>>>> You need to call resolve_shared_entities on the entities prior to
>>>>>>>>>>> doing ghost exchange. When we load the mesh from file, this
>>>>>>>>>>> happens
>>>>>>>>>>> automatically based on the read options but if you are forming
>>>>>>>>>>> a mesh
>>>>>>>>>>> in memory, you need to make sure that the shared vertices have
>>>>>>>>>>> the
>>>>>>>>>>> same GLOBAL_ID numbering consistently across processes. That
>>>>>>>>>>> is shared
>>>>>>>>>>> vertices are unique. Once that is set, shared entities
>>>>>>>>>>> resolution will
>>>>>>>>>>> work correctly out of the box and you will have shared
>>>>>>>>>>> edges/entities
>>>>>>>>>>> query working correctly. A call to get ghosted layers once
>>>>>>>>>>> this is
>>>>>>>>>>> done would be the way to go.
>>>>>>>>>>>
>>>>>>>>>>> I assume ESMF has a unique global numbering for the vertices ?
>>>>>>>>>>> Use
>>>>>>>>>>> that to set the GLOBAL_ID tag. Let us know if you are still
>>>>>>>>>>> having
>>>>>>>>>>> issues.
>>>>>>>>>>>
>>>>>>>>>>> Vijay
>>>>>>>>>>> On Thu, Oct 25, 2018 at 11:17 AM Ryan O'Kuinghttons
>>>>>>>>>>> <ryan.okuinghttons at noaa.gov> wrote:
>>>>>>>>>>>> Hi Vijay,
>>>>>>>>>>>>
>>>>>>>>>>>> I've had some time to play with this now, and I have another
>>>>>>>>>>>> question.
>>>>>>>>>>>> Thank you for sending along the example code, it has been
>>>>>>>>>>>> extremely
>>>>>>>>>>>> helpful.
>>>>>>>>>>>>
>>>>>>>>>>>> The example shows the ghost exchange happening between shared
>>>>>>>>>>>> owned
>>>>>>>>>>>> entities. In my experiments I've been unable to make the
>>>>>>>>>>>> ghost exchange
>>>>>>>>>>>> work, and I think that might be because my entities are not
>>>>>>>>>>>> shared. The
>>>>>>>>>>>> situation I have is entities that are owned wholly on a single
>>>>>>>>>>>> processor, which need to be communicated to other processors
>>>>>>>>>>>> which
>>>>>>>>>>>> require them as part of a halo region for mesh based
>>>>>>>>>>>> computation. In
>>>>>>>>>>>> this situation would I need to "share" my entities across the
>>>>>>>>>>>> whole
>>>>>>>>>>>> processor space, before requesting a ghost exchange? I'm not
>>>>>>>>>>>> even really
>>>>>>>>>>>> sure what shared entities mean, is there a good place to look
>>>>>>>>>>>> in the
>>>>>>>>>>>> documentation to learn more about the terminology? Thanks,
>>>>>>>>>>>>
>>>>>>>>>>>> Ryan
>>>>>>>>>>>>
>>>>>>>>>>>> On 10/8/18 12:17, Vijay S. Mahadevan wrote:
>>>>>>>>>>>>> Ryan,
>>>>>>>>>>>>>
>>>>>>>>>>>>> You need to use the ParallelComm object in MOAB to call
>>>>>>>>>>>>> exchange_ghost_cells [1] with appropriate levels of ghost
>>>>>>>>>>>>> layers for
>>>>>>>>>>>>> your mesh. This needs to be done after the mesh is loaded,
>>>>>>>>>>>>> or you can
>>>>>>>>>>>>> pass this information also as part of the options when
>>>>>>>>>>>>> loading a file
>>>>>>>>>>>>> and MOAB will internally load the file with ghosted layers.
>>>>>>>>>>>>> Here's an
>>>>>>>>>>>>> example [2].
>>>>>>>>>>>>>
>>>>>>>>>>>>> Vijay
>>>>>>>>>>>>>
>>>>>>>>>>>>> [1]
>>>>>>>>>>>>> http://ftp.mcs.anl.gov/pub/fathom/moab-docs-develop/classmoab_1_1ParallelComm.html#a55dfa308f56fd368319bfb4244428878
>>>>>>>>>>>>> [2]
>>>>>>>>>>>>> http://ftp.mcs.anl.gov/pub/fathom/moab-docs-develop/HelloParMOAB_8cpp-example.html
>>>>>>>>>>>>> On Mon, Oct 8, 2018 at 1:23 PM Ryan O'Kuinghttons
>>>>>>>>>>>>> <ryan.okuinghttons at noaa.gov> wrote:
>>>>>>>>>>>>>> Hi, I am wondering if there is a way to create ghost
>>>>>>>>>>>>>> elements in MOAB.
>>>>>>>>>>>>>> By this I mean a list of copies of elements surrounding a
>>>>>>>>>>>>>> specific MOAB
>>>>>>>>>>>>>> element, ghost elements may exist on a different processor
>>>>>>>>>>>>>> than the
>>>>>>>>>>>>>> source element. I see a ghost_elems class in the appData
>>>>>>>>>>>>>> namespace, but
>>>>>>>>>>>>>> there is not much documentation on how to use it.. Thank you,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Ryan O'Kuinghttons
>>>>>>>>>>>>>> Cherokee Nation Management and Consulting
>>>>>>>>>>>>>> NESII/NOAA/Earth System Research Laboratory
>>>>>>>>>>>>>> ryan.okuinghttons at noaa.gov
>>>>>>>>>>>>>> https://www.esrl.noaa.gov/gsd/nesii/
>>>>>>>>>>>>>>
More information about the moab-dev
mailing list