[MOAB-dev] ghost/halo elements
Ryan O'Kuinghttons
ryan.okuinghttons at noaa.gov
Fri Oct 26 08:36:15 CDT 2018
Hi Vijay,
I resolved the issue with not getting any shared, owned entities back by
using the other version of resolve_shared_ents, as demonstrated in the
example you indicated. I created a Range containing all elements (faces
for moab), and used that as input.
Range range_ent;
merr=mb->get_entities_by_dimension(0,this->sdim,range_ent);
MBMESH_CHECK_ERR(merr, localrc);
merr = pcomm->resolve_shared_ents(0, range_ent, this->sdim, 1);
MBMESH_CHECK_ERR(merr, localrc);
The first version of the interface (which does not use a Range input)
does not work for me:
merr = pcomm->resolve_shared_ents(0, this->sdim, 1);
Otherwise, everything else seems to be resolved with this issue, thanks
again for all of your help!!
Ryan
On 10/25/18 15:13, Vijay S. Mahadevan wrote:
> Ryan, I don't see anything obviously wrong in the code below. Does
> your shared_ents not have any entities after parallel resolve ? If you
> can replicate your issue in a small standalone test, we should be able
> to quickly find out what's going wrong.
>
> Look at the test_reduce_tag_explicit_dest routine in
> test/parallel/parallel_unit_tests.cpp:1617. It has the workflow where
> you create a mesh in memory, resolve shared entities, get shared
> entities and even set/reduce tag data.
>
> Vijay
> On Thu, Oct 25, 2018 at 4:25 PM Ryan O'Kuinghttons
> <ryan.okuinghttons at noaa.gov> wrote:
>> OK, that allows the resolve_shared_ents routine to complete without
>> error, but I still don't get any shared, owned entities back from the
>> get_shared_entities/filter_pstatus calls:
>>
>> Range shared_ents;
>> // Get entities shared with all other processors
>> merr = pcomm->get_shared_entities(-1, shared_ents);
>> MBMESH_CHECK_ERR(merr, localrc);
>>
>> // Filter shared entities with not not_owned, which means owned
>> Range owned_entities;
>> merr = pcomm->filter_pstatus(shared_ents, PSTATUS_NOT_OWNED,
>> PSTATUS_NOT, -1, &owned_entities);
>> MBMESH_CHECK_ERR(merr, localrc);
>>
>> unsigned int nums[4] = {0}; // to store the owned entities per dimension
>> for (int i = 0; i < 4; i++)
>> nums[i] = (int)owned_entities.num_of_dimension(i);
>>
>> vector<int> rbuf(nprocs*4, 0);
>> MPI_Gather(nums, 4, MPI_INT, &rbuf[0], 4, MPI_INT, 0, mpi_comm);
>> // Print the stats gathered:
>> if (0 == rank) {
>> for (int i = 0; i < nprocs; i++)
>> cout << " Shared, owned entities on proc " << i << ": " <<
>> rbuf[4*i] << " verts, " <<
>> rbuf[4*i + 1] << " edges, " << rbuf[4*i + 2] << " faces, " <<
>> rbuf[4*i + 3] << " elements" << endl;
>> }
>>
>> OUTPUT:
>>
>> Shared, owned entities on proc 0: 0 verts, 0 edges, 0 faces, 0 elements
>> Shared, owned entities on proc 1: 0 verts, 0 edges, 0 faces, 0 elements
>> Shared, owned entities on proc 2: 0 verts, 0 edges, 0 faces, 0 elements
>> Shared, owned entities on proc 3: 0 verts, 0 edges, 0 faces, 0 elements
>>
>>
>> Note: I also tried changing the -1 to 2 in the
>> get_shared_entities/filter_pstatus calls to only retrieve elements.
>>
>>
>> On 10/25/18 13:57, Vijay S. Mahadevan wrote:
>>> No problem. Can you try 1 for the last argument to look for edges as
>>> the shared_dim.
>>>
>>> Vijay
>>> On Thu, Oct 25, 2018 at 3:41 PM Ryan O'Kuinghttons
>>> <ryan.okuinghttons at noaa.gov> wrote:
>>>> OK, thanks again for that explanation, I never would have been able to figure out that this_set was the root set. I apologize for all the dumb questions, hopefully they are all getting me closer to self sufficiency..
>>>>
>>>> I'm getting an error that doesn't make sense to me. I specifying the resolve_dim and shared_dim in the call:
>>>>
>>>> // Get the ParallelComm instance
>>>> ParallelComm* pcomm = new ParallelComm(mb, mpi_comm);
>>>> int nprocs = pcomm->proc_config().proc_size();
>>>> int rank = pcomm->proc_config().proc_rank();
>>>>
>>>> // get the root set handle
>>>> EntityHandle root_set = mb->get_root_set();
>>>>
>>>> merr = pcomm->resolve_shared_ents(root_set, 2, -1);
>>>>
>>>>
>>>> but I'm getting this message:
>>>>
>>>> [0]MOAB ERROR: Unable to guess shared_dim or resolve_dim!
>>>>
>>>>
>>>> what am i missing?
>>>>
>>>>
>>>> On 10/25/18 12:51, Vijay S. Mahadevan wrote:
>>>>
>>>> It is your root set or fileset if you added entities to it. You can get the root set handle from Core object.
>>>>
>>>> Vijay
>>>>
>>>> On Thu, Oct 25, 2018, 14:38 Ryan O'Kuinghttons <ryan.okuinghttons at noaa.gov> wrote:
>>>>> Thanks again, Vijay. However, I still don't understand what this_set
>>>>> should be, is it an output maybe?
>>>>>
>>>>> Ryan
>>>>>
>>>>> On 10/25/18 12:27, Vijay S. Mahadevan wrote:
>>>>>> You can try either the first or the second variant instead [1] with
>>>>>> following arguments.
>>>>>>
>>>>>> ErrorCode moab::ParallelComm::resolve_shared_ents(EntityHandle this_set,
>>>>>> int resolve_dim = 2,
>>>>>> int shared_dim = -1,
>>>>>> const Tag * id_tag = 0
>>>>>> )
>>>>>>
>>>>>> That should resolve 2-dim entities with shared edges across
>>>>>> partitions. You can leave id_tag pointer as zero since the default is
>>>>>> GLOBAL_ID.
>>>>>>
>>>>>>> This brings up a more general question I've had about the moab
>>>>>> documentation for a while. In the doc for this routine, it only lists 2
>>>>>> parameters, proc_ents and shared_dim, even though in the function
>>>>>> signature above it clearly shows more. I've had trouble understanding
>>>>>> which parameters are relevant in the past, or what they do because I'm
>>>>>> not quite sure how to read the documentation.
>>>>>>
>>>>>> This is an oversight. We will go through and rectify some of the
>>>>>> inconsistencies in the documentation. We are preparing for an upcoming
>>>>>> release and I'll make sure that routines in Core/ParallelComm have
>>>>>> updated documentation that match the interfaces. Meanwhile, if you
>>>>>> have questions, feel free to shoot an email to the list.
>>>>>>
>>>>>> Hope that helps.
>>>>>>
>>>>>> Vijay
>>>>>>
>>>>>> [1] http://ftp.mcs.anl.gov/pub/fathom/moab-docs-develop/classmoab_1_1ParallelComm.html#a29a3b834b3fc3b4ddb3a5d8a78a37c8a
>>>>>> On Thu, Oct 25, 2018 at 1:58 PM Ryan O'Kuinghttons
>>>>>> <ryan.okuinghttons at noaa.gov> wrote:
>>>>>>> Hi Vijay,
>>>>>>>
>>>>>>> Thanks again for that explanation.
>>>>>>>
>>>>>>> ESMF does use unique global ids for the vertices and set them to the
>>>>>>> GLOBAL_ID in the moab mesh. So I think we are good there.
>>>>>>>
>>>>>>> I can't quite figure out how to use resolve_shared_entities though.
>>>>>>> There are three versions of the call in the documentation, and I assume
>>>>>>> that I should use the first and pass in a Range containing all entities
>>>>>>> for proc_ents:
>>>>>>>
>>>>>>> http://ftp.mcs.anl.gov/pub/fathom/moab-docs-develop/classmoab_1_1ParallelComm.html#a59e35d9906f2e33fe010138a144a5cb6
>>>>>>>
>>>>>>> However, I'm not sure what the this_set EntityHandle should be.
>>>>>>>
>>>>>>> This brings up a more general question I've had about the moab
>>>>>>> documentation for a while. In the doc for this routine, it only lists 2
>>>>>>> parameters, proc_ents and shared_dim, even though in the function
>>>>>>> signature above it clearly shows more. I've had trouble understanding
>>>>>>> which parameters are relevant in the past, or what they do because I'm
>>>>>>> not quite sure how to read the documentation.
>>>>>>>
>>>>>>> Any example or explanation you give me for resolve_shared_entities is
>>>>>>> much appreciated!
>>>>>>>
>>>>>>> Ryan
>>>>>>>
>>>>>>> On 10/25/18 09:40, Vijay S. Mahadevan wrote:
>>>>>>>> Hi Ryan,
>>>>>>>>
>>>>>>>> Glad the example helped.
>>>>>>>>
>>>>>>>>> The example shows the ghost exchange happening between shared owned
>>>>>>>> entities. In my experiments I've been unable to make the ghost exchange
>>>>>>>> work, and I think that might be because my entities are not shared.
>>>>>>>>
>>>>>>>> You need to call resolve_shared_entities on the entities prior to
>>>>>>>> doing ghost exchange. When we load the mesh from file, this happens
>>>>>>>> automatically based on the read options but if you are forming a mesh
>>>>>>>> in memory, you need to make sure that the shared vertices have the
>>>>>>>> same GLOBAL_ID numbering consistently across processes. That is shared
>>>>>>>> vertices are unique. Once that is set, shared entities resolution will
>>>>>>>> work correctly out of the box and you will have shared edges/entities
>>>>>>>> query working correctly. A call to get ghosted layers once this is
>>>>>>>> done would be the way to go.
>>>>>>>>
>>>>>>>> I assume ESMF has a unique global numbering for the vertices ? Use
>>>>>>>> that to set the GLOBAL_ID tag. Let us know if you are still having
>>>>>>>> issues.
>>>>>>>>
>>>>>>>> Vijay
>>>>>>>> On Thu, Oct 25, 2018 at 11:17 AM Ryan O'Kuinghttons
>>>>>>>> <ryan.okuinghttons at noaa.gov> wrote:
>>>>>>>>> Hi Vijay,
>>>>>>>>>
>>>>>>>>> I've had some time to play with this now, and I have another question.
>>>>>>>>> Thank you for sending along the example code, it has been extremely
>>>>>>>>> helpful.
>>>>>>>>>
>>>>>>>>> The example shows the ghost exchange happening between shared owned
>>>>>>>>> entities. In my experiments I've been unable to make the ghost exchange
>>>>>>>>> work, and I think that might be because my entities are not shared. The
>>>>>>>>> situation I have is entities that are owned wholly on a single
>>>>>>>>> processor, which need to be communicated to other processors which
>>>>>>>>> require them as part of a halo region for mesh based computation. In
>>>>>>>>> this situation would I need to "share" my entities across the whole
>>>>>>>>> processor space, before requesting a ghost exchange? I'm not even really
>>>>>>>>> sure what shared entities mean, is there a good place to look in the
>>>>>>>>> documentation to learn more about the terminology? Thanks,
>>>>>>>>>
>>>>>>>>> Ryan
>>>>>>>>>
>>>>>>>>> On 10/8/18 12:17, Vijay S. Mahadevan wrote:
>>>>>>>>>> Ryan,
>>>>>>>>>>
>>>>>>>>>> You need to use the ParallelComm object in MOAB to call
>>>>>>>>>> exchange_ghost_cells [1] with appropriate levels of ghost layers for
>>>>>>>>>> your mesh. This needs to be done after the mesh is loaded, or you can
>>>>>>>>>> pass this information also as part of the options when loading a file
>>>>>>>>>> and MOAB will internally load the file with ghosted layers. Here's an
>>>>>>>>>> example [2].
>>>>>>>>>>
>>>>>>>>>> Vijay
>>>>>>>>>>
>>>>>>>>>> [1] http://ftp.mcs.anl.gov/pub/fathom/moab-docs-develop/classmoab_1_1ParallelComm.html#a55dfa308f56fd368319bfb4244428878
>>>>>>>>>> [2] http://ftp.mcs.anl.gov/pub/fathom/moab-docs-develop/HelloParMOAB_8cpp-example.html
>>>>>>>>>> On Mon, Oct 8, 2018 at 1:23 PM Ryan O'Kuinghttons
>>>>>>>>>> <ryan.okuinghttons at noaa.gov> wrote:
>>>>>>>>>>> Hi, I am wondering if there is a way to create ghost elements in MOAB.
>>>>>>>>>>> By this I mean a list of copies of elements surrounding a specific MOAB
>>>>>>>>>>> element, ghost elements may exist on a different processor than the
>>>>>>>>>>> source element. I see a ghost_elems class in the appData namespace, but
>>>>>>>>>>> there is not much documentation on how to use it.. Thank you,
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Ryan O'Kuinghttons
>>>>>>>>>>> Cherokee Nation Management and Consulting
>>>>>>>>>>> NESII/NOAA/Earth System Research Laboratory
>>>>>>>>>>> ryan.okuinghttons at noaa.gov
>>>>>>>>>>> https://www.esrl.noaa.gov/gsd/nesii/
>>>>>>>>>>>
More information about the moab-dev
mailing list