[MOAB-dev] ghost/halo elements
Ryan O'Kuinghttons
ryan.okuinghttons at noaa.gov
Thu Oct 25 15:25:05 CDT 2018
OK, that allows the resolve_shared_ents routine to complete without
error, but I still don't get any shared, owned entities back from the
get_shared_entities/filter_pstatus calls:
Range shared_ents;
// Get entities shared with all other processors
merr = pcomm->get_shared_entities(-1, shared_ents);
MBMESH_CHECK_ERR(merr, localrc);
// Filter shared entities with not not_owned, which means owned
Range owned_entities;
merr = pcomm->filter_pstatus(shared_ents, PSTATUS_NOT_OWNED,
PSTATUS_NOT, -1, &owned_entities);
MBMESH_CHECK_ERR(merr, localrc);
unsigned int nums[4] = {0}; // to store the owned entities per dimension
for (int i = 0; i < 4; i++)
nums[i] = (int)owned_entities.num_of_dimension(i);
vector<int> rbuf(nprocs*4, 0);
MPI_Gather(nums, 4, MPI_INT, &rbuf[0], 4, MPI_INT, 0, mpi_comm);
// Print the stats gathered:
if (0 == rank) {
for (int i = 0; i < nprocs; i++)
cout << " Shared, owned entities on proc " << i << ": " <<
rbuf[4*i] << " verts, " <<
rbuf[4*i + 1] << " edges, " << rbuf[4*i + 2] << " faces, " <<
rbuf[4*i + 3] << " elements" << endl;
}
OUTPUT:
Shared, owned entities on proc 0: 0 verts, 0 edges, 0 faces, 0 elements
Shared, owned entities on proc 1: 0 verts, 0 edges, 0 faces, 0 elements
Shared, owned entities on proc 2: 0 verts, 0 edges, 0 faces, 0 elements
Shared, owned entities on proc 3: 0 verts, 0 edges, 0 faces, 0 elements
Note: I also tried changing the -1 to 2 in the
get_shared_entities/filter_pstatus calls to only retrieve elements.
On 10/25/18 13:57, Vijay S. Mahadevan wrote:
> No problem. Can you try 1 for the last argument to look for edges as
> the shared_dim.
>
> Vijay
> On Thu, Oct 25, 2018 at 3:41 PM Ryan O'Kuinghttons
> <ryan.okuinghttons at noaa.gov> wrote:
>> OK, thanks again for that explanation, I never would have been able to figure out that this_set was the root set. I apologize for all the dumb questions, hopefully they are all getting me closer to self sufficiency..
>>
>> I'm getting an error that doesn't make sense to me. I specifying the resolve_dim and shared_dim in the call:
>>
>> // Get the ParallelComm instance
>> ParallelComm* pcomm = new ParallelComm(mb, mpi_comm);
>> int nprocs = pcomm->proc_config().proc_size();
>> int rank = pcomm->proc_config().proc_rank();
>>
>> // get the root set handle
>> EntityHandle root_set = mb->get_root_set();
>>
>> merr = pcomm->resolve_shared_ents(root_set, 2, -1);
>>
>>
>> but I'm getting this message:
>>
>> [0]MOAB ERROR: Unable to guess shared_dim or resolve_dim!
>>
>>
>> what am i missing?
>>
>>
>> On 10/25/18 12:51, Vijay S. Mahadevan wrote:
>>
>> It is your root set or fileset if you added entities to it. You can get the root set handle from Core object.
>>
>> Vijay
>>
>> On Thu, Oct 25, 2018, 14:38 Ryan O'Kuinghttons <ryan.okuinghttons at noaa.gov> wrote:
>>> Thanks again, Vijay. However, I still don't understand what this_set
>>> should be, is it an output maybe?
>>>
>>> Ryan
>>>
>>> On 10/25/18 12:27, Vijay S. Mahadevan wrote:
>>>> You can try either the first or the second variant instead [1] with
>>>> following arguments.
>>>>
>>>> ErrorCode moab::ParallelComm::resolve_shared_ents(EntityHandle this_set,
>>>> int resolve_dim = 2,
>>>> int shared_dim = -1,
>>>> const Tag * id_tag = 0
>>>> )
>>>>
>>>> That should resolve 2-dim entities with shared edges across
>>>> partitions. You can leave id_tag pointer as zero since the default is
>>>> GLOBAL_ID.
>>>>
>>>>> This brings up a more general question I've had about the moab
>>>> documentation for a while. In the doc for this routine, it only lists 2
>>>> parameters, proc_ents and shared_dim, even though in the function
>>>> signature above it clearly shows more. I've had trouble understanding
>>>> which parameters are relevant in the past, or what they do because I'm
>>>> not quite sure how to read the documentation.
>>>>
>>>> This is an oversight. We will go through and rectify some of the
>>>> inconsistencies in the documentation. We are preparing for an upcoming
>>>> release and I'll make sure that routines in Core/ParallelComm have
>>>> updated documentation that match the interfaces. Meanwhile, if you
>>>> have questions, feel free to shoot an email to the list.
>>>>
>>>> Hope that helps.
>>>>
>>>> Vijay
>>>>
>>>> [1] http://ftp.mcs.anl.gov/pub/fathom/moab-docs-develop/classmoab_1_1ParallelComm.html#a29a3b834b3fc3b4ddb3a5d8a78a37c8a
>>>> On Thu, Oct 25, 2018 at 1:58 PM Ryan O'Kuinghttons
>>>> <ryan.okuinghttons at noaa.gov> wrote:
>>>>> Hi Vijay,
>>>>>
>>>>> Thanks again for that explanation.
>>>>>
>>>>> ESMF does use unique global ids for the vertices and set them to the
>>>>> GLOBAL_ID in the moab mesh. So I think we are good there.
>>>>>
>>>>> I can't quite figure out how to use resolve_shared_entities though.
>>>>> There are three versions of the call in the documentation, and I assume
>>>>> that I should use the first and pass in a Range containing all entities
>>>>> for proc_ents:
>>>>>
>>>>> http://ftp.mcs.anl.gov/pub/fathom/moab-docs-develop/classmoab_1_1ParallelComm.html#a59e35d9906f2e33fe010138a144a5cb6
>>>>>
>>>>> However, I'm not sure what the this_set EntityHandle should be.
>>>>>
>>>>> This brings up a more general question I've had about the moab
>>>>> documentation for a while. In the doc for this routine, it only lists 2
>>>>> parameters, proc_ents and shared_dim, even though in the function
>>>>> signature above it clearly shows more. I've had trouble understanding
>>>>> which parameters are relevant in the past, or what they do because I'm
>>>>> not quite sure how to read the documentation.
>>>>>
>>>>> Any example or explanation you give me for resolve_shared_entities is
>>>>> much appreciated!
>>>>>
>>>>> Ryan
>>>>>
>>>>> On 10/25/18 09:40, Vijay S. Mahadevan wrote:
>>>>>> Hi Ryan,
>>>>>>
>>>>>> Glad the example helped.
>>>>>>
>>>>>>> The example shows the ghost exchange happening between shared owned
>>>>>> entities. In my experiments I've been unable to make the ghost exchange
>>>>>> work, and I think that might be because my entities are not shared.
>>>>>>
>>>>>> You need to call resolve_shared_entities on the entities prior to
>>>>>> doing ghost exchange. When we load the mesh from file, this happens
>>>>>> automatically based on the read options but if you are forming a mesh
>>>>>> in memory, you need to make sure that the shared vertices have the
>>>>>> same GLOBAL_ID numbering consistently across processes. That is shared
>>>>>> vertices are unique. Once that is set, shared entities resolution will
>>>>>> work correctly out of the box and you will have shared edges/entities
>>>>>> query working correctly. A call to get ghosted layers once this is
>>>>>> done would be the way to go.
>>>>>>
>>>>>> I assume ESMF has a unique global numbering for the vertices ? Use
>>>>>> that to set the GLOBAL_ID tag. Let us know if you are still having
>>>>>> issues.
>>>>>>
>>>>>> Vijay
>>>>>> On Thu, Oct 25, 2018 at 11:17 AM Ryan O'Kuinghttons
>>>>>> <ryan.okuinghttons at noaa.gov> wrote:
>>>>>>> Hi Vijay,
>>>>>>>
>>>>>>> I've had some time to play with this now, and I have another question.
>>>>>>> Thank you for sending along the example code, it has been extremely
>>>>>>> helpful.
>>>>>>>
>>>>>>> The example shows the ghost exchange happening between shared owned
>>>>>>> entities. In my experiments I've been unable to make the ghost exchange
>>>>>>> work, and I think that might be because my entities are not shared. The
>>>>>>> situation I have is entities that are owned wholly on a single
>>>>>>> processor, which need to be communicated to other processors which
>>>>>>> require them as part of a halo region for mesh based computation. In
>>>>>>> this situation would I need to "share" my entities across the whole
>>>>>>> processor space, before requesting a ghost exchange? I'm not even really
>>>>>>> sure what shared entities mean, is there a good place to look in the
>>>>>>> documentation to learn more about the terminology? Thanks,
>>>>>>>
>>>>>>> Ryan
>>>>>>>
>>>>>>> On 10/8/18 12:17, Vijay S. Mahadevan wrote:
>>>>>>>> Ryan,
>>>>>>>>
>>>>>>>> You need to use the ParallelComm object in MOAB to call
>>>>>>>> exchange_ghost_cells [1] with appropriate levels of ghost layers for
>>>>>>>> your mesh. This needs to be done after the mesh is loaded, or you can
>>>>>>>> pass this information also as part of the options when loading a file
>>>>>>>> and MOAB will internally load the file with ghosted layers. Here's an
>>>>>>>> example [2].
>>>>>>>>
>>>>>>>> Vijay
>>>>>>>>
>>>>>>>> [1] http://ftp.mcs.anl.gov/pub/fathom/moab-docs-develop/classmoab_1_1ParallelComm.html#a55dfa308f56fd368319bfb4244428878
>>>>>>>> [2] http://ftp.mcs.anl.gov/pub/fathom/moab-docs-develop/HelloParMOAB_8cpp-example.html
>>>>>>>> On Mon, Oct 8, 2018 at 1:23 PM Ryan O'Kuinghttons
>>>>>>>> <ryan.okuinghttons at noaa.gov> wrote:
>>>>>>>>> Hi, I am wondering if there is a way to create ghost elements in MOAB.
>>>>>>>>> By this I mean a list of copies of elements surrounding a specific MOAB
>>>>>>>>> element, ghost elements may exist on a different processor than the
>>>>>>>>> source element. I see a ghost_elems class in the appData namespace, but
>>>>>>>>> there is not much documentation on how to use it.. Thank you,
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Ryan O'Kuinghttons
>>>>>>>>> Cherokee Nation Management and Consulting
>>>>>>>>> NESII/NOAA/Earth System Research Laboratory
>>>>>>>>> ryan.okuinghttons at noaa.gov
>>>>>>>>> https://www.esrl.noaa.gov/gsd/nesii/
>>>>>>>>>
More information about the moab-dev
mailing list