[MOAB-dev] ghost/halo elements
Ryan O'Kuinghttons
ryan.okuinghttons at noaa.gov
Mon Oct 29 16:57:43 CDT 2018
Hi Iulian, I know this isn't a clean or easy problem, I really
appreciate you looking into it with me.
The tag is created like this:
merr=moab_mesh->tag_get_handle("elem_coords", mbmp->sdim,
MB_TYPE_DOUBLE, mbmp->elem_coords_tag, MB_TAG_EXCL|MB_TAG_DENSE,
&dbl_def_val);
and it is set like this:
merr=moab_mesh->tag_set_data(mbmp->elem_coords_tag, &new_elem, 1,
elemCoords+mbmp->sdim*e);
This is a small test mesh, distributed like this:
//
// 3.0 13 ------ 14 ------ 15 [15] ----------- 16
// | | | | |
// | | | | |
// | 8 | 9 | | 10 |
// | | | | |
// | | | | |
// 2.0 [9] ----- [10] ---- [11] [11] ---------- [12]
//
// 1.0 1.5 2.0 2.0 3.0
//
// PET 2 PET 3
//
//
// 2.0 9 ------- 10 ------ 11 [11] ----------- 12
// | | | | |
// | 5 | 6 | | 7 |
// | | | | |
// 1.5 5 ------- 6 ------- 7 [7] ----------- 8
// | | | | |
// | 1 | 2 | | 4 |
// | | | | |
// 1.0 1 ------- 2 ------- 3 [3] ------------ 4
//
// 1.0 1.5 2.0 2.0 3.0
//
// PET 0 PET 1
//
// Node Id labels at corners
// Element Id labels in centers
It would be difficult to create a small reproducer for you, given that
I'm working on a development branch, but I could probably send you a
tarball if we get to that. Likewise, it would be some work to create a
similar test on 2 processors, but not out of the question.
Is there anything that comes to mind with the above information that I
could look into first? Is it possible that this is duplicating a default
tag in moab? (given the message MB_ALREADY_ALLOCATED)..
On 10/29/18 15:31, Grindeanu, Iulian R. wrote:
> Sorry, I still don't know what is wrong.
> What kind of tag is it? is it fixed length, does it have a default value?
> How was this tag created/set ?
>
> the sizes of the buffers sent seem to be small, is this a small test mesh? how is this partitioned?
> Can you replicate the bug in a way I can run on my machine? I mean, I do have an old version of esmf. I assume this is on newer code. Or can you replicate on smaller number of tasks (maybe 2?)
>
> Iulian
>
> ________________________________________
> From: Ryan O'Kuinghttons <ryan.okuinghttons at noaa.gov>
> Sent: Monday, October 29, 2018 4:10:16 PM
> To: Grindeanu, Iulian R.; Vijay S. Mahadevan
> Cc: moab-dev at mcs.anl.gov
> Subject: Re: [MOAB-dev] ghost/halo elements
>
> 0 ParallelComm(0.03 s) Entering exchange_tags
> 3 ParallelComm(0.03 s) Entering exchange_tags
> 1 ParallelComm(0.03 s) Entering exchange_tags
> 2 ParallelComm(0.03 s) Entering exchange_tags
> 3 ParallelComm(0.03 s) Irecv, 1<-3, buffer ptr = 0x13a5a20, tag=7,
> size=1024, incoming=1
> 1 ParallelComm(0.03 s) Irecv, 0<-1, buffer ptr = 0x1115b30, tag=7,
> size=1024, incoming=1
> 0 ParallelComm(0.03 s) Irecv, 1<-0, buffer ptr = 0x11cc7e0, tag=7,
> size=1024, incoming=1
> 1 ParallelComm(0.03 s) Irecv, 2<-1, buffer ptr = 0x17a4330, tag=7,
> size=1024, incoming=2
> 0 ParallelComm(0.03 s) Irecv, 2<-0, buffer ptr = 0x11cdf60, tag=7,
> size=1024, incoming=2
> 0 ParallelComm(0.03 s) Irecv, 3<-0, buffer ptr = 0xb4b9e0, tag=7,
> size=1024, incoming=3
> 3 ParallelComm(0.03 s) Irecv, 2<-3, buffer ptr = 0x13a6360, tag=7,
> size=1024, incoming=2
> 3 ParallelComm(0.03 s) Irecv, 0<-3, buffer ptr = 0xd23290, tag=7,
> size=1024, incoming=3
> 1 ParallelComm(0.03 s) Irecv, 3<-1, buffer ptr = 0x11135a0, tag=7,
> size=1024, incoming=3
> 2 ParallelComm(0.03 s) Irecv, 0<-2, buffer ptr = 0x16505f0, tag=7,
> size=1024, incoming=1
> 2 ParallelComm(0.03 s) Irecv, 1<-2, buffer ptr = 0x2362770, tag=7,
> size=1024, incoming=2
> 2 ParallelComm(0.03 s) Irecv, 3<-2, buffer ptr = 0x1651720, tag=7,
> size=1024, incoming=3
> 1 ParallelComm(0.03 s) Packing tag "elem_coords"(0.03 s)
> 0 ParallelComm(0.03 s) Packing tag "elem_coords"(0.03 s)
> 3 ParallelComm(0.03 s) Packing tag "elem_coords"(0.03 s)
> 2 ParallelComm(0.03 s) Packing tag "elem_coords"(0.03 s)
> 1 ParallelComm(0.03 s) Done packing tags.(0.03 s) Isend, 1->0,
> buffer ptr = 0x17a5500, tag=7, size=83
> 0 ParallelComm(0.03 s) Done packing tags.(0.03 s) Isend, 0->1,
> buffer ptr = 0x11ceee0, tag=7, size=83
> 3 ParallelComm(0.03 s) Done packing tags.(0.03 s) Isend, 3->1,
> buffer ptr = 0x1a27200, tag=7, size=83
> 2 ParallelComm(0.03 s) Done packing tags.(0.03 s) Isend, 2->0,
> buffer ptr = 0x2363980, tag=7, size=107
> 0 ParallelComm(0.03 s) Packing tag "elem_coords"(0.03 s)
> 3 ParallelComm(0.03 s) Packing tag "elem_coords"(0.03 s)
> 1 ParallelComm(0.03 s) Packing tag "elem_coords"(0.03 s)
> 2 ParallelComm(0.03 s) Packing tag "elem_coords"(0.03 s)
> 0 ParallelComm(0.03 s) Done packing tags.(0.03 s) Isend, 0->2,
> buffer ptr = 0xb4a0d0, tag=7, size=83
> 3 ParallelComm(0.03 s) Done packing tags.(0.03 s) Isend, 3->2,
> buffer ptr = 0xd22a20, tag=7, size=131
> 1 ParallelComm(0.03 s) Done packing tags.(0.03 s) Isend, 1->2,
> buffer ptr = 0x1114230, tag=7, size=107
> 2 ParallelComm(0.03 s) Done packing tags.(0.03 s) Isend, 2->1,
> buffer ptr = 0x1652330, tag=7, size=107
> 0 ParallelComm(0.03 s) Packing tag "elem_coords"(0.03 s)
> 1 ParallelComm(0.03 s) Packing tag "elem_coords"(0.03 s)
> 3 ParallelComm(0.03 s) Packing tag "elem_coords"(0.03 s)
> 1 ParallelComm(0.03 s) Done packing tags.(0.03 s) Isend, 1->3,
> buffer ptr = 0x11163b0, tag=7, size=107
> 0 ParallelComm(0.03 s) Done packing tags.(0.03 s) Isend, 0->3,
> buffer ptr = 0xb4bdf0, tag=7, size=59
> 3 ParallelComm(0.03 s) Done packing tags.(0.03 s) Isend, 3->0,
> buffer ptr = 0xd22610, tag=7, size=59
> 0 ParallelComm(0.03 s) Unpacking tag elem_coords
> 1 ParallelComm(0.03 s) Unpacking tag elem_coords
> 2 ParallelComm(0.03 s) Packing tag "elem_coords"(0.03 s)
> 3 ParallelComm(0.03 s) Unpacking tag elem_coords
> 2 ParallelComm(0.03 s) Done packing tags.(0.03 s) Isend, 2->3,
> buffer ptr = 0x1652bd0, tag=7, size=107
> 2 ParallelComm(0.03 s) Unpacking tag elem_coords
> [3]MOAB ERROR: --------------------- Error Message
> ------------------------------------
> [0]MOAB ERROR: --------------------- Error Message
> ------------------------------------
> [0]MOAB ERROR: Failed to recv-unpack-tag message!
> [1]MOAB ERROR: --------------------- Error Message
> ------------------------------------
> [1]MOAB ERROR: Failed to recv-unpack-tag message!
> [3]MOAB ERROR: Failed to recv-unpack-tag message!
> [3]MOAB ERROR: exchange_tags() line 7414 in ParallelComm.cpp
> [0]MOAB ERROR: exchange_tags() line 7414 in ParallelComm.cpp
> [1]MOAB ERROR: exchange_tags() line 7414 in ParallelComm.cpp
> [2]MOAB ERROR: --------------------- Error Message
> ------------------------------------
> [2]MOAB ERROR: Failed to recv-unpack-tag message!
> [2]MOAB ERROR: exchange_tags() line 7414 in ParallelComm.cpp
> terminate called after throwing an instance of 'int'
> terminate called after throwing an instance of 'int'
> terminate called after throwing an instance of 'int'
> terminate called after throwing an instance of 'int'
>
> Program received signal SIGABRT: Process abort signal.
>
> On 10/29/18 15:02, Grindeanu, Iulian R. wrote:
>> Hi Ryan,
>> I don't know what could be wrong; can you set verbosity level in your code?
>> I mean, can you use something like
>>
>> pcomm->set_debug_verbosity(4)
>>
>> before calling exchange tags? And send me the output?
>>
>> I don't think MB_ALREADY_ALLOCATED is a problem in this case; I could be wrong
>>
>> Iulian
>> ________________________________________
>> From: Ryan O'Kuinghttons <ryan.okuinghttons at noaa.gov>
>> Sent: Monday, October 29, 2018 3:44:01 PM
>> To: Grindeanu, Iulian R.; Vijay S. Mahadevan
>> Cc: moab-dev at mcs.anl.gov
>> Subject: Re: [MOAB-dev] ghost/halo elements
>>
>> Hi Iulian, thanks for responding :)
>>
>> So I did get the exchange_tags call to work for all but one of our tags.
>> However, when I add the last tag to the vector that is passed into
>> exchange_tags I get the following segfault:
>>
>> [3]MOAB ERROR: --------------------- Error Message
>> ------------------------------------
>> [3]MOAB ERROR: Failed to recv-unpack-tag message!
>> [3]MOAB ERROR: exchange_tags() line 7414 in ParallelComm.cpp
>> terminate called after throwing an instance of 'int'
>>
>> Program received signal SIGABRT: Process abort signal.
>>
>> Backtrace for this error:
>> [0]MOAB ERROR: --------------------- Error Message
>> ------------------------------------
>> [0]MOAB ERROR: Failed to recv-unpack-tag message!
>> [1]MOAB ERROR: --------------------- Error Message
>> ------------------------------------
>> [2]MOAB ERROR: --------------------- Error Message
>> ------------------------------------
>> [2]MOAB ERROR: Failed to recv-unpack-tag message!
>> [2]MOAB ERROR: exchange_tags() line 7414 in ParallelComm.cpp
>> [0]MOAB ERROR: exchange_tags() line 7414 in ParallelComm.cpp
>> [1]MOAB ERROR: Failed to recv-unpack-tag message!
>> [1]MOAB ERROR: exchange_tags() line 7414 in ParallelComm.cpp
>> terminate called after throwing an instance of 'int'
>> terminate called after throwing an instance of 'int'
>> terminate called after throwing an instance of 'int'
>>
>>
>>
>> Do you have any ideas what I could be doing wrong?
>>
>> Ryan
>>
>> On 10/29/18 12:52, Grindeanu, Iulian R. wrote:
>>> Hi Ryan,
>>> Sorry for missing the messages, my mistake;
>>> If you have your tags defined on your entities, you will need to call the "exchange_tags" method, after your ghost method;
>>> (call this one:
>>> ErrorCode ParallelComm::exchange_tags(const std::vector<Tag> &src_tags,
>>> const std::vector<Tag> &dst_tags,
>>> const Range &entities_in)
>>> usually, source and dest tags are the same lists)
>>>
>>>
>>> the augment method is for "membership" to specific moab sets (material, boundary conditions, partition );
>>>
>>> Entities (cells, usually), do not have a tag defined on them that would signal they are part of those sets, this is why we had to transfer that information in a different way.
>>>
>>> What is exactly your use case? Do you need to transfer information about a set or about a tag ?
>>>
>>> Iulian
>>>
>>> ________________________________________
>>> From: Vijay S. Mahadevan <vijay.m at gmail.com>
>>> Sent: Monday, October 29, 2018 1:05:53 PM
>>> To: Ryan OKuinghttons - NOAA Affiliate
>>> Cc: Grindeanu, Iulian R.; moab-dev at mcs.anl.gov
>>> Subject: Re: [MOAB-dev] ghost/halo elements
>>>
>>> Hi Ryan,
>>>
>>> The method `augment_default_sets_with_ghosts` [1] in ParallelComm
>>> enables this for some of the default tags. We haven't exposed this as
>>> a very general mechanism as the user could define the tag data layout
>>> in many possible ways and it would be difficult to cover all use cases
>>> from a library standpoint. Remember that for custom defined Tags, you
>>> can also use exchange_tag/reduce_tags in ParallelComm to get the data
>>> that you need on shared entities. If exchange_tags does not suit your
>>> needs, you can see if you can adapt augment_default_sets_with_ghosts
>>> method for your use case. Let me know if that helps.
>>>
>>> Vijay
>>>
>>> [1] http://ftp.mcs.anl.gov/pub/fathom/moab-docs-develop/classmoab_1_1ParallelComm.html#a76dced0b6dbe2fb99340945c7ca3c0f2
>>> On Mon, Oct 29, 2018 at 1:49 PM Ryan O'Kuinghttons
>>> <ryan.okuinghttons at noaa.gov> wrote:
>>>> Hi Iulian, Vijay,
>>>>
>>>> We have a set of custom tags that we use in ESMF, would it be possible
>>>> to transfer these as well?
>>>>
>>>> Ryan
>>>>
>>>> On 10/29/18 10:54, Vijay S. Mahadevan wrote:
>>>>> The default set of tags do get transferred. These would be the
>>>>> MATERIAL_SET, DIRICHLET/NEUMANN sets etc.
>>>>>
>>>>> Iulian, did we add a separate method to transfer these to the ghosted
>>>>> entities ? I vaguely remember something like that.
>>>>>
>>>>> Vijay
>>>>> On Mon, Oct 29, 2018 at 12:42 PM Ryan O'Kuinghttons
>>>>> <ryan.okuinghttons at noaa.gov> wrote:
>>>>>> Hi Vijay,
>>>>>>
>>>>>> I ran into another issue with the call to exchange_ghost_cells, but this
>>>>>> is probably just a misunderstanding on my part. I find that the tag
>>>>>> information that is part of the mesh before the the call does not
>>>>>> transfer to the newly created faces. Is there some way to ask moab to
>>>>>> transfer tags when creating ghost entities? (FWIW, I could not find this
>>>>>> in the example you sent me) Thanks,
>>>>>>
>>>>>> Ryan
>>>>>>
>>>>>> On 10/26/18 07:36, Ryan O'Kuinghttons wrote:
>>>>>>> Hi Vijay,
>>>>>>>
>>>>>>> I resolved the issue with not getting any shared, owned entities back
>>>>>>> by using the other version of resolve_shared_ents, as demonstrated in
>>>>>>> the example you indicated. I created a Range containing all elements
>>>>>>> (faces for moab), and used that as input.
>>>>>>>
>>>>>>> Range range_ent;
>>>>>>> merr=mb->get_entities_by_dimension(0,this->sdim,range_ent);
>>>>>>> MBMESH_CHECK_ERR(merr, localrc);
>>>>>>>
>>>>>>> merr = pcomm->resolve_shared_ents(0, range_ent, this->sdim, 1);
>>>>>>> MBMESH_CHECK_ERR(merr, localrc);
>>>>>>>
>>>>>>> The first version of the interface (which does not use a Range input)
>>>>>>> does not work for me:
>>>>>>>
>>>>>>> merr = pcomm->resolve_shared_ents(0, this->sdim, 1);
>>>>>>>
>>>>>>> Otherwise, everything else seems to be resolved with this issue,
>>>>>>> thanks again for all of your help!!
>>>>>>>
>>>>>>> Ryan
>>>>>>>
>>>>>>> On 10/25/18 15:13, Vijay S. Mahadevan wrote:
>>>>>>>> Ryan, I don't see anything obviously wrong in the code below. Does
>>>>>>>> your shared_ents not have any entities after parallel resolve ? If you
>>>>>>>> can replicate your issue in a small standalone test, we should be able
>>>>>>>> to quickly find out what's going wrong.
>>>>>>>>
>>>>>>>> Look at the test_reduce_tag_explicit_dest routine in
>>>>>>>> test/parallel/parallel_unit_tests.cpp:1617. It has the workflow where
>>>>>>>> you create a mesh in memory, resolve shared entities, get shared
>>>>>>>> entities and even set/reduce tag data.
>>>>>>>>
>>>>>>>> Vijay
>>>>>>>> On Thu, Oct 25, 2018 at 4:25 PM Ryan O'Kuinghttons
>>>>>>>> <ryan.okuinghttons at noaa.gov> wrote:
>>>>>>>>> OK, that allows the resolve_shared_ents routine to complete without
>>>>>>>>> error, but I still don't get any shared, owned entities back from the
>>>>>>>>> get_shared_entities/filter_pstatus calls:
>>>>>>>>>
>>>>>>>>> Range shared_ents;
>>>>>>>>> // Get entities shared with all other processors
>>>>>>>>> merr = pcomm->get_shared_entities(-1, shared_ents);
>>>>>>>>> MBMESH_CHECK_ERR(merr, localrc);
>>>>>>>>>
>>>>>>>>> // Filter shared entities with not not_owned, which means owned
>>>>>>>>> Range owned_entities;
>>>>>>>>> merr = pcomm->filter_pstatus(shared_ents, PSTATUS_NOT_OWNED,
>>>>>>>>> PSTATUS_NOT, -1, &owned_entities);
>>>>>>>>> MBMESH_CHECK_ERR(merr, localrc);
>>>>>>>>>
>>>>>>>>> unsigned int nums[4] = {0}; // to store the owned entities per
>>>>>>>>> dimension
>>>>>>>>> for (int i = 0; i < 4; i++)
>>>>>>>>> nums[i] = (int)owned_entities.num_of_dimension(i);
>>>>>>>>>
>>>>>>>>> vector<int> rbuf(nprocs*4, 0);
>>>>>>>>> MPI_Gather(nums, 4, MPI_INT, &rbuf[0], 4, MPI_INT, 0, mpi_comm);
>>>>>>>>> // Print the stats gathered:
>>>>>>>>> if (0 == rank) {
>>>>>>>>> for (int i = 0; i < nprocs; i++)
>>>>>>>>> cout << " Shared, owned entities on proc " << i << ": " <<
>>>>>>>>> rbuf[4*i] << " verts, " <<
>>>>>>>>> rbuf[4*i + 1] << " edges, " << rbuf[4*i + 2] << " faces,
>>>>>>>>> " <<
>>>>>>>>> rbuf[4*i + 3] << " elements" << endl;
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> OUTPUT:
>>>>>>>>>
>>>>>>>>> Shared, owned entities on proc 0: 0 verts, 0 edges, 0 faces, 0
>>>>>>>>> elements
>>>>>>>>> Shared, owned entities on proc 1: 0 verts, 0 edges, 0 faces, 0
>>>>>>>>> elements
>>>>>>>>> Shared, owned entities on proc 2: 0 verts, 0 edges, 0 faces, 0
>>>>>>>>> elements
>>>>>>>>> Shared, owned entities on proc 3: 0 verts, 0 edges, 0 faces, 0
>>>>>>>>> elements
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Note: I also tried changing the -1 to 2 in the
>>>>>>>>> get_shared_entities/filter_pstatus calls to only retrieve elements.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 10/25/18 13:57, Vijay S. Mahadevan wrote:
>>>>>>>>>> No problem. Can you try 1 for the last argument to look for edges as
>>>>>>>>>> the shared_dim.
>>>>>>>>>>
>>>>>>>>>> Vijay
>>>>>>>>>> On Thu, Oct 25, 2018 at 3:41 PM Ryan O'Kuinghttons
>>>>>>>>>> <ryan.okuinghttons at noaa.gov> wrote:
>>>>>>>>>>> OK, thanks again for that explanation, I never would have been
>>>>>>>>>>> able to figure out that this_set was the root set. I apologize for
>>>>>>>>>>> all the dumb questions, hopefully they are all getting me closer
>>>>>>>>>>> to self sufficiency..
>>>>>>>>>>>
>>>>>>>>>>> I'm getting an error that doesn't make sense to me. I specifying
>>>>>>>>>>> the resolve_dim and shared_dim in the call:
>>>>>>>>>>>
>>>>>>>>>>> // Get the ParallelComm instance
>>>>>>>>>>> ParallelComm* pcomm = new ParallelComm(mb, mpi_comm);
>>>>>>>>>>> int nprocs = pcomm->proc_config().proc_size();
>>>>>>>>>>> int rank = pcomm->proc_config().proc_rank();
>>>>>>>>>>>
>>>>>>>>>>> // get the root set handle
>>>>>>>>>>> EntityHandle root_set = mb->get_root_set();
>>>>>>>>>>>
>>>>>>>>>>> merr = pcomm->resolve_shared_ents(root_set, 2, -1);
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> but I'm getting this message:
>>>>>>>>>>>
>>>>>>>>>>> [0]MOAB ERROR: Unable to guess shared_dim or resolve_dim!
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> what am i missing?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On 10/25/18 12:51, Vijay S. Mahadevan wrote:
>>>>>>>>>>>
>>>>>>>>>>> It is your root set or fileset if you added entities to it. You
>>>>>>>>>>> can get the root set handle from Core object.
>>>>>>>>>>>
>>>>>>>>>>> Vijay
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Oct 25, 2018, 14:38 Ryan O'Kuinghttons
>>>>>>>>>>> <ryan.okuinghttons at noaa.gov> wrote:
>>>>>>>>>>>> Thanks again, Vijay. However, I still don't understand what this_set
>>>>>>>>>>>> should be, is it an output maybe?
>>>>>>>>>>>>
>>>>>>>>>>>> Ryan
>>>>>>>>>>>>
>>>>>>>>>>>> On 10/25/18 12:27, Vijay S. Mahadevan wrote:
>>>>>>>>>>>>> You can try either the first or the second variant instead [1] with
>>>>>>>>>>>>> following arguments.
>>>>>>>>>>>>>
>>>>>>>>>>>>> ErrorCode moab::ParallelComm::resolve_shared_ents(EntityHandle
>>>>>>>>>>>>> this_set,
>>>>>>>>>>>>> int resolve_dim = 2,
>>>>>>>>>>>>> int shared_dim = -1,
>>>>>>>>>>>>> const Tag * id_tag = 0
>>>>>>>>>>>>> )
>>>>>>>>>>>>>
>>>>>>>>>>>>> That should resolve 2-dim entities with shared edges across
>>>>>>>>>>>>> partitions. You can leave id_tag pointer as zero since the
>>>>>>>>>>>>> default is
>>>>>>>>>>>>> GLOBAL_ID.
>>>>>>>>>>>>>
>>>>>>>>>>>>>> This brings up a more general question I've had about the moab
>>>>>>>>>>>>> documentation for a while. In the doc for this routine, it only
>>>>>>>>>>>>> lists 2
>>>>>>>>>>>>> parameters, proc_ents and shared_dim, even though in the function
>>>>>>>>>>>>> signature above it clearly shows more. I've had trouble
>>>>>>>>>>>>> understanding
>>>>>>>>>>>>> which parameters are relevant in the past, or what they do
>>>>>>>>>>>>> because I'm
>>>>>>>>>>>>> not quite sure how to read the documentation.
>>>>>>>>>>>>>
>>>>>>>>>>>>> This is an oversight. We will go through and rectify some of the
>>>>>>>>>>>>> inconsistencies in the documentation. We are preparing for an
>>>>>>>>>>>>> upcoming
>>>>>>>>>>>>> release and I'll make sure that routines in Core/ParallelComm have
>>>>>>>>>>>>> updated documentation that match the interfaces. Meanwhile, if you
>>>>>>>>>>>>> have questions, feel free to shoot an email to the list.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hope that helps.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Vijay
>>>>>>>>>>>>>
>>>>>>>>>>>>> [1]
>>>>>>>>>>>>> http://ftp.mcs.anl.gov/pub/fathom/moab-docs-develop/classmoab_1_1ParallelComm.html#a29a3b834b3fc3b4ddb3a5d8a78a37c8a
>>>>>>>>>>>>> On Thu, Oct 25, 2018 at 1:58 PM Ryan O'Kuinghttons
>>>>>>>>>>>>> <ryan.okuinghttons at noaa.gov> wrote:
>>>>>>>>>>>>>> Hi Vijay,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks again for that explanation.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ESMF does use unique global ids for the vertices and set them
>>>>>>>>>>>>>> to the
>>>>>>>>>>>>>> GLOBAL_ID in the moab mesh. So I think we are good there.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I can't quite figure out how to use resolve_shared_entities
>>>>>>>>>>>>>> though.
>>>>>>>>>>>>>> There are three versions of the call in the documentation, and
>>>>>>>>>>>>>> I assume
>>>>>>>>>>>>>> that I should use the first and pass in a Range containing all
>>>>>>>>>>>>>> entities
>>>>>>>>>>>>>> for proc_ents:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> http://ftp.mcs.anl.gov/pub/fathom/moab-docs-develop/classmoab_1_1ParallelComm.html#a59e35d9906f2e33fe010138a144a5cb6
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> However, I'm not sure what the this_set EntityHandle should be.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> This brings up a more general question I've had about the moab
>>>>>>>>>>>>>> documentation for a while. In the doc for this routine, it only
>>>>>>>>>>>>>> lists 2
>>>>>>>>>>>>>> parameters, proc_ents and shared_dim, even though in the function
>>>>>>>>>>>>>> signature above it clearly shows more. I've had trouble
>>>>>>>>>>>>>> understanding
>>>>>>>>>>>>>> which parameters are relevant in the past, or what they do
>>>>>>>>>>>>>> because I'm
>>>>>>>>>>>>>> not quite sure how to read the documentation.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Any example or explanation you give me for
>>>>>>>>>>>>>> resolve_shared_entities is
>>>>>>>>>>>>>> much appreciated!
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Ryan
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 10/25/18 09:40, Vijay S. Mahadevan wrote:
>>>>>>>>>>>>>>> Hi Ryan,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Glad the example helped.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> The example shows the ghost exchange happening between shared
>>>>>>>>>>>>>>>> owned
>>>>>>>>>>>>>>> entities. In my experiments I've been unable to make the ghost
>>>>>>>>>>>>>>> exchange
>>>>>>>>>>>>>>> work, and I think that might be because my entities are not
>>>>>>>>>>>>>>> shared.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> You need to call resolve_shared_entities on the entities prior to
>>>>>>>>>>>>>>> doing ghost exchange. When we load the mesh from file, this
>>>>>>>>>>>>>>> happens
>>>>>>>>>>>>>>> automatically based on the read options but if you are forming
>>>>>>>>>>>>>>> a mesh
>>>>>>>>>>>>>>> in memory, you need to make sure that the shared vertices have
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> same GLOBAL_ID numbering consistently across processes. That
>>>>>>>>>>>>>>> is shared
>>>>>>>>>>>>>>> vertices are unique. Once that is set, shared entities
>>>>>>>>>>>>>>> resolution will
>>>>>>>>>>>>>>> work correctly out of the box and you will have shared
>>>>>>>>>>>>>>> edges/entities
>>>>>>>>>>>>>>> query working correctly. A call to get ghosted layers once
>>>>>>>>>>>>>>> this is
>>>>>>>>>>>>>>> done would be the way to go.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I assume ESMF has a unique global numbering for the vertices ?
>>>>>>>>>>>>>>> Use
>>>>>>>>>>>>>>> that to set the GLOBAL_ID tag. Let us know if you are still
>>>>>>>>>>>>>>> having
>>>>>>>>>>>>>>> issues.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Vijay
>>>>>>>>>>>>>>> On Thu, Oct 25, 2018 at 11:17 AM Ryan O'Kuinghttons
>>>>>>>>>>>>>>> <ryan.okuinghttons at noaa.gov> wrote:
>>>>>>>>>>>>>>>> Hi Vijay,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I've had some time to play with this now, and I have another
>>>>>>>>>>>>>>>> question.
>>>>>>>>>>>>>>>> Thank you for sending along the example code, it has been
>>>>>>>>>>>>>>>> extremely
>>>>>>>>>>>>>>>> helpful.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> The example shows the ghost exchange happening between shared
>>>>>>>>>>>>>>>> owned
>>>>>>>>>>>>>>>> entities. In my experiments I've been unable to make the
>>>>>>>>>>>>>>>> ghost exchange
>>>>>>>>>>>>>>>> work, and I think that might be because my entities are not
>>>>>>>>>>>>>>>> shared. The
>>>>>>>>>>>>>>>> situation I have is entities that are owned wholly on a single
>>>>>>>>>>>>>>>> processor, which need to be communicated to other processors
>>>>>>>>>>>>>>>> which
>>>>>>>>>>>>>>>> require them as part of a halo region for mesh based
>>>>>>>>>>>>>>>> computation. In
>>>>>>>>>>>>>>>> this situation would I need to "share" my entities across the
>>>>>>>>>>>>>>>> whole
>>>>>>>>>>>>>>>> processor space, before requesting a ghost exchange? I'm not
>>>>>>>>>>>>>>>> even really
>>>>>>>>>>>>>>>> sure what shared entities mean, is there a good place to look
>>>>>>>>>>>>>>>> in the
>>>>>>>>>>>>>>>> documentation to learn more about the terminology? Thanks,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Ryan
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On 10/8/18 12:17, Vijay S. Mahadevan wrote:
>>>>>>>>>>>>>>>>> Ryan,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> You need to use the ParallelComm object in MOAB to call
>>>>>>>>>>>>>>>>> exchange_ghost_cells [1] with appropriate levels of ghost
>>>>>>>>>>>>>>>>> layers for
>>>>>>>>>>>>>>>>> your mesh. This needs to be done after the mesh is loaded,
>>>>>>>>>>>>>>>>> or you can
>>>>>>>>>>>>>>>>> pass this information also as part of the options when
>>>>>>>>>>>>>>>>> loading a file
>>>>>>>>>>>>>>>>> and MOAB will internally load the file with ghosted layers.
>>>>>>>>>>>>>>>>> Here's an
>>>>>>>>>>>>>>>>> example [2].
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Vijay
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>> http://ftp.mcs.anl.gov/pub/fathom/moab-docs-develop/classmoab_1_1ParallelComm.html#a55dfa308f56fd368319bfb4244428878
>>>>>>>>>>>>>>>>> [2]
>>>>>>>>>>>>>>>>> http://ftp.mcs.anl.gov/pub/fathom/moab-docs-develop/HelloParMOAB_8cpp-example.html
>>>>>>>>>>>>>>>>> On Mon, Oct 8, 2018 at 1:23 PM Ryan O'Kuinghttons
>>>>>>>>>>>>>>>>> <ryan.okuinghttons at noaa.gov> wrote:
>>>>>>>>>>>>>>>>>> Hi, I am wondering if there is a way to create ghost
>>>>>>>>>>>>>>>>>> elements in MOAB.
>>>>>>>>>>>>>>>>>> By this I mean a list of copies of elements surrounding a
>>>>>>>>>>>>>>>>>> specific MOAB
>>>>>>>>>>>>>>>>>> element, ghost elements may exist on a different processor
>>>>>>>>>>>>>>>>>> than the
>>>>>>>>>>>>>>>>>> source element. I see a ghost_elems class in the appData
>>>>>>>>>>>>>>>>>> namespace, but
>>>>>>>>>>>>>>>>>> there is not much documentation on how to use it.. Thank you,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>> Ryan O'Kuinghttons
>>>>>>>>>>>>>>>>>> Cherokee Nation Management and Consulting
>>>>>>>>>>>>>>>>>> NESII/NOAA/Earth System Research Laboratory
>>>>>>>>>>>>>>>>>> ryan.okuinghttons at noaa.gov
>>>>>>>>>>>>>>>>>> https://www.esrl.noaa.gov/gsd/nesii/
>>>>>>>>>>>>>>>>>>
More information about the moab-dev
mailing list