[MOAB-dev] ghost/halo elements
Ryan O'Kuinghttons
ryan.okuinghttons at noaa.gov
Tue Oct 30 09:27:33 CDT 2018
Hi Iulian,
I'll answer your questions one at a time:
- sdim is 2 in this test, it the simple 2D mesh in the ascii drawing in
the last mail
- Yes, elem_coords is a tag with 2 doubles per entity
- Yes, merr is checked after both tag creation and tag set, both are fine
- The MB_ALREADY_ALLOCATED comes from the moab error code return after
exchange_tags, but only when the elem_coords tag is in the vector of
tags to exchange
- I have written out the vtk files already, the tags that are not
exchanged (elem_coords) come back with value of 0, so the mesh has a
bunch of connections that go back to (0,0)
- elemCoords+mbmp->sdim*e is a reference to the e'th element in the
elemCoords array of coordinate values. This code is widely tested in a
variety of scenarios and passes everywhere. Additionally, the
elem_coords tag is created and set as part of the original mesh creation
in my test code, and writing out the mesh before the ghost_exchange
shows a healthy looking mesh.
I'm a bit hesitant to ask you to look at the esmf moab code directly
because it requires building esmf, which can be a hassle. However, if
you think that is the best way to go, I will send instructions on where
to get the code and how to build.
Ryan
On 10/29/18 19:54, Grindeanu, Iulian R. wrote:
> Hi Ryan,
> Yes, nothing comes out right away; it would be easier if I could debug this on my computer;
> what is sdim? Is it 2? So is this a tag with 2 doubles per entity?
> did you check merr after tag creaton?
> If you use MB_TAG_EXCL , it means the tag should not have been created before;
>
> When did you get MB_ALREADY_ALLOCATED? after you called tag exchange? I did not see in output, maybe cerr is redirected?
>
> You can try to write out the file (in serial), from each task; to check that ghosting did what you expect:
> something like this:
>
> std::ostringstream ent_str;
> ent_str << "mesh." <<pcomm->rank() << ".vtk";
> moab_mesh->write_mesh(ent_str.str().c_str());
>
>
> what is elemCoords+mbmp->sdim*e in your code? I assume you are using some pointer arithmetic, are all those variables OK?
> Is elemCoords a double pointer, and e an index ?
>
> Iulian
>
> ________________________________________
> From: Ryan O'Kuinghttons <ryan.okuinghttons at noaa.gov>
> Sent: Monday, October 29, 2018 4:57:43 PM
> To: Grindeanu, Iulian R.; Vijay S. Mahadevan
> Cc: moab-dev at mcs.anl.gov
> Subject: Re: [MOAB-dev] ghost/halo elements
>
> Hi Iulian, I know this isn't a clean or easy problem, I really
> appreciate you looking into it with me.
>
> The tag is created like this:
>
> merr=moab_mesh->tag_get_handle("elem_coords", mbmp->sdim,
> MB_TYPE_DOUBLE, mbmp->elem_coords_tag, MB_TAG_EXCL|MB_TAG_DENSE,
> &dbl_def_val);
>
> and it is set like this:
>
> merr=moab_mesh->tag_set_data(mbmp->elem_coords_tag, &new_elem, 1,
> elemCoords+mbmp->sdim*e);
>
>
> This is a small test mesh, distributed like this:
>
> //
> // 3.0 13 ------ 14 ------ 15 [15] ----------- 16
> // | | | | |
> // | | | | |
> // | 8 | 9 | | 10 |
> // | | | | |
> // | | | | |
> // 2.0 [9] ----- [10] ---- [11] [11] ---------- [12]
> //
> // 1.0 1.5 2.0 2.0 3.0
> //
> // PET 2 PET 3
> //
> //
> // 2.0 9 ------- 10 ------ 11 [11] ----------- 12
> // | | | | |
> // | 5 | 6 | | 7 |
> // | | | | |
> // 1.5 5 ------- 6 ------- 7 [7] ----------- 8
> // | | | | |
> // | 1 | 2 | | 4 |
> // | | | | |
> // 1.0 1 ------- 2 ------- 3 [3] ------------ 4
> //
> // 1.0 1.5 2.0 2.0 3.0
> //
> // PET 0 PET 1
> //
> // Node Id labels at corners
> // Element Id labels in centers
>
> It would be difficult to create a small reproducer for you, given that
> I'm working on a development branch, but I could probably send you a
> tarball if we get to that. Likewise, it would be some work to create a
> similar test on 2 processors, but not out of the question.
>
> Is there anything that comes to mind with the above information that I
> could look into first? Is it possible that this is duplicating a default
> tag in moab? (given the message MB_ALREADY_ALLOCATED)..
>
>
> On 10/29/18 15:31, Grindeanu, Iulian R. wrote:
>> Sorry, I still don't know what is wrong.
>> What kind of tag is it? is it fixed length, does it have a default value?
>> How was this tag created/set ?
>>
>> the sizes of the buffers sent seem to be small, is this a small test mesh? how is this partitioned?
>> Can you replicate the bug in a way I can run on my machine? I mean, I do have an old version of esmf. I assume this is on newer code. Or can you replicate on smaller number of tasks (maybe 2?)
>>
>> Iulian
>>
>> ________________________________________
>> From: Ryan O'Kuinghttons <ryan.okuinghttons at noaa.gov>
>> Sent: Monday, October 29, 2018 4:10:16 PM
>> To: Grindeanu, Iulian R.; Vijay S. Mahadevan
>> Cc: moab-dev at mcs.anl.gov
>> Subject: Re: [MOAB-dev] ghost/halo elements
>>
>> 0 ParallelComm(0.03 s) Entering exchange_tags
>> 3 ParallelComm(0.03 s) Entering exchange_tags
>> 1 ParallelComm(0.03 s) Entering exchange_tags
>> 2 ParallelComm(0.03 s) Entering exchange_tags
>> 3 ParallelComm(0.03 s) Irecv, 1<-3, buffer ptr = 0x13a5a20, tag=7,
>> size=1024, incoming=1
>> 1 ParallelComm(0.03 s) Irecv, 0<-1, buffer ptr = 0x1115b30, tag=7,
>> size=1024, incoming=1
>> 0 ParallelComm(0.03 s) Irecv, 1<-0, buffer ptr = 0x11cc7e0, tag=7,
>> size=1024, incoming=1
>> 1 ParallelComm(0.03 s) Irecv, 2<-1, buffer ptr = 0x17a4330, tag=7,
>> size=1024, incoming=2
>> 0 ParallelComm(0.03 s) Irecv, 2<-0, buffer ptr = 0x11cdf60, tag=7,
>> size=1024, incoming=2
>> 0 ParallelComm(0.03 s) Irecv, 3<-0, buffer ptr = 0xb4b9e0, tag=7,
>> size=1024, incoming=3
>> 3 ParallelComm(0.03 s) Irecv, 2<-3, buffer ptr = 0x13a6360, tag=7,
>> size=1024, incoming=2
>> 3 ParallelComm(0.03 s) Irecv, 0<-3, buffer ptr = 0xd23290, tag=7,
>> size=1024, incoming=3
>> 1 ParallelComm(0.03 s) Irecv, 3<-1, buffer ptr = 0x11135a0, tag=7,
>> size=1024, incoming=3
>> 2 ParallelComm(0.03 s) Irecv, 0<-2, buffer ptr = 0x16505f0, tag=7,
>> size=1024, incoming=1
>> 2 ParallelComm(0.03 s) Irecv, 1<-2, buffer ptr = 0x2362770, tag=7,
>> size=1024, incoming=2
>> 2 ParallelComm(0.03 s) Irecv, 3<-2, buffer ptr = 0x1651720, tag=7,
>> size=1024, incoming=3
>> 1 ParallelComm(0.03 s) Packing tag "elem_coords"(0.03 s)
>> 0 ParallelComm(0.03 s) Packing tag "elem_coords"(0.03 s)
>> 3 ParallelComm(0.03 s) Packing tag "elem_coords"(0.03 s)
>> 2 ParallelComm(0.03 s) Packing tag "elem_coords"(0.03 s)
>> 1 ParallelComm(0.03 s) Done packing tags.(0.03 s) Isend, 1->0,
>> buffer ptr = 0x17a5500, tag=7, size=83
>> 0 ParallelComm(0.03 s) Done packing tags.(0.03 s) Isend, 0->1,
>> buffer ptr = 0x11ceee0, tag=7, size=83
>> 3 ParallelComm(0.03 s) Done packing tags.(0.03 s) Isend, 3->1,
>> buffer ptr = 0x1a27200, tag=7, size=83
>> 2 ParallelComm(0.03 s) Done packing tags.(0.03 s) Isend, 2->0,
>> buffer ptr = 0x2363980, tag=7, size=107
>> 0 ParallelComm(0.03 s) Packing tag "elem_coords"(0.03 s)
>> 3 ParallelComm(0.03 s) Packing tag "elem_coords"(0.03 s)
>> 1 ParallelComm(0.03 s) Packing tag "elem_coords"(0.03 s)
>> 2 ParallelComm(0.03 s) Packing tag "elem_coords"(0.03 s)
>> 0 ParallelComm(0.03 s) Done packing tags.(0.03 s) Isend, 0->2,
>> buffer ptr = 0xb4a0d0, tag=7, size=83
>> 3 ParallelComm(0.03 s) Done packing tags.(0.03 s) Isend, 3->2,
>> buffer ptr = 0xd22a20, tag=7, size=131
>> 1 ParallelComm(0.03 s) Done packing tags.(0.03 s) Isend, 1->2,
>> buffer ptr = 0x1114230, tag=7, size=107
>> 2 ParallelComm(0.03 s) Done packing tags.(0.03 s) Isend, 2->1,
>> buffer ptr = 0x1652330, tag=7, size=107
>> 0 ParallelComm(0.03 s) Packing tag "elem_coords"(0.03 s)
>> 1 ParallelComm(0.03 s) Packing tag "elem_coords"(0.03 s)
>> 3 ParallelComm(0.03 s) Packing tag "elem_coords"(0.03 s)
>> 1 ParallelComm(0.03 s) Done packing tags.(0.03 s) Isend, 1->3,
>> buffer ptr = 0x11163b0, tag=7, size=107
>> 0 ParallelComm(0.03 s) Done packing tags.(0.03 s) Isend, 0->3,
>> buffer ptr = 0xb4bdf0, tag=7, size=59
>> 3 ParallelComm(0.03 s) Done packing tags.(0.03 s) Isend, 3->0,
>> buffer ptr = 0xd22610, tag=7, size=59
>> 0 ParallelComm(0.03 s) Unpacking tag elem_coords
>> 1 ParallelComm(0.03 s) Unpacking tag elem_coords
>> 2 ParallelComm(0.03 s) Packing tag "elem_coords"(0.03 s)
>> 3 ParallelComm(0.03 s) Unpacking tag elem_coords
>> 2 ParallelComm(0.03 s) Done packing tags.(0.03 s) Isend, 2->3,
>> buffer ptr = 0x1652bd0, tag=7, size=107
>> 2 ParallelComm(0.03 s) Unpacking tag elem_coords
>> [3]MOAB ERROR: --------------------- Error Message
>> ------------------------------------
>> [0]MOAB ERROR: --------------------- Error Message
>> ------------------------------------
>> [0]MOAB ERROR: Failed to recv-unpack-tag message!
>> [1]MOAB ERROR: --------------------- Error Message
>> ------------------------------------
>> [1]MOAB ERROR: Failed to recv-unpack-tag message!
>> [3]MOAB ERROR: Failed to recv-unpack-tag message!
>> [3]MOAB ERROR: exchange_tags() line 7414 in ParallelComm.cpp
>> [0]MOAB ERROR: exchange_tags() line 7414 in ParallelComm.cpp
>> [1]MOAB ERROR: exchange_tags() line 7414 in ParallelComm.cpp
>> [2]MOAB ERROR: --------------------- Error Message
>> ------------------------------------
>> [2]MOAB ERROR: Failed to recv-unpack-tag message!
>> [2]MOAB ERROR: exchange_tags() line 7414 in ParallelComm.cpp
>> terminate called after throwing an instance of 'int'
>> terminate called after throwing an instance of 'int'
>> terminate called after throwing an instance of 'int'
>> terminate called after throwing an instance of 'int'
>>
>> Program received signal SIGABRT: Process abort signal.
>>
>> On 10/29/18 15:02, Grindeanu, Iulian R. wrote:
>>> Hi Ryan,
>>> I don't know what could be wrong; can you set verbosity level in your code?
>>> I mean, can you use something like
>>>
>>> pcomm->set_debug_verbosity(4)
>>>
>>> before calling exchange tags? And send me the output?
>>>
>>> I don't think MB_ALREADY_ALLOCATED is a problem in this case; I could be wrong
>>>
>>> Iulian
>>> ________________________________________
>>> From: Ryan O'Kuinghttons <ryan.okuinghttons at noaa.gov>
>>> Sent: Monday, October 29, 2018 3:44:01 PM
>>> To: Grindeanu, Iulian R.; Vijay S. Mahadevan
>>> Cc: moab-dev at mcs.anl.gov
>>> Subject: Re: [MOAB-dev] ghost/halo elements
>>>
>>> Hi Iulian, thanks for responding :)
>>>
>>> So I did get the exchange_tags call to work for all but one of our tags.
>>> However, when I add the last tag to the vector that is passed into
>>> exchange_tags I get the following segfault:
>>>
>>> [3]MOAB ERROR: --------------------- Error Message
>>> ------------------------------------
>>> [3]MOAB ERROR: Failed to recv-unpack-tag message!
>>> [3]MOAB ERROR: exchange_tags() line 7414 in ParallelComm.cpp
>>> terminate called after throwing an instance of 'int'
>>>
>>> Program received signal SIGABRT: Process abort signal.
>>>
>>> Backtrace for this error:
>>> [0]MOAB ERROR: --------------------- Error Message
>>> ------------------------------------
>>> [0]MOAB ERROR: Failed to recv-unpack-tag message!
>>> [1]MOAB ERROR: --------------------- Error Message
>>> ------------------------------------
>>> [2]MOAB ERROR: --------------------- Error Message
>>> ------------------------------------
>>> [2]MOAB ERROR: Failed to recv-unpack-tag message!
>>> [2]MOAB ERROR: exchange_tags() line 7414 in ParallelComm.cpp
>>> [0]MOAB ERROR: exchange_tags() line 7414 in ParallelComm.cpp
>>> [1]MOAB ERROR: Failed to recv-unpack-tag message!
>>> [1]MOAB ERROR: exchange_tags() line 7414 in ParallelComm.cpp
>>> terminate called after throwing an instance of 'int'
>>> terminate called after throwing an instance of 'int'
>>> terminate called after throwing an instance of 'int'
>>>
>>>
>>>
>>> Do you have any ideas what I could be doing wrong?
>>>
>>> Ryan
>>>
>>> On 10/29/18 12:52, Grindeanu, Iulian R. wrote:
>>>> Hi Ryan,
>>>> Sorry for missing the messages, my mistake;
>>>> If you have your tags defined on your entities, you will need to call the "exchange_tags" method, after your ghost method;
>>>> (call this one:
>>>> ErrorCode ParallelComm::exchange_tags(const std::vector<Tag> &src_tags,
>>>> const std::vector<Tag> &dst_tags,
>>>> const Range &entities_in)
>>>> usually, source and dest tags are the same lists)
>>>>
>>>>
>>>> the augment method is for "membership" to specific moab sets (material, boundary conditions, partition );
>>>>
>>>> Entities (cells, usually), do not have a tag defined on them that would signal they are part of those sets, this is why we had to transfer that information in a different way.
>>>>
>>>> What is exactly your use case? Do you need to transfer information about a set or about a tag ?
>>>>
>>>> Iulian
>>>>
>>>> ________________________________________
>>>> From: Vijay S. Mahadevan <vijay.m at gmail.com>
>>>> Sent: Monday, October 29, 2018 1:05:53 PM
>>>> To: Ryan OKuinghttons - NOAA Affiliate
>>>> Cc: Grindeanu, Iulian R.; moab-dev at mcs.anl.gov
>>>> Subject: Re: [MOAB-dev] ghost/halo elements
>>>>
>>>> Hi Ryan,
>>>>
>>>> The method `augment_default_sets_with_ghosts` [1] in ParallelComm
>>>> enables this for some of the default tags. We haven't exposed this as
>>>> a very general mechanism as the user could define the tag data layout
>>>> in many possible ways and it would be difficult to cover all use cases
>>>> from a library standpoint. Remember that for custom defined Tags, you
>>>> can also use exchange_tag/reduce_tags in ParallelComm to get the data
>>>> that you need on shared entities. If exchange_tags does not suit your
>>>> needs, you can see if you can adapt augment_default_sets_with_ghosts
>>>> method for your use case. Let me know if that helps.
>>>>
>>>> Vijay
>>>>
>>>> [1] http://ftp.mcs.anl.gov/pub/fathom/moab-docs-develop/classmoab_1_1ParallelComm.html#a76dced0b6dbe2fb99340945c7ca3c0f2
>>>> On Mon, Oct 29, 2018 at 1:49 PM Ryan O'Kuinghttons
>>>> <ryan.okuinghttons at noaa.gov> wrote:
>>>>> Hi Iulian, Vijay,
>>>>>
>>>>> We have a set of custom tags that we use in ESMF, would it be possible
>>>>> to transfer these as well?
>>>>>
>>>>> Ryan
>>>>>
>>>>> On 10/29/18 10:54, Vijay S. Mahadevan wrote:
>>>>>> The default set of tags do get transferred. These would be the
>>>>>> MATERIAL_SET, DIRICHLET/NEUMANN sets etc.
>>>>>>
>>>>>> Iulian, did we add a separate method to transfer these to the ghosted
>>>>>> entities ? I vaguely remember something like that.
>>>>>>
>>>>>> Vijay
>>>>>> On Mon, Oct 29, 2018 at 12:42 PM Ryan O'Kuinghttons
>>>>>> <ryan.okuinghttons at noaa.gov> wrote:
>>>>>>> Hi Vijay,
>>>>>>>
>>>>>>> I ran into another issue with the call to exchange_ghost_cells, but this
>>>>>>> is probably just a misunderstanding on my part. I find that the tag
>>>>>>> information that is part of the mesh before the the call does not
>>>>>>> transfer to the newly created faces. Is there some way to ask moab to
>>>>>>> transfer tags when creating ghost entities? (FWIW, I could not find this
>>>>>>> in the example you sent me) Thanks,
>>>>>>>
>>>>>>> Ryan
>>>>>>>
>>>>>>> On 10/26/18 07:36, Ryan O'Kuinghttons wrote:
>>>>>>>> Hi Vijay,
>>>>>>>>
>>>>>>>> I resolved the issue with not getting any shared, owned entities back
>>>>>>>> by using the other version of resolve_shared_ents, as demonstrated in
>>>>>>>> the example you indicated. I created a Range containing all elements
>>>>>>>> (faces for moab), and used that as input.
>>>>>>>>
>>>>>>>> Range range_ent;
>>>>>>>> merr=mb->get_entities_by_dimension(0,this->sdim,range_ent);
>>>>>>>> MBMESH_CHECK_ERR(merr, localrc);
>>>>>>>>
>>>>>>>> merr = pcomm->resolve_shared_ents(0, range_ent, this->sdim, 1);
>>>>>>>> MBMESH_CHECK_ERR(merr, localrc);
>>>>>>>>
>>>>>>>> The first version of the interface (which does not use a Range input)
>>>>>>>> does not work for me:
>>>>>>>>
>>>>>>>> merr = pcomm->resolve_shared_ents(0, this->sdim, 1);
>>>>>>>>
>>>>>>>> Otherwise, everything else seems to be resolved with this issue,
>>>>>>>> thanks again for all of your help!!
>>>>>>>>
>>>>>>>> Ryan
>>>>>>>>
>>>>>>>> On 10/25/18 15:13, Vijay S. Mahadevan wrote:
>>>>>>>>> Ryan, I don't see anything obviously wrong in the code below. Does
>>>>>>>>> your shared_ents not have any entities after parallel resolve ? If you
>>>>>>>>> can replicate your issue in a small standalone test, we should be able
>>>>>>>>> to quickly find out what's going wrong.
>>>>>>>>>
>>>>>>>>> Look at the test_reduce_tag_explicit_dest routine in
>>>>>>>>> test/parallel/parallel_unit_tests.cpp:1617. It has the workflow where
>>>>>>>>> you create a mesh in memory, resolve shared entities, get shared
>>>>>>>>> entities and even set/reduce tag data.
>>>>>>>>>
>>>>>>>>> Vijay
>>>>>>>>> On Thu, Oct 25, 2018 at 4:25 PM Ryan O'Kuinghttons
>>>>>>>>> <ryan.okuinghttons at noaa.gov> wrote:
>>>>>>>>>> OK, that allows the resolve_shared_ents routine to complete without
>>>>>>>>>> error, but I still don't get any shared, owned entities back from the
>>>>>>>>>> get_shared_entities/filter_pstatus calls:
>>>>>>>>>>
>>>>>>>>>> Range shared_ents;
>>>>>>>>>> // Get entities shared with all other processors
>>>>>>>>>> merr = pcomm->get_shared_entities(-1, shared_ents);
>>>>>>>>>> MBMESH_CHECK_ERR(merr, localrc);
>>>>>>>>>>
>>>>>>>>>> // Filter shared entities with not not_owned, which means owned
>>>>>>>>>> Range owned_entities;
>>>>>>>>>> merr = pcomm->filter_pstatus(shared_ents, PSTATUS_NOT_OWNED,
>>>>>>>>>> PSTATUS_NOT, -1, &owned_entities);
>>>>>>>>>> MBMESH_CHECK_ERR(merr, localrc);
>>>>>>>>>>
>>>>>>>>>> unsigned int nums[4] = {0}; // to store the owned entities per
>>>>>>>>>> dimension
>>>>>>>>>> for (int i = 0; i < 4; i++)
>>>>>>>>>> nums[i] = (int)owned_entities.num_of_dimension(i);
>>>>>>>>>>
>>>>>>>>>> vector<int> rbuf(nprocs*4, 0);
>>>>>>>>>> MPI_Gather(nums, 4, MPI_INT, &rbuf[0], 4, MPI_INT, 0, mpi_comm);
>>>>>>>>>> // Print the stats gathered:
>>>>>>>>>> if (0 == rank) {
>>>>>>>>>> for (int i = 0; i < nprocs; i++)
>>>>>>>>>> cout << " Shared, owned entities on proc " << i << ": " <<
>>>>>>>>>> rbuf[4*i] << " verts, " <<
>>>>>>>>>> rbuf[4*i + 1] << " edges, " << rbuf[4*i + 2] << " faces,
>>>>>>>>>> " <<
>>>>>>>>>> rbuf[4*i + 3] << " elements" << endl;
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>> OUTPUT:
>>>>>>>>>>
>>>>>>>>>> Shared, owned entities on proc 0: 0 verts, 0 edges, 0 faces, 0
>>>>>>>>>> elements
>>>>>>>>>> Shared, owned entities on proc 1: 0 verts, 0 edges, 0 faces, 0
>>>>>>>>>> elements
>>>>>>>>>> Shared, owned entities on proc 2: 0 verts, 0 edges, 0 faces, 0
>>>>>>>>>> elements
>>>>>>>>>> Shared, owned entities on proc 3: 0 verts, 0 edges, 0 faces, 0
>>>>>>>>>> elements
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Note: I also tried changing the -1 to 2 in the
>>>>>>>>>> get_shared_entities/filter_pstatus calls to only retrieve elements.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 10/25/18 13:57, Vijay S. Mahadevan wrote:
>>>>>>>>>>> No problem. Can you try 1 for the last argument to look for edges as
>>>>>>>>>>> the shared_dim.
>>>>>>>>>>>
>>>>>>>>>>> Vijay
>>>>>>>>>>> On Thu, Oct 25, 2018 at 3:41 PM Ryan O'Kuinghttons
>>>>>>>>>>> <ryan.okuinghttons at noaa.gov> wrote:
>>>>>>>>>>>> OK, thanks again for that explanation, I never would have been
>>>>>>>>>>>> able to figure out that this_set was the root set. I apologize for
>>>>>>>>>>>> all the dumb questions, hopefully they are all getting me closer
>>>>>>>>>>>> to self sufficiency..
>>>>>>>>>>>>
>>>>>>>>>>>> I'm getting an error that doesn't make sense to me. I specifying
>>>>>>>>>>>> the resolve_dim and shared_dim in the call:
>>>>>>>>>>>>
>>>>>>>>>>>> // Get the ParallelComm instance
>>>>>>>>>>>> ParallelComm* pcomm = new ParallelComm(mb, mpi_comm);
>>>>>>>>>>>> int nprocs = pcomm->proc_config().proc_size();
>>>>>>>>>>>> int rank = pcomm->proc_config().proc_rank();
>>>>>>>>>>>>
>>>>>>>>>>>> // get the root set handle
>>>>>>>>>>>> EntityHandle root_set = mb->get_root_set();
>>>>>>>>>>>>
>>>>>>>>>>>> merr = pcomm->resolve_shared_ents(root_set, 2, -1);
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> but I'm getting this message:
>>>>>>>>>>>>
>>>>>>>>>>>> [0]MOAB ERROR: Unable to guess shared_dim or resolve_dim!
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> what am i missing?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On 10/25/18 12:51, Vijay S. Mahadevan wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> It is your root set or fileset if you added entities to it. You
>>>>>>>>>>>> can get the root set handle from Core object.
>>>>>>>>>>>>
>>>>>>>>>>>> Vijay
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, Oct 25, 2018, 14:38 Ryan O'Kuinghttons
>>>>>>>>>>>> <ryan.okuinghttons at noaa.gov> wrote:
>>>>>>>>>>>>> Thanks again, Vijay. However, I still don't understand what this_set
>>>>>>>>>>>>> should be, is it an output maybe?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Ryan
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 10/25/18 12:27, Vijay S. Mahadevan wrote:
>>>>>>>>>>>>>> You can try either the first or the second variant instead [1] with
>>>>>>>>>>>>>> following arguments.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ErrorCode moab::ParallelComm::resolve_shared_ents(EntityHandle
>>>>>>>>>>>>>> this_set,
>>>>>>>>>>>>>> int resolve_dim = 2,
>>>>>>>>>>>>>> int shared_dim = -1,
>>>>>>>>>>>>>> const Tag * id_tag = 0
>>>>>>>>>>>>>> )
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> That should resolve 2-dim entities with shared edges across
>>>>>>>>>>>>>> partitions. You can leave id_tag pointer as zero since the
>>>>>>>>>>>>>> default is
>>>>>>>>>>>>>> GLOBAL_ID.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> This brings up a more general question I've had about the moab
>>>>>>>>>>>>>> documentation for a while. In the doc for this routine, it only
>>>>>>>>>>>>>> lists 2
>>>>>>>>>>>>>> parameters, proc_ents and shared_dim, even though in the function
>>>>>>>>>>>>>> signature above it clearly shows more. I've had trouble
>>>>>>>>>>>>>> understanding
>>>>>>>>>>>>>> which parameters are relevant in the past, or what they do
>>>>>>>>>>>>>> because I'm
>>>>>>>>>>>>>> not quite sure how to read the documentation.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> This is an oversight. We will go through and rectify some of the
>>>>>>>>>>>>>> inconsistencies in the documentation. We are preparing for an
>>>>>>>>>>>>>> upcoming
>>>>>>>>>>>>>> release and I'll make sure that routines in Core/ParallelComm have
>>>>>>>>>>>>>> updated documentation that match the interfaces. Meanwhile, if you
>>>>>>>>>>>>>> have questions, feel free to shoot an email to the list.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hope that helps.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Vijay
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>> http://ftp.mcs.anl.gov/pub/fathom/moab-docs-develop/classmoab_1_1ParallelComm.html#a29a3b834b3fc3b4ddb3a5d8a78a37c8a
>>>>>>>>>>>>>> On Thu, Oct 25, 2018 at 1:58 PM Ryan O'Kuinghttons
>>>>>>>>>>>>>> <ryan.okuinghttons at noaa.gov> wrote:
>>>>>>>>>>>>>>> Hi Vijay,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks again for that explanation.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> ESMF does use unique global ids for the vertices and set them
>>>>>>>>>>>>>>> to the
>>>>>>>>>>>>>>> GLOBAL_ID in the moab mesh. So I think we are good there.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I can't quite figure out how to use resolve_shared_entities
>>>>>>>>>>>>>>> though.
>>>>>>>>>>>>>>> There are three versions of the call in the documentation, and
>>>>>>>>>>>>>>> I assume
>>>>>>>>>>>>>>> that I should use the first and pass in a Range containing all
>>>>>>>>>>>>>>> entities
>>>>>>>>>>>>>>> for proc_ents:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> http://ftp.mcs.anl.gov/pub/fathom/moab-docs-develop/classmoab_1_1ParallelComm.html#a59e35d9906f2e33fe010138a144a5cb6
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> However, I'm not sure what the this_set EntityHandle should be.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> This brings up a more general question I've had about the moab
>>>>>>>>>>>>>>> documentation for a while. In the doc for this routine, it only
>>>>>>>>>>>>>>> lists 2
>>>>>>>>>>>>>>> parameters, proc_ents and shared_dim, even though in the function
>>>>>>>>>>>>>>> signature above it clearly shows more. I've had trouble
>>>>>>>>>>>>>>> understanding
>>>>>>>>>>>>>>> which parameters are relevant in the past, or what they do
>>>>>>>>>>>>>>> because I'm
>>>>>>>>>>>>>>> not quite sure how to read the documentation.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Any example or explanation you give me for
>>>>>>>>>>>>>>> resolve_shared_entities is
>>>>>>>>>>>>>>> much appreciated!
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Ryan
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 10/25/18 09:40, Vijay S. Mahadevan wrote:
>>>>>>>>>>>>>>>> Hi Ryan,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Glad the example helped.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> The example shows the ghost exchange happening between shared
>>>>>>>>>>>>>>>>> owned
>>>>>>>>>>>>>>>> entities. In my experiments I've been unable to make the ghost
>>>>>>>>>>>>>>>> exchange
>>>>>>>>>>>>>>>> work, and I think that might be because my entities are not
>>>>>>>>>>>>>>>> shared.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> You need to call resolve_shared_entities on the entities prior to
>>>>>>>>>>>>>>>> doing ghost exchange. When we load the mesh from file, this
>>>>>>>>>>>>>>>> happens
>>>>>>>>>>>>>>>> automatically based on the read options but if you are forming
>>>>>>>>>>>>>>>> a mesh
>>>>>>>>>>>>>>>> in memory, you need to make sure that the shared vertices have
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> same GLOBAL_ID numbering consistently across processes. That
>>>>>>>>>>>>>>>> is shared
>>>>>>>>>>>>>>>> vertices are unique. Once that is set, shared entities
>>>>>>>>>>>>>>>> resolution will
>>>>>>>>>>>>>>>> work correctly out of the box and you will have shared
>>>>>>>>>>>>>>>> edges/entities
>>>>>>>>>>>>>>>> query working correctly. A call to get ghosted layers once
>>>>>>>>>>>>>>>> this is
>>>>>>>>>>>>>>>> done would be the way to go.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I assume ESMF has a unique global numbering for the vertices ?
>>>>>>>>>>>>>>>> Use
>>>>>>>>>>>>>>>> that to set the GLOBAL_ID tag. Let us know if you are still
>>>>>>>>>>>>>>>> having
>>>>>>>>>>>>>>>> issues.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Vijay
>>>>>>>>>>>>>>>> On Thu, Oct 25, 2018 at 11:17 AM Ryan O'Kuinghttons
>>>>>>>>>>>>>>>> <ryan.okuinghttons at noaa.gov> wrote:
>>>>>>>>>>>>>>>>> Hi Vijay,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I've had some time to play with this now, and I have another
>>>>>>>>>>>>>>>>> question.
>>>>>>>>>>>>>>>>> Thank you for sending along the example code, it has been
>>>>>>>>>>>>>>>>> extremely
>>>>>>>>>>>>>>>>> helpful.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> The example shows the ghost exchange happening between shared
>>>>>>>>>>>>>>>>> owned
>>>>>>>>>>>>>>>>> entities. In my experiments I've been unable to make the
>>>>>>>>>>>>>>>>> ghost exchange
>>>>>>>>>>>>>>>>> work, and I think that might be because my entities are not
>>>>>>>>>>>>>>>>> shared. The
>>>>>>>>>>>>>>>>> situation I have is entities that are owned wholly on a single
>>>>>>>>>>>>>>>>> processor, which need to be communicated to other processors
>>>>>>>>>>>>>>>>> which
>>>>>>>>>>>>>>>>> require them as part of a halo region for mesh based
>>>>>>>>>>>>>>>>> computation. In
>>>>>>>>>>>>>>>>> this situation would I need to "share" my entities across the
>>>>>>>>>>>>>>>>> whole
>>>>>>>>>>>>>>>>> processor space, before requesting a ghost exchange? I'm not
>>>>>>>>>>>>>>>>> even really
>>>>>>>>>>>>>>>>> sure what shared entities mean, is there a good place to look
>>>>>>>>>>>>>>>>> in the
>>>>>>>>>>>>>>>>> documentation to learn more about the terminology? Thanks,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Ryan
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On 10/8/18 12:17, Vijay S. Mahadevan wrote:
>>>>>>>>>>>>>>>>>> Ryan,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> You need to use the ParallelComm object in MOAB to call
>>>>>>>>>>>>>>>>>> exchange_ghost_cells [1] with appropriate levels of ghost
>>>>>>>>>>>>>>>>>> layers for
>>>>>>>>>>>>>>>>>> your mesh. This needs to be done after the mesh is loaded,
>>>>>>>>>>>>>>>>>> or you can
>>>>>>>>>>>>>>>>>> pass this information also as part of the options when
>>>>>>>>>>>>>>>>>> loading a file
>>>>>>>>>>>>>>>>>> and MOAB will internally load the file with ghosted layers.
>>>>>>>>>>>>>>>>>> Here's an
>>>>>>>>>>>>>>>>>> example [2].
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Vijay
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>> http://ftp.mcs.anl.gov/pub/fathom/moab-docs-develop/classmoab_1_1ParallelComm.html#a55dfa308f56fd368319bfb4244428878
>>>>>>>>>>>>>>>>>> [2]
>>>>>>>>>>>>>>>>>> http://ftp.mcs.anl.gov/pub/fathom/moab-docs-develop/HelloParMOAB_8cpp-example.html
>>>>>>>>>>>>>>>>>> On Mon, Oct 8, 2018 at 1:23 PM Ryan O'Kuinghttons
>>>>>>>>>>>>>>>>>> <ryan.okuinghttons at noaa.gov> wrote:
>>>>>>>>>>>>>>>>>>> Hi, I am wondering if there is a way to create ghost
>>>>>>>>>>>>>>>>>>> elements in MOAB.
>>>>>>>>>>>>>>>>>>> By this I mean a list of copies of elements surrounding a
>>>>>>>>>>>>>>>>>>> specific MOAB
>>>>>>>>>>>>>>>>>>> element, ghost elements may exist on a different processor
>>>>>>>>>>>>>>>>>>> than the
>>>>>>>>>>>>>>>>>>> source element. I see a ghost_elems class in the appData
>>>>>>>>>>>>>>>>>>> namespace, but
>>>>>>>>>>>>>>>>>>> there is not much documentation on how to use it.. Thank you,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>> Ryan O'Kuinghttons
>>>>>>>>>>>>>>>>>>> Cherokee Nation Management and Consulting
>>>>>>>>>>>>>>>>>>> NESII/NOAA/Earth System Research Laboratory
>>>>>>>>>>>>>>>>>>> ryan.okuinghttons at noaa.gov
>>>>>>>>>>>>>>>>>>> https://www.esrl.noaa.gov/gsd/nesii/
>>>>>>>>>>>>>>>>>>>
More information about the moab-dev
mailing list