[MOAB-dev] ghost/halo elements
Ryan O'Kuinghttons
ryan.okuinghttons at noaa.gov
Tue Oct 30 12:32:27 CDT 2018
Hi Iulian,
First, I pushed this code with a reproducer to a remote branch. You can
clone esmf here: (read only)
git clone git://git.code.sf.net/p/esmf/esmf esmf
and the branch is called moab4.
Building ESMF for this case should only require mpi and a fortran/c
compiler. Set the following environment variables and type make at the
top level esmf directory:
ESMF_DIR=<path to top level esmf dir>
ESMF_COMPILER=gfortran
ESMF_COMM=mpich3
ESMF_BOPT=g
If the build goes awry, please send me the output of 'make info'. Then
once that is built, the reproducer is here:
esmf/src/Infrastructure/Mesh/tests/ESMC_MeshMOABGhostUTest.C
you can build it by typing make in the tests directory, and run it with:
mpirun -n 2 $ESMF_DIR/test/*/*/ESMC_MeshMOABGhostUTest
-----------------------------------------------------------------------------------
As to your questions:
- Yes, the elem_coords tag is what stores the coordinates of an ESMF
MBMesh element, this is what we need to access on the ghost elements
- It is actually the elem_coords that are not exchanged in this case.
The vertex coordinates are in a different tag (node_coords) and those
ones go through the exchange_tags just fine.
- I don't know what you mean by 'instanced'.. but before the ghost
exchange is called on the mesh it is a valid mesh and the vtk file is
created correctly. after the ghost exchange most, but not all, of the
shared cells have 0 values for all tags which have not been exchanged
yet. and then during the tag exchange we get the segv when trying to
operate on the elem_coords tag.
- I will need to check further to see if we are handling adjacencies in
the way that you describe. I will do that now.
FYI, I have an appointment now, I will bring my laptop and keep looking
at this as I am able, should be back online in about an hour.
Ryan
On 10/30/18 10:25, Grindeanu, Iulian R. wrote:
> Hi Ryan,
> So you are using this to exchange actual coordinates?
> so the vertex coordinates of the nodes that are not "exchanged" yet are 0?
> When the mesh is instanced on each task, is it a valid "mesh", or not yet? I understand that some coordinates could be 0 (not defined?), but the rest of the vtk file should be "valid"
> One issue is that we do use adjacency information to build up the ghosting layer; adjacency should work fine even if the coordinates of the nodes are wrong, as long as the connectivity information is correct; For the adjacency logic to work, the vertex adjacencies have to be defined/set correctly;
> If you create an element/a cell somewhere in the middle, make sure that the "adjacency" information is updated for the affected vertices;
>
> for example, if you use
> ReadUtilIface::get_element_connect to create elements, you need to also call ReadUtilIface::update_adjacencies
>
> What entities are you passing to tag exchange, are they including cells, edges, and/or vertices ?
> Is this tag defined on vertices and cells too, or just on vertices? Does it make sense to be created on cells? Maybe the center coordinate of cell?
>
> Iulian
>
> ________________________________________
> From: Ryan O'Kuinghttons <ryan.okuinghttons at noaa.gov>
> Sent: Tuesday, October 30, 2018 9:27:33 AM
> To: Grindeanu, Iulian R.; Vijay S. Mahadevan
> Cc: moab-dev at mcs.anl.gov
> Subject: Re: [MOAB-dev] ghost/halo elements
>
> Hi Iulian,
>
> I'll answer your questions one at a time:
>
> - sdim is 2 in this test, it the simple 2D mesh in the ascii drawing in
> the last mail
>
> - Yes, elem_coords is a tag with 2 doubles per entity
>
> - Yes, merr is checked after both tag creation and tag set, both are fine
>
> - The MB_ALREADY_ALLOCATED comes from the moab error code return after
> exchange_tags, but only when the elem_coords tag is in the vector of
> tags to exchange
>
> - I have written out the vtk files already, the tags that are not
> exchanged (elem_coords) come back with value of 0, so the mesh has a
> bunch of connections that go back to (0,0)
>
> - elemCoords+mbmp->sdim*e is a reference to the e'th element in the
> elemCoords array of coordinate values. This code is widely tested in a
> variety of scenarios and passes everywhere. Additionally, the
> elem_coords tag is created and set as part of the original mesh creation
> in my test code, and writing out the mesh before the ghost_exchange
> shows a healthy looking mesh.
>
> I'm a bit hesitant to ask you to look at the esmf moab code directly
> because it requires building esmf, which can be a hassle. However, if
> you think that is the best way to go, I will send instructions on where
> to get the code and how to build.
>
> Ryan
>
>
> On 10/29/18 19:54, Grindeanu, Iulian R. wrote:
>> Hi Ryan,
>> Yes, nothing comes out right away; it would be easier if I could debug this on my computer;
>> what is sdim? Is it 2? So is this a tag with 2 doubles per entity?
>> did you check merr after tag creaton?
>> If you use MB_TAG_EXCL , it means the tag should not have been created before;
>>
>> When did you get MB_ALREADY_ALLOCATED? after you called tag exchange? I did not see in output, maybe cerr is redirected?
>>
>> You can try to write out the file (in serial), from each task; to check that ghosting did what you expect:
>> something like this:
>>
>> std::ostringstream ent_str;
>> ent_str << "mesh." <<pcomm->rank() << ".vtk";
>> moab_mesh->write_mesh(ent_str.str().c_str());
>>
>>
>> what is elemCoords+mbmp->sdim*e in your code? I assume you are using some pointer arithmetic, are all those variables OK?
>> Is elemCoords a double pointer, and e an index ?
>>
>> Iulian
>>
>> ________________________________________
>> From: Ryan O'Kuinghttons <ryan.okuinghttons at noaa.gov>
>> Sent: Monday, October 29, 2018 4:57:43 PM
>> To: Grindeanu, Iulian R.; Vijay S. Mahadevan
>> Cc: moab-dev at mcs.anl.gov
>> Subject: Re: [MOAB-dev] ghost/halo elements
>>
>> Hi Iulian, I know this isn't a clean or easy problem, I really
>> appreciate you looking into it with me.
>>
>> The tag is created like this:
>>
>> merr=moab_mesh->tag_get_handle("elem_coords", mbmp->sdim,
>> MB_TYPE_DOUBLE, mbmp->elem_coords_tag, MB_TAG_EXCL|MB_TAG_DENSE,
>> &dbl_def_val);
>>
>> and it is set like this:
>>
>> merr=moab_mesh->tag_set_data(mbmp->elem_coords_tag, &new_elem, 1,
>> elemCoords+mbmp->sdim*e);
>>
>>
>> This is a small test mesh, distributed like this:
>>
>> //
>> // 3.0 13 ------ 14 ------ 15 [15] ----------- 16
>> // | | | | |
>> // | | | | |
>> // | 8 | 9 | | 10 |
>> // | | | | |
>> // | | | | |
>> // 2.0 [9] ----- [10] ---- [11] [11] ---------- [12]
>> //
>> // 1.0 1.5 2.0 2.0 3.0
>> //
>> // PET 2 PET 3
>> //
>> //
>> // 2.0 9 ------- 10 ------ 11 [11] ----------- 12
>> // | | | | |
>> // | 5 | 6 | | 7 |
>> // | | | | |
>> // 1.5 5 ------- 6 ------- 7 [7] ----------- 8
>> // | | | | |
>> // | 1 | 2 | | 4 |
>> // | | | | |
>> // 1.0 1 ------- 2 ------- 3 [3] ------------ 4
>> //
>> // 1.0 1.5 2.0 2.0 3.0
>> //
>> // PET 0 PET 1
>> //
>> // Node Id labels at corners
>> // Element Id labels in centers
>>
>> It would be difficult to create a small reproducer for you, given that
>> I'm working on a development branch, but I could probably send you a
>> tarball if we get to that. Likewise, it would be some work to create a
>> similar test on 2 processors, but not out of the question.
>>
>> Is there anything that comes to mind with the above information that I
>> could look into first? Is it possible that this is duplicating a default
>> tag in moab? (given the message MB_ALREADY_ALLOCATED)..
>>
>>
>> On 10/29/18 15:31, Grindeanu, Iulian R. wrote:
>>> Sorry, I still don't know what is wrong.
>>> What kind of tag is it? is it fixed length, does it have a default value?
>>> How was this tag created/set ?
>>>
>>> the sizes of the buffers sent seem to be small, is this a small test mesh? how is this partitioned?
>>> Can you replicate the bug in a way I can run on my machine? I mean, I do have an old version of esmf. I assume this is on newer code. Or can you replicate on smaller number of tasks (maybe 2?)
>>>
>>> Iulian
>>>
>>> ________________________________________
>>> From: Ryan O'Kuinghttons <ryan.okuinghttons at noaa.gov>
>>> Sent: Monday, October 29, 2018 4:10:16 PM
>>> To: Grindeanu, Iulian R.; Vijay S. Mahadevan
>>> Cc: moab-dev at mcs.anl.gov
>>> Subject: Re: [MOAB-dev] ghost/halo elements
>>>
>>> 0 ParallelComm(0.03 s) Entering exchange_tags
>>> 3 ParallelComm(0.03 s) Entering exchange_tags
>>> 1 ParallelComm(0.03 s) Entering exchange_tags
>>> 2 ParallelComm(0.03 s) Entering exchange_tags
>>> 3 ParallelComm(0.03 s) Irecv, 1<-3, buffer ptr = 0x13a5a20, tag=7,
>>> size=1024, incoming=1
>>> 1 ParallelComm(0.03 s) Irecv, 0<-1, buffer ptr = 0x1115b30, tag=7,
>>> size=1024, incoming=1
>>> 0 ParallelComm(0.03 s) Irecv, 1<-0, buffer ptr = 0x11cc7e0, tag=7,
>>> size=1024, incoming=1
>>> 1 ParallelComm(0.03 s) Irecv, 2<-1, buffer ptr = 0x17a4330, tag=7,
>>> size=1024, incoming=2
>>> 0 ParallelComm(0.03 s) Irecv, 2<-0, buffer ptr = 0x11cdf60, tag=7,
>>> size=1024, incoming=2
>>> 0 ParallelComm(0.03 s) Irecv, 3<-0, buffer ptr = 0xb4b9e0, tag=7,
>>> size=1024, incoming=3
>>> 3 ParallelComm(0.03 s) Irecv, 2<-3, buffer ptr = 0x13a6360, tag=7,
>>> size=1024, incoming=2
>>> 3 ParallelComm(0.03 s) Irecv, 0<-3, buffer ptr = 0xd23290, tag=7,
>>> size=1024, incoming=3
>>> 1 ParallelComm(0.03 s) Irecv, 3<-1, buffer ptr = 0x11135a0, tag=7,
>>> size=1024, incoming=3
>>> 2 ParallelComm(0.03 s) Irecv, 0<-2, buffer ptr = 0x16505f0, tag=7,
>>> size=1024, incoming=1
>>> 2 ParallelComm(0.03 s) Irecv, 1<-2, buffer ptr = 0x2362770, tag=7,
>>> size=1024, incoming=2
>>> 2 ParallelComm(0.03 s) Irecv, 3<-2, buffer ptr = 0x1651720, tag=7,
>>> size=1024, incoming=3
>>> 1 ParallelComm(0.03 s) Packing tag "elem_coords"(0.03 s)
>>> 0 ParallelComm(0.03 s) Packing tag "elem_coords"(0.03 s)
>>> 3 ParallelComm(0.03 s) Packing tag "elem_coords"(0.03 s)
>>> 2 ParallelComm(0.03 s) Packing tag "elem_coords"(0.03 s)
>>> 1 ParallelComm(0.03 s) Done packing tags.(0.03 s) Isend, 1->0,
>>> buffer ptr = 0x17a5500, tag=7, size=83
>>> 0 ParallelComm(0.03 s) Done packing tags.(0.03 s) Isend, 0->1,
>>> buffer ptr = 0x11ceee0, tag=7, size=83
>>> 3 ParallelComm(0.03 s) Done packing tags.(0.03 s) Isend, 3->1,
>>> buffer ptr = 0x1a27200, tag=7, size=83
>>> 2 ParallelComm(0.03 s) Done packing tags.(0.03 s) Isend, 2->0,
>>> buffer ptr = 0x2363980, tag=7, size=107
>>> 0 ParallelComm(0.03 s) Packing tag "elem_coords"(0.03 s)
>>> 3 ParallelComm(0.03 s) Packing tag "elem_coords"(0.03 s)
>>> 1 ParallelComm(0.03 s) Packing tag "elem_coords"(0.03 s)
>>> 2 ParallelComm(0.03 s) Packing tag "elem_coords"(0.03 s)
>>> 0 ParallelComm(0.03 s) Done packing tags.(0.03 s) Isend, 0->2,
>>> buffer ptr = 0xb4a0d0, tag=7, size=83
>>> 3 ParallelComm(0.03 s) Done packing tags.(0.03 s) Isend, 3->2,
>>> buffer ptr = 0xd22a20, tag=7, size=131
>>> 1 ParallelComm(0.03 s) Done packing tags.(0.03 s) Isend, 1->2,
>>> buffer ptr = 0x1114230, tag=7, size=107
>>> 2 ParallelComm(0.03 s) Done packing tags.(0.03 s) Isend, 2->1,
>>> buffer ptr = 0x1652330, tag=7, size=107
>>> 0 ParallelComm(0.03 s) Packing tag "elem_coords"(0.03 s)
>>> 1 ParallelComm(0.03 s) Packing tag "elem_coords"(0.03 s)
>>> 3 ParallelComm(0.03 s) Packing tag "elem_coords"(0.03 s)
>>> 1 ParallelComm(0.03 s) Done packing tags.(0.03 s) Isend, 1->3,
>>> buffer ptr = 0x11163b0, tag=7, size=107
>>> 0 ParallelComm(0.03 s) Done packing tags.(0.03 s) Isend, 0->3,
>>> buffer ptr = 0xb4bdf0, tag=7, size=59
>>> 3 ParallelComm(0.03 s) Done packing tags.(0.03 s) Isend, 3->0,
>>> buffer ptr = 0xd22610, tag=7, size=59
>>> 0 ParallelComm(0.03 s) Unpacking tag elem_coords
>>> 1 ParallelComm(0.03 s) Unpacking tag elem_coords
>>> 2 ParallelComm(0.03 s) Packing tag "elem_coords"(0.03 s)
>>> 3 ParallelComm(0.03 s) Unpacking tag elem_coords
>>> 2 ParallelComm(0.03 s) Done packing tags.(0.03 s) Isend, 2->3,
>>> buffer ptr = 0x1652bd0, tag=7, size=107
>>> 2 ParallelComm(0.03 s) Unpacking tag elem_coords
>>> [3]MOAB ERROR: --------------------- Error Message
>>> ------------------------------------
>>> [0]MOAB ERROR: --------------------- Error Message
>>> ------------------------------------
>>> [0]MOAB ERROR: Failed to recv-unpack-tag message!
>>> [1]MOAB ERROR: --------------------- Error Message
>>> ------------------------------------
>>> [1]MOAB ERROR: Failed to recv-unpack-tag message!
>>> [3]MOAB ERROR: Failed to recv-unpack-tag message!
>>> [3]MOAB ERROR: exchange_tags() line 7414 in ParallelComm.cpp
>>> [0]MOAB ERROR: exchange_tags() line 7414 in ParallelComm.cpp
>>> [1]MOAB ERROR: exchange_tags() line 7414 in ParallelComm.cpp
>>> [2]MOAB ERROR: --------------------- Error Message
>>> ------------------------------------
>>> [2]MOAB ERROR: Failed to recv-unpack-tag message!
>>> [2]MOAB ERROR: exchange_tags() line 7414 in ParallelComm.cpp
>>> terminate called after throwing an instance of 'int'
>>> terminate called after throwing an instance of 'int'
>>> terminate called after throwing an instance of 'int'
>>> terminate called after throwing an instance of 'int'
>>>
>>> Program received signal SIGABRT: Process abort signal.
>>>
>>> On 10/29/18 15:02, Grindeanu, Iulian R. wrote:
>>>> Hi Ryan,
>>>> I don't know what could be wrong; can you set verbosity level in your code?
>>>> I mean, can you use something like
>>>>
>>>> pcomm->set_debug_verbosity(4)
>>>>
>>>> before calling exchange tags? And send me the output?
>>>>
>>>> I don't think MB_ALREADY_ALLOCATED is a problem in this case; I could be wrong
>>>>
>>>> Iulian
>>>> ________________________________________
>>>> From: Ryan O'Kuinghttons <ryan.okuinghttons at noaa.gov>
>>>> Sent: Monday, October 29, 2018 3:44:01 PM
>>>> To: Grindeanu, Iulian R.; Vijay S. Mahadevan
>>>> Cc: moab-dev at mcs.anl.gov
>>>> Subject: Re: [MOAB-dev] ghost/halo elements
>>>>
>>>> Hi Iulian, thanks for responding :)
>>>>
>>>> So I did get the exchange_tags call to work for all but one of our tags.
>>>> However, when I add the last tag to the vector that is passed into
>>>> exchange_tags I get the following segfault:
>>>>
>>>> [3]MOAB ERROR: --------------------- Error Message
>>>> ------------------------------------
>>>> [3]MOAB ERROR: Failed to recv-unpack-tag message!
>>>> [3]MOAB ERROR: exchange_tags() line 7414 in ParallelComm.cpp
>>>> terminate called after throwing an instance of 'int'
>>>>
>>>> Program received signal SIGABRT: Process abort signal.
>>>>
>>>> Backtrace for this error:
>>>> [0]MOAB ERROR: --------------------- Error Message
>>>> ------------------------------------
>>>> [0]MOAB ERROR: Failed to recv-unpack-tag message!
>>>> [1]MOAB ERROR: --------------------- Error Message
>>>> ------------------------------------
>>>> [2]MOAB ERROR: --------------------- Error Message
>>>> ------------------------------------
>>>> [2]MOAB ERROR: Failed to recv-unpack-tag message!
>>>> [2]MOAB ERROR: exchange_tags() line 7414 in ParallelComm.cpp
>>>> [0]MOAB ERROR: exchange_tags() line 7414 in ParallelComm.cpp
>>>> [1]MOAB ERROR: Failed to recv-unpack-tag message!
>>>> [1]MOAB ERROR: exchange_tags() line 7414 in ParallelComm.cpp
>>>> terminate called after throwing an instance of 'int'
>>>> terminate called after throwing an instance of 'int'
>>>> terminate called after throwing an instance of 'int'
>>>>
>>>>
>>>>
>>>> Do you have any ideas what I could be doing wrong?
>>>>
>>>> Ryan
>>>>
>>>> On 10/29/18 12:52, Grindeanu, Iulian R. wrote:
>>>>> Hi Ryan,
>>>>> Sorry for missing the messages, my mistake;
>>>>> If you have your tags defined on your entities, you will need to call the "exchange_tags" method, after your ghost method;
>>>>> (call this one:
>>>>> ErrorCode ParallelComm::exchange_tags(const std::vector<Tag> &src_tags,
>>>>> const std::vector<Tag> &dst_tags,
>>>>> const Range &entities_in)
>>>>> usually, source and dest tags are the same lists)
>>>>>
>>>>>
>>>>> the augment method is for "membership" to specific moab sets (material, boundary conditions, partition );
>>>>>
>>>>> Entities (cells, usually), do not have a tag defined on them that would signal they are part of those sets, this is why we had to transfer that information in a different way.
>>>>>
>>>>> What is exactly your use case? Do you need to transfer information about a set or about a tag ?
>>>>>
>>>>> Iulian
>>>>>
>>>>> ________________________________________
>>>>> From: Vijay S. Mahadevan <vijay.m at gmail.com>
>>>>> Sent: Monday, October 29, 2018 1:05:53 PM
>>>>> To: Ryan OKuinghttons - NOAA Affiliate
>>>>> Cc: Grindeanu, Iulian R.; moab-dev at mcs.anl.gov
>>>>> Subject: Re: [MOAB-dev] ghost/halo elements
>>>>>
>>>>> Hi Ryan,
>>>>>
>>>>> The method `augment_default_sets_with_ghosts` [1] in ParallelComm
>>>>> enables this for some of the default tags. We haven't exposed this as
>>>>> a very general mechanism as the user could define the tag data layout
>>>>> in many possible ways and it would be difficult to cover all use cases
>>>>> from a library standpoint. Remember that for custom defined Tags, you
>>>>> can also use exchange_tag/reduce_tags in ParallelComm to get the data
>>>>> that you need on shared entities. If exchange_tags does not suit your
>>>>> needs, you can see if you can adapt augment_default_sets_with_ghosts
>>>>> method for your use case. Let me know if that helps.
>>>>>
>>>>> Vijay
>>>>>
>>>>> [1] http://ftp.mcs.anl.gov/pub/fathom/moab-docs-develop/classmoab_1_1ParallelComm.html#a76dced0b6dbe2fb99340945c7ca3c0f2
>>>>> On Mon, Oct 29, 2018 at 1:49 PM Ryan O'Kuinghttons
>>>>> <ryan.okuinghttons at noaa.gov> wrote:
>>>>>> Hi Iulian, Vijay,
>>>>>>
>>>>>> We have a set of custom tags that we use in ESMF, would it be possible
>>>>>> to transfer these as well?
>>>>>>
>>>>>> Ryan
>>>>>>
>>>>>> On 10/29/18 10:54, Vijay S. Mahadevan wrote:
>>>>>>> The default set of tags do get transferred. These would be the
>>>>>>> MATERIAL_SET, DIRICHLET/NEUMANN sets etc.
>>>>>>>
>>>>>>> Iulian, did we add a separate method to transfer these to the ghosted
>>>>>>> entities ? I vaguely remember something like that.
>>>>>>>
>>>>>>> Vijay
>>>>>>> On Mon, Oct 29, 2018 at 12:42 PM Ryan O'Kuinghttons
>>>>>>> <ryan.okuinghttons at noaa.gov> wrote:
>>>>>>>> Hi Vijay,
>>>>>>>>
>>>>>>>> I ran into another issue with the call to exchange_ghost_cells, but this
>>>>>>>> is probably just a misunderstanding on my part. I find that the tag
>>>>>>>> information that is part of the mesh before the the call does not
>>>>>>>> transfer to the newly created faces. Is there some way to ask moab to
>>>>>>>> transfer tags when creating ghost entities? (FWIW, I could not find this
>>>>>>>> in the example you sent me) Thanks,
>>>>>>>>
>>>>>>>> Ryan
>>>>>>>>
>>>>>>>> On 10/26/18 07:36, Ryan O'Kuinghttons wrote:
>>>>>>>>> Hi Vijay,
>>>>>>>>>
>>>>>>>>> I resolved the issue with not getting any shared, owned entities back
>>>>>>>>> by using the other version of resolve_shared_ents, as demonstrated in
>>>>>>>>> the example you indicated. I created a Range containing all elements
>>>>>>>>> (faces for moab), and used that as input.
>>>>>>>>>
>>>>>>>>> Range range_ent;
>>>>>>>>> merr=mb->get_entities_by_dimension(0,this->sdim,range_ent);
>>>>>>>>> MBMESH_CHECK_ERR(merr, localrc);
>>>>>>>>>
>>>>>>>>> merr = pcomm->resolve_shared_ents(0, range_ent, this->sdim, 1);
>>>>>>>>> MBMESH_CHECK_ERR(merr, localrc);
>>>>>>>>>
>>>>>>>>> The first version of the interface (which does not use a Range input)
>>>>>>>>> does not work for me:
>>>>>>>>>
>>>>>>>>> merr = pcomm->resolve_shared_ents(0, this->sdim, 1);
>>>>>>>>>
>>>>>>>>> Otherwise, everything else seems to be resolved with this issue,
>>>>>>>>> thanks again for all of your help!!
>>>>>>>>>
>>>>>>>>> Ryan
>>>>>>>>>
>>>>>>>>> On 10/25/18 15:13, Vijay S. Mahadevan wrote:
>>>>>>>>>> Ryan, I don't see anything obviously wrong in the code below. Does
>>>>>>>>>> your shared_ents not have any entities after parallel resolve ? If you
>>>>>>>>>> can replicate your issue in a small standalone test, we should be able
>>>>>>>>>> to quickly find out what's going wrong.
>>>>>>>>>>
>>>>>>>>>> Look at the test_reduce_tag_explicit_dest routine in
>>>>>>>>>> test/parallel/parallel_unit_tests.cpp:1617. It has the workflow where
>>>>>>>>>> you create a mesh in memory, resolve shared entities, get shared
>>>>>>>>>> entities and even set/reduce tag data.
>>>>>>>>>>
>>>>>>>>>> Vijay
>>>>>>>>>> On Thu, Oct 25, 2018 at 4:25 PM Ryan O'Kuinghttons
>>>>>>>>>> <ryan.okuinghttons at noaa.gov> wrote:
>>>>>>>>>>> OK, that allows the resolve_shared_ents routine to complete without
>>>>>>>>>>> error, but I still don't get any shared, owned entities back from the
>>>>>>>>>>> get_shared_entities/filter_pstatus calls:
>>>>>>>>>>>
>>>>>>>>>>> Range shared_ents;
>>>>>>>>>>> // Get entities shared with all other processors
>>>>>>>>>>> merr = pcomm->get_shared_entities(-1, shared_ents);
>>>>>>>>>>> MBMESH_CHECK_ERR(merr, localrc);
>>>>>>>>>>>
>>>>>>>>>>> // Filter shared entities with not not_owned, which means owned
>>>>>>>>>>> Range owned_entities;
>>>>>>>>>>> merr = pcomm->filter_pstatus(shared_ents, PSTATUS_NOT_OWNED,
>>>>>>>>>>> PSTATUS_NOT, -1, &owned_entities);
>>>>>>>>>>> MBMESH_CHECK_ERR(merr, localrc);
>>>>>>>>>>>
>>>>>>>>>>> unsigned int nums[4] = {0}; // to store the owned entities per
>>>>>>>>>>> dimension
>>>>>>>>>>> for (int i = 0; i < 4; i++)
>>>>>>>>>>> nums[i] = (int)owned_entities.num_of_dimension(i);
>>>>>>>>>>>
>>>>>>>>>>> vector<int> rbuf(nprocs*4, 0);
>>>>>>>>>>> MPI_Gather(nums, 4, MPI_INT, &rbuf[0], 4, MPI_INT, 0, mpi_comm);
>>>>>>>>>>> // Print the stats gathered:
>>>>>>>>>>> if (0 == rank) {
>>>>>>>>>>> for (int i = 0; i < nprocs; i++)
>>>>>>>>>>> cout << " Shared, owned entities on proc " << i << ": " <<
>>>>>>>>>>> rbuf[4*i] << " verts, " <<
>>>>>>>>>>> rbuf[4*i + 1] << " edges, " << rbuf[4*i + 2] << " faces,
>>>>>>>>>>> " <<
>>>>>>>>>>> rbuf[4*i + 3] << " elements" << endl;
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>> OUTPUT:
>>>>>>>>>>>
>>>>>>>>>>> Shared, owned entities on proc 0: 0 verts, 0 edges, 0 faces, 0
>>>>>>>>>>> elements
>>>>>>>>>>> Shared, owned entities on proc 1: 0 verts, 0 edges, 0 faces, 0
>>>>>>>>>>> elements
>>>>>>>>>>> Shared, owned entities on proc 2: 0 verts, 0 edges, 0 faces, 0
>>>>>>>>>>> elements
>>>>>>>>>>> Shared, owned entities on proc 3: 0 verts, 0 edges, 0 faces, 0
>>>>>>>>>>> elements
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Note: I also tried changing the -1 to 2 in the
>>>>>>>>>>> get_shared_entities/filter_pstatus calls to only retrieve elements.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On 10/25/18 13:57, Vijay S. Mahadevan wrote:
>>>>>>>>>>>> No problem. Can you try 1 for the last argument to look for edges as
>>>>>>>>>>>> the shared_dim.
>>>>>>>>>>>>
>>>>>>>>>>>> Vijay
>>>>>>>>>>>> On Thu, Oct 25, 2018 at 3:41 PM Ryan O'Kuinghttons
>>>>>>>>>>>> <ryan.okuinghttons at noaa.gov> wrote:
>>>>>>>>>>>>> OK, thanks again for that explanation, I never would have been
>>>>>>>>>>>>> able to figure out that this_set was the root set. I apologize for
>>>>>>>>>>>>> all the dumb questions, hopefully they are all getting me closer
>>>>>>>>>>>>> to self sufficiency..
>>>>>>>>>>>>>
>>>>>>>>>>>>> I'm getting an error that doesn't make sense to me. I specifying
>>>>>>>>>>>>> the resolve_dim and shared_dim in the call:
>>>>>>>>>>>>>
>>>>>>>>>>>>> // Get the ParallelComm instance
>>>>>>>>>>>>> ParallelComm* pcomm = new ParallelComm(mb, mpi_comm);
>>>>>>>>>>>>> int nprocs = pcomm->proc_config().proc_size();
>>>>>>>>>>>>> int rank = pcomm->proc_config().proc_rank();
>>>>>>>>>>>>>
>>>>>>>>>>>>> // get the root set handle
>>>>>>>>>>>>> EntityHandle root_set = mb->get_root_set();
>>>>>>>>>>>>>
>>>>>>>>>>>>> merr = pcomm->resolve_shared_ents(root_set, 2, -1);
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> but I'm getting this message:
>>>>>>>>>>>>>
>>>>>>>>>>>>> [0]MOAB ERROR: Unable to guess shared_dim or resolve_dim!
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> what am i missing?
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 10/25/18 12:51, Vijay S. Mahadevan wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> It is your root set or fileset if you added entities to it. You
>>>>>>>>>>>>> can get the root set handle from Core object.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Vijay
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Thu, Oct 25, 2018, 14:38 Ryan O'Kuinghttons
>>>>>>>>>>>>> <ryan.okuinghttons at noaa.gov> wrote:
>>>>>>>>>>>>>> Thanks again, Vijay. However, I still don't understand what this_set
>>>>>>>>>>>>>> should be, is it an output maybe?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Ryan
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 10/25/18 12:27, Vijay S. Mahadevan wrote:
>>>>>>>>>>>>>>> You can try either the first or the second variant instead [1] with
>>>>>>>>>>>>>>> following arguments.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> ErrorCode moab::ParallelComm::resolve_shared_ents(EntityHandle
>>>>>>>>>>>>>>> this_set,
>>>>>>>>>>>>>>> int resolve_dim = 2,
>>>>>>>>>>>>>>> int shared_dim = -1,
>>>>>>>>>>>>>>> const Tag * id_tag = 0
>>>>>>>>>>>>>>> )
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> That should resolve 2-dim entities with shared edges across
>>>>>>>>>>>>>>> partitions. You can leave id_tag pointer as zero since the
>>>>>>>>>>>>>>> default is
>>>>>>>>>>>>>>> GLOBAL_ID.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> This brings up a more general question I've had about the moab
>>>>>>>>>>>>>>> documentation for a while. In the doc for this routine, it only
>>>>>>>>>>>>>>> lists 2
>>>>>>>>>>>>>>> parameters, proc_ents and shared_dim, even though in the function
>>>>>>>>>>>>>>> signature above it clearly shows more. I've had trouble
>>>>>>>>>>>>>>> understanding
>>>>>>>>>>>>>>> which parameters are relevant in the past, or what they do
>>>>>>>>>>>>>>> because I'm
>>>>>>>>>>>>>>> not quite sure how to read the documentation.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> This is an oversight. We will go through and rectify some of the
>>>>>>>>>>>>>>> inconsistencies in the documentation. We are preparing for an
>>>>>>>>>>>>>>> upcoming
>>>>>>>>>>>>>>> release and I'll make sure that routines in Core/ParallelComm have
>>>>>>>>>>>>>>> updated documentation that match the interfaces. Meanwhile, if you
>>>>>>>>>>>>>>> have questions, feel free to shoot an email to the list.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hope that helps.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Vijay
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>> http://ftp.mcs.anl.gov/pub/fathom/moab-docs-develop/classmoab_1_1ParallelComm.html#a29a3b834b3fc3b4ddb3a5d8a78a37c8a
>>>>>>>>>>>>>>> On Thu, Oct 25, 2018 at 1:58 PM Ryan O'Kuinghttons
>>>>>>>>>>>>>>> <ryan.okuinghttons at noaa.gov> wrote:
>>>>>>>>>>>>>>>> Hi Vijay,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks again for that explanation.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> ESMF does use unique global ids for the vertices and set them
>>>>>>>>>>>>>>>> to the
>>>>>>>>>>>>>>>> GLOBAL_ID in the moab mesh. So I think we are good there.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I can't quite figure out how to use resolve_shared_entities
>>>>>>>>>>>>>>>> though.
>>>>>>>>>>>>>>>> There are three versions of the call in the documentation, and
>>>>>>>>>>>>>>>> I assume
>>>>>>>>>>>>>>>> that I should use the first and pass in a Range containing all
>>>>>>>>>>>>>>>> entities
>>>>>>>>>>>>>>>> for proc_ents:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> http://ftp.mcs.anl.gov/pub/fathom/moab-docs-develop/classmoab_1_1ParallelComm.html#a59e35d9906f2e33fe010138a144a5cb6
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> However, I'm not sure what the this_set EntityHandle should be.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> This brings up a more general question I've had about the moab
>>>>>>>>>>>>>>>> documentation for a while. In the doc for this routine, it only
>>>>>>>>>>>>>>>> lists 2
>>>>>>>>>>>>>>>> parameters, proc_ents and shared_dim, even though in the function
>>>>>>>>>>>>>>>> signature above it clearly shows more. I've had trouble
>>>>>>>>>>>>>>>> understanding
>>>>>>>>>>>>>>>> which parameters are relevant in the past, or what they do
>>>>>>>>>>>>>>>> because I'm
>>>>>>>>>>>>>>>> not quite sure how to read the documentation.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Any example or explanation you give me for
>>>>>>>>>>>>>>>> resolve_shared_entities is
>>>>>>>>>>>>>>>> much appreciated!
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Ryan
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On 10/25/18 09:40, Vijay S. Mahadevan wrote:
>>>>>>>>>>>>>>>>> Hi Ryan,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Glad the example helped.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> The example shows the ghost exchange happening between shared
>>>>>>>>>>>>>>>>>> owned
>>>>>>>>>>>>>>>>> entities. In my experiments I've been unable to make the ghost
>>>>>>>>>>>>>>>>> exchange
>>>>>>>>>>>>>>>>> work, and I think that might be because my entities are not
>>>>>>>>>>>>>>>>> shared.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> You need to call resolve_shared_entities on the entities prior to
>>>>>>>>>>>>>>>>> doing ghost exchange. When we load the mesh from file, this
>>>>>>>>>>>>>>>>> happens
>>>>>>>>>>>>>>>>> automatically based on the read options but if you are forming
>>>>>>>>>>>>>>>>> a mesh
>>>>>>>>>>>>>>>>> in memory, you need to make sure that the shared vertices have
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> same GLOBAL_ID numbering consistently across processes. That
>>>>>>>>>>>>>>>>> is shared
>>>>>>>>>>>>>>>>> vertices are unique. Once that is set, shared entities
>>>>>>>>>>>>>>>>> resolution will
>>>>>>>>>>>>>>>>> work correctly out of the box and you will have shared
>>>>>>>>>>>>>>>>> edges/entities
>>>>>>>>>>>>>>>>> query working correctly. A call to get ghosted layers once
>>>>>>>>>>>>>>>>> this is
>>>>>>>>>>>>>>>>> done would be the way to go.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I assume ESMF has a unique global numbering for the vertices ?
>>>>>>>>>>>>>>>>> Use
>>>>>>>>>>>>>>>>> that to set the GLOBAL_ID tag. Let us know if you are still
>>>>>>>>>>>>>>>>> having
>>>>>>>>>>>>>>>>> issues.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Vijay
>>>>>>>>>>>>>>>>> On Thu, Oct 25, 2018 at 11:17 AM Ryan O'Kuinghttons
>>>>>>>>>>>>>>>>> <ryan.okuinghttons at noaa.gov> wrote:
>>>>>>>>>>>>>>>>>> Hi Vijay,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I've had some time to play with this now, and I have another
>>>>>>>>>>>>>>>>>> question.
>>>>>>>>>>>>>>>>>> Thank you for sending along the example code, it has been
>>>>>>>>>>>>>>>>>> extremely
>>>>>>>>>>>>>>>>>> helpful.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> The example shows the ghost exchange happening between shared
>>>>>>>>>>>>>>>>>> owned
>>>>>>>>>>>>>>>>>> entities. In my experiments I've been unable to make the
>>>>>>>>>>>>>>>>>> ghost exchange
>>>>>>>>>>>>>>>>>> work, and I think that might be because my entities are not
>>>>>>>>>>>>>>>>>> shared. The
>>>>>>>>>>>>>>>>>> situation I have is entities that are owned wholly on a single
>>>>>>>>>>>>>>>>>> processor, which need to be communicated to other processors
>>>>>>>>>>>>>>>>>> which
>>>>>>>>>>>>>>>>>> require them as part of a halo region for mesh based
>>>>>>>>>>>>>>>>>> computation. In
>>>>>>>>>>>>>>>>>> this situation would I need to "share" my entities across the
>>>>>>>>>>>>>>>>>> whole
>>>>>>>>>>>>>>>>>> processor space, before requesting a ghost exchange? I'm not
>>>>>>>>>>>>>>>>>> even really
>>>>>>>>>>>>>>>>>> sure what shared entities mean, is there a good place to look
>>>>>>>>>>>>>>>>>> in the
>>>>>>>>>>>>>>>>>> documentation to learn more about the terminology? Thanks,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Ryan
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On 10/8/18 12:17, Vijay S. Mahadevan wrote:
>>>>>>>>>>>>>>>>>>> Ryan,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> You need to use the ParallelComm object in MOAB to call
>>>>>>>>>>>>>>>>>>> exchange_ghost_cells [1] with appropriate levels of ghost
>>>>>>>>>>>>>>>>>>> layers for
>>>>>>>>>>>>>>>>>>> your mesh. This needs to be done after the mesh is loaded,
>>>>>>>>>>>>>>>>>>> or you can
>>>>>>>>>>>>>>>>>>> pass this information also as part of the options when
>>>>>>>>>>>>>>>>>>> loading a file
>>>>>>>>>>>>>>>>>>> and MOAB will internally load the file with ghosted layers.
>>>>>>>>>>>>>>>>>>> Here's an
>>>>>>>>>>>>>>>>>>> example [2].
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Vijay
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>> http://ftp.mcs.anl.gov/pub/fathom/moab-docs-develop/classmoab_1_1ParallelComm.html#a55dfa308f56fd368319bfb4244428878
>>>>>>>>>>>>>>>>>>> [2]
>>>>>>>>>>>>>>>>>>> http://ftp.mcs.anl.gov/pub/fathom/moab-docs-develop/HelloParMOAB_8cpp-example.html
>>>>>>>>>>>>>>>>>>> On Mon, Oct 8, 2018 at 1:23 PM Ryan O'Kuinghttons
>>>>>>>>>>>>>>>>>>> <ryan.okuinghttons at noaa.gov> wrote:
>>>>>>>>>>>>>>>>>>>> Hi, I am wondering if there is a way to create ghost
>>>>>>>>>>>>>>>>>>>> elements in MOAB.
>>>>>>>>>>>>>>>>>>>> By this I mean a list of copies of elements surrounding a
>>>>>>>>>>>>>>>>>>>> specific MOAB
>>>>>>>>>>>>>>>>>>>> element, ghost elements may exist on a different processor
>>>>>>>>>>>>>>>>>>>> than the
>>>>>>>>>>>>>>>>>>>> source element. I see a ghost_elems class in the appData
>>>>>>>>>>>>>>>>>>>> namespace, but
>>>>>>>>>>>>>>>>>>>> there is not much documentation on how to use it.. Thank you,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>> Ryan O'Kuinghttons
>>>>>>>>>>>>>>>>>>>> Cherokee Nation Management and Consulting
>>>>>>>>>>>>>>>>>>>> NESII/NOAA/Earth System Research Laboratory
>>>>>>>>>>>>>>>>>>>> ryan.okuinghttons at noaa.gov
>>>>>>>>>>>>>>>>>>>> https://www.esrl.noaa.gov/gsd/nesii/
>>>>>>>>>>>>>>>>>>>>
More information about the moab-dev
mailing list