[MOAB-dev] EntityHandle?
Lukasz Kaczmarczyk
Lukasz.Kaczmarczyk at glasgow.ac.uk
Thu Jul 14 09:07:54 CDT 2016
Hello,
Pleas attached fix, is small fix the problem. All test in moab passed, works for me as well.
With that change behaviour is deterministic, for any sequence of creation-deletion-creation entities the set of entity handlers and adjacencies is the same and independent on platform.
I made pull request to moab I hope that is ok.
Kind regards,
Lukasz
> On 14 Jul 2016, at 10:53, Lukasz Kaczmarczyk <Lukasz.Kaczmarczyk at glasgow.ac.uk> wrote:
>
> Hello,
>
> I apologise for being not very precise. However I finally found a problem and potential solution.
>
> The problem:
> If you create entities and delete some of them and then create entities once again order of handler will be different for different processes run sequentially.
>
> That create difficulties for example if you do parallel computations, where you like to use the same non-distributed mesh on several processors. Since entities handlers have different order on different processors, you don’t know how to attach GID to each entity without some additional communication.
>
> Moreover if you run the same program twice, sequentially, and you attach GIDs based on range of entity handlers is not guaranteed that you will get the same adjacency matrix, some rows and columns will be swapped. That make debugging difficult.
>
> If you attach GIDs in separate process and run your program then adjacency matrix will be the same and problem will be solved. However that demand some sequential run prior calculations.
>
> Where is the problem:
>
> I track this problem to TypeSequenceManager and how available list of unused entries is managed, i.e. availableList. This list is defined as
>
> typedef std::set<SequenceData*> data_set_type;
> data_set_type availableList; //!< SequenceData containing unused entries
>
> Since this is set of pointers, it is not guaranteed that order of pointer will much sequence how memory is allocated to those pointers. This is managed by operating system itself which has build in some psudo-randomnes, since we do not what other processes in system do.
>
> Now if you look how TypeSequenceManager::find_free_handle works, you will notice that behaviour of that method depend how SequenceData are ordered in availableList. Since it is set of pointer, on each process behaviour of that function is different. Not wrong just different.
>
> Partial solution, which in general not work,
>
> If you will use get_element_connect form ReadUtil and attached larger set of entities, is is guaranteed that order of handlers will be the same for each run. Moreover If set of created entities is big (or moab database is empty) is is very probable that the entities will be in separate sequence at the end and handlers will be the same for each run. That way if you red mesh to empty moab database set of handlers is always the same.
>
> However problem will exist if you build adjacencies, f.e.
> m_field.get_moab().get_adjacencies(tets,2,true,faces);
> then handlers of adjacent entities will be attached differently for different runs if you deleted entities before and this problem will persist.
>
> The optimal,
>
> The optimal solution is to make availableList deterministic. One solution is to have static variable which counts is increased each time SequenceData are created. And use that unique counter to create
>
> typedef std::set<SequenceData*,SequencePtrDataSets> data_set_type;
>
> That will resolve problem.
>
> In principle if your program behaviour is random in any sense is not good and cause the problems with debugging. I am aware that this is not problem for majority of problems solved with moab however form time to time it can make problems.
>
> I can have a go, I have forked moab repository on bitbucket, I could do some changes, hopefully that you will like it enough to accept pull-request.
>
> Kind regards,
> Lukasz
>
>
>> On 12 Jul 2016, at 08:36, Lukasz Kaczmarczyk <Lukasz.Kaczmarczyk at glasgow.ac.uk> wrote:
>>
>> If you use following (order by set containment) in last two line it works,
>>
>> // do reorder
>> ReorderTool tool(&mb_instance);
>> Tag mapping;
>> // rval = tool.handle_order_from_int_tag(th_gid, -1, mapping );
>> rval = tool.handle_order_from_sets_and_adj(meshsets,mapping);
>> rval = tool.reorder_entities( mapping );
>>
>> The problem is when you have meshsets and order entities by tag, yiu run into execution error.
>>
>> Regards,
>> Lukasz
>>
>>> On 12 Jul 2016, at 08:12, Lukasz Kaczmarczyk <Lukasz.Kaczmarczyk at glasgow.ac.uk> wrote:
>>>
>>> Hello,
>>>
>>> Pleas see attached quick check, this generate run time error. If all meshsets are removed, for example if you convert attached file to vtk and run it for VTK it work differently.
>>>
>>> Kind regards,
>>> Lukasz
>>>
>>>
>>>> On 11 Jul 2016, at 21:45, Vijay S. Mahadevan <vijay.m at gmail.com> wrote:
>>>>
>>>> Thanks Lukasz. We will see if there is a better way to optimize the
>>>> assignment in SequenceManager. However, the failure in ReorderTool may
>>>> be fixable I think. If you can send a minimal example, even a mesh
>>>> that's generated in memory, we can use that to debug this case.
>>>>
>>>> Vijay
>>>>
>>>> On Mon, Jul 11, 2016 at 3:43 PM, Lukasz Kaczmarczyk
>>>> <Lukasz.Kaczmarczyk at glasgow.ac.uk> wrote:
>>>>> Thanks,
>>>>>
>>>>> I make hierarchical meshes when I can, however I have cases were topological changes are needed. When I need to create and delete entireties, moab generate difficulties. This is problem which can be solved, however is problem more fundamental how sequence manager works not related to parallel processing. For example If I create mesh, delete some entities, and try to use ReorderTool to reorder entities with given tag, I get error.
>>>>>
>>>>> Avoiding of deletion of entities will work for many cases, however with substantial changes in mesh topology capability for safe deletion of entities could be essential.
>>>>>
>>>>>
>>>>> Kind regards,
>>>>> Lukasz
>>>>>
>>>>>> On 11 Jul 2016, at 15:43, Grindeanu, Iulian R. <iulian at mcs.anl.gov> wrote:
>>>>>>
>>>>>> Hi Lukasz,
>>>>>>
>>>>>> If you delete entities, you should be careful about entity sets those entities belong to.
>>>>>> Unless the entity sets have the tracking bit, you should remove the deleted entities from the entity sets yourself.
>>>>>>
>>>>>> Also, if the entities you delete are shared between processors, you can run into additional problems too.
>>>>>>
>>>>>> One strategy I use is to delay the deletion until the end. In this way, the new entities that get created will not reuse the entity handle of one of the deleted entities.
>>>>>>
>>>>>> (but then I have to use a bit tag to mark the entity as "deleted", so I have an extra check for an entity)
>>>>>>
>>>>>> We need to do some local refinements ourselves, and we do need to have probably a common strategy.
>>>>>>
>>>>>> Still, if you delete the remeshed entities, it will be hard to go back to the original mesh, as part of a hierarchical structure / hierarchical solver, or to even transfer results between meshes at different (local or global) refinement levels.
>>>>>>
>>>>>> Our strategy, at least for uniform refinement, is to keep the coarse meshes in different sets, according to levels.
>>>>>> I would say it would be easier to keep the coarser mesh elements, than to delete them, if you organize them correctly.
>>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>> Iulian
>>>>>>
>>>>>>
>>>>>> ________________________________________
>>>>>> From: moab-dev-bounces at mcs.anl.gov [moab-dev-bounces at mcs.anl.gov] on behalf of Lukasz Kaczmarczyk [Lukasz.Kaczmarczyk at glasgow.ac.uk]
>>>>>> Sent: Monday, July 11, 2016 9:19 AM
>>>>>> Cc: moab-dev at mcs.anl.gov
>>>>>> Subject: Re: [MOAB-dev] EntityHandle?
>>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> I working on example where I do remeshing part of the mesh. Results are consistent when I do not delete entities, however if I do remove entities I start to run into problems. EntityHandles are not consistent across runs.
>>>>>>
>>>>>> It is something strange as well, I run ReorderTool at it works ok fine, however if I delete some entities from database I get error;
>>>>>> Unreconverable error during mesh reorder.
>>>>>> Error Code 4 at ReorderTool.cpp:486
>>>>>> [likask:88241] *** Process received signal ***
>>>>>>
>>>>>> In general I stick to GID, one of the data models is that I have non-distributed mesh, but distributed algebra, i.e. mesh is the same on each proc, however I do all algebra as distributed, inconsistent handlers makes a problem for me.
>>>>>>
>>>>>> All problems start when I delete something form mesh, then if I create entity and adjacencies to those entities have handlers inconsistent cross processors, so I can not attache GID or guarantee order without some additional communication.
>>>>>>
>>>>>> Kind regards,
>>>>>> Lukasz
>>>>>>
>>>>>>> On 11 Jul 2016, at 15:03, Grindeanu, Iulian R. <iulian at mcs.anl.gov> wrote:
>>>>>>>
>>>>>>> Hi Lukasz,
>>>>>>> The entity handle is associated with the moab instance. Of course, different tasks have different moab instances.
>>>>>>> Shared entities between processors can have different handles.
>>>>>>> If the options used to read the mesh are consistent between runs, you should expect that the entity handles are consistent, reproducible between runs.
>>>>>>> Global IDs, which are just tags on entities, if read from the file, should be the same for shared entities.
>>>>>>> If you use ghosting for reading, global ids will be correctly propagated, if they exist in the file.
>>>>>>>
>>>>>>> Most of the readers use ReadUtil interface, which means that entity handles are assigned in bulk, for the entity sequence created ;
>>>>>>>
>>>>>>> vertices:
>>>>>>> ErrorCode get_node_coords(const int num_arrays,
>>>>>>> const int num_nodes,
>>>>>>> const int preferred_start_id,
>>>>>>> EntityHandle& actual_start_handle,
>>>>>>> std::vector<double*>& arrays,
>>>>>>> const int sequence_size = -1)
>>>>>>> elements:
>>>>>>> ErrorCode get_element_connect(const int num_elements,
>>>>>>> const int verts_per_element,
>>>>>>> const EntityType mdb_type,
>>>>>>> const int preferred_start_id,
>>>>>>> EntityHandle& actual_start_handle,
>>>>>>> EntityHandle*& array,
>>>>>>> int sequence_size = -1)
>>>>>>>
>>>>>>>
>>>>>>> ghosted entities are usually created one by one, so that means the first "free" entity handle is used, in ascending order, for the ghosted entities.
>>>>>>>
>>>>>>> Do you have a particular example for which you have issues with the entity handles or global ids?
>>>>>>>
>>>>>>> Iulian
>>>>>>>
>>>>>>>
>>>>>>> ________________________________________
>>>>>>> From: moab-dev-bounces at mcs.anl.gov [moab-dev-bounces at mcs.anl.gov] on behalf of Lukasz Kaczmarczyk [Lukasz.Kaczmarczyk at glasgow.ac.uk]
>>>>>>> Sent: Sunday, July 10, 2016 4:08 PM
>>>>>>> To: moab-dev at mcs.anl.gov
>>>>>>> Subject: [MOAB-dev] EntityHandle?
>>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> I like to ask how entity handle is given to created entity.
>>>>>>>
>>>>>>> In particular if I the same mesh is read, then on that mesh the some entities are create and delete in particular order, do resulting mesh has entities with the same handles for each execution of program.
>>>>>>>
>>>>>>> I understand that mesh distributed on several processes, has different entities handles on shared entities. That is clear.
>>>>>>>
>>>>>>> However what if exactly the same (non-distributed) mesh is stored on the several processor, should I expect that such mesh has that entities with the same GID has the same entity handle. What if next I apply the same sequence of operations, deletion and creation of entities. Should I expect the same handles on all meshes?
>>>>>>>
>>>>>>> Kind regards,
>>>>>>> Lukasz
>>>>>>>
>>>>>>
>>>>>
>>> <reorder_tool.cpp><4seasons.h5m>
>>
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: TypeSequenceManager.hpp
Type: application/octet-stream
Size: 17737 bytes
Desc: TypeSequenceManager.hpp
URL: <http://lists.mcs.anl.gov/pipermail/moab-dev/attachments/20160714/d2b1852e/attachment-0001.obj>
More information about the moab-dev
mailing list