[MOAB-dev] EntityHandle?
Vijay S. Mahadevan
vijay.m at gmail.com
Thu Jul 14 09:25:16 CDT 2016
Lukasz, thanks much for getting to the bottom of the issue. Yes loss of
determinism is a bad thing for reproducibility of results and this should
be fixed.
I just saw that you submitted a PR for the change. Perhaps if you can also
add the explanation of the issue from your previous email, that would add
more context to the PR submission. We will take a look at it, test it on
various configurations and if everything passes, will merge to master soon.
Best,
Vijay
On Jul 14, 2016 9:07 AM, "Lukasz Kaczmarczyk" <
Lukasz.Kaczmarczyk at glasgow.ac.uk> wrote:
> Hello,
>
> Pleas attached fix, is small fix the problem. All test in moab passed,
> works for me as well.
>
> With that change behaviour is deterministic, for any sequence of
> creation-deletion-creation entities the set of entity handlers and
> adjacencies is the same and independent on platform.
>
> I made pull request to moab I hope that is ok.
>
> Kind regards,
> Lukasz
>
>
>
> > On 14 Jul 2016, at 10:53, Lukasz Kaczmarczyk <
> Lukasz.Kaczmarczyk at glasgow.ac.uk> wrote:
> >
> > Hello,
> >
> > I apologise for being not very precise. However I finally found a
> problem and potential solution.
> >
> > The problem:
> > If you create entities and delete some of them and then create entities
> once again order of handler will be different for different processes run
> sequentially.
> >
> > That create difficulties for example if you do parallel computations,
> where you like to use the same non-distributed mesh on several processors.
> Since entities handlers have different order on different processors, you
> don’t know how to attach GID to each entity without some additional
> communication.
> >
> > Moreover if you run the same program twice, sequentially, and you attach
> GIDs based on range of entity handlers is not guaranteed that you will get
> the same adjacency matrix, some rows and columns will be swapped. That make
> debugging difficult.
> >
> > If you attach GIDs in separate process and run your program then
> adjacency matrix will be the same and problem will be solved. However that
> demand some sequential run prior calculations.
> >
> > Where is the problem:
> >
> > I track this problem to TypeSequenceManager and how available list of
> unused entries is managed, i.e. availableList. This list is defined as
> >
> > typedef std::set<SequenceData*> data_set_type;
> > data_set_type availableList; //!< SequenceData containing unused
> entries
> >
> > Since this is set of pointers, it is not guaranteed that order of
> pointer will much sequence how memory is allocated to those pointers. This
> is managed by operating system itself which has build in some
> psudo-randomnes, since we do not what other processes in system do.
> >
> > Now if you look how TypeSequenceManager::find_free_handle works, you
> will notice that behaviour of that method depend how SequenceData are
> ordered in availableList. Since it is set of pointer, on each process
> behaviour of that function is different. Not wrong just different.
> >
> > Partial solution, which in general not work,
> >
> > If you will use get_element_connect form ReadUtil and attached larger
> set of entities, is is guaranteed that order of handlers will be the same
> for each run. Moreover If set of created entities is big (or moab database
> is empty) is is very probable that the entities will be in separate
> sequence at the end and handlers will be the same for each run. That way if
> you red mesh to empty moab database set of handlers is always the same.
> >
> > However problem will exist if you build adjacencies, f.e.
> > m_field.get_moab().get_adjacencies(tets,2,true,faces);
> > then handlers of adjacent entities will be attached differently for
> different runs if you deleted entities before and this problem will persist.
> >
> > The optimal,
> >
> > The optimal solution is to make availableList deterministic. One
> solution is to have static variable which counts is increased each time
> SequenceData are created. And use that unique counter to create
> >
> > typedef std::set<SequenceData*,SequencePtrDataSets> data_set_type;
> >
> > That will resolve problem.
> >
> > In principle if your program behaviour is random in any sense is not
> good and cause the problems with debugging. I am aware that this is not
> problem for majority of problems solved with moab however form time to
> time it can make problems.
> >
> > I can have a go, I have forked moab repository on bitbucket, I could do
> some changes, hopefully that you will like it enough to accept pull-request.
> >
> > Kind regards,
> > Lukasz
> >
> >
> >> On 12 Jul 2016, at 08:36, Lukasz Kaczmarczyk <
> Lukasz.Kaczmarczyk at glasgow.ac.uk> wrote:
> >>
> >> If you use following (order by set containment) in last two line it
> works,
> >>
> >> // do reorder
> >> ReorderTool tool(&mb_instance);
> >> Tag mapping;
> >> // rval = tool.handle_order_from_int_tag(th_gid, -1, mapping );
> >> rval = tool.handle_order_from_sets_and_adj(meshsets,mapping);
> >> rval = tool.reorder_entities( mapping );
> >>
> >> The problem is when you have meshsets and order entities by tag, yiu
> run into execution error.
> >>
> >> Regards,
> >> Lukasz
> >>
> >>> On 12 Jul 2016, at 08:12, Lukasz Kaczmarczyk <
> Lukasz.Kaczmarczyk at glasgow.ac.uk> wrote:
> >>>
> >>> Hello,
> >>>
> >>> Pleas see attached quick check, this generate run time error. If all
> meshsets are removed, for example if you convert attached file to vtk and
> run it for VTK it work differently.
> >>>
> >>> Kind regards,
> >>> Lukasz
> >>>
> >>>
> >>>> On 11 Jul 2016, at 21:45, Vijay S. Mahadevan <vijay.m at gmail.com>
> wrote:
> >>>>
> >>>> Thanks Lukasz. We will see if there is a better way to optimize the
> >>>> assignment in SequenceManager. However, the failure in ReorderTool may
> >>>> be fixable I think. If you can send a minimal example, even a mesh
> >>>> that's generated in memory, we can use that to debug this case.
> >>>>
> >>>> Vijay
> >>>>
> >>>> On Mon, Jul 11, 2016 at 3:43 PM, Lukasz Kaczmarczyk
> >>>> <Lukasz.Kaczmarczyk at glasgow.ac.uk> wrote:
> >>>>> Thanks,
> >>>>>
> >>>>> I make hierarchical meshes when I can, however I have cases were
> topological changes are needed. When I need to create and delete
> entireties, moab generate difficulties. This is problem which can be
> solved, however is problem more fundamental how sequence manager works not
> related to parallel processing. For example If I create mesh, delete some
> entities, and try to use ReorderTool to reorder entities with given tag, I
> get error.
> >>>>>
> >>>>> Avoiding of deletion of entities will work for many cases, however
> with substantial changes in mesh topology capability for safe deletion of
> entities could be essential.
> >>>>>
> >>>>>
> >>>>> Kind regards,
> >>>>> Lukasz
> >>>>>
> >>>>>> On 11 Jul 2016, at 15:43, Grindeanu, Iulian R. <iulian at mcs.anl.gov>
> wrote:
> >>>>>>
> >>>>>> Hi Lukasz,
> >>>>>>
> >>>>>> If you delete entities, you should be careful about entity sets
> those entities belong to.
> >>>>>> Unless the entity sets have the tracking bit, you should remove the
> deleted entities from the entity sets yourself.
> >>>>>>
> >>>>>> Also, if the entities you delete are shared between processors, you
> can run into additional problems too.
> >>>>>>
> >>>>>> One strategy I use is to delay the deletion until the end. In this
> way, the new entities that get created will not reuse the entity handle of
> one of the deleted entities.
> >>>>>>
> >>>>>> (but then I have to use a bit tag to mark the entity as "deleted",
> so I have an extra check for an entity)
> >>>>>>
> >>>>>> We need to do some local refinements ourselves, and we do need to
> have probably a common strategy.
> >>>>>>
> >>>>>> Still, if you delete the remeshed entities, it will be hard to go
> back to the original mesh, as part of a hierarchical structure /
> hierarchical solver, or to even transfer results between meshes at
> different (local or global) refinement levels.
> >>>>>>
> >>>>>> Our strategy, at least for uniform refinement, is to keep the
> coarse meshes in different sets, according to levels.
> >>>>>> I would say it would be easier to keep the coarser mesh elements,
> than to delete them, if you organize them correctly.
> >>>>>>
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Iulian
> >>>>>>
> >>>>>>
> >>>>>> ________________________________________
> >>>>>> From: moab-dev-bounces at mcs.anl.gov [moab-dev-bounces at mcs.anl.gov]
> on behalf of Lukasz Kaczmarczyk [Lukasz.Kaczmarczyk at glasgow.ac.uk]
> >>>>>> Sent: Monday, July 11, 2016 9:19 AM
> >>>>>> Cc: moab-dev at mcs.anl.gov
> >>>>>> Subject: Re: [MOAB-dev] EntityHandle?
> >>>>>>
> >>>>>> Hello,
> >>>>>>
> >>>>>> I working on example where I do remeshing part of the mesh. Results
> are consistent when I do not delete entities, however if I do remove
> entities I start to run into problems. EntityHandles are not consistent
> across runs.
> >>>>>>
> >>>>>> It is something strange as well, I run ReorderTool at it works ok
> fine, however if I delete some entities from database I get error;
> >>>>>> Unreconverable error during mesh reorder.
> >>>>>> Error Code 4 at ReorderTool.cpp:486
> >>>>>> [likask:88241] *** Process received signal ***
> >>>>>>
> >>>>>> In general I stick to GID, one of the data models is that I have
> non-distributed mesh, but distributed algebra, i.e. mesh is the same on
> each proc, however I do all algebra as distributed, inconsistent handlers
> makes a problem for me.
> >>>>>>
> >>>>>> All problems start when I delete something form mesh, then if I
> create entity and adjacencies to those entities have handlers inconsistent
> cross processors, so I can not attache GID or guarantee order without some
> additional communication.
> >>>>>>
> >>>>>> Kind regards,
> >>>>>> Lukasz
> >>>>>>
> >>>>>>> On 11 Jul 2016, at 15:03, Grindeanu, Iulian R. <iulian at mcs.anl.gov>
> wrote:
> >>>>>>>
> >>>>>>> Hi Lukasz,
> >>>>>>> The entity handle is associated with the moab instance. Of course,
> different tasks have different moab instances.
> >>>>>>> Shared entities between processors can have different handles.
> >>>>>>> If the options used to read the mesh are consistent between runs,
> you should expect that the entity handles are consistent, reproducible
> between runs.
> >>>>>>> Global IDs, which are just tags on entities, if read from the
> file, should be the same for shared entities.
> >>>>>>> If you use ghosting for reading, global ids will be correctly
> propagated, if they exist in the file.
> >>>>>>>
> >>>>>>> Most of the readers use ReadUtil interface, which means that
> entity handles are assigned in bulk, for the entity sequence created ;
> >>>>>>>
> >>>>>>> vertices:
> >>>>>>> ErrorCode get_node_coords(const int num_arrays,
> >>>>>>> const int num_nodes,
> >>>>>>> const int preferred_start_id,
> >>>>>>> EntityHandle& actual_start_handle,
> >>>>>>> std::vector<double*>& arrays,
> >>>>>>> const int sequence_size = -1)
> >>>>>>> elements:
> >>>>>>> ErrorCode get_element_connect(const int num_elements,
> >>>>>>> const int verts_per_element,
> >>>>>>> const EntityType mdb_type,
> >>>>>>> const int preferred_start_id,
> >>>>>>> EntityHandle&
> actual_start_handle,
> >>>>>>> EntityHandle*& array,
> >>>>>>> int sequence_size = -1)
> >>>>>>>
> >>>>>>>
> >>>>>>> ghosted entities are usually created one by one, so that means the
> first "free" entity handle is used, in ascending order, for the ghosted
> entities.
> >>>>>>>
> >>>>>>> Do you have a particular example for which you have issues with
> the entity handles or global ids?
> >>>>>>>
> >>>>>>> Iulian
> >>>>>>>
> >>>>>>>
> >>>>>>> ________________________________________
> >>>>>>> From: moab-dev-bounces at mcs.anl.gov [moab-dev-bounces at mcs.anl.gov]
> on behalf of Lukasz Kaczmarczyk [Lukasz.Kaczmarczyk at glasgow.ac.uk]
> >>>>>>> Sent: Sunday, July 10, 2016 4:08 PM
> >>>>>>> To: moab-dev at mcs.anl.gov
> >>>>>>> Subject: [MOAB-dev] EntityHandle?
> >>>>>>>
> >>>>>>> Hello,
> >>>>>>>
> >>>>>>> I like to ask how entity handle is given to created entity.
> >>>>>>>
> >>>>>>> In particular if I the same mesh is read, then on that mesh the
> some entities are create and delete in particular order, do resulting mesh
> has entities with the same handles for each execution of program.
> >>>>>>>
> >>>>>>> I understand that mesh distributed on several processes, has
> different entities handles on shared entities. That is clear.
> >>>>>>>
> >>>>>>> However what if exactly the same (non-distributed) mesh is stored
> on the several processor, should I expect that such mesh has that entities
> with the same GID has the same entity handle. What if next I apply the same
> sequence of operations, deletion and creation of entities. Should I expect
> the same handles on all meshes?
> >>>>>>>
> >>>>>>> Kind regards,
> >>>>>>> Lukasz
> >>>>>>>
> >>>>>>
> >>>>>
> >>> <reorder_tool.cpp><4seasons.h5m>
> >>
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/moab-dev/attachments/20160714/f3b1cc5d/attachment.html>
More information about the moab-dev
mailing list