[MOAB-dev] thread safety

[Grindeanu, Iulian R.]

Hi Lorenzo,
Thanks for the example.
So you are using c++11 thread library.

The fundamental problem with thread safety in moab is due to the sequences.


different threads (with different entity handles) will find different sequences.

it is implemented something like this:
  get_tag_data (entity, *value)
{
    SequenceData  * seq = find_sequence(entity)
    *value = seq->value_for_entity(entity)
}

So we need a mechanism to make this thread safe.
I am not sure what is the best solution here. (make seq private?)

It would be much better to use *_iterate methods.

(you have also other advantages, like not needing to do a deep copy)

Iulian


________________________________
From: Lorenzo Alessio Botti [bottilorenzo at gmail.com]
Sent: Friday, October 03, 2014 10:21 AM
To: Grindeanu, Iulian R.
Subject: Re: [MOAB-dev] thread safety

Basically each thread should work on subset of a Range of entities.
What I was trying know is simply one thread per element, it is not very efficient but trivial to implement.

Each thread should initialise data I need to retrive from moab (node coords, elem connectivity, tags) and use it to compute shape functions needed to build local matrices and residuals.
Actually if data from moab is initialised on the main thread and copied in each thread the code works.

  std::atomic<int> nAssemblers(0);
  std::mutex buildersMut;
  std::condition_variable condVar;
  const moab::Range& ents = feTOmoab.get<TopologyMOABType::OwnedElements>();
   while (it != ents.end())
      {
        while(nAssemblers.load() == nThreads)
        {
          std::unique_lock<std::mutex> uLock(buildersMut);
          condVar.wait(uLock, [&nAssemblers,&nThreads] {return nAssemblers.load() < nThreads;});
          //std::cout<<"wait ends"<<std::endl;
        }
        while (nAssemblers.load() < nThreads && it != ents.end())
        {
          EntityHandle elem = *it;
          int elem_local_id, elem_global_id;
          result = feTOmoab.get<TopologyMOABType::LocalId>(elem, elem_local_id);
            assert(result == MB_SUCCESS);
          result = feTOmoab.get<TopologyMOABType::GlobalId>(elem, elem_global_id);
            assert(result == MB_SUCCESS);
          TopologyMOAB& feDataAccessor = feTOmoab.getFEDataAccessor();
          ErrorCode result = feDataAccessor.initData(elem);
          feDataAccessor.setLocalIndex(elem_local_id);
          feDataAccessor.setGlobalIndex(elem_global_id);
          nAssemblers++;
          //std::cout<<nAssemblers.load()<<" ";
          voidF = (std::async(std::launch::async,
          [&matrix, &pRhs, feDataAccessor, &FEManager, &oldSolution, &nlSolution, dt, &condVar, &nAssemblers]
          () mutable -> void
          {
            INSElemAssembler iNSElemAssembler;
            FEStorage<InputData::dim> feStorage;
            iNSElemAssembler.assemble_element_contribution (matrix, pRhs, feStorage, feDataAccessor, FEManager, oldSolution, nlSolution, dt);
            nAssemblers--;
            condVar.notify_one();
          }));
          it++;
          elemNumber++;
          //std::cout<<"elemNumber "<<elemNumber<<std::endl;
        }
      }

In the loop above you can see that feDataAccessor.initData(elem) (which calls get_coords() and get_connectivity()) and the tags getter are executed on the main thread and captured by copy in the lambda.
As opposite feDataAccessor should be instantiated and used inside the lambda, this would also allow each thread to deal with more than one element without coping a lot of memory.



On 03 Oct 2014, at 16:46, Grindeanu, Iulian R. <iulian at mcs.anl.gov<mailto:iulian at mcs.anl.gov>> wrote:

How are you planing to use these?
can you share some code?

I think we are using some hints for the next similar calls, which may not be thread safe.
One solution would be to use direct access to memory calls, which return pointers in memory, for contiguous access.

so we could use tag_iterate, coords_iterate, connect_iterate.
The disadvantage is that an extra loop is needed if the sequences are broken.

Iulian

________________________________________
From: moab-dev-bounces at mcs.anl.gov<mailto:moab-dev-bounces at mcs.anl.gov> [moab-dev-bounces at mcs.anl.gov<mailto:moab-dev-bounces at mcs.anl.gov>] on behalf of Paul Wilson [wilsonp at engr.wisc.edu<mailto:wilsonp at engr.wisc.edu>]
Sent: Friday, October 03, 2014 8:38 AM
To: Lorenzo Alessio Botti; MOAB-dev at mcs.anl.gov<mailto:MOAB-dev at mcs.anl.gov>
Subject: Re: [MOAB-dev] thread safety

Hi Lorenzo,

We also have quite urgent needs for this and have begun looking at it
for our use cases.

Paul

On 10/03/2014 04:44 AM, Lorenzo Alessio Botti wrote:
Dear all,
trying to playing around with some threaded code I seem to understand that some moab interface methods are not thread safe.

I’m concurrently using, for example
tag_get_data
get_connectivity
get_coords

Is my assertion correct or am I messing things up on my side?
If so, do you have any plans to implement a set, or a subset, of thread safe methods?

Thanks for the infos.
Lorenzo

--
-- ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ --
Paul Wilson ~ UW-Madison ~ 608-263-0807 ~ cal: http://bit.ly/pphw-cal
Professor, Engineering Physics. ~ http://cnerg.engr.wisc.edu
Faculty Director, Advanced Computing Infrastructure



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/moab-dev/attachments/20141003/cb0e732f/attachment-0001.html>