<html dir="ltr">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<style id="owaParaStyle" type="text/css">P {margin-top:0;margin-bottom:0;}</style>
</head>
<body ocsi="0" fpstyle="1" style="word-wrap:break-word">
<div style="direction: ltr;font-family: Tahoma;color: #000000;font-size: 10pt;">Hi Lorenzo,<br>
Thanks for the example.<br>
So you are using c++11 thread library. <br>
<br>
The fundamental problem with thread safety in moab is due to the sequences.<br>
<br>
<br>
different threads (with different entity handles) will find different sequences. <br>
<br>
it is implemented something like this:<br>
get_tag_data (entity, *value)<br>
{<br>
SequenceData * seq = find_sequence(entity)<br>
*value = seq->value_for_entity(entity)<br>
}<br>
<br>
So we need a mechanism to make this thread safe. <br>
I am not sure what is the best solution here. (make seq private?)<br>
<br>
It would be much better to use *_iterate methods.<br>
<br>
(you have also other advantages, like not needing to do a deep copy)<br>
<br>
Iulian<br>
<br>
<br>
<div style="font-family: Times New Roman; color: #000000; font-size: 16px">
<hr tabindex="-1">
<div style="direction: ltr;" id="divRpF417459"><font color="#000000" face="Tahoma" size="2"><b>From:</b> Lorenzo Alessio Botti [bottilorenzo@gmail.com]<br>
<b>Sent:</b> Friday, October 03, 2014 10:21 AM<br>
<b>To:</b> Grindeanu, Iulian R.<br>
<b>Subject:</b> Re: [MOAB-dev] thread safety<br>
</font><br>
</div>
<div></div>
<div>
<div>Basically each thread should work on subset of a Range of entities. </div>
<div>What I was trying know is simply one thread per element, it is not very efficient but trivial to implement.</div>
<div><br>
</div>
<div>Each thread should initialise data I need to retrive from moab (node coords, elem connectivity, tags) and use it to compute shape functions needed to build local matrices and residuals.</div>
<div>Actually if data from moab is initialised on the main thread and copied in each thread the code works. </div>
<div><br>
</div>
<div style="margin:0px; font-size:14px; font-family:Menlo; color:rgb(245,245,245); background-color:rgb(0,0,0)">
std::atomic<<span style="color:#34bd26">int</span>> nAssemblers(<span style="color:#ce7924">0</span>);</div>
<div style="margin:0px; font-size:14px; font-family:Menlo; color:rgb(245,245,245); background-color:rgb(0,0,0)">
std::mutex buildersMut;</div>
<div style="margin:0px; font-size:14px; font-family:Menlo; color:rgb(245,245,245); background-color:rgb(0,0,0); position:static; z-index:auto">
std::condition_variable condVar;</div>
<div><span style="background-color:rgb(0,0,0); color:rgb(245,245,245); font-family:Menlo; font-size:14px"> </span><span style="font-family:Menlo; font-size:14px; color:rgb(52,189,38)">const</span><span style="background-color:rgb(0,0,0); color:rgb(245,245,245); font-family:Menlo; font-size:14px">
moab::Range& ents = feTOmoab.get<TopologyMOABType::OwnedElements>();</span><span style="background-color:rgb(0,0,0); color:rgb(245,245,245); font-family:Menlo; font-size:14px"> </span></div>
<div><span style="background-color:rgb(0,0,0); color:rgb(245,245,245); font-family:Menlo; font-size:14px"> </span><span style="font-family:Menlo; font-size:14px; color:rgb(175,173,36)">while</span><span style="background-color:rgb(0,0,0); color:rgb(245,245,245); font-family:Menlo; font-size:14px">
(it != ents.end())</span></div>
<div>
<div style="margin:0px; font-size:14px; font-family:Menlo; color:rgb(245,245,245); background-color:rgb(0,0,0); position:static; z-index:auto">
{</div>
<div style="margin:0px; font-size:14px; font-family:Menlo; color:rgb(245,245,245); background-color:rgb(0,0,0)">
<span style="color:#afad24">while</span>(nAssemblers.load() == nThreads)</div>
<div style="margin:0px; font-size:14px; font-family:Menlo; color:rgb(245,245,245); background-color:rgb(0,0,0)">
{</div>
<div style="margin:0px; font-size:14px; font-family:Menlo; color:rgb(245,245,245); background-color:rgb(0,0,0)">
std::unique_lock<std::mutex> uLock(buildersMut);</div>
<div style="margin:0px; font-size:14px; font-family:Menlo; color:rgb(245,245,245); background-color:rgb(0,0,0)">
condVar.wait(uLock, [&nAssemblers,&nThreads] <span style="color:#bfbfbf; background-color:#991200">
<b>{</b></span><span style="color:#afad24">return</span> nAssemblers.load() < nThreads;<span style="color:#bfbfbf; background-color:#991200"><b>}</b></span>);</div>
<div style="margin:0px; font-size:14px; font-family:Menlo; color:rgb(52,187,199); background-color:rgb(0,0,0)">
<span style="color:#f5f5f5"> </span>//std::cout<<"wait ends"<<std::endl;</div>
<div style="margin:0px; font-size:14px; font-family:Menlo; color:rgb(245,245,245); background-color:rgb(0,0,0)">
}</div>
<div style="margin:0px; font-size:14px; font-family:Menlo; color:rgb(245,245,245); background-color:rgb(0,0,0)">
<span style="color:#afad24">while</span> (nAssemblers.load() < nThreads && it != ents.end())</div>
<div style="margin:0px; font-size:14px; font-family:Menlo; color:rgb(245,245,245); background-color:rgb(0,0,0)">
{</div>
<div style="margin:0px; font-size:14px; font-family:Menlo; color:rgb(245,245,245); background-color:rgb(0,0,0)">
EntityHandle elem = *it;</div>
<div style="margin:0px; font-size:14px; font-family:Menlo; color:rgb(245,245,245); background-color:rgb(0,0,0)">
<span style="color:#34bd26">int</span> elem_local_id, elem_global_id;</div>
<div style="margin:0px; font-size:14px; font-family:Menlo; color:rgb(245,245,245); background-color:rgb(0,0,0)">
result = feTOmoab.get<TopologyMOABType::LocalId>(elem, elem_local_id);</div>
<div style="margin:0px; font-size:14px; font-family:Menlo; color:rgb(245,245,245); background-color:rgb(0,0,0)">
assert(result == MB_SUCCESS);</div>
<div style="margin:0px; font-size:14px; font-family:Menlo; color:rgb(245,245,245); background-color:rgb(0,0,0)">
result = feTOmoab.get<TopologyMOABType::GlobalId>(elem, elem_global_id);</div>
<div style="margin:0px; font-size:14px; font-family:Menlo; color:rgb(245,245,245); background-color:rgb(0,0,0)">
assert(result == MB_SUCCESS);</div>
<div style="margin:0px; font-size:14px; font-family:Menlo; color:rgb(245,245,245); background-color:rgb(0,0,0)">
TopologyMOAB& feDataAccessor = feTOmoab.getFEDataAccessor();</div>
<div style="margin:0px; font-size:14px; font-family:Menlo; color:rgb(245,245,245); background-color:rgb(0,0,0)">
ErrorCode result = feDataAccessor.initData(elem);</div>
<div style="margin:0px; font-size:14px; font-family:Menlo; color:rgb(245,245,245); background-color:rgb(0,0,0)">
feDataAccessor.setLocalIndex(elem_local_id);</div>
<div style="margin:0px; font-size:14px; font-family:Menlo; color:rgb(245,245,245); background-color:rgb(0,0,0)">
feDataAccessor.setGlobalIndex(elem_global_id);</div>
<div style="margin:0px; font-size:14px; font-family:Menlo; color:rgb(245,245,245); background-color:rgb(0,0,0)">
nAssemblers++;</div>
<div style="margin:0px; font-size:14px; font-family:Menlo; color:rgb(52,187,199); background-color:rgb(0,0,0)">
<span style="color:#f5f5f5"> </span>//std::cout<<nAssemblers.load()<<" ";</div>
<div style="margin:0px; font-size:14px; font-family:Menlo; color:rgb(245,245,245); background-color:rgb(0,0,0)">
voidF = (std::async(std::launch::async,</div>
<div style="margin:0px; font-size:14px; font-family:Menlo; color:rgb(245,245,245); background-color:rgb(0,0,0)">
[&matrix, &pRhs, feDataAccessor, &FEManager, &oldSolution, &nlSolution, dt, &condVar, &nAssemblers]</div>
<div style="margin:0px; font-size:14px; font-family:Menlo; color:rgb(245,245,245); background-color:rgb(0,0,0)">
() <span style="color:#34bd26">mutable</span> -> <span style="color:#34bd26">
void</span></div>
<div style="margin:0px; font-size:14px; font-family:Menlo; color:rgb(245,245,245); background-color:rgb(0,0,0)">
<span style="color:#bfbfbf; background-color:#991200"><b>{</b></span></div>
<div style="margin:0px; font-size:14px; font-family:Menlo; color:rgb(245,245,245); background-color:rgb(0,0,0)">
INSElemAssembler iNSElemAssembler;</div>
<div style="margin:0px; font-size:14px; font-family:Menlo; color:rgb(245,245,245); background-color:rgb(0,0,0)">
FEStorage<InputData::dim> feStorage;</div>
<div style="margin:0px; font-size:14px; font-family:Menlo; color:rgb(245,245,245); background-color:rgb(0,0,0)">
iNSElemAssembler.assemble_element_contribution (matrix, pRhs, feStorage, feDataAccessor, FEManager, oldSolution, nlSolution, dt);</div>
<div style="margin:0px; font-size:14px; font-family:Menlo; color:rgb(245,245,245); background-color:rgb(0,0,0)">
nAssemblers--;</div>
<div style="margin:0px; font-size:14px; font-family:Menlo; color:rgb(245,245,245); background-color:rgb(0,0,0)">
condVar.notify_one();</div>
<div style="margin:0px; font-size:14px; font-family:Menlo; color:rgb(245,245,245); background-color:rgb(0,0,0)">
<span style="color:#bfbfbf; background-color:#991200"><b>}</b></span>));</div>
<div style="margin:0px; font-size:14px; font-family:Menlo; color:rgb(245,245,245); background-color:rgb(0,0,0)">
it++;</div>
<div style="margin:0px; font-size:14px; font-family:Menlo; color:rgb(245,245,245); background-color:rgb(0,0,0)">
elemNumber++;</div>
<div style="margin:0px; font-size:14px; font-family:Menlo; color:rgb(52,187,199); background-color:rgb(0,0,0)">
<span style="color:#f5f5f5"> </span>//std::cout<<"elemNumber "<<elemNumber<<std::endl;</div>
<div style="margin:0px; font-size:14px; font-family:Menlo; color:rgb(245,245,245); background-color:rgb(0,0,0)">
}</div>
<div style="margin:0px; font-size:14px; font-family:Menlo; color:rgb(245,245,245); background-color:rgb(0,0,0); position:static; z-index:auto">
}</div>
</div>
<div><br>
</div>
<div>In the loop above you can see that feDataAccessor.initData(elem) (which calls get_coords() and get_connectivity()) and the tags getter are executed on the main thread and captured by copy in the lambda.</div>
<div>As opposite feDataAccessor should be instantiated and used inside the lambda, this would also allow each thread to deal with more than one element without coping a lot of memory. </div>
<div><br>
</div>
<div><br>
</div>
<br>
<div>
<div>On 03 Oct 2014, at 16:46, Grindeanu, Iulian R. <<a href="mailto:iulian@mcs.anl.gov" target="_blank">iulian@mcs.anl.gov</a>> wrote:</div>
<br class="Apple-interchange-newline">
<blockquote type="cite">How are you planing to use these?<br>
can you share some code?<br>
<br>
I think we are using some hints for the next similar calls, which may not be thread safe.<br>
One solution would be to use direct access to memory calls, which return pointers in memory, for contiguous access.<br>
<br>
so we could use tag_iterate, coords_iterate, connect_iterate.<br>
The disadvantage is that an extra loop is needed if the sequences are broken.<br>
<br>
Iulian<br>
<br>
________________________________________<br>
From: <a href="mailto:moab-dev-bounces@mcs.anl.gov" target="_blank">moab-dev-bounces@mcs.anl.gov</a> [<a href="mailto:moab-dev-bounces@mcs.anl.gov" target="_blank">moab-dev-bounces@mcs.anl.gov</a>] on behalf of Paul Wilson [<a href="mailto:wilsonp@engr.wisc.edu" target="_blank">wilsonp@engr.wisc.edu</a>]<br>
Sent: Friday, October 03, 2014 8:38 AM<br>
To: Lorenzo Alessio Botti; <a href="mailto:MOAB-dev@mcs.anl.gov" target="_blank">
MOAB-dev@mcs.anl.gov</a><br>
Subject: Re: [MOAB-dev] thread safety<br>
<br>
Hi Lorenzo,<br>
<br>
We also have quite urgent needs for this and have begun looking at it<br>
for our use cases.<br>
<br>
Paul<br>
<br>
On 10/03/2014 04:44 AM, Lorenzo Alessio Botti wrote:<br>
<blockquote type="cite">Dear all,<br>
trying to playing around with some threaded code I seem to understand that some moab interface methods are not thread safe.<br>
<br>
I’m concurrently using, for example<br>
tag_get_data<br>
get_connectivity<br>
get_coords<br>
<br>
Is my assertion correct or am I messing things up on my side?<br>
If so, do you have any plans to implement a set, or a subset, of thread safe methods?<br>
<br>
Thanks for the infos.<br>
Lorenzo<br>
</blockquote>
<br>
--<br>
-- ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ --<br>
Paul Wilson ~ UW-Madison ~ 608-263-0807 ~ cal: <a href="http://bit.ly/pphw-cal" target="_blank">
http://bit.ly/pphw-cal</a><br>
Professor, Engineering Physics. ~ <a href="http://cnerg.engr.wisc.edu" target="_blank">
http://cnerg.engr.wisc.edu</a><br>
Faculty Director, Advanced Computing Infrastructure<br>
<br>
<br>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</body>
</html>