<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>Hi Matthew,</p>
<p>In the worst case, every node/cell may have different label. <br>
</p>
<p>Below is one of the worst scenario with 102299 nodes and 102299
different labels for test. I found the time cost increase during
the loop. The first 9300 loop takes least time (<0.5) while the
last 9300 loops takes much more time (>7.7), as shown below. If
I use larger mesh with >1 million nodes, it runs very very
slowly in this part. The PETSc is configured with optimization on.</p>
<p>Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran
--download-mpich --download-scalapack --download-parmetis
--download-metis --download-ptscotch --download-fblaslapack
--download-hypre --download-superlu_dist --download-hdf5=yes
--download-ctetgen --with-debugging=0 COPTFLAGS="-O3 -march=native
-mtune=native" CXXOPTFLAGS="-O3 -march=native -mtune=native"
FOPTFLAGS="-O3 -march=native -mtune=native"</p>
<p>
<style type="text/css">
body,div,table,thead,tbody,tfoot,tr,th,td,p { font-family:"Liberation Sans"; font-size:x-small }
a.comment-indicator:hover + comment { background:#ffd; position:absolute; display:block; border:1px solid black; padding:0.5em; }
a.comment-indicator { background:red; display:inline-block; border:1px solid black; width:0.5em; height:0.5em; }
comment { display:none; }
</style> </p>
<table cellspacing="0" width="816" border="0" height="245">
<colgroup span="6" width="85"></colgroup> <tbody>
<tr>
<td align="left" height="32">istart</td>
<td align="left">iend</td>
<td align="left">progress</td>
<td align="left">CPU_Time</td>
<td align="left">time cost - old (sec) </td>
<td align="left">time cost - new (sec)</td>
</tr>
<tr>
<td align="right" height="17">0</td>
<td align="right">9299</td>
<td align="right">0</td>
<td align="right">1524670045.51166</td>
<td align="left"><br>
</td>
<td align="left"><br>
</td>
</tr>
<tr>
<td align="right" height="17">9300</td>
<td align="right">18599</td>
<td align="right">0.100010753</td>
<td align="right">1524670045.99605</td>
<td align="right">0.4843890667</td>
<td align="right">0.497246027</td>
</tr>
<tr>
<td align="right" height="17">18600</td>
<td align="right">27899</td>
<td align="right">0.200010747</td>
<td align="right">1524670047.32635</td>
<td align="right">1.330302</td>
<td align="right">1.3820912838</td>
</tr>
<tr>
<td align="right" height="17">27900</td>
<td align="right">37199</td>
<td align="right">0.300010741</td>
<td align="right">1524670049.3066</td>
<td align="right">1.9802515507</td>
<td align="right">2.2439446449</td>
</tr>
<tr>
<td align="right" height="17">37200</td>
<td align="right">46499</td>
<td align="right">0.400010765</td>
<td align="right">1524670052.1594</td>
<td align="right">2.852804184</td>
<td align="right">3.0739262104</td>
</tr>
<tr>
<td align="right" height="17">46500</td>
<td align="right">55799</td>
<td align="right">0.500010729</td>
<td align="right">1524670055.90961</td>
<td align="right">3.7502081394</td>
<td align="right">3.9270553589</td>
</tr>
<tr>
<td align="right" height="17">55800</td>
<td align="right">65099</td>
<td align="right">0.600010753</td>
<td align="right">1524670060.47654</td>
<td align="right">4.5669286251</td>
<td align="right">4.7571902275</td>
</tr>
<tr>
<td align="right" height="17">65100</td>
<td align="right">74399</td>
<td align="right">0.700010777</td>
<td align="right">1524670066.0941</td>
<td align="right">5.6175630093</td>
<td align="right">5.7428796291</td>
</tr>
<tr>
<td align="right" height="17">74400</td>
<td align="right">83699</td>
<td align="right">0.800010741</td>
<td align="right">1524670072.53886</td>
<td align="right">6.44475317</td>
<td align="right">6.5761549473</td>
</tr>
<tr>
<td align="right" height="17">83700</td>
<td align="right">92998</td>
<td align="right">0.900010765</td>
<td align="right">1524670079.99072</td>
<td align="right">7.4518604279</td>
<td align="right">7.4606924057</td>
</tr>
<tr>
<td align="right" height="17">92999</td>
<td align="right">102298</td>
<td align="right">1</td>
<td align="right">1524670087.71066</td>
<td align="right">7.7199423313</td>
<td align="right">8.2424075603</td>
</tr>
</tbody>
</table>
<br>
<br>
old code<br>
<br>
do ipoint = 0, istart-1<br>
!c output time cost, use 1 processor to test<br>
if (b_enable_output .and. rank == 0) then<br>
if (mod(ipoint,iprogress) == 0 .or. ipoint == istart-1)
then<br>
!write(*,'(f3.1,1x)',advance="no") (ipoint+1.0)/istart<br>
write(*,*) ipoint,
(ipoint+1.0)/istart,"time",MPI_Wtime()<br>
end if<br>
end if<br>
<br>
call
DMSetLabelValue(dmda_flow%da,"cid_lg2g",ipoint, &<br>
ipoint+1,ierr)<br>
CHKERRQ(ierr)<br>
end do<br>
<br>
<br>
new code<br>
<br>
call DMCreateLabel(dmda_flow%da,'cid_lg2g',ierr)<br>
CHKERRQ(ierr)<br>
<br>
call DMGetLabel(dmda_flow%da,'cid_lg2g',label, ierr)<br>
CHKERRQ(ierr)<br>
<br>
do ipoint = 0, istart-1<br>
!c output time cost, use 1 processor to test<br>
if (b_enable_output .and. rank == 0) then<br>
if (mod(ipoint,iprogress) == 0 .or. ipoint == istart-1)
then<br>
!write(*,'(f3.1,1x)',advance="no") (ipoint+1.0)/istart<br>
write(*,*) ipoint,
(ipoint+1.0)/istart,"time",MPI_Wtime()<br>
end if<br>
end if<br>
<br>
call DMLabelSetValue(label,ipoint,ipoint+1,ierr)<br>
CHKERRQ(ierr)<br>
end do<br>
<br>
Thanks,<br>
<br>
Danyang<br>
<br>
<div class="moz-cite-prefix">On 2018-04-25 03:16 AM, Matthew Knepley
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAMYG4G=SDwJLJ=f-oV9tnVYxqjkJEjy6RUsxaPOgLmCyzm7S-w@mail.gmail.com">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">On Tue, Apr 24, 2018 at 11:57 PM,
Danyang Su <span dir="ltr"><<a
href="mailto:danyang.su@gmail.com" target="_blank"
moz-do-not-send="true">danyang.su@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">Hi All,<br>
<br>
I use DMPlex in unstructured grid code and recently found
DMSetLabelValue takes a lot of time for large problem,
e.g., num. of cells > 1 million. In my code, I use<br>
</blockquote>
<div><br>
</div>
<div>I read your code wrong. For large loop, you should not
use the convenience function. You should use</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
DMPlexCreateFromCellList ()</blockquote>
<div><br>
</div>
<div>DMGetLabel(dm, name, &label)</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex"> </blockquote>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Loop over all cells/nodes{<br>
<br>
DMSetLabelValue<br>
</blockquote>
<div><br>
</div>
<div>Replace this by DMLabelSetValue(label, point, val)</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
}<br>
<br>
DMPlexDistribute<br>
<br>
The code works fine except DMSetLabelValue takes a lot of
time for large problem. I use DMSetLabelValue to set
material id for all the nodes or cells so that each
subdomain has a copy of material id. Is there any other
functions that can be used more efficient, e.g. set labels
by array, not 1 by 1?<br>
</blockquote>
<div><br>
</div>
<div>That should take much less time.</div>
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
Thanks,<br>
<br>
Danyang<br>
<br>
</blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
-- <br>
<div class="gmail_signature" data-smartmail="gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>What most experimenters take for granted before
they begin their experiments is infinitely more
interesting than any results to which their
experiments lead.<br>
-- Norbert Wiener</div>
<div><br>
</div>
<div><a href="http://www.caam.rice.edu/%7Emk51/"
target="_blank" moz-do-not-send="true">https://www.cse.buffalo.edu/~knepley/</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<br>
</body>
</html>