<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body dir="auto"><div></div><div>That'a reasonable. The core compete for shared resources (L3 and DDR). </div><div><br>On 6 Dec 2016, at 09:25, <a href="mailto:nek5000-users@lists.mcs.anl.gov">nek5000-users@lists.mcs.anl.gov</a> wrote:<br><br></div><blockquote type="cite"><div><div dir="ltr">Dear Stefan,<div><br></div><div>Thank you for your reply. There are 139056 elements and the polynomial order is 7 (lx1=8). I measure the solver time to compute the speedup. For example, the serial job takes 2564.59s and the same job with 24 tasks takes 302.18s. The speedup is about 8.5. This is on a single Tianhe-2 node. The speedup between nodes is quite good.</div><div><br></div><div>I also test the code on my 36-core computer (Dual Xeon E5 18-Core). I can only get about 12 times speedup when I use 36 tasks. It is also about 1/3.</div><div><br></div><div>Best regards,</div><div>Wei XU</div><div><br></div><div><br></div><div><span style="font-size:12.8px">From: </span><a href="mailto:nek5000-users@lists.mcs.anl.gov" style="font-size:12.8px">nek5000-users@lists.mcs.anl.<wbr>gov</a><br style="font-size:12.8px"><span style="font-size:12.8px">To: <</span><a href="mailto:nek5000-users@lists.mcs.anl.gov" style="font-size:12.8px">nek5000-users@lists.mcs.anl.<wbr>gov</a><span style="font-size:12.8px">></span><br style="font-size:12.8px"><span style="font-size:12.8px">Subject: Re: [Nek5000-users] Parallel speedup on supercomputer</span><br style="font-size:12.8px"><span style="font-size:12.8px"> Tianhe-2</span><br style="font-size:12.8px"><span style="font-size:12.8px">Message-ID:</span><br style="font-size:12.8px"><span style="font-size:12.8px"> <</span><a href="mailto:mailman.7164.1480933423.3602.nek5000-users@lists.mcs.anl.gov" style="font-size:12.8px">mailman.7164.1480933423.3602.<wbr>nek5000-users@lists.mcs.anl.<wbr>gov</a><span style="font-size:12.8px">></span><br style="font-size:12.8px"><span style="font-size:12.8px">Content-Type: text/plain; charset="utf-8"</span><br style="font-size:12.8px"><br style="font-size:12.8px"><span style="font-size:12.8px">What's your problem size (number of elements and polynomial order)?</span><br style="font-size:12.8px"><br style="font-size:12.8px"><span style="font-size:12.8px">Let's assume t_MPI << t (this holds if your problem size is reasonably large). Even in this limit you don't get a linear intra-node speedup simply because Nek5000 is not purely compute bound and the cumulative memory bandwidth is saturated with N cores (N < total number of cores).</span><br style="font-size:12.8px"><br style="font-size:12.8px"><span style="font-size:12.8px">Cheers,</span><br style="font-size:12.8px"><br style="font-size:12.8px"><span style="font-size:12.8px">Stefan</span><br style="font-size:12.8px"><br><br style="font-size:12.8px"><span style="font-size:12.8px">From: <</span><a href="mailto:nek5000-users-bounces@lists.mcs.anl.gov" style="font-size:12.8px">nek5000-users-bounces@lists.<wbr>mcs.anl.gov</a><span style="font-size:12.8px">> on behalf of <</span><a href="mailto:nek5000-users@lists.mcs.anl.gov" style="font-size:12.8px">nek5000-users@lists.mcs.anl.<wbr>gov</a><span style="font-size:12.8px">></span><br style="font-size:12.8px"><span style="font-size:12.8px">Reply-To: <</span><a href="mailto:nek5000-users@lists.mcs.anl.gov" style="font-size:12.8px">nek5000-users@lists.mcs.anl.<wbr>gov</a><span style="font-size:12.8px">></span><br style="font-size:12.8px"><span style="font-size:12.8px">Date: Monday, December 5, 2016 at 6:12 AM</span><br style="font-size:12.8px"><span style="font-size:12.8px">To: <</span><a href="mailto:nek5000-users@lists.mcs.anl.gov" style="font-size:12.8px">nek5000-users@lists.mcs.anl.<wbr>gov</a><span style="font-size:12.8px">></span><br style="font-size:12.8px"><span style="font-size:12.8px">Subject: [Nek5000-users] Parallel speedup on supercomputer Tianhe-2</span><br style="font-size:12.8px"><br style="font-size:12.8px"><span style="font-size:12.8px">Dear Neks,</span><br style="font-size:12.8px"><br style="font-size:12.8px"><span style="font-size:12.8px">I'm using Nek5000 to simulate turbulent Rayleigh-Benard convection, which is governed by the coupled Navier-Stokes equations and convective heat equation. I'm running the code on a supercomputer, Tianhe-2, located in Guangzhou, China. Each computer node in Tianhe-2 has 24 cores (2 Xeon E5 12-core CPUs) and 64GB memory. I find the speedup curve is not linear on a single node. For example, a 24-task job is only 8 times faster than the serial one. However, the performance with an increasing number of nodes is quite good. I don't know whether there is any parameter in nek500 that I can change in order to improve the speedup performance of the individual nodes.</span><br style="font-size:12.8px"><br style="font-size:12.8px"><span style="font-size:12.8px">Thanks in advance!</span><br style="font-size:12.8px"><br style="font-size:12.8px"><span style="font-size:12.8px">Best regards,</span><br style="font-size:12.8px"><br style="font-size:12.8px"><span style="font-size:12.8px">Wei XU</span><br style="font-size:12.8px"><br style="font-size:12.8px"></div></div>
</div></blockquote><blockquote type="cite"><div><span>_______________________________________________</span><br><span>Nek5000-users mailing list</span><br><span><a href="mailto:Nek5000-users@lists.mcs.anl.gov">Nek5000-users@lists.mcs.anl.gov</a></span><br><span><a href="https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users">https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users</a></span><br></div></blockquote></body></html>