<div dir="ltr">Thank you Dr. Fischer, I think it became so clear, maybe the problem for me happened since I first ran nek with 32 processors and then I went to run it on 1024! Which I found it little bit problematic on couple of cases that I have tested!<br><br>Thank you,<br>Ami<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Nov 11, 2014 at 8:25 AM, <span dir="ltr"><<a href="mailto:nek5000-users@lists.mcs.anl.gov" target="_blank">nek5000-users@lists.mcs.anl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<div style="direction:ltr;font-family:Tahoma;color:#000000;font-size:10pt">
<div><br>
</div>
Hi Ami,
<div><br>
</div>
<div>Great to hear the problem is fixed!</div>
<div><br>
</div>
<div>In principle, it should suffice to simply</div>
<div><br>
</div>
<div>makenek blah</div>
<div><br>
</div>
<div>after editing the SIZE file, since make is smart enough to know that you've touched SIZE.</div>
<div><br>
</div>
<div>Normally I don't recompile when changing processor count (unless it's by a huge factor).</div>
<div><br>
</div>
<div>So, suppose my job has 4000 elements and I'm running on anywhere from 16 to 2048</div>
<div>processors. It's sufficient to set lelt=(4000/16) and lp=2048. Then you can run on that</div>
<div>range P=16 to 2048 without recompile.</div>
<div><br>
</div>
<div>Paul</div>
<div><br>
</div>
<div>
<div style="font-family:Times New Roman;color:#000000;font-size:16px"><br>
<hr>
<div style="direction:ltr"><font face="Tahoma" color="#000000"><span class=""><b>From:</b> <a href="mailto:nek5000-users-bounces@lists.mcs.anl.gov" target="_blank">nek5000-users-bounces@lists.mcs.anl.gov</a> [<a href="mailto:nek5000-users-bounces@lists.mcs.anl.gov" target="_blank">nek5000-users-bounces@lists.mcs.anl.gov</a>] on behalf of <a href="mailto:nek5000-users@lists.mcs.anl.gov" target="_blank">nek5000-users@lists.mcs.anl.gov</a> [<a href="mailto:nek5000-users@lists.mcs.anl.gov" target="_blank">nek5000-users@lists.mcs.anl.gov</a>]<br>
</span><b>Sent:</b> Tuesday, November 11, 2014 3:23 AM<br>
<b>To:</b> <a href="mailto:nek5000-users@lists.mcs.anl.gov" target="_blank">nek5000-users@lists.mcs.anl.gov</a><br>
<b>Subject:</b> Re: [Nek5000-users] problem with running nek5000 on supercomputers<br>
</font><br>
</div><div><div class="h5">
<div></div>
<div>
<div dir="ltr">
<div>It seems that for each time I had to clean compiled files before changing the processor count and run it again! It works now!<br>
<br>
</div>
Thanks,<br>
Ami<br>
<div>
<div>
<div>
<div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Mon, Nov 10, 2014 at 4:29 PM, Amirreza Hashemi <span dir="ltr">
<<a href="mailto:amirezahashemi@gmail.com" target="_blank">amirezahashemi@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div>
<div>
<div>Dr. Fishcer,<br>
<br>
</div>
Thanks for your answer, for sure, I change the lelt and lelg and keep them as lowest even number that I can pick when I increase the number of CPUs. I picked also lp=32768, but I still get this "allocation failed" error. Is there any thing else that effect
on the allocation which I have to change it?!<br>
<br>
</div>
Thanks,<br>
</div>
Ami<br>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div class="gmail_extra"><br>
<div class="gmail_quote">
<div>
<div>On Mon, Nov 10, 2014 at 8:55 AM, <span dir="ltr"><<a href="mailto:nek5000-users@lists.mcs.anl.gov" target="_blank">nek5000-users@lists.mcs.anl.gov</a>></span> wrote:<br>
</div>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<div>
<div>
<div style="direction:ltr;font-family:Tahoma;color:#000000;font-size:10pt">
<div><br>
</div>
Hi Ami,
<div><br>
</div>
<div>When you increase the processor count, you can recompile with a smaller lelt.</div>
<div><br>
</div>
<div>Suppose you have lelt=1000 and you go from P=64 to P=128. Then set lelt=500 and</div>
<div>recompile.</div>
<div><br>
</div>
<div>Make certain lelg >= 500*P and lp >= 128.</div>
<div><br>
</div>
<div>I would suggest lelg="big enough" (and an even number) and lp=32768.</div>
<div><br>
</div>
<div>Paul</div>
<div><br>
<div style="font-family:Times New Roman;color:#000000;font-size:16px">
<hr>
<div style="direction:ltr"><font face="Tahoma" color="#000000"><b>From:</b> <a href="mailto:nek5000-users-bounces@lists.mcs.anl.gov" target="_blank">
nek5000-users-bounces@lists.mcs.anl.gov</a> [<a href="mailto:nek5000-users-bounces@lists.mcs.anl.gov" target="_blank">nek5000-users-bounces@lists.mcs.anl.gov</a>] on behalf of
<a href="mailto:nek5000-users@lists.mcs.anl.gov" target="_blank">nek5000-users@lists.mcs.anl.gov</a> [<a href="mailto:nek5000-users@lists.mcs.anl.gov" target="_blank">nek5000-users@lists.mcs.anl.gov</a>]<br>
<b>Sent:</b> Monday, November 10, 2014 5:05 AM<br>
<b>To:</b> <a href="mailto:nek5000-users@lists.mcs.anl.gov" target="_blank">nek5000-users@lists.mcs.anl.gov</a><br>
<b>Subject:</b> [Nek5000-users] problem with running nek5000 on supercomputers<br>
</font><br>
</div>
<div>
<div>
<div></div>
<div>
<div dir="ltr">
<div>
<div>
<div>Hi Neks,<br>
<br>
</div>
I am trying to run my simulation with nek5000 on supercomputers, the size of the problem is something like:
<pre> text data bss dec hex filename
2233955 187412 2554984280 2557405647 986ee9cf nek5000</pre>
</div>
I don't have any problem when I use 32~64 CPUs to run parallel on supercomputers, but once I go for more than 64 CPUs, I always get "allocation failed" error. I have tried to run on different supercomputer like as Gordon SDSC, OSC, TACC, but I got the same
type of error each time. I have checked the active memories, it seems for each of them memories have large unused portion. Does any one have any idea why when I increase the number of CPUs I get this allocation error?!<br>
</div>
<div><br>
</div>
Thank you,<br>
Ami<br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
</div>
</div>
_______________________________________________<br>
Nek5000-users mailing list<br>
<a href="mailto:Nek5000-users@lists.mcs.anl.gov" target="_blank">Nek5000-users@lists.mcs.anl.gov</a><br>
<a href="https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users" target="_blank">https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users</a><br>
<br>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div></div></div>
</div>
</div>
</div>
<br>_______________________________________________<br>
Nek5000-users mailing list<br>
<a href="mailto:Nek5000-users@lists.mcs.anl.gov">Nek5000-users@lists.mcs.anl.gov</a><br>
<a href="https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users" target="_blank">https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users</a><br>
<br></blockquote></div><br></div>