<html dir="ltr">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=gb2312">
<style type="text/css" id="owaParaStyle"></style>
</head>
<body fpstyle="1" ocsi="0">
<div style="direction: ltr;font-family: Tahoma;color: #000000;font-size: 10pt;">
<div><br>
</div>
<div>Hi Tom,</div>
<div><br>
</div>
<div>Josh is correct --- one other thing to keep in mind is that the memory layout, controlled by the product</div>
<div><br>
</div>
<div> lx1*ly1*lz1*lelt</div>
<div><br>
</div>
<div>is the _per_processor layout. So, if you need more memory for your problem, simply adding more</div>
<div>processors (or even processes, in some cases) will suffice to meet the memory demands.</div>
<div>Suppose for example that you wanted to run a 1 million point problem (which is of modest</div>
<div>size, btw). You could do this with</div>
<div><br>
</div>
<div> lx1=11,ly1=11,lz1=11, lelt=35 and use 32 processors.</div>
<div><br>
</div>
<div>Net result is an upper bound of the number of elements being 32 x 35, each element of order N=10.</div>
<div>(The lx1=11 comes from the fact that you need N+1 points to prescribe a polynomial of degree N.)</div>
<div><br>
</div>
<div>There is more explanation of the memory issues in the online manual --- I cut and paste some of</div>
<div>it below.</div>
<div><br>
</div>
<div>Paul</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div class="column">
<p><span style="font-size: 12.000000pt; font-family: 'CMR12'">Per-processor memory requirements for Nek5000 scale roughly as 400 8-byte words per allocated gridpoint. The number of
</span><span style="font-size: 12.000000pt; font-family: 'CMTI12'">allo- cated </span>
<span style="font-size: 12.000000pt; font-family: 'CMR12'">gridpoints per processor is
</span><span style="font-size: 12.000000pt; font-family: 'CMMI12'">nmax=lx1*ly1*lz1*lelt.
</span><span style="font-size: 12.000000pt; font-family: 'CMR12'">(For 3D, lz1=ly1=lx1; for 2D, lz1=1, ly1=lx1.) If required for a particular simulation, more memory may be made available by using addi- tional processors. For example, suppose one needed to
run a simu- lation with 6000 elements of order </span><span style="font-size: 12.000000pt; font-family: 'CMMI12'">N
</span><span style="font-size: 12.000000pt; font-family: 'CMR12'">= 9. To leading order, the total memory requirements would be
</span><span style="font-size: 12.000000pt; font-family: 'CMSY10'">¡Ö </span><span style="font-size: 12.000000pt; font-family: 'CMMI12'">E(N
</span><span style="font-size: 12.000000pt; font-family: 'CMR12'">+1)3points¡Á400(wds/pt)¡Á 8bytes/wd = 6000
</span><span style="font-size: 12.000000pt; font-family: 'CMSY10'">¡Á </span><span style="font-size: 12.000000pt; font-family: 'CMR12'">103
</span><span style="font-size: 12.000000pt; font-family: 'CMSY10'">¡Á </span><span style="font-size: 12.000000pt; font-family: 'CMR12'">400
</span><span style="font-size: 12.000000pt; font-family: 'CMSY10'">¡Á </span><span style="font-size: 12.000000pt; font-family: 'CMR12'">8 = 19.2 GB. Assuming there is 400 MB of memory per core available to the user (after account- ing for OS requirements), then
one could run this simulation with </span><span style="font-size: 12.000000pt; font-family: 'CMMI12'">P
</span><span style="font-size: 12.000000pt; font-family: 'CMSY10'">¡Ý </span><span style="font-size: 12.000000pt; font-family: 'CMR12'">19, 200MB/(400MB/proc) = 48 processors. To do so, it would be necessary to set lelt
</span><span style="font-size: 12.000000pt; font-family: 'CMSY10'">¡Ý </span><span style="font-size: 12.000000pt; font-family: 'CMR12'">6000/48 = 125.
</span></p>
<p><span style="font-size: 12.000000pt; font-family: 'CMR12'">We note two other parameters of interest in the parallel context:
</span><span style="font-size: 12.000000pt; font-family: 'CMBX12'">lp, </span><span style="font-size: 12.000000pt; font-family: 'CMR12'">the maximum number of processors that can be used.
</span></p>
<p><span style="font-size: 12.000000pt; font-family: 'CMBX12'">lelg, </span><span style="font-size: 12.000000pt; font-family: 'CMR12'">an upper bound on the number of elements in the simulation.
</span></p>
<p><span style="font-size: 12.000000pt; font-family: 'CMR12'">There is a slight memory penalty associated with these variables, so one generally does not want to have them excessively large. It is common, however, to have lp be as large as anticipated for a
given case so that the executable can be run without recompiling on any admissable number of processors (Pmem
</span><span style="font-size: 12.000000pt; font-family: 'CMSY10'">¡Ü </span><span style="font-size: 12.000000pt; font-family: 'CMMI12'">P
</span><span style="font-size: 12.000000pt; font-family: 'CMSY10'">¡Ü </span><span style="font-size: 12.000000pt; font-family: 'CMMI12'">E,
</span><span style="font-size: 12.000000pt; font-family: 'CMR12'">where </span><span style="font-size: 12.000000pt; font-family: 'CMMI12'">Pmem
</span><span style="font-size: 12.000000pt; font-family: 'CMR12'">is the value computed above).</span></p>
<p><span style="font-size: 12.000000pt; font-family: 'CMR12'"><br>
</span></p>
<p><span style="font-size: 12.000000pt; font-family: 'CMR12'"> </span></p>
</div>
<div style="font-family: Times New Roman; color: #000000; font-size: 16px">
<hr tabindex="-1">
<div id="divRpF756533" style="direction: ltr;"><font face="Tahoma" size="2" color="#000000"><b>From:</b> nek5000-users-bounces@lists.mcs.anl.gov [nek5000-users-bounces@lists.mcs.anl.gov] on behalf of nek5000-users@lists.mcs.anl.gov [nek5000-users@lists.mcs.anl.gov]<br>
<b>Sent:</b> Tuesday, February 04, 2014 5:51 PM<br>
<b>To:</b> nek5000-users<br>
<b>Subject:</b> Re: [Nek5000-users] relocation errors on small example?<br>
</font><br>
</div>
<div></div>
<div>
<div dir="ltr">Hi Tom,
<div><br>
</div>
<div><br>
</div>
<div>The variable lx1 in the SIZE file essentially specifies the number of grid points used for _each_ element in the "x" direction (ly1 for "y", lz1 for "z". Note here that by "x", "y", "z", we are really talking about the coordinates defined on the master
element, not in physical space). Thus, the total number of grid points you are reserving for each variable (u,v,w, t, etc) is lx1*ly1*lz1*lelt. So for your specification, you are actually attempting to allocate space for a very very large problem. Even
if you could allocate that amount of space (as you were able to by removing some variables while compiling), I believe NEK by default is only setup to handle lx1 ~ 30 or below.</div>
<div><br>
</div>
<div>I'm assuming you are wanting to have 192 points in each direction for your problem? You can either change the number of elements (contained in the .rea file) or vary lx1 in the SIZE file so that lx1*(number of element in each direction) ~= 192.</div>
<div><br>
</div>
<div>Hopefully this helps!</div>
<div><br>
</div>
<div>Josh</div>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Tue, Feb 4, 2014 at 5:14 PM, <span dir="ltr"><<a href="mailto:nek5000-users@lists.mcs.anl.gov" target="_blank">nek5000-users@lists.mcs.anl.gov</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex; border-left:1px #ccc solid; padding-left:1ex">
I am trying to set up an example problem with Nek5000. Unfortunately,<br>
I'm hitting the notorious:<br>
<br>
relocation truncated to fit: R_X86_64_PC32 against symbol<br>
<br>
error with it. I gather from past posts that this is an issue with<br>
grid/problem sizes. However, I'm just using eddy_uv at the moment, with<br>
minor modifications to bump the grid size up. Diff appended.<br>
<br>
By removing some (seemingly) unused common blocks for this problem I<br>
managed to get it to link with a 192x192x1 grid, but it aborts quickly,<br>
complaining that the problem size is too large. I feel like I must be<br>
doing something, because 192x192 of 32bit floats is tiny: 144K.<br>
<br>
Could anyone point me towards what I've messed up, here?<br>
<br>
Thanks,<br>
<br>
-tom<br>
<br>
<br>
_______________________________________________<br>
Nek5000-users mailing list<br>
<a href="mailto:Nek5000-users@lists.mcs.anl.gov" target="_blank">Nek5000-users@lists.mcs.anl.gov</a><br>
<a href="https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users" target="_blank">https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users</a><br>
<br>
</blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
-- <br>
Josh Camp<br>
<br>
"All that is necessary for the triumph of evil is that good men do nothing" -- Edmund Burke
</div>
</div>
</div>
</div>
</body>
</html>